Optical design and optimization techniques for 3D bright-field displays

Abstract

A method and framework for designing and optimizing high-performance bright-field displays, such as integral imaging head-mounted displays, is disclosed. The disclosed techniques enable user-defined metrics for characterizing the performance and optimization of such systems. One design method relates to a three-dimensional (3D) display device including an array optical system, an array display device for generating multiple elemental images, a first reference plane representing a virtual central depth plane (CDP), a second reference plane representing a viewing window for viewing a reconstructed 3D scene, and an optical subsection representing a model of the human eye. The method includes tracing a ray of light originating at the array display device through the array optical system to the optical subsection and adjusting one or more parameters to obtain at least a first metric value within a predetermined value.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 885,460, entitled "Optical Design and Optimization Techniques for 3D Light Field Displays," filed August 12, 2019, the entire contents of which are incorporated by reference into the present disclosure.

[0002] (Language regarding federally sponsored research) This invention was made with Government support under Grant No. 1422653 awarded by the NSF. The Government has certain rights in this invention.

[0003] (Technical field) The disclosed technology relates generally to three-dimensional (3D) displays, and more specifically to a method and framework for designing and optimizing high performance bright-field displays. [Background technology]

[0004] Conventional stereoscopic three-dimensional displays (S3D) stimulate the perception of 3D space and shape from a pair of two-dimensional (2D) perspective images (one for each eye) at a fixed distance, using binocular disparity and other image depth cues of a 3D scene viewed from two slightly different viewing positions. An important limitation associated with S3D type displays is the well-known Vergence / Accommodation Collision (VAC) problem. In binocular vision, when observing an object, the eyes must rotate about a horizontal axis so that the projection of the image is centered on the retinas of both eyes. Vergence is the simultaneous movement of both eyes in opposite directions to obtain or maintain single binocular vision. Accommodation is the process by which the eyes change their refractive power to maintain a clear image or to focus on an object as its distance varies. Under normal viewing conditions of an object, changing the focus of the eyes to view an object at a different distance will automatically cause vergence and accommodation. In the context of 3D displays, VAC occurs when the brain receives mismatched cues between the distance of a virtual 3D object (vergence) and the focal distance at which the eyes are required to focus on that object (accommodation). The problem results from an inability to render correct focus cues for a 3D scene (including accommodation and retinal blur effects). It causes a cue conflict, which is considered one of the important causes of various visual artifacts associated with viewing S3D displays. Summary of the Invention [Means for solving the problem]

[0005] The disclosed embodiments relate to three-dimensional (3D) displays, and more particularly, to methods and frameworks for the design and optimization of high performance bright-field displays, including, but not limited to, bright-field head-mounted displays.

[0006] In one exemplary embodiment, a method for designing an integral imaging (InI)-based three-dimensional (3D) display system. The system includes an array optical system, an array display device capable of generating a plurality of element images, a first reference plane representing a virtual central depth plane (CDP) where light rays emitted by a point source on the display converge to form an image point, a second reference plane representing a viewing window for viewing the reconstructed 3D scene, and an optical subdivision representing a model of the human eye. The method for designing the system includes tracking a set of light rays associated with a bright field in the InI-based 3D system, the tracking starting at the array display device, through the array optical system, and to an optical subdivision for each element of the array display device and the array optical system. The method further includes adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values. The first metric value corresponds to a light ray direction sampling of the bright field.

[0007] In the disclosed InI-based 3D system, the bright field reconstructs a 4D bright field of the 3D scene by angularly sampling the directions of the rays apparently emitted by the 3D scene. According to some embodiments, the optical design process involves optimizing the mapping of both ray positions and ray directions in the 4D bright field rendering, rather than simply optimizing the 2D mapping between the object-image conjugate planes in traditional HMD designs. The present invention provides, for example, the following: (Item 1) 1. A method for designing an integral imaging (InI) based three-dimensional (3D) display system, the method comprising: tracing a set of rays associated with a bright field in the InI-based 3D system, the system comprising: An array optical system; an array display device capable of generating a plurality of elemental images; a first reference plane representing a virtual central depth plane (CDP), where light rays emitted by a point source on the display converge to form an image point on the CDP; a second reference plane representing a viewing window for viewing the reconstructed 3D scene; An optical subdivision representing a model of the human eye; Including, the tracking is performed starting at the array display device, through the array optics, to the optical subsections for each element of the array display device and the array optics; adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values; Including, The method of claim 1, wherein the first metric value corresponds to a ray direction sampling of the bright field. (Item 2) 2. The method of claim 1, wherein the first metric value quantifies a deformation of a ray footprint of a given ray bundle of the bright field from its paraxial footprint. (Item 3) 3. The method of claim 2, wherein the first metric value is determined according to a relative ratio of an average deviation distance between an actual position and a theoretical position of a marginal ray on the second reference plane to a diagonal width of the paraxial footprint. (Item 4) Item 2. The method of item 1, wherein the first metric value is determined based on a difference between actual positions of marginal rays on the viewing window obtained by ray tracing and their corresponding paraxial positions on the viewing window. (Item 5) 2. The method of claim 1, wherein adjusting the one or more parameters associated with the InI-based 3D system is performed to further obtain a second metric value within another predetermined value or range of values, the second metric value corresponding to a ray position sampling of the bright field that takes into account at least deformations induced by neighboring elements of the array optical system. (Item 6) 6. The method according to claim 5, wherein the second metric value is determined according to an angular deviation between an actual position of a chief ray of a central object field and a theoretical position thereof measured from the second reference plane. (Item 7) Item 6. The method of item 5, wherein the second metric value represents a global distortion measure. (Item 8) 6. The method according to item 5, wherein the second metric value is calculated as a deviation of a central position of a virtual element image of the plurality of element images on the virtual CDP from its paraxial position. (Item 9) 6. The method of claim 5, wherein adjusting the one or more parameters associated with the InI-based 3D system is performed on the ray position sampling of the bright field to further individually optimize imaging of each element image. (Item 10) 2. The method of claim 1, wherein the InI-based 3D system further includes an eyepiece positioned between the array optical system and the second reference plane, and tracking the set of rays includes tracking the set of rays through the eyepiece. (Item 11) 2. The method of claim 1, wherein the array display device is a microdisplay device. (Item 12) Item 2. The method of item 1, wherein the array optical system comprises one or more lenslet arrays, each of which includes a plurality of microlenses. (Item 13) 2. The method according to claim 1, wherein the InI-based 3D system is an InI-based head mounted display (InI-based HMD) system. (Item 14) 6. The method of claim 1 or 5, wherein the predetermined value or range of values ​​for one or both of the first or second metrics is selected to achieve a particular image quality. (Item 15) 6. The method of claim 1 or 5, wherein the predetermined value or range of values ​​for one or both of the first or second metrics represents a maximum or minimum value that provides an optimal design criteria for the first or second metric. (Item 16) 1. A method of improving the design of an integral imaging optical system that includes a lenslet array that angularly samples directions of a bright field to produce an array of two-dimensional elemental images (EIs), each of which represents a different viewpoint in three dimensions (3D), the method comprising: determining a first metric corresponding to a ray direction sampling of the bright field; determining a second metric corresponding to a ray position sampling of the bright field; and performing a ray tracing operation to determine a design for the integral imaging optical system based on the first and second metrics; A method comprising: (Item 17) Item 17. The method of item 16, wherein the ray tracing operation is performed based on one or more constraints including maintaining the first or second metrics within corresponding values ​​or ranges of values. (Item 18) Item 17. The method of item 16, wherein the integral imaging optical system comprises an InI-based head mounted display (InI-based HMD). (Item 19) A device, the device comprising: A processor; a memory having processor executable code; Equipped with The code, when executed by the processor, 1. Tracing a set of rays associated with a bright field in an InI based 3D system, the system comprising: An array optical system; an array display device capable of generating a plurality of elemental images; a first reference plane representing a virtual central depth plane (CDP), where light rays emitted by a point source on the display converge to form an image point on the CDP; a second reference plane representing a viewing window for viewing the reconstructed 3D scene; An optical subdivision representing a model of the human eye; Including, the tracking is performed starting at the array display device, through the array optics, to the optical subsections for each element of the array display device and the array optics; adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values; causing said processor to perform The first metric value corresponds to a ray direction sampling of the bright field. (Item 20) A non-transitory computer readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to perform a method for designing an integral imaging (InI) based three-dimensional (3D) display system, comprising: The instruction: 11. A program code for tracing a set of rays associated with a bright field in the InI based 3D system, the system comprising: An array optical system; an array display device capable of generating a plurality of elemental images; a first reference plane representing a virtual central depth plane (CDP), where light rays emitted by a point source on the display converge to form an image point on the CDP; a second reference plane representing a viewing window for viewing the reconstructed 3D scene; An optical subdivision representing a model of the human eye; Including, program code for performing said tracking starting at said array display device, through said array optics, to said optical subsections for each element of said array display device and said array optics; program code for adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values; Equipped with The first metric value corresponds to a ray direction sampling of the bright field. [Brief description of the drawings]

[0008] [Figure 1A]FIG. 1A is an example of a conventional head-mounted display (HMD) configuration that renders two-dimensional (2D) images on a microdisplay to form a sharply focused retinal image.

[0009] [Figure 1B] FIG. 1B illustrates a conventional HMD configuration as shown in FIG. 1A when accommodation depth is displaced from the distance of the virtual display.

[0010] [Figure 2A] FIG. 2A is an example configuration illustrating the image formation process of an integral imaging-based HMD (InI-HMD) for bright-field reconstruction for a first accommodation depth.

[0011] [Figure 2B] FIG. 2B illustrates the configuration of FIG. 2A and the image formation corresponding to a second accommodation depth.

[0012] [Figure 3A] FIG. 3A illustrates the mapping of brightfield functions and associated annotations for an integral imaging-based HMD (InI-HMD) configuration.

[0013] [Figure 3B] FIG. 3B illustrates the optical principle of reconstructing a 3D scene in an InI-based bright-field 3D display.

[0014] [Figure 4A] FIG. 4A illustrates a three-dimensional view of various aspects of the InI-HMD system, including the footprint on the viewing window and the imaged aperture of the lenslets of the microlens array (MLA).

[0015] [Figure 4B] FIG. 4B illustrates the division of elemental images (EI) on a microdisplay associated with an InI-HMD system.

[0016] [Figure 4C] FIG. 4C illustrates a subsystem including the EI, the lenslets of the MLA, and the shared ocular group.

[0017] [Figure 4D] FIG. 4D illustrates partially overlapping virtual images of four EIs on a virtual central depth plane (CDP).

[0018] [Figure 4E] FIG. 4E illustrates a partially overlapping virtual image of an EI on a CDP as illustrated in FIG. 4D with certain features removed for enhanced clarity.

[0019] [Diagram 5] FIG. 5 illustrates the effect of the global disparity metric on the global distortion of ray positions against a full-field distortion grid for an exemplary InI-HMD.

[0020] [Figure 6] FIG. 6 illustrates the effect of deformation of the ray footprint of a given ray bundle from its paraxial shape for an exemplary InI-HMD system.

[0021] [Figure 7-1] 7A illustrates an example layout of a binocular InI-HMD system designed in accordance with the disclosed technology, and FIG. 7B illustrates further details regarding the optical layout of the sections of FIG. 7A.

[0022] [Figure 7-2] FIG. 7C shows the integrated display path design configuration in CodeV® plotted with actual ray tracing from a portion of the sampled EI and lenslets.

[0023] [Figure 8A] FIG. 8A illustrates a plot of image contrast for a display path associated with an exemplary InI-HMD system.

[0024] [Figure 8B] FIG. 8B illustrates a modulation transfer function (MTF) plot for on-axis field points corresponding to a portion of sampled lenslets covering the field of view of an exemplary InI-HMD system.

[0025] [Figure 8C] FIG. 8C illustrates MTF plots for on-axis field points of each EI whose virtual CDPs correspond to central lenslets sampled 3 diopters to 0 diopters away from the viewing window of an exemplary InI-HMD system.

[0026] [Figure 9-1] FIG. 9A illustrates a plot of a distortion grid of a display path covering the entire field of view for an exemplary InI-HMD system.

[0027] [Figure 9-2] 9B illustrates a footprint diagram at the viewing window of an exemplary InI-HMD system before and after optimization of pupil aberrations. FIG. 9C illustrates a footprint diagram at the viewing window of an exemplary InI-HMD system after optimization of pupil aberrations.

[0028] [Figure 10] FIG. 10 illustrates a set of operations for designing an InI-based 3D display system, according to an example embodiment.

[0029] [Figure 11] FIG. 11 illustrates a set of operations for improving the design of an integral imaging optical system, according to an example embodiment.

[0030] [Figure 12] FIG. 12 illustrates a block diagram of a device that can be used to implement certain aspects of the disclosed technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] There are several efforts to address the vergence / accommodation collision (VAC) problem, among which is integral imaging-based (InI-based) bright-field 3D (LF-3D) displays. Integral imaging generally refers to a three-dimensional imaging technique that captures and reproduces the bright field by using a two-dimensional array of microlenses or optical apertures. This configuration allows the reconstruction of a 3D scene by rendering directional light rays apparently emitted by the scene through an array optical system seen through a pre-designed viewing window. However, a systematic design approach suitable for the optimization of 3D display systems, including InI-based LF-3D head-mounted displays (HMDs), is lacking. Therefore, designing a high-performance 3D display system remains a challenge. In particular, without a well-optimized design, an InI-HMD will not be able to correctly render the depth and accommodation cues of the reconstructed 3D scene, resulting in an image of impaired quality and comfort.

[0032] In the following description, the InI-HMD is used as an exemplary system to illustrate the above-mentioned problems and the disclosed solutions. However, it should be understood that the disclosed embodiments are equally applicable to other types of 3D display systems, such as non-head-mounted direct-view type brightfield displays in which the optical system for rendering the brightfield is not directly mounted on the viewer's head, 3D brightfield displays in one angular direction (typically sampling only the brightfield in the horizontal direction) (more commonly known as displays that render only horizontal parallax, i.e., the multiple views are arranged as vertical stripes on the viewing window), or super multi-view displays or autostereoscopic displays in which the component views are generated by an array of projectors or imaging units.

[0033] In systems that utilize integral imaging, a 3D image is typically displayed by placing a microlens array (MLA) in front of the image, with each lenslet of the MLA appearing differently depending on the viewing angle. InI-HMD systems require that different elemental views created by multiple elements of the MLA are rendered and viewed through each of the eye's pupils. Thus, the light rays emitted by multiple spatially separated pixels on these elemental views are received by the eye's pupil and summed together to form the perception of a 3D reconstructed point, which is essentially a key difference between InI-HMDs and conventional HMDs. In conventional HMD systems, the viewing optics simply projects a 2D image on the microdisplay onto a 2D virtual display, and thus the light rays from a single pixel are imaged together by the eye's optics to form the perception of a 2D point. Due at least to this inherent difference in the image formation process, existing optical design methods for conventional HMDs are inadequate for designing true 3D LF-HMD systems that require light ray integration from multiple individual sources.

[0034] The disclosed embodiments provide, among other features and benefits, an improved optical design methodology that allows (1) generating LF-3D HMD designs for precisely performing real ray tracing, and (2) optimizing the design for precisely sampling and rendering the bright field of the reconstructed 3D scene, which facilitates the accommodation status of the viewer's eyes and is therefore important for solving the VAC problem. In embodiments of the disclosed technology, one or more new design constraints or metrics are established that facilitate optimization of bright field ray position sampling and / or bright field ray direction sampling. For example, one constraint or metric for position sampling considers global distortions (e.g., aberrations) associated with the lateral position of the virtual elemental images (EI) relative to the entire aberrated FOV of the reconstructed 3D scene. Another constraint or metric for direction sampling provides a measure of deviation or deformation of the ray footprints from their paraxial shape.

[0035] Use of the disclosed constraints and metrics improves the optical design process, allowing designers to assess the quality of the generated image (e.g., with respect to solving the VAC problem) and improve the design of the optical system. In some embodiments, optimal designs may be generated by minimizing the disclosed metrics during the design process. The disclosed metrics further provide an assessment of the achievable image quality, and thus, in some embodiments, desired image quality goals may be achieved for a particular optical system based on (as opposed to minimizing) target values ​​of the disclosed metrics.

[0036] 1A and 1B illustrate an exemplary conventional HMD configuration that renders a 2D image on a microdisplay. As with other 2D displays, each pixel value represents the sum of the intensity or radiance of light rays over a range of angles. An eyepiece inserted between the microdisplay (labeled "display" in FIGS. 1A and 1B) and the eye simply magnifies the 2D image, forming a magnified 2D virtual display at a distance optically conjugate to the microdisplay plane through the eyepiece. Thus, all of the light rays from a single pixel are imaged together by the eye's optics to form the perception of a 2D point. In FIGS. 1A and 1B, the "red" and "green" labels are used to help illustrate the different bundles of light rays propagating through the system. As shown in FIG. 1A, when the accommodation depth of the eye's optics matches the apparent distance of the virtual display, a sharply focused retinal image is formed. Otherwise, as shown in FIG. 1B, when the accommodation depth is displaced from the distance of the virtual display, an equally blurred retinal image is formed.

[0037] Due to the nature of 2D image formation described above, the optical design process for such systems only needs to focus on the 2D mapping between pixels on the microdisplay and their corresponding images on the virtual display, and optimization strategies are centered on controlling optical aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the virtual display. For this, light rays from every single pixel on the display are imaged by a common optical path or series of optical elements. Thus, conventional HMD systems can be modeled by a shared optical configuration. In such systems, the retinal image of a rendered point is the projection of light rays emitted by a single pixel on the microdisplay or the augmented virtual display, allowing the optical performance of conventional 2D HMD systems to be properly evaluated by characterizing the 2D image patterns projected by light rays from a small number of viewing positions on the microdisplay.

[0038] In contrast, LF-HMD reconstructs a 4D bright field of a 3D scene by angularly sampling the direction of the light rays apparently emitted by the 3D scene. Figures 2A and 2B are exemplary configurations illustrating the image formation process of an integral imaging-based HMD (InI-HMD) for bright field reconstruction. Figures 3A and 3B illustrate a three-dimensional view of the same system with further annotations to facilitate understanding of the bright field function. As illustrated in Figure 2A, the system includes a microdisplay, an array optical system (e.g., a lenslet array), and an eyepiece. In Figure 2A, the element images (EI) of the 2D array represent different viewpoints of the 3D scene rendered on the microdisplay (e.g., elements A1-A3 are used for reconstructed point A, and elements B1-B3 are used for reconstructed point B). There are two unique and important conceptual reference planes in InI-HMDs that are widely recognized. The first plane is the virtual central depth plane (CDP) where the light rays emitted by a point source on the microdisplay converge to form an image point after propagating through the MLA and the eyepiece (see also FIG. 3A). It is considered as a reference plane in visual space optically conjugate to the microdisplay. The second reference plane is the viewing window that defines the area where the viewer observes the reconstructed 3D scene. By design, it coincides with the entrance pupil plane of the eye's optics and is commonly known as the system's exit pupil or eyebox in conventional HMDs.

[0039] Each pixel on these EIs is considered as an image source that defines the positional information (s,t) of a 4D bright-field function. An array optical system such as an MLA is associated with an array of EIs, each of which defines the directional information (u,v) of the bright-field function (see also FIG. 3A, which illustrates a 4D bright-field with parameters s, t, u, v). Each 2D EI is imaged as a separate imaging path by its corresponding microlenslet. To reconstruct the bright-field of a 3D point, the ray bundles emitted by multiple pixels (each on different EIs) are modulated by their corresponding MLA elements and intersect at the 3D location of the reconstruction. As a result, the rays from these spatially separated pixels collectively create a 3D point, which appears to emit light in different directions. By properly arranging the EIs on the display panel, a 4D bright-field of a 3D scene with points located at different viewing angles and depths (e.g., points A and B in FIGS. 2A and 2B) can thus be rendered correspondingly.

[0040] In the magnified view configuration, an eyepiece is inserted to further magnify the miniature 3D scene into a large 3D volume with extended depth in the virtual space (e.g., A' and B' in Figs. 2A and 2B are magnified renderings of reconstructed points A and B, respectively). In a true LF-3D HMD, multiple elemental views from a 3D point are projected to different locations on the entrance pupil of the eye, and the retinal images of these elemental views collectively form the perception of a 3D point in space. As illustrated in Fig. 2A, when the eye is accommodated at the depth of the reconstructed point A', the light rays from its corresponding elemental pixels (e.g., pixels A1-A3) will overlap each other and naturally form a sharply focused image on the retina, while the light rays from pixels reconstructing point B' located at a different depth from point A (e.g., light rays from pixels B1-B3) will be spatially displaced from each other and create retinal blur. The apparent amount of retinal blur at point B varies with the difference between the reconstruction depth and the eye's accommodation. As illustrated in FIG. 2B, when the eye's accommodation depth is switched to point B', the retinal image at point B' becomes in focus, while the retinal image at point A' becomes blurred. Under such circumstances, the retinal image of the reconstructed 3D scene by the InI-HMD will approximate the visual effect of viewing a natural 3D scene.

[0041] Due to the nature of 3D image formation described above, the bright-field reconstruction of a 3D point is the integral effect of rays emitted by multiple spatially separated pixels, each of which is located on a different element image and imaged by a different optical unit of the array optical system. Each pixel provides a sample of the bright-field position, and its corresponding unit of the imaging optical system provides a sample of the bright-field direction. Thus, the optical design process requires optimizing the mapping of both ray positions and directions in 4D bright-field rendering, rather than simply optimizing the 2D mapping between the object-image conjugate plane in traditional HMD design. The optimization strategy not only needs to properly control and evaluate optical aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the 2D element images (taking into account the ray sampling aspects of the bright field), but also requires methods and metrics to control and evaluate optical aberrations that reduce the accuracy of the directional ray sampling.

[0042] As illustrated in FIG. 3A, pixels on different element images are imaged by different optical paths through different optical elements. Therefore, the LF-HMD system needs to be modeled by a multi-configuration array system or an array of subsystems distributed horizontally and vertically. Each of the subsystems represents one single imaging path from a given element image through its corresponding optical system unit and possibly an eyepiece group (if present). To clearly distinguish the shape of the EI from the lenslets, for illustration purposes, it is noted that a circular shape is used to represent the apertures of the lenslets and their corresponding footprints on the viewing window, while a square aperture is utilized for the lenslets in the examples provided herein.

[0043] The retinal image of a rendered 3D point is the integral sum of projected rays emitted by multiple pixels on different elemental images, and the appearance of the image varies significantly with the state of accommodation of the eye. Therefore, the optical performance of an LF-HMD system cannot be properly evaluated by only characterizing the 2D image pattern projected by rays from a few viewing positions on the microdisplay, but needs to be evaluated by characterizing the integral images on the retina for different states of accommodation of the eye. For this purpose, an eye model is a necessary part of the imaging system.

[0044] FIG. 3A further illustrates the simplified process of light-field rendering in visual space, where the ray positions of the light-field function correspond to projected virtual pixels (x c ,y c ), and the ray direction is the projection coordinate (x v ,y v Theoretically, the ideal light field function L'(x c ,y c ,x v ,y v), the corresponding EIs reconstructing the bright field points will be imaged as point sources on the virtual CDP, and their ray bundles will completely intersect at the lateral and longitudinal positions of the points and be projected in their corresponding viewing areas on the viewing window, as illustrated in FIG. 3B. Note that unlike conventional HMDs, the ray bundles emitted by the element pixels reconstructing the 3D points will be projected onto different locations on the viewing window, as illustrated in FIG. 3B. However, any deviation in rendering the bright field function due to imperfections in the imaging process of the InI-HMD will lead to impaired EIs on the virtual CDP and / or incorrect directions and footprints of the ray bundles projected onto the viewing window. As a result, such deviations lead to impaired reconstruction of the 3D scene and reduced perceived image quality of the InI-HMD by the viewer. Therefore, in designing an InI-HMD system, the light field L(s,t,u,v) physically rendered by the display panel and MLA in object space is transformed into the light field L'(x c ,y c ,x v ,y v ) must be mapped exactly to

[0045] To achieve a good mapping, (1) the bright field function (s,t) is converted to (x c ,y c ), and (2) a good control of the position sampling mapping from (u,v) of the bright-field function to (x v ,y v It is important to obtain good control of the directional sampling mapping to

[0046] In particular, from (s,t) to (x c ,y cIn optimizing the ray position mapping from (u,v) to (x v ,y v ) requires an entirely new design metric capable of precisely assessing the quality of the directional sampling of the reconstructed light field of a 3D scene and its effect on the display system.

[0047] 4A-4D illustrate three-dimensional views of various aspects of the InI-HMD system, including the footprint on the viewing window and the imaged aperture of the lenslets of the MLA (real exit pupil), and the division of the EI on the microdisplay. It should be noted that in some of the InI-HMD systems, additional optical elements that can further improve the overall display performance, such as adjustable relay groups, can be added in the display path, which adds additional complexity in designing the InI-HMD, but can still follow the same design methodology as a system consisting of only an eyepiece. Without loss of generality, in some cases, these added optical elements can be combined with the eyepiece, and are referred to herein as an "eyepiece group."

[0048] FIG. 4A illustrates the footprint on the viewing window and the imaged aperture of the lenslets of the MLA (real exit pupil). The LF-HMD system can be divided into M×N subsystems, where M and N are the total number of optical units in the array optics or the number of EIs rendered on the microdisplay in the horizontal and vertical directions, respectively. FIG. 4B illustrates the division of the EIs on the microdisplay. FIG. 4C illustrates a subsystem including an EI, a lenslet of the MLA, and a shared eyepiece group. Each of the subsystems represents one single imaging path from the EI through its corresponding optical unit of the array optics and the shared eyepiece group. The imaging path of each subsystem is off-axis non-rotationally symmetric with respect to the main optical axis, as illustrated in FIG. 4C. Each of the subsystems can be configured as a zoom configuration in the optical design software. FIG. 4D illustrates the partially overlapping virtual images of the EIs on the virtual CDP.

[0049] In conventional 2D HMD designs, the HMD optical system is generally configured to trace rays backwards from a shared exit pupil (or eye entrance pupil) towards the microdisplay, and no eye model is required. In contrast, the subsystem according to the disclosed embodiment is configured such that ray tracing starts from the microdisplay or from the EI towards the viewing window. In this way, ray tracing failures are avoided due to the fact that the projection of the array of apertures of the optical unit of the array optical system on the viewing window does not form a commonly shared exit pupil as in conventional HMDs. In addition, an ideal lens or an established eye model (e.g., the Arizona eye model) that emulates the optical system of the viewer's eye is inserted with its entrance pupil coinciding with the viewing window for better optimization convergence and convenient assessment of the retinal image of the bright field reconstruction. It should be noted that the use of a standard eye model is one non-limiting example that allows the design of a 3D system for mass-produced products. In some embodiments, individualized or customized eye models may be used.

[0050] 4A-4D, the zoom configurations between the subsystems differ from each other primarily by the surface geometry and lateral position of the corresponding optics units, as well as the lateral position of the corresponding EI relative to the optical axis of the eyepiece group. In one embodiment, all lenslets in the MLA have the same surface geometry and are arranged in a rectangular array with equal lens pitch in both the horizontal and vertical directions. For this embodiment, the lateral position (u,v) of each lenslet is simply the displacement Δp between neighboring lenslets. MLA , i.e., lens pitch p MLA According to the same coordinate system as shown in FIG. 3A, for a given zoom configuration indexed as (m,n), the lateral coordinates of its corresponding lenslet can be expressed as:

number

[0051] The microdisplay is divided into an M×N array of EIs, one for each lenslet, but the lateral location and size of each EI is more complicated and depends on some other specifications of the display system. For example, the viewing window (see, for example, FIG. 4A, where X indicates a non-conjugate relationship), which is not necessarily an optical conjugate of the MLA but can be shifted vertically along the optical axis according to the design requirements, plays an important role in dividing and structuring the whole system into a series of subsystems. An array of imaged apertures is actually formed in the visual space, which can be considered as an exit pupil in the sense of a conventional display or imaging system. However, placing the viewer's eye at this exit pupil location would indicate that the viewer cannot see all the EIs at the same time, since the imaged apertures are spatially separated. To ensure that all of the EIs can be properly seen at the viewing window, the microdisplay ensures that the central chief ray of each EI through the entire display optics, including the corresponding lenslet of the MLA and the shared eyepiece group, is aligned with the center O of the viewing window. v4B ) (e.g., the solid line in FIG. 4B starting at the display and converging at the viewing window). Under such constraints, the displacement Δp between adjacent EIs EI and the size of the element image on the microdisplay, p EI can equally be written as:

number

[0052] In equation (2), g is the gap between the display panel and the MLA, l is the gap between the MLA and the intermediate CDP, and z IC is the gap between the intermediate CDP and the eyepiece group, and z' xp is introduced to refer to the distance between the eyepiece group and the viewing window imaged by the eyepiece group, which can be further given as:

number

[0053] In equation (3), f ep is the equivalent focal length of the eyepiece group. Therefore, for a given subsystem unit indexed as (m,n), the lateral coordinate of the center of the corresponding EI can be expressed as:

number

[0054] Footprint size d of the light beam from a pixel of EI projected through the entire optical system onto the viewing window v The view density (which determines the total number of views enclosed by the eye pupil) can be determined by tracing the ray bundle emitted by the center point of EI (e.g., the shaded ray bundle in FIG. 4C) and the overall size of the viewing window D vcan be determined by tracing the chief ray of the edge point of EI (e.g., a line starting at the edge of EI in FIG. 4C) throughout the display optics. The beam footprint diameter and overall viewing window size can be expressed as:

number

number

[0055] In equations (5) and (6), |···| denotes the absolute value symbol.

[0056] According to paraxial geometry, both the footprint diameter and the viewing window size are the same for any of the subsystems, and footprints corresponding to the same field object in different subsystems have the same coordinates (x v ,y v ) will intersect on the viewing window. For example, as mentioned above, the central chief ray of each EI is x v0 (m,n) and y v0 The optical axis will intersect at the center of the viewing window such that both (m,n) are equal to zero for any of the subsystems.

[0057] In an InI-HMD system, the EIs are seen as an array of spatially displaced virtual images observed through a viewing window. Figure 4D illustrates a simple example in which four neighboring EIs rendered on the microdisplay are imaged through their corresponding lenslets and a shared eyepiece of the MLA and projected onto the virtual CDP as four partially overlapping virtual EIs, as illustrated by the dashed boxes overlapping each other in the central shaded square. Figure 4E is the same as Figure 4D, with the light rays associated with two of the EIs removed to facilitate understanding of the basic principle. The displacement Δp between the centers of neighboring virtual EIs on the virtual CDP EIc is no longer the size of the virtual EI p EIcand their paraxial values, respectively, are further expressed as:

number

number

[0058] In equations (7) and (8), Z CDP is the distance between the virtual CDP and the viewing window. For a given subsystem unit indexed as (m,n), the lateral coordinate of the paraxial center of the corresponding virtual EI can be expressed as:

number

[0059] Equations (7) and (9) essentially provide a paraxial calculation of the image dimensions and center position for each of the subsystems. The image size of the virtual EI on the virtual CDP will usually be much larger than the displacement of the neighboring virtual EIs, and therefore the virtual EIs on the virtual CDP will partially overlap each other. As illustrated in FIG. 4D, the shaded overlap area on the virtual CDP corresponds to the region where all four virtual EIs overlap each other, where four distinctive elemental views, one from each of the EIs, can be rendered to reconstruct the light field of the sub-volume of the scene seen through the viewing window. Due to this nature, the entire FOV of the InI-HMD is mosaicked by the sub-volumes rendered by each of the individual EIs, and therefore cannot be calculated as easily as in a conventional HMD. For simplicity, by referencing the center of the virtual image of the edge EI on the virtual CDP, the diagonal field of view (FOV) of the display system can be estimated as follows:

number

[0060] The above steps demonstrate the disclosed method of modeling an LF-HMD system and analytical methods for calculating linear relationships of system parameters. These steps are distinct from the steps of modeling a traditional 2D HMD and are important for developing a suitable optimization strategy.

[0061] As mentioned before, an optimization strategy for an LF-HMD needs to properly control and evaluate optical aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the 2D elemental images, individually and collectively, to account for the bright field ray position sampling aspects. The optimization strategy also requires methods and metrics to control and evaluate optical aberrations that reduce the accuracy of directional ray sampling.

[0062] (Ray Position Sampling Considerations): Optimizing the ray position sampling of the brightfield function can be achieved by controlling optical aberrations that affect the contrast and resolution of the virtual display or distort the geometry of the 2D element images. It is beneficial to obtain clearly imaged EIs on the virtual CDP from the display panel through their corresponding lenslets and eyepiece groups of the MLA.

[0063] The optimization strategies according to the disclosed embodiments for ray position sampling are manifold and include optimizing the imaging process of each EI by each of the subsystems individually. For example, optimization constraints and performance metrics available in optimizing 2D image-conjugates for conventional HMDs can be used. The exact constraints and performance metrics vary widely from system to system, depending heavily on the complexity of the optical components utilized in the optical system for InI-HMDs. Examples of typical constraints include, but are not limited to, minimum and maximum element thickness or spacing between adjacent elements, total path length of the system, allowable component size, shape of each of the optical surfaces, surface sag deviation from a reference plane, type of optical material to be used, amount of tolerable aberration, or amount of optical power. Examples of performance metrics include, but are not limited to, root-mean-square (RMS) spot size, wavefront error, amount of residual aberration, modulation transfer function, or acceptable image contrast. With this initial stem, the entire FOV of the LF display composed by the individual EIs is optimized separately, instead of being treated as a whole as in conventional HMD design. However, such individual optimization for each of the EIs overlooks the corresponding connections between neighboring EIs, and more importantly, overlooks the relative position and size of the virtual EI with respect to the total FOV. For the InI-HMD, as shown in FIG. 4D, the imaged EIs on the virtual CDP will be perfectly aligned and partially overlap with the neighboring ones, and the paraxial relationship will be given by equations (7)-(9). However, the image aberrations induced by both the MLA and the eyepiece group will distort the virtual EIs on the virtual CDP, and the distorted virtual EIs will cause potential failure of view convergence when reconstructing the bright field of a 3D scene at different depths.

[0064] In order to take into account the effects of EI-induced distortion locally and globally, two different types of constraints during optimization can be applied. The first constraint is the control of local distortion aberrations for each of the subsystems representing a single EI, which can be easily implemented by adopting distortion-related optimization constraints already available in the optical design software for each zoom configuration. The exact constraints for local distortion control vary widely from system to system, depending heavily on the complexity of the optical components utilized in the optical system for the InI-HMD. Examples of typical controls on distortion include, but are not limited to, maximum allowable deviations of image heights of sampled object fields from their paraxial values, allowable percentiles of image height and shape differences from a regular grid, allowable magnification differences of different object fields, or allowable shape deformations of images from a desired shape, etc. These controls are typically applied as constraints to each subsystem individually to ensure that each subsystem forms an image with an acceptable local distortion. These local controls of distortion in each subsystem ensure that the dimensions and shapes of the virtual EIs remain within threshold levels compared to their paraxial undistorted images.

[0065] The second constraint is the control of the global distortion associated with the lateral position of the virtual EI relative to the entire FOV of the reconstructed 3D scene. To optimize this global distortion, the chief ray of the central object field of each EI on the microdisplay should be specially tracked and its occlusion on the virtual CDP needs to be extracted and constrained within a threshold level compared to its paraxial position in the global coordinates. For a given subsystem indexed as (m,n), the global deviation of the central position of the virtual EI on the virtual CDP from its paraxial position, together with the corresponding constraints, can be quantified by a possible metric GD, which can be expressed as follows:

number

[0066] The GD metric in Equation (11) examines the angular deviation between the actual position of the chief ray of the central object field measured from the viewing window and the theoretical position. For example, as illustrated in FIG. 5, different values ​​of the GD metric correspond to different amounts of distortion, and thus can enable a system designer to select an appropriate target GD value that is suitable for a particular system. For example, in one particular system (e.g., due to the cost and quality of components), a larger amount of distortion can be tolerated, which can inform the designer regarding the selection of an appropriate target GD value.

[0067] A constraint corresponding to the metric can thus be created by taking the maximum value of the metric for all EIs across all subsystems. By adding constraints to the optimization process and modifying the value of the constraint, the maximum allowed global distortion can be adjusted and optimized. Figure 5 further demonstrates the overall correlation between the global distortion and the value of GD by utilizing an example of barrel distortion and keystone distortion simulated in a 40°×40° InI-HMD system with a CDP depth of 1 diopter. In each of the subfigures, both a perfect theoretical FOV grid without global distortion (solid black line) and a distorted FOV grid (slightly offset in grey) corresponding to a particular type and value of distortion are plotted, with the numbers representing the maximum and average values ​​of GD calculated from Equation (11) for a total of 11×11 sampled centers (grid intersections) of EIs. For example, a barrel distortion of 1% results in a maximum GD of only 0.36°, while a barrel distortion of 5% results in a maximum GD of 1.78° in magnitude. Clearly, the value of GD provides good control of the global distortion of the EI central position, either as a conventional distortion pattern (e.g., barrel or pincushion distortion) or an unconventional one (e.g., keystone distortion), when optimizing the ray position of the brightfield function.

[0068] Ray Direction Sampling Considerations: Due to the inherent characteristics of LF-HMDs, the ray direction of the bright field plays a very important role in designing such a display system. Inaccurate sampling of the ray direction will not only affect the integration of EI, but also potentially lead to a non-uniform number of elemental views on the reconstructed bright field target and therefore to misrepresented focal cues. In the case of severe pupil aberrations, it is even possible that the number of elemental views encompassed by the pupil of the viewer's eye is reduced to less than two, such that the system is no different from a conventional stereoscopic display system and is unable to properly render the true bright field.

[0069] As mentioned above, the viewing window is where all the chief rays through the central pixels of all the EIs intersect with the optical axis, as shown in FIG. 3A, to ensure that all of the EIs can be properly viewed at the same time. The footprints of the ray bundles projected from each of the elemental views, characterized by equation (5), collectively define the viewing window, whose paraxial dimension is characterized by equation (6). In order to optimize the bright field ray direction sampling in designing an LF-HMD, appropriate constraints must be provided on the footprint of each elemental view projected onto the viewing window, rather than directly optimizing the exit pupil of the elemental view. For an InI-HMD with ideal ray directions of the bright field function, i.e., without pupil aberrations, the merged footprint diagram at the viewing window should have two notable properties. First, chief rays from different object fields on a single EI passing through corresponding lenslets and eyepiece groups of the MLA should converge at the center of the imaged aperture and intersect the viewing window in a regular grid with uniform spacing similar to the pixel array of the EI. Second, chief rays from the same object field (relative to their own EI) passing through their corresponding lenslets and eyepieces should converge at the viewing window, and the footprints of the ray bundles from these pixels should form the same shape and overlap perfectly with each other on the viewing window.

[0070] To optimize the bright field ray direction sampling in designing the LF-HMD, appropriate constraints on the footprint of each elemental view projected onto the viewing window must be provided. To consider the effect of ray footprint and direction-induced pupil aberrations on the viewing window during optimization, the disclosed optimization strategy (1) extracts the exact footprint of the ray bundle from any given pixel of a given EI on the viewing window, and (2) establishes a metric function that appropriately quantifies any deviation of the ray footprints from their paraxial shape and position, so that constraints can be applied during the optimization process to control the deviation within a threshold level.

[0071] In some embodiments, for each given object field, four marginal rays are sampled through the lenslet aperture to avoid exhaustive computation time during the optimization process. The coordinates of these marginal rays on the viewing window define the envelope of the ray footprint of the sampled field of view on a given EI in a given subsystem. For a sampled object field indexed as (i,j) on a given EI corresponding to a sampled subsystem indexed as (m,n), the deformation of the ray footprint from its paraxial shape can be quantified by a metric function PA, for example, using the following:

number

[0072] In equation (12), x' v and y' v are the real positions of the marginal rays on the viewing window obtained via real ray tracing horizontally and vertically, respectively, while x v and y v are their corresponding paraxial positions on the viewing window. k is the index of the four marginal rays for the sampled object on a given EI corresponding to the sampled subsystem.

[0073] The metric PA in equation (12) quantifies the deformation of the ray footprint of a given ray bundle from its paraxial shape by investigating the relative ratio of the average deviation distance between the actual and theoretical positions of the peripheral rays on the viewing window to the diagonal width of the paraxial footprint. By adding constraints to the optimization process and modifying values, the maximum allowed deviation and deformation of the footprint, i.e., pupil aberrations affecting the ray directions of the bright field of the InI-HMD, can be adjusted and optimized.

[0074] Figure 6 illustrates the overall correlation between the footprint diagram on the viewing window, pupil aberration, and metric function values ​​of the PA. A single constraint can therefore be created by taking the maximum value of the metric from all sampled object fields on each of the sampled subsystems. The figure plots simulated ray footprint diagrams for the central field point of the EI centered on the optical axis of the InI-HMD, with and without pupil aberration. For simplicity, in the simulations, the lenslets were treated as ideal lenses, and the eyepiece group was modeled with different aberration terms and magnitudes applied as pupil aberrations (e.g., spherical aberration of 0.25 to 1 wave peak-to-valley (λPV) and tilt of 0.25 to 1°). Specifically, the theoretical footprint diameter d vwas set as 1 mm. In each of the sub-figures, both the theoretical footprint diagram without pupil aberrations (a set of 10×10 star points forming a square bounded by solid lines) and the deformed or displaced footprint diagram (the remaining star points) corresponding to a particular type and value of the pupil aberration terms were plotted. The numbers under each of the sub-figures represent the value of PA calculated from equation (12) for each case. It can be observed that due to the presence of pupil aberrations, the actual footprint diagrams can be significantly deformed (e.g., by pupil spherical aberration) or displaced (e.g., by tilt) from their theoretical footprints. Equation (12) gives a good estimate of the severity of the pupil aberrations in terms of PA based on the footprint diagrams. For example, a pupil spherical aberration of 0.25 λPV results in a PA of only 0.059, while a pupil spherical aberration of 1 λPV results in a PA of 0.23.

[0075] Using metrics such as those described above, a system design can be performed to determine an optimal (or generally desired or target) design, including determining the distance and angular alignment of components, component sizes (e.g., lenslet pitch, lenslet area, lenslet and / or eyepiece surface profile, and lenslet array and / or eyepiece aperture, etc.). The system can further include additional optical elements such as relay optics, elements for folding or changing the optical path, and others, which may undergo ray tracing and optimization as part of the system design.

[0076] FIG. 7A illustrates an example layout of a binocular setup of an InI-HMD system generated according to the disclosed design setup and optimization method. In FIG. 7A, the system is illustrated relative to the viewer's head. FIG. 7B provides further details on the optical layout of FIG. 7A for its monocular setup (right eye) with important elements labeled. The upper section of FIG. 7A follows a similar configuration to FIG. 7B. It should be noted that the distances and specific numbers of components illustrated are provided by way of example, not limitation, to facilitate understanding of the disclosed technology. As illustrated, the optical system for the display path of particular interest includes three main subsections: a micro-INI unit including a high-resolution microdisplay, a custom-designed aspheric MLA, a custom aperture array, an adjustable relay group with four lenses (e.g., an off-the-shelf spherical lens with an Optotune® EL-10-30 adjustable lens sandwiched inside), and a freeform waveguide-like prism. Essentially, a waveguide-like prism formed by four freeform surfaces designated as S1-S4 further magnifies the reconstructed intermediate compact scene and projects the light towards an exit pupil or viewing window, where the viewer sees the magnified 3D scene reconstruction.

[0077] At the beginning of the design for the example configuration, the MLA and relay eyepiece groups were optimized separately to get a good starting point due to the complexity of the system. For the initial iterations of the MLA design, special attention was paid to the peripheral rays being constrained not to go beyond the edge of the lenslets to prevent crosstalk between nearby EIs. The two surfaces of the lenslets were optimized as aspheric polynomials with coefficients up to the sixth order. In the initial iterations of the relay and eyepiece group design, the design was inversely set up by tracing the rays backwards from the viewing window towards the eyepiece and relay lenses. Each of the four freeform surfaces of the prism was described by an x-plane symmetric XY polynomial and optimized with their coefficients up to the tenth order.

[0078] After obtaining the initial design of both the MLA and the relay eyepiece group, the two parts were integrated and an array of 7x3 zoom configurations was created. Figure 7C shows the design configuration of the integrated display path in CodeV® plotted with real ray tracing from some of the sampled EIs and lenslets. A viewing window was placed at the back focus of the freeform eyepiece. An ideal lens with a focal length equivalent to the focal power of the eye corresponding to the depth of the virtual CDP was inserted at the viewing window to simulate the retinal image of the EI. In the figure, only the rays from the central pixel of each EI were traced. The MLA contains 17x9 identical lenslets with a lens pitch of 1 mm, and the microdisplay is divided into 17x9 EIs, each with 125x125 pixels. Considering the planar symmetry of the freeform prism, a total of 7x3 subsystems (EIs with corresponding lenslets of the MLA) for the upper half of the total FOV were sampled, and the distribution of the sampled subsystems is shown in Figure 7C as zoomed lenslets. For each subsystem, nine field points covering the entire EI were further sampled. In addition, to account for DOF expansion and virtual CDP depth with various refractive powers of the adjustable lenses, the system was further configured to optimize its performance for virtual CDP depths of 0, 1, and 3 diopters. The focal lengths of the ideal and adjustable lenses in the viewing window were therefore correspondingly adjusted to correctly focus the light rays onto the image plane. In summary, combining the zooms of the 21 sampled MLA-EI subsystems and the zooms for different virtual CDP depths, the overall system was modeled with a total of 63 zoom configurations and a total of 567 field points.

[0079] FIG. 8A plots the image contrast of sampled 7×3 subsystems covering the entire field of view of the display path evaluated at a Nyquist frequency of 3 arc minutes or 10 cycles per degree (cpd), with a virtual CDP set 1 diopter away from the viewing window. In each of the subsystems, five object fields on their corresponding EI are sampled, and their contrast values ​​are represented by circles, each at a specific location corresponding to each of the five object fields. Over the entire 30°×18° of the FOV, all of the image contrasts are well above the threshold of 0.2 at the Nyquist frequency, with an average value of 0.53.

[0080] FIG. 8B plots the modulation transfer function (MTF) of on-axis field points on three EIs corresponding to a lenslet (index (9,5)) centered on the optical axis covering the entire FOV of the display path, a lenslet in the upper left corner (index (1,1)), and a lenslet in the upper right corner (index (17,1)). FIG. 7C further plots the MTF of on-axis field points on three EIs corresponding to a lenslet (index (9,5)) centered on the optical axis, but with their virtual CDPs adjusted 3 diopters to 0 diopters away from the viewing window by adjusting the refractive power of the adjustable lens. It is clear that the optical system demonstrates uniform image contrast and MTF performance across the entire FOV and a depth range of more than 3 diopters, with image contrast degradation evaluated at Nyquist angular frequencies below 0.15.

[0081] FIG. 9A further plots the global distortion grid of sampled 7×3 subsystems of the display path covering the entire display FOV by extracting the chief ray coordinates of the central object field on the corresponding EI from each of the subsystems from the actual ray tracing of the design example, where the paraxial coordinates of the chief rays are plotted in the solid grid and the actual ray coordinates are plotted in asterisks. Although the display path suffers from a small amount of keystone distortion due to the folded optical path, in general, the global distortion for the entire display field of view is relatively small, especially for designs with freeform optics that easily introduce higher order distortion terms. The design target for the global distortion GD was set as 0.75°, which corresponds to about 2% of the distortion for the entire FOV. All of the 7×3 subsystems were optimized within the design target with an average value of GD of 0.22°, which corresponds to an average distortion for the entire FOV of less than 1%.

[0082] Figures 9B and 8C compare the ray footprint diagrams in the viewing window before and after optimization of the ray direction of the bright field function. Figure 9B plots the envelope of the ray footprints on the viewing window for nine sampled object fields of an on-axis lenslet (solid line, index (9,5)) and the envelope for nine object fields of an edge lenslet located in the upper right corner (solid line with diagonal dash, index (17,1)) from the actual design setting before constraining the pupil aberration of the system. The ray footprint envelopes for these two lenslets are not only distorted but also largely separated. In comparison, Figure 9C plots the merged envelope of the footprint diagrams extracted from the actual design setting after optimization. In this case, the ray footprints through nine sampled lenslets of the MLA are plotted (thin solid lines, index (1,1), (9,1), (17,1), (1,3), (9,3), (17,3), (1,5), (9,5), and (17,5)). FIG. 8C plots (in thick solid lines) the theoretical envelope of ray footprints of the same field on lenslets obtained from paraxial calculations perfectly aligned with each other across lenslets and fields of view, as suggested above. The design target PA was set as 0.3, since the human visual system will be less sensitive to ray direction than position. Nineteen of the 7×3 subsystems were optimized within the design target with an average ratio of 0.145, well below the design target, which corresponds to an average throughput or size deviation of the real footprints of less than 0.27, of which the projected footprint deviations and deformations are still acceptable, as shown in FIG. 9C.

[0083] In the experiment, test results of a prototype InI-HMD system designed according to the disclosed technology were obtained by placing a camera in the viewing window and capturing a real image of the scene displayed through the system. The test scene included a slanted wall with a water drop texture ranging from a depth of about 500 mm (2 diopters) to 1,600 mm (0.6 diopters) that was computationally rendered and displayed as a test target. A central 15×7 elemental view rendered on the microdisplay was acquired, and in addition, a real captured image of the rendered light field of such a continuous 3D scene was acquired by adjusting the focal depth of the camera from the near side (about 600 mm) to the middle part (about 1,000 mm) and far side (about 1,400 mm) of the scene, which simulates the accommodation adjustment of the eye from near to far distances. The prototype virtual CDP was shifted and fixed at a depth of 750 mm (1.33 diopters). The portions of the 3D scene within the same depth of the camera focus remained sharply in focus with high fidelity compared to the target. In contrast, other portions of the 3D scene outside the camera focus depth were blurred, and the further the depth of the 3D scene deviates from the camera focus, the more blurred the portions of the 3D scene became, which resembles what one would observe from a real-world scene. Such results clearly demonstrate the ability of the prototype designed in accordance with the disclosed embodiments to render high quality bright field content and, more importantly, to render the correct focus cues and facilitate accommodation of the viewer's eyes.

[0084] FIG. 10 illustrates a set of operations that may be performed to implement a method for designing an integral imaging (InI)-based three-dimensional (3D) display system according to an exemplary embodiment. The method includes, at 1002, tracking a set of light rays associated with a bright field in an InI-based 3D system. The system includes an array optical system, an array display device capable of generating a plurality of element images, a first reference plane representing a virtual central depth plane (CDP) where light rays emitted by a point source on the display converge to form an image point, a second reference plane representing a viewing window for viewing the reconstructed 3D scene, and an optical subdivision representing a model of the human eye. The tracking starts at the array display device and is performed through the array optical system to the optical subdivision for each element of the array display device and the array optical system. The method further includes, at 1004, adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values, the first metric value corresponding to a light ray direction sampling of the bright field.

[0085] In one exemplary embodiment, the first metric value quantifies the deformation of the ray footprint of a given ray bundle of bright field from its paraxial footprint. In another exemplary embodiment, the first metric value is determined according to the relative ratio of the average deviation distance between the actual and theoretical positions of the peripheral rays on the second reference plane to the diagonal width of the paraxial footprint. In yet another exemplary embodiment, the first metric value is determined according to Equation (12). For example, the first metric value can be determined based on the difference between the actual positions of the peripheral rays on the viewing window obtained by ray tracing and their corresponding paraxial positions on the viewing window.

[0086] According to another exemplary embodiment, in the method as described above, adjusting one or more parameters associated with the InI-based 3D system is performed to further obtain a second metric value within another predetermined value or range of values, the second metric value corresponding to a bright field ray position sampling that takes into account at least the deformation induced by neighboring elements of the array optical system. In one exemplary embodiment, the second metric value is determined according to an angular deviation between the actual position and the theoretical position of the chief ray of the central object field measured from a second reference plane. In yet another exemplary embodiment, the second metric value represents a global distortion measure. In yet another exemplary embodiment, the second metric value is determined according to Equation (11). For example, the second metric value is calculated as a deviation of a center position of a virtual element image of a plurality of element images on the virtual CDP from its paraxial position. In another exemplary embodiment, adjusting one or more parameters associated with the InI-based 3D system is performed on the bright field ray position sampling to further individually optimize the imaging of each EI.

[0087] In one exemplary embodiment, the InI-based 3D system further includes an eyepiece positioned between the array optics and the second reference plane, and tracking the set of rays includes tracking the set of rays through the eyepiece. In some embodiments, the array display device is a microdisplay device. In some embodiments, the array optics comprises one or more lenslet arrays, each of which includes a plurality of microlenses. In another embodiment, the InI-based 3D system is an InI-based head-mounted display (InI-based HMD) system.

[0088] In some embodiments, the predetermined value or range of values ​​for one or both of the first or second metrics is selected to achieve a particular image quality, hi some embodiments, the predetermined value or range of values ​​for one or both of the first or second metrics represents a maximum or minimum value that provides an optimal design criteria for the first or second metric.

[0089] 11 illustrates a set of operations that may be performed to improve the design of an integral imaging optical system, according to certain example embodiments. These operations may be performed for an integral imaging optical system that includes a lenslet array that angularly samples a bright field direction to generate an array of two-dimensional elemental images (EI), each of which represents a different viewpoint in three dimensions (3D). The method includes determining, at 1102, a first metric corresponding to ray direction sampling of the bright field, determining, at 1104, a second metric corresponding to ray position sampling of the bright field, and performing, at 1106, a ray tracing operation to determine a design for the integral imaging optical system based on the first and second metrics. In some embodiments, the ray tracing operation is performed based on one or more constraints, including maintaining the first or second metrics within corresponding values ​​or ranges of values.

[0090] FIG. 12 illustrates a block diagram of a device 1200 that may be used to implement certain aspects of the disclosed technology. For example, the device of FIG. 12 may be used to receive, process, store, provide for display, and / or transmit various data and signals associated with the disclosed image sensor. The device 1200 includes at least one processor 1204 and / or controller, at least one memory 1202 unit in communication with the processor 1204, and at least one communication unit 1206 that enables exchange of data and information, directly or indirectly, with other entities, devices, databases, and networks through a communication link 1208. The communication unit 1206 may provide wired and / or wireless communication capabilities according to one or more communication protocols, and thus may include suitable transmitters / receivers, antennas, circuits and ports, and encoding / decoding capabilities that may be necessary for proper transmission and / or reception of data and other information. The example device 1200 of FIG. 12 may be integrated as part of a larger component (e.g., a server, computer, tablet, smartphone, etc.) that may be used to perform various calculations, methods, or algorithms disclosed herein, such as to implement a ray tracing program (e.g., Code V or Zemax) that is extended to accommodate the improvements disclosed herein.

[0091] Processor 1204 may include, for example, a central processing unit (CPU) for controlling the overall operation of the host computer. In one embodiment, processor 1204 accomplishes this by executing software or firmware stored in memory 1202. Processor 1204 may be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), graphics processing units (GPUs), etc., or a combination of such devices.

[0092] The memory 1202 may be or may include the main memory of the computer system. The memory 1202 represents any suitable form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of such devices. In use, the memory 1202 may include, among other things, a set of machine instructions that, when executed by the processor 1204, cause the processor 1204 to perform operations to implement certain aspects of the disclosed technology.

[0093] It should be understood that the various disclosed embodiments may be implemented individually or collectively in devices comprised of various optical components, electronic hardware and / or software modules and components. These devices may comprise, for example, processors, memory units, interfaces communicatively connected to each other and may range from desktop and / or laptop computers to mobile devices, etc. The processors and / or controllers may perform the various disclosed operations based on execution of program codes stored on storage media. The processors and / or controllers may, for example, communicate with at least one memory and directly or indirectly communicate with at least one communication unit that enables exchange of data and information with other entities, devices, and networks through communication links. The communication unit may provide wired and / or wireless communication capabilities according to one or more communication protocols, and thus, it may comprise appropriate transmitters / receivers, antennas, circuits and ports, and encoding / decoding capabilities that may be necessary for proper transmission and / or reception of data and other information.

[0094] Various information and data processing operations described herein may be implemented, in one embodiment, by a computer program product embodied in a computer-readable medium including computer-executable instructions, such as program code, executed by a computer in a networked environment. The computer-readable medium may include removable and non-removable storage devices, including, but not limited to, read-only memory (ROM), random access memory (RAM), compact disks (CDs), digital versatile disks (DVDs), and the like. Thus, the computer-readable medium described herein comprises a non-transitory storage medium. Generally, program modules may include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. A particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0095] The foregoing description of the embodiments has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the embodiments of the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments. The embodiments discussed herein have been chosen and described to explain the principles and nature of various embodiments and their practical applications, and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications as suited to the particular use envisaged. Although operations are depicted in the figures in a particular order, this should not be construed as requiring such operations to be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, to achieve desirable results. Features of the embodiments described herein may be combined in all possible combinations of methods, apparatus, modules, and systems.

Claims

1. 1. A method for designing an integral imaging (InI) based three-dimensional (3D) display system, the method comprising: tracing a set of rays associated with a bright field in the InI based 3D display system, the InI based 3D display system comprising: An array optical system; an array display device capable of generating a plurality of elemental images; a first reference plane representing a virtual central depth plane (CDP), where light rays emitted by a point source on the array display device converge to form an image point on the virtual CDP; a second reference plane representing a viewing window for viewing the reconstructed 3D scene; An optical subdivision representing a model of the human eye; the tracking is performed starting at the array display device, through the array optics, to the optical subsections for each element of the array display device and the array optics; adjusting one or more parameters associated with the InI-based 3D display system to obtain at least a first metric value within a predetermined value or range of values; Including, The method of claim 1, wherein the first metric value corresponds to a ray direction sampling of the bright field and quantifies a deformation of a ray footprint of a given ray bundle of the bright field from its paraxial footprint, the first metric value being determined according to a relative ratio of an average shift distance between actual and theoretical positions of marginal rays on the second reference plane to a diagonal width of the paraxial footprint, and adjusting the one or more parameters associated with the InI-based 3D display system is performed to further obtain a second metric value within another predetermined value or another range of values, the second metric value corresponding to a ray position sampling of the bright field that takes into account at least deformations induced by neighboring elements of the array optical system.

2. The method of claim 1 , wherein the first metric value is determined based on a difference between actual positions of marginal rays on the viewing window obtained by ray tracing and their corresponding paraxial positions on the viewing window.

3. The method of claim 1 , wherein the second metric value is determined according to an angular deviation between an actual position of a chief ray of a central object field and a theoretical position measured from the second reference plane.

4. The method of claim 1 , wherein the second metric value represents a global distortion measure.

5. The method of claim 1 , wherein the second metric value is calculated as a deviation of a central position of a virtual elemental image of the plurality of elemental images on the virtual CDP from its paraxial position.

6. 2. The method of claim 1, wherein adjusting the one or more parameters associated with the InI-based 3D display system is performed on the ray position sampling of the bright field to further individually optimize imaging of each element image.

7. 2. The method of claim 1, wherein the InI-based 3D display system further includes an eyepiece positioned between the array optics and the second reference plane, and wherein tracing the set of rays includes tracing the set of rays through the eyepiece.

8. The method of claim 1 , wherein the array optics comprises one or more lenslet arrays, each of the one or more lenslet arrays including a plurality of microlenses.

9. The method of claim 1 , wherein the InI based 3D display system is an InI based head mounted display (InI based HMD) system.

10. The method of claim 1 , wherein the predetermined value or range of values ​​for one or both of the first metric value or the second metric value is selected to achieve a particular image quality.

11. 2. The method of claim 1 , wherein the predetermined value or range of values ​​for one or both of the first metric value or the second metric value represents a maximum or minimum value that provides an optimum design criteria for the first metric value or the second metric value.

12. A device, the device comprising: A processor; a memory for storing processor-executable code; Equipped with A device, wherein the processor executable code, when executed by the processor, causes the processor to perform the method for designing an InI-based 3D display system according to any one of claims 1 to 11.

Images