Optical design and optimization technology for 3D light field displays

Abstract

Methods and frameworks for designing and optimizing high-performance light field displays such as integrated imaging head-mounted displays are disclosed. The disclosed techniques enable user-defined metrics to characterize the performance and optimization of such systems. A design method relates to an integrated imaging-based three-dimensional (3D) display system, the display system comprising an array optical body, an array display device for generating multiple element images, a first reference plane representing a virtual central depth plane (CDP), a second reference plane representing a viewing window for viewing a reconstructed 3D scene, and an optical subsegment representing a model of a human eye. The method includes tracing rays starting from the array display device, passing through the array optical body, and reaching the optical subsegment for each element of the array display device and the array optical body, and adjusting one or more parameters to obtain at least a first metric value within a predetermined value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 885,460, entitled “Optical Design and Optimization Techniques for 3D Light Field Displays,” filed on August 12, 2019. The entire contents of the above-mentioned provisional application are incorporated by reference as part of the disclosure of this document.

[0003] Statement Regarding Federally Funded Research

[0004] This invention was made with government support under Grant No. 1,422,653 awarded by NSF. The government has certain rights in this invention. Technical Field

[0005] The disclosed technology relates generally to three-dimensional (3D) displays and, more particularly, to methods and frameworks for designing and optimizing high-performance light field displays. Background Art

[0006] Conventional stereoscopic 3D displays (S3D) stimulate the perception of 3D space and shape from a pair of two-dimensional (2D) perspective images (one for each eye) at a fixed distance through binocular disparity and other pictorial depth cues of the 3D scene seen from two slightly different viewing positions. A key limitation associated with S3D type displays is the known vergence-accommodation conflict (VAC) problem. In binocular vision, when viewing an object, the eyes must rotate about a horizontal axis so that the image is projected onto the center of the retina of both eyes. Vergence is the simultaneous movement of the two eyes in opposite directions to achieve or maintain single binocular vision. Accommodation is the process by which the eyes change optical power as distance changes to maintain a sharp image or focus on an object. Under normal viewing conditions of objects, changes in the focus of the eyes to view objects at different distances automatically cause vergence and accommodation. In the context of 3D displays, VAC occurs when the brain receives mismatched cues between the distance of a virtual 3D object (vergence) and the focal distance required for the eyes to focus on that object (accommodation). This problem stems from the inability to render correct focus cues for 3D scenes, including accommodation and retinal blur effects. It causes multiple cues to conflict and is believed to be one of the key factors causing various visual artifacts associated with viewing S3D displays. Summary of the invention

[0007] The disclosed embodiments relate to three-dimensional (3D) displays, and more particularly to methods and frameworks for designing and optimizing high-performance light field displays, including but not limited to light field head-mounted displays.

[0008] In one example embodiment, a method for designing a three-dimensional (3D) display system based on integrated imaging (InI). The system includes: an array optical body; an array display device capable of producing a plurality of element images; a first reference plane representing a virtual central depth plane (CDP) on which light rays emitted by a point source on the display converge to form an image point; a second reference plane representing a viewing window for viewing a reconstructed 3D scene; and an optical sub-segment representing a model of a human eye. The method for designing the system includes tracing a set of rays associated with a light field in an InI-based 3D system, wherein for each element of the array display device and the array optical body, the tracing starts from the array display device and is performed through the array optical body to the optical sub-segment. The method also includes adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values. The first metric value corresponds to a ray-directed sampling of the light field.

[0009] In the disclosed InI-based 3D system, the light field reconstructs a 4-D light field of a 3D scene by angularly sampling the directions of light rays apparently emitted by the 3D scene. According to some embodiments, the optical design process includes optimizing the mapping of both ray positions and ray directions in 4-D light field rendering, rather than simply optimizing the 2D mapping between object-image conjugate planes in conventional HMD designs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is an example of a conventional head mounted display (HMD) configuration that renders a two-dimensional (2D) image on a microdisplay to form a sharply focused retinal image.

[0011] Figure 1B The diagram shows how the depth adjustment differs from the virtual display. Figure 1A Conventional HMD configuration shown.

[0012] Figure 2A is an example configuration illustrating an image formation process of an integrated imaging based HMD (InI-HMD) for light field reconstruction of a first adjusted depth.

[0013] Figure 2B Picture shows Figure 2A Configuration and image formation corresponding to the second adjustment depth.

[0014] Figure 3A Illustrated is a mapping of associated symbols and light field functions for an integrated imaging based HMD (InI-HMD) configuration.

[0015] Figure 3B The optical principle of reconstructing a 3D scene for an InI-based light field 3D display is illustrated.

[0016] Figure 4A A three-dimensional view of various aspects of the InI-HMD system is illustrated, including the footprint on the viewing window and the imaging apertures of the lenslets of the microlens array (MLA).

[0017] Figure 4B Illustrated is the partitioning of an elemental image (EI) on a microdisplay associated with an InI-HMD system.

[0018] Figure 4C The diagram shows a subsystem including EI, MLA lenslets and a shared eyepiece set.

[0019] Figure 4D Partially overlapping virtual images of four EIs on a virtual central depth plane (CDP) are shown.

[0020] Figure 4E The diagram shows how some features are removed to improve clarity. Figure 4D Partially overlapping virtual images of the EI on the CDP are shown.

[0021] Figure 5 Illustrated is the effect of a global deviation metric of global distortion with respect to ray positions on the full-field distortion grid for an example InI-HMD system.

[0022] Figure 6 The effect of deformation of the ray footprint of a given ray beam from its paraxial shape is illustrated for an example InI-HMD system.

[0023] Fig. 7A An example layout of a binocular InI-HMD system designed according to the disclosed technology is illustrated.

[0024] Figure 7B Illustration about Fig. 7A Further details of the optical layout of the section.

[0025] Figure 7C Show Figure 2. The design configuration of the integrated display path is plotted, with actual ray traces from a portion of the sampled EI and lenslet.

[0026] Fig. 8A An image contrast diagram of a display path associated with an example InI-HMD system is illustrated.

[0027] Figure 8B Illustrated are modulation transfer function (MTF) plots of on-axis field points corresponding to a subset of sampled lenslets covering the field of view of an example InI-HMD system.

[0028] Figure 8CThe MTF plots for the on-axis field points of each EI corresponding to the central lenslet are illustrated, where their virtual CDPs are sampled from 3 diopters to 0 diopters away from the viewing window of the example InI-HMD system.

[0029] Fig.9A A distortion grid diagram of the display path covering the full field of view of an example InI-HMD system is illustrated.

[0030] Fig. 9B Illustrated is a footprint diagram at the viewing window before optimizing the pupil aberration of an example InI-HMD system.

[0031] Fig. 9C Illustrated is a footprint diagram at the viewing window after optimizing the pupil aberration of an example InI-HMD system.

[0032] Fig.10 A set of operations for designing an InI-based 3D display system according to an example embodiment is illustrated.

[0033] Fig.11 A set of operations for improving the design of an integrated imaging optical system is illustrated according to an example embodiment.

[0034] Fig.12 Illustrated is a block diagram of an apparatus that may be used to implement certain aspects of the disclosed technology. DETAILED DESCRIPTION

[0035] There have been some efforts to address the vergence-accommodation conflict (VAC) problem, among which is light field 3D (LF-3D) displays based on integrated imaging (InI-based). Integrated imaging generally refers to a three-dimensional imaging technique that captures and reproduces a light field using a two-dimensional array of optical apertures or microlenses. This configuration allows the reconstruction of a 3D scene by rendering directional light rays emitted from a scene seen through an array optical body from a pre-designed viewing window. However, there is a lack of system design methods suitable for optimizing 3D display systems, including InI-based LF-3D head-mounted displays (HMDs). Therefore, designing high-performance 3D display systems remains a challenge. In particular, without an optimized design, the InI-HMD will not be able to correctly render the depth and accommodation cues of the reconstructed 3D scene, resulting in impaired image quality and comfort.

[0036] In the following description, InI-HMD is used as an example system to illustrate the above problems and the disclosed solutions. However, it should be understood that the disclosed embodiments are similarly applicable to other types of 3D display systems, such as non-head-mounted, direct-view light field displays, in which the optical system for rendering the light field is not worn directly on the viewer's head, 3D light field displays that sample the light field only in one angular direction, typically in the horizontal direction, (this is better known as a display that only renders horizontal parallax - that is, multiple views are arranged as vertical stripes on the viewing window), or super multi-view displays or autostereoscopic displays, in which the base views are generated by a projector or imaging unit.

[0037] In systems utilizing integrated imaging, 3D images are typically displayed by placing a microlens array (MLA) in front of the image, where each lenslet of the MLA looks different depending on the viewing angle. The InI-HMD system requires rendering different element views created by multiple elements of the MLA and requires them to be seen through each eye pupil. Therefore, the light emitted by multiple spatially separated pixels on these element views is received by the eye pupil and integrated and added to form the perception of a 3D reconstructed point, which is essentially the key difference between the InI-HMD and conventional HMDs. In conventional HMD systems, the viewing optics simply projects the 2D image on the microdisplay onto a 2D virtual display, and therefore the light from a single pixel is imaged together by the eye optics to form the perception of a 2D point. At least due to this inherent difference in the image formation process, existing optical design methods for conventional HMDs become insufficient to design a true 3D LF-HMD system, which requires ray integration from multiple separate sources.

[0038] Among other features and advantages, the disclosed embodiments provide an improved optical design method that is capable of (1) generating a LF-3D HMD design to accurately perform actual ray tracing, and (2) optimizing the design to accurately sample and render the light field of the reconstructed 3D scene, which is key to driving the accommodation state of the viewer's eyes and thereby solving the VAC problem. In an embodiment of the disclosed technology, one or more new design constraints or metrics are established that help optimize the ray position sampling of the light field and / or the ray orientation sampling of the light field. For example, one constraint or metric for position sampling accounts for the global distortion (e.g., aberration) associated with the lateral position of the virtual element image (EI) relative to the entire FOV of the reconstructed 3D scene disparity. Another constraint or metric for orientation sampling provides a measure of the deviation or deformation of the ray footprint from its paraxial shape.

[0039] The use of the disclosed constraints and metrics improves the optical design process and allows the designer to evaluate the quality of the resulting image (e.g., in terms of solving VAC problems) and improve the design of the optical system. In some embodiments, by minimizing the disclosed metrics during the design process, an optimal design can be produced. The disclosed metrics further provide an assessment of achievable image quality, and thus, in some embodiments, a desired image quality target can be achieved for a particular optical system based on a target value (as opposed to minimization) of the disclosed metrics.

[0040] Figure 1A and 1B An exemplary conventional HMD configuration for rendering a 2D image on a microdisplay is shown. Similar to other 2D displays, each pixel value represents the intensity or sum of radiation of light within an angular range. Inserting the microdisplay (in Figure 1A and 1B The eyepiece between the microdisplay (labeled "display" in the figure) and the eye simply magnifies the 2D image and forms an enlarged 2D virtual display at a distance optically conjugate to the microdisplay plane via the eyepiece. Therefore, all light rays from a single pixel are imaged together by the eye optics to form the perception of a 2D point. Figure 1A and 1B In the diagram, the labels "red" and "green" are used to help illustrate the different beams of rays propagating through the system. When the depth of accommodation of the eye's optics matches the apparent distance of the virtual display, a sharply focused retinal image is formed, such as Figure 1A Otherwise, Figure 1B As shown, when the accommodation depth deviates from the distance of the virtual display, the same blurred retinal image is formed.

[0041] Due to the nature of 2D image formation described above, the optical design process of such a system only needs to focus on the 2D mapping between pixels on the microdisplay and their corresponding images on the virtual display; the optimization strategy focuses on controlling optical aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the virtual display. To this end, the rays from each pixel on the display are imaged by a sequence of optical elements or a common optical path. Therefore, a conventional HMD system can be modeled by a shared optical configuration. In such a system, the retinal image of the rendered point is the projection of the ray emitted by a single pixel on the microdisplay or the magnified virtual display, allowing the optical performance of a conventional 2D HMD system to be fully evaluated by characterizing the 2D image pattern projected by the rays of a few field positions on the microdisplay.

[0042] In contrast, the LF-HMD reconstructs the 4-D light field of a 3D scene by angularly sampling the directions of light rays apparently emitted by the 3D scene. Figure 2A and2B is an example configuration illustrating an image formation process of an integrated imaging based HMD (InI-HMD) for light field reconstruction. Figure 3A and 3B The figure shows a three-dimensional view of the same system, with additional symbols to help understand the light field function. Figure 2A As shown, the system includes a microdisplay, an array optical body (e.g., a lenslet array), and an eyepiece. Figure 2A In , the element images (EI) of the 2D array represent different perspectives of the 3D scene rendered on the microdisplay (e.g., elements A1 to A3 for reconstructed point A; elements B1 to B3 for reconstructed point B). There are two unique and important conceptual reference planes that are widely recognized in InI-HMD. The first plane is the virtual central depth plane (CDP), where light rays emitted by point sources on the microdisplay converge to form image points after propagating through the MLA and the eyepiece (see also Figure 3A The second reference plane is the viewing window that defines the area where the viewer observes the reconstructed 3D scene. It coincides with the entrance pupil plane of the eye optics by design and is usually referred to as the exit pupil or eye box of a conventional HMD system.

[0043] Each pixel on these EIs is considered to be an image source defining position information (s, t) of a 4-D light field function. Associated with the array of EIs is an array optical body, such as an MLA, each of which defines directional information (u, v) of a light field function (see also Figure 3A , which illustrates a 4-D light field, including parameters s, t, u, v). Each 2D EI is imaged by its corresponding tiny lens as a separate imaging path. To reconstruct the light field of a 3D point, the ray beams emitted by multiple pixels (each pixel at a different EI) are modulated by their corresponding MLA elements to intersect at the reconstructed 3D location. Therefore, the light rays from these spatially separated pixels collectively create a 3-D point that appears to emit light in different directions. By properly arranging the EIs on the display panel, points located at different field angles and depths (e.g. Figure 2A and 2B The 4-D light field of the 3D scene for points A and B in FIG. 1 can thus be rendered accordingly.

[0044] In the magnified view configuration, the eyepiece is inserted to further magnify the miniature 3D scene into a large 3D volume with extended depth in the virtual space (e.g. Figure 2A and 2BA' and B' in the figure are magnified renderings of the reconstructed points A and B, respectively). In a true LF-3D HMD, multiple element views from a 3D point are projected at different locations of the eye's entrance pupil, and the retinal images of these element views collectively form the perception of a 3D point in space. Figure 2A As shown, when the eye is accommodated at the depth of reconstruction point A', the rays from their corresponding element pixels (e.g., pixels A1 to A3) will overlap each other and naturally form a sharply focused image on the retina, while the rays from pixel reconstruction point B' (e.g., rays from pixels B1 to B3) - which are located at a different depth than point A - will spatially deviate from each other and produce retinal blur. The apparent amount of retinal blur at point B varies with the difference between the reconstruction depth and the eye's accommodation. Figure 2B As shown, when the eye adjusts the depth to point B'

[0045] When , the retinal image of point B' becomes clear, while the retinal image of point A' becomes blurred. In this case, the retinal image of the reconstructed 3D scene by the InI-HMD will be close to the visual effect of watching a natural 3D scene.

[0046] Due to the nature of 3D image formation described above, the light field reconstruction of a 3D point is the integrated effect of light emitted by multiple spatially separated pixels, each located on a different element image and imaged by a different optical unit of the array optical body. Each pixel provides a sample of the light field position, and its corresponding unit of the imaging optical body provides a sample of the light field direction. Therefore, the optical design process needs to optimize the mapping of both ray position and direction in 4-D light field rendering, rather than simply optimizing the 2D mapping between object-image conjugate planes in conventional HMD design. The optimization strategy requires not only the proper control and evaluation of optical aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the 2D element image, which accounts for the ray position sampling aspect of the light field, but also methods and metrics for controlling and evaluating optical aberrations that reduce the accuracy of directional ray sampling.

[0047] like Figure 3A As shown, pixels on different element images are imaged through different optical paths passing through different optical elements. Therefore, the LF-HMD system needs to be modeled by a multi-configuration array system or an array of subsystems distributed in horizontal and vertical directions. Each subsystem represents a single imaging path from a given element image through its corresponding optical unit and possible eyepiece set (if there is one). It should be noted that in order to clearly distinguish the EI from the shape of the lenslet, for illustration purposes, a circular shape is used to represent the aperture of the lenslet and its corresponding footprint on the viewing window, while a square aperture has been used for the lenslets in the examples provided herein.

[0048] The retinal image of a rendered 3D point is the integral sum of projected rays emitted by multiple pixels on different element images, and the appearance of the image varies greatly with the state of eye accommodation. Therefore, the optical performance of the LF-HMD system cannot be fully evaluated by only characterizing the 2D image pattern projected onto the microdisplay from rays from a few field positions, but needs to be evaluated by characterizing the integrated image on the retina for different eye accommodation states. For this purpose, the eye model is an essential part of the imaging system.

[0049] Figure 3A Further illustration is a simplified process of light field rendering in view space, where the ray position of the light field function is represented by the projected virtual pixel (x c ,y c ) is sampled on the virtual CDP, and the ray direction is determined by the projection coordinate (x v ,y v Theoretically, the ideal light field function L'(x c ,y c ,x v ,y v ), the corresponding EI of the reconstructed light field point will be imaged as a point source on the virtual CDP, and their ray beams will perfectly intersect at the lateral and longitudinal positions of the point and will be Figure 3B It should be noted that, significantly different from conventional HMDs, the ray beams emitted by the element pixels of the reconstructed 3D points are projected to different positions of the viewing window, such as Figure 3B However, any deviation in the rendering of the light field function due to imperfections in the InI-HMD imaging process will result in corrupted EI on the virtual CDP and / or erroneous directions and footprints of the ray beams projected onto the viewing window. As a result, such deviations lead to corrupted reconstruction of the 3D scene and degrade the perceived image quality of the InI-HMD by the viewer. Therefore, when designing an InI-HMD system, the light field L(s,t,u,v)—physically rendered by the MLA and display panel in object space—must be accurately mapped to the light field L'(x c ,y c ,x v ,y v )middle.

[0050] In order to obtain a good mapping, it is crucial to obtain (1) the light field function from (s, t) to (x c ,y c ) so that each EI rendered on the display panel is well imaged onto the virtual CDP, and (2) the light field function from (u, v) to (x v ,yv )'s directional sampling map is well controlled so that the ray beam from each imaged EI is projected onto the viewing window with the correct direction and footprint, so that the element views are well integrated without deviating from each other.

[0051] In particular, when optimizing from (s, t) to (x c ,y c ), additional considerations regarding the interactions between individual elements must be taken into account when mapping ray positions from (u,v) to (xv,yv). Furthermore, the optimization of the ray direction mapping from (u,v) to (xv,yv) requires new design metrics that can accurately evaluate the quality of the directional sampling of the reconstructed light field of the 3D scene and its impact on the display system.

[0052] Figures 4A to 4D A three-dimensional view of various aspects of the InI-HMD system is illustrated, including the footprint on the viewing window and the imaging aperture (actual exit pupil) of the lenslet of the MLA, and the division of the EI on the microdisplay. It is worth noting that in some InI-HMD systems, more optical elements that can improve the overall display performance can be added in the display path, such as an adjustable relay group, which adds more complexity to the InI-HMD design, but can still follow the same design method as a system consisting of only the eyepiece. Without loss of generality, in some instances, these added optical elements can be combined with the eyepiece and are referred to as an "eyepiece group" in this article.

[0053] Figure 4A The footprint on the viewing window and the imaging aperture (actual exit pupil) of the lenslets of the MLA are illustrated. The LF-HMD system can be divided into M times N subsystems, where M and N are the total number of optical elements in the array optical body and equivalently the number of EIs rendered on the microdisplay in the horizontal and vertical directions, respectively. Figure 4B The division of EI on the microdisplay is illustrated. Figure 4C The diagram shows subsystems including EI, MLA lenslets, and a shared eyepiece set. Each subsystem represents a single imaging path from the EI through the corresponding optical unit of the array optical body and the shared eyepiece set. The imaging path of each subsystem is off-axis and non-rotationally symmetric with the main optical axis, such as Figure 4C Each subsystem can be configured as a zoom configuration in the optical design software. Figure 4D The diagram shows a partially overlapping virtual image of the EI on the virtual CDP.

[0054] In conventional 2D HMD designs, the HMD optical system is typically configured to trace rays from a shared exit pupil (or entrance pupil of the eye) back to the microdisplay, and no eye model is required. In contrast, the subsystem according to the disclosed embodiments is configured such that ray tracing starts from the microdisplay, or equivalently from the EI, towards the viewing window. In this way, ray tracing failures are avoided due to the fact that the projection of the aperture array of the optical elements of the array optical body onto the viewing window does not form a common shared exit pupil as in conventional HMDs. In addition, an ideal lens simulating the viewer's eye optical body or an established eye model (e.g., the Arizona eye model) is inserted, whose entrance pupil coincides with the viewing window to better optimize convergence and facilitate evaluation of the retinal image of the light field reconstruction. It should be noted that the use of a standard eye model is a non-limiting example that allows the design of a 3D system for a mass-produced product. In some embodiments, a personalized or customized eye model may be used.

[0055] Reference back Figures 4A to 4D As shown, the zoom configurations between the subsystems differ from each other mainly in the surface shape specifications and lateral positions of the corresponding optical units and the lateral positions of the corresponding EIs relative to the optical axis of the eyepiece group. In one embodiment, all lenslets in the MLA have the same surface shape and are arranged in a rectangular array with equal lens pitch in both the horizontal and vertical directions. For this embodiment, the lateral position (u, v) of each lenslet is determined only by the displacement Δp between adjacent lenslets. MLA or equivalently the lens pitch p MLA and the arrangement of the lenslets. Figure 3A In the same coordinate system as shown in Figure 2, for a given zoom configuration with index (m,n), the abscissa of the corresponding small lens can be expressed as

[0056]

[0057] Although the microdisplay is also divided into an M by N array of EIs, one EI per lenslet, the lateral position and size of each EI is more complex and depends on several other specifications of the display system. For example, the viewing window - which is not necessarily optically conjugate to the MLA and can be shifted longitudinally along the optical axis according to design requirements (see, for example, Figure 4A, where X indicates a non-conjugate relationship)—plays a vital role in partitioning and configuring the entire system into a series of subsystems. An array of imaging apertures is actually formed in the visual space, which is considered to be the exit pupil in the sense of a conventional display or imaging system. However, placing the viewer's eye at this exit pupil location will dictate that the viewer cannot see all EIs simultaneously because the imaging apertures are spatially separated. To ensure that all EIs can be properly seen at the viewing window, the microdisplay is partitioned so that the chief ray through the center of each EI of the entire display optical body (including the corresponding lenslets of the MLA and the shared eyepiece set) will intersect the optical axis at the center of the viewing window. v (for example, Figure 4B ). Under this constraint, the displacement Δp between adjacent EIs on the microdisplay is EI and the size p of the element image EI are equivalent and can be written as:

[0058]

[0059] In equation (2), g is the gap between the display panel and the MLA, l is the gap between the MLA and the middle CDP, and z IC is the gap between the intermediate CDP and the eyepiece group, and introduces z′ xp denoted by the distance between the eyepiece set and the imaged viewing window through the eyepiece set, which can be further given as:

[0060]

[0061] In equation (3), f ep is the equivalent focal length of the eyepiece assembly. Therefore, for a given subsystem unit with index (m,n), the abscissa corresponding to the EI center can be expressed as:

[0062]

[0063] The footprint size d of the beam from the EI pixel projected onto the viewing window through the entire optical body v - which determines the view density or equivalently the total number of views enclosed by the eye pupil - can be obtained by tracing ray beams emitted from the center point of the EI (e.g. Figure 4C The shadow ray bundle in the image is determined by the overall size of the viewing window D v can be determined by tracing the principal ray of the edge point of EI throughout the display optical volume (e.g., Figure 4C The ray coverage area diameter and the overall viewing window size can be expressed as:

[0064]

[0065] In equations (5) and (6), |...| represents an absolute value sign.

[0066] According to the paraxial geometry, the footprint diameter and viewing window size of any subsystem are both the same, and the footprints corresponding to the same field objects of different subsystems will intersect on the viewing window so that they share the same coordinates (x v ,y v ). For example, as described above, the chief ray at the center of each EI will intersect the optical axis at the center of the viewing window, so that for any subsystem, x v0 (m,n) and y v0 (m,n) are both equal to zero.

[0067] In an InI-HMD system, the EI is viewed as an array of spatially displaced virtual images observed from a viewing window. Figure 4D A simple example is illustrated where four adjacent EIs rendered on a microdisplay are imaged through their corresponding lenslets in the MLA and a shared eyepiece and projected onto the virtual CDP as four partially overlapping virtual EIs, as shown by the dashed boxes overlapping each other at the center shaded square. Figure 4E In the case of removing the rays associated with both EIs Figure 4D The displacement Δp between the centers of adjacent virtual EIs on the virtual CDP is the same as EIc is no longer equal to the size p of the virtual EI EIc , and their paraxial values ​​are further expressed as,

[0068]

[0069] In equations (7) and (8), Z CDP is the distance between the virtual CDP and the viewing window. For a given subsystem unit with index (m,n), the abscissa of the paraxial center of the corresponding virtual EI can be expressed as:

[0070]

[0071] Equations (7) and (9) essentially provide the paraxial calculation of the image size and the center position of each subsystem. The image size of the virtual EI on the virtual CDP is usually much larger than the displacement of the adjacent virtual EI, so that the virtual EIs on the virtual CDP will partially overlap each other. Figure 4DAs shown, the shadow overlap area on the virtual CDP corresponds to the area where all four virtual EIs overlap each other, and four different element views can be rendered, one for each EI, to reconstruct the light field of a sub-volume of the scene seen from the viewing window. Due to this nature, the overall FOV of the InI-HMD is stitched together from the sub-volumes rendered by each individual EI, and therefore cannot be directly calculated as in a conventional HMD. For simplicity, by referencing the center of the virtual image of the edge EI on the virtual CDP, the diagonal field of view (FOV) of the display system can be estimated as:

[0072]

[0073] The above steps demonstrate the disclosed method of modeling LF-HMD systems and analytical methods for calculating first-order relations of system parameters. These steps are different from modeling conventional 2D HMDs and are critical for developing appropriate optimization strategies.

[0074] As mentioned previously, optimization strategies for LF-HMDs require appropriate control and evaluation of optical aberrations that degrade the contrast and resolution of the virtual display or that individually and collectively distort the geometry of the 2D element images to account for aspects of the ray position sampling of the light field. Optimization strategies also require methods and metrics for controlling and evaluating optical aberrations that degrade the accuracy of directional ray sampling.

[0075] Ray Position Sampling Notes : Optimization of the ray position sampling of the light field function can be achieved by controlling the optical aberrations that affect the aberrations that reduce the contrast and resolution of the virtual display or distort the geometry of the 2D element image. It is helpful to obtain the EI of good imaging on the virtual CDP from the display panel through the corresponding small lens of the MLA and the eyepiece set.

[0076] The optimization strategy for ray position sampling according to the disclosed embodiment is multifaceted and includes optimizing the imaging process of each EI individually by each subsystem. For example, optimization constraints and performance metrics that can be used to optimize the 2D image conjugation of a conventional HMD can be used. The exact constraints and performance metrics vary greatly from system to system, depending largely on the complexity of the optical components utilized in the optical system of the InI-HMD. Examples of typical constraints include, but are not limited to, the minimum and maximum values ​​of element thickness or the spacing between adjacent elements, the total path length of the system, the allowable component size, the shape of each optical surface, the surface sag (sag) deviating from the reference surface, the type of optical material to be used, the tolerable amount of aberration, or the amount of focal length. Examples of performance metrics include, but are not limited to, root mean square (RMS) spot size, wavefront error, residual aberration amount, modulation transfer function, or acceptable image contrast. Using this initial source, the entire FOV of the LF display composed of a single EI is optimized separately, rather than being considered as a whole as in conventional HMD designs. However, such individual optimization for each EI ignores the corresponding connections between neighboring EIs, and more importantly ignores the relative position and size of the virtual EI with respect to the total FOV. Figure 4D In the InI-HMD shown, the imaged EI on the virtual CDP will be perfectly aligned with and partially overlapped with the neighboring EIs, where the paraxial relationship is given by equations (7) to (9). However, the image aberrations caused by the MLA and the eyepiece group will distort the virtual EI on the virtual CDP, and when reconstructing the light field of a 3D scene at different depths, the distorted virtual EI will cause the view convergence to potentially fail.

[0077] In order to illustrate the effect of induced distortion locally and globally on EI, two different types of constraints can be applied during optimization. The first constraint is the control of local distortion aberrations for each subsystem representing a single EI, which can be easily achieved by applying the existing distortion-related optimization constraints in the optical design software to each zoom configuration. The exact constraints of local distortion control vary greatly with the system, largely depending on the complexity of the optical components used in the optical system of the InI-HMD. Examples of typical controls of distortion include, but are not limited to, the maximum allowable deviation of the image height of the sampled object field from its paraxial value, the shape difference from the regular grid and the allowable percentage of the image height, the allowable magnification difference of different object fields, or the allowable shape deformation of the image from the desired shape, etc. These controls are typically applied to each subsystem separately as constraints to ensure that each subsystem forms an image with acceptable local distortion. These local distortion controls in each subsystem ensure that the size and shape of the virtual EI remain within the threshold level compared to its paraxial non-distorted image.

[0078] The second constraint is the control of global distortion, which is related to the lateral position of the virtual EI relative to the entire FOV of the reconstructed 3D scene. In order to optimize this global distortion, the chief ray of the central object field of each EI on the microdisplay should be tracked exclusively, and its intercept on the virtual CDP needs to be extracted and limited to a threshold level compared to its paraxial position in global coordinates. For a given subsystem with index (m,n), the global deviation of the central position of the virtual EI on the virtual CDP from its paraxial position can be quantified by a possible metric GD together with the corresponding constraints, which can be expressed as:

[0079]

[0080] The GD metric in equation (11) checks the angular deviation between the actual position of the chief ray of the central object field measured from the viewing window and the theoretical position. Figure 5 As shown, different GD metric values ​​can correspond to different amounts of distortion, thus allowing the system designer to select an appropriate target GD value for a particular system. For example, in a particular system (e.g., due to component cost and quality), a higher amount of distortion can be tolerated, which can inform the designer to select an appropriate target GD value.

[0081] Therefore, a constraint corresponding to a metric can be created by obtaining the maximum value of the metric across all EIs of all subsystems. By adding constraints and modifying constraint values ​​during the optimization process, the maximum allowed global distortion can be adjusted and optimized. Figure 5 The overall correlation between global distortion and GD value is further demonstrated by using the example of barrel distortion and keystone distortion simulated in a 40° by 40° InI-HMD system, where the CDP depth is 1 diopter. In each sub-figure, a completely theoretical FOV grid without global distortion (solid black line) and a FOV grid corresponding to a specific type and distortion value (slightly offset gray) are plotted, where the numbers represent the maximum and average values ​​of GD calculated according to equation (11) for a total of 11 by 11 EI sampling centers (intersections of the grid). For example, while the maximum GD produced by 1% barrel distortion is only 0.36°, the maximum GD produced by 5% barrel distortion is 1.78°. Obviously, when optimizing the ray position of the light field function, the GD value provides good control over the global distortion of the EI center position, whether as a regular distortion mode (such as barrel or pincushion distortion) or an unconventional distortion mode (such as keystone distortion)

[0082] Notes on Ray Directed Sampling: Due to the unique properties of LF-HMDs, the ray directions of the light field play a very important role in designing such display systems. Incorrect sampling of ray directions not only affects the integration of the EI, but can also result in an uneven number of element views of the reconstructed light field target, leading to misrepresented focus cues. In the case of severe pupil aberrations, it is even possible that the number of element views encompassed by the viewer's eye pupil is reduced to less than two, making the system become indistinguishable from a conventional stereoscopic display system and not properly rendering the actual light field.

[0083] As mentioned above, the viewing window is the location where all principal rays passing through the central pixels of all EIs intersect the optical axis, such as Figure 3A As shown, to ensure that all EIs can be correctly viewed simultaneously. The footprint of the ray beams projected from each element view, characterized by equation (5), jointly defines the viewing window, whose paraxial size is characterized by equation (6). In order to optimize the ray-directed sampling of the light field when designing a LF-HMD, it is necessary to provide appropriate constraints for the footprint of each element view projected on the viewing window, rather than directly optimizing the exit pupil of the element view. For ray directions with ideal light field functions or equivalently an InI-HMD without pupil aberration, the merged footprint diagram at the viewing window should have two significant features. First, the chief rays of different object fields on a single EI should converge at the center of the imaging aperture after passing through the corresponding lenslets of the MLA and the eyepiece group, and intersect the viewing window in a regular grid with uniform spacing, which is similar to the pixel array of the EI. Second, the chief rays from the same object field (relative to the EI of the chief ray itself) passing through its corresponding lenslets and eyepieces should converge at the viewing window, and the footprints of the ray beams from these pixels should form the same shape and perfectly overlap each other on the viewing window.

[0084] In order to optimize the ray-directed sampling of the light field when designing a LF-HMD, it is necessary to provide appropriate constraints on the footprint of each element view projected on the viewing window. To illustrate the impact of pupil aberration caused by ray footprint and direction on the viewing window during optimization, the disclosed optimization strategy

[0085] (1) extracting the exact footprint of the ray beam for any given pixel of a given EI on the viewing window; and

[0086] (2) Establish a metric function that appropriately quantifies any deviation of the ray footprint from its paraxial shape and position, so that constraints can be applied during the optimization process to control the deviation within a threshold level.

[0087] In some embodiments, for each given object field, four marginal rays are sampled through the lenslet aperture to avoid exhausting computation time during the optimization process. The coordinates of these marginal rays on the viewing window define the envelope of the ray footprint of the sampled field at a given EI in a given subsystem. For a sampled object field indexed (i, j) at a given EI corresponding to a sampling subsystem indexed (m, n), the deformation of the ray footprint from its paraxial shape can be quantified by the metric function PA using the following formula:

[0088]

[0089]

[0090] In equation (12), x' v and y' v are the actual positions of the upper edge rays of the viewing window obtained horizontally and vertically through actual ray tracing, and x v and v are their corresponding paraxial positions on the viewing window. k is the index of the four edge rays of the sampled object at a given EI corresponding to the sampling subsystem.

[0091] The metric PA in equation (12) quantifies the deformation of the ray footprint of a given ray bundle from its paraxial shape by examining the relative ratio of the average deviation distance between the actual and theoretical positions of the edge rays on the viewing window to the diagonal width of the paraxial footprint. By adding constraints and modifying the values ​​during the optimization process, the maximum allowed deviation and deformation of the footprint can be adjusted and optimized, or equivalently, the pupil aberration that affects the ray direction of the InI-HMD light field can be adjusted and optimized.

[0092] Figure 6 The overall relationship between the footprint diagram on the viewing window, the pupil aberration and the metric function value of the PA is illustrated. Therefore, a single constraint can be created by obtaining the maximum value of the metric from all sampled object fields on each sampling subsystem. The accompanying figure plots the simulated ray footprint diagram of the central field point of the EI centered on the optical axis of the InI-HMD with and without pupil aberration. For simplicity, in the simulation, the small lens is regarded as an ideal lens, and the eyepiece group is modeled using different aberration terms and amplitudes used as pupil aberrations (e.g., spherical aberration from 0.25 to 1 peak-to-valley (λPV), and tilt from 0.25 to 1°). Specifically, the diameter d of the theoretical footprint vis set to 1 mm. In each sub-figure, a theoretical footprint map without pupil aberration (a set of 10 by 10 star points forming a square surrounded by a solid line) and a deformed or displaced footprint map corresponding to a specific type and value of pupil aberration terms (the remaining star points) are depicted. The numbers below each sub-figure represent the PA values ​​calculated according to formula (12) for each case. It can be observed that due to the presence of pupil aberration, the real footprint maps can be significantly deformed (e.g., by pupil spherical aberration) or shifted (e.g., by tilt) from their theoretical footprint. Equation (12) provides a good estimate of the severity of pupil aberration in terms of PA based on the footprint map. For example, a PA produced by a 0.25λ PV pupil spherical aberration is only 0.059, while a PA produced by a 1λ PV pupil spherical aberration is as high as 0.23.

[0093] Using the above metrics, a system design can be performed to determine an optimal (or typical, desired, or target) design, which includes determining distances and angular alignments of components, dimensions of components (e.g., lenslet pitch, lenslet area, surface profile of lenslets and / or eyepiece, focal length, and aperture of lenslet array and / or eyepiece, etc.). The system can also include additional optical elements, such as relay optics, elements for folding or altering optical paths, and others, which can be subject to ray tracing and optimization as part of the system design.

[0094] Fig. 7A An example layout of a binocular setup of an InI-HMD system produced according to the disclosed design setup and optimization method is shown. Fig. 7A , the system is illustrated relative to the viewer's head. Figure 7B Provides information about Fig. 7A Additional details of the optical layout relative to its monocular setup (right eye), with key elements labeled. Fig. 7A The top section follows the Figure 7B It should be noted that the distances and specific numbers of components shown are provided as examples and not limitations to facilitate understanding of the disclosed technology. As shown, the optical body of the display path of particular interest includes three main subsections: a micro-InI unit including a high-resolution microdisplay, a custom-designed aspheric MLA, a custom aperture array, a lens array including four lenses (e.g., The tunable repeater set of the EL-10-30 tunable lens) and the free-form waveguide prism. The waveguide prism, essentially formed by 4 free-form surfaces denoted as S1 to S4, further magnifies the reconstructed intermediate micro scene and projects the light to the exit pupil or viewing window, where the viewer sees the magnified 3D scene reconstruction.

[0095] At the beginning of the design of the example configuration, the MLA and the relay eyepiece set were optimized separately to get a good starting point due to the complexity of the system. For the initial iterations of the MLA design, special attention was paid to limiting the edge rays to not exceed the edge of the lenslet to prevent crosstalk between adjacent EIs. The two surfaces of the lenslets were optimized as aspheric polynomials with coefficients up to order 6. In the initial iterations of the relay and eyepiece set design, the design was set up in reverse by tracing rays from the viewing window back to the eyepiece and relay. Each of the four freeform surfaces of the prism was described by an XY polynomial with x-plane symmetry and optimized using coefficients up to order 10.

[0096] After obtaining initial designs for both the MLA and the relay eyepiece assembly, the two parts were integrated together and an array of 7 by 3 zoom configurations was created. Figure 7C Shows The actual ray tracing from the sampled EI and a portion of the lenslets are plotted in the design configuration of the integrated display path. The viewing window is placed at the back focus of the freeform eyepiece. An ideal lens with a focal length equal to the focal length of the eye corresponding to the depth of the virtual CDP is inserted at the viewing window to simulate the retinal image of the EI. In the figure, only rays from the central pixel of each EI are traced. The MLA includes 17 times 9 identical lenslets with a lens pitch of 1 mm, and the microdisplay is divided into 17 times 9 EIs, each with 125 times 125 pixels. Considering the planar symmetry of the freeform prism, a total of 7 times 3 subsystems (corresponding lenslets of EI and MLA) in the upper half of the total FOV are sampled, and the distribution of the sampling subsystems is shown in Figure 7C , shown as a zoom lenslet. In each subsystem, 9 field points covering the entire EI are further sampled. In addition, in order to illustrate the DOF extension with changes in the optical focal length of the tunable lens as well as the virtual CDP depth, the system is also configured to optimize its performance at virtual CDP depths of 0, 1 and 3 diopters. Therefore, the focal lengths of the ideal lens and the tunable lens at the viewing window are adjusted accordingly to correctly focus the rays onto the image plane. In summary, the zoom of the 21 sampled MLA-EI subsystems and the zoom at different virtual CDP depths are combined to model the entire system with a total of 63 zoom configurations and a total of 567 field points.

[0097] Fig. 8AThe image contrast of the sampled 7 by 3 subsystems covering the full field of the display path is depicted, evaluated at a Nyquist angular frequency of 3 arc minutes or 10 cycles / degree (cpd), with the virtual CDP set to 1 diopter from the viewing window. In each subsystem, five object fields on its corresponding EI are sampled, and their contrast values ​​are represented by circles, each located at a specific position corresponding to each of the five object fields. Throughout the 30° by 18° FOV, the image contrast is well above the threshold of 0.2 for the Nyquist angular frequency, with an average value of 0.53.

[0098] Figure 8B The modulation transfer functions (MTFs) of on-axis field points on three EIs covering the entire FOV of the display path are plotted, corresponding to the lenslet centered on the optical axis (index (9,5)), the left corner lenslet (index (1,1)), and the right corner lenslet (index (17,1)), respectively. Figure 7C The MTF of the on-axis field point is also plotted for three EIs corresponding to the lenslets (index (9,5)) centered on the optical axis, but with their virtual CDPs away from the viewing window adjusted from 3 diopters to 0 diopters by adjusting the optical power of the tunable lens. It is clear that the optical system exhibits uniform image contrast and MTF performance over the entire FOV and depth range beyond 3 diopters, with a drop in image contrast evaluated at a Nyquist corner frequency of less than 0.15.

[0099] Fig.9A The global distortion grids for a sample of 7 by 3 subsystems covering the display path of the full display FOV are further plotted by extracting the principal ray coordinates of the central object field on the EI from each subsystem of the actual ray tracing of the design example, where the paraxial coordinates of the principal ray are plotted as a solid line grid and the true ray coordinates are plotted as asterisks. Although the display path suffers a small amount of keystone distortion due to the folded optical path, the global distortion of the full display field is generally relatively small, especially for designs involving freeform optics that are prone to introducing high-order distortion terms. The design target for the global distortion GD is set to 0.75°, which corresponds to a distortion of approximately 2% relative to the full FOV. All of the 7 by 3 subsystems were optimized within the design target, with an average value of GD of 0.22°, which corresponds to an average distortion of less than 1% relative to the full FOV.

[0100] Fig. 9B and 8C The ray coverage diagram at the viewing window before and after the optimization of the ray direction with respect to the light field function is compared. Fig. 9BThe envelope of the ray footprint over the viewing window of the 9 sampled object fields from the actual design setup before constraining the system's pupil aberrations is plotted for the on-axis lenslet (solid line, index (9,5)) and the edge lenslet located in the top right corner (solid line with diagonal dashes, index (17,1)). The ray footprint envelopes of these two lenslets are not only distorted but also severely separated. In contrast, Fig. 9C The merged envelope of the footprint plots extracted from the actual design setup after optimization is plotted. In this case, the footprints of the rays passing through the 9 sampled lenslets of the MLA are plotted (thin solid lines, indices (1,1), (9,1), (17,1), (1,3), (9,3), (17,3), (1,5), (9,5), and (17,5)). Figure 8C The theoretical envelope of the ray footprint for the same field on the lenslet obtained from the paraxial calculation is also plotted (thick solid line), which is perfectly aligned with each other on the lenslet and field as suggested above. The design target PA is set to 0.3, because the human visual system is less sensitive to ray direction than position. 19 of the 7 by 3 subsystems were optimized within the design target, with an average ratio of 0.145, well below the design target, which corresponds to an average throughput or size deviation of the actual footprint of less than 0.27, where as Fig. 9C As shown, deviations and deformations of the projected footprint are still acceptable.

[0101] In the experiment, the test results of the prototype InI-HMD system designed according to the disclosed technology were obtained by placing a camera in the viewing window and capturing an actual image of the display scene through the system. The test scene includes an inclined wall with a water drop texture, which spans a depth from about 500mm (2 diopters) to 1600mm (0.6 diopters), which is computationally rendered and displayed as a test target. By adjusting the focal depth of the camera from the near side (~600mm) to the middle part of the scene (~1000mm) and the far side (~1400mm), respectively, which simulates the adjustment of the eye accommodation from near to far distance, a central 15×7 element view rendered on the microdisplay and a real captured image of the rendered light field of this continuous 3D scene are obtained. The virtual CDP of the prototype is shifted and fixed at a depth of 750mm (1.33 diopters). Compared with the target, the part of the 3D scene within the same depth of the camera focus remains in sharp focus with high fidelity. In contrast, other parts of the 3D scene outside the camera's focal depth are blurry; the further the depth of the 3D scene is from the camera's focal point, the blurrier the part of the 3D scene is, similar to what we observe from real-world scenes. Such results clearly demonstrate the ability of the prototype designed according to the disclosed embodiments to render high-quality light field content and, more importantly, to present the correct focus cues to drive the viewer's eye accommodation.

[0102] Fig.10 A set of operations that can be performed to implement a method for designing a three-dimensional (3D) display system based on integrated imaging (InI) according to an example embodiment is illustrated. The method includes, at 1002, tracing a set of rays associated with a light field in an InI-based 3D system. The system includes: an array optical body; an array display device capable of generating a plurality of element images; a first reference plane representing a virtual central depth plane (CDP) on which light rays emitted by a point source on the display converge to form an image point; a second reference plane representing a viewing window for viewing a reconstructed 3D scene, and an optical subsegment representing a human eye model. For each element of the array display device and the array optical body, tracing starts from the array display device and is performed as an optical subsegment that passes through the array optical body and reaches. The method also includes, at 1004, adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values, wherein the first metric value corresponds to a ray-directed sampling of the light field.

[0103] In one example embodiment, the first metric value quantifies the deformation of the ray footprint of a given ray beam of the light field from its paraxial footprint. In another example embodiment, the first metric value is determined based on the relative ratio of the average deviation distance between the actual position and the theoretical position of the edge ray on the second reference plane and the diagonal width of the paraxial footprint. In yet another example embodiment, the first metric value is determined according to equation (12). For example, the first metric value can be determined based on the difference between the actual position of the edge rays on the viewing window obtained by ray tracing and their corresponding paraxial positions on the viewing window.

[0104] According to another example embodiment, in the above method, one or more parameters associated with the InI-based 3D system are adjusted to further obtain a second metric value within another predetermined value or range of values, wherein the second metric value corresponds to a ray position sampling of the light field, which accounts for deformation caused by at least adjacent elements of the array optical body. In an example embodiment, the second metric value is determined based on an angular deviation between an actual position and a theoretical position of a principal ray of a central object field measured from a second reference plane. In yet another example embodiment, the second metric value represents a global distortion metric. In yet another example embodiment, the second metric value is determined according to equation (11). For example, the second metric value is calculated as a deviation between a center position of a virtual element image of a plurality of element images on a virtual CDP and its paraxial position. In another example embodiment, one or more parameters associated with the InI-based 3D system are adjusted relative to the ray position sampling of the light field to additionally optimize the imaging of each EI individually.

[0105] In one example embodiment, the InI-based 3D system further comprises an eyepiece located between the array optical body and the second reference plane, and tracing the set of rays comprises tracing the set of rays through the eyepiece. In some embodiments, the array display device is a microdisplay device. In some embodiments, the array optical body comprises one or more lenslet arrays, each lenslet array comprising a plurality of microlenses. In another embodiment, the InI-based 3D system is an InI-based head mounted display (InI-based HMD) system.

[0106] In some embodiments, a predetermined value or range of values ​​for one or both of the first metric and the second metric is selected to achieve a particular image quality. In some embodiments, the predetermined value or range of values ​​for one or both of the first metric and the second metric represents a maximum or minimum value that provides an optimal design criterion with respect to the first metric or the second metric.

[0107] Fig.11 A set of operations that may be performed to improve the design of an integrated imaging optical system according to example embodiments is illustrated. The operations may be performed for an integrated imaging optical system including an array of lenslets that angularly samples the direction of a light field that produces an array of two-dimensional elemental images (EI), each elemental image representing a different viewing angle in three dimensions (3D). The method includes, at 1102, determining a first metric corresponding to a sampling of ray orientations of the light field; at 1104, determining a second metric corresponding to a sampling of ray positions of the light field, and at 1106, performing a ray tracing operation for determining a design of the integrated imaging optical system based on the first metric and the second metric. In some embodiments, the ray tracing operation is performed based on one or more constraints, the constraints including maintaining the first metric or the second metric at a corresponding value or range of values.

[0108] Fig.12 1 shows a block diagram of an apparatus 1200 that can be used to implement certain aspects of the disclosed technology. For example, Fig.12 The device 1200 may be used to receive, process, store, provide a display, and / or transmit various data and signals associated with the disclosed image sensor. The device 1200 includes at least one processor 1204 and / or controller, at least one memory 1202 unit in communication with the processor 1204, and at least one communication unit 1206 capable of exchanging data and information directly or indirectly via a communication link 1208 with other entities, devices, databases, and networks. The communication unit 1206 may provide wired and / or wireless communication capabilities according to one or more communication protocols, and thus it may include appropriate transmitters / receivers, antennas, circuits, and ports, as well as encoding / decoding capabilities necessary to properly transmit and / or receive data and other information. Fig.12The exemplary apparatus 1200 can be integrated as part of a larger component (e.g., a server, a computer, a tablet computer, a smartphone, etc.), which can be used to perform various calculations, methods, or algorithms disclosed herein, such as implementing ray tracing programs (e.g., Code V or Zemax) that are enhanced to accommodate the improvements disclosed in this document.

[0109] The processor 1204 may include a central processing unit (CPU) to control the overall operation of, for example, a host computer. In some embodiments, the processor 1204 implements this by executing software or firmware stored in the memory 1202. The processor 1204 may be or may include one or more programmable general or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), graphics processing units (GPUs), etc., or a combination of such devices.

[0110] The memory 1202 may be or may include the main memory of a computer system. The memory 1202 represents any suitable form of random access memory (RAM), read-only memory (ROM), flash memory, etc., or a combination of these devices. In use, the memory 1202 may contain, among other things, a set of machine instructions that, when executed by the processor 1204, causes the processor 1204 to perform operations to implement certain aspects of the currently disclosed technology.

[0111] It should be understood that various disclosed embodiments can be implemented individually or jointly in a device consisting of various optical components, electronic hardware and / or software modules and components. For example, these devices may include a processor, a memory unit, an interface that is mutually communicatively connected, and the scope may be from a desktop and / or laptop computer to a mobile device, etc. Processors and / or controllers may perform various disclosed operations based on the execution of a program code stored on a storage medium. For example, processors and / or controllers may communicate with at least one memory and at least one communication unit, and the communication unit may directly or indirectly exchange data and information through communication links with other entities, devices and networks. The communication unit may provide wired and / or wireless communication capabilities according to one or more communication protocols, so it may include appropriate transmitter / receiver antennas, circuits and ports, and the encoding / decoding capabilities that may be required to correctly send and / or receive data and other information.

[0112] In one embodiment, the various information and data processing operations described herein can be implemented by a computer program product, which is implemented in a computer-readable medium, including computer executable instructions, such as program codes executed by a computer in a network environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. Therefore, the computer-readable medium described in this application includes non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. Such specific sequences of executable instructions or associated data structures represent examples of corresponding actions for implementing the functions described in these steps or processes.

[0113] The foregoing description of the embodiments of the present invention has been presented for the purpose of illustration and description. The foregoing description is not intended to be exhaustive or to limit the embodiments of the present invention to the precise form disclosed, and modifications and variations are possible according to the above teachings, or can be obtained from the practice of the various embodiments. The embodiments discussed herein are selected and described to explain the principles and essence of the various embodiments and their practical applications, so that other technical personnel in the art can utilize the present invention in various embodiments and by various modifications such as those suitable for the intended specific use. Although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown or to perform all illustrated operations to achieve the desired results. The features of the embodiments described herein can be combined in all possible combinations of methods, devices, modules and systems.

Claims

1. A method for designing a three-dimensional (3D) display system based on integrated imaging (InI), the method comprising: Tracing a set of rays associated with a light field in an InI-based 3D system, the system comprising: Array optical body, An array display device capable of generating a plurality of element images, a first reference plane, representing a virtual central depth plane (CDP) on which light rays emitted by a point source on the display converge to form an image point, a second reference plane representing a viewing window for viewing the reconstructed 3D scene, and an optical sub-segment representing a model of a human eye, wherein for each element of the array display device and the array optical body, the tracing starts from the array display device and is performed through the array optical body and reaches the optical sub-segment; and One or more parameters associated with the InI-based 3D system are adjusted to obtain at least a first metric value within a predetermined value or range of values, wherein the first metric value corresponds to a ray-oriented sampling of the light field, quantifying the deformation of a ray coverage area of ​​a given ray beam of the light field and its paraxial coverage area, wherein the first metric value is determined based on a relative ratio of an average deviation distance between an actual position and a theoretical position of an edge ray on a second reference plane and a diagonal width of the paraxial coverage area, wherein one or more parameters associated with the InI-based 3D system are adjusted to further obtain a second metric value within another predetermined value or range of values, wherein the second metric value corresponds to a ray position sampling of the light field, which accounts for deformation caused by at least adjacent elements of the array optical body.

2. The method according to claim 1, wherein: The first metric value is determined based on a difference between an actual position of an edge ray on the viewing window obtained by ray tracing and a corresponding paraxial position of the edge ray on the viewing window.

3. The method according to claim 1, wherein: The second metric value is determined based on an angular deviation between an actual position and a theoretical position of a chief ray of the central object field measured from the second reference plane.

4. The method according to claim 1, wherein: The second metric value represents a global distortion metric.

5. The method according to claim 1, wherein: The second metric value is calculated as a deviation between a central position of a virtual element image of a plurality of element images on the virtual CDP and a paraxial position thereof.

6. The method according to claim 1, wherein: Adjustments are made to one or more parameters associated with the InI-based 3D system relative to the ray position sampling of the light field to further optimize imaging of each element image individually.

7. The method according to claim 1, wherein: The InI-based 3D system also includes an eyepiece located between the array optical body and the second reference plane, and wherein tracing the set of rays includes tracing the set of rays through the eyepiece.

8. The method according to claim 1, wherein: The array display device is a micro display device.

9. The method according to claim 1, wherein: The array optical body includes one or more small lens arrays, and each small lens array includes a plurality of micro lenses.

10. The method according to claim 1, wherein: The InI-based 3D system is an InI-based head mounted display (InI-based HMD) system.

11. The method according to claim 1, wherein: A predetermined value or range of values ​​of one or both of the first metric and the second metric is selected to achieve a particular image quality.

12. The method according to claim 1, wherein: The predetermined value or range of values ​​of one or both of the first metric and the second metric represents a maximum value or a minimum value that provides an optimal design criterion with respect to the first metric or the second metric.

13. A method for improving the design of an integrated imaging optical system, the system comprising an array of lenslets that angularly samples the direction of a light field producing an array of two-dimensional elemental images (EI), each of which represents a different viewing angle in three dimensions (3D), the method comprising: determining a first metric corresponding to a ray-directed sampling of the light field; determining a second metric corresponding to ray position samples of the light field; and A ray tracing operation is performed based on the first metric and the second metric to determine the design of the integrated imaging optical system, wherein the first metric value corresponds to a ray orientation sampling of the light field, quantifying the deformation of a ray coverage area of ​​a given ray beam of the light field and its paraxial coverage area, the first metric value is determined based on a relative ratio of an average deviation distance between an actual position and a theoretical position of an edge ray on a second reference plane and a diagonal width of the paraxial coverage area, wherein the second reference plane represents a viewing window for viewing a reconstructed 3D scene, and wherein the second metric value corresponds to a ray position sampling of the light field, which accounts for deformation caused by adjacent elements of at least an array optical body of the integrated imaging optical system.

14. The method according to claim 13, wherein: The ray tracing operation is performed based on one or more constraints, the constraints including maintaining the first metric or the second metric at a corresponding value or range of values.

15. The method according to claim 13, wherein: The integrated imaging optical system includes an InI-based head mounted display (InI-based HMD).

16. An apparatus for designing a three-dimensional (3D) display system based on integrated imaging (InI), comprising: processor; as well as A memory comprising processor executable code which, when executed by the processor, causes the processor to: Tracing a set of rays associated with a light field in an InI-based 3D system, the system comprising: Array optical body, An array display device capable of generating a plurality of element images, a first reference plane, representing a virtual central depth plane (CDP) on which light rays emitted by a point source on the display converge to form an image point, a second reference plane representing a viewing window for viewing the reconstructed 3D scene, and an optical sub-segment representing a model of a human eye, wherein for each element of the array display device and the array optical body, the tracing starts from the array display device and is performed through the array optical body and reaches the optical sub-segment; and One or more parameters associated with the InI-based 3D system are adjusted to obtain at least a first metric value within a predetermined value or range of values, wherein the first metric value corresponds to a ray-oriented sampling of the light field, quantifying the deformation of a ray coverage area of ​​a given ray beam of the light field and its paraxial coverage area, wherein the first metric value is determined based on a relative ratio of an average deviation distance between an actual position and a theoretical position of an edge ray on a second reference plane and a diagonal width of the paraxial coverage area, wherein one or more parameters associated with the InI-based 3D system are adjusted to further obtain a second metric value within another predetermined value or range of values, wherein the second metric value corresponds to a ray position sampling of the light field, which accounts for deformation caused by at least adjacent elements of the array optical body.

17. A non-transitory computer readable medium comprising instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to perform a method for designing an integrated imaging (InI) based three-dimensional (3D) display system, the instructions comprising: Program code for tracing a set of rays associated with a light field in an InI-based 3D system, wherein the system comprises: Array optical body, An array display device capable of generating a plurality of element images, a first reference plane, representing a virtual central depth plane (CDP) on which light rays emitted by a point source on the display converge to form an image point, a second reference plane representing a viewing window for viewing the reconstructed 3D scene, and an optical sub-segment representing a model of a human eye, wherein for each element of the array display device and the array optical body, the tracing starts from the array display device and is performed through the array optical body and reaches the optical sub-segment; and Program code for adjusting one or more parameters associated with the InI-based 3D system to obtain at least a first metric value within a predetermined value or range of values, wherein the first metric value corresponds to a ray-oriented sampling of the light field, quantifying the deformation of a ray coverage area of ​​a given ray beam of the light field and its paraxial coverage area, wherein the first metric value is determined based on a relative ratio of an average deviation distance between an actual position and a theoretical position of an edge ray on a second reference plane and a diagonal width of the paraxial coverage area, wherein one or more parameters associated with the InI-based 3D system are adjusted to further obtain a second metric value within another predetermined value or range of values, wherein the second metric value corresponds to a ray position sampling of the light field, which accounts for deformation caused by at least adjacent elements of the array optical body.

Images