Large eye box light field holographic display system

Abstract

The invention relates to a large eye box light field holographic display system, the display system comprises a controller, an optical imaging system and a spatial light modulator, and a virtual holographic surface is a Fourier surface of a human eye retina. The method comprises the following steps: firstly, recording light field information of a three-dimensional object by using a light field camera to obtain a plurality of two-dimensional observation sub-graphs containing different directions; according to the angles corresponding to the obtained two-dimensional observation sub-images, the frequency spectrum information of the two-dimensional observation sub-images is placed at the spatial positions corresponding to the virtual holographic surfaces, and a frequency spectrum extended hologram is synthesized through complex amplitude addition; and then a hologram synthesized by the virtual holographic surface is subjected to back propagation through an optical imaging system to obtain a hologram displayed on the spatial light modulator, and the hologram on the spatial light modulator is projected on the retina of the human eye to realize reconstruction of light field information of a three-dimensional object. Compared with the prior art, the invention has the advantages of realizing the imaging effect of eye box enlargement and the like.

Description

Technical Field

[0001] The invention relates to the field of holographic display, and in particular to a large-eye box light field holographic display system. Background Art

[0002] Traditional flat-panel displays can usually only display two-dimensional images and cannot accurately convey the depth and position information of objects, which leads to difficulties in virtual reality, three-dimensional modeling and other applications that require accurate spatial perception, because they cannot present the distance and relative position between objects, limiting the user's interaction and understanding of the content. In recent years, light field holographic display technology has attracted much attention in the display of multi-angle and multi-depth stereoscopic images, and is considered to be one of the most ideal technologies for achieving true three-dimensional reconstruction.

[0003] Light field display focuses on capturing and simulating the direction and intensity of light to calculate three-dimensional images from different viewpoints. Its core is to capture the information of light propagation and generate images through mathematical calculations; holographic images are produced by interfering the light waves of the object with the reference light waves, which can directly modulate the wavefront information of the light and provide complete clues to the depth of the three-dimensional scene, which enables the observer to see images with real depth. Therefore, combining light field holographic display technology can provide richer image information and provide a more realistic visual experience in VR, AR and vehicle-mounted HUD display applications.

[0004] Holographic display technology can complete the projection display of the human eye retina by converging the modulated light beam into the pupil of the human eye. However, due to the size limitation of the device, the current liquid crystal spatial light modulator has a small diffraction angle. When the field of view is fixed, the size of the eye box is very limited, and image information can only be received in a small range, which limits the development of holographic display technology. Summary of the invention

[0005] The purpose of the present invention is to provide a large-eye box light field holographic display system.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A large eye box light field holographic display system comprises an optical imaging system and a spatial light modulator, wherein the optical imaging system is arranged between a human eye and the spatial light modulator, and the Fourier surface of the retina of the human eye is defined as a virtual holographic surface.

[0008] The system further includes a light field camera and a controller, wherein the controller is connected to the light field camera and the spatial light modulator, respectively, and is configured to perform the following steps:

[0009] Acquire light field information of a three-dimensional object collected by a light field camera, wherein the light field information includes a plurality of two-dimensional observation sub-images in different directions;

[0010] According to each two-dimensional observation sub-image and its corresponding angle, the spectrum information is placed at the corresponding spatial position of the virtual holographic surface, and the spectrum-expanded hologram is synthesized by complex amplitude addition;

[0011] The hologram synthesized by the virtual holographic surface is back-propagated to a spatial light modulator via an optical imaging system to obtain a hologram for display;

[0012] The spatial light modulator is controlled to project the displayed hologram onto the retina of the human eye to reconstruct the light field information of the three-dimensional object.

[0013] The system also includes an optical waveguide component for performing pupil expansion processing on the virtual holographic surface.

[0014] The spectrum expansion process of the hologram synthesized by complex amplitude addition spectrum expansion is specifically: different light field images in the spatial domain are phase-shifted by displacement in the frequency domain, thereby corresponding to different angle information of the light field images.

[0015] The sum of the complex amplitude accumulation is specifically:

[0016]

[0017] FU i (u, v) = W (u, v) · U i (u, v), i=1,...n

[0018] Where: FU(u, v) is the sum of the complex amplitudes, U i (u, v), FU i (u, v) are the spectrum information of the light field image and the complex amplitude obtained after filtering, W(u, v) is a low-pass filter, n represents the number of light field images, and u and v are the horizontal and vertical coordinates in the frequency domain.

[0019] The mathematical expression of the hologram for display is specifically:

[0020] h comp (x,y)=F-1{FU(u,v)·H(u,v)}

[0021] Where: h comp (x, y) is the complex amplitude hologram finally loaded by the spatial light modulator, F -1 is the back propagation operation of the optical system, and H(u, v) is the transfer function of the optical system.

[0022] The optical waveguide component includes an in-coupling element, a steering element and an out-coupling element. The spectrum-expanded hologram is incident on the steering element through the in-coupling element to undergo total reflection and propagate to the out-coupling element region, where the eye box is enlarged.

[0023] The optical imaging system includes an aperture and a convex lens, the focal length of the convex lens is f 01 , the distance from the spatial light modulator is f 01 .

[0024] The optical imaging system comprises an aperture and two convex lenses, wherein the spatial light modulator, the first convex lens, the aperture and the second convex lens are arranged in sequence, wherein the focal length of the first convex lens is f 11 The distance between the spatial light modulator and the aperture and the first convex lens is f 11 , the focal length of the second convex lens is f 12 , the distance between the aperture and the second convex lens is greater than f 12 .

[0025] The optical imaging system comprises an aperture and three convex lenses, wherein the spatial light modulator, the first convex lens, the aperture, the second convex lens and the third convex lens are arranged in sequence, wherein the focal length of the first convex lens is f 21 The distance between the spatial light modulator and the aperture and the first convex lens is f 21 , the focal length of the second convex lens is f 22 , the distance between the aperture and the second convex lens is f 22 , the focal length of the third convex lens is f 23 , the distance between the second convex lens and the third convex lens is f 23 .

[0026] The light field camera is composed of multiple cameras, and all the cameras are placed at different positions on the same plane.

[0027] Compared with the prior art, the present invention has the following beneficial effects: the spectrum information of different light field images is placed at the spatial position corresponding to the virtual holographic surface, and a spectrum-expanded hologram is synthesized by complex amplitude addition; it is then reversely propagated through the optical imaging system to obtain a hologram displayed on the spatial light modulator, and the image is diffracted and propagated and finally projected onto the retina of the human eye to realize the reconstruction of the light field information of the three-dimensional object and achieve the imaging effect of enlarged eye box. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the optical path of the holographic display system;

[0029] Figure 2 A schematic diagram of the optical path of the large-eye box light field holographic display system of the present invention;

[0030] Figure 3 A schematic diagram of the optical path of a large-eye box light field holographic display system using an optical waveguide component;

[0031] Figure 4 Schematic diagram of the structure of the optical waveguide component, where (a) the incident light and the outgoing light propagate in the same direction, and (b) the incident light and the outgoing light propagate in the opposite direction;

[0032] Figure 5 is an optical imaging system, wherein (a) is a single-lens solution, (b) is a double-lens solution, and (c) is a triple-lens solution;

[0033] Figure 6 Schematic diagram of the principle of retinal imaging of the human eye, where (a) shows that the virtual holographic surface is located on the Fourier surface of the retina of the human eye, (b) shows that the virtual holographic surface is located before the Fourier surface of the retina of the human eye, and (c) shows that the virtual holographic surface is located after the Fourier surface of the retina of the human eye;

[0034] Figure 7 Schematic diagram of the light field image acquisition principle, where (a) is multi-camera acquisition and (b) is single-camera microlens array acquisition. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0036] like Figure 1 As shown, the existing holographic retinal imaging technology can often achieve imaging of a single viewpoint. When the field of view is fixed, it cannot guarantee a sufficient eye box size and the viewing range is small. In order to achieve imaging display with enlarged eye box, it also has complete depth clues and multi-angle information of the object.

[0037] A large eye box light field holographic display system, such as Figure 2 As shown, it includes an optical imaging system 3 and a spatial light modulator 4. The optical imaging system 3 is arranged between the human eye 1 and the spatial light modulator 4. The Fourier surface of the retina on the human eye 1 is defined as a virtual holographic surface 2. The human eye can be understood as a human eye optical system.

[0038] The system further includes a light field camera and a controller, wherein the controller is connected to the light field camera and the spatial light modulator 4, respectively, and is configured to perform the following steps:

[0039] Acquire light field information of a three-dimensional object collected by a light field camera, wherein the light field information includes a plurality of two-dimensional observation sub-images in different directions;

[0040] According to each two-dimensional observation sub-image and its corresponding angle, the spectrum information is placed at the corresponding spatial position of the virtual holographic surface 2, and the spectrum-expanded hologram is synthesized by complex amplitude addition;

[0041] The hologram synthesized by the virtual holographic surface 2 is back-propagated to the spatial light modulator 4 through the optical imaging system 3 to obtain a hologram for display;

[0042] The spatial light modulator 4 is controlled to project the displayed hologram onto the retina of the human eye to achieve reconstruction of the light field information of the three-dimensional object.

[0043] The system also includes an optical waveguide component for expanding the pupil of the virtual holographic surface, such as Figure 3 As shown. The optical waveguide component includes an in-coupling element 1-1, a steering element and an out-coupling element 1-2. The spectrum-expanded hologram is incident on the steering element through the in-coupling element, undergoes total reflection and propagates to the out-coupling element area. The eye box is enlarged in the out-coupling element area. At the rear side of the out-coupling element area, the human eye 1 can view a large-size continuous light field image.

[0044] Furthermore, the optical waveguide component can be any one of PB liquid crystal grating, surface relief grating or volume holographic grating according to the principle; it can be a transmission grating or a reflection grating according to the light wave emission mode.

[0045] Specifically, in the optical waveguide component, the in-coupling element and the out-coupling element may be reflective gratings with surface relief gratings. Figure 4 As shown in part (a) of FIG. 1 , the in-coupling element 1-1 and the out-coupling element 1-2 are located on different sides of the optical waveguide substrate, and the incident light and the outgoing light propagate in the same direction; Figure 4 As shown in part (b), the in-coupling element 1-1 and the out-coupling element 1-2 are located on the same side of the optical waveguide substrate, and the incident light and the outgoing light propagate in opposite directions.

[0046] Specifically, the virtual holographic surface 2 is placed in the coupling region of the optical waveguide component, the spectrum-expanded light field information enters the coupling element 1-1, the light that meets the total reflection condition of the optical waveguide is coupled into the optical waveguide sheet, and propagates to the steering element in the optical waveguide sheet, that is, after multiple diffractions and reflections in the horizontal direction (i.e., horizontal pupil expansion), it reaches the coupling region, and the light is coupled out of the waveguide while completing the longitudinal pupil expansion through the coupling element 1-2, to obtain the two-dimensional expanded light field information. At this time, a large eye movement range, large size, and continuously viewable holographic display can be achieved after the coupling region of the optical waveguide.

[0047] The spectrum expansion process of the hologram synthesized by complex amplitude addition spectrum expansion is specifically as follows: different light field images in the spatial domain are phase-shifted by displacement in the frequency domain, thereby corresponding to different angle information of the light field image.

[0048] The sum of complex amplitudes is specifically:

[0049]

[0050] FU i (u, v) = W(u, v) · U i (u, v), i = 1,...n

[0051] Where: FU(u, v) is the sum of complex amplitudes, U i (u, v), FU i (u, v) are respectively the spectral information of the light field image and the complex amplitude obtained after filtering it, W(u, v) is a low-pass filter, n represents the number of light field images, and u and v are respectively the horizontal and vertical coordinates in the frequency domain.

[0052] The mathematical expression of the hologram for display is specifically:

[0053] h comp (x, y) = F -1 {FU(u, v) · H(u, v)}

[0054] Where: h comp (x, y) is the complex amplitude hologram finally loaded on the spatial light modulator, F -1 is the inverse propagation operation of the optical system, and H(u, v) is the optical system transfer function.

[0055] In one embodiment, as shown in part (a) of Figure 5 the optical imaging system 3 includes a diaphragm 3-2 and a convex lens 3-1, and the focal length of the convex lens 3-1 is f 01 , and the distance from the spatial light modulator 4 is f 01 .

[0056] In another embodiment, as shown in part (b) of Figure 5 the optical imaging system 3 includes a diaphragm 3-2 and two convex lenses. The spatial light modulator 4, the first convex lens 3-1, the diaphragm 3-2, and the second convex lens 3-3 are arranged in sequence. Among them, the focal length of the first convex lens 3-1 is f 11 , and the distances from the spatial light modulator 4 and the diaphragm 3-2 to the first convex lens 3-1 are both f 11 , the focal length of the second convex lens 3-3 is f 12 , and the distance from the diaphragm 3-2 to the second convex lens 3-3 is greater than f 12 .

[0057] In yet another embodiment, as shown in Figure 5As shown in part (c) of FIG. 1 , the optical imaging system 3 includes an aperture 3-2 and three convex lenses, and the spatial light modulator 4, the first convex lens 3-1, the aperture 3-2, the second convex lens 3-3 and the third convex lens 3-4 are arranged in sequence, wherein the focal length of the first convex lens 3-1 is f 21 The distance between the spatial light modulator 4 and the aperture 3-2 and the first convex lens 3-1 is f 21 , the focal length of the second convex lens 3-3 is f 22 , the distance between the aperture 3-2 and the second convex lens 3-3 is f 22 , the focal length of the third convex lens 3-3 is f 23 , the distance between the second convex lens 3-3 and the third convex lens 3-3 is f 23 .

[0058] The optical imaging systems all satisfy the thin lens imaging formula:

[0059] Specifically, the distance between the reconstructed image after passing through the optical imaging system and the principal plane of the optical lens group (or single lens) is u, and the distance between the virtual image after passing through the optical lens group (or single lens) and the principal plane of the optical lens group (or single lens) is v, and u and v satisfy the following relationship:

[0060]

[0061] Wherein: f is the equivalent focal length of the optical imaging system 3 and satisfies 0<u<f.

[0062] Furthermore, the spatial light modulator:

[0063] It may be a phase-type spatial light modulator; it may be an amplitude-type spatial light modulator; it may be a complex-amplitude-type spatial light modulator.

[0064] Specifically, in this embodiment, a phase-type spatial light modulator is used to load the complex amplitude light field hologram, and the complex amplitude hologram is encoded into a phase-type hologram and then loaded onto the phase-type modulation device.

[0065] Specifically, double-phase encoding is used, and the specific steps are as follows:

[0066] make Encoding a complex amplitude hologram into two pure phase holograms Among them, B is a constant term, A max =max(A) is the maximum amplitude of h(x,y), and They are:

[0067] Then, a checkerboard is used to sample two pure-phase holograms, and the two decomposed pure-phase holograms are superimposed to obtain the phase-type hologram;

[0068] Specifically, a checkerboard pattern M 1 , M 2 are respectively expressed as:

[0069]

[0070]

[0071] and M 1 and M 2 satisfy M 1 (iΔx, jΔy)+M 2 (iΔx, jΔy) = 1.

[0072] Where Δx and Δy are pixel intervals and should be consistent with the pixel intervals of the hologram generation unit used; the obtained phase-type hologram is expressed as:

[0073] Furthermore, the virtual holographic plane and the retina plane of the human eye are in a Fourier transform relationship, as shown in part (a) of Figure 6 ; assuming the focal length of the human eye is f (f = 17mm - 23mm), the distance between the virtual holographic plane 2 and the human eye 1 is z;

[0074] In addition, in some other embodiments, the virtual holographic plane can be located in front of the Fourier plane of the human eye retina, that is, z > f. By adding a converging spherical wave quadratic phase factor to the virtual holographic plane, for example: exp(-ik(x 2 +y 2 ) / 2f), the phase compensation of the complex amplitude information is performed, so as to obtain a clear reconstructed image of the light field images in different directions on the human eye retina plane, as shown in part (b) of Figure 6 ;

[0075] Or, in some other embodiments, the virtual holographic plane can be located behind the Fourier plane of the human eye retina, that is, z < f. By adding a diverging spherical wave quadratic phase factor to the virtual holographic plane, for example: exp(ik(x 2 +y 2 ) / 2f), the phase compensation of the complex amplitude information is performed, so as to obtain a clear reconstructed image of the light field images in different directions on the human eye retina plane, as shown in part (c) of Figure 6 ;

[0076] Furthermore, the acquisition method of the light field information:

[0077] Specifically, multiple cameras may be placed at different positions on the same plane to collect intensity and depth clues of a three-dimensional object at different angles and directions, such as Figure 7 As shown in part (a) of

[0078] Specifically, multiple sub-lenses in a microlens array on a single camera can receive scattered light from different angles of a three-dimensional object and project it on a CCD plane at different angles, thereby recording the light field information of the three-dimensional object. Figure 7 As shown in part (b) of .

[0079] The implementation process of the present invention is:

[0080] A light field camera is used to record the light field information of a three-dimensional object, that is, a single sub-lens in a microlens array receives scattered light from different angles of the three-dimensional object, and projects it on a CCD plane at different angles to obtain a plurality of two-dimensional observation sub-images containing different directions; according to the angles corresponding to the obtained two-dimensional observation sub-images, its spectrum information is placed on the corresponding spatial positions of a virtual holographic surface (i.e., the Fourier plane of the human eye retina), and a spectrum-expanded hologram is synthesized by complex amplitude addition; the hologram synthesized by the virtual holographic surface is reversely propagated through an optical imaging system to obtain a hologram displayed on a spatial light modulator, and the aforementioned complex amplitude hologram is encoded into a phase form using a double-phase encoding method, and loaded on a phase-type spatial light modulator, and the hologram is finally projected on the human eye retina to realize the reconstruction of the light field information of the three-dimensional object.

[0081] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A large eye box light field holographic display system, comprising an optical imaging system (3) and a spatial light modulator (4), wherein the optical imaging system (3) is arranged between a human eye (1) and the spatial light modulator (4), and the Fourier surface of the retina on the human eye (1) is defined as a virtual holographic surface (2); It is characterized in that The system further comprises a light field camera and a controller, wherein the controller is connected to the light field camera and the spatial light modulator (4) respectively and is configured to perform the following steps: Acquire light field information of a three-dimensional object collected by a light field camera, wherein the light field information includes a plurality of two-dimensional observation sub-images in different directions; According to each two-dimensional observation sub-image and its corresponding angle, the spectrum information is placed at the corresponding spatial position of the virtual holographic surface (2), and a spectrum-expanded hologram is synthesized by complex amplitude addition; The hologram synthesized by the virtual holographic surface (2) is back-propagated to the spatial light modulator (4) via the optical imaging system (3) to obtain a hologram for display; The spatial light modulator (4) is controlled to project the displayed hologram onto the retina of the human eye to achieve reconstruction of the light field information of the three-dimensional object.

2. A large-eye box light field holographic display system according to claim 1, It is characterized in that The system also includes an optical waveguide component for performing pupil expansion processing on the virtual holographic surface.

3. A large-eye box light field holographic display system according to claim 2, It is characterized in that The optical waveguide component includes an in-coupling element, a steering element and an out-coupling element. The spectrum-expanded hologram is incident on the steering element through the in-coupling element to undergo total reflection and propagate to the out-coupling element region, where the eye box is enlarged.

4. The large-eye box light field holographic display system according to claim 1, It is characterized in that The spectrum expansion process of the hologram synthesized by complex amplitude addition spectrum expansion is specifically: different light field images in the spatial domain are phase-shifted by displacement in the frequency domain, thereby corresponding to different angle information of the light field images.

5. A large-eye box light field holographic display system according to claim 4, It is characterized in that The sum of the complex amplitude accumulation is specifically: FU i (u,v)=W(u,v)·U i (u,v),i=1,…n Where: FU(v,v) is the sum of the complex amplitudes, U i (u,v),FU i (u, v) are the spectrum information of the light field image and the complex amplitude obtained after filtering, W(u, v) is a low-pass filter, n represents the number of light field images, and u and v are the horizontal and vertical coordinates in the frequency domain.

6. A large-eye box light field holographic display system according to claim 5, It is characterized in that The mathematical expression of the hologram for display is specifically: h comp (x,y)=F -1 {FU(u,v)·H(u,v)} Where: h comp (x, y) is the complex amplitude hologram finally loaded by the spatial light modulator, F -1 is the back propagation operation of the optical system, and H(u,v) is the transfer function of the optical system.

7. The large-eye box light field holographic display system according to claim 1, It is characterized in that The optical imaging system (3) comprises an aperture and a convex lens, the focal length of the convex lens is f 01 , the distance from the spatial light modulator (4) is f 01 .

8. The large-eye box light field holographic display system according to claim 1, It is characterized in that The optical imaging system (3) comprises an aperture and two convex lenses, wherein the spatial light modulator (4), the first convex lens, the aperture and the second convex lens are arranged in sequence, wherein the focal length of the first convex lens is f 11 The distance between the spatial light modulator (4) and the aperture and the first convex lens is f 11 , the focal length of the second convex lens is f 12 , the distance between the aperture and the second convex lens is greater than f 12 .

9. The large-eye box light field holographic display system according to claim 1, It is characterized in that The optical imaging system (3) comprises an aperture and three convex lenses, wherein the spatial light modulator (4), the first convex lens, the aperture, the second convex lens and the third convex lens are arranged in sequence, wherein the focal length of the first convex lens is f 21 The distance between the spatial light modulator (4) and the aperture and the first convex lens is f 21 , the focal length of the second convex lens is f 22 , the distance between the aperture and the second convex lens is f 22 , the focal length of the third convex lens is f 23 , the distance between the second convex lens and the third convex lens is f 23 .

10. The large-eye box light field holographic display system according to claim 1, It is characterized in that The light field camera is composed of multiple cameras, and all the cameras are placed at different positions on the same plane.

Images