Hybrid multi-layer compressed light field near-eye display system and image generation method thereof

Abstract

The invention discloses a mixed multi-layer compressed light field near-to-eye display system and an image generation method thereof. The system comprises a first multiplication display, a second multiplication display, a variable-focus lens, a doublet achromatic lens, a control unit and a gasket, the two multiplication displays modulate the intensity of the light; the variable-focus lens and the doublet achromatic lens are arranged in light paths of the first multiplication display and the second multiplication display; the control unit is electrically connected with the first multiplication display, the second multiplication display and the variable-focus lens; the control unit controls the variable-focus lens to be in at least two different focal length states so as to meet frequency switching of the human vision persistence effect. The upper limit of the depth of field of the system is obviously expanded, the thickness of the system is greatly reduced and the brightness loss is reduced by about 50% compared with the traditional four-layer compressed light field display while the adjustment range of 2.0 D and above is realized, and a new way is provided for developing light and thin near-to-eye display equipment with high visual comfort.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of compressed light field near-eye display, and particularly relates to a hybrid multi-layer compressed light field near-eye display system and an image generation method thereof. BACKGROUND

[0002] Near-eye display is the core display form of virtual reality (VR) and augmented reality (AR) systems, which provides immersive visual experience by placing a miniature display and imaging optical system close to the user's eyes. The existing near-eye display technology generally has a vergence-accommodation conflict (VAC) problem: the physical focal plane of the display is fixed, which causes the mismatch between the accommodation distance and the convergence angle of the human eye, resulting in visual fatigue and dizziness. In addition, the depth of field range of the traditional system is limited, which makes it difficult to maintain clear imaging at different depth positions at the same time, affecting the realism of three-dimensional display. To improve visual comfort, researchers have proposed various solutions. Among them, multi-layer compressed light field display uses multiple liquid crystal panels to modulate light flux, which can reconstruct the light field under a limited number of layers, and has the advantages of compact structure and easy implementation. However, the effective depth of field of this method is limited by the number of physical layers, layer spacing, and system bandwidth product. Increasing the number of layers can expand the depth of field, but it significantly increases the thickness and complexity of the device, which is not conducive to the lightweight of head-mounted displays. SUMMARY

[0003] The application aims to solve the problems of severe vergence-accommodation conflict, limited depth of field range, and complex system structure in traditional near-eye display. The application proposes a hybrid multi-layer compressed light field near-eye display system to achieve multi-focal plane display without increasing the number of physical layers, expand the depth of field range, and improve visual comfort.

[0004] The technical scheme of the application is a hybrid multi-layer compressed light field near-eye display system, which includes a first multiplicative display, a second multiplicative display, a variable focus lens, a double cemented achromatic lens, a control unit, and a spacer.

[0005] The first multiplicative display and the second multiplicative display modulate the intensity of light. The variable focus lens and the double cemented achromatic lens are arranged in the optical path of the first multiplicative display and the second multiplicative display. The control unit is electrically connected to the first multiplicative display, the second multiplicative display, and the variable focus lens.

[0006] The control unit controls the variable focus lens to switch between at least two different focal length states at a frequency that meets the human visual persistence effect. When the variable focus lens is in different focal length states, the first multiplicative display and the second multiplicative display are loaded with image content that matches the depth of the corresponding virtual imaging plane. Through time division multiplexing, the display images from different focal length states are integrated in time sequence, and at least four virtual imaging planes are formed in vision.

[0007] Furthermore, the multiplication display is a liquid crystal display, or a multiplication modulator group consisting of two liquid crystal panels.

[0008] Furthermore, the variable focus lens is a liquid variable focus lens.

[0009] Furthermore, the spacer controls the spacing between the two multiplication displays and the spacing between the first multiplication display and the cemented doublet achromatic lens.

[0010] The image generation method of the present invention using a hybrid multilayer compressed light field near-eye display system includes the following steps:

[0011] S1: Obtain the four-dimensional light field data of the 3D scene to be displayed;

[0012] S2: Decompose the four-dimensional light field data of the 3D scene into light fields corresponding to each display state of the system to obtain the four-dimensional light field; the display state is determined by the combination of the focal length state of the variable focal lens and the corresponding multiplication display. Using the rank-1 light field decomposition algorithm, calculate the attenuation layer map corresponding to the first multiplication display and the second multiplication display under the light field conditions corresponding to each display state.

[0013] S3: Set the optimization objective function, and use a non-negative update algorithm to iteratively optimize the attenuation layers corresponding to the first multiplication display and the second multiplication display under the corresponding light field conditions of each display state, so as to ensure that all pixel values ​​are non-negative;

[0014] S4: The optimized multi-layer attenuation layer diagram is loaded onto the first multiplication display and the second multiplication display respectively according to their timing correspondence.

[0015] Furthermore, the four-dimensional light field data of the 3D scene to be displayed mentioned in step S1 is calculated and generated by the camera array of the rendering software according to the views captured from different perspectives by perspective projection or orthographic projection.

[0016] Furthermore, the beam splitting field corresponding to each display state described in step S2 is:

[0017] ;

[0018] Where m is 1 or 2; This refers to the light field formed by the first and second multiplication displays when the zoom lens adjusts the light at the first focal length. This represents the light field formed by the two displays adjusted at the second focal length using a variable-focus lens, where y represents the coordinates on the light field plane. These are the coordinates on the pupil plane. and It is the pixel transmittance of the multiplicative display. It is the distance from the eye to the zoom lens. It is the distance between the zoom lens and the light field plane, and It is the distance between the virtual image of the microdisplay and the variable focus lens, the mapping function. : It includes the intersection of the light field and the pattern displayed on the physical display.

[0019] Furthermore, the four-dimensional light field described in step S2 is:

[0020]

[0021] in, These are vectors representing the fully synthesized light field and the partially synthesized light field, respectively. The partially synthesized light field refers to the light field emitted by each group of multiplicative displays at different focal lengths using a variable focal length lens. This refers to the Hadamard product; This refers to the vectorized form of the displayed pattern, where the index indicates the placement order of the virtual panels, with 1 representing closest to the zoom lens and 4 representing farthest; a sparse binary projection matrix. The structure is defined as By discretizing the number of light field positions on the pupil plane into V positions, the submatrix... Each submatrix in the model models how each pixel on the display is mapped to a specific view of the light field.

[0022] Furthermore, the optimization objective function described in step S3 is to minimize the difference between the target light field and the emitted light field. - Norm error:

[0023] .

[0024] Furthermore, step S3, which iteratively optimizes the attenuation layers corresponding to the first and second multiplication displays under the beam splitting field conditions corresponding to each display state using a non-negative update algorithm, specifically involves:

[0025] The initial condition is that the light field target of a group of multiplication displays is half the value of the original light field. When the number of iterations is less than a set number, the iteration order is as follows: first, update the display pattern of Group1 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the input light field of Group2 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the initial value for the next iteration of the display pattern of Group1 multiplication display, until the total number of iterations reaches the specified value; the update rules for Group1 and Group2 are as follows:

[0026] Group 1 update rules:

[0027] ,

[0028] Group 2 update rules:

[0029] ,

[0030] Where i represents the iteration number. It is the smallest positive number to prevent the denominator from being 0.

[0031] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention combines additive and multiplicative light field display and utilizes the dynamic focusing capability of a variable focus lens to achieve time-division multiplexing multifocal plane display, significantly expanding the depth of field range without increasing the number of physical display layers; the present invention effectively alleviates the convergence-accommodation conflict problem in traditional near-eye displays, improving visual comfort and immersion; at the same time, by replacing the physical layer structure with a virtual layer, the system thickness and hardware complexity are reduced, making the device lighter and more suitable for near-eye head-mounted display applications. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the hybrid multilayer compressed light field near-eye display system proposed in this invention;

[0033] Figure 2 This is a schematic diagram of the optical positions of each component in the hybrid multilayer compressed light field near-eye display system proposed in this invention;

[0034] Figure 3 This is a diagram illustrating the position of the rendered model.

[0035] Figure 4 This is a side view of the principle of the hybrid multilayer compressed light field near-eye display system proposed in this invention;

[0036] Figure 5 This is a top view of the principle of the hybrid multilayer compressed light field near-eye display system proposed in this invention;

[0037] Figure 6 Iterative flowchart of the nonnegative update optimization algorithm;

[0038] Figure 7 A photograph of a hybrid multilayer compressed light field near-eye display system taken by modifying the camera's adjustment depth.

[0039] Figure 8 A photograph of a two-layer display taken by adjusting the camera's depth settings. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the present invention provides a hybrid multilayer compressed light field near-eye display system 100, which mainly includes: a variable focus lens 101, a cemented doublet achromatic lens 102, a spacer 103, a multiplication display 104, and a controller 105; each component is arranged in the optical position required by the designed optical path.

[0042] The zoom lens 101 is a liquid zoom lens with a diopter range of -10D to +10D, corresponding to a focal length range of approximately 100 mm to infinity. It features electrically adjustable focal length and a fast response time (millisecond level). The cemented doublet achromatic lens 102 effectively corrects chromatic aberration in the visible light band. The combination of the zoom lens 101 and the cemented doublet achromatic lens 102 is responsible for controlling the system's focal length, which is set to 155.28 mm (Group 1) and 147.06 mm (Group 2) in the corresponding display modes, respectively.

[0043] The spacer 103 controls the spacing between the two multiplication displays 104 and the spacing between the first multiplication display and the cemented doublet achromatic lens 102. Both multiplication displays 104 employ high refresh rate micro-liquid crystal displays, responsible for intensity modulation of the light. The controller 105 is responsible for electrically connecting the multiplication displays 104 to the computer.

[0044] like Figure 2 As shown, the distance between the variable focus lens 101 and the cemented doublet achromatic lens 102 is 16.275 mm, the distance to the nearest multiplication display 104 is 52 mm, and the distance between the two multiplication displays 104 is 8 mm. In Group 1 display mode, the virtual image distances of the two multiplication displays 104 to the human eye are 437.621 mm (2.29D) and 1024.659 mm (0.98D), respectively. In Group 2 display mode, the virtual image distances to the human eye are 511.355 mm (1.96D) and 1587.044 mm (0.63D), respectively.

[0045] During a complete cycle of the system, in Group 1 display mode: the focal length of the zoom lens 103 is fixed at 155.28 mm. Simultaneously, the multiplication display 104 closer to the human eye loads a display image corresponding to the first virtual imaging plane, while the multiplication display 104 farther from the human eye loads a display image corresponding to the third virtual imaging plane. At this time, the human eye observes a three-dimensional image with depth information, formed by the fusion of these two virtual planes. In Group 2 display mode: the focal length of the zoom lens 103 is fixed at 147.06 mm. Simultaneously, the multiplication display 104 closer to the human eye loads a display image corresponding to the second virtual imaging plane, while the multiplication display 104 farther from the human eye loads a display image corresponding to the fourth virtual imaging plane. At this time, the human eye observes a three-dimensional image at a greater depth.

[0046] The switching process described above repeats cyclically at a rate higher than the persistence of vision in the human eye (e.g., 120Hz). As a result, the observer's brain merges the images seen in Group 1 and Group 2 to perceive a 3D scene covering a range from 0.63D to 2.29D, with continuous depth perception and realistic focus cues.

[0047] This invention also proposes an image generation method suitable for the proposed hybrid multilayer compressed light field near-eye display system, the specific implementation process of which is as follows:

[0048] The original light field data was generated using POV-RAY software. The rendered model was then placed according to the corresponding virtual layer positions. An orthogonal projection camera array was used to simulate the human eye and capture corresponding views from different perspectives. Figure 3 As shown, four-dimensional light field data for the left and right eyes of the corresponding person are calculated and generated based on the obtained view.

[0049] Considering the imaging geometry of a two-layer multiplicative display and a variable-focus lens, the position of the virtual image formed by each display layer at different focal lengths is determined using the thin-lens formula, such as... Figure 4 , Figure 5 As shown, the beam splitting field is determined. The expression:

[0050]

[0051] in, This refers to adjusting the light field formed by the two displays at the first focal length using a variable focus lens, where... This represents the light field formed by the two displays when the zoom lens is adjusted to the second focal length. y represents the coordinates on the light field plane. These are the absolute coordinates on the pupil plane. and It is the pixel transmittance of the multiplicative display. It is the distance from the eye to the zoom lens. It is the distance defined between the zoom lens and the light field plane, and It is the distance between the virtual image of the microdisplay and the variable focus lens, the mapping function. : It includes the intersection of the light field and the pattern displayed on the physical display.

[0052] Discretize the light field expression into matrix form. :

[0053]

[0054] in, These are vectors representing the fully synthesized light field and the partially synthesized light field, respectively. The partially synthesized light field refers to the light field emitted by each group of multiplicative displays at different focal lengths of the variable focal lens. This refers to the Hadamard product. This refers to the vectorized form of the displayed pattern, where the index indicates the placement order of the virtual panels (1 indicates closest to the zoom lens, 4 indicates farthest from the zoom lens). Sparse binary projection matrix. The structure is defined as By discretizing the number of light field positions on the pupil plane into V positions, the submatrix... Each submatrix in the model models how each pixel on the display is mapped to a specific view of the light field.

[0055] Using a non-negative update rule, minimize the difference between the target light field and the emitted light field. - Norm error, to obtain a set of optimal display patterns. The optimized simplified formula is as follows:

[0056]

[0057] The nonnegative matrix update rule iterates through the attenuation layers of Group1 and Group2, with the initial condition being that the light field target of a set of multiplicative displays is half the value of the original light field. The update rules for Group1 and Group2 are as follows:

[0058] Group 1 update rules:

[0059] ,

[0060] Group 2 update rules:

[0061] ,

[0062] Where i represents the iteration number. It must be the smallest positive number to prevent the denominator from being 0. When the number of iterations is less than the set number, the iteration order is as follows: first, update the display pattern of the Group1 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the input light field of the Group2 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the initial value for the next iteration of the Group1 multiplication display pattern, until the total number of iterations reaches the specified value, and then ends. Specifically, as shown below... Figure 6 As shown.

[0063] The resulting display pattern is synchronously loaded onto the corresponding multiplication display. The result is captured by modifying the focal plane of the SLR digital camera to focus on the depth of the object, such as... Figure 7 As shown. Due to the limitation of the aperture of the zoom lens, multiple photos are taken using a digital SLR camera to represent the entire scene. Because different layers correspond to different magnifications, the pixel size of each layer is different, and consequently, the size of each attenuation layer map is different. The further away from the lens the virtual layer is, the fewer pixels it needs to display the image, resulting in the appearance of a gray background and flickering, which does not affect human visual perception. Based on the depth of focus of the digital SLR camera and the placement of the object, magnified photos confirm the support for adjustment cues. In contrast, such as... Figure 8 As shown, reconstructing the light field of the same object at the same depth using only two attenuation layers without time-division multiplexing results in a significantly lower upper limit of depth of field compared to the proposed solution. A prototype of a hybrid multilayer compressed light field near-eye display system was constructed using given specifications. When the optimized image was displayed in the prototype, experimental results verified that the system can provide sufficient accommodation cues for a monocular eye within a 2.0D range. Compared to traditional four-layer multiplicative light field displays, the system thickness is significantly reduced, and brightness loss is reduced by approximately 50%.

[0064] The above description is merely an embodiment of the present invention and does not limit the scope of protection of the present invention. Any equivalent transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, should be included within the scope of protection of the present invention.

Claims

1. A hybrid multilayer compressed light field near-eye display system, characterized in that, include: First multiplication display, second multiplication display, zoom lens, cemented doublet achromatic lens, control unit and gasket; The first multiplication display and the second multiplication display modulate the intensity of the light. A variable focus lens and a cemented doublet achromatic lens are disposed in the optical paths of the first multiplication display and the second multiplication display; The control unit is electrically connected to the first multiplication display, the second multiplication display, and the variable focus lens; The control unit controls the variable focal length lens to switch between at least two different focal length states to satisfy the persistence of vision effect of the human eye; when the variable focal length lens is in different focal length states, image content that matches the depth of the corresponding virtual imaging plane is synchronously loaded onto the first multiplication display and the second multiplication display. By using time-division multiplexing, display images from different focal length states are integrated in time sequence to visually form at least four virtual imaging planes.

2. The hybrid multilayer compressed light field near-eye display system according to claim 1, characterized in that, The multiplication display is a liquid crystal display, or a multiplication modulator group consisting of two liquid crystal panels.

3. The hybrid multilayer compressed light field near-eye display system according to claim 1, characterized in that, The variable focus lens is a liquid variable focus lens.

4. The hybrid multilayer compressed light field near-eye display system according to claim 1, characterized in that, The spacer controls the spacing between the two multiplication displays and the spacing between the first multiplication display and the cemented doublet achromatic lens.

5. An image generation method employing the system described in any one of claims 1 to 4, characterized in that, Includes the following steps: S1: Obtain the four-dimensional light field data of the 3D scene to be displayed; S2: Decompose the four-dimensional light field data of the 3D scene into light fields corresponding to each display state of the system to obtain the four-dimensional light field; the display state is determined by the combination of the focal length state of the variable focus lens and the corresponding multiplication display. Using the rank-1 light field decomposition algorithm, calculate the attenuation layer map corresponding to the first multiplication display and the second multiplication display under the light field conditions corresponding to each display state. S3: Set the optimization objective function, and use a non-negative update algorithm to iteratively optimize the attenuation layers corresponding to the first multiplication display and the second multiplication display under the corresponding light field conditions of each display state, so as to ensure that all pixel values ​​are non-negative; S4: The optimized multi-layer attenuation layer map is loaded onto the first multiplication display and the second multiplication display respectively according to its timing correspondence.

6. The image generation method for a hybrid multilayer compressed light field near-eye display system according to claim 5, characterized in that, The four-dimensional light field data of the 3D scene to be displayed mentioned in step S1 is calculated and generated by the camera array of the rendering software according to the views captured from different perspectives by perspective projection or orthographic projection.

7. The image generation method for a hybrid multilayer compressed light field near-eye display system according to claim 5, characterized in that, The beam splitting fields corresponding to each display state described in step S2 are: ； Where m is 1 or 2; This refers to the light field formed by the first and second multiplication displays when the zoom lens adjusts the light at the first focal length. This represents the light field formed by the two displays adjusted at the second focal length using a variable-focus lens, where y represents the coordinates on the light field plane. These are the coordinates on the pupil plane. and It is the pixel transmittance of the multiplicative display. It is the distance from the eye to the zoom lens. It is the distance between the zoom lens and the light field plane, and It is the distance between the virtual image of the microdisplay and the variable focus lens, the mapping function. : It includes the intersection of the light field and the pattern displayed on the physical display.

8. The image generation method for a hybrid multilayer compressed light field near-eye display system according to claim 5, characterized in that, The four-dimensional light field described in step S2 is: ； in, These are vectors representing the fully synthesized light field and the partially synthesized light field, respectively. The partially synthesized light field refers to the light field emitted by each group of multiplicative displays at different focal lengths using a variable focal length lens. This refers to the Hadamard product; This refers to the vectorized form of the displayed pattern, where the index indicates the placement order of the virtual panels, with 1 representing closest to the zoom lens and 4 representing farthest; a sparse binary projection matrix. The structure is defined as By discretizing the number of light field positions on the pupil plane into V positions, the submatrix... Each submatrix in the model models how each pixel on the display is mapped to a specific view of the light field.

9. The image generation method for a hybrid multilayer compressed light field near-eye display system according to claim 5, characterized in that, The optimization objective function described in step S3 is to minimize the difference between the target light field and the emitted light field. - Norm error: 。 10. The image generation method for a hybrid multilayer compressed light field near-eye display system according to claim 5, characterized in that, Step S3, which iteratively optimizes the attenuation layers corresponding to the first and second multiplication displays under the beam splitting field conditions corresponding to each display state using a non-negative update algorithm, specifically involves: The initial condition is that the light field target of a group of multiplication displays is half the value of the original light field. When the number of iterations is less than a set number, the iteration order is as follows: first, update the display pattern of Group1 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the input light field of Group2 multiplication display; after the sub-iteration is completed, the difference between the split light field and the original light field is used as the initial value for the next iteration of the display pattern of Group1 multiplication display, until the total number of iterations reaches the specified value; the update rules for Group1 and Group2 are as follows: Group 1 update rules: ， ； Group2 update rules: ， ； Where i represents the iteration number. It is the smallest positive number to prevent the denominator from being 0.

Images