Human eye follow-up light field 3D display method

Through the 3D display method of following the light field of the human eye, the viewpoint spatial distribution of the light field 3D display is adjusted in real time by using the camera internal parameters and distortion coefficients, which solves the problems of limited viewing angle improvement and image distortion in the prior art, and achieves high 3D The image resolution and large 3D depth simultaneously expand the 3D viewing angle.

CN119071467APending Publication Date: 2024-12-03BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411354238.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-27
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing 3D display methods of light field cannot guarantee high 3D image resolution, large 3D depth and large 3D viewing angle at the same time, and the human eye tracking technology has image distortion problems when improving the 3D viewing angle.

Method used

Using the 3D light field display method of following human eyes, the camera's internal parameters and distortion coefficients are obtained and calibrated through the initial configuration stage, and the face recognition and tracking algorithm are combined to obtain the pixel coordinates of the midpoint of the viewer's binocular line in real time, and converted to offset. Angle, recalculate the spatial distribution of the viewpoint and dynamically adjust the 3D film source of the light field to respond to changes in the human eye position.

Benefits of technology

Without changing the hardware conditions, the 3D viewing angle is greatly improved, avoiding image distortion caused by the translation film source, and ensuring high 3D image resolution and large 3D depth performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071467A_ABST
    Figure CN119071467A_ABST
Patent Text Reader

Abstract

The invention provides a human eye follow-up light field 3D display method, and the method comprises the steps: tracking the positions of two eyes of a viewer, obtaining a deviation angle of a midpoint of a connecting line of the two eyes of the viewer relative to an optical axis of a camera in real time, dynamically adjusting the viewpoint space distribution of light field 3D display, and updating a light field 3D film source in real time. According to the technical scheme of the invention, on the premise of ensuring high 3D image resolution and large 3D depth, the 3D viewing angle can be greatly improved, and meanwhile, image distortion caused by responding to the position change of human eyes in a film source translation mode is effectively avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to 3D display technology, and in particular to a light field 3D display method for human eye tracking. Background Art

[0002] Light field 3D display is an important 3D display technology. By restoring the light distribution on the surface of each object, it forms a slight difference in the viewer's eyes that changes with depth. Through human brain fusion, it forms a sense of depth for 3D objects at different distances. Since light field 3D display is essentially a reconstruction of the light field, compared to conventional parallax-type 3D display that directly projects parallax information into the left and right eyes, it can avoid stereoscopic fatigue caused by convergence adjustment conflicts. The performance and viewing experience of light field 3D display can be evaluated through indicators such as 3D viewing angle, 3D image resolution and 3D depth. Due to the limited amount of spatial information, existing light field 3D display methods cannot simultaneously guarantee high 3D image resolution, large 3D depth and large 3D viewing angle.

[0003] To solve this problem, eye tracking technology is gradually being introduced into light field 3D display. By tracking the position of the human eye, the display content can be switched in real time, thereby improving the viewing angle, image resolution and other performance of the 3D display without changing the hardware conditions. However, the existing light field 3D display technology based on eye tracking is still imperfect. On the one hand, the existing methods have limited improvement in 3D viewing angle under the premise of high 3D image resolution and large 3D depth. On the other hand, the existing methods mostly use the method of translating the film source to respond to changes in the position of the human eye. This direct projection will cause the angle relationship between the eye and the 3D scene to be incorrect, resulting in image distortion. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a light field 3D display method with human eye tracking, which specifically includes two stages: initialization configuration and human eye tracking cyclic rendering.

[0005] Furthermore, the initialization configuration phase includes the following steps:

[0006] A camera used to obtain a real-time video stream of a viewer is fixed on the light field 3D display screen, the camera is pre-calibrated, the intrinsic parameters and distortion coefficients of the camera are obtained, and stored in a computer;

[0007] Before the light field 3D display screen is put into operation, the intrinsic parameters and distortion coefficients of the camera are read to initialize the system.

[0008] Furthermore, the intrinsic parameters and distortion coefficients of the camera include image principal point, focal length, radial distortion coefficient and tangential distortion coefficient.

[0009] Furthermore, the camera must have a function of acquiring video streams in real time. The camera can be any one of a network camera, a USB camera, a smart camera, and a digital camera.

[0010] Furthermore, the human eye follows the rendering in a loop, and a rendering loop includes the following steps:

[0011] The camera obtains a real-time video stream of the viewer;

[0012] Use face recognition and tracking algorithms to process the viewer's real-time video stream and obtain the pixel coordinates of the midpoint of the viewer's binocular line in real time;

[0013] Using the camera internal parameters and distortion coefficients, the pixel coordinates of the midpoint of the viewer's binocular line are converted into an offset angle of the midpoint of the viewer's binocular line relative to the camera optical axis, including a horizontal offset angle and a vertical offset angle;

[0014] Recalculating the viewpoint spatial distribution according to the horizontal offset angle, the vertical offset angle and the hardware parameters of the light field 3D display screen;

[0015] According to the new viewpoint spatial distribution, the parallax image is rendered to regenerate the light field 3D film source. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above aspects and advantages of the present invention will be further clarified and easily understood from the following detailed description in conjunction with the accompanying drawings and embodiments, in which:

[0017] Figure 1 This is a flow chart of a light field 3D display method for human eye tracking according to an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a deflection viewing area according to an embodiment of the present invention;

[0019] Figure 3 A viewing model for a conventional ultra-multi-viewpoint 3D display according to an embodiment of the present invention;

[0020] Figure 4 An idealized model of a light field 3D display method for human eye tracking according to an embodiment of the present invention;

[0021] The diagram numbers in the above drawings are:

[0022] 201 imaging plane, 202 human eye tracking camera lens, 203 viewing frame, 204 optical axis, 205 camera imaging plane, 206 human eye in the camera imaging plane, 301 light field 3D display screen, 302 reconstructed light field, 303 viewing field, 304 viewer, 401 camera, 402 camera captureable area, 403 3D viewing field, 404 total 3D viewing field.

[0023] It should be understood that the above drawings are only schematic and are not drawn to scale. DETAILED DESCRIPTION

[0024] In order to facilitate the understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings. Optional embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the invention pertains. The terms used herein in the specification of the invention are for the purpose of describing specific embodiments only and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0026] The following will describe in detail a light field 3D display method with human eye tracking proposed in the present application in combination with the embodiments and drawings disclosed in the present application.

[0027] In one embodiment, the light field 3D display screen is an ultra-multi-viewpoint light field 3D display screen with M viewpoints, and its main structure is a liquid crystal display screen with a color bit depth of 8 bits and a cylindrical lens grating. The cylindrical lens grating only splits light in the horizontal direction, and is used to reconstruct the light distribution at different angles to cause binocular parallax of the human eye and produce a three-dimensional sense; the cylindrical lens grating does not split light in the vertical direction, so the image content viewed at various angles in this direction does not change. The camera used to obtain the viewer's real-time video stream has a resolution of 1080p.

[0028] Figure 1 A flow chart of a light field 3D display method with human eye tracking according to an embodiment of the present invention is shown, which specifically includes two stages: initialization configuration and human eye tracking loop rendering.

[0029] The initialization configuration phase includes the following steps:

[0030] S1: Fix the camera 401 on the light field 3D display screen 301 in a specific orientation, pre-calibrate the camera 401 for obtaining the real-time video stream of the viewer 304, and obtain the internal parameters and distortion coefficients of the camera 401, including the principal point (u 0 ,v 0 ), focal length f x and f y , radial distortion coefficient k 1 , k 2 , k 3 And the tangential distortion coefficient P 1, P 2 , and store these parameters in the computer.

[0031] S2: Before the light field 3D display screen 301 is operated, the internal parameters and distortion coefficients of the camera 401 are read to initialize the system. Specifically, the internal parameters and distortion coefficients of the camera 401 are imported into the human eye tracking loop rendering system.

[0032] In the human eye following loop rendering stage, a rendering cycle includes the following steps:

[0033] S3: Using the calibrated camera 401 to obtain the real-time video stream of the viewer 304 .

[0034] S4: Using the real-time video stream of the viewer 304 obtained by processing with the human eye recognition and tracking algorithm, the coordinates (u, v) of the midpoint pixel of the binocular line connecting the viewer 304 in the pixel coordinate system are obtained in real time.

[0035] S5: Using the intrinsic parameters and distortion coefficients of the camera 401, the pixel coordinates (u, v) of the midpoint of the binocular line of the viewer 304 are converted into the horizontal offset angle θ of the midpoint of the binocular line of the viewer 304 relative to the optical axis 204 of the camera 401. In this embodiment, firstly, the distortion correction is performed on (u, v) using the distortion coefficients of the camera 401 obtained above. Secondly, using the intrinsic parameters of the camera 401 obtained above, the coordinates (u, v) of the midpoint of the binocular line of the viewer 304 in the pixel coordinate system are converted into the coordinates (x) in the camera coordinate system. c ,y c ,z c ), the specific coordinate transformation formula is:

[0036] x c =(uu 0 )·z c / f x (1)

[0037] y c =(vv 0 )·z c / f y (2)

[0038] where z c is the depth of the imaging surface 205 of the camera 401. Next, the horizontal offset angle θ of the midpoint of the binocular line of the viewer 304 relative to the optical axis 204 of the camera is calculated. The specific calculation formula is:

[0039] θ=arctan[(uu 0 ) / f x ] (3)

[0040] According to the geometric relationship between the camera 401 and the light field 3D display screen 301 , the horizontal offset angle of the midpoint of the binocular line connecting the viewer 304 with respect to the center of the light field 3D display screen 301 is also θ.

[0041] S6: The obtained horizontal offset angle θ of the midpoint of the binocular line of the viewer 304 relative to the center of the light field 3D display screen 301 is transmitted to the rendering pipeline in real time through Socket communication. According to the horizontal offset angle θ and the hardware parameters of the light field 3D display screen 301, the viewpoint offset η and the camera 401 offset Δx are solved in real time in the rendering pipeline. Figure 2 A schematic diagram of a deflection viewing area according to an embodiment of the present invention is shown.

[0042] The specific calculation formula of the viewpoint offset η is:

[0043]

[0044] Among them, p L is the pitch of the cylindrical lens grating, N view is the total number of viewpoints. The specific calculation formula of the camera offset Δx is:

[0045]

[0046] Then recalculate the viewpoint spatial distribution and change the viewpoint index. The specific calculation formula of the viewpoint index is:

[0047]

[0048] i′ view =(i view +η+Δ′) mod N s (7)

[0049] Among them, i view is the viewpoint index corresponding to the current index sub-pixel before correction, i' view is the viewpoint index corresponding to the current index sub-pixel after correction, k is the cylindrical lens slope, Δ and Δ' are the positive values, N s is the number of sub-pixels that the lens spans horizontally.

[0050] S7: According to the new viewpoint spatial distribution, in the rendering pipeline, the parallax image is rendered to regenerate the light field 3D film source.

[0051] During the system operation phase, the human eye tracking loop rendering will be executed continuously.

[0052] Figure 3 The viewing model of the conventional super multi-viewpoint 3D display according to the embodiment of the present invention is shown. Figure 4An idealized model of a light field 3D display method with human eye tracking according to an embodiment of the present invention is shown. The original 3D viewing angle of the light field 3D display screen 301 is γ, the maximum shooting angle of the camera 401 is ε, and the total 3D viewing angle of the ideal model of the light field 3D display method with human eye tracking is α. Compared with the traditional light field 3D display, the 3D viewing angle enlargement value of the present method is theoretically the maximum shooting angle γ of the camera. The calculation formula of the total 3D viewing angle is:

[0053] α=γ+ε (8)

[0054] The present invention proposes a light field 3D display method with human eye tracking. Through human eye tracking, the 3D viewing angle can be greatly improved without changing the hardware conditions, thereby achieving a large 3D viewing angle while ensuring high 3D image resolution and large 3D depth. In addition, by dynamically adjusting the viewpoint spatial distribution of the light field 3D display in response to changes in the human eye position, image distortion caused by translating the film source can be effectively avoided.

Claims

1. A light field 3D display method with human eye tracking, characterized in that: The method includes two stages: initialization configuration and human eye following loop rendering. The initialization configuration stage includes the following steps: A camera used to obtain a real-time video stream of a viewer is fixed on the light field 3D display screen, the camera is pre-calibrated, the intrinsic parameters and distortion coefficients of the camera are obtained, and stored in a computer; Before the light field 3D display screen is put into operation, the camera's internal parameters and distortion coefficients are read to initialize the system. The human eye follows the loop rendering, and a rendering cycle includes the following steps: The camera obtains a real-time video stream of the viewer; Use face recognition and tracking algorithms to process the viewer's real-time video stream and obtain the pixel coordinates of the midpoint of the viewer's binocular line in real time; Using the camera internal parameters and distortion coefficients, the pixel coordinates of the midpoint of the viewer's binocular line are converted into an offset angle of the midpoint of the viewer's binocular line relative to the camera optical axis, including a horizontal offset angle and a vertical offset angle; Recalculating the viewpoint space distribution according to the horizontal offset angle, the vertical offset angle and the hardware parameters of the light field 3D display screen; According to the new viewpoint spatial distribution, the parallax image is rendered to regenerate the light field 3D film source.