Naked eye 3D ray tracing rendering method and device, parameter design method and display device

Abstract

The present application relates to naked eye 3D light ray tracing rendering method, device, parameter design method and display equipment, belong to naked eye 3D rendering and holographic image field. Including: the establishment of light field distribution model;According to the current eye coordinates and light field distribution model obtains the current equivalent lens pitch, equivalent offset and equivalent pixel density of optical structure, and then obtains the current viewable area, view angle, the total number of light rays corresponding to the viewable area and the parameter space of light rays;Based on the parameter space of light rays, human eye characteristics, eye coordinates, the probability density cloud corresponding to the current parallax map is obtained, and then the light rays corresponding to the current parallax map of each viewer's each eye at the viewing position are obtained;The light field map is obtained by summing the light rays, and is projected to the display screen to realize naked eye 3D rendering. The method determines the light rays corresponding to the current parallax map for rendering, solves the problems of low display resolution, poor definition and dizziness caused by the current naked eye 3D rendering and display method.

Description

Technical Field

[0001] This invention relates to the field of naked-eye 3D rendering and holographic imaging technology, and in particular to naked-eye 3D ray tracing rendering methods, devices, parameter design methods, and display devices. Background Technology

[0002] Current naked-eye 3D rendering methods typically use multiple virtual cameras to capture virtual 3D scene views from different angles, and render the 3D scene view corresponding to each virtual camera at the same resolution; during rendering, the 3D scene is projected onto a 2D plane according to the projection relationship between the 3D model and the display screen coordinates in the world coordinate system.

[0003] Currently, some methods for naked-eye 3D rendering and display employ binocular left and right views. Based on the eye coordinates obtained through pupil tracking, the left and right views are arranged at the positions of the views after refraction by the cylindrical lens array along the viewer's left and right lines of sight. However, when there is a certain difference between the eye coordinates obtained through pupil tracking and the actual eye coordinates, dizziness can occur. Other methods divide the period of the cylindrical lens array into multiple viewing zones, periodically arranging views at different angles within each zone. However, when the viewer's eye crosses the boundary of the viewing zone, 3D imaging distortion occurs. Furthermore, due to the splitting of light across multiple viewing zones, the equivalent pixel distance between the same view increases, resulting in reduced display resolution and poor display clarity.

[0004] In summary, existing naked-eye 3D rendering and display methods have the following drawbacks:

[0005] 1. Low display resolution and poor display clarity. Improving display resolution without increasing costs has always been one of the challenges in designing glasses-free 3D rendering and display methods.

[0006] 2. The binocular naked-eye 3D rendering and display method based on pupil tracking has high requirements for pupil tracking performance (i.e., the temporal resolution of the pupil tracking system). When there is a certain difference between the human eye coordinates obtained by pupil tracking and the actual human eye coordinates, it will cause dizziness. In addition, the left and right eye views will overlap due to factors such as angular resolution, spherical aberration and other spatial resolution of the optical system.

[0007] 3. Multi-viewpoint naked-eye 3D rendering and display methods: When the human eye crosses the boundary of the viewpoint, it will cause distortion in 3D imaging. In addition, due to the limited spatial resolution of the grating structure, it will also cause blurring and dizziness when entering and leaving the screen.

[0008] 4. Rasterization rendering is usually used to generate multiple subviews arranged in a grid pattern. It does not provide ray-level rendering that reflects the mapping relationship of the light field distribution. Even when ray tracing rendering is used, it usually simulates a virtual camera to generate subviews and does not directly generate a light field map based on the mapping relationship of the light field distribution, resulting in serious waste of resources. Summary of the Invention

[0009] Based on the above analysis, the present invention aims to provide a naked-eye 3D ray tracing rendering method, apparatus, parameter design method and display device to solve the problems of low display resolution, poor clarity and dizziness, jump, view overlap and blurring when entering and leaving the screen caused by current naked-eye 3D rendering and display methods.

[0010] This invention provides a naked-eye 3D ray tracing rendering method, apparatus, parameter design method, and display device. The naked-eye 3D ray tracing rendering method includes the following steps:

[0011] Step S1: Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the parameters of the 2D display device and the light field mapping relationship corresponding to the optical structure.

[0012] Step S2: Real-time acquisition of human eye coordinates; based on the current human eye coordinates and light field distribution model, obtain the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure; based on the current equivalent lens pitch and equivalent offset, obtain the current visible area and viewing angle; based on the current equivalent pixel density and application scenario, obtain the total number of rays corresponding to the current visible area; based on the current human eye coordinates and viewing angle, obtain the parameter space of the rays.

[0013] Step S3: Set the probability density function of the prior distribution of the parameter space of the light rays; establish the probability density function of the human eye characteristics related to the light density based on the characteristics of the human eye; obtain the probability density function of the human eye coordinates based on the pupil tracking algorithm and the pupil tracking system delay; multiply the above three probability density functions together to obtain the probability density cloud corresponding to the current disparity map; and obtain the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position based on the probability density cloud and the total number of light rays.

[0014] Step S4: Establish a rendering model, sum the light rays corresponding to the current parallax map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0015] Furthermore, the step of obtaining the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current visible area and viewing angle based on the equivalent lens pitch and equivalent offset, includes:

[0016] The current viewing distance is obtained based on the current human eye coordinates, and the current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model; the current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system; the current human eye parallax is obtained based on the current human eye coordinates, and the current equivalent pixel density is obtained based on the 2D display device parameters and the current human eye parallax;

[0017] The current reference visible area and the viewing angle range are obtained based on the current equivalent lens pitch; the current visible area is obtained by sliding the current reference visible area according to the current equivalent offset, and the current viewing angle range is obtained by sliding the current equivalent offset.

[0018] Furthermore, the step of setting the probability density function of the parameter space prior distribution of the light ray includes: taking the center point of the visible area corresponding to the observation space coordinate system as the mean position of the Gaussian distribution, and taking the size of the visible area as three times the standard deviation of the Gaussian distribution, thereby obtaining the probability density function of the parameter space prior distribution of the light ray.

[0019] Furthermore, the step of obtaining the probability density function of the human eye coordinates based on the pupil-tracking algorithm and the pupil-tracking system delay includes: inferring the probability distribution function of the human eye position from the pupil-tracking algorithm; performing Kalman filtering on the probability distribution function of the human eye position to obtain a smooth real-time pupil-tracking curve; obtaining the pupil-tracking system delay based on the factory settings or manual calibration of the pupil-tracking device; establishing an empirical curve of the pupil-tracking system delay based on the pupil-tracking system delay; and multiplying the real-time pupil-tracking curve and the empirical curve of the pupil-tracking system delay to obtain the probability density function of the human eye coordinates.

[0020] Furthermore, a color difference model is obtained based on the characteristics of the human eye, which represents the relationship between color difference and viewing angle within a certain angle of the pupil's main viewing angle. A color correction matrix based on the spatial distribution on the human eye's visual coordinate system is obtained through the color difference model.

[0021] Furthermore, the step of obtaining the light corresponding to the current disparity map of each viewer's eyes at the viewing position based on the probability density cloud and the total number of light rays includes:

[0022] Based on the probability density cloud, the probability corresponding to each point in the parameter space is obtained. The probability corresponding to each point in the parameter space is multiplied by the total number of light rays to obtain the number of light rays corresponding to each point in the parameter space. Based on each point in the parameter space and the number of light rays corresponding to it, the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position are obtained.

[0023] Furthermore, for a viewing position, the expression for the rendering model is:

[0024]

[0025] Among them, res S The trace represents the resolution of the accumulated and post-accumulation light field map, where t represents time; field This refers to the process of ray tracing performed by a 3D display device; N light Indicates the total number of light rays, o Ob o W These represent the light rays in the observation space coordinate system Ob. xyz The actual position below, and the corresponding 3D virtual space world coordinate system W xyz The starting point below, d Ob (o Ob ,F Ob (r')) indicates that light originates from the observation space Ob. xyz The starting point o Ob To the surface of the light field structure F Ob The normalized ray direction of (r'), d W (o W ,S xy (r') indicates that light passes through the optical structure at point S. xy (r') corresponds to the 3D virtual space world coordinate system W xyz Normalized ray direction under, χ(o) Ob ,d Ob (o Ob ,F Ob (r')),t))χ(o W ,d W (o W ,S xy (r'),t)) represent the probability density cloud in the observation space coordinate system and the probability density cloud in the 3D virtual space coordinate system, respectively.

[0026] Furthermore, a parameter design method for the optical structure of a glasses-free 3D display device applied to the aforementioned glasses-free 3D ray tracing rendering method, the method comprising:

[0027] Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees.

[0028] Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees;

[0029] Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating. Spend.

[0030] Furthermore, a naked-eye 3D rendering device based on light field distribution, the device comprising:

[0031] Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module;

[0032] The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. Based on the current human eye coordinates, the human eye characteristic model, and the rendering model, it obtains a light field map and sends it to the naked-eye 3D display device.

[0033] A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

[0034] Furthermore, a glasses-free 3D display device, wherein the glasses-free 3D display device applies a glasses-free 3D ray tracing rendering method, including:

[0035] 2D display devices or light sources used to provide 2D images;

[0036] Optical structures are used to provide viewers with different perspectives for each eye at the same time, thereby enabling naked-eye 3D displays and holographic displays;

[0037] Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. This invention acquires six-DOF human eye coordinates through real-time pupil tracking and determines the parameters of each ray in the light field map based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It considers the attributes of each observer's eye in the light field at any given moment and the full parameter mapping relationship through the optical structure, adjusting the equivalent offset of the corresponding sub-pixels on the screen. This ensures that the visible area always aligns with the center of the human eye coordinates, even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience.

[0040] 2. This invention directly generates the light field map required for human eye parallax, without generating subviews and interweaving them, thus eliminating crosstalk between subviews and avoiding problems such as 3D image tearing; the performance overhead of naked-eye 3D ray tracing rendering compared to 2D ray tracing rendering is almost negligible, far less than the performance overhead of naked-eye 3D rasterization rendering compared to 2D rasterization rendering.

[0041] 3. This invention obtains the probability density distribution of various parameters of light rays in the light field map required by each viewer based on the light field distribution and the parallax required by the human eye characteristic model. It modulates the probability density distribution according to the pupil-tracking delay empirical curve to obtain all the light rays in the light field map, thereby generating a light field map. This enhances the realism of the 3D visual effect. While ensuring the naked-eye 3D display effect, it effectively reduces the resource consumption of the light field map in terms of storage, transmission and rendering, and saves bandwidth significantly.

[0042] 4. This invention obtains the design parameters of the light field screen of a naked-eye 3D display device through a naked-eye 3D ray tracing rendering method. By using dynamic light field mapping relationships, it ensures that the physical limits of viewing angle, clarity, and imaging quality are reached on a given optical structure, thus providing a foundation for the design of the optical structure of naked-eye 3D display devices.

[0043] 5. This invention takes into account both the density of the light rays required for the light field map and the virtual spatial position corresponding to each observation position provided to the user. Therefore, the method and system provided by this invention can support both providing naked-eye 3D rendering for only two eyes, that is, providing a 3D display effect for one viewer when moving in space, and providing multiple observation positions at the same time, that is, providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0044] 6. This invention predicts the human eye movement trajectory by real-time pupil tracking combined with unscented Kalman filtering. Under special conditions such as low light, it uses visual characteristic empirical curves to modulate real-time pupil tracking to obtain the human eye coordinates, making the human eye coordinates closer to the true value and reducing problems such as dizziness and ghosting.

[0045] 7. This invention utilizes the depth information of the rendering to apply a depth-based Gaussian blur to the target to be rendered, applying a depth-of-field effect to objects far from the focal plane, thereby compensating for the appearance of distant objects and reducing crosstalk.

[0046] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from the description and drawings, which are particularly pointed out. Attached Figure Description

[0047] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0048] Figure 1 This is a flowchart of a naked-eye 3D ray tracing rendering method according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the spatial light field according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of view alignment in an embodiment of the present invention;

[0051] Figure 4 This is a block diagram of a naked-eye 3D ray tracing rendering device according to an embodiment of the present invention;

[0052] Figure 5 This is a block diagram of a naked-eye 3D display device according to an embodiment of the present invention. Detailed Implementation

[0053] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0054] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0055] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0058] One specific embodiment of the present invention discloses a naked-eye 3D ray tracing rendering method. For example... Figure 1 As shown, the method includes the following steps:

[0059] S1. Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the light field mapping relationship corresponding to the 2D display device parameters and the optical structure.

[0060] S2. Real-time acquisition of human eye coordinates. Based on the current human eye coordinates and the light field distribution model, obtain the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure. Based on the current equivalent lens pitch and equivalent offset, obtain the current visible area and viewing angle. Based on the current equivalent pixel density and application scenario, obtain the total number of rays corresponding to the current visible area. Based on the current human eye coordinates and viewing angle, obtain the parameter space of the rays.

[0061] S3. Set the probability density function of the prior distribution of the parameter space of the light rays, establish the probability density function of the human eye characteristics related to the light density based on the characteristics of the human eye, and obtain the probability density function of the human eye coordinates based on the pupil tracking algorithm and the pupil tracking system delay; multiply the above three probability density functions together to obtain the probability density cloud corresponding to the current disparity map, and obtain the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position based on the probability density cloud and the total number of light rays;

[0062] S4. Establish a rendering model, sum the light rays corresponding to the current parallax map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0063] Specifically, in step S1, the 2D display device parameters include the screen size, aspect ratio, resolution, pixel pitch, etc. The optical structure parameters include the refractive index, fill spacing, lens focal length, viewing angle, spatial resolution, lens pitch, lens thickness, tilt angle, etc.

[0064] It should be noted that the naked-eye 3D display device includes a 2D display device and an optical structure added to the 2D display device. The 2D display device includes a plurality of periodically arranged light-emitting units and pixels capable of displaying red, green, and blue colors. The optical structure is used to provide different viewing angles to the viewer's eyes at the same time, thereby forming a 3D effect. The optical structure includes slit gratings, lenticular gratings, adjustable liquid crystal gratings, spatial light modulators, etc. Preferably, when providing a 3D display effect to multiple viewers simultaneously, i.e., supporting multiple viewing positions simultaneously, a lenticular grating is used as the optical structure to provide a larger field of view.

[0065] Furthermore, obtaining the light field mapping relationship corresponding to the optical structure based on the optical structure parameters includes: obtaining the light field mapping relationship corresponding to the optical structure through one or more methods such as theoretical or numerical calculation, Monte Carlo simulation, neural network learning, and standard light source calibration test;

[0066] The light field mapping relationship includes the light field mapping relationship from the light-emitting unit on the screen coordinate system of the 2D display device through the optical structure to the observation space, or the light field mapping relationship from the observation space through the optical structure to the light-emitting unit on the screen coordinate system of the 2D display device.

[0067] Specifically, in some embodiments, the light field mapping relationship is obtained through theoretical or numerical calculations (including empirical function models) of geometric optics. For example, the periodic viewpoint arrangement relationship after refraction by the lens is obtained by calculating the Fresnel theorem theory of cylindrical lens grating.

[0068] In some embodiments, Monte Carlo simulation tools are used to simulate the absorption, scattering, and refraction effects of each wavelength in the visible light band, obtaining the full-parameter optical field mapping relationship, i.e., the response matrix of each parameter of the optical field. For example, using the Geant4 Monte Carlo simulation tool, the absorption, scattering, and refraction effects of each wavelength in the visible light band are simulated using parallel light sources of each wavelength and optical structures modeled in 3D within Geant4, obtaining the full-parameter optical field mapping relationship.

[0069] In some embodiments, the light field mapping relationship is learned through a neural network. Specifically, this process includes the following main steps:

[0070] Step S11: Use multiple imaging devices to acquire imaging images of different 3D scenes at multiple human eye coordinates, and construct a dataset; the dataset covers a wide range of 3D scenes and viewpoints;

[0071] Step S12: Construct a neural network to output 3D scene rendering parameters corresponding to the input human eye coordinates; the 3D scene rendering parameters are used to control image generation during real-time rendering.

[0072] Step S13: Train the neural network based on the dataset to minimize the loss function and obtain a trained neural network;

[0073] Step S14: Capture the current human eye coordinates, input the current human eye coordinates into the trained neural network, and infer the 3D scene rendering parameters corresponding to the current human eye coordinates.

[0074] It should be noted that neural networks can learn the implicit light field mapping relationship from the light field in the human eye coordinates through the optical structure to the light-emitting unit in the screen coordinate system of the 2D display device.

[0075] In some embodiments, the optical field mapping relationship is obtained by performing response tests in the visible light band using a standard light source. For example, the optical field mapping relationship is obtained by testing the refraction angle, dispersion, and intensity distribution of the optical structure at different angles using a standard light source.

[0076] In some embodiments, one or more of the above methods can be used at different stages to obtain the light field mapping relationship. For example, since the cylindrical lens grating has different thermal deformations at different stages such as design, production, before bonding, and after bonding, the mapping relationship of each parameter can be obtained by theoretical, numerical, and simulation methods at the design stage; and the mapping relationship of each parameter can be calibrated by testing with a standard light source after bonding.

[0077] It is understandable that the design parameters of the optical structure may deviate from the actual parameters due to factors such as manufacturing process. Therefore, it is necessary to use the above method to obtain the light field mapping relationship corresponding to the optical structure, improve the accuracy of the light field mapping relationship, and provide a foundation for improving the quality of naked-eye 3D display in the future.

[0078] Furthermore, the light field distribution model represents the light field distribution generated in the observation space due to the influence of the optical characteristics of the 2D display device and the light field mapping relationship corresponding to the optical structure on the emitted light of the 2D display device.

[0079] It should be noted that, for ease of description, the world coordinate system of the virtual space where the target to be rendered is defined as W. xyz The screen coordinate system represented by the two-dimensional screen coordinates of the light-emitting unit of the 2D display device is S. xy The observation space coordinate system describing the spatial location of the human eye and the naked-eye 3D display device is Ob. xyz The human visual coordinate system is V xyz Among them, the screen coordinate system S xy It conforms to the definition of the screen coordinate system in general rasterization, ray tracing, graphics processing systems, and graphics pipelines. The observation space coordinate system Ob xyzWith the geometric center of the light-emitting unit of the 2D display device as the origin, the positive direction of the light-emitting unit passing through this origin is the positive z-axis direction of the right-hand coordinate system. The human eye visual coordinate system V... xyz A right-handed coordinate system is established with the visual center of each eye as the origin and the positive z-axis pointing in the direction of the eye center. Define T. Ob-V Indicates from Ob xyz coordinate system to V xyz The transformation matrix of the coordinate system, T V-Ob Indicates from V xyz coordinate system to Ob xyz The transformation matrix of a coordinate system can be used to derive the transformation matrix representation between other coordinate systems.

[0080] like Figure 2 As shown, in the screen coordinate system, the light-emitting unit S(x, y) is emitted from the surface of the optical structure at an angle α(θ, φ) in the observation space coordinate system Ob(x', y', z'). Each light-emitting point Ob0(x′, y′, z′|α), Ob1(x′, y′, z′|α), Ob2(x′, y′, z′|α), ... emitting at an angle α(θ, φ) constitutes the equivalent pixel of the view I(α) corresponding to that emission angle α(θ, φ). Six degrees of freedom in space are defined for each viewer at each moment, including the distance from each eye to the screen of the naked-eye 3D display device and the pupil orientation; that is, the human eye coordinates are... Where p represents the p-th viewer; i represents the i-th eye of each viewer; This represents the horizontal position of the i-th eye of the p-th viewer on the screen, i.e., the coordinates in the left and right directions; This represents the vertical position of the i-th eye of the p-th viewer from the screen, i.e., the coordinates in the up and down direction; Represents the vertical distance from the i-th eye of the p-th viewer to the screen, i.e., the coordinates in the near and far directions; This represents the yaw angle, which is the horizontal rotation angle around the vertical axis (z′ axis), corresponding to the viewer's eye turning to the left or right; It represents the pitch angle, which is the vertical rotation angle around the horizontal axis (y′ axis), corresponding to the viewer's head-up or head-down movement. The 't' represents the roll angle, which is the axial rotation around the front-to-back axis (x' axis), corresponding to the viewer's head tilting to the side; 't' represents the change of the coordinate over time.

[0081] To better illustrate the content of this invention, the optical field and the optical field mapping relationship after passing through the optical structure are described. The following description is not the only one available in this application. The general form of the optical field can be expressed as:

[0082]

[0083] Where E(r,t) represents the general plane wave representation of the light field, r=(x,y,z) represents the position vector, and A(r,t) represents the amplitude of the light field. ω represents the unit vector characterizing the polarization direction, k represents the wave vector, and φ represents the initial phase.

[0084] Assuming the initial light field does not pass through the optical structure (i.e., the outgoing light is only affected by the optical properties of the 2D display device), it can be expressed as:

[0085]

[0086] The light field changes after passing through the optical structure, as shown below:

[0087]

[0088] Where Ω represents a spatial angle, and dΩ represents the integral over the spatial angle.

[0089] The initial light field described above can be expressed in matrix form as follows:

[0090]

[0091] The light field after passing through the optical structure can be represented in matrix form as follows:

[0092]

[0093] The mapping relationship between the initial light field and the light field after passing through the optical structure can be expressed in matrix form as follows:

[0094] V(r,Ω)=M(r,Ω)V0(r)

[0095] Where M(r,Ω) represents the optical field mapping relationship corresponding to the optical structure.

[0096]

[0097] A(r,Ω)=a(r,Ω)A0(r), k(r,Ω)=R(r,Ω)k0(r), ω(r,Ω)=b(r,Ω)ω0(r), φ(r,Ω)=c(r,Ω)φ0(r),

[0098] The light field after passing through the optical structure is summed in all directions in space, i.e., integrated, to obtain the input light field in the observation space, i.e., the light field distribution model, which is expressed as:

[0099]

[0100] Its discrete form is expressed as:

[0101]

[0102] Where, N θ N represents the number of discrete points representing the polar angle θ. φ Δθ represents the number of discrete points representing the azimuth angle φ, Δθ represents the step size of the polar angle, and Δφ represents the step size of the azimuth angle. Unless otherwise stated, I(α) represents the view that should be seen at the observation position α, and β represents the virtual camera in the human visual coordinate system V. xyz The position below.

[0103] It should be noted that the above method can represent both the light field formed by light rays from the emitting unit to the viewing space and the light field formed by tracing rays from the viewer's pupil to the emitting unit. The matrices involved are related by convolution and deconvolution. The mapping relationship from the emitting unit in the screen coordinate system to the light field after passing through the optical structure is denoted as M. S-Ob (r, Ω), the mapping relationship of the light field from the light field in the observation space through the optical structure to the light-emitting unit in the screen coordinate system is denoted as M. Ob-S (r, Ω). Since some optical structures can actively adjust their corresponding optical field mapping relationship with time t, therefore, M S-Ob (r,Ω) can also be denoted as M S-Ob (r, Ω, t), M Ob-S (r,Ω) can also be denoted as M Ob-S (r, Ω, t).

[0104] Specifically, in step S2, the real-time acquisition of human eye coordinates includes: using existing pupil tracking algorithms (such as pupil distance tracking AI algorithms in visual solutions, infrared pupil recognition algorithms, etc.) to acquire the human eye plane coordinates and pupil orientation in real time, and using one of the following methods—ToF, structured light, multi-camera cross-positioning, etc.—to acquire depth values, thereby obtaining the human eye coordinates, i.e., the human eye six-degree-of-freedom coordinates.

[0105] Optionally, an empirical curve of visual characteristics can be determined using a pupil-tracking device. Since the empirical curve of human eye movement differs at different device sizes and viewing distances, the pre-determined empirical curve of visual characteristics will reduce the computational load during real-time pupil-tracking rendering. Under special conditions such as low light, when the uncertainty of real-time pupil-tracking position is large and the confidence level is low, the empirical curve of visual characteristics can be used instead of real-time pupil-tracking to obtain the human eye coordinates.

[0106] It should be noted that human eye coordinates represent a specific viewing position of the human eye in the observation space, looking at the screen from a certain viewing angle.

[0107] Furthermore, the step of obtaining the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current visible area and viewing angle based on the current equivalent lens pitch and equivalent offset, includes:

[0108] The current viewing distance is obtained based on the current human eye coordinates, and the current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model; the current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system; the current human eye parallax is obtained based on the current human eye coordinates, and the current equivalent pixel density is obtained based on the 2D display device parameters and the current human eye parallax;

[0109] The current reference visible area and the viewing angle range are obtained based on the current equivalent lens pitch; the current visible area is obtained by sliding the current reference visible area according to the current equivalent offset, and the current viewing angle range is obtained by sliding the current equivalent offset.

[0110] Specifically, the distance from the human eye to the display screen is obtained based on the current human eye coordinates, i.e., the viewing distance.

[0111] Specifically, the current equivalent lens pitch refers to the periodic cycle of light dispersion in all directions on the 2D display screen, formed by the refraction angle of the optical structure and the physical pitch of the lens, at the current viewing distance; that is, the current reference viewing area size. Taking a cylindrical lens grating as an example, the current equivalent lens pitch is obtained through the following formula:

[0112] A = A0 + ΔA,

[0113]

[0114] Where A represents the current equivalent lens pitch, which is also the current reference visible area size; ΔA represents the current equivalent lens pitch increment, which is also the current reference visible area size increment; A0 represents the original lens pitch in units of sub-pixels, which is also the original visible area period (i.e., the original reference visible area); H0 represents the optimal viewing distance preset according to the optical structure, and H represents the current viewing distance. The value represents the viewing angle within the original viewing zone period, and W represents the average width of the lens covering the 2D display screen (unit: millimeters).

[0115] It should be noted that the equivalent lens pitch or related equivalent parameters can be obtained by analogy for other optical structures.

[0116] Specifically, the current equivalent offset refers to the horizontal offset of the visible area and the viewing angle caused by the horizontal offset of the viewer's eye at the current viewing distance.

[0117] Specifically, assuming the tracking camera is installed above the horizontal center of the light field screen (i.e., the display screen) (the camera's optical axis installation error has been corrected, and the horizontal and vertical optical axes of the camera are parallel to the horizontal and vertical directions of the display screen, respectively), the current equivalent offset can be obtained using the following formula:

[0118]

[0119] Where ΔX is the current equivalent offset, x cam y cam These are the horizontal and vertical coordinates (normalized values ​​ranging from 0 to 1) of the current human eye within the frame of the tracking camera, fov. cam cam_ratio is the horizontal angle of the image sensor of the tracking camera, and cam_ratio is the aspect ratio of the image sensor of the tracking camera. It is the screen coordinate system S xy The horizontal coordinate (normalized value range 0-1), W OC It is the width of the display screen (unit: mm), α len It refers to the design tilt angle of the cylindrical lens on the 2D display screen.

[0120] It should be noted that due to the physical characteristics of the optical structure, the visible area is periodically arranged. When the viewing distance remains constant, the reference visible area remains unchanged. When the human eye moves laterally to the left relative to the display screen, the visible area and viewing angle shift laterally to the right. Therefore, the corresponding visible area can be obtained based on the equivalent offset and the current reference visible area, and the corresponding viewing angle can be obtained based on the equivalent offset and the current viewing angle range. According to real-time pupil tracking, when the viewing distance changes, the reference visible area changes, and the lens period in the screen coordinate system corresponding to the visible area changes. For example, the screen coordinate period facing the lens numbered m changes to the screen coordinate period facing the lenses numbered m-2, m-1, m+1, m+2, etc. If the viewing distance of the human eye coordinates at the current moment and the previous moment has not changed, then the reference visible area is the visible area at the previous moment. To achieve a good display effect, the center point of the visible area is aligned with the center of the human eye. When the viewing angle of the human eye obtained by pupil tracking exceeds the bottom screen period area corresponding to the front lens, light will reach the bottom screen through the adjacent lens. Assuming that the original equivalent pixel in the forward visible area is generated from a position on the m-th lenticular lens, when outside the forward visible area, the equivalent pixel may be generated from a position on the (m-1), m-2, or m+1, m+2-th lenticular lenses, depending on the mapping relationship of the optical structure. According to the light field mapping relationship, the sub-view has a mapping relationship with the pixels on the 2D screen, and the offset of the visible area is essentially the offset of its corresponding pixel on the 2D screen. Since the grating unit itself is approximately periodic, the pixel corresponding to the offset visible area is equivalent to the pixel corresponding to the original visible area.

[0121] Specifically, the current equivalent pixel density refers to the sum of the pixel densities (in sub-pixels per inch) of the theoretical resolution under different viewing angles of the light field display. The human eye parallax refers to the horizontal parallax between the left and right eyes.

[0122] Specifically, the current equivalent pixel density is obtained using the following formula:

[0123]

[0124] Where Dxpixel represents the current equivalent pixel density in the horizontal direction, Dypixel represents the current equivalent pixel density in the vertical direction, γ is the empirical correction coefficient for motion-sensing resolution, and N x-subpixel fov represents the horizontal screen resolution of the 2D display device (in subpixels per inch), A represents the current equivalent lens pitch (in subpixels), and fov A It is the viewing angle under the current equivalent lens pitch, and aspect_ratio represents the pixel aspect ratio of the 2D display device.

[0125] Preferably, 1≤γ≤5.

[0126] Specifically, the current reference viewing area refers to the area on the display screen where, at the current viewing distance, the center of the viewer's eye is directly facing the display screen, and the viewer sees the best 3D effect in the observation space; that is, the coordinate range of a sub-pixel in the screen coordinate system. It should be noted that the equivalent lens pitch can also be obtained for other optical structures based on their light field distribution characteristics.

[0127] Specifically, the current viewing angle range refers to the viewing angle of a viewer at the current viewing distance within the original viewing area period size. The current visible angle range when the human eye sees the best 3D effect is determined by the equivalent lens pitch (i.e., the size of the viewing area period); the total visible angle range due to the equivalent lens offset and sliding is the maximum angular range that this solution can provide for the 3D effect.

[0128] It should be noted that, through the light field mapping relationship, the mapping between the screen coordinate system, the virtual space world coordinate system and the observation space coordinate system can be realized, that is, the mapping between the current reference view area and the current viewable range can be realized.

[0129] It should be noted that, taking a cylindrical lens grating as an example of optical structure, a linear relationship is used to represent the refraction of light by the cylindrical lens, ignoring aberrations, off-axis nonlinearity, chromatic aberration, and brightness variations. Each pixel on the display screen is composed of multiple sub-pixels. Along the period of the cylindrical lens, each sub-pixel refracts light in a different direction on the focal plane, passing through the optical structure to reach the observation space. Assuming the sub-pixels are point light sources, and that the optical structure only changes the direction of the emitted light on the focal plane, the direction of refraction of each sub-pixel can be periodically obtained based on the relative positions of the sub-pixels and the optical structure on the display screen. Aligning the center of the viewer's eye with the center point of the current visible area yields the current dominant viewing angle.

[0130] Specifically, the current equivalent pixel density reflects the resolution of pixels in the horizontal and vertical directions. Based on the current equivalent pixel density, the number of sub-pixels corresponding to the current viewable area can be determined. Then, based on the actual application scenario, the computing power of the naked-eye 3D display system, and the equivalent pixel density, the total number of rays corresponding to the current viewable area can be obtained.

[0131] It should be noted that the parallax map refers to an image that utilizes the binocular parallax principle of the human eye, allowing viewers to perceive 3D visual effects without wearing auxiliary equipment. The total number of sub-pixels on the 2D display screen can be obtained based on the equivalent pixel density and the 2D display screen size. Each sub-pixel can emit several rays of light. The number of rays emitted by each sub-pixel is determined by the actual application scenario and the computing power of the naked-eye 3D display system. Therefore, the total number of sub-pixels contained in the current light field screen is obtained through the equivalent pixel density and the 2D display screen size. Multiplying the total number of sub-pixels by the number of rays emitted by each sub-pixel yields the total number of rays corresponding to the current light field screen, and thus the total number of rays corresponding to the current visible area.

[0132] Furthermore, the parameter space of the light rays includes the starting point and direction of each light ray. The starting point of the light ray refers to its coordinates in the observation space coordinate system corresponding to the emission of the light ray from the human eye. The direction of the light ray refers to its direction in the observation space coordinate system corresponding to the emission of the light ray from the human eye.

[0133] It should be noted that the pupil position is obtained based on the current human eye coordinates, the human eye size is obtained based on the characteristics of the human eye, and the value range corresponding to the starting point of the light ray is obtained based on the pupil position and the human eye size; the value range corresponding to the direction of the light ray is obtained based on the current visible field of view. The value ranges corresponding to the starting point and direction of the light ray constitute the parameter space of the light ray in the observation space coordinate system.

[0134] Understandably, based on the distribution area of ​​the light starting point (i.e. the range of values ​​corresponding to the light starting point) and the light field mapping relationship, the screen coordinate position in the 3D virtual space corresponding to each eye of each viewer at the viewing position is obtained in the observation space. The screen coordinate position is used as the position of the starting point of ray tracing after light field mapping; the direction pointing to the light field screen in the observation space is used as the direction of the light after light field mapping.

[0135] Specifically, in step S3, setting the probability density function of the parameter space prior distribution of the light ray includes: taking the center point of the visible area corresponding to the observation space coordinate system as the mean position of the Gaussian distribution, and taking the size of the visible area as three times the standard deviation of the Gaussian distribution, thereby obtaining the probability density function of the parameter space prior distribution of the light ray.

[0136] It should be noted that other prior distributions can be used instead of the Gaussian distribution. The visible area corresponding to the observation space coordinate system refers to the area that the human eye can observe in the observation space, which is composed of the starting point and the range of values ​​of the direction of the light rays.

[0137] Specifically, the human eye characteristics include the influence of the retina and cone cell distribution on the received light field. The probability density functions of the human eye characteristics related to light density include visual acuity curves, etc.; the visual acuity curve represents the functional relationship curve between light density (related to human eye resolution) and viewing angle, that is, the probability density function of viewing angle with respect to the amount of light in the visual coordinate system.

[0138] Specifically, the step of obtaining the probability density function of the human eye coordinates based on the pupil-tracking algorithm and the pupil-tracking system delay includes: inferring the probability distribution function of the human eye position from the pupil-tracking algorithm; performing Kalman filtering (including unscented Kalman filtering, extended Kalman filtering, and particle filtering) on ​​the probability distribution function of the human eye position to obtain a smooth real-time pupil-tracking curve; obtaining the pupil-tracking system delay based on the factory settings or manual calibration of the pupil-tracking device; establishing an empirical curve of the pupil-tracking system delay based on the pupil-tracking system delay; and multiplying the real-time pupil-tracking curve and the empirical curve of the pupil-tracking system delay to obtain the probability density function of the human eye coordinates.

[0139] It should be noted that due to factors such as the acquisition, transmission, and inference of the eye-tracking signal, as well as the eye-tracking frame rate, there is a certain error between the real-time eye coordinates obtained by the eye-tracking algorithm and the actual eye coordinates. Based on the eye-tracking frame rate and the predicted eye movement position and velocity, the probability distributions of both are obtained, forming the probability density function of each viewer at the current eye coordinates obtained by the eye-tracking algorithm. This function is then corrected using Kalman filtering to obtain the real-time eye-tracking curve corresponding to the algorithm. In particular, considering that human eye motion often tends to be a nonlinear system, the unscented Kalman filtering method has certain advantages in nonlinear estimation and processing speed.

[0140] Furthermore, the probability density function of the human eye characteristics, the probability density function of the human eye coordinates, and the probability density function of the prior distribution of the light parameter space are multiplied together to obtain the probability density cloud corresponding to the current disparity map.

[0141] It should be noted that a point in the probability density cloud represents the probability of a point in the parameter space of a ray, that is, the probability of a ray with that point as a parameter. Once the starting point and direction of the ray are determined, the color corresponding to the ray at that starting point and direction can be obtained through 2D ray tracing technology and a light field distribution model.

[0142] Preferably, a color difference model is obtained based on the characteristics of the human eye, representing the relationship between color difference and viewing angle within a certain angle (e.g., 2°, 10°) of the pupil's dominant visual field. This color difference model is then used to obtain a color difference model based on the human eye's visual coordinate system V. xyz Color correction matrix with spatial distribution.

[0143] It should be noted that the color correction matrix described above is used to correct the color of the corresponding light rays in order to improve the realism of the 3D display effect.

[0144] Furthermore, the step of obtaining the light corresponding to the current disparity map of each viewer's eyes at the viewing position based on the probability density cloud and the total number of light rays includes:

[0145] Based on the probability density cloud, the probability corresponding to each point in the parameter space is obtained. The probability corresponding to each point in the parameter space is multiplied by the total number of light rays to obtain the number of light rays corresponding to each point in the parameter space. Based on each point in the parameter space and the number of light rays corresponding to it, the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position are obtained.

[0146] It should be noted that a point in the parameter space contains the values ​​of each parameter of a ray, thus representing a ray with a specific starting point and direction. The number of rays corresponding to a point in the parameter space represents the number of rays with the parameter values ​​corresponding to that point as parameters. All rays with specific parameters and their quantities constitute the rays corresponding to the current disparity map of each viewer's eyes at the viewing position.

[0147] Specifically, in step S4, for a viewing position, the expression for rendering the model is:

[0148]

[0149] Among them, res S The trace represents the resolution of the accumulated and post-accumulation light field map, where t represents time; field This refers to the process of ray tracing performed by a 3D display device; N light Indicates the total number of light rays, o Ob o WThese represent the light rays in the observation space coordinate system Ob. xyz The actual position below, and the corresponding 3D virtual space world coordinate system W xyz The starting point below, d Ob (o Ob ,F Ob (r')) indicates that light originates from the observation space Ob. xyz The starting point o Ob To the surface of the light field structure F Ob The normalized ray direction of (r'), d W (o W ,S xy (r') indicates that light passes through the optical structure at point S. xy (r') corresponds to the 3D virtual space world coordinate system W xyz Normalized ray direction under, χ(o) Ob ,d Ob (o Ob ,F Ob (r')),t))χ(o W ,d W (o W ,S xy (r'),t)) represent the probability density cloud in the observation space coordinate system and the probability density cloud in the 3D virtual space coordinate system, respectively.

[0150] It should be noted that the process of performing ray tracing is that the 2D display screen emits rays based on the starting point and direction of the rays determined by the probability density cloud and the number of rays corresponding to each point in the parameter space. The rays are then mapped by the ray structure to change the starting point and direction of each ray on the 2D display screen (corresponding to the viewport plane of the 3D virtual space). The mapped rays are then accumulated in the observation space to form a ray field map.

[0151] Similarly, the expression for the rendering model at each viewing position can be obtained, and then the light field map can be obtained by summing the light rays corresponding to the current disparity map at each viewing position.

[0152] Furthermore, the light field map is projected onto a display screen to achieve naked-eye 3D display.

[0153] It should be noted that in the ray tracing rendering method, each ray of light in the light field map is refracted and focused onto the screen coordinates of the 2D display panel through the cylindrical lens grating in the paraxial region of the lens.

[0154] Furthermore, the present invention can either render directly using a parallax map, that is, provide a 3D effect directly based on the parallax between the left and right views seen by the human eye; or render by interweaving viewpoint maps, that is, provide a 3D effect based on the parallax generated in the virtual space by a virtual camera when the viewer moves.

[0155] Specifically, this invention directly provides a 3D effect through the parallax of the left and right views as perceived by the human eye. The method and system provided by this invention support both binocular naked-eye 3D rendering and naked-eye 3D rendering from multiple simultaneous viewing positions.

[0156] Understandably, this invention acquires six-DOF human eye coordinates through real-time pupil tracking and determines the parameters of each ray in the light field map based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It considers the attributes of each observer's eye in the light field at any given moment and the full parameter mapping relationship through the optical structure, adjusting the equivalent offset of the corresponding sub-pixels on the screen. This ensures that the visible area always aligns with the center of the human eye coordinates, even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience. This invention directly generates the light field map required for human eye parallax, eliminating the need to generate subviews and interweave them, thus avoiding crosstalk between subviews and problems such as 3D image tearing. The performance overhead of naked-eye 3D ray tracing rendering compared to 2D ray tracing rendering is negligible, far less than the performance overhead of naked-eye 3D rasterization rendering compared to 2D rasterization rendering. Based on the spatial resolution of the optical structure and the temporal resolution of the pupil-tracking system, this invention adjusts the density of light rays required for each viewer's light field map, reducing the light density at non-primary viewpoints and enhancing the realism of the 3D visual effect. While ensuring the naked-eye 3D display effect, it effectively reduces the resource consumption of the light field map in storage, transmission, and rendering, significantly saving bandwidth.

[0157] Another specific embodiment of the present invention, taking a cylindrical lens grating as an example, discloses a parameter design method for the optical structure of a naked-eye 3D display device applied to the naked-eye 3D ray tracing rendering method, the method comprising:

[0158] Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees.

[0159] Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees;

[0160] Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating. Spend.

[0161] Optionally, for large-size light field screens, due to the increased number of applications where multiple people view the screen simultaneously, an additional redundant angle of α3 degrees is needed to avoid crosstalk, resulting in the design degree of the light field screen grating. Spend.

[0162] It should be noted that the light field screen refers to the screen of a naked-eye 3D display device, including the screen of a 2D display device and the optical structure.

[0163] For example, design a 15.6-inch desktop monitor to achieve a 16:9 landscape multi-view stereoscopic display, providing users with a high-quality 3D visual experience without glasses. Within a specified cost, a 4K 2D resolution LCD screen is used, with traditional adjacent RGB pixel arrangement. Based on manufacturing capabilities and experience, the spatial resolution redundancy angle of each subpixel is estimated to be 4°. According to the design goals, the interpupillary distance angle in this application scenario is approximately 7.0°, and the temporal resolution redundancy angle is 5°. Therefore, the monitor needs to have a horizontal... The field of view is 10 degrees.

[0164] Understandably, the naked-eye 3D ray tracing rendering method of this invention obtains the design parameters of the light field screen of naked-eye 3D display devices. Through dynamic light field mapping relationships, it ensures that the physical limits of viewing angle, clarity, and image quality are reached on a given optical structure, providing a foundation for the design of the optical structure of naked-eye 3D display devices. It enables the design of holographic display devices with a larger field of view, higher clarity, greater depth of field, horizontal and vertical screen compatibility, and simultaneous viewing by multiple people at a lower cost, breaking the impossible triangle of clarity, depth of field, and FOV.

[0165] Another specific embodiment of the present invention discloses a naked-eye 3D ray tracing rendering device. For example... Figure 4 As shown, the device includes:

[0166] Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module;

[0167] The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. Based on the current human eye coordinates, the human eye characteristic model, and the rendering model, it obtains a light field map and sends it to the naked-eye 3D display device.

[0168] A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

[0169] Specifically, the data acquisition device includes a camera or sensor, an eye-tracking module, and a data acquisition module. The camera or sensor captures facial images in real time and sends them to the data acquisition module. The eye-tracking module acquires eye movement parameters in real time, such as pupil size, fixation point, and fixation direction, and sends them to the data acquisition module. The data acquisition module obtains the eye coordinates (i.e., pupil tracking values) from the facial image and eye movement parameters and sends them to the control module. It also acquires parameters for the glasses-free 3D display device and sends them to the control module. The parameters for the glasses-free 3D display device include 2D display device parameters and optical structure parameters.

[0170] Furthermore, the control module includes a memory and a computing unit. The memory includes ROM and RAM, used to store relevant parameters, images, textures, and other data. The computing unit includes a rendering parameter calculation module and a view processing module. The computing unit includes a GPU and a VPU.

[0171] Specifically, the rendering parameter calculation module is used to obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; to obtain the light field distribution model based on the 2D display device parameters and the light field mapping relationship corresponding to the optical structure; to obtain the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure based on the current human eye coordinates and the light field distribution model; to obtain the current visible area and viewing angle based on the current equivalent lens pitch and equivalent offset; to obtain the total number of rays corresponding to the current visible area based on the current equivalent pixel density and the application scenario; and to obtain the parameter space of the rays based on the current human eye coordinates and the viewing angle. The module sets the probability density function of the prior distribution of the ray parameter space; establishes a probability density function of human eye characteristics related to ray density based on human eye characteristics; and obtains the probability density function of human eye coordinates based on the pupil tracking algorithm and pupil tracking system delay. The module multiplies the above three probability density functions to obtain the probability density cloud corresponding to the current disparity map; and obtains the rays corresponding to the current disparity map for each eye of each viewer at the viewing position based on the probability density cloud and the total number of rays. Finally, the module sends the rays corresponding to the current disparity map to the view processing module.

[0172] Specifically, the view processing module is used to establish a rendering model, accumulate the light rays corresponding to the current parallax map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0173] Understandably, the naked-eye 3D rendering device based on the light field distribution model, which applies the naked-eye 3D ray tracing rendering method, supports both providing naked-eye 3D rendering for only two eyes, i.e., providing a 3D display effect for one viewer when moving in space, and providing multiple viewing positions simultaneously, i.e., providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0174] Another specific embodiment of the present invention discloses a glasses-free 3D display device. For example... Figure 5 As shown, the naked-eye 3D display device applies the naked-eye 3D ray tracing rendering method, including:

[0175] 2D display devices or light sources used to provide 2D images;

[0176] Optical structures are used to provide a parallax map for the viewer's eyes at the same time, thereby enabling naked-eye 3D display and holographic display;

[0177] Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

[0178] Specifically, the 2D display device includes a plurality of periodically arranged light-emitting units and pixels capable of displaying red, green, and blue colors. Preferably, the 2D display device includes an LCD panel or an OLED panel.

[0179] Specifically, the optical structure includes a slit grating, a cylindrical lens grating, an adjustable liquid crystal grating, and a spatial light modulator. Preferably, when providing a 3D display effect for multiple viewers simultaneously, i.e., supporting multiple viewing positions at the same time, a cylindrical lens grating is used as the optical structure to provide a larger field of view.

[0180] Specifically, the input and output interfaces include a USB interface, a Mini HDMI interface, and an HDMI interface.

[0181] Understandably, the naked-eye 3D display device, by applying the naked-eye 3D ray tracing rendering method, supports both providing naked-eye 3D rendering for only two eyes, i.e., providing a 3D display effect for one viewer while they move in space, and simultaneously providing multiple viewing positions, i.e., providing a 3D display effect for multiple viewers while they move in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0182] Another specific embodiment of the present invention discloses an electronic device for implementing embodiments of the present invention. The electronic device includes:

[0183] Memory, used to store computer programs;

[0184] A processor for executing the computer program to implement the naked-eye 3D ray tracing rendering method as described above.

[0185] It should be noted that the electronic device includes various forms of digital computers, such as desktop computers, workbenches, servers, etc., as well as various forms of mobile devices, such as mobile phones, wearable devices, etc. The processor includes a computing unit, an input unit, an output unit, and a communication unit. The computing unit includes various general-purpose and / or dedicated processing components with processing and computing capabilities, such as a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), controllers, microcontrollers, etc., used to execute corresponding processing according to computer programs. The input unit includes a keyboard and a mouse; the output unit includes various types of displays and speakers; the communication unit includes a network card, a modem, and a wireless communication transceiver, enabling the electronic device to exchange information / data with other devices through computer networks and / or telecommunications networks.

[0186] Another specific embodiment of the present invention provides a computer program product, the computer program product including a computer program that can be stored on a readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the above-described naked-eye 3D ray tracing rendering method.

[0187] Another specific embodiment of the present invention discloses a computer-readable storage medium for storing a computer program that executes the aforementioned naked-eye 3D ray tracing rendering method. It should be noted that the readable storage medium includes volatile memory and non-volatile memory.

[0188] Compared with existing technologies, the beneficial effects of the naked-eye 3D ray tracing rendering method, apparatus, and naked-eye 3D display device provided by this invention are as follows:

[0189] 1. This invention acquires six-DOF human eye coordinates through real-time pupil tracking and determines the parameters of each ray in the light field map based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It considers the attributes of each observer's eye in the light field at any given moment and the full parameter mapping relationship through the optical structure, adjusting the equivalent offset of the corresponding sub-pixels on the screen. This ensures that the visible area always aligns with the center of the human eye coordinates, even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience.

[0190] 2. This invention directly generates the light field map required for human eye parallax, without generating subviews and interweaving them, thus eliminating crosstalk between subviews and avoiding problems such as 3D image tearing; the performance overhead of naked-eye 3D ray tracing rendering compared to 2D ray tracing rendering is almost negligible, far less than the performance overhead of naked-eye 3D rasterization rendering compared to 2D rasterization rendering.

[0191] 3. This invention obtains the probability density distribution of various parameters of light rays in the light field map required by each viewer based on the light field distribution and the parallax required by the human eye characteristic model. It modulates the probability density distribution according to the pupil-tracking delay empirical curve to obtain all the light rays in the light field map, thereby generating a light field map. This enhances the realism of the 3D visual effect. While ensuring the naked-eye 3D display effect, it effectively reduces the resource consumption of the light field map in terms of storage, transmission and rendering, and saves bandwidth significantly.

[0192] 4. This invention obtains the design parameters of the light field screen of a naked-eye 3D display device through a naked-eye 3D ray tracing rendering method. By using dynamic light field mapping relationships, it ensures that the physical limits of viewing angle, clarity, and imaging quality are reached on a given optical structure, thus providing a foundation for the design of the optical structure of naked-eye 3D display devices.

[0193] 5. This invention takes into account both the density of the light rays required for the light field map and the virtual spatial position corresponding to each observation position provided to the user. Therefore, the method and system provided by this invention can support both providing naked-eye 3D rendering for only two eyes, that is, providing a 3D display effect for one viewer when moving in space, and providing multiple observation positions at the same time, that is, providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0194] 6. This invention predicts the human eye movement trajectory by real-time pupil tracking combined with unscented Kalman filtering. Under special conditions such as low light, it uses visual characteristic empirical curves to modulate real-time pupil tracking to obtain the human eye coordinates, making the human eye coordinates closer to the true value and reducing problems such as dizziness and ghosting.

[0195] 7. This invention utilizes the depth information of the rendering to apply a depth-based Gaussian blur to the target to be rendered, applying a depth-of-field effect to objects far from the focal plane, thereby compensating for the appearance of distant objects and reducing crosstalk.

[0196] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0197] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A naked-eye 3D ray tracing rendering method, characterized in that, The method includes the following steps: Step S1: Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the parameters of the 2D display device and the light field mapping relationship corresponding to the optical structure. Step S2: Real-time acquisition of human eye coordinates; based on the current human eye coordinates and light field distribution model, obtain the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure; based on the current equivalent lens pitch and equivalent offset, obtain the current visible area and viewing angle; based on the current equivalent pixel density and application scenario, obtain the total number of rays corresponding to the current visible area; based on the current human eye coordinates and viewing angle, obtain the parameter space of the rays. Step S3: Set the probability density function of the prior distribution of the parameter space of the light rays; establish the probability density function of the human eye characteristics related to the light density based on the characteristics of the human eye; obtain the probability density function of the human eye coordinates based on the pupil tracking algorithm and the pupil tracking system delay; multiply the above three probability density functions together to obtain the probability density cloud corresponding to the current disparity map; and obtain the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position based on the probability density cloud and the total number of light rays. Step S4: Establish a rendering model, sum the light rays corresponding to the current parallax map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

2. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, The process of obtaining the current equivalent lens pitch, equivalent offset, and equivalent pixel density of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current visible area and viewing angle based on the equivalent lens pitch and equivalent offset, includes: The current viewing distance is obtained based on the current human eye coordinates, and the current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model; the current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system; the current human eye parallax is obtained based on the current human eye coordinates, and the current equivalent pixel density is obtained based on the 2D display device parameters and the current human eye parallax; The current reference visible area and the viewing angle range are obtained based on the current equivalent lens pitch; the current visible area is obtained by sliding the current reference visible area according to the current equivalent offset, and the current viewing angle range is obtained by sliding the current equivalent offset.

3. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, The method of setting the probability density function of the parameter space prior distribution of the light ray includes: taking the center point of the visible area corresponding to the observation space coordinate system as the mean position of the Gaussian distribution, and taking the size of the visible area as three times the standard deviation of the Gaussian distribution, thereby obtaining the probability density function of the parameter space prior distribution of the light ray.

4. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, The process of obtaining the probability density function of human eye coordinates based on the pupil-tracking algorithm and the pupil-tracking system delay includes: inferring the probability distribution function of human eye position from the pupil-tracking algorithm; performing Kalman filtering on the probability distribution function of human eye position to obtain a smooth real-time pupil-tracking curve; obtaining the pupil-tracking system delay based on the factory settings or manual calibration of the pupil-tracking device; establishing an empirical curve of the pupil-tracking system delay based on the pupil-tracking system delay; and multiplying the real-time pupil-tracking curve and the empirical curve of the pupil-tracking system delay to obtain the probability density function of human eye coordinates.

5. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, Based on the characteristics of the human eye, a color difference model is obtained, which represents the relationship between color difference and viewing angle within a certain angle of the pupil's main viewing angle. A color correction matrix based on the spatial distribution on the human eye's visual coordinate system is obtained through the color difference model.

6. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, The light rays corresponding to the current disparity map of each viewer's eyes at the viewing position, obtained based on the probability density cloud and the total number of light rays, include: Based on the probability density cloud, the probability corresponding to each point in the parameter space is obtained. The probability corresponding to each point in the parameter space is multiplied by the total number of light rays to obtain the number of light rays corresponding to each point in the parameter space. Based on each point in the parameter space and the number of light rays corresponding to it, the light rays corresponding to the current disparity map of each eye of each viewer at the viewing position are obtained.

7. The naked-eye 3D ray tracing rendering method according to claim 1, characterized in that, For a given viewing position, the expression for rendering the model is: Among them, res S The trace represents the resolution of the accumulated and post-accumulation light field map, where t represents time; field This refers to the process of ray tracing performed by a 3D display device; N light Indicates the total number of light rays, o Ob o W These represent the light rays in the observation space coordinate system Ob. xyz The actual position below, and the corresponding 3D virtual space world coordinate system W xyz The starting point below, d Ob (o Ob ,F Ob (r')) indicates that light originates from the observation space Ob. xyz The starting point o Ob To the surface of the light field structure F Ob The normalized ray direction of (r'), d W (o W ,S xy (r') indicates that light passes through the optical structure at point S. xy (r') corresponds to the 3D virtual space world coordinate system W xyz Normalized ray direction under, χ(o) Ob ,d Ob (o Ob ,F Ob (r')),t))χ(o W ,d W (o W ,S xy (r'),t)) represent the probability density cloud in the observation space coordinate system and the probability density cloud in the 3D virtual space coordinate system, respectively.

8. A parameter design method for the optical structure of a glasses-free 3D display device applied to the glasses-free 3D ray tracing rendering method according to any one of claims 1-7, characterized in that, The method includes: Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees. Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees; Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating. Spend.

9. A naked-eye 3D rendering device based on light field distribution, characterized in that, The device includes: Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module; The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. It obtains a light field map based on the current human eye coordinates, the human eye characteristic model, and the rendering model and sends it to the naked-eye 3D display device. A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

10. A glasses-free 3D display device, characterized in that, The naked-eye 3D display device applies a naked-eye 3D ray tracing rendering method, including: 2D display devices or light sources used to provide 2D images; Optical structures are used to provide viewers with different perspectives for each eye at the same time, thereby enabling naked-eye 3D displays and holographic displays; Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

Images