Autostereoscopic 3D rendering method and device based on light field distribution, parameter design method and display device

Abstract

The present application relates to a naked-eye 3D rendering method and device based on light field distribution, a parameter design method and a display device, and belongs to the field of naked-eye 3D rendering and holographic images. The method comprises the following steps: obtaining a light field distribution model based on display device parameters and a light field mapping relationship; obtaining a current equivalent lens pitch and an equivalent offset of an optical structure according to a current eye coordinate and the light field distribution model, obtaining a current reference viewable area based on the current equivalent lens pitch, and arranging viewpoints in the current reference viewable area; obtaining a current viewable area based on the current equivalent offset, and arranging viewpoints in the current viewable area; rearranging the viewpoints in the current viewable area according to the arrangement of the viewpoints in the current reference viewable area to obtain a viewpoint map; interleaving the viewpoint map to obtain a light field map, and projecting the light field map to a display screen to realize naked-eye 3D rendering. The method solves the problems of low display resolution, poor definition and dizziness caused by the current naked-eye 3D rendering and display method through viewpoint rearrangement.

Description

Technical Field

[0001] This invention relates to the field of naked-eye 3D rendering and holographic imaging technology, and in particular to naked-eye 3D rendering methods, apparatus, parameter design methods and display devices based on light field distribution. Background Technology

[0002] Current naked-eye 3D rendering methods typically use multiple virtual cameras to capture virtual 3D scene views from different angles, and render the 3D scene view corresponding to each virtual camera at the same resolution; during rendering, the 3D scene is projected onto a 2D plane according to the projection relationship between the 3D model and the display screen coordinates in the world coordinate system.

[0003] Currently, some methods for naked-eye 3D rendering and display employ binocular left and right views. Based on the eye coordinates obtained through pupil tracking, the left and right views are arranged at the positions of the views after refraction by the cylindrical lens array along the viewer's left and right lines of sight. However, when there is a certain difference between the eye coordinates obtained through pupil tracking and the actual eye coordinates, dizziness can occur. Other methods divide the period of the cylindrical lens array into multiple viewing zones, periodically arranging views at different angles within each zone. However, when the viewer's eye crosses the boundary of the viewing zone, 3D imaging distortion occurs. Furthermore, due to the splitting of light across multiple viewing zones, the equivalent pixel distance between the same view increases, resulting in reduced display resolution and poor display clarity.

[0004] In summary, existing naked-eye 3D rendering and display methods have the following drawbacks:

[0005] 1. Low display resolution and poor display clarity. Improving display resolution without increasing costs has always been one of the challenges in designing glasses-free 3D rendering and display methods.

[0006] 2. The binocular naked-eye 3D rendering and display method based on pupil tracking has high requirements for pupil tracking performance (i.e., the temporal resolution of the pupil tracking system). When there is a certain difference between the human eye coordinates obtained by pupil tracking and the actual human eye coordinates, it will cause dizziness. In addition, the left and right eye views will overlap due to factors such as angular resolution, spherical aberration and other spatial resolution of the optical system.

[0007] 3. Multi-viewpoint naked-eye 3D rendering and display methods: When the human eye crosses the boundary of the viewpoint, it will cause distortion in 3D imaging. In addition, due to the limited spatial resolution of the grating structure, it will also cause blurring and dizziness when entering and leaving the screen.

[0008] 4. Render the view corresponding to each virtual camera using the same resolution. In practice, it is usually necessary to render the view corresponding to dozens of virtual cameras. Since a large number of views are rendered at the same resolution, especially as the number of viewpoints increases, the resource consumption generated by rendering will increase significantly, resulting in serious resource waste. Summary of the Invention

[0009] Based on the above analysis, the present invention aims to provide a naked-eye 3D rendering method, device, parameter design method and display device based on light field distribution, in order to solve the problems of low display resolution, poor clarity and dizziness, jump, view overlap and blurring when entering and leaving the screen caused by current naked-eye 3D rendering and display methods.

[0010] This invention provides a naked-eye 3D rendering method, apparatus, parameter design method, and display device based on light field distribution. The naked-eye 3D rendering method based on light field distribution includes the following steps:

[0011] Step S1: Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the parameters of the 2D display device and the light field mapping relationship corresponding to the optical structure.

[0012] Step S2: Obtain the human eye coordinates in real time. Based on the current human eye coordinates and the light field distribution model, obtain the current equivalent lens pitch and equivalent offset of the optical structure. Based on the current equivalent lens pitch, obtain the current reference viewing area and viewing angle range. Based on the current reference viewing area, obtain the number of virtual cameras corresponding to each eye of each viewer at the viewing position. Based on the current reference viewing area and viewing angle range, obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture. Arrange the viewpoints within the current reference viewing area.

[0013] Step S3: Based on the current equivalent offset, obtain the rendering parameters of each virtual camera or video texture corresponding to each eye of each viewer at the viewing position, and arrange the viewpoints in the current view area; rearrange the viewpoints in the current view area according to the viewpoint arrangement in the current reference view area to obtain the viewpoint map.

[0014] Step S4: Establish a rendering model, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0015] Furthermore, based on the characteristics of the human eye, a human eye characteristic model corresponding to the rendering parameters of the virtual camera or the rendering parameters of the video texture is established. Fourier transform or Gaussian convolution is performed on the human eye characteristic model, and the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture at each viewing position are obtained based on the current human eye coordinates.

[0016] Based on the weights or convolution kernels updated in step S3, the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position are used to arrange the viewpoints in the current view area; the viewpoints in the current view area are rearranged according to the arrangement of viewpoints in the current reference view area to obtain a viewpoint map; the viewpoint map is interwoven with the rendering model to obtain a light field map; and the light field map is projected onto the display screen to achieve naked-eye 3D rendering.

[0017] Furthermore, the step of obtaining the current equivalent lens pitch and equivalent offset of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current reference viewing area and viewing angle range based on the current equivalent lens pitch includes:

[0018] The current viewing distance is obtained based on the current human eye coordinates. The current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model. The current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system. The current reference viewing area and viewing angle range are obtained based on the current equivalent lens pitch.

[0019] The number of virtual cameras corresponding to each viewer's eye at the viewing position, obtained based on the current reference view area, includes:

[0020] Based on the current reference view area, the number of viewpoints corresponding to each eye of each viewer at the viewing position is obtained. One viewpoint corresponds to one virtual camera, thus obtaining the number of virtual cameras.

[0021] Furthermore, the step of obtaining the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position based on the current equivalent offset, and arranging the viewpoints within the current visible area, includes:

[0022] The current reference view area is obtained by sliding it according to the current equivalent offset. The current viewing angle is obtained by sliding it according to the current equivalent offset. The current main viewing angle is obtained by aligning the center position of the current human eye with the center point of the current view area.

[0023] Based on the current main viewpoint and the view area and view angle, the rendering parameters of each virtual camera or the rendering parameters of the video texture are adjusted to obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each viewer's eye at the viewing position.

[0024] Based on the rendering parameters of each virtual camera or the rendering parameters of the video texture, rasterization is performed to generate a sub-view of the target to be rendered;

[0025] All subviews corresponding to the current viewport are organized together according to a storage format, which includes a grid diagram, multi-view difference encoding, and texture array; wherein each subview corresponds to a viewpoint within the current viewport, and each subview has a corresponding number.

[0026] Further, the step of rearranging the viewpoints within the current view area according to the viewpoint arrangement within the current reference view area to obtain the viewpoint map includes:

[0027] Obtain the overlapping and non-overlapping areas of the current reference view area and the current view area. The subview numbers corresponding to the overlapping areas remain unchanged. Map the subview numbers corresponding to the non-overlapping areas of the current view area to the non-overlapping areas of the current reference view area. Rearrange the viewpoints in the current view area to obtain the viewpoint map.

[0028] Furthermore, the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture for each viewer at the viewing position, based on the current human eye coordinates, include:

[0029] The visual coordinate system is transformed to the observation space coordinate system based on the current human eye coordinates. This transforms the weights or convolution kernels of the rendering parameters of each virtual camera or video texture in the visual coordinate system into the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or video texture at each viewer's eye position.

[0030] Furthermore, for a viewing position, without enhancing the first-person perspective effect, the expression for the rendering model is:

[0031] in, Let represent the i-th sub-view in the viewpoint diagram corresponding to the main perspective α at time t. λ represents the resolution of the i-th subview; ij γ represents the weight of each rendering parameter of the i-th subview, where j = 0, ..., n, and n represents the number of rendering parameters for the virtual camera or the video texture; i This represents the weight of the i-th subview;

[0032] When enhancing the first-person perspective effect, the expression for rendering the model is:

[0033] The following formula is used to assign weights to the main viewpoint in the rendering model:

[0034]

[0035] Where β represents the viewpoint corresponding to the non-prime view subview. This means that the subview corresponding to viewpoint β is superimposed on the subview corresponding to the main viewpoint.

[0036] Furthermore, a parameter design method for the optical structure of a glasses-free 3D display device applied to the aforementioned glasses-free 3D rendering method based on light field distribution, the method comprising:

[0037] Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees.

[0038] Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees;

[0039] Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating. Spend.

[0040] Furthermore, a naked-eye 3D rendering device based on light field distribution, the device comprising:

[0041] Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module;

[0042] The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. It obtains a light field map based on the current human eye coordinates, the human eye characteristic model, and the rendering model and sends it to the naked-eye 3D display device.

[0043] A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

[0044] Furthermore, a glasses-free 3D display device, wherein the glasses-free 3D display device applies a glasses-free 3D rendering method based on light field distribution, including:

[0045] 2D display devices or light sources used to provide 2D images;

[0046] Optical structures are used to provide viewers with different perspectives for each eye at the same time, thereby enabling naked-eye 3D displays and holographic displays;

[0047] Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

[0048] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0049] 1. This invention acquires six-DOF human eye coordinates through real-time pupil tracking and rearranges the viewpoint map of the target to be rendered based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It improves the interlacing results by considering the attributes of each eye of each observer in the light field at any given moment and the full parameter mapping relationship through the optical structure. It adjusts the equivalent offset of corresponding sub-pixels on the screen, ensuring that the visible area always aligns with the center of the human eye coordinates even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience.

[0050] 2. This invention obtains the viewing distance based on the eye coordinates acquired through real-time pupil tracking, calculates the equivalent lens pitch based on the viewing distance, and recalculates the reference viewing area and the number and distribution of viewpoints within the viewing area. For example, when the viewing distance is smaller or larger than the standard distance, the equivalent lens pitch increases or decreases, the number of viewpoints within the viewing area increases or decreases, and the number of viewpoints participating in the interlacing increases or decreases accordingly. This allows for the rearrangement of more or fewer sub-views in the viewpoint map, precisely controlling the number of sub-views required for each viewer's eyes, thereby reducing the consumption of required resources and avoiding problems such as 3D imaging tearing.

[0051] 3. Based on the spatial resolution of the optical structure and the temporal resolution of the pupil-tracking system, this invention adjusts the weight of the rendering parameters of the virtual camera that each viewer's eye should use in each sub-view at each perspective. This reduces the resolution, virtual camera intrinsic and extrinsic parameters, contrast, brightness, etc. of the non-primary view sub-view, thereby reducing crosstalk between views and enhancing the realism of the 3D visual effect. While ensuring the naked-eye 3D display effect, it effectively reduces the resource consumption of viewpoint maps in terms of storage, transmission, and rendering, and significantly saves bandwidth.

[0052] 4. The present invention obtains the design parameters of the light field screen of naked-eye 3D display device based on the naked-eye 3D rendering method of light field distribution. Through dynamic light field mapping relationship, it ensures that the physical limits of viewing angle, clarity and imaging quality are reached on a given optical structure, thus providing a basis for the design of optical structure of naked-eye 3D display device.

[0053] 5. This invention can change the correspondence between subviews in the visible area after the observation position is shifted, and it can also change the intrinsic and extrinsic parameters of the virtual camera corresponding to each subview. At the same time, it considers the interlacing method corresponding to the viewpoint map and the virtual spatial position corresponding to each observation position provided to the user. Therefore, the method and system provided by this invention can support providing only binocular naked-eye 3D rendering, that is, providing a 3D display effect for one viewer when moving in space; and it can also support providing multiple observation positions at the same time, that is, providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0054] 6. This invention predicts the trajectory of human eye movement by real-time pupil tracking combined with unscented Kalman filtering. Under special conditions such as low light, it uses the empirical curve of visual characteristics to replace real-time pupil tracking to obtain the human eye coordinates, making the human eye coordinates closer to the true value and reducing problems such as dizziness and ghosting.

[0055] 7. This invention utilizes the depth information of the rendering to apply a depth-based Gaussian blur to the target to be rendered, applying a depth-of-field effect to objects far from the focal plane, thereby compensating for the appearance of distant objects and reducing crosstalk.

[0056] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0058] Figure 1 This is a flowchart of a naked-eye 3D rendering method based on light field distribution, as described in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the spatial light field according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of a viewpoint from an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of view alignment in an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the light field pattern according to an embodiment of the present invention;

[0063] Figure 6This is a block diagram of a naked-eye 3D rendering device based on a light field distribution model according to an embodiment of the present invention;

[0064] Figure 7 This is a block diagram of a naked-eye 3D display device according to an embodiment of the present invention. Detailed Implementation

[0065] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0066] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0067] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0070] One specific embodiment of the present invention discloses a naked-eye 3D rendering method based on light field distribution. For example... Figure 1 As shown, the method includes the following steps:

[0071] S1. Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the light field mapping relationship corresponding to the 2D display device parameters and the optical structure.

[0072] S2. Real-time acquisition of human eye coordinates; based on the current human eye coordinates and light field distribution model, obtain the current equivalent lens pitch and equivalent offset of the optical structure; based on the current equivalent lens pitch, obtain the current reference viewing area and viewing angle range; based on the current reference viewing area, obtain the number of virtual cameras corresponding to each eye of each viewer at the viewing position; based on the current reference viewing area and viewing angle range, obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture; and arrange the viewpoints within the current reference viewing area.

[0073] S3. Based on the current equivalent offset, obtain the rendering parameters of each virtual camera or video texture corresponding to each eye of each viewer at the viewing position, and arrange the viewpoints in the current view area; rearrange the viewpoints in the current view area according to the viewpoint arrangement in the current reference view area to obtain the viewpoint map.

[0074] S4. Establish a rendering model, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0075] Specifically, in step S1, the 2D display device parameters include the screen size, aspect ratio, resolution, pixel pitch, etc. The optical structure parameters include the refractive index, fill spacing, lens focal length, viewing angle, spatial resolution, lens pitch, lens thickness, tilt angle, etc.

[0076] It should be noted that the naked-eye 3D display device includes a 2D display device and an optical structure added to the 2D display device. The 2D display device includes a plurality of periodically arranged light-emitting units and pixels capable of displaying red, green, and blue colors. The optical structure is used to provide different viewing angles to the viewer's eyes at the same time, thereby forming a 3D effect. The optical structure includes slit gratings, lenticular gratings, adjustable liquid crystal gratings, spatial light modulators, etc. Preferably, when providing a 3D display effect to multiple viewers simultaneously, i.e., supporting multiple viewing positions simultaneously, a lenticular grating is used as the optical structure to provide a larger field of view.

[0077] Furthermore, obtaining the light field mapping relationship corresponding to the optical structure based on the optical structure parameters includes: obtaining the light field mapping relationship corresponding to the optical structure through one or more methods such as theoretical or numerical calculation, Monte Carlo simulation, neural network learning, and standard light source calibration test;

[0078] The light field mapping relationship includes the light field mapping relationship from the light-emitting unit on the screen coordinate system of the 2D display device through the optical structure to the observation space, or the light field mapping relationship from the observation space through the optical structure to the light-emitting unit on the screen coordinate system of the 2D display device.

[0079] Specifically, in some embodiments, the light field mapping relationship is obtained through theoretical or numerical calculations (including empirical function models) of geometric optics. For example, the periodic viewpoint arrangement relationship after refraction by the lens is obtained by calculating the Fresnel theorem theory of cylindrical lens grating.

[0080] In some embodiments, Monte Carlo simulation tools are used to simulate the absorption, scattering, and refraction effects of each wavelength in the visible light band, obtaining the full-parameter optical field mapping relationship, i.e., the response matrix of each parameter of the optical field. For example, using the Geant4 Monte Carlo simulation tool, the absorption, scattering, and refraction effects of each wavelength in the visible light band are simulated using parallel light sources of each wavelength and optical structures modeled in 3D within Geant4, obtaining the full-parameter optical field mapping relationship.

[0081] In some embodiments, the light field mapping relationship is learned through a neural network. Specifically, this process includes the following main steps:

[0082] Step S11: Use multiple imaging devices to acquire imaging images of different 3D scenes at multiple human eye coordinates, and construct a dataset; the dataset covers a wide range of 3D scenes and viewpoints;

[0083] Step S12: Construct a neural network to output 3D scene rendering parameters corresponding to the input human eye coordinates; the 3D scene rendering parameters are used to control image generation during real-time rendering.

[0084] Step S13: Train the neural network based on the dataset to minimize the loss function and obtain a trained neural network;

[0085] Step S14: Capture the current human eye coordinates, input the current human eye coordinates into the trained neural network, and infer the 3D scene rendering parameters corresponding to the current human eye coordinates.

[0086] It should be noted that neural networks can learn the implicit light field mapping relationship from the light field in the human eye coordinates through the optical structure to the light-emitting unit in the screen coordinate system of the 2D display device.

[0087] In some embodiments, the optical field mapping relationship is obtained by performing response tests in the visible light band using a standard light source. For example, the optical field mapping relationship is obtained by testing the refraction angle, dispersion, and intensity distribution of the optical structure at different angles using a standard light source.

[0088] In some embodiments, one or more of the above methods can be used at different stages to obtain the light field mapping relationship. For example, since the cylindrical lens grating has different thermal deformations at different stages such as design, production, before bonding, and after bonding, the mapping relationship of each parameter can be obtained by theoretical, numerical, and simulation methods at the design stage; and the mapping relationship of each parameter can be calibrated by testing with a standard light source after bonding.

[0089] It is understandable that the design parameters of the optical structure may deviate from the actual parameters due to factors such as manufacturing process. Therefore, it is necessary to use the above method to obtain the light field mapping relationship corresponding to the optical structure, improve the accuracy of the light field mapping relationship, and provide a foundation for improving the quality of naked-eye 3D display in the future.

[0090] Furthermore, the light field distribution model represents the light field distribution generated in the observation space due to the influence of the optical characteristics of the 2D display device and the light field mapping relationship corresponding to the optical structure on the emitted light of the 2D display device.

[0091] It should be noted that, for ease of description, the world coordinate system of the virtual space where the target to be rendered is defined as W. xyz The screen coordinate system represented by the two-dimensional screen coordinates of the light-emitting unit of the 2D display device is S. xy The observation space coordinate system describing the spatial location of the human eye and the naked-eye 3D display device is Ob. xyz The human visual coordinate system is V xyz Among them, the screen coordinate system S xy It conforms to the definition of the screen coordinate system in general rasterization, graphics processing systems, and graphics pipelines. The observation space coordinate system Ob... xyz With the geometric center of the light-emitting unit of the 2D display device as the origin, the positive direction of the light-emitting unit passing through this origin is the positive z-axis direction of the right-hand coordinate system. The human eye visual coordinate system V... xyz A right-handed coordinate system is established with the visual center of each eye as the origin and the positive z-axis pointing in the direction of the eye center. Define T. Ob-V Indicates from Ob xyz coordinate system to V xyz The transformation matrix of the coordinate system, T V-Ob Indicates from V xyz coordinate system to Ob xyz The transformation matrix of a coordinate system can be used to derive the transformation matrix representation between other coordinate systems.

[0092] like Figure 2As shown, in the screen coordinate system, the light-emitting unit S(x, y) is emitted from the surface of the optical structure at an angle α(θ, φ) in the observation space coordinate system Ob(x', y', z'). Each light-emitting point Ob0(x', y', z'|α), Ob1(x′, y', z'α), Ob2(x', y'z'|α), ... emitted at an angle α(θ, φ) constitutes the equivalent pixel of the view I(α) corresponding to that emission angle α(θ, φ). Six degrees of freedom in space are defined for each viewer at each moment, including the distance from each eye to the screen of the naked-eye 3D display device and the pupil orientation; that is, the human eye coordinates are... Where p represents the p-th viewer; i represents the i-th eye of each viewer; This represents the horizontal position of the i-th eye of the p-th viewer on the screen, i.e., the coordinates in the left and right directions; This represents the vertical position of the i-th eye of the p-th viewer from the screen, i.e., the coordinates in the up and down direction; Represents the vertical distance from the i-th eye of the p-th viewer to the screen, i.e., the coordinates in the near and far directions; This represents the yaw angle, which is the horizontal rotation angle around the vertical axis (z′ axis), corresponding to the viewer's eye turning to the left or right; It represents the pitch angle, which is the vertical rotation angle around the horizontal axis (y′ axis), corresponding to the viewer's head-up or head-down movement. The 't' represents the roll angle, which is the axial rotation around the front-to-back axis (x' axis), corresponding to the viewer's head tilting to the side; 't' represents the change of the coordinate over time.

[0093] To better illustrate the content of this invention, the optical field and the optical field mapping relationship after passing through the optical structure are described. The following description is not the only one available in this application. The general form of the optical field can be expressed as:

[0094]

[0095] Where E(r,t) represents the general plane wave representation of the light field, r=(x,y,z) represents the position vector, and A(r,t) represents the amplitude of the light field. ω represents the unit vector characterizing the polarization direction, k represents the wave vector, and φ represents the initial phase.

[0096] Assuming the initial light field does not pass through the optical structure (i.e., the outgoing light is only affected by the optical properties of the 2D display device), it can be expressed as:

[0097]

[0098] The light field changes after passing through the optical structure, as shown below:

[0099]

[0100] Where Ω represents a spatial angle, and dΩ represents the integral over the spatial angle.

[0101] The initial light field described above can be expressed in matrix form as follows:

[0102]

[0103] The light field after passing through the optical structure can be represented in matrix form as follows:

[0104]

[0105] The mapping relationship between the initial light field and the light field after passing through the optical structure can be expressed in matrix form as follows:

[0106] V(r,Ω)=M(r,Ω)V0(r)

[0107] Where M(r,Ω) represents the optical field mapping relationship corresponding to the optical structure.

[0108]

[0109] The light field after passing through the optical structure is summed in all directions in space, i.e., integrated, to obtain the input light field in the observation space, i.e., the light field distribution model, which is expressed as:

[0110]

[0111] Its discrete form is expressed as:

[0112]

[0113] Where, N θ N represents the number of discrete points representing the polar angle θ. φ Δθ represents the number of discrete points representing the azimuth angle φ, Δθ represents the step size of the polar angle, and Δφ represents the step size of the azimuth angle. Unless otherwise stated, I(α) represents the view that should be seen at the observation position α, and β represents the virtual camera in the human visual coordinate system V. xyz The position below.

[0114] It should be noted that the above method can represent both the light field formed by light rays from the emitting unit to the viewing space and the light field formed by tracing rays from the viewer's pupil to the emitting unit. The matrices involved are related by convolution and deconvolution. The mapping relationship from the emitting unit in the screen coordinate system to the light field after passing through the optical structure is denoted as M. S-Ob (r, Ω), the mapping relationship of the light field from the light field in the observation space through the optical structure to the light-emitting unit in the screen coordinate system is denoted as M. Ob-S(r, Ω). Since some optical structures can actively adjust their corresponding optical field mapping relationship with time t, therefore, M S-Ob (r,Ω) can also be denoted as M S-Ob (r,Ω,t), M Ob-s (r,Ω) can also be denoted as M Ob-S (r,Ω,t).

[0115] Specifically, in step S2, the real-time acquisition of human eye coordinates includes: using existing pupil tracking algorithms (such as pupil distance tracking AI algorithms in visual solutions, infrared pupil recognition algorithms, etc.) to acquire the human eye plane coordinates and pupil orientation in real time, and using one of the following methods—ToF, structured light, multi-camera cross-positioning, etc.—to acquire depth values, thereby obtaining the human eye coordinates, i.e., the human eye six-degree-of-freedom coordinates.

[0116] Optionally, an empirical curve of visual characteristics can be determined using a pupil-tracking device. Since the empirical curve of human eye movement differs at different device sizes and viewing distances, the pre-determined empirical curve of visual characteristics will reduce the computational load during real-time pupil-tracking rendering. Under special conditions such as low light, when the uncertainty of real-time pupil-tracking position is large and the confidence level is low, the empirical curve of visual characteristics can be used instead of real-time pupil-tracking to obtain the human eye coordinates.

[0117] It should be noted that human eye coordinates represent a specific viewing position of the human eye in the observation space, looking at the screen from a certain viewing angle.

[0118] Furthermore, the step of obtaining the current equivalent lens pitch and equivalent offset of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current reference viewing area and viewing angle range based on the current equivalent lens pitch includes:

[0119] The current viewing distance is obtained based on the current human eye coordinates. The current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model. The current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system. The current reference viewing area and viewing angle range are obtained based on the current equivalent lens pitch.

[0120] Specifically, the distance from the human eye to the display screen is obtained based on the current human eye coordinates, i.e., the viewing distance.

[0121] Specifically, the current equivalent lens pitch refers to the smallest detail that the optical structure can resolve at the current viewing distance. Taking a cylindrical lens grating as an example, the current equivalent lens pitch is obtained using the following formula:

[0122] A = A0 + ΔA,

[0123]

[0124] Where A represents the current equivalent lens pitch, which is also the current reference visible area size; ΔA represents the current equivalent lens pitch increment, which is also the current reference visible area size increment; A0 represents the original lens pitch in units of sub-pixels, which is also the original visible area period (i.e., the original reference visible area); H0 represents the optimal viewing distance preset according to the optical structure, and H represents the current viewing distance. The value represents the viewing angle within the original viewing zone period, and W represents the average width of the lens covering the 2D display screen (unit: millimeters).

[0125] It should be noted that the equivalent lens pitch or related equivalent parameters can be obtained by analogy for other optical structures.

[0126] The current equivalent offset refers to the horizontal offset of the visible area and the viewing angle caused by the horizontal offset of the viewer's eye at the current viewing distance.

[0127] Specifically, assuming the tracking camera is installed above the horizontal center of the light field screen (i.e., the display screen) (the camera's optical axis installation error has been corrected, and the horizontal and vertical optical axes of the camera are parallel to the horizontal and vertical directions of the display screen, respectively), the current equivalent offset can be obtained using the following formula:

[0128]

[0129]

[0130] Where ΔX is the current equivalent offset, x cam y cam These are the horizontal and vertical coordinates (normalized values ​​ranging from 0 to 1) of the current human eye within the frame of the tracking camera, fov. cam cam_ratio is the horizontal angle of the image sensor of the tracking camera, and cam_ratio is the aspect ratio of the image sensor of the tracking camera. It is the screen coordinate system S xy The horizontal coordinate (normalized value range 0-1), W OC It is the width of the display screen (unit: mm), α len It refers to the design tilt angle of the cylindrical lens on the 2D display screen.

[0131] Specifically, the current reference viewing area refers to the area on the display screen where, at the current viewing distance, the center of the viewer's eye is directly facing the display screen, and the viewer sees the best 3D effect in the observation space; that is, the coordinate range of a sub-pixel in the screen coordinate system. It should be noted that the equivalent lens pitch can also be obtained for other optical structures based on their light field distribution characteristics.

[0132] Specifically, the current viewing angle range refers to the viewing angle of a viewer at the current viewing distance within the original viewing area period size. The current visible angle range when the human eye sees the best 3D effect is determined by the equivalent lens pitch (i.e., the size of the viewing area period); the total visible angle range due to the equivalent lens offset and sliding is the maximum angular range that this solution can provide for the 3D effect.

[0133] It should be noted that, through the light field mapping relationship, the mapping between the screen coordinate system and the observation space coordinate system can be realized, that is, the mapping between the current reference view area and the current viewable range can be realized.

[0134] Furthermore, the step of obtaining the number of virtual cameras corresponding to each viewer's eye at the viewing position based on the current reference viewing area includes:

[0135] Based on the current reference view area, the number of viewpoints corresponding to each eye of each viewer at the viewing position is obtained. One viewpoint corresponds to one virtual camera, thus obtaining the number of virtual cameras.

[0136] It should be noted that, taking a cylindrical lens grating as the optical structure as an example, a linear relationship is used to represent the refraction of light by the cylindrical lens, ignoring aberrations, off-axis nonlinearity, chromatic aberration, and brightness variations. Each pixel on the display screen is composed of multiple sub-pixels. Along the period of the cylindrical lens, each sub-pixel refracts light in a different direction on the focal plane, passing through the optical structure to reach the observation space. Assuming that each sub-pixel is a point light source, and that the optical structure only changes the direction of the emitted light source on the focal plane, the direction of refraction of each sub-pixel can be periodically obtained based on the relative positions of the sub-pixels and the optical structure on the display screen. Within the current reference viewing area, all sub-pixels in a certain direction together constitute the viewpoint in that direction within the reference viewing area; therefore, the viewpoint direction represents the viewer's perspective. Using the viewpoint in the direction towards the center of the human eye as the center, a viewpoint is added horizontally every 0.8° to 1.6° to obtain the number of viewpoints within the current reference viewing area.

[0137] Furthermore, the rendering parameters of each virtual camera include virtual camera resolution, virtual camera intrinsic and extrinsic parameters, etc.; the rendering parameters of the video texture include video texture resolution, etc.

[0138] It should be noted that the resolution of a virtual camera is usually obtained based on the beam splitting characteristics of the optical structure, such as the resolution of a cylindrical lens grating, which is obtained based on the resolution of the display screen; the intrinsic and extrinsic parameters of a virtual camera are usually calculated from the viewpoint direction and the optimal viewing distance, that is, the rendering parameters of the virtual camera and the rendering parameters of the video texture can be obtained through existing technology.

[0139] Furthermore, the arrangement of viewpoints within the current reference viewing area includes:

[0140] Based on the rendering parameters of each virtual camera or the rendering parameters of the video texture, rasterization is performed to generate corresponding subviews, where each subview corresponds to a viewpoint within the current reference view area, i.e., a viewpoint of the human eye.

[0141] All subviews corresponding to the current reference viewport are organized together according to a storage format, which includes a grid diagram, multi-view differential encoding (such as MV-HEVC encoding), and a texture array; each subview in the reference viewpoint diagram has a corresponding number; similarly, the viewpoints in the reference viewport corresponding to each eye of each viewer are arranged.

[0142] For example, such as Figure 3 As shown, a 7×7 arrangement of subviews is used for viewpoint layout. From the top left to the bottom right, the subviews have a horizontal parallax of -24° to 24° with respect to the main viewpoint. The difference between two adjacent subviews is one degree. The subviews are numbered 0, 1, ... 48 from the top left to the bottom right. Each subview has the same resolution.

[0143] Specifically, in step S3, obtaining the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position based on the current equivalent offset, and arranging the viewpoints within the current visible area includes:

[0144] The current reference view area is obtained by sliding it according to the current equivalent offset. The current viewing angle is obtained by sliding it according to the current equivalent offset. The current main viewing angle is obtained by aligning the center position of the current human eye with the center point of the current view area.

[0145] Based on the current main viewpoint and the view area and view angle, the rendering parameters of each virtual camera or the rendering parameters of the video texture are adjusted to obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each viewer's eye at the viewing position.

[0146] Based on the rendering parameters of each virtual camera or the rendering parameters of the video texture, rasterization is performed to generate a sub-view of the target to be rendered;

[0147] All subviews corresponding to the current viewport are organized together according to a storage format, which includes a grid diagram, multi-view difference encoding, and texture array; wherein each subview corresponds to a viewpoint within the current viewport, and each subview has a corresponding number.

[0148] It should be noted that due to the physical characteristics of the optical structure, the visible area is periodically arranged. When the viewing distance remains constant, the reference visible area remains unchanged. When the human eye moves laterally to the left relative to the display screen, the visible area and viewing angle shift laterally to the right. Therefore, the corresponding visible area can be obtained based on the equivalent offset and the current reference visible area, and the corresponding viewing angle can be obtained based on the equivalent offset and the current viewing angle range. Several sub-views, i.e., sub-views with different viewing angles, are obtained from a viewing position, deviating from the main viewing angle. According to real-time pupil tracking, when the viewing distance changes, the reference visible area changes, and the lens period in the screen coordinate system corresponding to the visible area changes. For example, the screen coordinate period facing the lens numbered m changes to the screen coordinate period facing the lenses numbered m-2, m-1, m+1, m+2, etc.

[0149] Understandably, since the number of viewpoints in the viewport is set based on the viewing distance, if the viewing distance changes without changing the number of viewpoints in the viewport, it may lead to scene tearing in severe cases. For example... Figure 4 As shown, this invention obtains the current viewing distance based on real-time pupil tracking, calculates the equivalent lens pitch based on the current viewing distance, and then obtains the number of viewpoints in the current visible area and the number of subviews in the corresponding viewpoint diagram. When the viewer is closer to the screen relative to the preset optimal viewing distance, the equivalent lens pitch increases, requiring more subviews; when the viewer is farther from the screen relative to the preset optimal viewing distance, the equivalent lens pitch decreases, requiring fewer subviews.

[0150] Further, the step of rearranging the viewpoints within the current view area according to the viewpoint arrangement within the current reference view area to obtain the viewpoint map includes:

[0151] Obtain the overlapping and non-overlapping areas of the current reference view area and the current view area. The subview numbers corresponding to the overlapping areas remain unchanged. Map the subview numbers corresponding to the non-overlapping areas of the current view area to the non-overlapping areas of the current reference view area. Rearrange the viewpoints in the current view area to obtain the viewpoint map.

[0152] It should be noted that if the viewing distance of the human eye coordinates has not changed between the current moment and the previous moment, then the reference visual area is the visual area of ​​the previous moment. The viewpoints in the current visual area are rearranged according to the viewpoint arrangement in the visual area of ​​the previous moment to obtain the viewpoint map.

[0153] Specifically, such as Figure 4 As shown, assuming the viewing distance of the human eye coordinates remains unchanged between the current and previous moments, the subview number corresponding to the viewpoint arrangement within the visible area at the previous moment is α. -1 ~α1, the subview number corresponding to the viewpoint arrangement within the current visible area is α. -2~α2, then the subview number corresponding to the overlapping area between the previous visible area and the current visible area is obtained as α. -2 ~α1, and the subviews corresponding to the non-overlapping areas are numbered α1~α2, α -2 ~α1 remains unchanged, and α1~α2 are mapped to α -1 ~α -2 The positions are used to rearrange the viewpoint diagram to obtain α1~α2, α -2 ~α1.

[0154] For example, suppose the viewpoint map has 21 subviews. The subview numbers corresponding to the visible area at the previous moment are -10 to 10, and the subview numbers corresponding to the visible area at the current moment are -5 to 15. Then -5 to 10 remain unchanged, and 11 to 15 are mapped to the positions of -10 to -6. The viewpoint map is rearranged to get 11 to 15 and -5 to 10.

[0155] It should be noted that, for good display results, the center point of the visible area should be aligned with the center of the human eye. When the viewing angle obtained by pupil tracking exceeds the periodic area of ​​the bottom screen corresponding to the front lens, light will pass through adjacent lenses to reach the bottom screen. Assuming that the original equivalent pixel in the front visible area is generated from a position on the m-th cylindrical lens, when it exceeds the front visible area, the equivalent pixel may be generated from a position on the (m-1), m-2, or m+1, m+2-th cylindrical lenses, depending on the mapping relationship of the optical structure. According to the light field mapping relationship, the sub-view has a mapping relationship with the pixels on the 2D screen, and the offset of the visible area is essentially the offset of its corresponding pixel on the 2D screen. With a high pupil tracking frequency, each offset is small, and the overlapping area is large, so most sub-views remain unchanged; for non-overlapping areas, it essentially involves transferring pixels from one grating unit to another. Since the grating unit itself is approximately periodic, the pixels corresponding to the offset visible area are equivalent to the pixels corresponding to the original visible area.

[0156] Understandably, this invention acquires six-DOF human eye coordinates through real-time pupil tracking and rearranges the viewpoint map of the target to be rendered based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It improves the interlacing results by considering the attributes of each eye of each observer in the light field at any given moment, as well as the full parameter mapping relationship through the optical structure. It adjusts the equivalent offset of corresponding sub-pixels on the screen, ensuring that the visible area always aligns with the center of the human eye coordinates, even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience.

[0157] Understandably, this invention obtains the viewing distance based on the eye coordinates acquired through real-time pupil tracking, calculates the equivalent lens pitch based on the viewing distance, and recalculates the reference viewing area and the number and distribution of viewpoints within the viewing area. Because the changes in the equivalent lens pitch and equivalent lens offset caused by distance and the x and y positions of sub-pixels in the screen coordinate system rearrange the viewpoints within the current viewpoint map, for example, when the viewing distance is smaller or larger than the standard distance, the equivalent lens pitch increases or decreases, the number of viewpoints within the viewing area increases or decreases, and the number of viewpoints participating in the interlacing increases or decreases accordingly. This results in the rearrangement of more or fewer sub-views in the viewpoint map, precisely controlling the number of sub-views required for each viewer's eyes, thereby reducing resource consumption and avoiding problems such as 3D imaging tearing.

[0158] Specifically, in step S4, for a viewing position, the expression for rendering the model is:

[0159]

[0160] Among them, res S represents the resolution of the interlaced light field map, and t represents time. This indicates that the subviews numbered i = 0, 1, 2, ..., k are interleaved; I(α, t) represents the viewpoint map corresponding to the main viewpoint α at time t, which has k+1 subviews numbered i = 0, 1, 2, ..., k.

[0161] It should be noted that texture sampling, i.e., interleaving, is performed on the viewpoint map to obtain the light field map. At this time, sub-views deviating from the main viewpoint α are interleaved with the same weight and resolution. Similarly, the expression for the rendering model at each viewing position can be obtained, and then the viewpoint map corresponding to each viewing position can be interleaved to obtain the light field map.

[0162] For example, for Figure 3 The viewpoint map shown is interleaved, that is, 49 sub-views are sampled to obtain the interleaved light field map as shown. Figure 5 As shown.

[0163] Furthermore, the light field map is projected onto a display screen to achieve naked-eye 3D display.

[0164] It should be noted that in rasterization rendering, light rays from different viewpoints are refracted and focused onto the screen coordinates of the 2D display panel through the paraxial region of the cylindrical lens grating. The intrinsic and extrinsic parameters of the multi-view virtual camera in rendering are determined by the mapping relationship of the light ray directions within the period of the cylindrical lens array. When using video texture rendering, video data from multiple perspectives has already been fused into a single texture file offline, allowing for direct interleaving to obtain the light field map.

[0165] Furthermore, the present invention can either render directly using a parallax map, that is, provide a 3D effect directly based on the parallax between the left and right views seen by the human eye; or render by interweaving viewpoint maps, that is, provide a 3D effect based on the parallax generated in the virtual space by a virtual camera when the viewer moves.

[0166] Specifically, this invention can alter the correspondence between sub-views in the visible area after viewpoint shift, and it can also change the intrinsic and extrinsic parameters of the virtual camera corresponding to each sub-view. It simultaneously considers the viewpoint arrangement corresponding to the viewpoint map and the virtual spatial position corresponding to each viewpoint provided to the user. In extreme cases, even only one viewpoint can be provided to the user (similar to 2D binoculars), but by changing the intrinsic and extrinsic parameters of the virtual camera following the movement of the human eye, the user sees different occlusion relationships as they move. That is, the 3D effect is directly provided through the parallax of the left and right views seen by the human eye. Therefore, the method and system provided by this invention support both binocular naked-eye 3D rendering and naked-eye 3D rendering from multiple simultaneous viewing positions.

[0167] In another specific embodiment of the present invention, the method further includes:

[0168] Step S11: Establish a human eye characteristic model corresponding to the rendering parameters of the virtual camera or the rendering parameters of the video texture based on the human eye characteristics. Perform Fourier transform or Gaussian convolution on the human eye characteristic model. Based on the current human eye coordinates, obtain the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture at each viewing position for each eye of each viewer.

[0169] Step S12: Update the rendering parameters of each virtual camera or video texture corresponding to each eye of each viewer at the viewing position in step S3 according to the weights or convolution kernels, and arrange the viewpoints in the current view area; rearrange the viewpoints in the current view area according to the arrangement of viewpoints in the current reference view area to obtain a viewpoint map, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0170] Specifically, in step S11, the human eye characteristics include the influence of the retina and cone cell distribution on the received light field. The human eye characteristic model includes a color difference model, a visual acuity model, etc.; wherein the color difference model represents the color difference within a certain range of the pupil's main visual angle (e.g., within 2° to 10°), and the visual acuity model represents the relationship between human eye resolution and visual angle.

[0171] It should be noted that a corresponding human eye characteristic model can be established for each rendering parameter of the virtual camera or each rendering parameter of the video texture, and expressed as a function curve of human eye characteristics and the rendering parameter, so as to reflect the relationship between human eye characteristics and the rendering parameter.

[0172] Furthermore, performing a Fourier transform or Gaussian convolution on the human eye characteristic model includes:

[0173] For each viewer, Fourier transform or Gaussian convolution is performed on the human eye characteristic model to establish the weights or convolution kernels of the rendering parameters of each virtual camera or video texture in the visual coordinate system of each eye with the six degrees of freedom coordinates of the viewer as the origin.

[0174] It should be noted that the weights or convolution kernels are vectors or vector functions representing the spatial distribution of the human eye characteristic model in the visual coordinate system. By performing a Fourier transform or Gaussian convolution decomposition on the human eye characteristic model corresponding to a rendering parameter, the coefficients of each component of that rendering parameter are obtained and used as the weights or convolution kernels for that rendering parameter of each virtual camera or video texture. The coefficients of the largest component are used as the weights or convolution kernels for the primary viewpoint. By performing a Fourier transform or Gaussian convolution decomposition on the human eye characteristic model corresponding to each rendering parameter of each virtual camera or video texture, the weights or convolution kernels for each rendering parameter of each virtual camera or video texture are obtained, forming a weight matrix or convolution kernel matrix.

[0175] Understandably, as needed, only the human eye characteristic model corresponding to certain rendering parameters can be subjected to Fourier transform or Gaussian convolution to obtain the corresponding weights or convolution kernels, while the weights of the remaining rendering parameters remain at 1.

[0176] For example, in some embodiments, a visual coordinate system V is established based on the color difference model. xyzThe chromatic aberration transformation (i.e., wavelength) weights or convolution kernels distributed in the lower spatial region are used to establish a visual coordinate system V based on the aforementioned visual acuity model. xyz Resolution weights or convolution kernels with spatial distribution.

[0177] Furthermore, the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture for each viewer at the viewing position, based on the current human eye coordinates, include:

[0178] The visual coordinate system is transformed to the observation space coordinate system based on the current human eye coordinates. This transforms the weights or convolution kernels of the rendering parameters of each virtual camera or video texture in the visual coordinate system into the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or video texture at each viewer's eye position.

[0179] It should be noted that the weights or convolution kernels corresponding to the human eye characteristic model represent the spatial distribution in the non-light field rendering case. Therefore, it is necessary to transform them into the observation space domain after optical structure mapping. Assuming that in the non-light field rendering case without optical structures, the weights or convolution kernels are denoted as Ψ... W′ / F′ / V′ (r W′ / F′ / V′ Ω W′ / F′ / V′ W' / F′ / V′ represents the spatial domain function in the virtual world coordinate system, screen coordinate system, and observation space coordinate system without optical structure. Since the mapping properties of the observation space do not change with or without optical structure, V' is V. Therefore, after optical structure mapping, the mapping relationship from v′ to W′ / F′ and from v to W / F is determined by the light field mapping relationship, denoted as . Ψ W′ / F′ / V′ To Ψ W / F / V The mapping relationship is denoted as The weights or convolution kernels in the observation space coordinate system after optical structure mapping are denoted as:

[0180]

[0181] For example, in some embodiments, a Gaussian convolution kernel is employed. Taking the aforementioned visual acuity model as an example, a quantitative relationship between the Gaussian convolution kernel and spatial frequency is established through the visual acuity curve in the visual acuity model, and a quantitative relationship between the blurred image obtained using the Gaussian filter and the multi-resolution texture mapping (Mipmap) level is established through the root mean square error (RMSE) of pixels. The above quantitative relationship is expressed as follows: Wherein, the Gaussian kernel σ is the blurred image obtained by applying a Gaussian filter at the original center viewpoint resolution, and f c Represents spatial frequency, e(Ω) W' / F' / V' ) represents the retinal eccentricity, d(r) W' / F' / V' -r 0W' / F' / V'() represents the coordinates r in the virtual world coordinate system, screen coordinate system, and view space coordinate system under non-light field rendering without optical structures. W′ / F′ / V′ With coordinate r 0W′ / F′ / V′ The Euclidean distance between them. In some embodiments, empirically, σ = 0.4e 0.705m Furthermore, based on the optical structure mapping, the convolution kernel of the visual acuity model in the observation space coordinate system is obtained, expressed as: in, This represents the light field mapping relationship. Based on this, the Mipmap levels of different positional views relative to the human eye's primary viewpoint can be obtained.

[0182] It should be noted that a mipmap consists of a series of progressively smaller texture images, used to improve the efficiency and quality of texture mapping. Each texture image is called a mipmap level.

[0183] Specifically, in step S12, updating the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position in step S3 according to the weights or convolution kernels, and arranging the viewpoints within the current visible area, includes:

[0184] The weights or convolution kernels are used to update the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each eye of each viewer at the viewing position in step S3, thereby updating the rendering parameters of each virtual camera or the rendering parameters of the video texture.

[0185] Rasterization is performed to generate subviews of the target to be rendered based on the updated rendering parameters of each virtual camera or the rendering parameters of the video texture.

[0186] All subviews corresponding to the current viewport are organized together according to a storage format, which includes a grid diagram, multi-view difference encoding, and texture array; wherein each subview corresponds to a viewpoint within the current viewport, and each subview has a corresponding number.

[0187] Further, according to the method in step S3, the viewpoints in the current view area are rearranged according to the viewpoint arrangement in the current reference view area to obtain a viewpoint map; according to the method in step S4, the viewpoint map is interwoven based on the rendering model to obtain a light field map, and the light field map is projected onto the display screen to achieve naked-eye 3D rendering.

[0188] Specifically, at this point, for a viewing position, the expression for rendering the model is:

[0189]

[0190] in, Let represent the i-th sub-view in the viewpoint diagram corresponding to the main perspective α at time t. λ represents the resolution of the i-th subview; ij γ represents the weight of each rendering parameter of the i-th subview, where j = 0, ..., n, and n represents the number of rendering parameters for the virtual camera or the video texture; i This represents the weight of the i-th subview.

[0191] Specifically, the above λ ij This can be viewed as the probability density distribution of each viewpoint subview under the condition that the pupil tracking position is relatively accurate. Due to the influence of pupil tracking frame rate and pupil tracking spatial position uncertainty, it can also be combined with unscented Kalman filtering of pupil tracking position, pupil tracking frame prediction value, and multi-camera cross-pupil tracking positioning on the weight λ. ij The correction is made to represent the joint probability density distribution of the light field that each viewer's eye should see in the temporal and spatial domains. The position and velocity of the human eye movement can be predicted based on the pupil-tracking frame rate and the human eye coordinates, obtaining their respective probability distributions. The joint density distribution of the predicted pupil-tracking position and the rendering model at that position is then used to process I(α). During human eye movement, methods such as Kalman filtering with a control matrix, extended Kalman filtering, particle filtering, and unscented Kalman filtering are used to estimate the position and velocity states. In particular, considering that human eye movement often tends to be a nonlinear system, the unscented Kalman filtering method has certain advantages in nonlinear estimation and processing speed.

[0192] It should be noted that if the weight of a rendering parameter is not updated, its weight is 1. In the formula above, Quilt i The orthogonality between them is not further constrained here, and the orthogonality decomposition algorithm derived therefrom is also the same element described in this invention.

[0193] Understandably, this invention, based on the spatial resolution of the optical structure and the temporal resolution of the pupil-tracking system, can achieve 3D effects by adjusting the weights of the rendering parameters of the virtual camera used by each viewer's eye in each viewpoint (e.g., reducing the resolution, virtual camera intrinsic and extrinsic parameters, contrast, brightness, etc. of subviews corresponding to non-primary viewpoints), and by providing weights to subviews in the viewpoint map (e.g., reducing the weights of subviews corresponding to non-primary viewpoints). This fully utilizes the characteristics of each part of the naked-eye 3D system, including the optical structure, to achieve 3D effects, reducing crosstalk between views and enhancing the realism of 3D visual effects. While ensuring the naked-eye 3D display effect, it significantly reduces the resource consumption of the viewpoint map in terms of storage, transmission, and rendering, greatly saving bandwidth, and providing a foundation for the design of the optical structure and computing unit of naked-eye 3D display devices. This invention predicts the human eye movement trajectory through real-time pupil tracking combined with unscented Kalman filtering. Under special conditions such as low light, it uses visual characteristic empirical curves instead of real-time pupil tracking to obtain the human eye coordinates, making the human eye coordinates closer to the true value and reducing problems such as dizziness and ghosting.

[0194] For example, suppose that filtering and discretizing the human eye characteristic model yields the optimal discretization decomposition (optimal approximation) of the viewpoint image I(α) at ​​observation position α as follows: When the true value of the pupil-tracking prediction is α, I(α) is the true value of the signal, and I(α'|X(α')) caused by the probability distribution of the pupil-tracking position constitutes the spectral distribution curve, and X(α') is the probability density function of the pupil-tracking position.

[0195] In some embodiments, only the primary viewpoint is considered as the desired signal. The primary viewpoint within the pupil-tracking reliability threshold (the true position of the primary viewpoint is evaluated by the threshold and the probability distribution of human eye localization) is considered the desired signal, while the eccentric viewpoint is considered a noise signal. In some embodiments, using... The visual acuity curve model CT(f,e) is decomposed into frequency domain weights, λ ij It is by The weights generated by probability density function decomposition, the expected signal is defined as follows: Noise signal is defined as

[0196] In some embodiments, the weights obtained by the above weight calculation method, together with the configuration parameters, determine the rendering threshold for mitigating crosstalk, or the dynamic rendering threshold is determined based on the user's computing power or configuration parameters. In multi-viewer application scenarios, the eye coordinates of each viewer... The main view to be rendered is determined, and at least one main view for each eye is required to form a 3D effect. In traditional naked-eye 3D rendering, all multi-viewpoint maps participate in the interleaving process, resulting in crosstalk between different viewpoints and requiring high rendering computing power. This invention, through the aforementioned light field mapping relationship and the calculation of weights or convolution kernels, can better determine the multi-camera spacing and the off-center viewpoint threshold for disabling rendering when reducing crosstalk or lowering computing power requirements. For example, the threshold for the rendering configuration is set to CT for the forward acuity curve CT(f,e). cut The resolution threshold for each viewpoint is set by the computing power or configuration of the user device. Alternatively, set the total viewpoint resolution threshold to res. Q Therefore, based on the aforementioned filtering process using visual acuity curves, the total number of eyes and the view of each eye can be used as the basis for determining the total number of eyes and the view of each eye. The filtering threshold is determined by the levels and thresholds calculated based on visual acuity and other visual curves. These are used to decide which viewpoint to disable rendering. In extreme cases, only the primary viewpoint, where the human eye's coordinates are estimated using methods such as unscented Kalman filtering, is rendered.

[0197] In some embodiments, the joint probability density amplitude obtained from the aforementioned visual models such as visual acuity and pupil tracking uncertainty needs to be adjusted by a threshold to ensure reliable view selection under external configuration parameters such as view redundancy, rendering efficiency, computing power, and frame rate. When the joint probability density of non-primary view positions is increased, more sub-views are needed to ensure 3D effects; while when the joint probability density of non-primary view positions is decreased, more non-primary view views can be discarded.

[0198] In multi-view scenarios, the crosstalk metric needs to be determined based on the number of address conflicts at the sub-pixel mapping positions of each view. If the crosstalk limit is exceeded, it's necessary to decide whether to reduce the number of rendering images required for each viewer's glasses based on the configuration, until the crosstalk exceeds a certain threshold, at which point only a single eye is rendered, and each viewer sees a 2D effect. By employing the joint probability density amplitude and probability modulation model using visual models such as visual acuity and pupil tracking models, a smoother and wider crosstalk tolerance transition can be achieved between 3D multi-view for each viewer's eyes, to binocular vision for each eye, and finally to all viewers seeing a 2D scene.

[0199] Optionally, considering the latency of the pupil-tracking processing system, typically ranging from 20ms to 80ms, to eliminate the lag effect of the probability density cloud distribution caused by this latency, a pupil-tracking prediction algorithm is incorporated. This algorithm is trained using the fixed system latency of the device and eye movement tracking data from the naked-eye 3D viewing scene as training parameters. During rendering, the pupil-tracking prediction algorithm is used to predict the zero-latency eye coordinates corresponding to the pupil-tracking system in advance.

[0200] It should be noted that when the human eye moves vertically, the joint probability density determines the new sub-pixel arrangement in real time. Therefore, by changing the joint probability density in real time, horizontal and vertical screen compatibility can be achieved. The parameters such as viewing area, viewpoint, equivalent lens pitch, and equivalent lens offset in the aforementioned embodiments can be applied to one-dimensional gratings, i.e., gratings that are periodic only in one direction (horizontal) and completely consistent in the other direction (vertical), thus exhibiting parallax only in one direction; or to two-dimensional gratings, i.e., gratings with parallax in both directions. In this case, S1, S2, ... mentioned above represent two-dimensional pixels, and S(x,y) represents a two-dimensional pixel. -1 The range ~S1 becomes the set of sub-pixels corresponding to the normal visible area of ​​a certain unit of the raster. After the visible area is shifted, the range of the overlapping part of the new and old visible areas is the intersection of the sub-pixel sets of the new and old ranges, not the non-intersecting part of the sets corresponding to the overlapping part. For viewpoints α1 to α2, it is α(θ,φ), which represents the solid angle formed by the human eye to a certain point on the screen and the screen.

[0201] In some embodiments, to enhance the first-person perspective effect, a weighted weight is provided for the first-person view in the rendering model using the following formula:

[0202]

[0203] Where β represents the viewpoint corresponding to the non-prime view subview. This means that the subview corresponding to viewpoint β is superimposed on the subview corresponding to the main viewpoint.

[0204] In some embodiments, the view brightness weight at large angles is adjusted based on the principal axis angle during pupil tracking (i.e., the horizontal parallax that typically corresponds to the viewing angle), and the output brightness is increased during the interlacing process to compensate for the brightness decrease caused by the lens characteristics. That is, because the brightness decreases in the non-principal axis direction of the lens, the view brightness is increased when the deviation from the principal axis angle of the lens is large.

[0205] In some embodiments, concave filtering is performed in the spatial domain of the light field based on a human eye characteristic model to reduce off-axis imaging distortion.

[0206] In some embodiments, the viewpoint map is filtered to correct the weights in the rendering model so that the virtual camera array has the maximum gain in a certain direction. For example, a parametric multi-channel Wiener filter, a Wiener post-filter, or an optimal corrected logarithmic spectral amplitude post-filter are used for filtering.

[0207] In some embodiments, depth information is used to perform depth-based Gaussian blur on the target to be rendered, including simulating denser viewpoints through lateral Gaussian blur and reducing the vertical resolution at a distance through a two-dimensional Gaussian convolution kernel, avoiding Gaussian blurring in only one direction to improve the 3D display effect.

[0208] Understandably, reducing viewpoint density can lower rendering or transmission costs, but this might cause a single distant point to appear as several points, severely impacting the 3D display of distant objects. This invention utilizes the depth information from the rendering to apply a depth-based Gaussian blur to the target object, applying a depth-of-field effect to objects far from the focal plane, thereby compensating for the perceived distance of objects and reducing crosstalk.

[0209] In some embodiments, the depth-of-field boundary of the primary rendering target is used as a criterion for reducing the number of virtual cameras during rendering to mitigate crosstalk between subviews and make the primary rendering target clearer with a larger depth-of-field range. For example, in a rendering engine, N virtual cameras are normally used to capture images and render them to N viewpoint maps. To reduce crosstalk, some targets are captured by M of these virtual cameras, where N>M≥2. The advantage of this is that a portion of the image from the raster is of the same parallax, thus reducing crosstalk. However, to match the parallax of the left and right eye positions of the human eye, the pupil tracking requirements become relatively higher.

[0210] Understandably, traditional methods of arranging sub-views for each perspective using multiple virtual camera views lead to problems such as information redundancy, excessive performance consumption, shallow depth of field, and low field of view (FOV) utilization. This invention precisely controls the number of sub-views required for each viewer's eyes and the sub-view resolution, thereby saving computational resources, improving the utilization of hardware and software resources, reducing crosstalk, and expanding the visible area. The method of this invention supports both providing a single viewpoint (i.e., providing a 3D display effect for one viewer moving through space) and providing multiple viewpoints simultaneously (i.e., providing a 3D display effect for multiple viewers moving through space), thus it can be applied to scenarios such as holographic calls, glasses-free 3D video viewing, and glasses-free 3D games.

[0211] Another specific embodiment of the present invention, taking a cylindrical lens grating as an example, discloses a parameter design method for the optical structure of a glasses-free 3D display device applied to the aforementioned glasses-free 3D rendering method based on light field distribution, the method comprising:

[0212] Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees.

[0213] Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees;

[0214] Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating.

[0215] Optionally, for large-size light field screens, due to the increased number of applications where multiple people view the screen simultaneously, an additional redundant angle of α3 degrees is needed to avoid crosstalk, resulting in the design degree of the light field screen grating.

[0216] It should be noted that the light field screen refers to the screen of a naked-eye 3D display device, including the screen of a 2D display device and the optical structure.

[0217] For example, design a 15.6-inch desktop monitor to achieve a 16:9 landscape multi-view stereoscopic display, providing users with a high-quality 3D visual experience without glasses. Within a specified cost, a 4K 2D resolution LCD screen is used, with traditional adjacent RGB pixel arrangement. Based on manufacturing capabilities and experience, the spatial resolution redundancy angle of each subpixel is estimated to be 4°. According to the design goals, the interpupillary distance angle in this application scenario is approximately 7.0°, and the temporal resolution redundancy angle is 5°. Therefore, the monitor needs to have a horizontal...

[0218] Understandably, this invention uses a naked-eye 3D rendering method based on light field distribution to obtain the design parameters of the light field screen for naked-eye 3D display devices. Through dynamic light field mapping, it ensures that the physical limits of viewing angle, clarity, and image quality are reached within a given optical structure, providing a foundation for the design of optical structures for naked-eye 3D display devices. This allows for the design of holographic display devices with larger field of view, higher clarity, greater depth of field, horizontal and vertical screen compatibility, and simultaneous viewing by multiple users at a lower cost, breaking the impossible triangle of clarity, depth of field, and FOV.

[0219] Another specific embodiment of the present invention discloses a naked-eye 3D rendering device based on a light field distribution model. For example... Figure 6 As shown, the device includes:

[0220] Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module;

[0221] The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. It obtains a light field map based on the current human eye coordinates, the human eye characteristic model, and the rendering model and sends it to the naked-eye 3D display device.

[0222] A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

[0223] Specifically, the data acquisition device includes a camera or sensor, an eye-tracking module, and a data acquisition module. The camera or sensor captures facial images in real time and sends them to the data acquisition module. The eye-tracking module acquires eye movement parameters in real time, such as pupil size, fixation point, and fixation direction, and sends them to the data acquisition module. The data acquisition module obtains the eye coordinates (i.e., pupil tracking values) from the facial image and eye movement parameters and sends them to the control module. It also acquires parameters for the glasses-free 3D display device and sends them to the control module. The parameters for the glasses-free 3D display device include 2D display device parameters and optical structure parameters.

[0224] Furthermore, the control module includes a memory and a computing unit. The memory includes ROM and RAM, used to store relevant parameters, images, textures, and other data. The computing unit includes a rendering parameter calculation module and a view processing module.

[0225] Specifically, the rendering parameter calculation module is used to obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the 2D display device parameters and the light field mapping relationship corresponding to the optical structure; obtain the current equivalent lens pitch and equivalent offset of the optical structure according to the current human eye coordinates and the light field distribution model; obtain the current reference viewing area and viewing angle range based on the current equivalent lens pitch; obtain the number of virtual cameras corresponding to each viewer's eye at the viewing position according to the current reference viewing area; obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture according to the current reference viewing area and the viewing angle range, and arrange the viewpoints in the current reference viewing area; obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each viewer's eye at the viewing position according to the current equivalent offset, and arrange the viewpoints in the current viewing area; rearrange the viewpoints in the current viewing area according to the arrangement of viewpoints in the current reference viewing area to obtain a viewpoint map; and send the viewpoint map to the view processing module.

[0226] Specifically, the view processing module is used to establish a rendering model, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0227] Preferably, the computing unit includes a rendering parameter calculation module, a weight calculation module, and a view processing module. The computing unit may include a GPU and a VPU.

[0228] Specifically, the rendering parameter calculation module is used to obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the 2D display device parameters and the light field mapping relationship corresponding to the optical structure; obtain the current equivalent lens pitch and equivalent offset of the optical structure according to the current human eye coordinates and the light field distribution model; obtain the current reference viewing area and viewing angle range based on the current equivalent lens pitch; obtain the number of virtual cameras corresponding to each viewer's eye at the viewing position according to the current reference viewing area; obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture according to the current reference viewing area and the viewing angle range, and arrange the viewpoints in the current reference viewing area; obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each viewer's eye at the viewing position according to the current equivalent offset, and arrange the viewpoints in the current viewing area; rearrange the viewpoints in the current viewing area according to the arrangement of viewpoints in the reference viewing area to obtain a viewpoint map; and send the viewpoint map and the light field distribution model to the weight calculation module.

[0229] Specifically, the weight calculation module is used to establish a human eye characteristic model corresponding to the rendering parameters of the virtual camera or the rendering parameters of the video texture based on the characteristics of the human eye; perform Fourier transform or Gaussian convolution on the human eye characteristic model; obtain the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture at each viewing position for each eye of each viewer based on the current human eye coordinates; update the rendering parameters of each virtual camera or the rendering parameters of the video texture at each viewing position for each eye of each viewer according to the weights or convolution kernels; arrange the viewpoints in the current visible area; rearrange the viewpoints in the current visible area according to the viewpoint arrangement in the reference visible area to obtain a viewpoint map; and send the viewpoint map to the view processing module.

[0230] Specifically, the view processing module is used to establish a rendering model, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

[0231] Understandably, the naked-eye 3D rendering device based on the light field distribution model, which applies the naked-eye 3D rendering method based on the light field distribution, can support both providing naked-eye 3D rendering for only two eyes, i.e., providing a 3D display effect for one viewer when moving in space, and providing multiple viewing positions simultaneously, i.e., providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0232] Another specific embodiment of the present invention discloses a glasses-free 3D display device. For example... Figure 7As shown, the naked-eye 3D display device applies the naked-eye 3D rendering method based on light field distribution, including:

[0233] 2D display devices or light sources used to provide 2D images;

[0234] Optical structures are used to provide viewers with different perspectives for each eye at the same time, thereby enabling naked-eye 3D displays and holographic displays;

[0235] Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

[0236] Specifically, the 2D display device includes a plurality of periodically arranged light-emitting units and pixels capable of displaying red, green, and blue colors. Preferably, the 2D display device includes an LCD panel or an OLED panel.

[0237] Specifically, the optical structure includes a slit grating, a cylindrical lens grating, an adjustable liquid crystal grating, and a spatial light modulator. Preferably, when providing a 3D display effect for multiple viewers simultaneously, i.e., supporting multiple viewing positions at the same time, a cylindrical lens grating is used as the optical structure to provide a larger field of view.

[0238] Specifically, the input and output interfaces include a USB interface, a Mini HDMI interface, and an HDMI interface.

[0239] Understandably, the naked-eye 3D display device that applies the naked-eye 3D rendering method based on light field distribution can support both providing naked-eye 3D rendering for only two eyes, i.e., providing a 3D display effect for one viewer when moving in space, and providing multiple viewing positions simultaneously, i.e., providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0240] Another specific embodiment of the present invention discloses an electronic device for implementing embodiments of the present invention. The electronic device includes:

[0241] Memory, used to store computer programs;

[0242] A processor for executing the computer program to implement the naked-eye 3D rendering method based on light field distribution as described above.

[0243] It should be noted that the electronic device includes various forms of digital computers, such as desktop computers, workbenches, servers, etc., as well as various forms of mobile devices, such as mobile phones, wearable devices, etc. The processor includes a computing unit, an input unit, an output unit, and a communication unit. The computing unit includes various general-purpose and / or dedicated processing components with processing and computing capabilities, such as a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), a controller, a microcontroller, etc., used to execute corresponding processing according to computer programs. The input unit includes a keyboard and a mouse; the output unit includes various types of displays and speakers; the communication unit includes a network card, a modem, and a wireless communication transceiver, enabling the electronic device to exchange information / data with other devices through computer networks and / or telecommunications networks.

[0244] Another specific embodiment of the present invention provides a computer program product, the computer program product including a computer program that can be stored on a readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the above-described naked-eye 3D rendering method based on light field distribution.

[0245] Another specific embodiment of the present invention discloses a computer-readable storage medium for storing a computer program that executes the aforementioned naked-eye 3D rendering method based on light field distribution. It should be noted that the readable storage medium includes volatile memory and non-volatile memory.

[0246] Compared with existing technologies, the beneficial effects of the naked-eye 3D rendering method, apparatus, and naked-eye 3D display device based on light field distribution provided by this invention are as follows:

[0247] 1. This invention acquires six-DOF human eye coordinates through real-time pupil tracking and rearranges the viewpoint map of the target to be rendered based on a human eye characteristic model, naked-eye 3D display device parameters, and a light field distribution model caused by the optical structure. It improves the interlacing results by considering the attributes of each eye of each observer in the light field at any given moment and the full parameter mapping relationship through the optical structure. It adjusts the equivalent offset of corresponding sub-pixels on the screen, ensuring that the visible area always aligns with the center of the human eye coordinates even when the human eye moves. Therefore, when the human eye is outside the normal viewing angle range, it can still see a clear 3D scene without any jumps. The user's viewing area is expanded from a relatively small area to a larger range determined by real-time pupil tracking or the optical performance of the screen and grating. This method significantly improves the field of view, clarity, and resolution of naked-eye 3D devices without increasing costs, and enhances the holographic experience with the same computational load, or significantly reduces the computational system requirements for a near-holographic experience.

[0248] 2. This invention obtains the viewing distance based on the eye coordinates acquired through real-time pupil tracking, calculates the equivalent lens pitch based on the viewing distance, and recalculates the reference viewing area and the number and distribution of viewpoints within the viewing area. For example, when the viewing distance is smaller or larger than the standard distance, the equivalent lens pitch increases or decreases, the number of viewpoints within the viewing area increases or decreases, and the number of viewpoints participating in the interlacing increases or decreases accordingly. This allows for the rearrangement of more or fewer sub-views in the viewpoint map, precisely controlling the number of sub-views required for each viewer's eyes, thereby reducing the consumption of required resources and avoiding problems such as 3D imaging tearing.

[0249] 3. Based on the spatial resolution of the optical structure and the temporal resolution of the pupil-tracking system, this invention adjusts the weight of the rendering parameters of the virtual camera that each viewer's eye should use in each sub-view at each perspective. This reduces the resolution, virtual camera intrinsic and extrinsic parameters, contrast, brightness, etc. of the non-primary view sub-view, thereby reducing crosstalk between views and enhancing the realism of the 3D visual effect. While ensuring the naked-eye 3D display effect, it effectively reduces the resource consumption of viewpoint maps in terms of storage, transmission, and rendering, and significantly saves bandwidth.

[0250] 4. The present invention obtains the design parameters of the light field screen of naked-eye 3D display device based on the naked-eye 3D rendering method of light field distribution. Through dynamic light field mapping relationship, it ensures that the physical limits of viewing angle, clarity and imaging quality are reached on a given optical structure, thus providing a basis for the design of optical structure of naked-eye 3D display device.

[0251] 5. This invention can change the correspondence between subviews in the visible area after the observation position is shifted, and it can also change the intrinsic and extrinsic parameters of the virtual camera corresponding to each subview. At the same time, it considers the interlacing method corresponding to the viewpoint map and the virtual spatial position corresponding to each observation position provided to the user. Therefore, the method and system provided by this invention can support providing only binocular naked-eye 3D rendering, that is, providing a 3D display effect for one viewer when moving in space; and it can also support providing multiple observation positions at the same time, that is, providing a 3D display effect for multiple viewers when moving in space. Therefore, it can be applied to scenarios such as holographic calls, naked-eye 3D video viewing, holographic live streaming, and holographic entertainment.

[0252] 6. This invention predicts the trajectory of human eye movement by real-time pupil tracking combined with unscented Kalman filtering. Under special conditions such as low light, it uses the empirical curve of visual characteristics to replace real-time pupil tracking to obtain the human eye coordinates, making the human eye coordinates closer to the true value and reducing problems such as dizziness and ghosting.

[0253] 7. This invention utilizes the depth information of the rendering to apply a depth-based Gaussian blur to the target to be rendered, applying a depth-of-field effect to objects far from the focal plane, thereby compensating for the appearance of distant objects and reducing crosstalk.

[0254] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0255] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A naked-eye 3D rendering method based on light field distribution, characterized in that, The method includes the following steps: Step S1: Obtain the parameters of the naked-eye 3D display device, including the parameters of the 2D display device and the optical structure parameters; obtain the light field mapping relationship corresponding to the optical structure based on the optical structure parameters; obtain the light field distribution model based on the parameters of the 2D display device and the light field mapping relationship corresponding to the optical structure. Step S2: Obtain the human eye coordinates in real time. Based on the current human eye coordinates and the light field distribution model, obtain the current equivalent lens pitch and equivalent offset of the optical structure. Based on the current equivalent lens pitch, obtain the current reference viewing area and viewing angle range. Based on the current reference viewing area, obtain the number of virtual cameras corresponding to each eye of each viewer at the viewing position. Based on the current reference viewing area and viewing angle range, obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture. Arrange the viewpoints within the current reference viewing area. Step S3: Based on the current equivalent offset, obtain the rendering parameters of each virtual camera or video texture corresponding to each eye of each viewer at the viewing position, and arrange the viewpoints in the current view area; rearrange the viewpoints in the current view area according to the viewpoint arrangement in the current reference view area to obtain the viewpoint map. Step S4: Establish a rendering model, interweave the viewpoint map based on the rendering model to obtain a light field map, and project the light field map onto the display screen to achieve naked-eye 3D rendering.

2. The naked-eye 3D rendering method based on light field distribution according to claim 1, characterized in that, The method further includes: Based on the characteristics of the human eye, a human eye characteristic model is established corresponding to the rendering parameters of the virtual camera or the rendering parameters of the video texture. Fourier transform or Gaussian convolution is performed on the human eye characteristic model. Based on the current human eye coordinates, the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or the rendering parameters of the video texture at each viewing position of each eye of each viewer are obtained. Based on the weights or convolution kernels updated in step S3, the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position are used to arrange the viewpoints in the current view area; the viewpoints in the current view area are rearranged according to the arrangement of viewpoints in the current reference view area to obtain a viewpoint map; the viewpoint map is interwoven with the rendering model to obtain a light field map; and the light field map is projected onto the display screen to achieve naked-eye 3D rendering.

3. The naked-eye 3D rendering method based on light field distribution according to claim 1, characterized in that, The process of obtaining the current equivalent lens pitch and equivalent offset of the optical structure based on the current human eye coordinates and light field distribution model, and obtaining the current reference viewing area and viewing angle range based on the current equivalent lens pitch, includes: The current viewing distance is obtained based on the current human eye coordinates. The current equivalent lens pitch is obtained based on the current viewing distance and the light field distribution model. The current equivalent offset is obtained based on the coordinates of the current human eye in the horizontal and vertical directions of the display screen, the viewing distance, and the coordinates of the sub-pixels in the screen coordinate system. The current reference viewing area and viewing angle range are obtained based on the current equivalent lens pitch. The number of virtual cameras corresponding to each eye of each viewer at the viewing position, obtained based on the current reference view area, includes: Based on the current reference view area, the number of viewpoints corresponding to each eye of each viewer at the viewing position is obtained. One viewpoint corresponds to one virtual camera, thus obtaining the number of virtual cameras.

4. The naked-eye 3D rendering method based on light field distribution according to claim 1, characterized in that, The process of obtaining the rendering parameters of each virtual camera or video texture corresponding to each viewer's eye at the viewing position based on the current equivalent offset, and arranging the viewpoints within the current view area, includes: The current reference view area is obtained by sliding it according to the current equivalent offset. The current viewing angle is obtained by sliding it according to the current equivalent offset. The current main viewing angle is obtained by aligning the center position of the current human eye with the center point of the current view area. Based on the current main viewpoint and the view area and view angle, the rendering parameters of each virtual camera or the rendering parameters of the video texture are adjusted to obtain the rendering parameters of each virtual camera or the rendering parameters of the video texture corresponding to each viewer's eye at the viewing position. Based on the rendering parameters of each virtual camera or the rendering parameters of the video texture, rasterization is performed to generate a sub-view of the target to be rendered; All subviews corresponding to the current viewport are organized together according to a storage format, which includes a grid diagram, multi-view difference encoding, and texture array; wherein each subview corresponds to a viewpoint within the current viewport, and each subview has a corresponding number.

5. The naked-eye 3D rendering method based on light field distribution according to claim 1, characterized in that, The step of rearranging the viewpoints within the current view area according to the viewpoint arrangement within the current reference view area to obtain the viewpoint map includes: Obtain the overlapping and non-overlapping areas of the current reference view area and the current view area. The subview numbers corresponding to the overlapping areas remain unchanged. Map the subview numbers corresponding to the non-overlapping areas of the current view area to the non-overlapping areas of the current reference view area. Rearrange the viewpoints in the current view area to obtain the viewpoint map.

6. The naked-eye 3D rendering method based on light field distribution according to claim 2, characterized in that, The weights or convolution kernels corresponding to the rendering parameters of each virtual camera or video texture at each viewing position, obtained based on the current human eye coordinates, include: The visual coordinate system is transformed to the observation space coordinate system based on the current human eye coordinates. This transforms the weights or convolution kernels of the rendering parameters of each virtual camera or video texture in the visual coordinate system into the weights or convolution kernels corresponding to the rendering parameters of each virtual camera or video texture at each viewer's eye position.

7. The naked-eye 3D rendering method based on light field distribution according to claim 2, characterized in that, For a given viewing position, without enhancing the first-person perspective, the expression for the rendered model is: in, Let represent the i-th sub-view in the viewpoint diagram corresponding to the main perspective α at time t. λ represents the resolution of the i-th subview; ij γ represents the weight of each rendering parameter of the i-th subview, where j = 0, ..., n, and n represents the number of rendering parameters for the virtual camera or the video texture; i This represents the weight of the i-th subview; When enhancing the first-person perspective effect, the expression for rendering the model is: The following formula is used to assign weights to the main viewpoint in the rendering model: Where β represents the viewpoint corresponding to the non-prime view subview. This means that the subview corresponding to viewpoint β is superimposed on the subview corresponding to the main viewpoint.

8. A parameter design method for the optical structure of a glasses-free 3D display device applied to the glasses-free 3D rendering method based on light field distribution as described in any one of claims 1-7, characterized in that, The method includes: Step S21: Determine the corresponding viewing distance based on the size of the light field screen, the resolution of the 2D display device, and the application scenario. Based on the viewing distance and the interpupillary distance of the human eyes, obtain the angle subtended by the human eyes when the viewing distance is directly opposite the center of the light field screen as α0 degrees. Step S22: Based on the technological level and experience, the corresponding spatial resolution subtendant angle is estimated to be α1 degrees; Step S23: Based on the cost of the naked-eye 3D display device, the total latency of the hardware and software solution, the empirical function of human eye movement speed, and the viewing distance, the temporal resolution redundancy angle caused by the pupil tracking delay is α2 degrees, thus obtaining the design degree of the light field screen grating. Spend.

9. A naked-eye 3D rendering device based on light field distribution, characterized in that, The device includes: Data acquisition equipment is used to acquire parameters of the naked-eye 3D display device and real-time human eye coordinates and send them to the control module; The control module is used to establish a light field distribution model, a human eye characteristic model, and a rendering model based on the parameters of the naked-eye 3D display device. It obtains a light field map based on the current human eye coordinates, the human eye characteristic model, and the rendering model and sends it to the naked-eye 3D display device. A glasses-free 3D display device used to achieve glasses-free 3D display and holographic display of light field maps.

10. A glasses-free 3D display device, characterized in that, The naked-eye 3D display device applies a naked-eye 3D rendering method based on light field distribution, including: 2D display devices or light sources used to provide 2D images; Optical structures are used to provide viewers with different perspectives for each eye at the same time, thereby enabling naked-eye 3D displays and holographic displays; Input and output interfaces are used to transmit the data required for glasses-free 3D display and holographic display.

Images