Visual angle tracking grating stereo display system

Abstract

The invention discloses a visual angle tracking grating stereo display system comprising a 3D camera grating display device used for obtaining viewer 3D information; the eyeball position tracking module is used for acquiring real-time spatial position coordinates of the eyes of the audience relative to the 3D camera; the pose calibration module is used for acquiring a spatial relationship between the 3D camera and the display; and the visual angle rendering and grating interlaced graph generation module is used for calculating the spatial position of the human eyes on the display screen and the mapping of each grating strip in a screen pixel coordinate system after acquiring the positions of the eyes and the relative poses of the display screen and the 3D camera, generating a grating stereoscopic display image suitable for the eyes, and sending the grating stereoscopic display image to a display to be output. According to the method and the device, the viewpoint self-adaptive three-dimensional display of the eyes of one or more audiences is realized, the calculation complexity is reduced, the three-dimensional vision consistency can be ensured, the picture which should be seen by the viewing angle can be accurately rendered, and the problem of blurring or 3D picture distortion is avoided.

Description

Technical Field

[0001] The present invention relates to the field of stereoscopic display technology, and in particular to a grating stereoscopic display system for viewing angle tracking. Background Art

[0002] With the continuous advancement of display technology, people's demand for three-dimensional stereoscopic display is growing. At present, common three-dimensional display methods include binocular stereoscopic display (such as common VR and AR devices), grating stereoscopic display, etc. Among them, the grating stereoscopic display screen can present different pictures for different viewing angles, which can better meet the needs of naked-eye 3D display. However, in traditional grating stereoscopic display, dozens or even hundreds of pictures need to be rendered at the same time to provide naked-eye 3D pictures with a sense of stereo at all viewing angles, which places extremely high demands on the system's computational power. Even so, in this display scheme, not only can the best display effect not be obtained at certain viewing angles, but the depth of field is also limited. Summary of the invention

[0003] In order to improve the depth of field of naked-eye 3D stereoscopic display and produce a smoother picture, the present invention proposes a grating stereoscopic display system with viewpoint tracking. By introducing an eyeball position tracking mechanism into the grating stereoscopic display system, the system can dynamically render the display picture according to the real-time position of the user's eyes, thereby accurately controlling the viewpoint position used in rendering, reducing unnecessary viewpoint rendering, and reducing the amount of system calculation, and finally obtaining a stereoscopic display effect with a large depth of field and accurate rendering at the user's actual viewing angle. It includes: a 3D camera grating display device composed of a grating, a display screen, and a 3D camera, an eyeball position tracking module; a posture calibration module; a viewpoint rendering and grating interlaced map generation module; wherein,

[0004] 3D camera grating display device, used to obtain the viewer's 3D information and output stereoscopic images;

[0005] The eyeball position tracking module is used to obtain the real-time spatial position coordinates of the audience's eyes relative to the 3D camera, and transmit the tracked eye coordinate information to the perspective rendering and raster interlaced image generation module;

[0006] A pose calibration module is used to obtain the spatial relationship between the 3D camera and the display, and transmit the result to the perspective rendering and raster interlaced image generation module;

[0007] The perspective rendering and grating interlaced image generation module is connected to the 3D camera grating display device, the posture calibration module and the eye tracking module. After obtaining the eye position, the relative posture of the display screen and the 3D camera, it calculates the spatial position of the human eye on the display screen and the mapping of each grating strip in the screen pixel coordinate system, generates a grating stereo display image suitable for the eye, and sends it to the display for output.

[0008] Furthermore, the range of the grating spacing d is 0.1 mm to 1.0 mm, the range of the initial offset is 0 to d; and the range of the vertical distance h from the grating to the display screen is 0.5 mm to 2.0 mm.

[0009] Furthermore, the perspective rendering and grating interlaced image generation module includes:

[0010] S1, receiving the position of the human eye relative to the 3D display obtained by the eye position tracking module, and calculating the spatial position (x, y, z) of the human eye in the screen pixel coordinate system according to the relative position of the grating stereo display screen and the 3D camera of the calibration module;

[0011] S2. Perform viewpoint calculation and image rendering for each eye to obtain a scene pixel sub-image corresponding to each grating strip under the viewpoint of the eye, synthesize or overlay the scene pixel sub-images and output them to a display.

[0012] Preferably, in S2, the method of viewpoint calculation and rendering picture includes:

[0013] S21. According to the spatial position (x, y, z) of the eye, the position and viewing angle of the virtual camera are set;

[0014] S22, rendering the virtual 3D scene into a texture to obtain a scene image from an eye viewpoint;

[0015] S23, raster mapping calculation to obtain a scene pixel sub-image displayed by raster strips under the eye viewpoint.

[0016] Furthermore, S23 includes:

[0017] S231, calculating the screen area seen by the eye viewpoint through the grating strips;

[0018] S232: Map the texture corresponding to the screen area onto the screen area to form a scene pixel sub-image.

[0019] Further, in S231, the screen area seen by the eye through the grating is a long and narrow rectangle that makes an angle θ with the vertical angle, the length direction of the rectangle runs through the screen, and the width is Δu, which ranges from 0.05 mm to 0.1 mm;

[0020] In S232, on the screen corresponding to the rectangle, each pixel has a UV value, and the texture obtained by the UV value is used to fill the pixels on the grating strip corresponding to the rectangle; the texture coordinates used for each pixel are:

[0021] u coordinate: u=(Qx-x_min) / (x_max-x_min);

[0022] v coordinate: v=(Qy-y_min) / (y_max-y_min);

[0023] Where Q is the coordinate of the intersection of the line of sight through the pixel and the display screen, x_min and x_max represent the range of the rectangle's horizontal coordinates, and y_min and y_max represent the range of the rectangle's vertical coordinates.

[0024] Furthermore, S23 also includes:

[0025] Calculate the observation angle α and viewing angle range Δα of each grating strip:

[0026] If the observation angle α of the grating strip is greater than Δα, then ghosting is removed; preferably, the method for removing ghosting includes:

[0027] Note that the color value to be output at the current viewing angle is ψ1 = (r1, g1, b1), and the color value to be output at other viewing angles of the current grating is ψ2 = (r2, g2, b2). Then, for the current viewing angle of the current grating, the actual output value is:

[0028] ψ=ψ1+B1-ψ2×B2

[0029] Among them, B1 is the maximum light leakage amount, and B2 is the light leakage ratio.

[0030] Furthermore, it also includes a grating parameter calibration module for calibrating the parameters of the grating, including the relative geometric offset d0 of the grating relative to the upper left corner of the screen, the grating spacing d, the vertical distance h from the grating to the display screen, and the angle θ between the grating and the screen, and transmits the grating parameters to the perspective rendering and grating interlaced image generation module.

[0031] Furthermore, in the grating parameter calibration module, the method for calibrating the grating spacing d of the grating includes:

[0032] S1, displaying lines parallel to the grating on the display screen with a spacing of ai;

[0033] S2. Use a camera to shoot the screen at a distance Li to obtain image i;

[0034] S3. In the captured image i, detect the spacing Mi (Li) where the moire fringes appear; and convert it into the actual physical spacing M p (Li);

[0035] S4. Calculate the grating spacing d of the grating, the formula is:

[0036] 1 / M p (Li) = |1 / a i -1 / d eff (Li)|, d = d eff (Li)

[0037] Among them, d eff (Li) represents the effective grating pitch when observed at distance Li.

[0038] Further, in S4, steps S2-S3 are repeated to obtain the equation system: 1 / M p (Li) = |1 / a i -1 / d eff (Li)|,i=1,2,…,m;

[0039] D eff (Li) is simplified to: d eff (Li)≈d×(Li / (Li+h));

[0040] Fit and solve to obtain the spacing d of the grating and the vertical distance h from the grating to the display screen;

[0041] Further, the method for calibrating d0 includes:

[0042] (1) Fix the binocular camera as eyes;

[0043] (2) Generate a pixel sub-image of the scene seen by the binocular camera through a grating stereo display system;

[0044] (3) In S231, d0 in the formula Gn=n×d+d0 is the relative geometric offset of the calibrated grating relative to the upper left corner of the display screen. When d0 is adjusted so that the left eye and the right eye of the binocular camera receive the red sub-image and the blue sub-image respectively, d0 at this time is the calibration value.

[0045] The beneficial effects of the grating calibration method and the stereoscopic display system provided by the present invention include:

[0046] 1. The present invention is based on an eye tracking and grating stereoscopic display system, which realizes adaptive stereoscopic display of the viewpoints of single or multiple viewers' eyes by combining a 3D camera to track the user's eye position in real time and cooperating with the optical characteristics of the grating display, thereby reducing the computational complexity while maintaining a good stereoscopic display effect.

[0047] 2. Adjust the viewpoint image for each eye in real time to ensure the best stereoscopic vision consistency and better stereoscopic effect.

[0048] 3. High flexibility: The system can automatically adapt to the viewing environment of a single person or multiple people with multiple eyes, and different viewers or different viewing positions can obtain good stereoscopic display effects.

[0049] 4. Improved viewing effect: It can accurately render the image that should be seen by the actual viewing angle, avoiding the blur caused by the error between the rendering viewpoint and the actual viewpoint under fixed multi-viewpoint rendering, or the 3D image distortion caused by the actual viewpoint being outside the rendering viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0051] Figure 1 A working principle diagram of a stereoscopic grating display system for eye tracking according to an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structure of a stereoscopic grating display system for eye tracking according to an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the flow of a perspective rendering and grating interlacing module according to an embodiment of the present invention;

[0054] Figure 4 The figure is a flow chart of a grating parameter calibration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0056] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0058] The present invention proposes a grating stereoscopic display system for viewing angle tracking, such as Figure 1 , 2 As shown, it includes: a 3D camera grating display device, which is composed of a grating, a display screen, and a 3D camera; an eye position tracking module; a calibration module; a viewing angle rendering and grating interlaced image generation module; wherein,

[0059] 3D camera grating display device, used to obtain the viewer's 3D information and output stereoscopic images. Its display part consists of a display screen and a vertical columnar grating covering it. A 3D camera (such as a ToF camera or a binocular depth camera) is arranged near the display. The grating spacing d, the angle θ between the grating and the screen, the initial offset d0 of the grating relative to the upper left corner of the screen, and the 3D camera relative to the screen pose matrix M are important parameters affecting imaging. The screen resolution is set to w×h and the actual physical size is X×Y.

[0060] The eyeball position tracking module is used to obtain the real-time spatial position coordinates of the viewer's eyes relative to the 3D camera. It can output eye tracking information in real time according to the changes in the user's head posture, such as tracking and obtaining the spatial position and orientation of the human eye; if multiple people are watching, the corresponding information of each eye is obtained separately. This module passes the tracked eye coordinate information to the subsequent perspective rendering and grating interlaced map generation module to calculate the correct mapping of each grating strip in the screen pixel coordinate system.

[0061] The pose calibration module is used to obtain the spatial relationship between the 3D camera and the display, and transmit the result to the perspective rendering and grating interlaced image generation module. Conventional calibration methods such as chessboard calibration can be used.

[0062] The viewing angle rendering and grating interlaced image generation module (hereinafter referred to as the interlaced module) connects the 3D camera grating display device and the eye tracking module, obtains the eye position transmitted by the eye tracking module, and calls the 3D rendering program (such as OpenGL, DirectX, etc.) to calculate in real time and map and control the grating strips one by one, and generate a grating stereo display image suitable for the eye: to ensure that the pixel sub-images seen by each eye do not interfere with each other, so as to form a stereoscopic vision in space. Finally, the result is sent to the grating display for output.

[0063] In one embodiment, the grating spacing d ranges from 0.1 mm to 1.0 mm, preferably 0.3 mm to 0.5 mm; the initial offset ranges from 0 to d; the vertical distance h from the grating to the display screen ranges from 0.5 mm to 2.0 mm, preferably 0.8 mm to 1.2 mm.

[0064] In the interleaving module, Figure 3 For each eye, do the following:

[0065] S1, receiving the position of the human eye relative to the 3D display obtained by the eye position tracking module, and the relative position of the grating stereo display screen and the 3D camera sent by the position calibration module, and calculating the spatial position (x, y, z) of the human eye in the display screen coordinate system;

[0066] S2. Calculate the viewpoint and render the image for each eye, synthesize or overlay the rendered images, and output them to the grating display. Due to the cylindrical lens effect of the grating, different viewing angles will only see the pixel sub-images consistent with their line of sight, thus forming a stereoscopic vision at the user's eye position.

[0067] In S2, the method of viewpoint calculation and rendering includes:

[0068] S21. According to the spatial position of the eye E = (x, y, z), set the position and viewing angle (projection matrix and view matrix) of the virtual camera;

[0069] S22, scene rendering: rendering the virtual 3D scene into an off-screen frame buffer (Frame Buffer Object, FBO), i.e., a texture, to obtain a scene image from the eye's viewpoint;

[0070] These two steps can be completed in image rendering software such as openGL.

[0071] S23, raster mapping calculation to obtain a scene pixel sub-image displayed by raster strips at the eye viewpoint, specifically including:

[0072] S231, due to the reversibility of the optical path, the viewpoints are reversely projected onto the screen coordinate system through each grating, and the screen area seen by the eye through the grating is calculated for subsequent coloring purposes. Specifically including:

[0073] (1) Calculate the position and center of each grating strip: The grating is composed of multiple grating strips with equal spacing and an angle θ in the vertical direction. It is necessary to determine the precise position of each strip to prepare for the subsequent calculation of the display content of each grating strip. The distance formula of the nth grating strip is as follows:

[0074] Gn=n×d+d0

[0075] Where d is the grating spacing, d0 is the initial offset, n is the grating strip number, and the value range of n is: 0 to

[0076] In one embodiment, the center point of the nth grating is:

[0077]

[0078] in In the vertical direction of the grating, the compensation term is used to correct the cylindrical imaging distortion when the observation angle is large. It can generally be a linear or quadratic function of the observation angle α.

[0079] (2) The screen area is a long and narrow rectangle that makes an angle θ with the vertical. The midpoint of the rectangle is denoted by P. n :

[0080] P n =Cn-(x,y)×h / z,

[0081] The length direction of the rectangle runs through the screen, and the width is Δu, which ranges from 0.05mm to 0.1mm.

[0082] S232, interlaced coloring: On the screen corresponding to the rectangle, each pixel has a UV value, and the texture obtained by the UV value is used to fill the pixels on the raster strip corresponding to the rectangle. For each pixel, the texture coordinates used are as follows:

[0083] u coordinate: u=(Qx-x_min) / (x_max-x_min);

[0084] v coordinate: v=(Qy-y_min) / (y_max-y_min);

[0085] Where Q is the coordinate of the intersection of the line of sight through the pixel and the display screen, x_min and x_max represent the range of the rectangle's horizontal coordinates, and y_min and y_max represent the range of the rectangle's vertical coordinates.

[0086] In one embodiment, S23 also needs to determine the relationship between the observation angle α and the viewing angle Δα required for each grating. If α>Δα, it is easy to cause ghosting crosstalk between the left and right eye images, and de-ghosting processing is required.

[0087] (1) The calculation method of the observation angle α is as follows:

[0088] Project the eye position in the direction of the grating strip: in is the unit vector in the raster direction, and (x,y) is the components of the screen coordinate system in the X and Y directions in the eye's spatial position.

[0089] Calculate the viewing angle: in: is the display screen normal vector, and v=normalize(EE′) is the normalized sight line vector.

[0090] (2) Each grating strip is like a small cylindrical lens, which can only accurately display images within a specific angle range. Beyond the angle range, the image quality will deteriorate. The formula for calculating the viewing angle range is as follows:

[0091] Δα=2*arctan(d / (2h))

[0092] Where h is the vertical distance from the grating to the display screen.

[0093] Parameter range: The value range of h is from 0.5 mm to 2.0 mm, preferably from 0.8 mm to 1.2 mm; it can be accurately obtained through calibration.

[0094] (3) Methods for removing ghosting include:

[0095] The color value to be output at the current viewing angle is ψ1 = (r1, g1, b1), where r, g, and b represent red, green, and blue. The color value to be output at the other nearest viewing angles of the current grating is ψ2 = (r2, g2, b2). For the current viewing angle of the grating, the actual output value is:

[0096] ψ=ψ1+B1-ψ2×B2

[0097] Among them, B1 is the maximum light leakage amount, and B2 is the light leakage ratio, which can be obtained by manually specifying the value. This means that the current grating leaks light to other gratings because it exceeds the optimal viewing angle, so compensation is needed.

[0098] For multi-user viewing, it is only necessary to further increase the eye viewpoints so that multiple users can see the 3D scene at the same time. In practice, a small amount of crosstalk may occur due to the limitation of screen resolution.

[0099] Example 1: Single-person viewing scenario

[0100] 1. Hardware configuration:

[0101] o Display: 23.8-inch 4K resolution raster display;

[0102] o Grating parameters: d = 0.35 mm, h = 1.0 mm;

[0103] o 3D camera: 60fps sampling rate, ±3mm positioning accuracy.

[0104] 2. Test results:

[0105] oOptimal viewing distance: 500mm;

[0106] oView tracking delay: ≤16.7ms;

[0107] o System frame rate: stable 60fps;

[0108] o Depth of field: 10cm in front of the screen to about 3m behind the screen.

[0109] Example 2: Two-person viewing scenario

[0110] 1. Hardware configuration: Same as Example 1.

[0111] 2. Test scenario:

[0112] o The two viewers were located 45° to the left and right of the monitor;

[0113] o Viewing distance: 700mm.

[0114] 3. Test results:

[0115] o Frame rate per person: stable 30fps;

[0116] o Crosstalk rate: ≤5% (the ratio of the image from one viewing angle leaking to the image from another viewing angle to the total image);

[0117] o Effective viewing range: ±40°.

[0118] The hardware part of the stereoscopic display system of the present invention only needs to install a grating on the front end of the display (or directly purchase a display screen with a grating) and match it with a 3D camera. No additional equipment needs to be installed on the audience side, and the deployment process is simple and flexible. It can be applied to various fields such as advertising display, education and training, medical visualization, remote interaction, game entertainment, etc., and has good compatibility with large-screen and small and medium-sized display devices.

[0119] With the help of grating stereo display technology, this system can provide multi-viewpoint holographic 3D images within a certain range. After eye tracking, the system only renders the actual observation viewpoint, avoiding repeated calculation of useless viewpoints and greatly reducing the amount of calculation. The present invention can achieve real-time tracking of the position of the audience's eyes through a 3D camera, and dynamically update the stereoscopic image according to the eye position, so that the viewer can still maintain clear and stable stereoscopic vision when moving or turning his head.

[0120] The present invention also proposes a grating parameter calibration method, such as Figure 4As shown, it is used to calibrate grating parameters such as grating spacing d, relative geometric offset of grating relative to the upper left corner of the screen (i.e., initial offset d0), angle θ between grating and screen, and transmit grating parameters to the viewing angle rendering and grating interlaced graph generation module to ensure the accuracy of the optical model of rendering and display. This module only depends on the camera position rather than the camera posture and personnel participation. It makes the grating strips parallel to the lines (spacing a) displayed on the display screen and overlap in the same plane projection direction to produce a Moiré effect that is easy to analyze. The grating parameters are solved by analyzing the stripes.

[0121] Generally, the real physical spacing d of the grating, the angle θ between the grating and the screen, the relative geometric offset of the grating relative to the upper left corner of the bottom screen (i.e., the initial offset d0) and the distance h between the grating and the display screen are important parameters affecting imaging.

[0122] By using the grating parameter calibration method, the real physical spacing d of the grating, the relative geometric offset of the grating relative to the upper left corner of the display screen (i.e., the initial offset d0) and other parameters can be automatically calibrated. The calibration steps include:

[0123] S1, displaying lines parallel to the grating on the display screen with random spacing ai;

[0124] S2. Use a camera to shoot the screen at a distance Li to obtain image i;

[0125] S3, in the captured image i, detecting the spacing Mi(Li) where the moiré fringes appear, and converting it into an actual physical spacing;

[0126] S4. Calculate and obtain the real physical spacing of the grating.

[0127] In one embodiment, in order to obtain a more accurate effect, some preparations can be performed before S1. For example, turn off any interference pattern that is not related to the vertical direction of the grating, and ensure that only the lines with a spacing a are parallel to the grating. The camera records the "distance L" from the screen without recording the posture (R, t). If the camera can output three-dimensional coordinates, only the vertical distance to the screen plane is required.

[0128] In S1, lines are displayed on the display with a random spacing a. The effect is to display a set of parallel stripes in the center of the display, with a stripe spacing a and the stripe direction consistent with the grating direction, so as to avoid additional "diagonal distortion" in the final moiré fringes.

[0129] In one embodiment, in order to calibrate θ, a single-pixel-width oblique line can be displayed and the oblique line angle can be manually controlled. If the oblique line appears only on a single grating strip from any angle, it means that the direction of the oblique line is consistent with the grating direction, thereby calibrating θ.

[0130] In S2, at different distances Li, keep the camera roughly facing the screen (not strictly necessary, but ensure that the stripes are visible in most areas of the image), use the camera to shoot the screen, obtain image i, and record the distance Li (in mm) corresponding to each photo.

[0131] The grating itself has a spacing d. When observed or photographed from a certain distance L, Moiré fringes will appear. In S3, for each captured image i, the "macro fringes" spacing Mi (Li) (unit: pixel) where the Moiré fringes appear is detected. The detection method is:

[0132] Perform a two-dimensional Fourier transform on the captured image i and find the spacing corresponding to the peak point, which is the moiré fringe spacing Mi(Li).

[0133] Convert the moiré fringe spacing Mi(Li) into the actual physical spacing M p (L i ), we need to understand the imaging scaling relationship of the camera at distance Li, which can be calculated through simple camera calibration (only referring to focal length f and pixel size k):

[0134] M p (L i )≈Mi(L i )×(k / f)×Li,

[0135] Where k / f represents the conversion factor between pixels at a unit distance and actual mm, and Li is the distance from the camera to the screen.

[0136] The actual physical spacing M of the moiré fringes p (L i ) and the distance a between the stripes on the display i The relationship between the grating spacing d satisfies:

[0137] 1 / M p (Li) = |1 / a i - 1 / d eff (Li)| (1)

[0138] Among them, d eff (Li) represents the effective grating spacing when observed at distance Li. Because of factors such as the refraction / perspective of light passing through the grating, the effective spacing will change with the observation distance.

[0139] In S4, solve equation (1) to obtain d eff (Li), which can be considered as the actual physical spacing d of the grating.

[0140] In one embodiment, if the requirements for precision, accuracy, stability, etc. are higher, m points can be taken, steps S2-S3 can be repeated, multiple equations (1) can be obtained, and a system of equations can be formed, and fitting and solving can be performed to obtain the real physical spacing of the grating and the relative geometric offset between the grating and the display screen. Preferably, Li is selected to cover the range of 300mm to 1000mm, and the data is collected every 50 to 100mm. The specific steps are as follows:

[0141] (1) Assuming that the grating refraction effect can be regarded as a fixed offset, d eff (Li) is simplified to:

[0142] d eff (Li)≈d×(Li / (Li+h))

[0143] Where d is the actual physical spacing of the grating.

[0144] (2) The measured m M p (Li) into the above formula, we can get several sets of equations:

[0145] 1 / M p (Li) = |1 / a i -1 / d(Li / (Li+h))|,i=1,2,…,m

[0146] (3) Then, d and h are solved by fitting. For example, the least square method can be used, and then δ = 1 / M p (Li)-|1 / a i -1 / d(Li / (Li+h))|, hoping that ∑δ2 is minimized, thus solving for the global optimal d and h.

[0147] In the stereoscopic display system of the present invention, a grating parameter calibration module using the above calibration method is also included. After calibrating d and h, d0 can be calibrated. The initial offset d0 mainly affects the "phase" of the moiré fringe and has no effect on the fringe period; therefore, in one embodiment, a stereoscopic display system is used to generate pure red and pure blue images for a binocular camera at a fixed position, and d0 is adjusted so that the left and right cameras respectively capture pure red and pure blue images. Specifically including:

[0148] (1) Fix the binocular camera as the eyes; (2) Generate a pixel sub-image of the scene seen by the binocular camera through the grating stereo display system, wherein in S231, d0 in the formula Gn=n×d+d0 is continuously adjusted. If the image captured by the binocular camera is pure red and pure blue, then d0 is the relative geometric offset of the calibrated grating relative to the upper left corner of the display screen (i.e., the initial offset d0).

[0149] Those skilled in the art will appreciate that the foregoing embodiments are merely intended to illustrate the technical solutions of the present invention rather than to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments may be modified or some or all of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope defined by the claims of the present invention.

Claims

1. A grating stereoscopic display system with viewing angle tracking, characterized in that: include: 3D camera grating display device composed of grating, display screen and 3D camera, eye position tracking module; posture calibration module; Perspective rendering and grating interlaced image generation module; wherein, 3D camera grating display device, used to obtain the viewer's 3D information and output stereoscopic images; The eyeball position tracking module is used to obtain the real-time spatial position coordinates of the audience's eyes relative to the 3D camera, and transmit the tracked eye coordinate information to the perspective rendering and raster interlaced image generation module; A pose calibration module is used to obtain the spatial relationship between the 3D camera and the display, and transmit the result to the perspective rendering and raster interlaced image generation module; The perspective rendering and grating interlaced image generation module is connected to the 3D camera grating display device, the posture calibration module and the eye tracking module. After obtaining the eye position, the relative posture of the display screen and the 3D camera, it calculates the spatial position of the human eye on the display screen and the mapping of each grating strip in the screen pixel coordinate system, generates a grating stereo display image suitable for the eyes, and sends it to the display for output.

2. The grating stereoscopic display system according to claim 1, characterized in that: The range of the grating spacing d is 0.1mm to 1.0mm, the range of the initial offset is 0 to d; the range of the vertical distance h from the grating to the display screen is 0.5mm to 2.0mm.

3. The grating stereoscopic display system according to claim 1, characterized in that: The view rendering and raster interlacing generation modules include: S1, receiving the position of the human eye relative to the 3D display obtained by the eye position tracking module, and calculating the spatial position (x, y, z) of the human eye in the screen pixel coordinate system according to the relative position of the grating stereo display screen and the 3D camera of the calibration module; S2, performing viewpoint calculation and image rendering for each eye, obtaining a scene pixel sub-image corresponding to each grating strip under the viewpoint of the eye, synthesizing or superimposing the scene pixel sub-images and outputting them to a display; Preferably, in S2, the method of viewpoint calculation and rendering picture includes: S21. According to the spatial position (x, y, z) of the eye, the position and viewing angle of the virtual camera are set; S22, rendering the virtual 3D scene into a texture to obtain a scene image from an eye viewpoint; S23, raster mapping calculation to obtain a scene pixel sub-image displayed by raster strips under the eye viewpoint.

4. The grating stereoscopic display system according to claim 3, characterized in that: S23 includes: S231, calculating the screen area seen by the eye viewpoint through the grating strips; S232: Map the texture corresponding to the screen area onto the screen area to form a scene pixel sub-image.

5. The grating stereoscopic display system according to claim 4, characterized in that: In S231, the screen area seen by the eye through the grating is a long and narrow rectangle that makes an angle θ with the vertical, the length direction of the rectangle runs through the screen, and the width is Δu, which ranges from 0.05 mm to 0.1 mm; In S232, on the screen corresponding to the rectangle, each pixel has a UV value, and the texture obtained by the UV value is used to fill the pixels on the grating strip corresponding to the rectangle; the texture coordinates used for each pixel are: u coordinate: u=(Qx-x_min) / (x_max-x_min); v coordinate: v=(Qy-y_min) / (y_max-y_min); Where Q is the coordinate of the intersection of the line of sight through the pixel and the display screen, x_min and x_max represent the range of the rectangle's horizontal coordinates, and y_min and y_max represent the range of the rectangle's vertical coordinates.

6. The grating stereoscopic display system according to claim 4, characterized in that: S23 also includes: Calculate the observation angle α and viewing angle range Δα of each grating strip: If the observation angle α of the grating strip is greater than Δα, then ghosting is removed; preferably, the method for removing ghosting includes: Note that the color value to be output at the current viewing angle is ψ1 = (r1, g1, b1), and the color value to be output at other viewing angles of the current grating is ψ2 = (r2, g2, b2). Then, for the current viewing angle of the current grating, the actual output value is: ψ=ψ1+B1-ψ2×B2 Among them, B1 is the maximum light leakage amount, and B2 is the light leakage ratio.

7. The grating stereoscopic display system according to claim 1, characterized in that: It also includes a grating parameter calibration module for calibrating the parameters of the grating, including the relative geometric offset d0 of the grating relative to the upper left corner of the screen, the grating spacing d, the vertical distance h from the grating to the display screen, and the angle θ between the grating and the screen, and transmits the grating parameters to the viewing angle rendering and grating interlaced image generation module.

8. The grating stereoscopic display system according to claim 7, characterized in that: In the grating parameter calibration module, the method for calibrating the grating spacing d of the grating includes: S1, on the display screen with a spacing of i Display lines parallel to the grating; S2, at a distance L i At , use the camera to shoot the screen to obtain image i; S3. Detect the spacing M of moire fringes in the captured image i. i (L i ) and convert it into the actual physical distance M p (L i ); S4. Calculate the grating spacing d of the grating, the formula is: 1 / M p (L i ) = |1 / a i - 1 / d eff (L i )|, where d = d eff (L i ) Among them, d eff (L i ) indicates that at a distance L i The effective grating spacing during observation.

9. The grating stereoscopic display system according to claim 8, characterized in that: In S4, repeat steps S2-S3 to obtain the equation system: 1 / M p (L i )=|1 / a i -1 / d eff (L i )|,i=1,2,…,m; D eff (L i ) is simplified to: d eff (L i )≈d×(L i / (L i +h)); By fitting and solving, we can obtain the spacing d of the grating and the vertical distance h from the grating to the display screen.

10. The grating stereoscopic display system according to claim 8, characterized in that: Methods for calibrating d0 include: (1) Fix the binocular camera as eyes; (2) Generate a pixel sub-image of the scene seen by the binocular camera through a grating stereo display system; (3) In S231, for formula G n =n×d+d0, where d0 is the relative geometric offset of the calibrated grating relative to the upper left corner of the display screen. When d0 is adjusted so that the left eye and the right eye of the binocular camera receive the red sub-image and the blue sub-image respectively, d0 at this time is the calibration value.

Images