Method for displaying stereoscopic images on an autostereoscopic display device

Abstract

The present invention relates to a method for displaying stereoscopic images of a 3D scene to a viewer of an autostereoscopic display device, where the rendering of the stereoscopic images takes into account the viewer's viewing position relative to the autostereoscopic display device, so that the viewer can experience the 3D scene from a perspective that corresponds to their viewing position relative to the 3D scene. In the method of the present invention, rendering is performed on the center of the eyeball rather than on the pupil. This bypasses the saccadic movements of the viewer's eyes so that they do not play a role in image rendering, thereby solving the problem of saccadic movements being too fast to be considered in time due to system latency. The method of the present invention ensures that the viewer experiences the intended stereoscopic images in the correct manner and immediately after the saccadic movements of the eyes. The present invention also relates to an autostereoscopic display device configured to perform such a method.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to a method for displaying stereoscopic images of a 3D scene to a viewer on an autostereoscopic display device, and further to an autostereoscopic display device for displaying stereoscopic images of a 3D scene to a viewer. [Background technology]

[0002] Autostereoscopic displays are playing an increasingly important role in virtual reality and augmented reality applications. One of their most notable features is that they enable a viewer to perceive three-dimensional images even as the viewer moves relative to the display, without the need for dedicated eyewear devices or other wearables.

[0003] The key to this technology is the presence of an eye tracker in combination with a screen containing a lenticular lens or parallax barrier. This allows the autostereoscopic display to simultaneously direct a left-eye image to the viewer's left eye and a right-eye image to the viewer's right eye. The resulting stereoscopic image provides a depth perception, and elements in the image can appear in front of or further away than the display ("behind" the display).

[0004] Like virtually any electronic device that functions through the input of acquired data, autostereoscopic display systems suffer from latency, commonly understood as the time delay between user input and system response, also known as input lag. In an autostereoscopic display system configuration, this essentially means a delay between the viewer's head / eye movement (user input) and the corresponding adaptation to the displayed content (system response). Latency then manifests as inaccurate rendering of the stereoscopic image. It can also cause crosstalk when the left-eye image is intended for a portion of the viewer's screen that is also intended for the viewer's right eye, and similarly when the right-eye image is intended for a portion of the viewer's screen that is also intended for the viewer's left eye. When latency exceeds a critical threshold, user performance and experience degrade. This is typically related to a disturbing look-around effect and / or crosstalk. For example, displayed objects are perceived, at least temporarily, not in their correct positions.

[0005] Latency in autostereoscopic displays is typically addressed by extrapolating historical position data of the viewer's pupils to obtain estimates of future values ​​(extrapolation may also include data regarding velocity and acceleration). These predictions are used in rendering the stereoscopic images, so that rapid movements of the user's head in the field of view of the autostereoscopic display can be taken into account in time (latency compensation) and do not result in inaccurate rendering, or do so to a lesser extent.

[0006] In addition to the user's overall head movement, the pupils themselves may also move relative to the autostereoscopic display (e.g., when the head is stationary). This is related to the rotational movement of the eyeballs. In some cases, this movement is very fast, known as a saccade. Such saccades pose problems for latency compensation because the saccadic movement is already complete before the system can react to it. Furthermore, such sudden movements do not conform to any head movement models typically used to extrapolate past pupil positions into the future. For example, typical latencies that need to be compensated for range from 60 to 130 ms, thus requiring an estimate of pupil position at a time 60 to 130 ms into the future, while saccades typically occur within 20 to 60 ms. Therefore, saccades are either missed in their entirety when captured by the eye tracker or cause errors in pupil position prediction, as the extreme eye velocities measured during saccades can lead to unrealistic predictions. For example, movement of the pupil position is extrapolated as if it were a long-lasting movement, which causes overshoot and jitter in the position.

[0007] Eye rotation can be expected to result in small pupil displacements, especially for full head degrees of freedom, but they cannot be ignored. Eye rotation alone, when uncompensated, appears to result in noticeable rendering inaccuracies. This is especially true when displayed objects are assumed to be aligned with real-world objects, such as when a virtual object is placed on a stationary real-world object. If a viewer's eyes make saccadic movements to such a configuration that is not compensated for in time by the eye tracker, the viewer will notice some (small but noticeable) movement of the virtual object relative to the stationary real-world object, which, of course, is also supposed to be stationary. This effect is most noticeable for virtual objects that appear close to the viewer's eyes. Crosstalk can also be perceived by a viewer whose eyes make saccadic movements when a light intended for the left eye hits the right eye after a right-eye saccade, and when a light intended for the right eye hits the left eye after a left-eye saccade.

[0008] Therefore, there is a need to address the impact of saccades on the latency compensation of the system. However, to date, no satisfactory solution has been found. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0009] <Summary of the Invention> Therefore, it is an object of the present invention to find a solution to the problem of inaccurate stereoscopic image rendering as a result of saccades and the problem of perceived crosstalk as a result of saccades. It is also an object to reduce or even cancel inaccurate image display as a result of the latency of an autostereoscopic display device, in particular inaccurate image display characterized by an incorrect 3D scene position relative to the real world. More generally, it is an object of the present invention to improve the viewing experience of a viewer of an autostereoscopic display device, including improving the viewer's perception of the position of a 3D scene relative to the real world. [Means for solving the problem]

[0010] It has now been discovered that one or more of these objectives can be achieved by performing eye tracking in different ways.

[0011] The present invention therefore relates to a method for displaying a stereoscopic image of a 3D scene to a viewer on an autostereoscopic display device, said stereoscopic image being displayed by said autostereoscopic display device and consisting of a left-eye image to be viewed by the left eye of said viewer, and a right-eye image to be viewed by the right eye of said viewer, said method comprising: - providing 3D scene data representing a 3D scene; - rendering stereoscopic images from the 3D scene data, taking into account a viewer's viewing position relative to the autostereoscopic display device, to enable the viewer to experience the 3D scene from a perspective corresponding to their viewing position relative to the 3D scene, The method further comprises - defining a location of a left rendering spot within a viewer's left eyeball relative to an autostereoscopic display device; - defining a location of a right rendering spot within a viewer's right eyeball relative to an autostereoscopic display device; - using a position of a left rendering spot relative to an autostereoscopic display device and a position of a right rendering spot relative to an autostereoscopic display device to render a stereoscopic image from the 3D scene data; - displaying the rendered stereoscopic image; The left and right rendering spots have fixed positions relative to the viewer's entire head and are located at or no more than 8.0 mm from the centers of rotation of their respective eyes.

[0012] The present invention further provides an autostereoscopic display device for displaying to a viewer a stereoscopic image of a 3D scene, said stereoscopic image consisting of a left image to be seen by the viewer's left eye and a right image to be seen by the viewer's right eye, said autostereoscopic display device comprising: a left rendering spot and a right rendering spot tracking system, the location of the left rendering spot within the viewer's left eyeball relative to the autostereoscopic display device; configured to track the position of a right rendering spot within the viewer's right eye relative to the autostereoscopic display device; the left rendering spot and the right rendering spot have fixed positions relative to the viewer's entire head and are located at or no more than 8.0 mm from the center of rotation of each eye; a display unit, such as a screen, configured to display a left-eye image to be viewed by the viewer's left eye and a right-eye image to be viewed by the viewer's right eye, an array of display pixel elements for generating a display output; and the display portion including a lenticular device disposed over the array, the lenticular device comprising lenticular lens areas that can direct display output from different display pixel elements to different spatial locations within a field of view of the autostereoscopic display device to enable the display of a stereoscopic image composed of left-eye and right-eye images; a rendering module configured to render stereoscopic images from the 3D scene data, taking into account a viewing position of a viewer relative to the autostereoscopic display device, to enable a viewer to experience the 3D scene from a viewpoint corresponding to their viewing position relative to the 3D scene, The rendering module is further related to an autostereoscopic display device configured to use a position of a left rendering spot relative to the autostereoscopic display device and a position of a right rendering spot relative to the autostereoscopic display device to render a stereoscopic image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 shows a schematic series of top views of a setup in which an image is rendered according to conventional methods. [Figure 2] FIG. 2 shows a series of schematic top views of a setup in which an image is rendered according to the method of the present invention. [Figure 3] Figure 3 is a visual representation of events occurring in the real world along an eye timeline that reflects eyeball orientation and a display timeline that reflects delayed image display as a result of latency. DETAILED DESCRIPTION OF THE INVENTION

[0014] The drawings do not limit the invention to the particular embodiments disclosed and described herein. Elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale, emphasis instead being placed on clearly illustrating the principles of the invention. For example, the relative dimensions of the screen of an autostereoscopic display device, the virtual object presented by the screen, and the eyeballs perceiving the virtual object cannot be derived from the figures.

[0015] In the context of the present invention, the term "viewer" means a real-world person who can consume, and in particular view, content presented by an autostereoscopic display device. Throughout this text, references to the viewer will be made by masculine terms such as "he," "him," or "his." This is for purposes of clarity and brevity only, as it will be understood that feminine terms such as "she" and "her" apply equally.

[0016] Throughout this text, the term "3D" is used for brevity. This term is meant to be equivalent to the term "three-dimensional." For example, the terms "3D scene" and "3D object" are meant to denote a "three-dimensional scene" and a "three-dimensional object," respectively.

[0017] In the context of this invention, "rendering" means using computations to create or derive left-eye and right-eye images from the perspective of a particular virtual stereo camera in a 3D scene based on available data of the 3D scene. When viewed in combination, the left-eye and right-eye images cause a viewer to perceive the 3D scene as a 3D image from the perspective of the particular virtual stereo camera.

[0018] Thus, rendering according to the present invention can in fact be viewed as generating, by a virtual stereo camera, image data representing a stereoscopic view of a 3D scene from a viewpoint corresponding to a particular position of the virtual stereo camera relative to the 3D scene. The virtual stereo camera then has its position at the viewer's eye position, with a left virtual camera at the viewer's left eye and a right virtual camera at the viewer's right eye. Consequently, movement of the viewer's head in the real world causes the virtual stereo camera to move relative to the 3D scene. The rendering module then sequentially creates subsequent stereoscopic pairs of left-eye and right-eye images from the moving observation point (or standpoint) of the virtual stereo camera. Tracking the eye positions determines the position of the virtual stereo camera relative to the 3D scene to provide virtual stereo camera "recordings" that can be displayed as stereoscopic images.

[0019] In the context of the present invention, a "3D scene" refers to a specific environment shaped in three dimensions, including one or more elements for which one or more properties are known, such as properties selected from the group of shape, size, surface orientation, surface properties, relative position (e.g., with respect to other elements), and physical behavior. The behavior of incident light and / or sound in the 3D scene may also be known, leading, for example, to the occurrence of shadows and surface reflections in a rendered image. A 3D scene is typically a virtual environment, but may also be a (recorded) real environment.

[0020] In the context of the present invention, the term "3D-scene data" refers to information representing a 3D scene, such as the 3D characteristics of elements of the 3D scene, and information about how these elements relate to one another in the 3D scene (3D mapping of the 3D scene). The 3D scene data may include data about one or more characteristics of the scene elements selected from the group consisting of shape, dimensions, surface orientation, surface properties, relative position (e.g., with respect to other elements), and physical behavior. The method of the present invention uses the 3D scene data to generate a stereoscopic image of the 3D scene corresponding to a particular viewpoint relative to the 3D scene, a process known as "rendering." Such rendering can be performed for virtually any desired viewpoint of the scene. The rendered stereoscopic image is then displayed on an autostereoscopic display device and perceived as a 3D image from the particular viewpoint.

[0021] The 3D scene data may be stored in a memory portion associated with the autostereoscopic display device, or may be generated in real time, for example by inputting a live recording (typically of the real world) into the autostereoscopic display device, or it may be a combination of both, for example when a viewer or another person makes live modifications to the stored 3D scene data.

[0022] It is stated herein that the latency of an autostereoscopic display device is or can be compensated for, typically by extrapolating historical viewing positions into the future and rendering a stereoscopic image by using such extrapolated values. This means that the apparent latency, i.e., the latency perceived by the viewer, is now reduced. The latency of the autostereoscopic display device itself remains unaffected by such methods.

[0023] The method according to the present invention applies known principles for displaying stereoscopic images of a 3D scene to a viewer, such that the viewer experiences the 3D scene from a viewpoint corresponding to the viewer's viewing position relative to the 3D scene. Furthermore, if the viewer changes their position relative to the 3D scene, such changes are reflected in a new stereoscopic image that is subsequently displayed. In this way, a viewer moving laterally relative to the display device may experience motion parallax of the 3D scene (foreground objects appear to move more than background objects) when the 3D scene is fixed on the screen of an autostereoscopic display device. The user may also view (or inspect) the displayed 3D object from different angles, which may cause the user to perceive the so-called "look-around effect." In an ideal situation, as pursued by the present invention, the experienced motion parallax of the 3D scene matches that of the real world. For example, when a virtual object is positioned on a real object that exists between the screen and the viewer, both objects do not move relative to each other when viewed from different angles.

[0024] However, the 3D scene does not need to be fixed to the screen in order to be experienced from a viewpoint corresponding to the viewer's viewing position. In some embodiments, the 3D scene is fixed to the real world and is not affected by movement of the screen relative to the real world, in which case the screen acts as a virtual window or moving frame through which the 3D scene is viewed.

[0025] In other embodiments, the 3D scene is movable relative to the screen, allowing the viewer to view the 3D scene from different angles while still sitting in front of the screen. In such cases, the viewer may have a joystick, mouse, or other input method to control the movement of the 3D scene, for example its rotation or translation relative to the screen.

[0026] Known methods that apply the principle of adapting a 3D scene to a viewer's viewing position track pupil position so that the intended image information can be accurately presented to each eye. However, all display devices to which such methods are applied suffer from noticeable inaccuracies in stereoscopic image rendering when pupil movement is very fast. This means that the time it takes for the pupil to reach a new position that is far enough away to cause a noticeable inaccuracy in the image is shorter than the latency of the display device. Such latency is also known as the response time of the display device, which is the time delay between when the viewer's pupil assumes a new position and when the eye sees the displayed image taking into account the new pupil position.

[0027] For a given latency, relatively slow pupil movements, such as those caused by normal head movements or when the pupil is in a so-called smooth tracking mode, do not cause noticeable inaccuracies in the display of images. On the other hand, relatively fast pupil movements cannot be compensated for in a timely manner. Such fast pupil movements are almost always caused by saccades, not by common movements of the entire head. Saccades are generally too fast for autostereoscopic display devices to take them into account in a timely manner.

[0028] This is illustrated by the example situation in Figure 3, which shows eyeball The eye timeline (the lower timeline in the figure) reflects the eye direction at time t display The display timeline (top timeline in the figure) reflects the stereoscopic image display at time t. The display timeline lags behind the eye timeline due to a system latency of 80 ms. Between both timelines, a schematic top-view of the eye, including the pupil, is provided at specific intervals along the eye timeline, with different pupil positions indicating changing eye orientation over time (e.g., saccades occurring between 20 and 80 ms).

[0029] In this particular example, the display timeline is delayed by 80 ms relative to the eye timeline. The delay is a result of the latency of the autostereoscopic display device. This delay is 80 ms, and t eyeball The eye direction at t = 0 ms is eyeball =80ms(t display = 0 ms) is used to render images that appear to the viewer only at eyeball The 60 ms saccade of the eye that ends at t = 80 ms is not considered within these 80 ms. eyeball At t = 80 ms, the eye display = 0 ms), the image rendered at t eyeball The eye direction at t = 80 ms was display = 80 ms) to render the image that is visible to the viewer. Thus, for only 80 ms after the completion of a saccade, the viewer sees an image that matches the eye direction resulting from the saccade (assuming no new saccades have occurred).

[0030] Furthermore, during the saccade itself, the image does not match the eye orientation (the mismatch is due to the eyeball = 20ms from zero to t eyeball (The maximum value is reached at t = 80 ms.) Therefore, the time eyeball For a period of time, the viewer is presented with an image that is not rendered correctly.

[0031] Inaccurate rendering as a result of pupil position not being taken into account (in time) is visualized in Figure 1. Each of the four figures represents a top view of a setup where an image is rendered onto the pupil of a viewer, and each image contains a triangular object and a square object that the viewer perceives as being in front of the screen of an autostereoscopic display device (the eyeball with pupil is shown at the bottom of each figure, and the screen seen by the eyeball is shown at the top of each figure). The four figures correspond to different points in time in one and the same setup where saccades occur. These points correspond to the events visualized in the eye timeline of Figure 3. The eyeball is first directed towards the triangle (t eyeball =0ms). The eyeballs are eyeball When directed to the square at t = 80 ms (immediately after the saccade is completed), the rendering of the square at that time is still eyeball It can be seen that this is based on the pupil position at t = 0 ms. eyeball Only at =160ms is the rendered image rendered with the correct pupil position.

[0032] Rather than aiming to eliminate latency itself (i.e., true latency reduction), for example, by improving the display device hardware or software, the present invention provides an approach in which image rendering is performed without considering the occurrence of saccades (or take out of play) and independent of pupil position relative to the head. In this alternative approach, rendering is performed for a rendering spot at each of the viewer's eyes at a (largely) fixed position relative to the head as a whole. Thus, the rendering spot at each eye has a position that is independent of whether the eye rotates within its socket. According to the present invention, each of the two rendering spots is located at or near the center of the respective eyeball (typically within 5.0 mm of the center).

[0033] According to common theory, conventional rendering of a stereoscopic image on the pupil itself provides the best user perception of a 3D scene, in the sense that it is stable and fixed to the real world. This means that the pupil position acquired by the eye tracker is taken as the viewing position for rendering the stereoscopic image. However, surprisingly, deviation from this as applied in the present invention (i.e., by selecting a rendering spot at or near the center of the eyeball) seemed to have little effect on the viewer's perception of the 3D scene position relative to the real world. This may be due to the fact that the deviation applied by the present invention does not significantly affect light rays traveling through the lens normal or in a direction slightly off the normal, because these rays are not (or are barely) subjected to refraction. And it is this direction that is most important to the viewer, because the viewer's gaze is primarily in the direction of these largely unrefracted light rays. It is recognized that while all directions other than the normal will theoretically bring along some rendering inaccuracy, directions close to the normal, for example, within 10° of the normal, will not result in noticeable image degradation by the user. Only peripheral regions truly far from the viewer's point of gaze can introduce substantial errors in the 3D scene position relative to the real world, but the viewer typically does not perceive these because the human eye does not have many photoreceptors in the periphery of the field of view. Thus, the periphery is perceived with much less detail than areas the user is viewing in the more central parts of the field of view, where there are more photoreceptors.

[0034] The beneficial effect of this method of inaccurate rendering in peripheral regions relative to a particular gaze is that this inaccuracy disappears immediately when gaze is directed toward these regions after a saccade (assuming that the change in pupil position due to whole head movement during a saccade is small compared to the change in pupil position due to the saccade). This is because rendering according to the present method is performed on a rendering spot at or near the center of the eyeball. Regardless of eye orientation, light rays traveling perpendicular to the eye lens always pass through this rendering spot, as do light rays that are part of the viewer's gaze. In other words, all regions of an image rendered according to the present invention are rendered simultaneously, as if the viewer were gazing at all of them simultaneously. Therefore, all new gaze points following a saccade are automatically rendered correctly. After a saccade, the user does not perceive a change in rendering position after the actual display latency, since the rendering position is not affected by the saccade. This effectively removes saccades from consideration by the present method without adversely affecting image perception at the gaze point.

[0035] Rendering according to the invention is visualized in Figure 2 in a similar way to Figure 1. Each of the four figures represents a top view of a setup where an image is rendered onto the center of the viewer's eyeball (the rendering spot), and each image contains a triangular object and a square object that the viewer perceives as being in front of the screen (the eyeball with pupil is shown at the bottom of each figure, and the screen seen by the eyeball is shown at the top of each figure). The four figures correspond to different points in time in one and the same setup where saccades occur. These points correspond to the events visualized in the eyeball timeline of Figure 3. The eyeball is first directed towards the triangle (t eyeball =0ms). The eyeballs are eyeball We can see that when directed to the square at t = 80 ms (immediately after the completion of the saccade), the rendering of the square at that time is based on the rendering spot of the eye's center, which remains in the same position during eye rotation. In other words, eye rotation does not incur inaccurate rendering at the fixation point (square). However, at teyeball It can be seen that the rendering of the square at ≠0 ms is indeed inaccurate. As mentioned above, this inaccurate rendering occurs at the periphery of the visual field and is therefore preferable to inaccurate rendering at the fixation point.

[0036] Rendering therefore occurs at two rendering spots, the positions of which are defined relative to the entire head. There is a left rendering spot associated with the left eye and a right rendering spot associated with the right eye, each of which resides within the transparent inner portion of the respective eye, the vitreous body. Each rendering spot itself is not specifically defined or bounded by any particular physical feature of the viewer; it is a point within a volume element centered at the center of rotation of the respective eye. Each rendering spot has a fixed position relative to the entire head of the viewer and is located at (or no more than 8.0 mm from) the center of rotation of its respective eye.

[0037] It is recognized that the center of rotation of the eyeball may be slightly variable relative to the eye socket (and therefore relative to the head) depending on the actual eyeball orientation, since the eye and / or eye socket do not necessarily form a perfect sphere. Thus, in practice, the eye may have multiple centers of rotation. Since any of these position changes are so small, their effect is not noticeable to the viewer, and the perceived quality of the displayed stereoscopic image is therefore the same as if the method of the present invention were applied taking into account a variable center of rotation. Thus, if there are multiple centers of rotation for the eye, any one of them may be selected for the purpose of defining the rendering spot.

[0038] The rendering spots are typically located within 8.0 mm of such centers of rotation. Preferably, the rendering spots are located at a shorter distance, e.g., 7.0 mm or less, 6.0 mm or less, 5.0 mm or less, 4.0 mm or less, 3.0 mm or less, 2.0 mm or less, or 1.0 mm or less from the centers of rotation of their respective eyes. In preferred embodiments, their locations coincide with the centers of rotation of their respective eyes.

[0039] The rendering uses the position of a left rendering spot relative to the autostereoscopic display device and the position of a right rendering spot relative to the autostereoscopic display device, which may be obtained by (1) tracking the eyes of a particular viewer after an initial determination of the positions of the rendering spots in the viewer's eyes (or relative to the pupils), or (2) tracking one or more other facial features of a particular viewer when the positions of these features relative to the left and right rendering spots are known for the viewer, e.g., by initial determination.

[0040] Thus, in one embodiment, the position of the left rendering spot for an autostereoscopic display device and the position of the right rendering spot for an autostereoscopic display device are determined by the following steps: 1) identifying one or more facial features of the viewer other than the left and right rendering spots; 2) determining the location of a left rendering spot and a right rendering spot relative to one or more facial features; 3) determining the location of facial features relative to an autostereoscopic display device; 4) Using the positions determined in steps 2) and 3), determining a position of a left rendering spot for the autostereoscopic display device and / or a position of a right rendering spot for the autostereoscopic display device.

[0041] Facial features that can be used in this method are for example selected from the group of nose, mouth, ears, wrinkles and eyebrows.

[0042] The method of the present invention utilizes an autostereoscopic display device. Such a device may be a device that is largely stationary in the real world during its use, such as a desktop device or a wall-mounted device. For example, an autostereoscopic display device may be a television, a (desktop) computer with a monitor, a laptop, or a cinema display system. It may also be a portable device, such as a mobile phone, a tablet, or a game console.

[0043] In the method of the present invention, the autostereoscopic display device is preferably a device including a pixel array backed by a lenticular lens capable of directing pixel outputs (i.e., light) belonging to a left-eye image specifically to the left eye of a viewer and pixel outputs (i.e., light) belonging to a right-eye image specifically to the right eye of a viewer. The combined pixel outputs of the pixels in the pixel array form the output of the entire display, i.e., the display output.

[0044] Such an autostereoscopic display device preferably also comprises a tracking system configured to determine the position of a viewer-left rendering spot and a viewer-right rendering spot relative to the autostereoscopic display device, and these obtained position data are then used to render the stereoscopic image and control the pixels.

[0045] Thus, in one embodiment, the method of the present invention comprises: a left and right rendering spot tracking system configured to track a position of a left rendering spot and a position of the right rendering spot relative to an autostereoscopic display device; a display unit configured to display a left-eye image to be seen by the left eye of the viewer and a right-eye image to be seen by the right eye of the viewer, an array of display pixel elements for generating a display output; ○ An autostereoscopic display device including a display portion comprising a lenticular device provided on the array, the lenticular device having a lenticular lens area that can direct display output from different display pixel elements to different spatial positions within the field of view of the autostereoscopic display device, thereby enabling the display of a stereoscopic image composed of left eye and right eye images.

[0046] Typically, such an autostereoscopic display device also comprises a rendering module configured to render a stereoscopic image from the 3D scene data, taking into account the position of a left rendering spot relative to the autostereoscopic display device and the position of a right rendering spot relative to the autostereoscopic display device.

[0047] Data regarding the locations of the left and right rendering spots relative to the autostereoscopic display device can also serve as input for weaving the left and right eye images onto an array of display pixel elements, i.e., for selecting the correct display pixel elements for display of both images so that the stereoscopic image is presented to the viewer as intended. Thus, displaying the rendered stereoscopic image in the method of the present invention can include weaving the left and right eye images onto the array of display pixel elements, where the weaving selecting display pixel elements to generate pixel outputs for the left eye image and for the right eye image, taking into account a position of a left rendering spot relative to the autostereoscopic display device and a position of a right rendering spot relative to the autostereoscopic display, respectively; - controlling selected display pixel elements accordingly to present a stereoscopic image to the viewer.

[0048] Typically, the method of the present invention is performed multiple times in succession, taking into account new positions of the viewer relative to the autostereoscopic display device and allowing the viewer to perceive movement relative to the displayed 3D scene. For example, the method is repeated at least 10 times, at least 100 times, at least 1,000 times, at least 10,000 times, at least 100,000 times, or at least 1 million times.

[0049] Re-rendering the stereoscopic images at each new position of the viewer relative to the 3D scene gives the viewer the impression that they are truly moving relative to the 3D scene (or that the 3D scene is truly moving relative to them, when the viewer can control the position and orientation of the 3D scene relative to the screen of an autostereoscopic display), especially if the re-rendering is done with an appropriate frequency. For a realistic viewing experience, rendering is typically done at a frequency of at least 10 times per second. Preferably, the frequency is at least 20 times per second, more preferably at least 30 times per second, and even more preferably at least 50 times per second. For example, it may be in the range of 50 to 250 times per second, 55 to 150 times per second, or 60 to 120 times per second. This may in particular be 60 Hz, 120 Hz, 144 Hz, 165 Hz, or 240 Hz, preferably the same frequency as the refresh rate of the screen itself.

[0050] As emphasized above, the method of the present invention does not take saccades into account. This essentially means that if (1) a first stereoscopic image is displayed by an autostereoscopic display and (2) a saccade occurs before a subsequent second stereoscopic image is displayed, the second stereoscopic image will not differ from the first stereoscopic image. In other words, an autostereoscopic display implementing the method of the present invention is insensitive to saccades. Furthermore, after a saccade occurs, the viewer immediately experiences a good rendering of the stereoscopic image at the point of gaze.

[0051] Thus, rendering of stereoscopic images in the methods of the present invention takes into account only the movement of the entire head (and not the movement of the eyes relative to the rest of the head). Such movement is not as rapid as saccadic pupil movement and can be predicted using methods known in the art. Input for such prediction is formed by the historical head position and orientation, as well as the velocity and / or acceleration of the left and right rendering spots relative to the autostereoscopic display. Such prediction can substantially reduce the apparent latency of the autostereoscopic display (i.e., substantially compensate for the latency of the autostereoscopic display) and improve the viewer's viewing experience. Thus, the methods of the present invention can include reducing the apparent latency of the autostereoscopic display device (i.e., compensate for the latency of the autostereoscopic display device).

[0052] In doing so, the stereoscopic image is rendered more accurately in the region within the viewer's gaze and contains less jitter compared to stereoscopic images rendered by using pupil positions rather than rendering spots according to the present invention.

[0053] In particular, the method comprises: - obtaining data relating to the velocity and / or acceleration of a left rendering spot and / or a right rendering spot relative to an autostereoscopic display device; - using the acquired data to predict the position of a left rendering spot for an autostereoscopic display device and the position of a right rendering spot for an autostereoscopic display device; - using the predicted positions for rendering a stereoscopic image from the 3D scene data and displaying it to a viewer; Optionally, weaving the left-eye image and the right-eye image into an array of display pixel elements using the predicted positions.

[0054] In some cases, it is not necessary to predict the positions of both the left and right rendering spots: since both spots have a fixed distance, it may be sufficient to predict the velocity and / or acceleration of only one of them when (1) the head orientation is known and can be taken into account, and (2) the relative positioning of the two rendering spots on the head is known.

[0055] Methods for compensating for latency in autostereoscopic display devices are known in the art, and may be done, for example, by using a history of multiple viewing positions to the left and right of the viewer and fitting a model to these viewing positions to estimate positions at future times.

[0056] The 3D scene can in principle be any imaginable 3D scene. Since the 3D scene needs to be able to be perceived from different viewing positions and with viewpoints corresponding to these viewing positions, a 3D mapping of the 3D scene needs to be available, with more than just one stereoscopic image of the 3D scene from one particular viewpoint. For this reason, the 3D scene data preferably represents an artificially created 3D scene; such a scene usually includes an extensive or complete 3D mapping of the 3D scene, which in principle allows the generation of stereoscopic images from any viewpoint (i.e., from any virtual stereo camera position).

[0057] However, in principle, it is also possible for the 3D scene data to represent a 3D scene in the real world. Such a real scene (or real scene) is, for example, recorded from multiple different camera positions. The 3D scene data resulting from recordings from different camera positions allows the real scene to be perceived at viewpoints corresponding to different viewing positions (i.e., allows the generation of stereoscopic images from more than viewpoints). Therefore, even in this situation, stereoscopic images can be generated from any viewpoint (i.e., any virtual stereoscopic camera position).

[0058] The invention further relates to an autostereoscopic display device configured to carry out a method according to the invention (i.e. one of the methods described above).

[0059] In a preferred embodiment, the autostereoscopic display device comprises a latency compensating module (or in other words an apparent latency reducing module) configured to predict the position of a left rendering spot for the autostereoscopic display device and the position of a right rendering spot for the autostereoscopic display device, and the rendering module is configured to use the predicted positions for rendering the stereoscopic image.

[0060] In another embodiment, the autostereoscopic display device is configured to determine the positions of the left and right rendering spots relative to the autostereoscopic display device by determining the position of one or more facial features of a viewer having known positions relative to both rendering spots. configured to determine the location of one or more facial features of a viewer other than a left rendering spot and a right rendering spot relative to an autostereoscopic display device; - comprising a memory comprising stored data representing the positions of left and right rendering spots relative to one or more facial features;

[0061] Autostereoscopic display devices of the present invention typically display left-eye and right-eye images on an array of display pixel elements, - selecting display pixel elements to generate pixel outputs for a left-eye image and selecting display pixel elements to generate pixel outputs for a right-eye image; a weaving module configured to weave accordingly by controlling selected display pixel elements to display a stereoscopic image to a viewer.

[0062] In a preferred embodiment, the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device are taken into account to select display pixel elements that generate pixel outputs for the left eye image and the right eye image.

[0063] An autostereoscopic display device according to the present invention may be a device that is largely stationary in the real world during its use, such as a desktop device or a wall-mounted device. For example, the autostereoscopic display device may be a television, a (desktop) computer with a monitor, a laptop, or a cinema display system. It may also be a portable device, such as a mobile phone, a tablet, or a game console.

Claims

1. 1. A method for displaying a stereoscopic image of a 3D scene to a viewer on an autostereoscopic display device, the stereoscopic image consisting of a left-eye image to be viewed by the left eye of the viewer and a right-eye image to be viewed by the right eye of the viewer; The method comprises: - providing 3D scene data representative of said 3D scene; - rendering the stereoscopic image from the 3D scene data by creating left-eye and right-eye images from the 3D scene data, taking into account a viewing position of the viewer relative to the autostereoscopic display device, allowing the viewer to experience the 3D scene from a viewpoint corresponding to their viewing position relative to the 3D scene; The method further comprises: - defining the position of a left rendering spot in the viewer's left eyeball relative to the autostereoscopic display device; - defining the position of a right rendering spot in the viewer's right eye relative to the autostereoscopic display device; - using the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device to render the stereoscopic image from the 3D scene data; - displaying the rendered stereoscopic image to the viewer; The method, wherein the left rendering spot and the right rendering spot have fixed positions relative to the viewer's entire head and are located at or no more than 5.0 mm from the center of rotation of each eye.

2. 2. The method of claim 1, wherein the left and right rendering spots are located at a distance of 3.0 mm or less, preferably 1.0 mm or less, from the center of rotation of the respective eyeball.

3. The position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device are 1) identifying one or more facial features of the viewer other than the left rendering spot and the right rendering spot; 2) determining the positions of the left and right rendering spots relative to the one or more facial features; 3) determining the position of the one or more facial features relative to the autostereoscopic display device; 4) using the positions determined in steps 2) and 3) to determine the position of the left rendering spot relative to the autostereoscopic display device and / or the position of the right rendering spot relative to the autostereoscopic display device; The method of claim 1 or 2, wherein the value is determined by

4. The autostereoscopic display device comprises: a left and right rendering spot tracking system configured to track the position of the left and right rendering spot relative to the autostereoscopic display device; a display unit configured to display a left-eye image to be seen by the left eye of the viewer and a right-eye image to be seen by the right eye of the viewer, an array of display pixel elements for generating a display output; the display portion including a lenticular device provided on the array, the lenticular device comprising lenticular lens areas that can direct display output from different display pixel elements to different spatial locations within a field of view of the autostereoscopic display device to enable the display of a stereoscopic image composed of left-eye and right-eye images; The method according to any one of claims 1 to 3, comprising:

5. The displaying of the rendered stereoscopic image includes weaving the left-eye image and the right-eye image onto the array of display pixel elements, the weaving comprising: selecting display pixel elements to generate pixel outputs for the left-eye image and selecting display pixel elements to generate pixel outputs for the right-eye image taking into account the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device, respectively; - controlling the selected display pixel elements accordingly to display the stereoscopic image.

6. 6. The method of any one of claims 1 to 5, wherein the method is repeated multiple times, for example at least 10 times, at least 100 times, at least 1,000 times or at least 10,000 times.

7. The method according to any one of claims 1 to 6, wherein the method is repeated at a frequency of at least 50 times per second, preferably at a frequency in the range of 55 to 150 times per second.

8. The method according to any one of claims 1 to 7, wherein the method comprises compensating for latency of the autostereoscopic display device.

9. The method comprises: - obtaining data relating to the velocity and / or acceleration of the left rendering spot and / or the right rendering spot relative to the autostereoscopic display device; - using the acquired data to predict the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device; - using the predicted positions to render the stereoscopic image from the 3D scene data and display it to the viewer; - optionally displaying said rendered stereoscopic image using said predicted positions; The method according to any one of claims 1 to 8, comprising:

10. The method of any one of claims 1 to 9, wherein the 3D scene data represents an artificially created 3D scene.

11. The method of any one of claims 1 to 9, wherein the 3D scene data represents a 3D scene in the real world.

12. 1. An autostereoscopic display device for displaying stereoscopic images of a 3D scene to a viewer, the autostereoscopic display device comprising: a left and right rendering spot tracking system, the position of a left rendering spot within the viewer's left eyeball relative to the autostereoscopic display device; and configured to track the position of a right rendering spot within the viewer's right eye relative to the autostereoscopic display device; the left rendering spot and the right rendering spot have fixed positions relative to the viewer's entire head and are located at or no more than 5.0 mm from the center of rotation of each eye; a display unit configured to display a left-eye image to be seen by the left eye of the viewer and a right-eye image to be seen by the right eye of the viewer, an array of display pixel elements for generating a display output; the display portion comprising a lenticular device disposed above the array, the lenticular device comprising lenticular lens areas that can direct the display output from different display pixel elements to different spatial locations within a field of view of the autostereoscopic display device to enable the display of a stereoscopic image composed of left-eye and right-eye images; a rendering module configured to render stereoscopic images from 3D scene data, taking into account the viewing position of the viewer relative to the autostereoscopic display device, to enable the viewer to experience the 3D scene from a viewpoint corresponding to their viewing position relative to the 3D scene, the rendering module is further configured to use the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device to render the stereoscopic image.

13. 13. The autostereoscopic display device of claim 12, wherein the autostereoscopic display device further comprises a latency compensation module configured to predict the position of the left rendering spot relative to the autostereoscopic display device and the position of the right rendering spot relative to the autostereoscopic display device, and the rendering module is configured to use the predicted positions for the rendering of the stereoscopic image.

14. The autostereoscopic display device comprises: configured to determine the position of one or more facial features of the viewer other than the left and right rendering spots relative to the autostereoscopic display device; An autostereoscopic display device according to claim 12 or 13, comprising a memory containing stored data representing the positions of the left and right rendering spots relative to the one or more facial features.

15. The autostereoscopic display device displays the left-eye image and the right-eye image on the array of display pixel elements; selecting display pixel elements to generate pixel outputs for the left-eye image and selecting display pixel elements to generate pixel outputs for the right-eye image taking into account the position of the left rendering spot relative to the autostereoscopic display and the position of the right rendering spot relative to the autostereoscopic display device, respectively; - by controlling the selected display pixel elements accordingly to display the stereoscopic image to the viewer; An autostereoscopic display device according to any one of claims 12 to 14, comprising a weaving module configured for weaving.

Images