Autostereoscopic display device configured to display stereoscopic images as a stereoscopic cinema

Abstract

The present invention relates to an autostereoscopic display device configured to display stereoscopic images as a stereoscopic movie and taking into account the position data of a viewer of the autostereoscopic display device, wherein the image acquisition rate and the display frame rate are matched by a matching means so that both frequencies have the same value. In this way, the relationship between the two frequencies is forced so that one does not deviate from the other over time. Furthermore, a phase difference between the image acquisition cycle and the image display cycle is adapted in accordance with the present invention, thereby reducing the time between the moment position data becomes available and the moment the position data is actually used by the intended application. Such phase difference adaptation may solve problems associated with conventional autostereoscopic display devices, such as varying latency, inaccurate prediction of viewer position, position jitter, movement of displayed virtual objects, and crosstalk as the viewer moves relative to the display.

Description

[Technical Field]

[0001] <Technical field to which the invention pertains> The present invention relates to an autostereoscopic display device configured to display stereoscopic images as in a stereoscopic cinema. The present invention further relates to a method of operating an autostereoscopic display device. [Background technology]

[0002] <Background> Autostereoscopic displays are playing an increasingly important role in virtual reality and augmented reality applications. One of their most notable features is that they allow the viewer to perceive depth in the displayed image without the need for glasses or other specialized eyewear. Furthermore, this principle works even if the viewer moves relative to the autostereoscopic display.

[0003] The key to this technology is the presence of a screen with a lenticular lens or parallax barrier placed in front of an array of pixels. The lenticular lens or parallax barrier allows the pixel output (i.e., light) to be directed in a specific spatial direction, selectively illuminating one eye only. By precisely controlling the pixels, the screen can simultaneously direct a left-eye image to the viewer's left eye and a right-eye image to the viewer's right eye. The resulting stereoscopic image provides a depth perception in which elements in the image appear to be in front of the display or farther away than the display ("behind" it).

[0004] The performance of such autostereoscopic displays is improved when the position of the viewer's eyes relative to the display is tracked: the resulting position data improves the selectivity of illumination for one eye, thereby reducing crosstalk (the undesirable phenomenon where some of the light intended for one eye "leaks" into the other eye).

[0005] Such tracking is typically performed by an eye-tracking system operating at a constant rate, typically the image capture rate of the cameras that make up part of the eye-tracking system, making eye position data available in real time to the autostereoscopic display, allowing accurate real-time adaptation of the image display to the viewer's position.

[0006] The image display itself also occurs at a certain rate, the frame rate, which is preferably the same rate at which the stereoscopic images are woven into the display by the processor. Both rates are usually set to the same value (e.g., 60Hz) so that the processor can go through its usual process of generating each image frame using the most recent available eye position.

[0007] However, in conventional autostereoscopic display devices, even if both frequencies are set to the same value, both frequencies may fluctuate relative to each other. This appears to cause latency fluctuations in certain cases. Also, if the moment of image presentation to the user cannot be estimated, the viewer's position at the moment of image presentation cannot be accurately predicted. This can cause undesirable position fluctuations or movement of the displayed virtual objects. Another undesirable effect is that the viewer may experience crosstalk when moving relative to the display. Possible causes of both frequencies fluctuating relative to each other are fluctuations in external conditions such as temperature and voltage.

[0008] Therefore, it is necessary to find ways to counteract latency, especially its variations, and other disturbing effects caused by variations in the image acquisition and image display frequencies. However, to date, no satisfactory solution has been found to do this. For example, estimating when each frame will be displayed is unattractive because it requires feedback from the hardware (GPU / display) to the application, which is not easy to implement. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0009] <Brief summary of the invention> It is therefore an object of the present invention to provide an improved autostereoscopic display device with reduced latency, in particular with constant latency, and also to provide a method of operating an autostereoscopic display device with low latency, in particular with low latency variation.

[0010] A more general object of the present invention is to improve the viewing experience of a viewer of an autostereoscopic display device, including reducing the viewer's experience of crosstalk and improving the viewer's perception of the position and / or movement of displayed virtual objects. [Means for solving the problem]

[0011] It has been found that one or more of these objectives can be achieved by introducing specific rate matching means into an autostereoscopic display device. The present invention therefore relates to an autostereoscopic display device (1) configured to display stereoscopic images as a stereoscopic cinema taking into account position data of a viewer (2) of the autostereoscopic display device (1) (or the present invention relates to an autostereoscopic display device (1) configured to display stereoscopic images as a stereoscopic cinema and thereby taking into account position data of a viewer (2) of the autostereoscopic display device (1)), wherein the autostereoscopic display device (1) a camera (3) configured to acquire images of the viewer (2) at an image acquisition rate; a viewer tracking system (4) configured to track the position of a viewer (2) relative to the autostereoscopic display device (1) using images acquired by the camera (3); a display (5) configured to display the stereoscopic image at a display frame rate as a stereoscopic movie; an image weaving unit (6) configured to weave a stereoscopic image onto a display (5) at a weaving frame rate using position data of a viewer (2) relative to the autostereoscopic display device (1) obtained by a viewer tracking system (4); - matching means (7) configured to match the image acquisition rate and the display frame rate.

[0012] The present invention further relates to a method of operating an autostereoscopic display device, the method comprising: acquiring images of a viewer of an autostereoscopic display device at an image acquisition rate; - tracking a viewer's position relative to the autostereoscopic display device using the acquired images; - weaving a stereoscopic image into a display at a weaving frame rate using viewer position data; - displaying the stereoscopic images at a display frame rate as a stereoscopic movie; - matching the image acquisition rate with the display frame rate. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 shows a schematic diagram of two timelines representing the image acquisition rate and display frame rate in a conventional autostereoscopic display device. [Figure 2] FIG. 2 shows a schematic diagram of two timelines representing the image acquisition rate and display frame rate in an autostereoscopic display device according to the invention. [Figure 3] FIG. 3 shows a schematic representation of an autostereoscopic display device (1) according to the invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] The drawings are not intended to limit the invention to the particular embodiments disclosed and described herein. Elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale, emphasis instead being placed upon clearly illustrating the principles of the invention. For example, the representation of different rates (e.g., as their periods) does not quantitatively reflect their differences.

[0015] In the context of this invention, an autostereoscopic display device means a device configured to display stereoscopic images to a viewer, simultaneously directing a left-eye image to the viewer's left eye and a right-eye image to the viewer's right eye, allowing the viewer to perceive depth without the need for glasses or other specialized eyewear. Together, the left-eye image and the right-eye image form a stereoscopic pair (i.e., a stereoscopic image).

[0016] The display portion of an autostereoscopic display device of the present invention typically includes an array of display pixel elements for generating a display output, and a view-forming arrangement associated with the array. Typically, the view-forming arrangement has a fixed connection with the pixel arrangement (i.e., they cannot move relative to each other). The view-forming arrangement can direct the display output from different display pixel elements to different spatial locations within the field of view of the autostereoscopic display device, allowing left-eye and right-eye images to be displayed simultaneously. Typically, the view-forming arrangement includes a lenticular lens or a parallax barrier.

[0017] In the context of the present invention, image capture rate refers to the frequency at which repeated image captures occur, usually expressed as image captures per second. The time that elapses between a particular image capture and the next image capture defines an image capture cycle.

[0018] Similarly, the display frame rate refers to the frequency at which repeated image displays occur, usually expressed as the number of display frames per second. The time that elapses between the display of one particular frame and the display of the next defines the image display cycle.

[0019] Similarly, the weaving frame rate refers to the frequency at which repeated image weaving occurs, usually expressed as the number of weaved images per second. The time that elapses between a particular image weaving and the next image weaving defines the image weaving cycle.

[0020] In the context of the present invention, "weaving" refers to the process by which pixels of the left-eye image of a stereoscopic pair are assigned to display pixel elements of an autostereoscopic display device that directs output to the left eye of the viewer, and pixels of the right-eye image of the stereoscopic pair are assigned to display pixel elements of a display device that directs output to the right eye of the viewer.

[0021] In the field of autostereoscopic displays, weaving is a well-known concept for generating an output image by an autostereoscopic display based on three-dimensional image input data. Terms that are considered equivalent to weaving in the prior art in this field include, for example, "interleaving," "interlacing," and "interdigitating."

[0022] In the autostereoscopic display device of the present invention, weaving is performed in an image weaving unit. In practice, this unit resides in a personal computer (which is part of the autostereoscopic display device) or in the display section of the autostereoscopic display device (the image weaving unit is then integrated into the display section). In the latter case, the weaving frame rate is usually synchronized with the display frame rate. For example, the completion of a particular image weaving automatically triggers the image display on the display section, so that the weaving frame rate is the same as the display frame rate.

[0023] In the context of this invention, "rendering" refers to the process of generating a stereoscopic image of a particular three-dimensional scene that corresponds to a particular viewpoint for that scene. Rendering can, in principle, be performed for any viewpoint of the three-dimensional scene. Subsequent display of the rendered stereoscopic image requires image weaving. In this way, a viewer can perceive the three-dimensional scene in three dimensions from a particular viewpoint (i.e., the viewpoint used for rendering).

[0024] In the context of the present invention, "viewer" means a real-world person who can consume, and in particular view, content presented by an autostereoscopic display device. Throughout this text, references to the viewer will be made in male terms such as "he," "him," and "his." This is for the sake of clarity and brevity only, and it is understood that female terms such as "she," "her," and "her" apply equally.

[0025] Like nearly all electronic devices that function through the input of acquired data, autostereoscopic display systems suffer from latency, commonly understood as the time delay between user input and system response, also known as input lag. In the context of an autostereoscopic display system configuration according to the present invention, this essentially means a delay between the viewer's head / eye movement (user input) and the corresponding adaptation of the displayed content (system response). Latency then manifests itself as inaccurate rendering of the stereoscopic image. Crosstalk can occur when the image intended for the left eye is also aimed at the viewer's right eye, and vice versa, when the image intended for the right eye is also aimed at the viewer's left eye. When latency exceeds a critical threshold, user performance and experience are impaired. This is typically related to artifacts such as the look-around effect or crosstalk. For example, displayed objects may not be perceived in their correct positions, at least temporarily.

[0026] The inventors have realized that setting the image acquisition rate and the display frame rate to the same value does not guarantee that both frequencies will actually match perfectly. This manifests as a time offset between image acquisition events and frame display events, which varies over time. This means that the number of image acquisition cycles will not equal the number of image display cycles within a given time period. For example, there may be a difference of several cycles over the course of a minute. Furthermore, the magnitude of such a difference is not constant over time.

[0027] This is thought to be due to external conditions such as temperature and voltage fluctuations, which are known to affect the output frequency of crystal oscillators used to set the frequency of electronic equipment (known as frequency stability).

[0028] The effect of slight variations in both frequencies is shown in Figure 1. It shows events occurring at the weaving frame rate in the top timeline, representing the moment when an available eye position is used by the weaver. Typically, the weaving frame rate is the same as the display frame rate of the autostereoscopic display device. The bottom timeline shows events occurring at the image acquisition rate, representing the moment of image acquisition and the moment when an eye position becomes available to the weaver. The frequency difference is visualized in Figure 1 by the length of each cycle, with the weaving cycle being longer than the image acquisition cycle.

[0029] In Figure 1, we can see that time elapses before an available eye position is used by the weaver, which is the latency (denoted as wt#1, wt#2, etc.). We can also see that the latency varies from cycle to cycle due to differences in the length of the weaving cycle and the length of the image acquisition cycle. These lengths result from the difference between the weaving frame rate (which is the same as the display frame rate) and the image acquisition rate. Figure 1 also shows that some eye positions are not used by the weaver because the latency becomes so long that new eye positions become available during that time. This latency variation causes problems for conventional autostereoscopic display devices, including latency fluctuations, inaccurate prediction of the viewer's position, position jitter, movement of displayed virtual objects, and crosstalk when the viewer moves relative to the display.

[0030] The present invention solves this problem by introducing a matching means into the autostereoscopic display device, which forces the relationship between both frequencies so that one does not deviate from the other over time. Here, the display indicates to the camera its display frame rate, and the camera adopts that rate as its image capture rate. Therefore, the matching means is typically configured to trigger the camera at a rate equal to the display frame rate.

[0031] One way to match the image acquisition rate with the display frame rate is to use the display's vsync signal. The vsync signal indicates the start of a frame by lighting the first pixel of the frame. This signal can also be used to trigger the camera's shutter, creating a fixed relationship between the start of a new frame and image acquisition. Therefore, the matching means of the autostereoscopic display device of the present invention can include communication means, such as electrical, electronic, or optical communication, between one display and the other camera. This can be implemented by connecting one end of a wire to the display's vsync output pin and the other end of the wire to the camera's input means. The electrical communication can also be wireless.

[0032] FIG. 3 shows a schematic diagram of an autostereoscopic display device (1) according to the present invention, in which a viewer (2) is viewing a stereoscopic movie displayed by the autostereoscopic display device (1). The viewer (2)'s position relative to the autostereoscopic display device (1) is tracked by a viewer tracking system (4), which receives images of the viewer (2) captured by a camera (3) at an image acquisition rate. An image weaving unit (6) is present, which receives image data from an image data storage unit (8) and position data of the viewer (2) from the viewer tracking system (4). The image weaving unit (6) weaves image data for display on a display unit (5) at an weaving frame rate. To this end, the display unit (5) receives the weaved images from the image weaving unit (6) and displays them to the viewer (2) as a stereoscopic movie. There is also a matching means (7) for matching the image acquisition rate and the display frame rate during the display of the stereoscopic movie. In this particular embodiment, the matching means (7) is a wire between the vsync output pin of the display unit (5) and the input means of the camera (3). In the autostereoscopic display device (1) of Figure 1, the viewer tracking system (4), the image weaving unit (6) and the image data storage unit (8) are all part of a processor (9), for example a personal computer.

[0033] When the two rates are matched, another property emerges: a phase difference. Both rates have a period, which in the context of this invention is identified as a cycle. The image acquisition cycle can be offset relative to the image display cycle, meaning that image acquisition occurs a certain time after image display occurs. This delay in the occurrence of image acquisition relative to image display is known as an offset or phase difference. The mere (or sheer) existence of communication between the display and the camera causes both rates to match, resulting in an offset that is neither selected nor adapted.

[0034] However, it is possible to adapt the phase difference by delaying the arrival of the signal from the display to the camera. To this end, the matching means comprises adaptation means configured to adapt the phase difference between the image acquisition cycle and the image display cycle to a desired value. Such adaptation means may comprise, for example, a signal delay unit.

[0035] FIG. 2 illustrates the effect of the present invention when 1) both frequencies are matched and 2) a phase difference is adapted. The matched frequencies are visualized in FIG. 2 by the length of each cycle, with the weaving cycle being the same length as the image acquisition cycle. The phase difference is adapted so that the latency is very short (e.g., to arrive at a reduced constant latency), but not too short (to prevent the eye position from becoming available too late). Thus, this embodiment illustrates that the method of the present invention reduces the latency of an autostereoscopic display device.

[0036] In the autostereoscopic display device of the present invention, the viewer tracking system is typically an eye-tracking system.

[0037] The method of the present invention may include adapting a phase difference between an image acquisition cycle and an image display cycle to a desired value, where the image acquisition cycle represents one period of the image acquisition rate and the image display cycle represents one period of the display frame rate.

[0038] Phase difference adaptation can be advantageously used to reduce the latency of autostereoscopic display devices. Such latency is typically a first time point at which viewer position data relative to the autostereoscopic display device is available; and - a second point in time, later than the first point in time, at which the viewer's position data relative to the autostereoscopic display device is actually used by an application running on the autostereoscopic display device for which the position data is intended.

[0039] Such applications are, for example, applications that include an image weaving unit configured to weave a stereoscopic image into a display unit, or applications that include an image rendering unit configured to render a stereoscopic image for display on an autostereoscopic display device.

[0040] The latency reduction described above results in reduced latency for autostereoscopic display devices.

[0041] In a method or autostereoscopic display device according to the invention, the image acquisition rate and the display frame rate have the same value, preferably selected from the group of 60 Hz, 90 Hz, 100 Hz, 120 Hz, 144 Hz, 165 Hz and 200 Hz.

[0042] In one embodiment, the autostereoscopic display device includes a processor and a memory, the memory including computer-executable code that, when executed by the processor, causes the processor to perform a method for operating an autostereoscopic display device as described above.

[0043] The present invention further relates to a computer-readable medium containing transitory or non-transitory data representing instructions for causing a processor system to perform the method for operating an autostereoscopic display device as described above.

[0044] The invention further relates to software arranged, when executed on a computer, to carry out the method for operating an autostereoscopic display device as described above.

Claims

1. An autostereoscopic display device (1) configured to display stereoscopic images as a stereoscopic cinema taking into account position data of a viewer (2) of said autostereoscopic display device (1), a camera (3) configured to acquire images of said viewer (2) at an image acquisition rate; - a viewer tracking system (4) configured to track the position of the viewer (2) relative to the autostereoscopic display device (1) using images obtained by the camera (3); a display (5) adapted to display said stereoscopic images as a stereoscopic movie at a display frame rate; an image weaving unit (6) configured to weave a stereoscopic image for display on said display (5) at a weaving frame rate using position data of said viewer (2) obtained by said viewer tracking system (4); - matching means (7) configured to match said image acquisition rate and said display frame rate; An autostereoscopic display device (1) comprising:

2. 2. An autostereoscopic display device (1) according to claim 1, wherein the matching means (7) is configured to trigger the cameras (3) at a rate equal to the display frame rate.

3. 3. An autostereoscopic display device (1) according to claim 1 or 2, wherein the matching means (7) comprises an electrical, electronic or optical communication between the display (5) on the one hand and the camera (3) on the other hand.

4. 4. An autostereoscopic display device (1) according to any one of claims 1 to 3, wherein the matching means (7) comprises adaptation means configured to adapt a phase difference between an image acquisition cycle and an image display cycle to a desired value, an image acquisition cycle representing one period of the image acquisition rate and an image display cycle representing one period of the display frame rate.

5. 5. An autostereoscopic display device (1) according to claim 4, wherein the adaptation means comprises a signal delay unit.

6. The autostereoscopic display device (1) according to any one of claims 1 to 5, wherein the viewer tracking system (4) is an eye tracking system.

7. An autostereoscopic display device (1) according to any one of claims 1 to 6, wherein the interwoven frame rate is the same as the display frame rate.

8. 1. A method of operating an autostereoscopic display device, comprising: - acquiring images of a viewer of said autostereoscopic display device at an image acquisition rate; - using the captured images to track the viewer's position relative to the autostereoscopic display device; - weaving a stereoscopic image for a display portion of the autostereoscopic display device at a weaving frame rate using the viewer's position data; - displaying said stereoscopic images by said display unit at a display frame rate as a stereoscopic movie; - matching the image acquisition rate and the display frame rate by forcing a relationship between the two frequencies such that one rate does not deviate from the other rate over time; A method comprising:

9. 9. The method of claim 8, wherein the method includes adapting a phase difference between an image acquisition cycle and an image display cycle to a desired value, wherein an image acquisition cycle represents one period of the image acquisition rate and an image display cycle represents one period of the display frame rate.

10. The phase difference is adapted to reduce latency, the latency being: the first time that the position data of the viewer relative to an autostereoscopic display device becomes available; and 10. The method of claim 9, wherein the amount of time that elapses between a second point in time, later than the first point in time, at which the position data of the viewer relative to the autostereoscopic display device is actually used by an application running on the autostereoscopic display device.

11. The method of claim 10 , wherein the application includes an image weaver configured to weave a stereoscopic image for the display.

12. The method of claim 10 or 11, wherein the application includes an image renderer configured to render stereoscopic images for display on the autostereoscopic display device.

13. The method of any one of claims 10 to 12, wherein the latency reduction reduces the latency of the autostereoscopic display device.

14. The method of any one of claims 8 to 13, wherein the interwoven frame rate is the same rate as the display frame rate.

15. 15. A method according to any one of claims 8 to 14 or an autostereoscopic display device according to any one of claims 1 to 7, wherein both the image acquisition rate and the display frame rate have the same value, the value being selected from the group of 60 Hz, 90 Hz, 100 Hz, 120 Hz, 144 Hz, 165 Hz and 200 Hz.

16. A computer readable medium containing transitory or non-transitory data representing instructions for operating a processor system to perform the method of any one of claims 8 to 15.

17. Software adapted to carry out the method of any one of claims 8 to 15 when run on a computer.

Images