Scaling 3D content for display on autostereoscopic display devices

Abstract

The invention relates to a method of driving a screen of an autostereoscopic display device to present a three-dimensional image to an observer, the method comprising the steps of scaling the three-dimensional image taking into account 1) the viewing distance from the observer to the screen of the eyes, and 2) the recording distance of the object. In this way, the three-dimensional image of one or more particular objects in a scene can be scaled so that the scene and the objects therein become a realistic part of the observer's real environment. It is also possible to display a background image on the screen, the background image being scaled to real dimensions.

Description

[Technical field]

[0001] <Field of the Invention> The present invention relates to a method for driving the screen of an autostereoscopic display device to present a three-dimensional image to a viewer present within the field of view of the screen. [Background technology]

[0002] <Background> Autostereoscopic displays have attracted a great deal of attention in the past two decades. One of their most outstanding features is that they allow the observer to perceive a three-dimensional image without a dedicated eyewear device, even as the observer moves relative to the display. The key to this technology is the presence of a screen that includes a lenticular lens or a parallax barrier. This allows an autostereoscopic display to simultaneously direct a left-eye image to the observer's left eye and a right-eye image to the observer's right eye. The resulting three-dimensional image provides a depth perception, and elements in the image may appear in front of the display or further away than the display ("behind" the display). The absence of a dedicated eyewear device allows the observer to experience being physically present in the real world, while at the same time, an autostereoscopic display forms a virtual window into another world, i.e. a truly believable virtual world that is also three-dimensional.

[0003] However, a drawback of such virtual windows is that the dimensions of the displayed content are often not perceived by the observer as matching the dimensions of the recorded real scene. For example, when an observer views an item (e.g., an object or person) contained within the field of view of the virtual window (i.e., the item visible "through" the virtual window), the dimensions of this displayed item typically do not match the dimensions of the real item when viewed from the same distance through a real window. Of course, this does not match even when multiple items are present.

[0004] A further discrepancy with reality is that as the viewer moves towards or away from the virtual window, the size of the displayed items does not change accordingly.

[0005] Thus, known autostereoscopic displays appear to have certain shortcomings with respect to realistically presenting three-dimensionally recorded items to a viewer, particularly in settings where the autostereoscopic display acts as a virtual window. Summary of the Invention [Problem to be solved by the invention]

[0006] It is therefore an object of the present invention to provide a method for improving an observer's experience when viewing an autostereoscopic display, for example when the observer "looks through a virtual window" to see another object or person. In particular, it is an object of the present invention to provide a method for improving an observer's experience in a teleconference. [Means for solving the problem]

[0007] It has been found that one or more of these objectives can be achieved by appropriate scaling of the three-dimensional content to be displayed by an autostereoscopic display.

[0008] The present invention therefore relates to a method for driving a screen of an autostereoscopic display device to present to an observer present within the field of view of the screen a three-dimensional image of an object in a scene, comprising: - providing a three-dimensional recording of objects in a scene using a stereo camera; - displaying the three-dimensional recording of the object as a three-dimensional image on a screen, The method further comprises the steps of: - the viewing distance from the observer's eyes to the screen, - a recording distance from the object to the stereo camera, and a scaled step taking into account DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Throughout this specification and the claims, the terms "three-dimensional image" and "autostereoscopic image" are used interchangeably and refer to the same type of image. It is recognized that an autostereoscopic image is said not strictly the same as a three-dimensional image. An autostereoscopic image is an image that is perceived as three-dimensional only by the observer because it is composed of a left image presented to the observer's left eye and a right image presented to the observer's right eye.

[0010] In the context of the present invention, the term "left image" refers to an image displayed by an autostereoscopic display device intended for the left eye. Correspondingly, the term "right image" refers to an image displayed by an autostereoscopic intended for the right eye. It is understood herein that in practice there is always a (very) small portion of light that "leaks" into the other eye, this effect being known as crosstalk. However, observers are not always aware of this and may still rate their three-dimensional viewing experience as satisfactory.

[0011] In the context of the present invention, a three-dimensional recording is meant to include information representative of a three-dimensional visual image in that, when input to an autostereoscopic display device in a format processable by this device, it can be used to display such a three-dimensional image. A three-dimensional recording (e.g., of a scene, person or object in the real world) is a recording or live stream of a real scene or object captured by a stereo camera and including information about the three-dimensionality of the scene or object. It can be stored in a memory portion associated with the device so that it can be displayed on demand, or it can be displayed as a live video captured by a stereo camera associated with the autostereoscopic display device (such a camera can be remote from the device and can be configured to capture a scene or object in a different environment, for example).

[0012] In the context of the present invention, the term "stereo camera" means a camera capable of providing a three-dimensional recording of a real scene or object, from which a stereoscopic or three-dimensional image can be recorded. For the purposes of the present invention, a stereo camera is meant to include a stereoscopic camera and a plenoptic camera. Furthermore, the term stereo camera can include a plurality of (stereo) cameras, such stereo cameras collectively providing the capabilities of such a stereo camera.

[0013] In the context of the present invention, the term "viewer" refers to a person who consumes the content presented to him according to the method of the present invention. In addition to seeing a three-dimensional image, the viewer may also experience other sensory stimuli, such as sound or tactile stimuli. However, for convenience, such a person is thus referred to as an "observer", but it is understood that at the same time he may also be, for example, a "listener".

[0014] Throughout the text, references to the observer will be made by masculine terms such as "he," "him," or "his." This is for purposes of clarity and brevity only, it being understood that feminine terms such as "she" and "her" apply equally.

[0015] It is understood that the method of the present invention utilizes an autostereoscopic display device. This is typically a device (or apparatus) that is mostly stationary in the real world during its use, such as a desktop device or a wall-mounted device. For example, an autostereoscopic display device is a television, a (desktop) computer with a monitor, a laptop, or a cinema display system. However, it may also be a portable device, such as a mobile phone, an in-car display, a tablet, or a game console, so that the observer can move (freely) in the real world together with the autostereoscopic display device.

[0016] Autostereoscopic display devices are known in the art, for example from WO2013120785A2. The main components of an autostereoscopic display device used in the method of the present invention are typically an eye tracking system, a screen, a processing unit and optional audio means.

[0017] The eye tracking system comprises means for tracking the position of an observer's eyes relative to the autostereoscopic display device and is in electronic communication with the processing unit. The eye positions are required for accurately weaving the left-eye and right-eye images into an array of pixels so that each image hits the intended eye, even as the observer moves relative to the screen of the autostereoscopic display device.

[0018] The viewing distance from the observer's eyes to the screen is typically also obtained from an eye tracking system, alternatively there may be a separate means for determining this viewing distance.

[0019] The recorded distance of the observer relative to the stereo camera can be obtained by using an eye-tracking system or a different system associated with or integrated into the stereo camera.

[0020] The screen includes means for displaying a three-dimensional image to an observer whose gaze is tracked by an eye-tracking system, such means comprising an array of pixels for generating a display output, and a parallax barrier or lenticular lens mounted over the array for directing a left image to the observer's left eye and a right image to the observer's right eye.

[0021] The processing unit is configured to receive the 3D recording and to drive the screen taking into account the data acquired by the eye tracking system. Thus, a key component of the processing unit is the so-called "weaver", which weaves the left and right images into an array of pixels, thereby determining which pixels should generate a pixel output corresponding to each image. In this way, a 3D image can be displayed to the observer at a specific position.

[0022] The processing unit is typically also configured to perform scaling of the three-dimensional image in accordance with the methods of the present invention.

[0023] The optional audio means includes means for playing sound to the observer, for example the audio means comprises one or more devices selected from the group of stereo loudspeakers, loudspeaker arrays, headphones, and ear buds.

[0024] An autostereoscopic display device used in the method of the invention typically comprises a receiver for receiving a three-dimensional recording of a scene or object in the real world. This also includes receiving a live video stream of a scene or object in the real world. Audio recordings and / or audio streams may also be received by such a receiver. The transfer of the video and / or audio recordings may for example be via a wireless connection or a telecommunications line.

[0025] An autostereoscopic display device used in the method of the present invention may include a memory for storing a three-dimensional recording of a scene or object in the real world.

[0026] In the scene (or scenario) in which the actual 3D recording takes place (or has been made), one or more devices (or apparatuses) are also present. Such devices are a stereo camera and a means for determining the recording distance from the object to be recorded to the stereo camera. The latter device may be integrated in the stereo camera. Optionally, an audio recording device and / or a lighting device are present in the scene of the 3D recording.

[0027] The devices in the scene of the 3D recording are typically not physically part of the autostereoscopic display apparatus, but may be associated with the autostereoscopic display apparatus, for example via telecommunications lines.

[0028] The inventors have realised that known autostereoscopic display devices may, in some aspects, display three-dimensional images just like normal television displays, i.e. the field of view of the means for recording the image essentially fits the screen of the device, i.e. the edges of the recorded scene also form the edges of the displayed image of the scene. In some cases the displayed content may be scaled for convenience, but this usually does not follow the apparent size. Furthermore, different objects at different recording distances require different degrees of scaling, which is not possible when the image is scaled as a whole. These drawbacks with respect to scaling appeared to have the effect that the displayed content is not experienced as a natural part of the observer's real environment. In other words, such displayed content detracts from the observer's immersive experience.

[0029] It has now been found that immersion is increased when one or more particular objects in a scene are recorded and measured separately, and when their display as a three-dimensional image includes a scaling that takes into account not only the recording distance of the object relative to the stereoscopic camera, but also the viewing distance of the observer's eyes relative to the screen. In this way, the dimensions of the object perceived from the displayed three-dimensional image can be adjusted to match the dimensions of the object as it is perceived in the real world (the screen acts as a virtual window).

[0030] The perception of the size of such an object has everything to do with the distance at which the object is observed, not only on the viewing side but also on the recording side. In the display of a real object on a screen there are two types of observation distance. The first type is the recording distance, i.e. the distance between the (stereo) camera and the object. The second type is the viewing distance, i.e. the distance between the observer's eyes and the screen on which the image of the object is displayed.

[0031] At a short viewing distance, the angle of view (field of view) is large and the object covers a large part of the surface of the observer's retina or the image sensor of the camera. Conversely, at a long viewing distance, the angle of view (field of view) is small and the object covers a part of the surface of the observer's retina or the image sensor of the camera. For a realistic perception at the viewing side of the dimensions occurring in the real world (i.e., at the recording side), the viewing angle at the recording side must be the same as that at the viewing side. Given a certain recording distance and viewing distance (depending on the time and circumstances), the most preferred way to arrive at a matching viewing angle is to scale (in real time) the image at the viewing side (i.e., on the screen). This is reflected in the method of the present invention. Using general mathematical principles, a person skilled in the art can calculate, without any inventive effort, how much scaling is applied when the recording distance and the viewing distance are known.

[0032] For the purposes of the present invention, the object to be scaled in this way is typically at a distance from the stereo camera that is within the range where stereoscopic vision is possible. Typically, this is at a distance of less than 25 m, preferably less than 10 m, more preferably less than 7 m, even more preferably less than 5 m. For example, it is less than 4 m, less than 3 m, or less than 2 m.

[0033] The preferred way to achieve such scaling is by defining the apparent size of the object to be recorded in the real world and converting this to the representation of the image of the recorded object on the screen (or translating). For this purpose, an apparent size is also defined for the object as it is displayed on the screen. The term "apparent size" of an object is meant to indicate the angular distance from one side of the object to the other side of the object. This can be thought of as the angular displacement that the eye or camera must rotate to see from one side to the other.

[0034] Within this specification, the apparent real-world size of an object is the angular size perceived by a person in the real world whose distance to the object is the distance from the stereo cameras to the object recorded by the stereo cameras (i.e., the person's viewpoint is the viewpoint of the stereo cameras). Correspondingly, the apparent display size of an object is the angular size perceived by an observer of an autostereoscopic display device when a three-dimensional image is displayed on a screen.

[0035] In the method of the present invention, the three-dimensional image is preferably scaled so that the apparent display size matches the apparent real-world size. Such matching means that the three-dimensional image and the real object are perceived as having the same size (e.g., both are perceived as having the same height, both are perceived as having the same width). In this way, it can be determined how much the three-dimensional image must be scaled to have a matching apparent size. As a result, the scaled three-dimensional image becomes a realistic part of the observer's real environment.

[0036] However, the apparent display size may be set to a particular desired percentage of the apparent real-world size (i.e., a percentage other than 100%) so that both apparent sizes match. This may be the case when the realistic dimensions are less important and / or when it is undesirable for the object to cover an exceptionally large or small portion of the screen. For example, the desired percentage may be in the range of 50-150%, particularly in the range of 80-120%, more particularly in the range of 95-105%, and even more particularly in the range of 99-101%.

[0037] Equivalent to determining the apparent size of an object (displayed or real) is determining the apparent size of a feature of such object. For example, when the object is a knife, a feature might be its blade, or when the object is a human head, it might be the distance between the eyes.

[0038] Thus, the method of the present invention comprises: a) determining a recorded distance from an object to a stereo camera; b) determining the viewing distance from the observer's eyes to the screen; c) defining features of the object, the features being included in the three-dimensional image; d) determining the apparent real-world size of the feature, which is the angular size of the feature when viewed in the real world from the recorded distance obtained under a); e) determining the apparent display size of the feature, which is the angular size of the feature when viewed as a three-dimensional image on a screen from the viewing distance obtained under b); f) scaling the three-dimensional image by adjusting the apparent display size determined under e) in the three-dimensional image to a desired percentage (or proportion) of the apparent real-world size obtained under d); g) fitting the 3D image scaled in step f) to the screen by cropping the 3D image when the 3D image is larger than the screen.

[0039] The method of the present invention is not limited to scaling one object. It is also possible to scale several objects and display them simultaneously. Thus, the method includes: -There are multiple objects in the scene, - the method is performed for each object of the plurality of objects; A method in which several three-dimensional images are displayed simultaneously on a screen, each image being scaled independently of the others, taking into account the viewing distance from the observer's eyes to the screen and the recording distance from the object to the stereo camera.

[0040] In another expression, the method comprises: - providing a first 3D recording of a first object in a scene and a second 3D recording of a second object in the scene, both recordings being obtained by a stereo camera; - simultaneously displaying the first three-dimensional record on a screen as a first three-dimensional image and the second three-dimensional record on a screen as a second three-dimensional image, The first and second three-dimensional images are - the viewing distance from the observer's eyes to the screen, - the recording distance from each object to the stereo camera is taken into account (i.e. for scaling the first 3D image, the recording distance from the first object to the stereo camera is taken into account, and for scaling the second 3D image, the recording distance from the second object to the stereo camera is taken into account), and are scaled independently of each other to the desired extent.

[0041] It should be noted that displaying separate recordings of one or more objects (simultaneously) on a screen usually requires that the objects have been previously segmented from each other and from the background, since the recordings usually include the complete scene of which the one or more objects form part. This applies even more when the objects are scaled independently of each other. Segmentation of objects from the rest of the scene to which they belong is known in the art. Any such segmentation is therefore a standard procedure for those skilled in the art.

[0042] The scene in which the actual 3D recording is performed may contain, in addition to one or more objects, a background. In the context of the present invention, background means any part of the scene that is at a distance from the stereo camera in the range of 7 m to infinity, 10 m to infinity, 15 m to infinity, 25 m to infinity, or 50 m to infinity. Due to its relative remoteness from the stereo camera, the background may be scaled as a whole. The method of the present invention therefore comprises: - providing a recording of the background of the scene, obtained by a camera or a stereo camera; - simultaneously displaying 1) a recording of a background image as a background image on a screen, and 2) a three-dimensional recording of an object as a three-dimensional foreground image on the screen, or a three-dimensional recording of a plurality of objects as a three-dimensional foreground image on the screen, The above method is such that the background image is - the viewing distance from the observer's eyes to the screen, - the field of view of the camera or stereo camera, and a scaling step taking into account.

[0043] In a preferred embodiment, the method comprises the steps of: a) determining a viewing distance from an observer's eyes to a screen; b) determining the field of view of a camera or a stereo camera; c) virtually positioning the background image within the display plane of the screen such that the background image plane coincides with the display plane of the screen; d) determining to what extent the background image must be virtually scaled from the viewing distance obtained under a) to enable the observer to virtually see a background image having a field of view that corresponds to a desired percentage of the field of view of the camera or stereo camera obtained under b); e) scaling the background image to the extent obtained under d) and displaying it on a screen; f) fitting the background image scaled in step e) to the screen by cropping the background image where the background image is larger than the screen.

[0044] Virtually positioning the background image in the display plane of the screen means that the background image is imaginarily or virtually displayed in a plane coinciding with the display plane of the screen, in a setting where the observer is in front of the screen at a distance obtained under a). Here, the dimensions (e.g. length and width) of the virtually displayed background image are not limited to the screen dimensions. Given the viewing distance from the observer to the screen and the field of view in which the background image needs to be seen (which is a desired percentage of the field of view of the camera or stereo camera obtained under a), the degree to which the background image must be virtually scaled is determined. A person skilled in the art can use general mathematical principles and without any inventive effort calculate the degree of such scaling when the viewing distance from the observer to the screen and the (desired percentage of) the field of view of the camera or stereo camera are known.

[0045] This virtual scaling measure is then applied to the actual display of the background image. As a result, if the background image is larger than the screen, it will be cropped where it is larger than the screen. If it is smaller, it may be determined that the background image is missing in some locations, or a different image may be displayed where the background image is missing.

[0046] In the method of the present invention, the background image is preferably scaled so that the field of view from which the observer sees the background image matches the field of view of the camera or stereo camera. Such matching means that the background image and the real background are perceived as having the same size (e.g., both are perceived as having the same height, both are perceived as having the same width). This setting (i.e., matching field of view) can be used to determine how much the background image must be scaled. As a result, the scaled background image becomes a realistic part of the observer's real environment.

[0047] However, the field of view from which the observer sees the background may be set to a particular desired percentage of the field of view of the camera or stereo camera (i.e., a percentage other than 100% where both fields of view coincide). This may be the case when real-world dimensions are less important and / or when it is desired that the background be enlarged or reduced. For example, the desired percentage may be in the range of 50-150%, specifically in the range of 80-120%, more specifically in the range of 95-105%, and even more specifically in the range of 99-101%.

[0048] The method of the invention is advantageously applied in teleconferencing, enabling another person remote from the observer to communicate with the observer, and vice versa. In such a setup, the other person and the observer preferably both have means by which the method of the invention can be carried out, i.e. the observer's autostereoscopic display device is equipped with a stereo camera which operates together with the other person's autostereoscopic display device according to the method of the invention. This provides a three-dimensional recording of the observer so that the other person can see a three-dimensional image of the observer, and vice versa.

[0049] However, it is also possible that the stereo camera of the observer displays a recording of the observer to the observer (and not to another person). Thus, in the method of the invention, the stereo camera can be configured to record the observer in the field of view of the screen, so that the screen can display a recording of the observer to the observer. In such a case, the observer sees a three-dimensional image of himself, and therefore the observer's autostereoscopic display device can be considered as a virtual mirror. For this purpose, the displayed three-dimensional image is preferably mirrored.

[0050] In one embodiment, scaling according to the method of the present invention may be repeated one or more times during the display of a three-dimensional image to account for changes in the viewer's eye position relative to the screen.

[0051] In another embodiment, the entire method is repeated one or more times to take into account possible changes in the scene, such as 1) changes in the recording distance from the object to the stereo camera, and 2) possible changes in the position of the observer's eyes relative to the screen. For example, at a rate of at least 1 repetition per second, at least 10 repetitions per second, at least 25 repetitions per second, at least 40 repetitions per second, or at least 50 repetitions per second. In particular, the rate is in the range of 27-33 repetitions per second, in the range of 57-63 repetitions per second, or in the range of 87-93 repetitions per second. A high rate produces a high frequency of successive images, which is perceived by the observer as a movie. A high rate also means that there is a more timely accommodation to changes in the viewing distance from the observer's eyes to the screen, and to changes in the recording distance from the object to the stereo camera. For example, when the observer makes a fast movement relative to the autostereoscopic display device and / or when the object makes a fast movement relative to the stereo camera, these movements are taken into account in a timely manner when the method of the present invention is performed at a high repetition rate.

[0052] In the method of the invention, the 3D recording may be stored on a data carrier such as a memory stick or a hard disk. The autostereoscopic display device then retrieves the 3D recording from such a storage device. Alternatively, the autostereoscopic display device retrieves the 3D recording "live" without reading it out of memory. Thus, in the method of the invention, the 3D recording may be contained in a memory portion associated with the autostereoscopic display device, or may be a live video stream originating from the scene where the actual 3D recording is made.

Claims

1. 1. A method of driving a screen of an autostereoscopic display device to present a three-dimensional image of an object in a scene to an observer present within a field of view of the screen, the method comprising: - providing a three-dimensional record of the objects in the scene, obtained by a stereo camera; - displaying said three-dimensional recording of said object as a three-dimensional image on said screen, The three-dimensional image of the object the viewing distance from the observer's eyes to the screen; - the recorded distance from the object to the stereo camera, and is scaled to the desired extent.

2. The method comprises: a) determining a recorded distance from the object to the stereo camera; b) determining a viewing distance from the observer's eyes to the screen; c) defining features of the object, the features being included in the three-dimensional image; d) determining the apparent real-world size of the feature, which is the angular size of the feature when viewed in the real world from the recorded distance obtained under a); e) determining the apparent display size of the feature, which is the angular size of the feature when viewed as a three-dimensional image on the screen from the viewing distance obtained under b); f) scaling the three-dimensional image by adjusting the apparent display size determined under e) in the three-dimensional image to a desired percentage of the apparent real-world size obtained under d); g) fitting the three-dimensional image scaled in step f) to the screen by cropping the three-dimensional image where the three-dimensional image is larger than the screen.

3. The method of claim 1 , wherein the three-dimensional recording is contained in a memory portion associated with the autostereoscopic display device, or the three-dimensional recording is a live video stream.

4. - there are multiple objects in the scene, - the method is performed for each object of the plurality of objects; The method of claim 1, wherein a plurality of three-dimensional images are displayed simultaneously on the screen, each image being scaled independently of the others.

5. The method comprises: - providing a first three-dimensional recording of a first object in a scene and a second three-dimensional recording of a second object in said scene, both recordings being obtained by a stereo camera; - simultaneously displaying said first three-dimensional record as a first three-dimensional image on said screen and said second three-dimensional record as a second three-dimensional image on said screen, The first and second three-dimensional images are the viewing distance from the observer's eyes to the screen; The method of claim 1, wherein the distances are scaled to take into account the recorded distances from the respective objects to the stereo cameras.

6. 2. The method of claim 1, wherein a background in the scene is present at a distance ranging from 10 m to infinity, specifically 25 m to infinity, from the stereo camera, said method comprising: - providing a recording of the background in the scene, obtained by a camera or a stereo camera; - simultaneously displaying 1) the recording of the background as a background image on the screen, and 2) the three-dimensional recording of the object as a three-dimensional foreground image on the screen, or of multiple objects as three-dimensional foreground images on the screen, The background image is the viewing distance from the observer's eyes to the screen; the field of view of said camera or stereo camera, How it is scaled to take into account

7. The method comprises: a) determining a viewing distance from the observer's eyes to the screen; b) determining the field of view of the camera or stereo camera; c) virtually positioning the background image within a display plane of the screen such that the background image plane coincides with a display plane of the screen; d) determining to what extent the background image must be virtually scaled from the viewing distance obtained under a) to enable the observer to virtually see the background image having a field of view corresponding to a desired percentage of the field of view of the camera or stereo camera obtained under b); e) scaling the background image to the extent obtained under d) and displaying it on the screen; f) fitting the background image scaled in step e) to the screen by cropping the background image where the background image is larger than the screen.

8. The method of claim 1 , wherein the object is a human head.

9. The method of claim 1 , wherein the method is used in a teleconferencing session.

10. The method of claim 1 , wherein the stereo camera is configured to record the observer within the field of view of the screen.

11. The method of claim 10 , wherein the displayed three-dimensional image is mirrored.

12. 3. The method of claim 2, wherein the desired percentage is in the range of 50-150%, specifically in the range of 80-120%, more specifically in the range of 95-105%, and even more specifically in the range of 99-101%.

13. The method of claim 1 , wherein the scaling is repeated one or more times during the display of the three-dimensional image to account for changes in the position of the observer's eyes relative to the screen.

14. The method of claim 1 , wherein the method is repeated one or more times to take into account changes in the recording distance from the object to the stereo camera and / or changes in the position of the observer's eyes relative to the screen.

15. The method of claim 1 , wherein the autostereoscopic display device is selected from the group of a television, a desktop computer, a laptop, a cinema display system, a mobile phone, an in-car display, a tablet, and a game console.