Autostereoscopic display device presenting 3D views and 3D-audio

Abstract

The present invention relates to a method for simultaneously presenting three-dimensional images and three-dimensional sound to a viewer of an autostereoscopic display device, comprising adapting the three-dimensional images and the three-dimensional sound to the position of the viewer's eyes and ears relative to the autostereoscopic display device. In this way, images and sounds of one and the same virtual scene can be presented to the viewer simultaneously as if they were from a real scene, both of which are experienced in three dimensions and according to the viewer's movements within the scene. Furthermore, this is also achieved for the viewer's interaction with the scene, for example with virtual objects displayed on the autostereoscopic display device. In this way, the viewer is provided with an increased immersion in the scene, which can find useful applications, for example, in teleconferencing and computer games.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for simultaneously presenting a viewer with three-dimensional images and three-dimensional sound displayed by an autostereoscopic display device. [Background technology]

[0002] Autostereoscopic displays have attracted a great deal of attention in the last two decades. One of their most striking features is that they allow the viewer to perceive three-dimensional images without a dedicated eyewear device, even when the viewer moves relative to the display. The key to this technology is the presence of an eye tracker in combination with a screen equipped with a lenticular lens or a parallax barrier. This ensures that the autostereoscopic display simultaneously orients the left eye image to the viewer's left eye and the right eye image to the viewer's right eye. The resulting three-dimensional image may appear to be in front of the display or further away than the display ("behind" the display). This technology also allows the viewer to experience perspective changes in the displayed content as he moves relative to the display, for example to view the surroundings of some virtual object. The viewer is thus able to perceive the three-dimensional reality without the need for unnatural peripherals such as headsets, head-mounted displays, or glasses. This allows the viewer to directly see and hear important parts of the real environment (such as his own hands and feet, the floor, other people interacting with the viewer, etc.), and the autostereoscopic display forms a window into the virtual world. In such a setting, the viewer experiences that he is physically present in the real world, part of which is a truly believable three-dimensional virtual environment. Such experiences are often called "immersive experiences".

[0003] To improve the viewer's experience of such a virtual environment, autostereoscopic displays can be extended with other features, such as features that allow the viewer to interact with the displayed scene, or features that stimulate senses other than vision, in particular hearing. However, the combination of sound and 3D display has some drawbacks.

[0004] Part of the desired virtual audiovisual experience is that when a viewer of a scene (who in this case is also a listener) moves his / her head relative to the autostereoscopic display, this is reflected not only in the visual viewpoint on the scene, but also in the way that sounds are perceived. For example, when a particular visible virtual item generates a sound, the viewer should be able to recognize that the sound is actually emanating from the particular item. Furthermore, when turning one ear towards the item, the viewer should perceive a stronger sound with that ear and less with the more distant ear, just as in real life.

[0005] However, this is not the case for conventional autostereoscopic displays, where the sound is optimized for a viewer in a specific position, usually in the center in front of the display. This is conventionally achieved by placing multiple speakers (e.g., 5 or 7) at specific positions around the viewer (colloquially known as a home cinema system). Another conventional effort to improve the sound perception involves the use of headphones or earphones. However, wearing such unnatural peripherals is undesirable (as also highlighted above) since they interfere with the viewer's audiovisual experience.

[0006] Another drawback experienced with conventional autostereoscopic displays relates to the viewer's interaction with the scene: for example, when a viewer moves (e.g., by swiping) a virtual object that they perceive in front of the display, the sound associated with such action does not correspond properly to the sound of the displayed image or movie.

[0007] Thus, the immersive experience provided to the viewer by conventional means appears to be deficient, particularly with respect to providing a real-world sense of the virtual environment. Summary of the Invention

[0008] It is therefore an object of the present invention to provide means and / or methods for improving the audiovisual experience provided by an autostereoscopic display, in particular to provide means and / or methods for achieving a more realistic sound sensation when the viewer moves his / her head relative to the autostereoscopic display and / or when the viewer interacts with the displayed scene.

[0009] A further objective is to provide the viewer with a more immersive experience than is possible with conventional autostereoscopic displays.

[0010] It has been found that one or more of these objectives may be achieved by adapting not only the view but also the sound to the viewer's head position relative to the autostereoscopic display.

[0011] Thus, in a first aspect, the present invention provides a method for providing a viewer of an autostereoscopic display device with: - a three-dimensional image displayed by a naked-eye stereoscopic display device; - A method for simultaneously presenting three-dimensional sound generated by a naked-eye stereoscopic display device, The method involves adapting three-dimensional images and three-dimensional sound to the positions of a viewer's eyes and ears, respectively, relative to an autostereoscopic display device.

[0012] In a second aspect, the present invention relates to an autostereoscopic display device, comprising: an eye tracking system for determining the position of a viewer's eyes relative to the display device; - means for determining the position of a viewer's ears relative to the display device; a display unit configured to display a three-dimensional image consisting of a left image presented to a viewer's left eye and a right image presented to the viewer's right eye; - audio means configured to generate three-dimensional sound; a processor configured to generate image data for driving the display and to generate sound data for driving the audio means; The processor includes an audiovisual application; - two virtual cameras configured to generate left and right image data for driving a display, taking into account the position of a viewer's eyes relative to the display device; - two virtual microphones configured to generate left and right sound data for driving the audio means, taking into account the position of the viewer's ears relative to the display device. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 shows a schematic top view of two setups where a viewer sits in front of an autostereoscopic display device. [Diagram 2] FIG. 1 is a first schematic diagram of an architecture for carrying out the method of the present invention. [Diagram 3] FIG. 2 is a second schematic diagram of an architecture for carrying out the method of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, dimensions of some elements in the figures may be exaggerated relative to other elements to help improve understanding of various exemplary embodiments of the present invention. In particular, the relative dimensions of a viewer, an autostereoscopic display device, and a virtual object in a particular setting cannot be derived from the figures. Furthermore, terms such as "first", "second", etc. in this specification, if any, are generally used to distinguish between similar elements and not necessarily to describe a sequential or chronological order.

[0015] Throughout the description and claims, the terms "three-dimensional image" and "autostereoscopic display image" are used interchangeably and refer to the same type of image. It is recognized here that an autostereoscopic display image is said not strictly the same as a three-dimensional image. An autostereoscopic display image is an image that is only perceived as three-dimensional by a viewer, since it is composed of a left image to be presented to the viewer's left eye and a right image to be presented to the viewer's right eye. The same applies to the terms "three-dimensional view" and "autostereoscopic view", which refer to the same type of view throughout this description.

[0016] In the context of the present invention, the term "left image" refers to the image displayed by the autostereoscopic display device for the left eye. Correspondingly, the term "right image" refers to the image displayed by the autostereoscopic display device for the right eye. The left and right images may also be displayed in areas close to each eye, as long as it does not hit the other eye. However, in practice, there is always a small (or very small) portion of light that "leaks" into the other eye (crosstalk), but viewers are not always aware of this and still consider their -3D viewing experience satisfactory.

[0017] In the context of the present invention, the term "image data" refers to image information that is input in some way to the autostereoscopic display device of the method of the present invention and converted into a format that can be processed by this device. It may be recorded in a memory portion associated with the device or captured by a camera associated with the device (such a camera may be a camera remote from the device that can capture images of the scene around the person participating in the video conference in different environments). The image data includes information that represents a visible image in that the image data can be used to generate a two-dimensional or three-dimensional image on the device. The left image data and the right image data are specific forms of image data that fall under the description of image data above. In particular, the left image data and the right image data include information that represents a visible left image or right image, respectively. A three-dimensional image is typically generated on the autostereoscopic display device using the left image data and the corresponding right image data.

[0018] In the context of the present invention, the term "sound data" refers to sound information that is input in some way to the autostereoscopic display device of the method of the present invention and converted into a format processable by this device. It may be recorded in a memory portion associated with the device or captured by a microphone associated with the device (such a microphone may be a microphone remote from the device that can capture the sound of the scene around the person participating in the video conference in a different environment). The sound data includes information representing an audible sound in that the sound data can be used by the device to generate two-dimensional or three-dimensional sound. The left sound data and the right sound data are specific forms of sound data that fall under the description of sound data above. In particular, the left sound data and the right sound data include information representing an audible left sound or right sound, respectively. The three-dimensional sound is reproduced by the autostereoscopic display device using the left sound data and the corresponding right sound data.

[0019] In the context of the present invention, the term "viewer" refers to a person who consumes the presented content generated by the method of the present invention. Such a person consumes not only three-dimensional images but also three-dimensional sound. For convenience, the person is called a "viewer", but it is understood that he is also a "listener". Throughout the text, references to the viewer are made by masculine words such as "he", "him" or "his". This is for the purposes of clarity and brevity only, it being understood that feminine words such as "she" and "her" apply equally.

[0020] In the context of the present invention, "three-dimensional sound" refers to the way humans experience sound in the real world, in everyday life. Sound reaches the ear from all directions and from different distances, and undergoes modifications in the pinna (ear) and inside the human head. These factors contribute to the three-dimensional auditory image that humans hear. In the context of the present invention, three-dimensional sound may be the actual sound in the environment in which humans live, or it may be a simulation thereof by technological means that gives humans the perception of the actual sound.

[0021] In the context of the present invention, "three-dimensional sound" refers to recordings of sound waves, e.g., live-recorded, streamed, or artificially created, from which three-dimensional sound may be generated. As a result, different sounds are perceived from different three-dimensional positions, even if the sounds are generated from just two speakers. Thus, a three-dimensional sound system can simulate realistic auditory events by convincing the brain that the sounds originate not from speakers or headphones, but from any point in three-dimensional space.

[0022] 3D audio may be live recorded 3D audio, or it may be computer generated using sound processing algorithms to modify pre-recorded sounds or sound effects as they move through the virtual space and wrap around a virtual head within it. In the case of live recorded 3D audio, the recording is called "binaural audio" recording. This is a method of recording sound that uses two microphones representing both ears of the listener. This is also known as dummy head recording, where for example a mannequin head is fitted with a microphone in each ear. Binaural audio is therefore the term used to describe audio content that can generate a 3D audio image in a way that humans hear sounds exactly as they do in the real world.

[0023] Although binaural audio is originally intended for playback using headphones (as a result of technical limitations), recent new technological advances have allowed for a suitable conversion via loudspeakers, where they do not have to surround the listener. This requires an array of two or more loudspeakers, typically four or more loudspeakers, driven by crosstalk cancellation units. Particular developments concern the adaptability of the crosstalk cancellation to the position of the listener, so that the listener is not limited to being in the so-called sweet spot of the sound field (see, for example, WO2017158338A1).

[0024] The method of the present invention utilizes an autostereoscopic display device, which is typically a device that is mostly stationary in the real world during its use, such as a desktop device or a wall-mounted device. For example, the autostereoscopic display device is a television, a (desktop) computer with a monitor, a laptop, or a cinema display system. However, it may also be a portable device, such as a mobile phone, a tablet, or a game console, which allows the viewer to move (freely) in the real world together with the autostereoscopic display device.

[0025] Autostereoscopic display devices are known in the art, for example from WO2013120785A2. The main components of an autostereoscopic display device are an eye tracking system, a processor, a display unit and audio means.

[0026] The eye-tracking system comprises a means for tracking the position of a viewer's eyes relative to an autostereoscopic display device and is in electronic communication with a processor. Its main components are 1) a camera aimed at the field of view of the display portion, and 2) an eye-tracker that processes the data generated by the camera.

[0027] The display portion comprises means for displaying a three dimensional image to a viewer whose eyes are tracked by an eye tracking system, such means comprising an array of pixels for generating a display output and a parallax barrier or lenticular lens provided over the array for directing a left image to the viewer's left eye and a right image to the viewer's right eye.

[0028] The audio means includes means for reproducing three-dimensional sound to a viewer, for example audio means includes items such as stereo speakers, a speaker array, headphones or earphones.

[0029] The processor is configured to take into account the data acquired by the eye-tracking system to generate left and right image data and use it to drive the display unit. Thus, a key component of the processor is the so-called "weaver", which weaves the left and right images into an array of pixels, thereby determining which pixels generate pixel outputs corresponding to the respective images. In this way, a three-dimensional image can be displayed from the left and right image data for a viewer at a specific position.

[0030] The processor is also configured to generate left and right sound data to generate three dimensional sound, and to drive the audio means taking into account data acquired by the eye (and optionally ear) tracking system.

[0031] An optional component of the processor is a so-called "crosstalk cancellation unit", which is used when the audio means do not include headphones, earphones, or other devices worn on the ears. It prevents sounds directed to the left ear from being perceived by the right ear, and vice versa.

[0032] In the method of the present invention, the autostereoscopic display device may further comprise additional features such as audiovisual applications, an ear tracking system, a system for tracking body parts other than the eyes and ears, a system for tracking objects associated with the viewer, and means for providing multiple autostereoscopic views (described in further detail below).

[0033] In the method of the present invention, the 3D image and the 3D sound are presented to the viewer simultaneously, so that not only the 3D image is adapted to the viewer's position, but also the 3D sound is adapted to the viewer's position. In particular, the 3D image is adapted to the viewer's eye position and the 3D sound is adapted to the viewer's ear position. The viewer's position here means the position relative to the autostereoscopic display device.

[0034] Adapting the 3D image to the viewer's eye position means that the perspective of the 3D image is modified according to the viewer's position relative to the autostereoscopic display device. In this way, the viewer can look around the object realistically (the so-called "look-around effect").

[0035] Similarly, adapting 3D sound to the viewer's eye position means that the 3D sound is modified according to the viewer's position relative to the autostereoscopic display device (this is also called the "hear-around effect").

[0036] The situation in which 3D images and 3D sounds are adapted to the viewer's position is illustrated in FIG. 1, which shows top views of two settings (A and B), where a viewer (1) sits in front of an autostereoscopic display device (2) equipped with a camera (5) and audio means (4) represented by an array of audio speakers. In setting A, the viewer (1) sees a virtual audiovisual object (3) in front of him / her to the right, which has a fixed position in the environment in front of the autostereoscopic display device. In setting B, the viewer has changed his / her position, so that he / she sees the same virtual object (3) in front of him / her to the left. During the change in the viewer's position, the auditory and visual perception of this virtual object (3) is adapted to the viewer's position. For example, the sound generated by the audio means (4) is perceived as coming from a virtual object (3) that is in front of the viewer's right in setting A and in front of the viewer's left in setting B (i.e. the viewer perceives a hear-around effect). At the same time, the view of the virtual object (3) provided by the autostereoscopic display (2) changes in the sense that some parts that were visible in setting A become invisible in setting B, and other parts only become visible for the first time in setting B (i.e. the viewer perceives a look-around effect). Figure 1 thus shows that the method of the invention allows the virtual content to be adapted to the viewer's position so that it is visually and aurally experienced as real. For completeness, it is noted that the same effect is perceived when, during the movement of the virtual object (3) relative to the virtual viewer (1), the viewer (1) remains stationary relative to the autostereoscopic display (2).

[0037] It must be emphasized that although the invention concerns the simultaneous presentation of a three-dimensional image and three-dimensional sound to a viewer, the three-dimensional sound or at least a part thereof is not necessarily associated with the three-dimensional image or any part thereof. Virtual objects invisible to the viewer can generate sounds perceived by the viewer, such as a virtual bug flying above the viewer's head, or a virtual clock ticking behind the viewer. Such objects are by definition invisible to the viewer, since they do not exist between the viewer and the displayed part, even if the viewer turns his / her head towards them. Nevertheless, when the viewer moves his / her head relative to the display, he has a different perception of the sound of the bug or clock. At the same time, the user perceives a changing perspective of the displayed three-dimensional image. That is, the area in which the three-dimensional image is presented to the viewer may be smaller than the area in which the three-dimensional sound is presented to the viewer.

[0038] In the method of the present invention, the autostereoscopic display device typically comprises: an eye tracking system for determining the position of a viewer's eyes relative to the display device; - means for determining the position of a viewer's ears relative to the display device; a display unit configured to display a three-dimensional image consisting of a left image presented to a viewer's left eye and a right image presented to the viewer's right eye; - audio means configured to generate three-dimensional sound; a processor configured to generate left and right image data for driving the display, and to generate left and right sound data for driving the sound means.

[0039] Such methods typically include: a) determining viewing positions of a viewer's eyes relative to an autostereoscopic display device using an eye tracking system; b) generating left image data and right image data for the display portion taking into account the position of the viewer's eyes relative to the autostereoscopic display device; c) determining the listening positions of the viewer's ears relative to the autostereoscopic display device using an eye tracking system and / or an optional ear tracking system; d) generating left and right sound data for the audio means, taking into account the position of the viewer's ears relative to the display device; e) driving a display unit and an audio means using the image data and the sound data, respectively, thereby simultaneously presenting three-dimensional images and three-dimensional sound to a viewer.

[0040] Typically, method steps a) to e) are repeated multiple times, for example at a repetition rate of at least 1 time per second, at least 10 times per second, at least 25 times per second, at least 40 times per second, or at least 50 times per second. In particular, the rate is in the range of 27 to 33 times per second, in the range of 57 to 63 times per second, or in the range of 87 to 93 times per second. A higher rate produces a continuous image with a higher frequency, which is perceived by the viewer as a movie. A higher rate also means that the three-dimensional image and the three-dimensional sound are more accurately adapted to the viewer's position. For example, when the viewer makes fast movements relative to the autostereoscopic display device, these movements are taken into account in time when the method of the present invention is performed at a high repetition rate.

[0041] In another embodiment, the present invention relates to a method of driving an autostereoscopic display device, the method comprising: - a three-dimensional image displayed by a naked-eye stereoscopic display device; - A method for simultaneously presenting three-dimensional sound generated by a naked-eye stereoscopic display device, The 3D images and 3D sound are adapted to the positions of the viewer's eyes and ears, respectively, relative to the autostereoscopic display device.

[0042] In one embodiment, the processor of the autostereoscopic display device includes an audiovisual application configured to generate image data and sound data from which a three-dimensional image and a three-dimensional sound are generated, respectively (the image data consists of left image data and right image data, and the sound data consists of left sound data and right sound data). The audiovisual application performs the actual adaptation of the three-dimensional image and the three-dimensional sound to the viewer's position (more specifically, the eye position and the ear position, respectively) relative to the autostereoscopic display device. This allows the viewer to simultaneously perceive the "look-around effect" and the "hear-around effect". The position data generated by the eye tracking system serves as an input to perform this function, optionally by using a separate ear tracking system.

[0043] Audiovisual applications typically combine a virtual three-dimensional environment with a real environment in front of (typically within) the display unit. This is done to provide the position of the viewer (a real entity) relative to the virtual three-dimensional environment, in particular the position of the viewer's eyes and ears, using at least position data on the viewer's eyes and ears obtained by executing the method of the invention. In other words, there is a virtual three-dimensional environment that is complemented with some real-world items that are (at least) the viewer's eyes and ears. Furthermore, the viewer's eyes and ears have a known position relative to the virtual three-dimensional environment. Other elements present in the real world, such as other body parts of the viewer (e.g. fingers or feet) or objects such as a stylus pen or pencil, may also be "added" to the virtual three-dimensional environment. Of course, this also occurs due to the known positions of such body parts or objects relative to the virtual three-dimensional environment.

[0044] The audiovisual application includes two virtual cameras and two virtual microphones, all of which have positions within a virtual three-dimensional environment, each of the two virtual cameras having a position at a viewer's eye, each of the two virtual microphones having a position at a viewer's ear, and the two virtual cameras generating (rendering) left and right image data and the two virtual microphones generating left and right sound data.

[0045] Input for these virtual cameras and virtual microphones is typically provided by an audiovisual memory (e.g., related libraries and support programs associated with an audiovisual application). The audiovisual memory typically includes a representation of the three-dimensional environment to be presented to the viewer and can be modified by input from the real world, such as user actions. Such actions can be, for example, pressing a virtual button located in the virtual three-dimensional environment, or hitting a flying bug in the virtual three-dimensional environment (a button or bug presented on the display portion and perceived to be in front of the display portion). Actions can also include, for example, the use of a stylus pen to press a button or to perform three-dimensional drawing.

[0046] By wearing virtual cameras and virtual microphones on their heads, the viewer can navigate in real time in the virtual 3D environment, on the one hand by moving themselves relative to the autostereoscopic display, and on the other hand by moving the virtual 3D environment relative to themselves, for example by using a mouse, a joystick, or keyboard shortcuts. The user can also manipulate the virtual 3D environment by pressing buttons or hitting bugs, which actions may cause the autostereoscopic display device to play sounds corresponding to such actions.

[0047] Virtual cameras are then defined as points in the virtual 3D environment at given positions and orientations (6 degrees of freedom). They can be moved and / or reoriented therein, thus defining the position, orientation and field of view for generating left and right image data.

[0048] Similar considerations apply for virtual microphones, since they are also defined as points in the virtual 3D environment that can be moved and / or reoriented: they define how sounds are perceived at their particular position and orientation in the virtual 3D environment, and generate left and right sound data accordingly.

[0049] As mentioned above, the input for the virtual camera and virtual microphone is usually formed by an audiovisual memory. It may comprise a database of elements in the virtual world, whose position and orientation with respect to the viewer is known in real time thanks to eye-tracking and ear-tracking systems. In addition to their position and orientation, other properties of the elements may also be known, such as surface properties, mechanical behavior (for example applied when the element collides), sound production, sound reflection and light reflection.

[0050] The input for the virtual camera and virtual microphone may be provided, for example, by an audiovisual live stream of a remote scene. Such a case finds advantageous application in video conferencing, where a viewer interacts with another person in a different (typically remote) scene with images and sounds from the respective scene. The position-matched three-dimensional images and sounds that each person experiences from the other person's scene make their interaction feel more natural than can be achieved with conventional teleconferencing means.

[0051] Thus, in the method of the present invention, the 3D images and 3D sound may be contained in a memory unit associated with the autostereoscopic display device or provided by live streaming of a particular scene.

[0052] An architecture on which the method of the invention can be implemented is shown diagrammatically in Fig. 2, which shows a top view of a scene in which a viewer (1) is positioned in front of a display unit (2) displaying virtual objects (3). Audio means (4) are also provided to the viewer (1) and are represented by an array of audio speakers (4a) or a set of headphones (4b). Furthermore, a camera (5) is directed towards the scene in front of the display unit (2) containing the viewer (1). The camera (5) is designed to capture data relating to the three dimensions of the scene. An audiovisual data provider (7) is present to provide an input of audiovisual data. A processor (6) is operatively connected to the display unit (2), the audio means (4), the camera (5) and the audiovisual data provider (7). The processor (6) processes the data received from the camera (5) together with the data received from the audiovisual data provider (7) to generate image data as input to the display unit (2) and sound data as input to the audio means (4).

[0053] FIG. 3 shows the architecture of FIG. 2 in detail, showing the different components of the processor and how they interact. The processor comprises a viewer tracker configured to generate position data for the eyes and ears, and eventually other body parts or objects. To do so, the viewer tracker receives the data captured by the camera. A weaver, a crosstalk cancellation unit and an audiovisual application are also part of the processor and obtain the position data from the viewer tracker. The audiovisual application comprises two virtual cameras and two virtual microphones. The virtual cameras generate left and right image data as input to the weaver. The virtual microphones generate left and right sound data as input to the audio means. If an audio speaker is used as the audio means, the left and right sound data are first processed by the crosstalk cancellation unit. If headphones are used as the audio means, such a unit is not necessary and the headphones can directly receive the left and right sound data.

[0054] In the method of the present invention, the position of the viewer's eyes relative to the autostereoscopic display device is obtained by tracking the viewer's eyes using an eye tracking system that typically comprises: 1) a camera pointed at the viewer in the field of view of the display unit of the autostereoscopic display device; and 2) an eye tracker that processes the data generated by the camera. The obtained eye position data allows the left and right images to be accurately projected to the viewer's respective eyes, and these data also allow the display of the three-dimensional image to be adapted to the viewer's position, so that the viewer can perceive a look-around effect.

[0055] A camera (also known in the art as a "range camera") in an eye-tracking system is generally a light sensor device that can gather information about the distance between one or more specific points in a scene (on the one hand) and the camera (on the other hand). The points are typically parts of the head or parts of the eye that are in the field of view of the camera. The device can operate according to one of several different techniques. Typically these techniques are stereo triangulation, light sheet triangulation, structured light and time-of-flight. There may be multiple cameras in an eye-tracking system.

[0056] The camera may in particular be a device capable of capturing visual or infrared images of a scene and gathering information about the distance between one or more particular points in the scene and the camera, which may be part of the head or part of the eye, typically by stereo triangulation.

[0057] The eye tracker in the eye tracking system that processes the camera data is configured to actually calculate the position of the eyes (specifically the eye sockets, more specifically the pupils) relative to the camera or another part fixed to the camera, such as a display part. If the camera is capable of capturing visual or infrared images of the scene, the eye tracker is typically also able to recognize the eyes in the captured visual or infrared images.

[0058] The eye tracking system used in the methods of the present invention may be capable of tracking the eyes of at least one individual, and may also be configured to track the eyes of multiple individuals, in which case facial recognition may be used to distinguish and identify individuals.

[0059] The eye tracking system may be configured to track body parts other than the eyes, such as hands, feet, arms, legs or ears. In addition to body parts of the viewer, the system may be configured to track objects, in particular objects controlled by the viewer. Such objects may be "random" objects, such as random pens or balls, or "adapted" objects that can be detected by the tracking system, for example by being equipped with a recognizable marker, such as a stylus pen.

[0060] When it is desired to describe the system's ability to track one or more body parts of the viewer in addition to the eyes, and / or final objects associated with the viewer, the terms "eye tracking system" and "eye tracker" are generalized in this description to "viewer tracking system" and "viewer tracker," respectively. Thus, the term "viewer tracking system" is meant to include at least an eye tracking system.

[0061] The position of the viewer's ears can be determined in different ways in the method of the invention: it can for example be derived from the position of the eyes, and since this position is known via an eye tracking system, there is no need to apply an ear tracking system.

[0062] However, it is also possible to specifically track the ears by an ear tracking system (if so, such a system is typically part of a viewer tracking system as defined above). However, this is not effective in all cases, since the ears may be out of the field of view of the cameras for ear tracking. In such cases, a combination of using eye tracking information and ear tracking may be effective. Thus, in the method of the present invention, the means for determining the position of the viewer's ears relative to the autostereoscopic display device may comprise an ear tracking system and / or a means for deriving the ear positions from the eye positions determined by the eye tracking system.

[0063] In the method of the invention, the three-dimensional sound is produced by audio means. These means can be divided into three groups: 1) those worn by the viewer and therefore have a substantially fixed position relative to the viewer's ears, 2) those attached to the autostereoscopic display device or have another stationary position with a substantially fixed position relative to the autostereoscopic display device, and 3) those that have a stationary position relative to the real world but not relative to the viewer and the autostereoscopic display device. In the first case, the three-dimensional sound is typically experienced through headphones or earphones (thus, in such cases, the audio means comprises headphones or earphones). In the second and third cases, the three-dimensional sound is typically experienced through remote speakers or speaker arrays (thus, in such cases, the audio means comprises speakers or speaker arrays).

[0064] In the second and third cases (i.e. with remote speakers), the autostereoscopic display device, in particular its processor, is required to have a crosstalk cancellation unit. Such a unit is configured to separate the sounds produced by the audio means between the two ears of the viewer (i.e. - to prevent sounds intended for the left ear from being perceived by the right ear and vice versa). This occurs otherwise because remote speakers are usually not selective in reaching one particular ear. On the other hand, headphones or earphones have this capability by design. Such a unit is also typically configured to cancel any distortion occurring between the ear and the speaker, since it compensates for sound distortion due to the asymmetric morphology of the pinnae and their internal structure (also called Head Related Transfer Function (HRTF)). An effective crosstalk cancellation unit creates virtual headphones at the listener's ears, allowing a set of binaural signals to be delivered cleanly and without distortion. Thus, the crosstalk cancellation unit is capable of 1) separating the sounds reproduced by the audio means between the two ears of the viewer, and 2) canceling distortion occurring between the ear and the speaker.

[0065] The autostereoscopic display device used in the method of the present invention is typically capable of displaying two views to provide a single autostereoscopic view. However, it may also be possible to display more than two views, thereby allowing multiple autostereoscopic views and therefore multiple simultaneous viewers to participate. In that case, the autostereoscopic display device may be configured to distinguish or identify individual persons by facial recognition.

[0066] For three-dimensional sound in the case of multiple viewers, the audio means are preferably worn on the ears by the viewers, as this prevents them from hearing sounds intended for other viewers.

[0067] The method of the invention demonstrates that images and sounds of one and the same virtual scene can be presented to the viewer simultaneously as if it were a real scene. Both are experienced three-dimensionally and according to the viewer's movements in the scene. It is also possible that the scene is not a real scene (in the sense that it is not a recording of a real scene) but an artificially created scene such as an animation. The invention is therefore also advantageously applied to the playing of computer games. This immersion of the viewer in the (real or artificial) scene forms a significant improvement over known methods for achieving immersion in a scene.

[0068] The method of the invention can be particularly advantageously applied in situations where a viewer interacts with a displayed scene and when such interaction has the intention of generating a particular sound. For example, the viewer may press a virtual button or break a virtual glass, for example with a hand or a foot. The sounds associated with such events may be presented to the viewer in a way that closely mimics the way the viewer actually perceives such events. To this end, the method of the invention may be supplemented with an additional tracking system for tracking further body parts, as already mentioned above. Such further body parts are for example body parts other than the eyes or ears, for example the hands or feet. These advantageous features of the invention make it an attractive means for playing computer games.

[0069] Further body parts are not necessarily in the field of view of the screen: for example, the feet may interact with a part of the scene that is not visible to the viewer, at least not at the moment of interaction. To this end, such body parts may be tracked by a tracking system, preferably having a camera remote from the autostereoscopic display device, so that they are better positioned in the field of view of the camera.

[0070] The method of the present invention may be supplemented with an additional tracking system for tracking an object associated with the viewer. A viewer holding such an object, for example a pencil or stylus pen, can thereby make a three-dimensional drawing that can be viewed on his or her autostereoscopic display device. Alternatively, in a teleconferencing setting with another person, the drawing can be viewed by this other person. The object may be used for various other purposes, such as pressing a virtual button, swiping a virtual object, or creating a three-dimensional selection space.

[0071] Therefore, in the method of the present invention, the naked eye stereoscopic display device may be equipped with an additional tracking system for tracking an object or a further body part, such as a hand or a foot, which is a body part other than an eye or an ear, and the method comprises: 1) determining the position and / or velocity of the object or further body part relative to the autostereoscopic display device by using an additional tracking system; 2) determining whether a specific 3D image and / or a specific 3D sound should be associated with the determined velocity and / or position of the object or further body part; 3) generating image data for the display unit and / or sound data for the audio means, taking into account the determination of step 2).

[0072] In this embodiment, typically the position and / or velocity of the object or further body part relative to the autostereoscopic display device, as determined by the additional tracking system in step 1), is also used. From these position and / or velocity data and from the position of the viewer's eyes and ears relative to the autostereoscopic display device, the position of the object or further body part relative to the viewer's eyes and ears can be derived. This position forms the input for the generation of image data and sound data. For example, if a virtual three-dimensional line drawing is drawn with a real pencil, the line drawn with the pencil needs to be displayed three-dimensionally (this is the image associated with the pencil, represented by the image data). Or, when a virtual bell is triggered by a real finger, the sound of the bell needs to be played in three dimensions (this is the sound associated with the finger, represented by the sound data). Thus, in these two examples, the image data for the display part and the sound data for the sound means are generated respectively, taking into account the respective positions of the pencil or finger relative to the viewer's eyes and ears. The image data then represents a three-dimensional image associated with the pencil, and the sound data then represents a three-dimensional sound associated with the tracked object or further body part.

[0073] The object may in particular be a pointing device such as a pen, pencil or stylus pen. Here, a "stylus pen" (or simply a "stylus") refers to a pen-shaped instrument that functions as a pointing device and / or drawing device, similar to a mouse or touchpad. To this end, the stylus pen can be detected by the autostereoscopic display device, in particular by a tracking system designed therefor, such as a viewer tracking system.

[0074] This ability to adapt images and sounds to the viewer's interaction with virtual content (including objects controlled by the viewer) makes the virtual experience provided by the method of the present invention even more immersive for the viewer.

[0075] The method of the invention can be used in particular in teleconferencing, which means the live exchange of information between several people and machines that are separated from each other but linked by a telecommunications system.

[0076] The present invention further relates to an autostereoscopic display device, an eye tracking system for determining the position of a viewer's eyes relative to the display device; - means for determining the position of a viewer's ears relative to the display device; a display unit configured to display a three-dimensional image consisting of a left image presented to a viewer's left eye and a right image presented to the viewer's right eye; - audio means configured to generate three-dimensional sound; a processor configured to generate image data for driving the display and to generate sound data for driving the audio means; The processor includes an audiovisual application; - two virtual cameras configured to generate left and right image data for driving a display, taking into account the position of a viewer's eyes relative to the display device; - two virtual microphones configured to generate left and right sound data for driving the audio means, taking into account the position of the viewer's ears relative to the display device.

[0077] In one embodiment, the audio means of the autostereoscopic display device of the present invention comprises headphones or earphones having a variable position relative to the autostereoscopic display device.

[0078] In a further embodiment, the means for determining the position of the viewer's ears relative to the naked eye stereoscopic display device comprises means for deriving the ear positions from the eye positions as determined using an ear tracking system and / or an eye tracking system.

[0079] In a further embodiment, the autostereoscopic display device of the present invention comprises 1) a speaker having a substantially fixed position relative to the autostereoscopic display device, and 2) a crosstalk cancellation unit which separates the sound generated by the audio means between the viewer's ears and cancels distortion occurring between the ears and the speaker.

[0080] In a further embodiment, the autostereoscopic display device of the present invention comprises an additional tracking system for tracking an object or a further body part, the further body part being a body part other than an eye or an ear, such as a hand or a foot. The autostereoscopic display device then typically: - reproducing three-dimensional sound associated with an object or body part, taking into account the position of the viewer's ears relative to the display device; - configured to display a three-dimensional image associated with an object or body part, taking into account the position of the viewer's eyes relative to the display device.

[0081] The invention further relates to a kit comprising such an autostereoscopic display device and a stylus pen, the additional tracking system being adapted to track the stylus pen. - Reproducing three-dimensional sounds associated with the stylus pen, taking into account the position of the viewer's ears relative to the display device; - configured to display a three-dimensional image associated with the stylus pen, taking into account the position of the viewer's eyes relative to the display device.

Claims

1. For viewers of the naked-eye 3D display device, - a three-dimensional image displayed by an autostereoscopic display device; - a three-dimensional sound generated by the naked eye stereoscopic display device, The method includes adapting the three-dimensional image and the three-dimensional sound to the positions of the viewer's eyes and ears, respectively, relative to an autostereoscopic display device.

2. The naked eye three-dimensional display device, an eye tracking system for determining the position of the viewer's eyes relative to the autostereoscopic display device; - means for determining the position of the viewer's ears relative to the autostereoscopic display device; a display unit adapted to display a three-dimensional image; - audio means arranged to generate three-dimensional sound; a processor configured to generate image data for driving said display and to generate sound data for driving said audio means; The method further comprising: a) determining a position of the viewer's eyes relative to the autostereoscopic display device using the eye tracking system; b) generating image data for the display unit taking into account the position of the viewer's eyes relative to the naked eye stereoscopic display device; c) determining the position of the viewer's ears relative to the autostereoscopic display device using the eye tracking system and / or ear tracking system; d) generating sound data for the audio means taking into account the position of the viewer's ears relative to the naked eye stereoscopic display device; e) driving the display unit and the audio means, respectively, using the image data and the sound data to present a three-dimensional image and a three-dimensional sound to a viewer at the same time; and f) optionally repeating steps a) to f) after a certain time interval.

3. 3. The method of claim 2, wherein the means for determining the position of the viewer's ears relative to the autostereoscopic display device comprises an ear tracking system, and / or means for deriving the ear positions from eye positions determined by the eye tracking system.

4. the processor further comprises an audiovisual application comprising two virtual cameras and two virtual microphones; the audiovisual application combines the virtual three-dimensional environment with a real environment in front of the display, at least to the extent that the virtual three-dimensional environment includes the eyes and ears of the viewer, the positions of the eyes and ears of the viewer relative to the virtual three-dimensional environment being known, - two virtual cameras each having an eye position of the viewer and jointly generating said image data from audiovisual input data representative of said virtual three-dimensional environment; A method according to claim 2 or 3, wherein the two virtual microphones each have an ear position of the viewer and jointly generate the sound data from audiovisual input data representative of the virtual three-dimensional environment.

5. - said audio means comprises a speaker having a substantially fixed position relative to said autostereoscopic display device; A method according to any one of claims 2 to 4, wherein the autostereoscopic display device comprises a crosstalk cancellation unit which separates the sounds generated by the speakers between the viewer's ears and cancels distortions occurring between the ears and the speakers.

6. A method according to any one of claims 2 to 4, wherein the audio means comprises headphones or earphones having a substantially fixed position relative to the listener's ears.

7. 7. The method according to any one of claims 1 to 6, wherein steps a) to e) are repeated at a repetition rate of at least 1 time per second, preferably at a repetition rate of at least 10 times per second, more preferably at least 50 times per second.

8. A method according to any preceding claim, wherein at least a portion of the three-dimensional sound is not associated with the three-dimensional image or any part thereof.

9. The method according to any one of claims 1 to 8, wherein the autostereoscopic display device is selected from the group of a television, a desktop computer with a monitor, a laptop, a cinema display system, a mobile phone, a tablet and a gaming console.

10. The naked eye stereoscopic display device comprises an additional tracking system for tracking an object or a further body part, the further body part being a body part other than an eye or an ear, and the method further comprises: 1) determining the position and / or velocity of the object or further body part relative to the autostereoscopic display device by using an additional tracking system; 2) determining whether a particular 3D image and / or a particular 3D sound should be associated with said object or a further body part; 3) taking into account the determination of step 2), generating the image data for the display unit and / or the sound data for the audio means using the position and / or velocity of the object or further body part relative to the naked eye stereoscopic display device determined by the additional tracking system in step 1).

11. The method of claim 10 , wherein the object is a pointing device such as a pen, pencil or stylus.

12. The method according to any one of claims 1 to 11, wherein the 3D images and the 3D sound are contained in a memory unit associated with the autostereoscopic display device or are provided by live streaming of a particular scene.

13. The method according to any one of claims 1 to 12, wherein the method is used in a teleconferencing session.

14. A naked eye stereoscopic display device, - an eye tracking system for determining the position of the viewer's eyes relative to said autostereoscopic display device; - means for determining the position of the viewer's ears relative to said autostereoscopic display device; a display unit configured to display a three-dimensional image consisting of a left image presented to the left eye of a viewer and a right image presented to the right eye of the viewer; - audio means arranged to generate three-dimensional sound; a processor configured to generate image data for driving said display and to generate sound data for driving said audio means; the processor includes an audiovisual application; - two virtual cameras configured to generate left and right image data for driving the display, taking into account the position of the viewer's eyes relative to the naked eye stereoscopic display device; - two virtual microphones configured to generate left and right sound data for driving the audio means, taking into account the position of the viewer's ears relative to the autostereoscopic display device.

15. - said audio means comprises a speaker having a substantially fixed position relative to said autostereoscopic display device; The naked-eye 3D display device of claim 14, further comprising a crosstalk cancellation unit that separates the sound generated by the speakers between the viewer's ears and cancels distortion occurring between the ears and the speakers.

16. 15. The autostereoscopic display device according to claim 14, wherein the audio means comprises headphones or earphones having a variable position relative to the autostereoscopic display device.

17. The naked eye stereoscopic display device according to any one of claims 14 to 16, wherein the means for determining the position of the viewer's ears relative to the naked eye stereoscopic display device includes an ear tracking system, and / or a means for deriving the position of the ears from the position of the eyes determined by the eye tracking system.

18. The naked eye stereoscopic display device further comprises an additional tracking system configured to track an object or a further body part, which is a body part other than an eye or an ear, such as a hand or a foot, - reproducing three-dimensional sound associated with the object or the further body part taking into account the position of the viewer's ears relative to the autostereoscopic display device and the position of the object or the further body part relative to the autostereoscopic display device, - An autostereoscopic display device as described in any one of claims 14 to 17, configured to display a three-dimensional image associated with the object or further body part taking into account the position of the viewer's eyes relative to the autostereoscopic display device and the position of the object or further body part relative to the autostereoscopic display device.

19. A configuration including the autostereoscopic display device according to claim 18 and a stylus pen, wherein the additional tracking system is configured to track the stylus pen, and the autostereoscopic display device is - reproducing three-dimensional sound associated with the stylus pen, taking into account the position of the viewer's ears relative to the autostereoscopic display device and the position of the stylus pen relative to the autostereoscopic display device; - A configuration configured to display a three-dimensional image associated with the stylus pen, taking into account the position of the viewer's eyes relative to the autostereoscopic display device and the position of the stylus pen relative to the autostereoscopic display device.

Images