AUTOSTEREOSCOPIC DISPLAY DEVICE WITH REMOTE BODY TRACKING SYSTEM - Patent application

Abstract

The present invention relates to a method for tracking relevant body parts of a viewer of an autostereoscopic screen using a head tracking system integrated with the screen and a body tracking system remote from the screen. By tracking the viewer's head simultaneously with both systems, it is possible to relate the position data of all other body parts acquired with the remote body tracking system to the position of the screen. In this way, the positions of such body parts relative to the screen can be acquired even when they are outside the field of view of the integrated head tracking system. In this way, a more accurate and therefore more realistic interaction of the viewer with the displayed content can be achieved.

Description

[Technical field]

[0001] The present invention relates to a method for tracking relevant body parts of a viewer of an autostereoscopic display device. [Background technology]

[0002] Autostereoscopic displays have attracted a great deal of attention in the past two decades. One of their most prominent features is that they allow the viewer to perceive three-dimensional images without dedicated eyewear devices such as headsets, head-mounted displays, glasses, etc. In particular, it is possible to take into account the viewer's movements relative to the display. The key to this technology is the presence of a screen with a lenticular lens or parallax barrier. This allows the left eye image to be directed to the viewer's left eye and the right eye image to be directed to the viewer's right eye. This allows the viewer to perceive a three-dimensional image, some parts of which appear to be in front of the display and other parts of which appear to be further away than the display ("behind" the display). In such displays, the eye positions are preferably tracked with an eye-tracking system, since this allows the left and right eye images to be better directed to the respective eyes.

[0003] However, certain autostereoscopic displays not only display three-dimensional images, but also allow the viewer to interact with them. For example, a foot may hit a virtual object or a hand may press a virtual button. Such a combination may contribute to a truly believable three-dimensional virtual environment for the viewer. To achieve a realistic interaction of the viewer with the displayed content, it is necessary to track the body parts he uses for the interaction.

[0004] Since the eyes are typically always in front of the autostereoscopic display, the eye tracking system is usually placed near the display to have a similar field of view to that of the display. However, this is different for other body parts, especially extremities such as hands, legs and feet, as they may be out of the field of view of the display. Therefore, the tracking systems are usually integrated into a single system called a "body tracking system" and are preferably positioned at a location where the entire body is well visible. This is usually not achieved by attaching it to the autostereoscopic display, but is remote from it. This means that the body tracking system is generally positioned at a location at the discretion of the viewer, depending on the space available to the viewer. Therefore, it is not as easy for the body tracking system to track against the autostereoscopic display as it is for the eye tracking system when it is arbitrarily positioned relative to the autostereoscopic display. A possible solution to this problem should not lead to a delay in the response of the autostereoscopic display (i.e., there is no increase in latency). Furthermore, it is desirable for the autostereoscopic display to form a robust system with a remote body tracking system that is not easily disturbed by any accidental movement of the device. Convenience for the viewer is also important. Whenever he wants to use an autostereoscopic display and interact with the content displayed on it, he should not be faced with lengthy calibration procedures. Summary of the Invention

[0005] It is therefore an object of the present invention to provide an accurate and robust method for using a separate body tracking system in combination with an autostereoscopic display device. A further object is to achieve a more realistic interaction of the viewer with the displayed content.

[0006] It has been found that one or more of these objectives may be achieved by using head position data acquired with a separate head tracking system. Accordingly, the present invention relates to a method for tracking relevant body parts of a viewer of an autostereoscopic display device, comprising: -An autostereoscopic display device, an integrated head tracking system configured to determine a position of a viewer's head relative to the autostereoscopic display device; a remote body tracking system configured to determine a position of the viewer's head and associated body parts relative to the remote body tracking system; -The method is 1) determining a viewer's head position relative to an autostereoscopic display device using an integrated head tracking system; 2) using a remote body tracking system to determine a position of the viewer's head and associated body parts of the viewer relative to the remote body tracking system; 3) using the head position obtained in step 1) and the head position obtained in step 2) to determine a position of a relevant body part of the viewer relative to the autostereoscopic display device. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic top view of a setup in which the method of the present invention can be performed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, dimensions of some elements in the figures may be exaggerated relative to other elements to help improve understanding of various exemplary embodiments of the present invention. In particular, the relative dimensions of the viewer, the autostereoscopic screen and the virtual objects in a particular setting cannot be derived from the figures. Also, the figures do not provide any indication of the relative positioning of the different tracking systems, except that the remote body tracking system is depicted farther away from the autostereoscopic display device than the integrated tracking system. Furthermore, terms such as "first", "second", etc. in this specification, if any, are generally used to distinguish between similar elements and not necessarily to describe a sequential or chronological order.

[0009] Throughout the description and claims, the terms "three-dimensional image" and "autostereoscopic image" are used interchangeably and refer to the same type of image. With this, it is recognized that an autostereoscopic image is said not strictly the same as a three-dimensional image. An autostereoscopic image is an image that is only perceived as three-dimensional by a viewer because it is composed of a left image that is to be presented to the viewer's left eye and a right image that is to be presented to the viewer's right eye.

[0010] In the context of the present invention, the term "viewer" means a person who consumes the content presented to him according to the method of the present invention. In addition to viewing three-dimensional images, the viewer may experience other sensory stimuli, such as sound or tactile stimuli. However, although for convenience a person is consequently called a "viewer", it is understood that he may at the same time be, for example, a "listener".

[0011] Throughout the text, references to the reader will be made by masculine terms such as "he," "to him," or "his." This is for purposes of clarity and brevity only, it being understood that feminine terms such as "she" and "her" apply equally.

[0012] When referring to "integrated" tracking systems, such as an integrated eye tracking system, an integrated head tracking system, and an integrated hand tracking system, it is meant that the tracking hardware of such tracking systems forms an integrated part of the autostereoscopic display device. In particular, in an "integrated" tracking system, any sensor for determining the position of a body part relative to the sensor has a fixed and known position relative to the screen. Thus, the sensors in such tracking systems can be equally used as sensors for determining the position of a body part relative to the screen.

[0013] When referring to a "remote" tracking system, such as a body tracking system, it is meant that the tracking hardware of such a tracking system is still associated with the autostereoscopic display device, but does not form an integral part of it. This means that it is in electrical communication (e.g., wired or wireless) with the autostereoscopic display device, but does not necessarily have a fixed, known position relative to the screen. In particular, in a "remote" tracking system, any sensor for determining the position of the body part relative to the sensor is designed to be positioned by the viewer in a suitable, but not predetermined, position relative to the autostereoscopic display device. The positioning of the sensor is at the discretion of the viewer, and thus the position may change from one viewing session to another.

[0014] The method of the present invention utilizes an autostereoscopic display device. This is typically a display device that is mostly stationary in the real world during its use, such as a desktop device or a wall-mounted device. For example, the autostereoscopic display device is a television, a (desktop) computer with a monitor, a laptop, or a cinema display system. However, it may also be a portable display device, such as a mobile phone, a tablet, or a game console, which allows the viewer to move (freely) in the real world with the autostereoscopic display device.

[0015] Autostereoscopic display devices are known in the art, for example from WO2013120785A2. The main components of the autostereoscopic display device used in the method of the present invention are typically an autostereoscopic screen, an integrated eye tracking system, an integrated head tracking system, a remote body tracking system, an image data supply unit, and a processing unit. Optionally, the autostereoscopic display device also comprises an integrated hand tracking system.

[0016] The autostereoscopic screen comprises means for displaying three-dimensional images to a viewer driven by a processing unit and whose eyes are tracked by an integrated eye-tracking system, such means comprising an array of pixels for generating a display output and a parallax barrier or lenticular lens provided on the array for directing a left image to the viewer's left eye and a right image to the viewer's right eye.

[0017] The integrated eye tracking system comprises means for tracking the position of the viewer's eyes relative to the autostereoscopic display device (more specifically relative to the screen) and is in electrical communication with the processing unit, in particular the weaver therein. The integrated eye tracking system enables the screen to generate a left image that is specifically directed to the viewer's left eye and a right image that is specifically directed to the viewer's right eye. To this end, the integrated eye tracking system is typically a fixed part of the autostereoscopic display device in that it has a fixed and known position relative to the screen.

[0018] The integrated head tracking system comprises means for tracking the position of the viewer's head relative to the autostereoscopic display device (more specifically relative to the screen) and is in electrical communication with the processing unit. To this end, the integrated head tracking system is typically a fixed part of the autostereoscopic display device in that it has a fixed, known position relative to the screen.

[0019] The remote body tracking system comprises means for simultaneously tracking the positions of multiple body parts of a viewer relative to the autostereoscopic display device and is in electrical communication with the processing unit. The tracked body parts are at least the head and preferably one or more other body parts selected from the group of torso, knees, elbows, forearms, upper arms, shoulders, lower legs, upper legs, hips, hands and feet. Preferably, the system is configured such that tracking of the entire body can be performed by deriving it from the tracking of a selection of separate body parts, for example by interpolation. The remote body tracking system does not need to have a fixed position relative to the screen and may be positioned at any suitable location relative to the screen, and it is not required (or even useful) to know the location of such locations relative to the screen.

[0020] The image data providing unit provides image data to the processing unit. It may be a memory containing a recording of a scene or an animation. It may also be a unit configured to receive a live streaming video recording of a scene.

[0021] The processing unit receives position data from the integrated eye tracking system, the integrated head tracking system, the optional integrated hand tracking system, and the remote body tracking system, and it receives image data from an image data supply unit (image memory or live stream). The processing unit is configured to take into account the eye position data acquired by the eye tracking system and generate image data thereby to drive the screen. Thus, a key component of the processing unit is the so-called "weaver", which weaves the left and right images into an array of pixels, thereby determining which pixels generate pixel outputs corresponding to the respective images. In this way, a three-dimensional image can be displayed from the image data for a viewer at a specific position.

[0022] The processing unit typically includes an application for determining a body part position, where the position data of the different tracking systems are combined to provide true position data relating to a particular body part. The processing unit typically also includes an application for processing image data from the image data providing unit, where the application also receives position data from the application for determining the position of the body part.

[0023] An optional integrated hand tracking system comprises means for tracking the position of the viewer's hand (or hands) relative to the autostereoscopic display device (more specifically, relative to the screen) and is in electrical communication with the processing unit. To this end, the integrated hand tracking system is typically a fixed part of the autostereoscopic display device in that it has a fixed, known position relative to the screen.

[0024] In the method of the present invention, the device may further be provided with additional features such as means for recording the scene in the field of view of the screen, and audio means for playing sounds to the viewer.

[0025] The audio means, if present, comprises means for reproducing sound to the viewer, for example the audio means may comprise items such as stereo loudspeakers, a loudspeaker array, headphones or earphones.

[0026] Figure 1 is a schematic top view of a setup in which the method of the present invention can be implemented. It includes an autostereoscopic display device as outlined above, a virtual object displayed by the autostereoscopic display device, and a viewer in the field of view of the autostereoscopic display device. The viewer's head, hands and feet are depicted, with one foot interacting with the virtual object. The position of this foot is determined using a remote body tracking system (dotted line from the remote body tracking system to the foot).

[0027] The inventors have recognized that it is possible that certain body parts of a viewer can be observed and tracked by their respective integrated tracking systems and by a remote body tracking system. By simultaneously tracking at least one body part by both tracking systems, it is possible to associate the position data of all other body parts acquired with the remote body tracking system with the position of the screen. In this way, the positions of such body parts relative to the screen can be acquired even when they are outside the field of view of the integrated head tracking system (or any other possible integrated tracking system, since it is typically inherent in the position of any such system relative to the viewer that its field of view cannot adequately cover the desired body part). In the context of the present invention, the body parts whose positions are (should be) acquired in this way are called "associated body parts".

[0028] The body part tracked simultaneously by both tracking systems is the head in each case (which is always present when the viewer is looking at the screen). Thus, both the integrated head tracking system and the remote body tracking system track the head. The position of a particular relevant body part of the viewer relative to the autostereoscopic display device is then determined by using the head positions obtained by 1) the integrated head tracking system and 2) the body tracking system.

[0029] This can be done, for example, - determining the position of the relevant body part relative to the head by using the position of the relevant body part and the head relative to a remote body tracking system, and subsequently - the position of a relevant body part of a viewer relative to an autostereoscopic display device, a) the position of relevant body parts relative to the head; b) deriving the position of the head relative to the autostereoscopic display device.

[0030] As mentioned above, the remote body tracking system does not need to have a fixed position relative to the screen, and may be positioned at any suitable location relative to the screen, without knowing how its position relates to the position of the screen. The fact that there is no fixed and known position relative to the screen does not hinder the position determination of the relevant body part. This is because the remote body tracking system only needs to provide the spatial relationship between the head and the remote body part. Given the relative nature of this relationship, the position of the remote body tracking system relative to the screen is actually irrelevant to determining this relationship. Thus, the position of the relevant body part of the viewer relative to the screen can be determined in the method according to the invention, and it will be appreciated by those skilled in the art that this can be done when the position of the remote body tracking system relative to the screen is not fixed and is not known.

[0031] In the method of the present invention, the remote body tracking system may be configured to determine the position of one or more further relevant body parts of the viewer relative to the remote body tracking system. In such a case, the method may further comprise determining the position of one or more further relevant body parts of the viewer relative to the autostereoscopic display device. This opens up a way to achieve full body tracking with respect to the autostereoscopic display device.

[0032] Next to the head, an additional body part (i.e. a second body part) can be tracked by both tracking systems simultaneously (i.e. by both the integrated head tracking system and the remote body tracking system). The purpose of doing this may be to (again) determine the position of the relevant body part of the viewer relative to the autostereoscopic display device. In this way, the accuracy of the position of the relevant body part of the viewer may be improved, for example by determining the average of the two positions thus determined.

[0033] The additional body part that is tracked simultaneously is preferably a body part that is also dominant (just like the head) in the field of view of the screen. It is for example selected from the group: hand, armpit, elbow, upper arm, shoulder or torso. More preferably, it is the hand. Thus, such a method comprises: the autostereoscopic display device further comprising an integrated hand tracking system configured to determine a position of a viewer's hands relative to the autostereoscopic display device; a remote body tracking system configured to determine a position of a viewer's head, associated body parts, and hands relative to the remote body tracking system; -The method is 4) determining the position of the viewer's hands relative to the autostereoscopic display device using an integrated hand tracking system; 5) using the remote body tracking system to determine the position of the viewer's hands and associated body parts of the viewer relative to the remote body tracking system; 6) By using the hand positions obtained in step 4) and the hand positions obtained in step 5), - determining the position of a relevant body part of a viewer relative to an autostereoscopic display device, and / or - refining the location of the viewer's relevant body parts determined in step 3).

[0034] In this method, the hand may equally be replaced by another suitable body part, typically one selected from the group of armpit, elbow, upper arm, shoulder and torso. Thus, the method according to the invention may be a method in which the hand is replaced by a body part selected from the group of armpit, elbow, upper arm, shoulder and torso.

[0035] To further improve the accuracy of the measurement of the position of the relevant body part, the method of the invention may also include tracking even a third body part simultaneously by both tracking systems (i.e. by both the integrated head tracking system and the remote body tracking system). Preferably, such body part is also a hand. Thus, such a method comprises: the autostereoscopic display device further comprises an integrated hand tracking system configured to determine a position of a viewer's hands relative to the autostereoscopic display device; the remote body tracking system is configured to determine the position of the viewer's head, relevant body parts, and hands relative to the remote body tracking system, the relevant body parts being other than the hands; -The method is 7) determining the position of the viewer's hands relative to the autostereoscopic display device using an integrated hand tracking system; 8) using the remote body tracking system to determine the position of the viewer's hands and associated body parts of the viewer relative to the remote body tracking system; 9) Using the positions of both hands obtained in step 7) and the positions of both hands obtained in step 8), - determining the position of a relevant body part of a viewer relative to an autostereoscopic display device; and / or - improving the accuracy of the location of the relevant body parts determined in step 3) and / or step 6).

[0036] In this method, one or both hands may be equally replaced independently of each other by another suitable body part, typically one or two selected from the group of armpit, elbow, upper arm, shoulder and torso. Thus, the method according to the present invention may be a method in which one or both hands are replaced by a body part selected from the group of armpit, elbow, upper arm, shoulder and torso.

[0037] The determined position of the relevant body part, as determined via the method of the invention, may change after a certain time interval as a result of the viewer's movements. However, it is not necessary to repeat the entire method to take into account any changes in the position of the relevant body part of the viewer. Once performed, the position of the relevant body part relative to the screen is known. It may then be sufficient to simply determine any movements of the relevant body part relative to its position as determined by the method of the invention. Thus, following the method of the invention, after a certain time interval, a remote body tracking system is used to determine the shift in the position of the relevant body part or one or more further relevant body parts after the time interval. This may be done without the use of a head or hand tracking system. The advantage of such a method is that less computational power is required, leading to less power consumption and / or less latency experienced by the viewer.

[0038] In order to continuously track the relevant body part or one or more further relevant body parts, the determination of the shift is performed multiple times, typically with each iteration of the determination being performed after a certain time interval, for example in the range of 0.02 to 2.0 seconds, in particular in the range of 0.1 to 1.0 seconds.

[0039] In addition to repeating the determination of the shift, it may be necessary from time to time to repeat the entire method of the present invention one or more times, for example 10 or more, 100 or more, 1,000 or more, 10,000 or more, 100,000 or more, or 1,000,000 or more. This allows adapting to any changes in the positioning of the remote body tracking system relative to the autostereoscopic display device. For example, the method of the present invention is repeated at a time interval that is in the range of 0.050 to 60 seconds, in particular in the range of 0.50 to 10 seconds, more particularly in the range of 1.0 to 5.0 seconds. Depending on the settings in which the autostereoscopic display device (including the remote body tracking system) is used by the viewer, it may be repeated at a larger time interval, for example at a time interval in the range of 1.0 to 60 minutes, or at a time interval in the range of 1 to 10 hours. The method of the present invention may also be performed only once during a viewing session by the viewer, which is typically at the beginning of the session.

Claims

1. 1. A method for tracking relevant body parts of a viewer of an autostereoscopic display device, comprising: said autostereoscopic display device, - an integrated head tracking system configured to determine the position of the viewer's head relative to the autostereoscopic display device; a remote body tracking system configured to determine a position of the viewer's head and associated body parts relative to the remote body tracking system; - the method comprising: determining the position of the viewer's head relative to the autostereoscopic display device using the integrated head tracking system; determining, using the remote body tracking system, the position of the viewer's head and associated body parts relative to the remote body tracking system; and using the position of the head obtained in step 1) and the position of the head obtained in step 2) to determine the position of the relevant body part of the viewer relative to the autostereoscopic display device.

2. Step 3) - determining the position of the relevant body part relative to the head by using the position of the relevant body part and the head relative to the remote body tracking system determined in step 2; - determining the position of the relevant body part relative to the autostereoscopic display device; a) the position of the relevant body part relative to the head; and b) the position of the head relative to the autostereoscopic display device.

3. the autostereoscopic display device further comprises an integrated hand tracking system configured to determine a position of the viewer's hands relative to the autostereoscopic display device; the remote body tracking system is configured to determine the position of the viewer's head, the relevant body parts, and hands relative to the remote body tracking system; - the method comprising: determining the position of the viewer's hands relative to the autostereoscopic display device using the integrated hand tracking system; determining, using the remote body tracking system, the position of the hand and the associated body part relative to the remote body tracking system; By using the position of the hand obtained in step 4) and the position of the hand obtained in step 5), - determining the position of the relevant body part relative to the autostereoscopic display device, and / or - refining the location of the relevant body part determined in step 3), The method according to claim 1 or 2.

4. the autostereoscopic display device further comprises an integrated hand tracking system configured to determine the position of the viewer's hands relative to the autostereoscopic display device; the remote body tracking system is configured to determine the position of the viewer's head, the relevant body part, and both hands relative to the remote body tracking system, and the relevant body part is not a hand; - the method comprising: determining the positions of the viewer's hands relative to the autostereoscopic display device using the integrated hand tracking system; using the remote body tracking system to determine the positions of the hands and the associated body parts relative to the remote body tracking system; By using the positions of both hands obtained in step 7) and the positions of both hands obtained in step 8), - determining the position of the relevant body part relative to the autostereoscopic display device, and / or - refining the accuracy of the location of the relevant body part determined in step 3) and / or in step 6), The method according to any one of claims 1 to 3.

5. the remote body tracking system is further configured to determine a position of one or more further relevant body parts of the viewer relative to the remote body tracking system; the method further comprises determining the position of the one or more further relevant body parts of the viewer relative to the autostereoscopic display device; The method according to any one of claims 1 to 4.

6. 6. The method according to any one of claims 1 to 5, further comprising following the method after a certain time interval using the remote body tracking system to determine a shift in the position of the relevant body part or the one or more further relevant body parts after said time interval.

7. The method of claim 6 , wherein the determination of the shift is performed multiple times, each time after a specific time interval, such that the relevant body part or the one or more further relevant body parts can be tracked.

8. The method according to claim 7, wherein the time interval is in the range of 0.02 to 2.0 seconds, in particular in the range of 0.1 to 1.0 seconds.

9. 9. The method according to any one of claims 1 to 8, wherein the method is repeated one or more times, for example at a time interval in the range of 0.050 to 60 seconds, in particular in the range of 0.50 to 10 seconds, more particularly in the range of 1.0 to 5.0 seconds.

10. 10. The method of any one of claims 1 to 9, wherein the remote body tracking system is configured to track the position of one or more body parts selected from the group of torso, knees, elbows, forearms, upper arms, shoulders, lower legs, upper legs, hips, hands, and feet.

11. The method according to any one of claims 1 to 10, wherein the autostereoscopic display device is selected from the group of televisions, monitors, desktop computers, laptops, cinema display systems, mobile phones, tablets and game consoles.

Images