Method for scaling size and depth in video conference

Abstract

The present invention relates to a method for driving a screen of a first autostereoscopic display device to display a stereoscopic image of the first person to a second person, wherein the second person is a viewer residing within a field of view of the screen of the first autostereoscopic display device. The method includes scaling the stereoscopic image in terms of size and depth. In a preferred embodiment, the size and depth of the zoom coincide with those experienced by a person in the real world. The method of the present invention is advantageous for video conferences such that a stereoscopic image of a first person can be displayed to a second person and a stereoscopic image of the second person can be displayed to the first person.

Description

Technical Field

[0001] The present invention relates to a method for driving a screen of an autostereoscopic display device to display a stereoscopic image of a first person to a second person. Background Art

[0002] One of the areas where autostereoscopic displays have found useful and entertaining applications is video conferencing, which is generally understood as a technology-assisted meeting in which participants in different locations can communicate with each other using both audio and visuals. The great advantage of video conferencing is that people do not have to travel to be part of a scene where they can talk and see each other.

[0003] Autostereoscopic displays contribute to making the encounter between people more realistic than with monoscopic displays. Autostereoscopic displays allow viewers to perceive remote participants in a video conference as three-dimensional images, without the need for specialized eyewear such as glasses. This allows the viewer to experience themselves as being in the real environment, while also being part of the environment displayed by the autostereoscopic display—a virtual environment observable in three dimensions through a virtual window formed by the autostereoscopic display's screen. To the viewer, the remote participant may appear to be in front of the virtual window, behind it, or partially in front of it and partially behind it. However, current video conferencing systems still have some shortcomings in displaying remote participants in a realistic manner. Typically, they fail to present remote participants at their true size and comfortable distance to the viewer. In particular, the size of the remote participant does not match the perceived distance, for example, when viewed from the same distance through a real window.

[0004] Another difference from reality is that as the viewer moves towards or away from the autostereoscopic display, the size of the remote participant does not change accordingly. Summary of the Invention Therefore, the object of the present invention is to provide a method for improving the viewing experience of a viewer of an autostereoscopic display device when viewing a specific person on the autostereoscopic display device, in particular when the viewer is viewing the remote participant during a video conference with the remote participant. In particular, the object is to provide a method that allows the display of the person to be adjusted according to the viewer's personal preferences.

[0005] More particularly, it is an object of the present invention to provide a method which allows displaying persons, in particular remote video conference participants, in real size and at a desired distance from the viewer. The goal is also to adjust the perceived size and perceived distance to a certain degree, which is not necessarily the most realistic, but merely the most convenient and / or desirable to the viewer.

[0006] It has now been discovered that one or more of these objectives can be achieved by appropriately scaling the three-dimensional content to be displayed by an autostereoscopic display.

[0007] The present invention therefore relates to a method of driving a screen of a first autostereoscopic display device to display a stereoscopic image of the first person to a second person, wherein the second person is a viewer residing within the field of view of the screen of the first autostereoscopic display device, the method comprising - Provides first-person stereoscopic recording via stereo cameras; - Displaying the stereoscopic recording of the first person to the second person as a stereoscopic image consisting of a left-eye image and a right-eye image, wherein the second person perceives the first person ο in apparent display size, which is the angular size of the first person when viewed by a second person on the screen; and ο to display depth, where displayed depth is the depth perceived by the second person to the first person; The method includes - Adjust the apparent display size to the desired apparent display size by scaling the first person stereo image; - Adjust the display depth to the desired display depth by changing the parallax between the first person's left-eye image and right-eye image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A conventional arrangement to which the present invention is applied is schematically shown, from which the relationship between recording and display can be explained.

[0009] Figure 2 Possible results when applying the method of the invention are shown schematically.

[0010] Figure 3 A video conference setup is schematically shown in which the method according to the invention is applied. DETAILED DESCRIPTION

[0011] The accompanying drawings do not limit the present invention to the specific embodiments disclosed and described in this specification. Elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention.

[0012] In the context of the present invention, the term "stereoscopic image" is intended to mean an image consisting of a left-eye image presented to the viewer's left eye and a right-eye image presented to the viewer's right eye. In this way, the viewer can perceive the image as three-dimensional (although strictly speaking, it is not a true three-dimensional image). The left-eye and right-eye images can also be displayed in areas close to the respective eyes, as long as they do not reach the other eye. However, in practice, a small (or very small) amount of light will always "leak" to the other eye (crosstalk), although viewers may not always be aware of this and still consider their 3D viewing experience satisfactory.

[0013] In the context of the present invention, a stereoscopic recording is intended to include information representing a three-dimensional visible image, as it can be used to display such a stereoscopic image when input to an autostereoscopic display device in a format that can be processed by the device. A stereoscopic recording is a recording or live stream of a real scene, person, or object captured by a stereoscopic camera and containing information about the scene, person, or object in three dimensions. It can be stored in a memory portion associated with the device so that it can be displayed on demand, or it can be displayed as live video captured by the stereoscopic camera associated with the autostereoscopic display device. When the stereoscopic recording in the method of the present invention is of a person, the recording may initially include the surroundings of the person's environment. When such a recording is displayed as an image, the person or a part of their body (e.g., the head) can be segmented from any surroundings, resulting in an image of the person (or part of their body) devoid of any surroundings. This allows for the introduction of background into the image. In the context of the present invention, the term "stereo camera" refers to a camera capable of providing a stereoscopic recording of a real scene, person, or object. From this stereoscopic recording, a stereoscopic image can be generated. For the purposes of the present invention, stereo cameras are intended to include both stereoscopic cameras and plenoptic cameras. Furthermore, the term stereo camera may include multiple (stereo) cameras that together constitute and provide the functionality of a stereo camera as described above. In the context of the present invention, the term "viewer" refers to a person who consumes content presented to them according to the methods of the present invention. In addition to viewing stereoscopic images, the viewer may also experience other sensory stimulations, such as sound or tactile stimulation. However, for convenience, such a person will also be referred to as a "viewer," although it is understood that they may also be, for example, an "audience." In the method of the present invention, the "second person" is always characterized as a viewer. The "first person" is then recorded by the stereo camera and can be seen by the second viewer. If the second person is also recorded by the stereo camera so that he or she can be seen by the first person, both the first and second person are viewers.

[0014] Throughout this document, references to viewers will be made using the masculine gender, such as "he" or "his." This is for the sake of clarity and brevity, and it should be understood that feminine gender, such as "she" or "her," also applies. For the same reason, the terms "person" and "participant" will also be referred to in the masculine gender.

[0015] The method of the present invention utilizes one or more autostereoscopic display devices. Such devices are typically devices that remain largely stationary in the real world during use, such as desktop or wall-mounted devices. For example, an autostereoscopic display device is a television, a (desktop) computer with a display, a laptop computer, or a cinema display system. However, it can also be a portable device, such as a mobile phone, a car monitor, a tablet computer, or a gaming console, allowing the viewer to (freely) move around in the real world with the autostereoscopic display device. Autostereoscopic display devices are known in the art, for example from WO2013120785A2. The main components of the autostereoscopic display device used in the method of the present invention are typically an eye tracking system, a screen, a processing unit and optionally an audio device. The eye tracking system includes a device for tracking the position of a viewer's eyes relative to the autostereoscopic display device and is in electronic communication with a processing unit. The eye position is required to correctly interleave left-eye and right-eye images in the pixel array so that each image reaches the target eye, even when the viewer moves relative to the screen of the autostereoscopic display device.

[0016] The viewing distance from the viewer's eyes to the screen can also be obtained by using an eye tracking system. Alternatively, there may be an independent device for determining the viewing distance. The recorded distance of the viewer relative to the stereo camera may be obtained by using an eye tracking system or a different system associated with or integrated into the stereo camera.

[0017] The screen includes a device for displaying stereoscopic images to a viewer whose eyes are tracked by an eye tracking system. The device includes a pixel array for generating a display output and a parallax barrier or a lenticular lens disposed above the array to direct a left-eye image to the viewer's left eye and a right-eye image to the viewer's right eye. The processing unit is fed with stereoscopic recordings and configured to drive the screen, taking into account the data acquired by the eye-tracking system. A key component of the processing unit is the so-called "weaver," which weaves the left-eye and right-eye images into the pixel array, determining which pixels produce pixel outputs corresponding to the respective images. In this way, stereoscopic images can be displayed to viewers in specific locations. The processing unit is typically further configured to perform stereoscopic image adjustment according to the method of the present invention, ie to adjust the apparent display size and to adjust the display depth of the stereoscopic image.

[0018] The optional audio device includes a device for playing sound to the viewer. For example, the audio device includes one or more devices selected from stereo speakers, speaker arrays, headphones and earphones.

[0019] The autostereoscopic display device used in the method of the present invention typically includes a receiver for receiving a stereoscopic recording of a person in the real world. This also includes receiving a live video stream of the person in the real world. An audio recording or audio stream may also be received by such a receiver. Transmission of the video and / or audio recording may be performed, for example, via a wireless connection or telecommunication line.

[0020] The autostereoscopic display device used in the method of the present invention may include a memory to store a stereoscopic recording of a scene, person or object in the real world.

[0021] The autostereoscopic display device used in the method of the present invention may include a stereo camera for recording the viewer and / or a device for determining the recording distance of the viewer from the stereo camera. The latter device may be integrated into the stereo camera. Optionally, an audio recording device for recording the viewer and / or the sounds surrounding the viewer is also part of the autostereoscopic display device. The recordings made by the stereo camera and / or the audio recording device can be transmitted to a different environment for presentation to another viewer. The determined recording distance can also be transmitted to this different environment.

[0022] When a viewer of an autostereoscopic display device views a stereoscopic recording of another person as a stereoscopic image on the screen of the autostereoscopic display device (without applying the method of the present invention), the size and depth of the other person perceived by the viewer are typically unrealistic, taking into account viewing distance and perceived depth. Alternatively, the viewer may prefer to view the distant person at a specific size and depth that differs from the "real" size and depth, for example because this preferred size and depth is more appealing to the viewer.

[0023] The present invention provides a solution to this by allowing the viewer to adjust the scale and depth of the displayed stereoscopic image.

[0024] The solution provided by the present invention for achieving a certain size and depth takes into account the so-called "apparent size" of the person to be recorded in the real world and converts this into the image of the person being recorded displayed on the screen. The term "apparent size" of a person is intended to refer to the angular distance from one side of the person (or a specific point) to the other side of the person (or a second specific point). This can be thought of as the angular displacement through which an eye or camera must rotate to see from one side (or point) to the other. Because people are irregularly shaped, with two opposing sides that can be difficult to define, in practice, according to the present invention, the apparent size of a person is typically obtained by defining two facial features of the person, such as the eyes or ears, and determining their angular displacement. Therefore, to implement the method of the present invention, it is important to perform other determinations of the person's angular size on the same features of the person.

[0025] In the context of this invention, there are two types of apparent size; 1) apparent display size; 2) apparent real-world size.

[0026] The apparent display size of a person in an image is the angular size of the person as perceived by a viewer viewing the image as a displayed image on a screen. In this context, the viewed person can be perceived within the plane of the screen or at a specific distance from the screen (i.e., at a specific depth from the viewer's perspective).

[0027] The apparent real-world size of a person is the angular size of that person as perceived by a viewer viewing that person from a specific distance in the real world. In the context of this invention, a viewed person is a person whose image is recorded by a stereo camera at the recording distance (the distance between the person and the stereo camera recording the person). The apparent real-world size of a viewed person is the angular size of that person as perceived by the viewer when viewing that person in the real world from the recording distance (i.e., when the viewer's viewpoint is that of the stereo camera).

[0028] For the purposes of the present invention, a "desired apparent display size" is defined as the apparent display size achieved when the method of the present invention is performed. The term "desired display depth" applies mutatis mutandis. Both terms refer to adjusting a parameter to a value preferred by, for example, the method of the present invention and / or a person involved in the environment in which the display or recording is performed according to the method of the present invention, such as a second person.

[0029] In the method of the present invention, the person whose image is stereoscopically recorded and displayed on the screen of the autostereoscopic display device is referred to as the "first person." The person viewing the image of the first person is referred to as the "second person." In an alternative embodiment, the reverse can be applied when the second person is also stereoscopically recorded and the first person views the image of the second person. In such an embodiment, a video conference can be conducted.

[0030] A stereoscopic image of a person does not necessarily include the entire person. The image may also include a portion of the person, typically the head. Therefore, in the method of the present invention, a stereoscopic recording of a first person may be represented by a stereoscopic recording of the first person's head, such that a stereoscopic image of the first person is represented by a stereoscopic image of the first person's head.

[0031] In the method of the present invention, depth scaling is achieved by varying the parallax between a person's left-eye and right-eye images. It is recognized that this depth scaling is actually an approximation of true depth scaling, as the recording distance is not specifically controlled to achieve depth scaling. For many applications of the present invention, such as video conferencing, this approximation is acceptable to the viewer, especially when the parallax change is not extreme.

[0032] A preferred method of achieving a desired apparent display size for the first person includes deriving from the desired display depth by performing the following steps: a) provide the second person with the desired display depth perceived by the first person; b) determine the viewing distance from the second person’s eyes to the screen; c) determine the recording distance from the first person to the stereo camera; d) determining an apparent real-world size of the first person, the apparent real-world size of the first person being the angular size of the first person when viewed in the real world from the recorded distance obtained in c); e) Calculate the first person's expected apparent display size by using the following, where the first person's expected apparent display size is the angular size when the first person is viewed in the real world from the expected display depth provided in a). a) the desired display depth provided; ο b) the viewing distance obtained; ο d) the apparent real-world size obtained; f) scaling the stereoscopic image to adjust the apparent display size to the desired apparent display size; g) When the image is larger than the screen, the stereoscopic image scaled in step f) is adapted to the screen by cropping the image.

[0033] This document describes setting a desired apparent display size to a value consistent with the depth perceived by a second person when the first person is displayed at a desired display depth. To achieve this, the stereo image of the first person is scaled to a certain degree to achieve the desired apparent display size. As a result, the first person is perceived at a size that the second person would expect, based on the depth the second person perceives the first person to be.

[0034] If desired, the stereo image of the first person can also be scaled to a different size. This may be preferable when actual dimensions are less important and / or when it is undesirable for the person's image to cover too large or too small a portion of the screen. For example, the screen may be too small to properly display the stereo image (or, equivalently, the first person may be too close to the stereo camera, resulting in an image of the second person that is too large on the screen). For example, additional scaling may be performed according to a first percentage within the range of 50%-150%, particularly within the range of 80%-120%, more particularly within the range of 95%-105%, and even more particularly within the range of 99%-101%. In this manner, the second person is allowed to adjust the desired apparent display size to an even more desired apparent display size.

[0035] In the method of the present invention, wherein the second person views the first person, the zoom percentage desired by the second person is referred to as the first percentage. Additionally, in the opposite case, wherein the first person views the image of the second person, the percentage is referred to as the second percentage, as will be further explained below.

[0036] The stereoscopic image of a person to be scaled according to the present invention is typically located at a distance from the stereo camera that is within the range where stereoscopic vision is possible. Typically, this distance is less than 25 meters, preferably less than 10 meters, more preferably less than 7 meters, and even more preferably less than 5 meters. For example, it is less than 4 meters, less than 3 meters, or less than 2 meters. For example, it is within the range of 0.50-2.0 meters, particularly within the range of 0.80-1.6 meters.

[0037] The image of the first person may include the actual background behind the first person when the stereoscopic recording of the first person is performed, i.e., the original background of the portion of reality in which the first person is located. However, typically, the first person is segmented from the original background such that the displayed image of the first person does not include the original background. This segmentation of the image portion may be performed using methods known in the art. When the background is missing due to this segmentation of the first person (or, for example, the first person's head), another background is typically introduced behind the stereoscopic image of the first person. This background can be scaled to a first desired degree, typically independently of the scaling of the stereoscopic (foreground) image of the first person. Without segmenting the first person from the background, scaling the stereoscopic image of the first person would cause the background to be scaled to the same degree. This could give the viewer of the image an unsettling background appearance.

[0038] Therefore, the method of the present invention may also include - Provide a recording of the background, obtained by a video camera or stereo camera; - Show a second person on screen at the same time ο a record of the background as a background image; and ο a stereoscopic recording of the first person as a stereoscopic foreground image; - optionally scaling the background image to a first desired extent; - When the image is larger than the screen, the background image is cropped to fit the screen.

[0039] Typically, the method of the present invention is performed multiple times in sequence. In this manner, new positions of the first and / or second person relative to the autostereoscopic display and the stereoscopic camera can be calculated, respectively. For example, the method can be repeated at least 10 times, at least 100 times, at least 1,000 times, at least 10,000 times, at least 100,000 times, or at least 1,000,000 times. Any repetition of the method of the present invention, for example, by one of the aforementioned times, can reflect a particular viewing session of the viewer.

[0040] For a realistic viewing experience, the method is typically performed at a frequency of at least 10 times per second. Preferably, the frequency is at least 20 times per second, more preferably at least 30 times per second, and even more preferably at least 50 times per second. For example, the frequency is in the range of 50-250 times per second, in the range of 55-150 times per second, or in the range of 60-120 times per second. The frequency may in particular be 60 Hz, 120 Hz, 144 Hz, or 240 Hz. Preferably, it is at the same frequency as the refresh rate of the screen itself.

[0041] When the method of the invention is performed multiple times, this may be advantageous in allowing the second person to perceive the first person to be at (substantially) the same depth relative to the screen each time the method is performed. For example, the second person may prefer to constantly perceive the first person as being directly behind the screen (or with their eyes in the plane of the screen), wherein the second person observes a more or less constant coverage of the screen area by the first person, even when the first person has a varying position relative to the stereo camera, resulting in a varying recording distance. This gives the first person a more stable image to view, wherein the displayed image does not constantly jump between different sizes as the first person moves away from or towards the stereo camera.

[0042] However, the perceived depth may also vary to a lesser extent with the changing recording distance, which may improve the viewing comfort experienced by the second viewer. For example, the perceived depth may vary between a lower value and a higher value, wherein the higher value is up to twice the lower value. For example, the higher value is up to 1.6 times the lower value, up to 1.5 times the lower value, up to 1.4 times the lower value, up to 1.3 times the lower value, up to 1.2 times the lower value, up to 1.1 times the lower value, up to 1.07 times the lower value, up to 1.05 times the lower value, up to 1.03 times the lower value, up to 1.02 times the lower value, or up to 1.01 times the lower value.

[0043] In particular, when the recording distance is reduced by a large percentage (e.g., when the first person moves closer to the stereo camera that is recording him), the screen coverage observed by the second person typically increases significantly. This increase exceeds the magnification that the viewer inherently perceives when observing objects at close range, because the recording distance becomes much smaller than the display depth.

[0044] Figure 1 The effect of a first person (1) moving closer to a stereo camera (3) on his display to a second person (2) on an autostereoscopic screen (4) is shown. The upper side of the figure shows two top view display situations, left side (I) and right side (II), in which the first person (1) is recorded by the stereo camera (3). In the left recording situation (I), the first person (1) is further away from the stereo camera (3) than in the right recording situation (II). The lower side of the figure shows two top view display situations, left side (III) and right side (IV), in which the first person (1) is displayed as a displayed first person (1') on the screen (4) to the second person (2). The displayed first person (1') in the left (III) and right (IV) display situations corresponds to the first person (1) recorded in the left (I) and right (II) recording situations, respectively. Figure 1 The two schematic display situations (III and IV) on the lower side reflect that the first person (1') is displayed as a stereoscopic image on the screen (4). This is because the displayed first person (1') has been drawn in a dimension extending perpendicular to the screen (4) in situations (III and IV), reflecting that the second person (2) may perceive that part of the displayed first person (1') is closer (for example, in front of the screen) and another part is farther away (for example, behind the screen). In addition, Figure 1 The two schematic display situations (III and IV) on the lower side show a first person (1') with a displayed depth (5), which is the depth at which the second person (2) perceives the first person (1'). In the left recording situation (I), the first person (1) is at an initial recording distance (6) from the camera (3), where it has an initial apparent real-world size (α). In the left display situation (III), the second person (2) observes the displayed first person (1') at an initial display depth (5) and at an initial apparent display size (α'), both of which are not significantly different from the initial recording distance (6) and initial apparent real-world size (α) of the left recording situation (I), respectively. The movement of the first person (1) towards the camera (3) is shown by the transition from the left recording situation (I) to the right recording situation (II) (indicated by the arrows). In the latter case, the first person (1) is at a final recording distance (6) from the camera (3), where it has a final apparent real-world size (α). The effect of this movement of the first person is shown in the right display situation (IV), where the second person (2) observes the displayed first person (1') at a final display depth (5) and at a final apparent display size (α'). Since the second person (2) now perceives the first person (1') to be directly in front of him, the display depth is greatly reduced. In addition, the apparent display size has also experienced a significant increase, with the autostereoscopic image of the first person (1') being much larger than that of the second person (2). Therefore, in the right display case (IV), the second person (2) perceives the first person (1') to be very large and at a very short distance.

[0045] However, the method of the present invention is able to compensate for this effect, allowing the first person (1') to be displayed with a reduced final apparent display size (α') and an increased final display depth (5). In other words, the present invention allows the display of a real person (or scene or object) with a certain (more or less) constant perceived depth and a certain (more or less) constant perceived size.

[0046] The method of the present invention is operated as follows Figure 2 As shown, Figure 1 The right side of the diagram shows case (IV) which constitutes the starting point of the method of the present invention ( Figure 2 (top). The arrows represent different embodiments of the method of the present invention, resulting in three possible display scenarios. From left to right, these three display scenarios show a progressive change in display depth and apparent display size, i.e., the display depth increases while the apparent display size decreases. Of course, the number of possible display depths and apparent display sizes is virtually endless; which ones are actually used depends on the setup of the autostereoscopic display device and the preferences of its users, in particular the preferences of the second person (2). A particular setup of display depth and apparent display size results in a scaled size and scaled depth that are consistent with those experienced by the second person (2) in the real world when he sees the first person (1) in the real world. In this setup, the screen of the autostereoscopic display device can be considered a virtual window into another world.

[0047] Thus, an overall advantage of the present invention is that it allows both the size and depth of the displayed stereoscopic image to be adjusted to values ​​that together best suit the viewing needs and / or viewing preferences of the viewer.

[0048] For example, a viewer can adjust the depth at which another person is viewed to achieve a depth that he or she deems convenient (e.g., he or she may not want to perceive the displayed person as too close in front of him or her, but also not too far away—the exact desired depth is a matter of personal preference). Furthermore, he or she can simultaneously adjust the size of such a person or object to achieve a size that he or she deems optimal (e.g., because he or she wishes to keep another portion of the screen available for displaying other content, or because he or she desires a certain perceived size relative to the perceived depth he or she has set—viewers may have different preferences). Thus, the method of the present invention allows a viewer to personalize certain viewing characteristics (i.e., perceived depth and perceived size) of a displayed stereoscopic image, resulting in a more enjoyable viewing experience for the viewer.

[0049] The method allows a viewer to perceive a stereoscopic image of another person at a specific preferred size and at a specific perceived depth. More specifically, the scaled size and scaled depth correspond to what the viewer experiences when seeing the person, as if the person also existed in the real world.

[0050] A particular advantage is that the present invention can compensate for the movement of people recorded by the stereo camera and observed by a viewer. This means that the person's movement toward and away from the stereo camera is not fully translated into the displayed stereo image, but rather is canceled or at least attenuated. This provides the viewer with a more stable view of the people in the video when displayed on an autostereoscopic display device. In its most basic form, the method of the present invention involves one-way communication, i.e., from a first person to a second person, such as might occur in a lecture, online education, webinar, tutorial, etc. However, the method can be supplemented by communication in the opposite direction, i.e., from the second person to the first person. Thus, the method involves two-way communication. Therefore, the method of the present invention can be advantageously applied to video conferencing, allowing a first person and a second person, who are located at a distance from each other, to communicate using both audio and (three-dimensional) vision. In this setup, both the first person and the second person participate in the (online) conference and have autostereoscopic display devices equipped to perform the method of the present invention. The autostereoscopic display device of the first person includes a stereo camera, which operates in conjunction with the autostereoscopic display device of the second person according to the method of the present invention. This also applies mutatis mutandis to the autostereoscopic display device of the first person. Figure 3 Such a video conferencing setup is shown schematically and is further explained below.

[0051] The above-described embodiments for unidirectional communication from a first person to a second person form corresponding embodiments for communication in the opposite direction, ie from the second person to the first person. Therefore, the method of the present invention may further comprise driving the screen of the second autostereoscopic display device to display a stereoscopic image of the second person to the first person, wherein the first person is a viewer residing within the field of view of the screen of the second autostereoscopic display device, the method comprising - Provides stereoscopic recording of the second person via a stereo camera; - showing the stereoscopic recording of the second person to the first person as a stereoscopic image consisting of a left-eye image and a right-eye image, wherein the first person perceives the second person ο in apparent display size, which is the angular size of the second person when the first person views the second person on the screen; and ο to display depth, where the displayed depth is the depth perceived by the first person to the second person; The method includes - Adjust the apparent display size to the desired apparent display size by scaling the stereo image of the second person; - Adjust the display depth to the desired display depth by changing the parallax between the second person's left-eye image and right-eye image.

[0052] In this method, a preferred method of achieving a desired apparent display size for a second person comprises deriving from the desired display depth by performing the following steps: h) providing the desired display depth perceived by the first person to the second person; i) determining the viewing distance from the first person's eyes to the screen; j) determine the recording distance of the second person from the stereo camera; k) determining the apparent real-world size of the second person, the apparent real-world size of the second person being the angular size of the second person when viewed in the real world from the recorded distance obtained in c); l) calculating the second person's expected apparent display size by using the following, the second person's expected apparent display size being the angular size when the second person is viewed from the expected display depth provided in a) in the real world; a) the desired display depth provided; ο b) the viewing distance obtained; ο d) the apparent real-world size obtained; m) scaling the stereoscopic image to adjust the apparent display size to the desired apparent display size; n) When the image is larger than the screen, the stereoscopic image scaled in step m) is adapted to the screen by cropping the image.

[0053] In this method, additional scaling can be performed according to a second percentage, the second percentage being in the range of 50%-150%, particularly in the range of 80%-120%, more particularly in the range of 95%-105%, even more particularly in the range of 99%-101%. In this way, the first person is allowed to adjust the desired apparent display size to an even more desired apparent display size.

[0054] In the method, the stereoscopic recording of the second person may be represented by a stereoscopic recording of the second person's head, such that the stereoscopic image of the second person is represented by a stereoscopic image of the second person's head.

[0055] The image of the second person may include the actual background behind the second person when the second person is stereoscopically recorded, i.e., the original background of the portion of reality in which the second person is located. However, typically, the second person is segmented from the original background such that the displayed image of the second person does not include the original background. This segmentation of portions of the image may be performed using methods known in the art.

[0056] When the background is missing due to this segmentation of the second person (or, for example, the first person's head), another background is typically introduced behind the stereoscopic image of the second person. This background can be scaled to a second desired degree, typically independently of the scaling of the stereoscopic (foreground) image of the second person. Without segmenting the second person from the background, scaling the stereoscopic image of the second person would cause the background to be scaled to the same degree. This could give the viewer of the image an unsettling background appearance.

[0057] Therefore, the program may also include - Provide a recording of the background, obtained by a video camera or stereo camera; - Show to the first person simultaneously on the screen ο a record of the background as a background image; and ο a stereoscopic recording of the second person as a stereoscopic foreground image; - optionally scaling the background image to a second desired extent; - When the image is larger than the screen, the background image is cropped to fit the screen.

[0058] During the video conference, the method of the present invention is typically performed multiple times in sequence. In this manner, the new positions of the first and / or second persons relative to the autostereoscopic display and the stereoscopic camera can be calculated, respectively. For example, the method can be repeated at least 10 times, at least 100 times, at least 1,000 times, at least 10,000 times, at least 100,000 times, or at least 1,000,000 times. Any repetition of the method of the present invention, for example, at one of the aforementioned times, can reflect a particular video conference session for the viewer.

[0059] For a realistic viewing experience, the method is typically performed at a frequency of at least 10 times per second. Preferably, the frequency is at least 20 times per second, more preferably at least 30 times per second, and even more preferably at least 50 times per second. For example, the frequency is in the range of 50-250 times per second, in the range of 55-150 times per second, or in the range of 60-120 times per second. The frequency may in particular be 60 Hz, 120 Hz, 144 Hz, or 240 Hz. Preferably, it is at the same frequency as the refresh rate of the screen itself.

[0060] When the method of the invention is performed in a video conference manner, it may be advantageous to allow each person to perceive the other person to be at (substantially) the same depth relative to the screen each time the method is performed. For example, both parties may prefer to constantly perceive the other party as being directly behind the screen (or with their eyes in the plane of the screen), with the other party having more or less constant coverage of the screen area, even when they have varying positions relative to the stereo cameras resulting in varying recording distances. This gives both parties a more stable image to view, with the displayed image not constantly jumping between different sizes as one of the parties moves away from or towards their respective camera.

[0061] As with the single viewer scenario described above, in a video conferencing embodiment, the perceived depth may also, to a lesser extent, follow the changing recording distance, as this will improve the viewing comfort experienced by both viewers. For example, the perceived depth may exhibit a variation between a lower value and a higher value, where the higher value is up to twice the lower value. For example, the higher value may be up to 1.6 times the lower value, up to 1.5 times the lower value, up to 1.4 times the lower value, up to 1.3 times the lower value, up to 1.2 times the lower value, up to 1.1 times the lower value, up to 1.07 times the lower value, up to 1.05 times the lower value, up to 1.03 times the lower value, up to 1.02 times the lower value, or up to 1.01 times the lower value.

[0062] Especially when the recording distance is reduced by a large percentage (e.g., when the person being recorded moves closer to the stereo camera that is recording him at a distance), the screen coverage next to the viewer usually increases significantly. This increase exceeds the magnification that the viewer inherently perceives when observing objects at close range, because the recording distance becomes much smaller than the display depth. However, the method of the present invention is able to compensate for this effect, allowing both parties to be displayed at a (more or less) constant depth and a (more or less) constant perceived size during a video conference.

[0063] In the method of the present invention, the first and / or second autostereoscopic display device can be a device that is largely stationary in the real world during use, such as a desktop device (e.g., a desktop computer monitor or laptop computer) or a wall-mounted device (e.g., a television or cinema display system). However, it can also be a portable device, such as a mobile phone, tablet computer, or game console, that allows the viewer to (freely) move in the real world. It can also be a display in a vehicle, vessel, or aircraft (e.g., a car, train, ship, or airplane).

[0064] In the method of the present invention, the stereoscopic recording can be stored on a data medium (e.g., a memory stick or hard drive). The autostereoscopic display device then retrieves the stereoscopic recording from this storage. Alternatively, the autostereoscopic display device can retrieve the stereoscopic recording "live" without reading it from memory. Thus, in the method of the present invention, the stereoscopic recording can be contained in a memory portion associated with the autostereoscopic display device, or it can be a live video stream.

[0065] Figure 3 A video conferencing setup is schematically shown in which the method according to the invention is applied. Two different environments are shown. In the left-hand environment, a second person (2) is sitting in front of a first autostereoscopic display device (11); in the right-hand environment, a first person (1) is sitting in front of a second autostereoscopic display device (12). The different components of the two autostereoscopic display devices (11, 12) are schematically drawn, namely the autostereoscopic screens (4a, 4b), the stereo cameras, the eye tracking system, the weaver, and the processor. A stereo camera in the second autostereoscopic display device (12) provides a left-eye image and a right-eye image recording of the first person (1). These images are sent to a processor in the first autostereoscopic display device (11), where the images are scaled taking into account the eye position (viewing distance) of the second person (2) and the eye position (recording distance) of the first person (1). This produces scaled left- and right-eye images, which are sent to the weaver. With the eye position input of the second person (2) provided by the eye tracking system of the first autostereoscopic display device (11), the weaver controls the screen of the first autostereoscopic display device (11) so as to display a stereoscopic image of the first person (1) to the second person (2) on the autostereoscopic screen (4a). The same process occurs when the second person (2) is recorded by the first autostereoscopic display device (11) and displayed to the first person (1) on the autostereoscopic screen (4b) at the second autostereoscopic display device (12). In this example, the processor activities of the two processors are not performed in the corresponding autostereoscopic display device ( 12 ), but rather “in the cloud”.

Claims

1. A method for driving a screen of a first autostereoscopic display device to display a stereoscopic image of the first person to a second person, wherein the second person is a viewer residing within a field of view of the screen of the first autostereoscopic display device, the method comprising: - Provides first-person stereoscopic recording via stereo cameras; - Displaying the stereoscopic recording of the first person to the second person as a stereoscopic image consisting of a left-eye image and a right-eye image, wherein the second person perceives the first person ο the apparent display size of the first person, which is the angular size of the first person when the second person views the first person on the screen; as well as ο The displayed depth of the first person is the depth perceived by the second person; wherein the method comprises - adjusting the apparent display size of the first person to the first person's desired apparent display size by scaling the stereoscopic image of the first person; - Adjusting the display depth of the first person to the first person's desired display depth by changing the parallax between the first person's left-eye image and the right-eye image.

2. The method according to claim 1, wherein The first person's desired apparent display size is derived from the first person's desired display depth by performing the following steps: a) providing a second person with the first person's desired display depth as perceived by the first person; b) determine the viewing distance from the second person’s eyes to the screen; c) determine the recording distance from the first person to the stereo camera; d) determining an apparent real-world size of the first person, the apparent real-world size of the first person being the angular size of the first person when viewed in the real world from the recorded distance obtained in c); e) Calculate the first person's expected apparent display size by using the following, where the first person's expected apparent display size is the angular size when the first person is viewed in the real world from the expected display depth provided in a). a) the desired display depth provided; ο b) the viewing distance obtained; ο d) the apparent real-world size obtained; f) scaling the stereoscopic image to adjust the apparent display size of the first person to the desired apparent display size of the first person; g) When the image is larger than the screen, the stereoscopic image scaled in step f) is adapted to the screen by cropping the image.

3. The method according to claim 1 or 2, wherein the method comprises, after step f), scaling the stereoscopic image to a first percentage, the first percentage being in the range of 50-150%, particularly in the range of 80-120%, more particularly in the range of 95-105%, even more particularly in the range of 99-101%.

4. The method according to any one of claims 1 to 3, wherein the stereoscopic recording of the first person is represented by a stereoscopic recording of the first person's head, such that the stereoscopic image of the first person is represented by a stereoscopic image of the first person's head.

5. The method according to any one of claims 1 to 4, further comprising - Provide a recording of the background, obtained by a video camera or stereo camera; - Show a second person on screen at the same time ο a record of the background as a background image; as well as ο a stereoscopic recording of the first person as a stereoscopic foreground image; - optionally scaling the background image to a first desired extent; - When the image is larger than the screen, the background image is cropped to fit the screen.

6. The method according to any one of claims 1 to 5, wherein the method is repeated one or more times.

7. The method according to claim 6 , wherein the method is performed in such a way that, on each repetition of the method, the second person perceives the first person to be at substantially the same depth relative to the first autostereoscopic display device, or at a depth that varies between a lower value and a higher value, wherein the higher value is up to 1.5 times the lower value, in particular up to 1.2 times the lower value, more in particular up to 1.1 times the lower value, even more in particular up to 1.05 times the lower value.

8. The method according to any one of claims 1-7, wherein the method is used in a video conference.

9. The method according to any one of claims 1 to 8, wherein the method further comprises driving a screen of a second autostereoscopic display device to display a stereoscopic image of the second person to a first person, wherein the first person is a viewer residing within a field of view of the screen of the second autostereoscopic display device, the method comprising - Provides stereoscopic recording of the second person via a stereo camera; - showing the stereoscopic recording of the second person to the first person as a stereoscopic image consisting of a left-eye image and a right-eye image, wherein the first person perceives the second person ο the apparent display size of the second person, which is the angular size of the second person when viewed by the first person on the screen; and ο The displayed depth of the second person is the depth perceived by the first person; wherein the method comprises - adjusting the apparent display size of the second person to the second person's desired apparent display size by scaling the stereoscopic image of the second person; - Adjusting the display depth of the second person to the second person's desired display depth by changing the parallax between the second person's left-eye image and the right-eye image.

10. The method of claim 9, wherein the second person's desired apparent display size is derived from the second person's desired display depth by performing the following steps: h) providing the second person's desired display depth as perceived by the first person; i) determining the viewing distance from the first person's eyes to the screen; j) determine the recording distance of the second person from the stereo camera; k) determining the apparent real-world size of the second person, the apparent real-world size of the second person being the angular size of the second person when viewed in the real world from the recorded distance obtained in c); l) Calculate the second person's expected apparent display size by using the following, where the second person's expected apparent display size is the angular size of the second person when viewed from the expected display depth provided in a) in the real world. a) the desired display depth provided; ο b) the viewing distance obtained; ο d) the apparent real-world size obtained; m) scaling the stereoscopic image to adjust the apparent display size of the second person to the second person's desired apparent display size; n) When the image is larger than the screen, the stereoscopic image scaled in step m) is adapted to the screen by cropping the image.

11. The method according to claim 9 or 10, wherein the method comprises, after step m), scaling the stereoscopic image to a second percentage, the second percentage being in the range of 50-150%, particularly in the range of 80-120%, more particularly in the range of 95-105%, even more particularly in the range of 99-101%.

12. The method according to any of claims 9-11, wherein the stereoscopic recording of the second person is represented by a stereoscopic recording of the second person's head, such that the stereoscopic image of the second person is represented by a stereoscopic image of the second person's head.

13. The method according to any one of claims 9 to 12, further comprising: - Provide a recording of the background, obtained by a video camera or stereo camera; - Show to the first person simultaneously on the screen ο a record of the background as a background image; as well as ο a stereoscopic recording of the second person as a stereoscopic foreground image; - optionally scaling the background image to a second desired extent; - When the image is larger than the screen, the background image is cropped to fit the screen.

14. The method according to any one of claims 9 to 13, wherein the method is repeated one or more times.

15. The method according to claim 14 , wherein the method is performed in such a way that, on each repetition of the method, the first person perceives the second person to be at substantially the same depth relative to the second autostereoscopic display device, or at a depth that varies between a lower value and a higher value, wherein the higher value is up to 1.5 times the lower value, in particular up to 1.2 times the lower value, more in particular up to 1.1 times the lower value, even more in particular up to 1.05 times the lower value.

16. The method according to any one of claims 1 to 15, wherein the first and / or second autostereoscopic display device are independently selected from the group consisting of a television, a desktop computer monitor, a laptop computer, a cinema display system, a mobile phone, a display in a car, a tablet computer, and a game console.

Images