METHOD FOR CONTROLLING A DISPLAY DEVICE

Abstract

A method for controlling a display device (10) is disclosed, wherein the display device (10) displays a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42), when superimposed, result in an overall representation (40) from a predetermined standard perspective (50). A user perspective (51) is determined, wherein for this purpose a starting point of a viewing direction of a user (1) onto the display device (10) is detected by means of a detection device (30).A corrected second partial representation (42') is then determined as a function of a deviation of the user perspective (51) from the standard perspective (50), wherein for this purpose at least a partial area of ​​the second partial representation (42) is changed in such a way that the first partial representation (41) and the corrected second partial representation (42') superimposed from the user perspective (50) result in the overall representation (40) at least with regard to the changed partial area of ​​the second partial representation (42).

Description

The present invention relates to a method and a system for controlling a display device, in particular in a vehicle. In particular, a user perspective on the display device may be taken into account in order to compensate for optical effects such as a parallax effect.Modern vehicles often have a large range of functions, which usually includes various vehicle or comfort functions, such as settings of the navigation system, the air conditioning system, seat settings, lighting settings and the like. Various functions of an infotainment system can also be operated, such as playing music, guiding telephone records, and the like. For the purpose of displaying and controlling the functions, usually at least one display is provided as part of the user interface, for example centrally in the dashboard. Here, the individual functions, menus, and the like may be displayed. These displays are often touch-sensitive, so that the desired function can be controlled by touching the display. For this purpose, the operating elements are designed in such a way that they can be reached and operated with a finger.Modern vehicles, however, often have multiple displays and increasingly larger displays, which may no longer be readily accessible by hand. For example, a plurality of distributed displays or a large display, which optionally extends in a curved manner over the width of the vehicle, can be provided. It is therefore also known to control certain functions in contactless fashion by gestures. For this purpose, a user performs certain predefined gestures in free space, for example in a spatial area of the vehicle cabin in front of the display. Gestures of this type can be detected for this purpose with corresponding sensors, which are capable of determining hand postures and movement of a hand in three-dimensional space.To expand the display of contents on a display, in addition to conventional 2D displays, 3D displays are also known which are capable of making displayed contents appear three-dimensionally. However, three-dimensional representations are often limited to a relatively small display. For example, individual objects in the dashboard may be highlighted directly in front of the driver by generating a 3D effect. In the case of such 3D displays, the three-dimensional effect can be produced, for example, by the combination of two 2D displays. For this purpose, a transparent specular surface can be provided in the display, for example, in order to generate a 3D effect from two two two-dimensional representations matched to one another in the perception of a viewer. The observer sees two superimposed representations through the specular surface.In such 3D displays with two image planes spaced apart from one another, the so-called parallax effect can occur. The parallax effect is a phenomenon that occurs when an object is viewed from different viewing angles, thereby giving an apparent displacement of the object from the background. In particular in the case of larger 3D displays, displacements or distortions of the display can therefore occur, depending on the user's perspective, i.e. the viewing angle from which a user views the display, which are usually undesirable.It is an object of the present invention to provide a method of controlling a display device having a plurality of image planes. In particular, a user perspective on the display device is to be taken into account in this case.The solution of this object is achieved according to the teaching of the independent claims. Various embodiments and developments of the invention are the subject matter of the dependent claims.A first aspect of the invention relates to an, in particular computer-implemented, method for controlling a display device, in particular in a vehicle, wherein the display device displays a first partial representation in a first image plane and a second partial representation in a second image plane spaced apart from the first image plane, such that the first partial representation and the second partial representation form an overall representation superimposed from a predefined standard perspective. In the method, a user perspective is determined, wherein for this purpose a starting point of a viewing direction of a user onto the display device is detected by means of a detection device, in particular in a three-dimensional spatial region. A corrected second partial representation is then determined as a function of a deviation of the user perspective from the standard perspective, wherein for this purpose at least a partial region of the second partial representation is changed in such a way that the first partial representation and the corrected second partial representation result in the overall representation superimposed from the user perspective at least with respect to the changed partial region of the second partial representation. Finally, the first partial representation and the corrected second partial representation are displayed.The aforementioned method according to the first aspect is thus based in particular on the fact that the display of contents on the display device is corrected as a function of a user perspective. An undesirable effect which may arise due to the distance between the two image planes is the parallax effect. Therefore, according to the method of the first aspect, it is provided that a starting point of the line of sight of the user is determined so as to ascertain the current user perspective on the display device. If the user perspective deviates from a standard perspective, for example a perpendicular view of the display device, i.e. if the user is looking at the display device at an angle, for example, this can be compensated for by correcting the second partial representation. This correction can then be applied to the display device at each glance, so that the user always perceives the correct overall representation independently of the viewing angle. In other words, the overall representation always appears the same for the user at least for the sub-area.As used herein, the term "display device" refers more particularly to a device by which content such as a user interface (GUI) can be graphically presented. This can be, in particular, a display or a screen, in particular in a vehicle. In the sense of the present invention, a display device is configured in particular to display an overall representation which is composed of two partial representations which are displayed in different image planes, in particular at a distance from one another. By the distance of the image planes and the superposition of the corresponding partial representations, a three-dimensional effect can be generated for the user ("3D display"). One of the two image planes can be referred to as "virtual" if the view of the overall representation is obtained by looking through this "virtual" image plane onto an image plane lying behind it. Technically, this can be achieved, for example, by two displays at an angle to one another (for example 90 degrees) with a transparent specular optical element lying in between (for example at 45 degrees).The term "detection device" as used herein refers in particular to a device which can detect contactless objects in three-dimensional space and determine their position. In particular, the sensing device may sense and locate a user's head or eyes. For example, optical methods can be used to detect the position of a user's eyes as the starting point of his viewing direction in space. The detection device can consist of one part or more parts, which can be dependent on which detection area is to be covered. For example, the capturing device may be an image capturing device such as a camera.The term "three-dimensional spatial area" as used herein refers in particular to an area which can be described by three-dimensional coordinates. A position in the three-dimensional spatial region has unique three-dimensional coordinates. The coordinate system can be selected arbitrarily. For example, a coordinate system of the detection device may be selected, or a coordinate system with respect to the display device. In particular, there is a relationship between a detection range of the detection device and the display device in order to be able to assign the position of the hand to corresponding locations on the GUI.As used herein, the term "vehicle" refers specifically to a passenger car, including any type of motor vehicle, hybrid and battery electric vehicles, as well as vehicles such as sedan, vans, buses, trucks, vans, and the like.As used herein, the term "user interface" or "graphical user interface" refers more particularly to a graphical representation of controls associated with a particular function that allow a user to control the function. The user interface (UI) may include "controls" (UIs) that a user can operate, such as input buttons, buttons, icons, buttons, icons, sliders, symbol bars, selection menus, and the like. The (graphical) user interface can also be referred to as a (graphical) user interface.The terms "comprises," "includes," "includes," "has," "has," "with," or any other variant thereof, as used herein, are intended to cover a non-exclusive inclusion. For example, a method or apparatus comprising or having a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a method or apparatus.Further, unless expressly stated to the contrary, "or" refers to an inclusive or not an exclusive "or.". For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).The terms "a" or "an" as used herein are defined in the sense of "one / one or more.". The terms "another" and "a further" and any other variant thereof are to be understood in the sense of "at least one further".The term "plurality" as used herein is to be understood as meaning "two or more".The term "configured" or "configured" to fulfil a specific function (and respective modifications thereof) is to be understood in the sense of the invention that the corresponding device is already present in a configuration or setting in which it can execute the function or it is at least adjustable-i.e. configurable-in such a way that it can execute the function after corresponding setting. The configuration can be effected, for example, by means of a corresponding setting of parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, such that the configuration can be effected by means of a selection of one of these configurations or operating modes.Preferred embodiments of the method are described below, which can be combined with one another as desired and with the further described other aspects of the invention, in each case, unless this is explicitly excluded or technically impossible.In some embodiments, a position of the user's eyes in three-dimensional space is determined as a starting point of the user's direction of gaze. The position of the user's eyes can be determined, for example, by means of image recognition. The detection device can then be in particular an image detection device. On the basis of the position of the user's eyes, the user perspective, i.e. its viewing angle onto the display device, can be determined. If a standard starting point, e.g. centered at a certain distance in front of the display device, is known for the standard perspective, a deviation and thus a correction to be carried out for the second partial representation can be determined in a simple manner by comparison with the eye position.In some embodiments, determining the user perspective includes determining a position and orientation of the user's head in three-dimensional space. In addition or alternatively to determining the position of the eyes, the user perspective on the display device can be determined in this way. The position and orientation of the head can also be determined by means of image analysis, in particular if a camera is used as the detection device. However, a 3D sensor can also be used, such as a ToF camera.In some embodiments, the starting point of the user's direction of gaze is determined continuously, i.e. in real time, wherein the corrected second partial representation is determined continuously accordingly. By displaying the representation corrected in real time, it can be ensured that the user always sees the correct overall representation without undesired parallax effects, which is generated on the basis of his viewing angle (the user perspective) on the display device by corresponding correction of the second partial representation.In some embodiments, altering the portion of the second portion comprises at least one of shifting, warping, and scaling the portion of the second portion. In the simplest case, shifting may already be sufficient to compensate for the parallax effect arising from the deviation between the user's perspective and the standard perspective. In the case of larger deviations or to increase the accuracy, it may be necessary to carry out warping or scaling (i.e. a change in size) of the sub-region of the second partial illustration to be changed.It is understood that the determination of the corrected second partial representation depends not only on the user perspective, but also on a distance of the first and second image planes, or even if the image planes are not parallel, on a position and orientation of the image planes relative to one another. However, these values are constant in a stable hardware setup that is usually assumed.In some embodiments, the method further comprises determining a first portion of the overall representation that is to appear unchanged from the standard perspective and the user perspective and a second portion of the overall representation. In this case, the determination of the corrected second partial representation can then take place in such a way that only a partial region of the second partial representation is changed, which partial region contributes to the first part of the overall representation, so that the first part of the overall representation from the standard perspective and the user perspective appears unchanged. Furthermore, the determination of the corrected second partial representation can then take place in such a way that a partial region of the second partial representation remains unchanged, which contributes to the second part of the overall representation, so that the second part of the overall representation appears different from the standard perspective and the user perspective. In other words, it can be provided that the complete overall representation, which results from superposition of the first and second partial representations, should not appear unchanged from each perspective. It can therefore be provided to undertake the change of the second partial representation only in partial regions in order to compensate for the deviation of the user perspective from the standard perspective.In some of the embodiments just described, the determination of the first and second parts of the overall representation is carried out depending on a semantic content of the overall representation, which is determined by image analysis. For example, with the aid of artificial intelligence, regions of the overall representation can thus be determined which are intended to appear the same (i.e. unchanged) for the user from each perspective. This can be the case, for example, when the second partial illustration is intended to be used to emphasize a corresponding element from the first partial illustration. For such regions, a correction of the second partial representation is then necessary, as described. If, on the other hand, a three-dimensional effect is intended, the parallax effect produced by the distance of the image planes may be desired, so that it may be provided for such contents not to perform a corresponding compensation depending on the viewing angle, i.e. that no correction of the second partial representation is then necessary for such regions.A second aspect of the invention relates to a system for data processing, comprising at least one processor configured to carry out the method according to the first aspect of the invention. The system also has at least one display device which is configured to display a first partial representation in a first image plane and a second partial representation in a second image plane spaced apart from the first image plane, such that the first partial representation and the second partial representation form an overall representation in a superimposed manner, and a detection device which is configured to detect a starting point of a viewing direction of a user, in particular in a three-dimensional spatial region.In some embodiments of the system, the display device comprises a first partial display device, a second partial display device and an optical element, wherein the first partial display device is configured to display the first partial representation and the second partial display device is configured to display the second partial representation, wherein the optical element is configured to optically superimpose the first and second partial representations. The first and second partial display devices can be arranged in particular at an angle to one another (for example approximately 90 degrees) and at a distance from one another. One of the two partial display devices, in particular the first, can be arranged in such a way that it lies in the field of vision of a user (first image plane). The second image plane can be realized by mirroring the second partial representation displayed by means of the second partial display device. The optical element can therefore be designed as a specular transparent element, which can be arranged at an angle between the partial display devices (for example at approximately 45 degrees), so that the user looks at the first image plane through the optical element (through which the second image plane is virtually generated). This superposition produces the overall representation.In some embodiments of the system, the capturing device comprises at least one image capturing device, in particular a camera. By means of a camera, the position of the eyes of a user in three-dimensional space can be determined in a simple manner. One camera or a plurality of cameras can be provided. The camera may be an infrared camera. The at least one camera is advantageously a time-of-flight camera (ToF camera). By using such a 3D sensor device, the position, the orientation and movements of the user's head in three-dimensional space can be directly detected. 2D sensors can also be combined accordingly.A third aspect of the invention relates to a computer program having instructions which, when executed on a system according to the second aspect, cause the latter to execute the method according to the first aspect.The computer program can be stored in particular on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This may be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local area network. In addition, the computer program can have a plurality of interacting individual program modules.The system according to the second aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the system can also be configured to access a computer program available externally, for example on one or more servers or other data processing units, via a communication connection, in particular to exchange with this data which are used during the course of the method or computer program or represent outputs of the computer program.The features and advantages explained with reference to the first aspect of the invention also apply correspondingly to the further aspects of the invention.Further advantages, features and possible applications of the present invention are evident from the following detailed description in conjunction with the drawings.The following shows: FIG. 1 schematically shows a display device having two image planes; FIG. 2 schematically shows a structure of a display device; FIG. 3 shows an example of an overall representation from a standard perspective; FIG. 4 shows the representation from FIG. 3 from a user perspective without correction; FIG. 5 shows the representation from FIG. 3 from a user perspective with correction; and FIG. 6 schematically shows the mode of operation of the correction.Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.FIG. 1 schematically shows a display device 10 having two image planes 11, 12. The top view shows a view of the display device 10, for example from a user perspective. The bottom view illustrates how a user 1 is looking at the display device 10, looking through the "virtual" second image plane 12 onto the first image plane 11 behind it. Depending on the user perspective, a three-dimensional effect can thus be generated. This is achieved, as is schematically illustrated in FIG. 2, by an optical element 24 which is arranged between two partial display devices 21, 22 which are designed as conventional two-dimensional displays.The optical element 24 is designed as a semi-transparent mirror surface, so that the two partial display devices 21, 22 are perceived as two parallel image planes 11, 12 which are arranged at a small distance 13 from one another and directly one behind the other. In this way, an overall representation is produced by superimposing a first partial representation, which is displayed by the first partial display device 21, and a second partial representation, which is displayed by the second partial display device 22, which the user 1 perceives three-dimensionally. Since the two partial display devices 21, 22 are not actually arranged parallel to one another, but only the image planes 11, 12 are perceived as such by the user 1, the second image plane 12 can be referred to as "virtual", in particular. The perceived distance 13 between the two partial display devices 21, 22 can also be referred to as a virtual distance. Since this arises due to an actual distance 23 between the second partial display device 22 and the semi-transparent optical element 24, this distance nevertheless also exists physically.Due to the virtual distance 13 between the two image planes 11, 12, a parallax effect arises. The strength of the parallax effect depends on the user's perspective, i.e. the position of the user's 1 eyes and on the positioning or orientation of the second (virtual) image plane 12 with respect to the first (real) image plane 11.In addition, a natural parallax effect occurs which arises due to the distance between the human eyes (and the brain) and which generates the 3D deception. However, this effect is not the subject matter of the present invention here and is to be distinguished from the abovementioned parallax effect which arises as a result of the distance 13 between the image planes 11, 12. It is assumed that both image planes 11, 12 are completely parallel and the 2D coordinates of the projection of an arbitrary target point of one image plane 12 onto the other image plane 11 are known.FIG. 3 illustrates an overall representation 40 which results from superposition of a first representation 41 (in the first image plane 11) and a second partial representation 42 (in the second image plane 12). The overall representation 40 is shown in FIG. 3 from a standard perspective, in particular a centered perpendicular perspective of the display device 10. In FIG. 4, this height map is shown from a user perspective, wherein the three-dimensional effect can be seen. On the other hand, when the overall display 41 aims at emphasizing edges or the convex hull of an object, the parallax effect needs to be removed, i.e., compensated depending on the user's perspective. A corresponding representation in which the second partial representation 42' has been corrected accordingly is illustrated in FIG. 5. Here, the user always sees the overall representation as in FIG. 3, regardless of the user perspective.FIG. 6 illustrates how the user perspective (the position of the user's eyes or the user's angle of view) is taken into account to correct the second partial representation 42. This is done dynamically in real time. The corrected second partial representation 42' can be compensated by a displacement of a partial region of the second partial representation 42 to be corrected within the image coordinate system. The extent of the displacement depends on the position of the user's eyes and also on the distance between the image planes 11, 12. In the example shown, the points of the virtual image (i.e. of the second partial representation 42) are shifted (cf. target points 42') such that they are correctly superimposed with the first partial representation 41 from the user perspective 51 as well as from the standard perspective 50. In addition to a pure shift, warping and / or scaling of the second partial representation 42 may also be required in order to remove the undesired parallax effect.Since the distance 13 between the two image planes is assumed to be fixed, only the position of the eyes has to be tracked in real time. This is done by means of a detection device 30 (see FIG. 1 ). For example, the eyes of the user can be tracked by means of a camera and corresponding image processing. It can also be provided to track the position and orientation of the user's head, for example by means of a 3D sensor, such as a ToF camera.However, the displacement does not have to be applied to the complete overall representation 40 (or the second partial representation 42). Semantic differentiation (e.g., context-based) may be performed to apply the displacement only to particular image segments. This makes it possible to present some objects with a desired lateral displacement (i.e. without compensating the parallax effect, cf. FIG. 4 ) in order to generate a 3D effect and at the same time to emphasize other objects, wherein the parallax effect is then compensated here as described (cf. FIG. 5 ).While at least one exemplary embodiment has been described above, it should be noted that a great number of variations thereto exist. It should also be appreciated that the described example embodiments are merely non-limiting examples, and are not intended to limit the scope, applicability, or configuration of the apparatuses and methods described herein. Rather, the foregoing description will provide guidance to those skilled in the art to implement at least one exemplary embodiment, it being understood that various changes may be made in the operation and arrangement of elements described in an exemplary embodiment without departing from the subject matter set forth in the appended claims, as well as the legal equivalents thereof.LIST OF REFERENCE CHARACTERS1 User 10 Display device 11 First image plane 12 Second image plane 13 Distance 21 First partial display device 22 Second partial display device 23 Distance 24 Optical element 30 Detection device 40 Overall representation 41 First partial representation 42 Second partial representation 42' Corrected second partial representation 50 Standard perspective 51 User perspective

Claims

Method for controlling a display device (10), wherein the display device (10) displays a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42) result in an overall representation (40) superimposed from a predefined standard perspective (50), wherein the method comprises: - determining a user perspective (51), wherein for this purpose a starting point of a viewing direction of a user (1) onto the display device (10) is detected by means of a detection device (30); determining a corrected second partial representation (42') as a function of a deviation of the user perspective (51) from the standard perspective (50), wherein for this purpose at least a partial region of the second partial representation (42) is changed in such a way that the first partial representation (41) and the corrected second partial representation (42') result in the overall representation (40) superimposed from the user perspective (50) at least with respect to the changed partial region of the second partial representation (42); and displaying the first partial representation (41) and the corrected second partial representation (42').Method according to claim 1, wherein a position of the eyes of the user in three-dimensional space is determined as the starting point of the viewing direction of the user (1).The method according to claim 1 or 2, wherein the determining of the user perspective (51) comprises determining a position and orientation of the head of the user (1) in three-dimensional space.Method according to one of the preceding claims, wherein the starting point of the viewing direction of the user is determined continuously, wherein the corrected second partial representation (42') is determined continuously accordingly.The method of any preceding claim, wherein altering the portion of the second partial representation (42) comprises at least one of shifting, warping, and scaling the portion of the second partial representation (42).The method according to any of the preceding claims, further comprising determining a first part of the overall representation (40), which is to appear unchanged from the standard perspective (50) and the user perspective (51), and a second part of the overall representation (40).Method according to claim 6, wherein the determination of the corrected second partial representation (42') is carried out in such a way that only a partial region of the second partial representation (42) is changed, which contributes to the first part of the overall representation (40), so that the first part of the overall representation (40) from the standard perspective (50) and the user perspective (51) appears unchanged.Method according to claim 6 or 7, wherein the determination of the corrected second partial representation (42') takes place in such a way that a partial region of the second partial representation (42) remains unchanged, which contributes to the second part of the overall representation (40), so that the second part of the overall representation (40) appears different from the standard perspective (50) and the user perspective (51).Method according to one of Claims 6 to 8, wherein the determination of the first and second part of the overall representation (40) takes place as a function of a semantic content of the overall representation (40), which content is determined by image analysis.A system for data processing, comprising at least one processor configured to execute the method according to any one of the preceding claims, as well as at least one display device (10) configured to display a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42) form an overall representation (40) in a superimposed manner, and a detection device (30) configured to detect a starting point of a viewing direction of a user (1).The system of claim 10, wherein the display device (10) comprises a first partial display device (21), a second partial display device (22) and an optical element (24), wherein the first partial display device (21) is configured to display the first partial representation (41) and the second partial display device (22) is configured to display the second partial representation (42), wherein the optical element (24) is configured to optically superimpose the first and second partial representations (41, 42).The system according to claim 10 or 11, wherein the detection device (30) comprises at least one camera and / or a 3D sensor.A computer program comprising instructions which, when executed on a system according to any one of claims 10 to 12, cause the system to carry out the method according to any one of claims 1 to 9.

Images