METHOD FOR CALIBRATING A DISPLAY DEVICE

Abstract

A method for calibrating a display device (10) is disclosed, wherein the display device (10) is configured to display a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42) are superimposed to form an overall representation (40). A first calibration graphic (51) is displayed in the first image plane (11) and a second calibration graphic (52) is displayed in the second image plane (12), wherein the first and second calibration graphics (51, 52) are configured such that, when superimposed, they are congruent from a predetermined standard perspective. An image capture device (30) is used to determine whether the second calibration graphic (52) is at least partially visible from the standard perspective in addition to the first calibration graphic (51).In the event that the second calibration graphic (52) is at least partially visible, a deviation between the first calibration graphic (51) and the second calibration graphic (52) is determined and, based on the deviation, a correction function is determined such that, by applying the correction function to the second calibration graphic (52), the deviation is compensated at least to the extent that the second calibration graphic (52) is no longer visible.

Description

The present invention relates to a method and a system for calibrating a display device, in particular in a vehicle. In particular, a display device having multiple image planes may be calibrated.Modern vehicles often have a large range of functions, which usually includes various vehicle or comfort functions, such as settings of the navigation system, the air conditioning system, seat settings, lighting settings and the like. Various functions of an infotainment system can also be operated, such as playing music, guiding telephone records, and the like. For the purpose of displaying and controlling the functions, usually at least one display is provided as part of the user interface, for example centrally in the dashboard. Here, the individual functions, menus, and the like may be displayed. These displays are often touch-sensitive, so that the desired function can be controlled by touching the display. For this purpose, the operating elements are designed in such a way that they can be reached and operated with a finger.Modern vehicles, however, often have multiple displays and increasingly larger displays, which may no longer be readily accessible by hand. For example, a plurality of distributed displays or a large display, which optionally extends in a curved manner over the width of the vehicle, can be provided. It is therefore also known to control certain functions in contactless fashion by gestures. For this purpose, a user performs certain predefined gestures in free space, for example in a spatial area of the vehicle cabin in front of the display. Gestures of this type can be detected for this purpose with corresponding sensors, which are capable of determining hand postures and movement of a hand in three-dimensional space.To expand the display of contents on a display, in addition to conventional 2D displays, 3D displays are also known which are capable of making displayed contents appear three-dimensionally. However, three-dimensional representations are often limited to a relatively small display. For example, individual objects in the dashboard may be highlighted directly in front of the driver by generating a 3D effect. In the case of such 3D displays, the three-dimensional effect can be produced, for example, by the combination of two 2D displays. For this purpose, a transparent specular surface can be provided in the display, for example, in order to generate a 3D effect from two two two-dimensional representations matched to one another in the perception of a viewer. The observer sees two superimposed representations through the specular surface.In such 3D displays with two image planes spaced apart from one another, the so-called parallax effect can occur. The parallax effect is a phenomenon that occurs when an object is viewed from different viewing angles, thereby giving an apparent displacement of the object from the background. Depending on the viewing angle, undesirable displacements or distortions of the representation can therefore occur. If the two two-dimensional representations are not correctly matched to one another, erroneous perceptions can also occur, regardless of the viewing angle, which can disturb or even destroy a 3D effect.The present invention is based on the object of providing a method for calibrating a display device having a plurality of image planes.The solution of this object is achieved according to the teaching of the independent claims. Various embodiments and developments of the invention are the subject matter of the dependent claims.A first aspect of the invention relates to a method, in particular a computer-implemented method, for calibrating a display device, wherein the display device is configured to display a first partial representation in a first image plane and a second partial representation in a second image plane spaced apart from the first image plane, such that the first partial representation and the second partial representation form an overall representation in a superimposed manner. In the method, a first calibration graphic is displayed in the first image plane and a second calibration graphic is displayed in the second image plane, wherein the first and second calibration graphics are configured such that they are congruent or at least substantially congruent in a superimposed manner from a predetermined standard perspective. An image capture device is then used to determine whether the second calibration graphic is visible from the standard perspective at least partially in addition to the first calibration graphic. In the event that the second calibration graphic is visible (from the standard perspective) at least in part (e.g. a part that is greater than a predetermined threshold value), a deviation between the first calibration graphic and the second calibration graphic is determined. Based on the deviation, a correction function is then determined such that by applying the correction function to the second calibration graphic, the deviation is compensated at least insofar as the second calibration graphic is no longer visible (from the standard perspective), or at most to a part which is less than or equal to a predetermined threshold value.The aforementioned method according to the first aspect is thus based in particular on the fact that a display device having a plurality of image planes can be calibrated automatically. As explained at the beginning, for the correct representation of, in particular, a three-dimensional content, it is important that the two image planes are correctly aligned with respect to one another, i.e. that the display device is calibrated. According to the method according to the first aspect, this can be realized in a simple manner by displaying a corresponding calibration graphic in each image plane. These are designed such that they appear congruent with correct calibration. If the second calibration graphic is therefore visible less than to a certain extent, preferably not at all, the calibration is correct. Otherwise, a calibration is carried out by means of a correction function. Therefore, no (physical) change in the alignment of at least one of the image planes is necessary, which would have to be carried out manually, for example, but rather the representation in the second image plane (that is to say the second calibration graphic during calibration), that is to say the display, is corrected accordingly. This allows the calibration to be automated and simplified.Congruence of the calibration graphics refers to the view in a standard perspective to be determinable regardless of a viewing angle. However, in other views or in a direct single view, the calibration graphics may have differences. It is understood that the first and / or second calibration graphics can also have portions which are not found in the respective other. For example, the second calibration graph may be only a portion of the first. In this case, parts of one of the graphics that are not found in the other can be ignored for the calibration. In other words, only the part which also has a correspondence in the other calibration graphic is designated as a calibration graphic in particular.As used herein, the term "display device" refers more particularly to a device by which content such as a user interface (GUI) can be graphically presented. This can be, in particular, a display or a screen, in particular in a vehicle. In the sense of the present invention, a display device is configured in particular to display an overall representation which is composed of two partial representations which are displayed in different image planes, in particular at a distance from one another. By the distance of the image planes and the superposition of the corresponding partial representations, a three-dimensional effect can be generated for the user ("3D display"). One of the two image planes can be referred to as "virtual" if the view of the overall representation is obtained by looking through this "virtual" image plane onto an image plane lying behind it. Technically, this can be achieved, for example, by two displays at an angle to one another (for example 90 degrees) with a transparent specular optical element lying in between (for example at 45 degrees).The term "detection device" as used here optionally refers in particular to a device which can detect objects in three-dimensional space and determine their position in a contactless manner. In particular, the sensing device may sense and locate a user's head or eyes. For example, optical methods can be used to detect the position of a user's eyes as the starting point of his viewing direction in space. The detection device can consist of one part or more parts, which can be dependent on which detection area is to be covered. For example, the capturing device may be an image capturing device such as a camera.The term "three-dimensional spatial area" as used herein refers in particular to an area which can be described by three-dimensional coordinates. A position in the three-dimensional spatial region has unique three-dimensional coordinates. The coordinate system can be selected arbitrarily. For example, a coordinate system of the detection device may be selected, or a coordinate system with respect to the display device. In particular, there is a relationship between a detection range of the detection device and the display device in order to be able to assign the position of the hand to corresponding locations on the GUI.As used herein, the term "vehicle" refers specifically to a passenger car, including any type of motor vehicle, hybrid and battery electric vehicles, as well as vehicles such as sedan, vans, buses, trucks, vans, and the like.As used herein, the term "user interface" or "graphical user interface" refers more particularly to a graphical representation of controls associated with a particular function that allow a user to control the function. The user interface (UI) may include "controls" (UIs) that a user can operate, such as input buttons, buttons, icons, buttons, icons, sliders, symbol bars, selection menus, and the like. The (graphical) user interface can also be referred to as a (graphical) user interface.The terms "comprises," "includes," "includes," "has," "has," "with," or any other variant thereof, as used herein, are intended to cover a non-exclusive inclusion. For example, a method or apparatus comprising or having a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a method or apparatus.Further, unless expressly stated to the contrary, "or" refers to an inclusive or not an exclusive "or.". For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).The terms "a" or "an" as used herein are defined in the sense of "one / one or more.". The terms "another" and "a further" and any other variant thereof are to be understood in the sense of "at least one further".The term "plurality" as used herein is to be understood as meaning "two or more".The term "configured" or "configured" to fulfil a specific function (and respective modifications thereof) is to be understood in the sense of the invention that the corresponding device is already present in a configuration or setting in which it can execute the function or it is at least adjustable-i.e. configurable-in such a way that it can execute the function after corresponding setting. The configuration can be effected, for example, by means of a corresponding setting of parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, such that the configuration can be effected by means of a selection of one of these configurations or operating modes.Preferred embodiments of the method are described below, which can be combined with one another as desired and with the further described other aspects of the invention, in each case, unless this is explicitly excluded or technically impossible.In some embodiments, determining the deviation includes determining first landmarks in the first calibration graph and corresponding second landmarks in the second calibration graph. Determining landmarks, which may also be called "landmarks," in the calibration graphics simplifies the calibration process. As will be explained in more detail in the following in particular, the effort can be reduced in this way by only taking into account the landmarks and not the complete calibration graphics. The landmarks occur in pairs, one in each of the two calibration graphics, so that in a view in which the landmarks are aligned, the calibration graphics appear congruent.In some embodiments, determining the correction function includes determining a function to minimize a distance between the first and second landmarks. In particular, the correction function can be selected such that a sum of all distances of corresponding landmarks in the first or second calibration graph is minimized. In the best case, this sum is zero, which means that the landmarks and thus the calibration graphics are congruent, which in the optimal case is the target of the calibration. For this purpose, the ICP algorithm (Iterative Closest Point Algorithm) can be used, for example. The first and second landmarks form point clouds, respectively. It can be assumed that these point clouds are already approximately aligned with one another at the beginning of the calibration.In some embodiments, determining the correction function includes determining a coordinate transformation that maps the second landmarks to the first landmarks. This can be derived in particular from the function just described for minimizing the distances.In some embodiments, the correction function includes at least one of shifting, warping, scaling (resizing), and rotating. Depending on a deviation from an optimal alignment of the two image planes with respect to one another, a calibration to a different extent is required. In the case of a lateral offset of the two image planes, a displacement of the second partial illustration may already be sufficient. If larger adjustments are necessary, the second partial representation may also be distorted, scaled, or rotated.In some embodiments, the correction function is stored as a map to be applied to the second partial image in operation of the display device. After calibration is complete, application of the correction function results in calibrated operation of the display device. The calibration usually only has to be carried out once before the display device is operated for the first time or when the hardware is changed.In some embodiments, the first and second calibration graphics are distinguishable. For example, the calibration graphics may be different in color. This simplifies identification in the image captured by the image capture device during calibration. It is understood that other distinguishing features may also be used. In the case of line patterns, a distinction can be simplified, for example, by a line drawing of one of the lines. By better distinguishing the two calibration graphics, deviations can be more easily determined, which can simplify the calibration process as a whole.In some embodiments, the first and second calibration graphics include first and second patterns of intersecting lines, respectively, such as a grid. In this case, the first and second landmarks may be crossing points of the lines. Grids as calibration graphics simplify the method. It should be understood, however, that any other graphic may serve as the calibration graphic.In some embodiments, the default standard perspective is an orthogonal and / or centered perspective on the display device. In particular, the starting point of the standard perspective can be a predefined point at a specific distance from the display device, more precisely the image planes. For example, for calibration, a camera can be directed centrally centered at a specific distance on the display device.A second aspect of the invention relates to a system for data processing, comprising at least one processor configured to carry out the method according to the first aspect of the invention. The system also has at least one display device which is configured to display a first partial representation in a first image plane and a second partial representation in a second image plane spaced apart from the first image plane, such that the first partial representation and the second partial representation form an overall representation in a superimposed manner, and an image capturing device, such as a camera, which is configured to capture the display device from a standard perspective.In some embodiments of the system, the display device comprises a first partial display device, a second partial display device and an optical element, wherein the first partial display device is configured to display the first partial representation and the second partial display device is configured to display the second partial representation, wherein the optical element is configured to optically superimpose the first and second partial representations. The first and second partial display devices can be arranged in particular at an angle to one another (for example approximately 90 degrees) and at a distance from one another. One of the two partial display devices, in particular the first, can be arranged in such a way that it lies in the field of vision of a user (first image plane). The second image plane can be realized by mirroring the second partial representation displayed by means of the second partial display device. The optical element can therefore be designed as a specular transparent element, which can be arranged at an angle between the partial display devices (for example at approximately 45 degrees), so that the user looks at the first image plane through the optical element (through which the second image plane is virtually generated). This superposition produces the overall representation.The image acquisition device for calibration can be directed onto the display device for the calibration process, for example be placed centrally in front of it. However, it can also be provided, in particular in a vehicle, to use a camera which is already installed in the vehicle for other purposes and which has the display device in the field of view.A third aspect of the invention relates to a computer program having instructions which, when executed on a system according to the second aspect, cause the latter to execute the method according to the first aspect.The computer program can be stored in particular on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This may be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local area network. In addition, the computer program can have a plurality of interacting individual program modules.The system according to the second aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the system can also be configured to access a computer program available externally, for example on one or more servers or other data processing units, via a communication connection, in particular to exchange with this data which are used during the course of the method or computer program or represent outputs of the computer program.The features and advantages explained with reference to the first aspect of the invention also apply correspondingly to the further aspects of the invention.Further advantages, features and possible applications of the present invention are evident from the following detailed description in conjunction with the drawings.The following shows: FIG. 1 schematically shows a display device having two image planes; FIG. 2 schematically shows a structure of a display device; FIG. 3 shows examples of an overall representation from a standard perspective and a user perspective; FIGS. 4-7 show examples of calibration graphics; and FIG. 8 schematically shows a structure for calibrating a display device.Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.FIG. 1 schematically shows a display device 10 having two image planes 11, 12. The top view shows a view of the display device 10, for example from a user perspective. The bottom view illustrates how a user 1 is looking at the display device 10, looking through the "virtual" second image plane 12 onto the first image plane 11 behind it. Depending on the user perspective, a three-dimensional effect can thus be generated. This is achieved, as is schematically illustrated in FIG. 2, by an optical element 24 which is arranged between two partial display devices 21, 22 which are designed as conventional two-dimensional displays.The optical element 24 is designed as a semi-transparent mirror surface, so that the two partial display devices 21, 22 are perceived as two parallel image planes 11, 12 which are arranged at a small distance 13 from one another and directly one behind the other. In this way, an overall representation is produced by superimposing a first partial representation, which is displayed by the first partial display device 21, and a second partial representation, which is displayed by the second partial display device 22, which the user 1 perceives three-dimensionally. Since the two partial display devices 21, 22 are not actually arranged parallel to one another, but only the image planes 11, 12 are perceived as such by the user 1, the second image plane 12 can be referred to as "virtual", in particular. The perceived distance 13 between the two partial display devices 21, 22 can also be referred to as a virtual distance. Since this arises due to an actual distance 23 between the second partial display device 22 and the semi-transparent optical element 24, this distance nevertheless also exists physically.Due to the virtual distance 13 between the two image planes 11, 12, a parallax effect arises. The strength of the parallax effect depends on the user's perspective, i.e. the position of the user's 1 eyes and on the positioning or orientation of the second (virtual) image plane 12 with respect to the first (real) image plane 11.In addition, a natural parallax effect occurs which arises due to the distance between the human eyes (and the brain) and which generates the 3D deception. However, this effect is not the subject matter of the present invention here and is to be distinguished from the abovementioned parallax effect which arises as a result of the distance 13 between the image planes 11, 12. It is assumed that both image planes 11, 12 are completely parallel and the 2D coordinates of the projection of an arbitrary target point of one image plane 12 onto the other image plane 11 are known. For this purpose, it is necessary for the two partial display devices 21, 22 and thus the image planes 11, 12 to be correctly aligned with respect to one another. For this purpose, the display device 10 is calibrated. In the case of a fixed construction, calibration is only necessary once. If the structure of the display device 10 is changed, recalibration is required.FIG. 3 a illustrates on the left an overall representation 40, which is produced by superposition of a first partial representation 41 and a second partial representation, from a standard perspective (centered and perpendicular), wherein the calibration is correct here. The right-hand representation illustrates a view from a user perspective, wherein a three-dimensional impression arises as a result of the parallax effect.In FIGS. 3 band 3 c, however, the two image planes 11, 12 are not correctly aligned with respect to one another, so that calibration is required, wherein the left view shows the overall representation in each case from the standard perspective, and the right view from a user perspective. In FIG. 3 b, the second image plane 12 (and thus the second partial illustration 42) is not parallel but at an angle to the first image plane 11 (illustrated in exaggerated form). In FIG. 3 c, the second image plane 12 is not centered with respect to the first image plane, but offset (likewise illustrated in exaggerated form). In the right-hand views in each case, it can be seen that an incorrect impression is then also produced from the user perspective.In order to calibrate the two partial display devices 21, 22 or the two image planes 11, 12, the partial representations 41, 42 must be matched to one another. For this purpose, a calibration graphic, for example in the form of a normalized grid, is shown on each of the partial display devices 21, 22. The calibration graphics 51, 52 preferably differ in their color and / or structure (cf. FIGS. 5 to 7 ) so that unambiguous identification is possible.FIG. 4 shows the view when calibration is successful. A grid (grid) 50 can be seen, which arises by exactly superimposing the first calibration graphic 51 (in the first image plane) and the second calibration graphic 52 (in the second image plane). The two calibration graphics 51, 52 are indistinguishable since they are correctly superimposed. Due to the distance 13, the superimposed second calibration graphic 52 is actually smaller than the first calibration graphic 51 since it appears closer to the user 1 (viewer). Therefore, the distance 13 is taken into account and should be known in advance.If the distance 13 is not taken into account, the impression illustrated in FIG. 5 arises. Although the calibration graphics 51, 52 are centered, the second calibration graphics 52 appear larger due to the smaller distance from the viewer. This is also the case when the distance between the two image planes 11, 12 is not correct, so that calibration is necessary. FIG. 6 shows a situation in which the image planes 11, 12 are not parallel (cf. FIG. 3 b). The impression illustrated in FIG. 7 arises in the case of a lateral offset of the image planes 11, 12 (cf. FIG. 3 c ). Calibration is also required in these cases.In order to calibrate the calibration graphics 51, 52 and thus the image planes 11, 12, i.e. to align them correctly with respect to one another, either the position and / or the alignment of one of the partial display devices 21, 22 must be changed manually in order to produce the view illustrated in FIG. 4, or the partial representation generated by one of the partial display devices 21, 22 (e.g. the second partial representation 42 or, during calibration, the second calibration graphics 52) must be adjusted with the aid of conventional image transformation algorithms (displacement, inclination, rotation, etc.). During the calibration process, the calibration graphics 51, 52 should be displayed as a reference until the process is complete.FIG. 8 schematically shows a structure for automatic calibration. Here, an image capturing device 30 (e.g. a camera) is installed at a known distance and centered with respect to the display device 10, in particular e.g. the partial display device 21. For this purpose, landmarks 53, e.g. the crossing points of the grid lines, are determined in the calibration graphics. The centered view at a known distance allows the system to capture the calibration graphic 51 undistorted and with expected dimensions. Likewise, the calibration graphic 52 is captured by the camera (not shown in FIG. 8 ). In order to automatically align the gratings of the calibration graphics 51, 52, a known algorithm such as an ICP (iterative closest point) or similar algorithm may be applied to the input image of the camera system. The ICP calculates the necessary transformation, which can then be mapped to a parameter set that yields the image transformation algorithm (e.g. a coordinate transformation), which is to be applied during operation to one of the partial representations (e.g. the second partial representation 42). The parameter set can also indicate the changes required for adjusting the partial display device 21, 22.While at least one exemplary embodiment has been described above, it should be noted that a great number of variations thereto exist. It should also be appreciated that the described example embodiments are merely non-limiting examples, and are not intended to limit the scope, applicability, or configuration of the apparatuses and methods described herein. Rather, the foregoing description will provide guidance to those skilled in the art to implement at least one exemplary embodiment, it being understood that various changes may be made in the operation and arrangement of elements described in an exemplary embodiment without departing from the subject matter set forth in the appended claims, as well as the legal equivalents thereof.LIST OF REFERENCE CHARACTERS1 User 10 Display device 11 First image plane 12 Second image plane 13 Distance 21 First partial display device 22 Second partial display device 23 Distance 24 Optical element 30 Image capturing device 40 Overall representation 41 First partial representation 42 Second partial representation 50 Calibrated raster 51 First calibration graphic 52 Second calibration graphic 53 Landmarks

Claims

Method for calibrating a display device (10), wherein the display device (10) is configured to display a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42) result in an overall representation (40) in a superimposed manner, wherein the method comprises: - displaying a first calibration graphic (51) in the first image plane (11) and a second calibration graphic (52) in the second image plane (12), wherein the first and second calibration graphics (51, 52) are designed such that they are congruent in a superimposed manner from a predetermined standard perspective; - determining, by means of an image acquisition device (30), whether the second calibration graphic (52) is visible from the standard perspective at least partially in addition to the first calibration graphic (51); and - in the case that the second calibration graphic (52) is at least partially visible: - determining a deviation between the first calibration graphic (51) and the second calibration graphic (52); and - determining a correction function based on the deviation such that by applying the correction function to the second calibration graphic (52) the deviation is compensated at least insofar as the second calibration graphic (52) is no longer visible.The method of claim 1, wherein determining the deviation comprises determining first landmarks (53) in the first calibration graphic (51) and corresponding second landmarks in the second calibration graphic (52).The method of claim 2, wherein determining the correction function comprises determining a function to minimize a distance between the first and second landmarks (53).The method of claim 2 or 3, wherein determining the correction function comprises determining a coordinate transformation mapping the second landmarks to the first landmarks (53).The method of any preceding claim, wherein the correction function comprises at least one of shifting, warping, and scaling.Method according to one of the preceding claims, wherein the correction function is stored as a mapping which is to be applied to the second partial representation (42) during operation of the display device (10).The method of any preceding claim, wherein the first and second calibration graphics (51, 52) are distinguishable.The method of any preceding claim, wherein the first and second calibration graphics (52, 53) comprise a first and second pattern of intersecting lines, respectively.The method of any preceding claim, wherein the predetermined standard perspective is an orthogonal and / or centered perspective on the display device.A system for data processing, comprising at least one processor configured to execute the method according to any one of the preceding claims, as well as at least one display device (10) configured to display a first partial representation (41) in a first image plane (11) and a second partial representation (42) in a second image plane (12) spaced apart from the first image plane (11), such that the first partial representation (41) and the second partial representation (42) form an overall representation (40) in a superimposed manner, and an image acquisition device (30) configured to acquire the display device (30) from a standard perspective.The system of claim 10, wherein the display device (10) comprises a first partial display device (21), a second partial display device (22) and an optical element (24), wherein the first partial display device (21) is configured to display the first partial representation (41) and the second partial display device (22) is configured to display the second partial representation (42), wherein the optical element (24) is configured to optically superimpose the first and second partial representations (41, 42).The system according to claim 10 or 11, wherein the detection device (30) comprises at least one camera.A computer program comprising instructions which, when executed on a system according to any one of claims 10 to 12, cause the system to carry out the method according to any one of claims 1 to 9.

Images