Sensing system and method for sensing contactless directed user inputs and method for calibrating the sensing system

Abstract

A calibration method for calibrating a sensing system for sensing contactless directed user inputs of a user with respect to a user interface (UI) of a target system, comprises: receiving or sensor-based generating of sensor data which represent, with reference to a first reference frame associated with the sensing system, a sensor-detected input pattern that a user inputs by means of at least one contactless directed user input with respect to the UI in order to copy, by means of a spatial course of the user input, a structure of a predefined recognition pattern depicted by a display device of the UI. A second reference frame is or becomes associated with the display device. Two or more different partial patterns of the input pattern are determined, for each of which: an associated mathematical mapping specification between the two reference frames is determined, on the basis of which mapping specification the partial pattern or the recognition pattern is mapped such that afterward the partial pattern and the recognition pattern are in a common reference frame. According to a mapping error indicating a measure of a deviation between the partial pattern and the recognition pattern, one of the partial patterns is selected and the mapping specification associated with said partial pattern is defined as a calibration variable for calibrating the sensing system for subsequent use.

Description

[0001] Acquisition system and method for acquiring contactless directed user inputs and method for calibrating the acquisition system

[0002] The present invention relates to a detection system and a method for detecting contactless directed user inputs with respect to a user interface of a target system, a method for calibrating the detection system, and a computer program or computer program product for executing at least one of the aforementioned methods.

[0003] Non-contact interaction with a user interface (UI), in particular a graphical user interface (GUI) of a computer or on a computer screen through hand gestures (e.g. pointing, dragging, grasping gestures, etc. regarding one or more GUI objects) and eye behavior (e.g. looking at a GUI object, blinking to select, ...) is a powerful approach for computer UI concepts, especially for environments where traditional interaction (via keyboard or other touch devices) is not possible.

[0004] The concept of such interaction is based on the fact that when a user points at a screen with their finger, the pointing ray (the ray emanating from the hand and extending along the pointing direction) virtually intersects the screen at a specific point. If this intersection is interpreted analogously to a touch on a touchscreen, this forms the basis for touchless screen control UIs.

[0005] Likewise, gaze direction can be used instead of pointing direction as the basis for eye-guided sensing systems. Gaze and gestures can even be combined for more powerful user input.

[0006] In practice, however, in order to reliably translate a sensor-detected gaze direction or gesture into correct user input, it is necessary to calibrate the sensors used for detection or any data processing of the raw data supplied by the sensors with respect to the screen, in particular with respect to its relative position to the sensors. For permanently installed detection systems, a one-time calibration is generally sufficient, as the relative position does not normally change thereafter. For variable detection systems, in which the relative position of the sensors to the screen or other display surface is variable, repeated or even continuous calibration is typically required so that subsequent user inputs can be correctly recorded even if the position has changed.

[0007] It is an object of the invention to further improve the calibration of a detection system for detecting contactless directed user inputs with respect to a user interface of a target system, in particular to enable reliable correct detection even when the relative position between the detection system and the target system or its user interface is variable.

[0008] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0009] A first aspect of the solution presented here concerns a calibration method, in particular a computer-implemented one, for calibrating a detection system for detecting contactless, directed user inputs from a user with respect to a user interface (U1) of a target system, in particular for controlling or configuring the system. The method comprises:

[0010] (i) Receiving or sensor-based generation of sensor data which, with reference to a first reference system assigned to the detection system, represent a sensor-detected input pattern which a user enters using at least one contactless, directed user input with respect to the UI in order to imitate, by means of a spatial progression of the user input, a structure of a predefined recognition pattern graphically displayed by a display device, e.g., a screen, of the UI. In this case, a second reference system is or will be assigned to the display device (generally, the first and second reference systems differ from one another at least with regard to their location (i.e., with regard to their position and / or orientation). The imitation of the recognition pattern can be carried out or have been carried out, in particular, by continuously tracking or point-by-point contactless scanning (e.g., in the sense of sampling) of the recognition pattern;

[0011] (ii) determining two or more different subpatterns of the input pattern;

[0012] (iii) For each of the subpatterns: - Determining, in particular estimating, a mathematical mapping rule, in particular homography, associated with the subpattern between the two reference systems, and mapping the subpattern or the recognition pattern using the mapping rule such that, after the mapping, the subpattern and the recognition pattern are present in a common reference system. The mapping rule can thus define a mapping from the first reference system into the second reference system or, alternatively, in the opposite direction, so that the reference system into which the mapping takes place simultaneously serves as the common reference system.The recognition pattern can in particular already be known to the detection system in advance or can be received dynamically by the detection system as part of the calibration process, for example via a direct or indirect, in particular at least partially wireless, data connection between the target system and the detection system.

[0013] - Determining an aberration, which indicates a measure of a deviation between the partial pattern and the recognition pattern in the common reference system; and

[0014] - selecting one of the partial patterns depending on a comparison of its associated aberration with the respective aberration of at least one other partial pattern; and

[0015] (iv) defining the mapping rule assigned to the selected partial pattern as a calibration variable for calibrating the acquisition system for subsequent use in transforming contactless directed user inputs of the user acquired with respect to the first reference system into corresponding user inputs related to the second reference system with respect to the Ul of the target system.

[0016] The acquisition system can be configured in particular as a vehicle-supported acquisition system (e.g. as a control unit for a vehicle) for acquiring user inputs in a vehicle, while the target system can be configured in particular as a vehicle-external system. The acquisition system can in particular be or comprise a program-controlled computer system. The term "reference system", as used herein, is to be understood in particular as an imaginary spatial or spatio-temporal structure that is required to unambiguously and completely describe the behavior of location-dependent variables. The reference system, in particular its spatial aspect, can in particular be described using a defined coordinate system. The first reference system can in particular be defined relative to a vehicle.The second reference system assigned to the display device can in particular be defined such that a plane spanned by two spatial dimensions of the second reference system coincides with a plane of the display device, in particular a flat screen or projection surface of the display device.

[0017] The term "subpattern," as used herein, is to be understood in particular as a pattern that simultaneously forms a subset of a higher-level pattern, here in particular the recognition pattern. For example, in a grid-like pattern consisting of a set of parallel horizontal lines and a set of vertical lines, each of these sets can be individually interpreted as a subpattern of the grid pattern. According to another example, in a line string composed of a plurality N of linked, in particular straight, line segments, in particular one - even each - subset of M <N, insbesondere verketteten, Liniensegmenten ein Teilmuster des gesamten Linienzugs interpretiert werden.

[0018] The term “calibration quantity” as used herein means information, in particular a mapping specification, which at least partially defines a specific calibration of the acquisition system.

[0019] The terms "comprises," "includes," "includes," "has," "has," "with," or any other variation thereof, as used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such a method or apparatus.

[0020] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive "or" and not an exclusive "or." For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

[0021] The terms "a" or "an" as used herein are defined as "one or more." The terms "another" and "another," and any other variations thereof, are defined as "at least one other."

[0022] The term "plurality" as used herein shall mean "two or more".

[0023] The term “configured” or “set up” to fulfil a specific function (and respective variations thereof) is to be understood within the meaning of the invention that the corresponding device is already in a design or setting in which it can carry out the function or is at least adjustable - i.e. configurable - so that it can carry out the function after being set accordingly. The configuration can be carried out, for example, by appropriately setting parameters of a process sequence or of switches or the like for activating or deactivating functionalities or settings. In particular, the device can have a plurality of predetermined configurations or operating modes, so that configuration can be carried out by selecting one of these configurations or operating modes.

[0024] The aforementioned method according to the first aspect makes it possible, particularly in situations in which the relative spatial position of a sensor used to acquire the sensor data, on the one hand, and the display device or its display area—and thus the relative position of the first and second reference systems to one another—are variable, in particular not known a priori, to achieve a suitable calibration of the acquisition system with respect to user inputs regarding the UI of the target system. Thus, even in such variable situations, reliable operability of the target system can be achieved by means of contactless, directed user inputs regarding the UI.

[0025] Such a case can occur in particular when the detection system is mobile, for example vehicle-based, while the target system is stationary.

[0026] In the following, various exemplary embodiments of the method are first described, which can each be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

[0027] In some embodiments, the calibration method is or is configured or used to calibrate a detection system arranged in or on a vehicle with respect to a target system external to the vehicle. The target system can be a wide variety of systems, in particular infrastructure systems, which are suitable or intended for contactless operation by a user from a vehicle. These include, in particular, parking management systems such as parking meters, barrier systems or other access control systems, drive-in terminals or drive-in kiosks, garage doors or electronic information boards with UI. Since the relative position of the vehicle to a UI of such a target system is typically variable or not clearly known in advance, the calibration method can be used particularly advantageously here in order to determine the relative position of the vehicle orof the user performing the interaction with the target system, so that the subsequent interaction can be reliably and correctly recorded.

[0028] In some embodiments, the sensor-based generation of sensor data is carried out by sensory detection of a free-space gesture, in particular a pointing gesture (e.g., by arm and / or hand) of the user, or by means of head or eye tracking. In this way, contactless, directed user inputs (in particular based on a measured or estimated gaze direction or pointing direction) that can be executed by most people with very high accuracy can be enabled as a basis for the calibration process and for the recording of subsequent interactions of the user with the UI of the target system using the calibrated detection system.

[0029] In some embodiments, the input pattern represents a sensor-detected path that a user enters using at least one contactless, directed user input with respect to the UI in order to imitate, by means of a spatial course of the user input, in particular a “pointing track” or “gaze track” defined by the user input, a structure in the form of a course of a predefined linear recognition pattern that is graphically represented by a display device of the user interface (e.g. screen or projection device for projecting an image onto a projection surface). At least two of the partial patterns of the input pattern each represent a different path section of the overall path. The selection of a linear path as the recognition pattern orThe input pattern is particularly advantageous in that it is particularly easy to imitate, even for an inexperienced user, within the context of contactless, directed user input of the input pattern, and is also particularly easy to process within the context of data processing. Such a path can be defined, in particular, as a concatenation of several straight-line path segments (line string), which allows it to be represented efficiently and easily by the start and end points of the segments, i.e., the end points and corner points (vertices) of the line string.

[0030] In some of these embodiments, the calibration method further comprises smoothing the path before determining the path sections as partial patterns. This allows, in particular, any disturbances in the intended spatial progression of the user input ("blurred" inputs) that occur during the user input to be corrected in order to reduce or even completely avoid incorrect interpretations or unnecessarily large data volumes for representing and processing the path captured as an input pattern. Thus, a more precise or optimized calibration can be achieved based on the smoothed path, which ultimately promotes particularly reliable detection of user inputs at the UI of the target system using the then calibrated detection system.

[0031] According to some embodiments, smoothing the path may, in particular, comprise section-by-section straightening of the path, in which the path is approximated using a plurality of straight-line segments. This allows, in particular, the aforementioned line string to be achieved with the associated advantages. Section-by-section straightening of the path may, in particular, be performed using a Hough transform, which allows a good approximation of the input pattern using a line string to be achieved in a particularly robust and reliable manner.

[0032] In some embodiments, the first reference system is defined as a first virtual two-dimensional surface or such a surface is or will be defined in it. Furthermore, the input pattern on the first virtual surface is defined by a linear beam-shaped projection of the user input onto the first virtual surface, following the respective current direction of the directed user input along its spatial course. The first virtual surface can in particular be the inner surface of a virtual spherical surface or partial spherical surface or lie on such a surface. It is also conceivable for the first virtual surface to be cumulatively composed of several separate partial surfaces. The second reference system is defined as a second virtual surface.For each subpattern, the associated mapping rule defines a mapping between the two virtual surfaces, whereby the surface serving as the target surface of the mapping rule is or will be defined as the common reference system. The use of two-dimensional surfaces, between which the mapping rule defines a transformation, has the particular advantage of reduced complexity compared to the general three-dimensional case. Consequently, the calculation of the mapping rule and its application are also simplified, which can also lead to increased efficiency and, given computing power, to faster, particularly real-time-capable implementation of the calibration procedure and subsequent acquisition of user inputs regarding the UL during interactions with the target system.

[0033] In some of these embodiments, the first reference system is defined as a function of one or more previously detected directed user inputs such that the user inputs detected by these one or more previously detected user inputs correspond to points and / or lines on the first virtual surface. In other words, this means that the first virtual surface is or will be defined such that these one or more previously detected user inputs “hit” the first virtual surface and do not miss it. In this way, the first virtual plane as such can be easily positioned in three-dimensional space based on the one or more user inputs, in particular such that it is easily accessible as a target area for directed user inputs that can be carried out by the user, for example when the user is strapped into a vehicle seat inside a vehicle and is therefore restricted in his freedom of movement and, if applicable,also with regard to the field of vision visible to him outside the vehicle.

[0034] In some embodiments, at least one of the two surfaces is or will be defined such that it has a virtual plane or a limited surface section thereof. This particularly simplifies the implementation of the mapping rule for determining the mapping rule and performing mapping between the reference systems according to the mapping rule.

[0035] In particular, according to some embodiments, the first virtual surface can be defined such that it has a virtual plane or a limited surface section thereof that is orthogonal to the direction of a previously detected directed user input. This allows for a particularly simple and efficient user-controlled definition of the virtual surface, in particular a spatial position of the surface, in space.

[0036] In some embodiments, the first virtual surface is defined to satisfy at least one of the following conditions:

[0037] (i) it forms a boundary surface which encloses a three-dimensional spatial area on at least two sides;

[0038] (ii) it has at least two separate, at least partially non-parallel, partial surfaces;

[0039] (iii) each linear ray projection of a directed user input along its current direction through the first virtual surface intersects the first virtual surface at exactly one point.

[0040] Option (i) is particularly advantageous in that the solid angle range within which the user can successfully execute his directed user inputs (so that they are recorded) can be extended, in particular beyond a half-space (as would be the case with a first virtual surface in the form of a single virtual plane). Option (ii) can also be used to extend the solid angle range available for user inputs, in particular if user inputs should be able to be made into separate solid angle ranges (e.g. through the windscreen of a vehicle and through a side window separated from it by a pillar). Option (iii) serves in particular to effectively avoid ambiguities in input from the outset.

[0041] In some embodiments, the subpatterns are determined such that they are each at least partially represented by a sequence of discrete points on the input pattern, particularly in the case of a path in the input pattern, along the course of the respective path section of the path. This allows for particularly efficient, particularly data-conserving, and fast implementations with low computational effort, since the processing of the input pattern can be reduced to calculations related to the discrete points (especially vertices).

[0042] In some embodiments, to determine the respective imaging error of a partial pattern, a comparison of the recognition pattern with the partial pattern in the common reference system is used on the basis of homogeneous coordinates (in the sense of projective geometry) to describe the recognition pattern on the one hand and the partial pattern on the other. Thus, due to the use of homogeneous coordinates, a spatial dimension can de facto be "compressed", namely along a direction of user input (in particular in the pointing or viewing direction), such that multiples of a coordinate value along the spatial dimension coincide upon transition to homogeneous coordinates, thus reducing complexity. The advantage of homogeneous coordinates lies in particular in the uniform representation of the elements of a projective space, in which distant elements (in particular multiples) no longer play a special role.In addition, by using homogeneous coordinates, all collineations, and thus also parallel shifts, can be described uniformly by linear mappings and thus efficiently and easily processable by matrices.

[0043] A second aspect of the solution presented here concerns a method for capturing contactless, directed user inputs from a user regarding a user interface, U1, of a target system using a capture system. The method comprises:

[0044] (i) calibrating the detection system according to the calibration method of the first aspect;

[0045] (ii) receiving or sensor-based generating input data representing, in the first reference system, which may in particular be defined in a vehicle-based manner, a sensor-detected, contactless directed user input of a user with respect to the UI;

[0046] (iii) transforming the input data into output data representing an input at the Ul corresponding to the user input and transformed into the second reference system, using the mapping rule determined in the calibration procedure or the inverse mapping rule thereto; and

[0047] (iv) Transmitting the output data to the target system, particularly one external to the vehicle, in particular for its operation, control or configuration.

[0048] The method according to the second aspect can thus be used to enable user-input-controlled operation, control or configuration of the target system after calibration according to process (i).

[0049] A third aspect of the solution presented here relates to a detection system for detecting contactless, directed user inputs from a user with respect to a user interface, U1, of a target system, wherein the detection system is configured to execute the calibration method according to the first aspect or the method according to the second aspect. In some embodiments, the detection system further comprises a sensor device for sensorically detecting the contactless user inputs and for generating sensor data and / or input data representing these user inputs in the first reference system. Thus, the sensor system for detecting the user inputs is already part of the detection system itself, for example, integrated into a housing thereof.

[0050] A fourth aspect of the solution presented here relates to a vehicle having a detection system according to the third aspect.

[0051] A fifth aspect of the solution presented here relates to a computer program or computer program product comprising instructions which, when executed on one or more processors of a detection system according to the third aspect, cause the system to carry out the calibration method according to the first aspect or the method according to the second aspect.

[0052] The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules. The modules can in particular be configured or at least be usable in such a way that they can be used in the sense of distributed computing (DC).“Distributed computing” is carried out on different devices (computers or processor units) that are geographically separated from each other and connected via a data network.

[0053] The recording system according to the third aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the recording system can also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0054] The features and advantages explained with regard to the first and second aspects of the solution presented here also apply accordingly to the other aspects of the solution.

[0055] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the figures.

[0056] It shows:

[0057] Fig. 1 schematically shows a first exemplary calibration and operating scenario with a first virtual plane;

[0058] Fig. 2 schematically shows a second exemplary calibration and operating scenario with a first virtual level and a different third virtual level;

[0059] Fig. 3A schematically shows in a perspective view an exemplary spatial relationship between input pattern and recognition pattern;

[0060] Fig. 3B again shows the spatial relationship between input pattern and recognition pattern from Fig. 3A, but here the two patterns are transferred to a common representation level for the purpose of better comparability;

[0061] Fig. 4 is a flowchart illustrating an exemplary embodiment of the calibration method according to the first aspect;

[0062] Fig. 5 schematically shows an illustration of a determination of exemplary partial patterns within the scope of the calibration method from Fig. 4; and

[0063] Fig. 6 is a flowchart illustrating an exemplary embodiment of the method according to the second aspect for detecting contactless directed user inputs. In the figures, identical reference numerals designate identical, similar, or corresponding elements. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are depicted in such a way that their function and general purpose will be understood by those skilled in the art. Connections and couplings between functional units and elements shown in the figures can, unless expressly stated otherwise, also be implemented as an indirect connection or coupling. Functional units, in particular the detection system and its components, can, in particular, be implemented as hardware, software, or a combination of hardware and software.

[0064] Fig. 1 schematically illustrates a first exemplary calibration and operating scenario 100. In this case, operation of a target system is to take place via a user interface U1 of the same from a vehicle 105, in which an occupant of the vehicle makes a user input regarding the U1 in a contactless manner, i.e. without physically coming into contact with the target system. For this purpose, the vehicle 105 has a detection system 110 with a sensor system 115 signal-coupled thereto, in particular with an image sensor system (such as a camera, in particular a video camera). The detection system 110 can in particular be a program-controlled data processing device which is set up or programmed to receive sensor data supplied by the sensor system 115 and to process it within the framework of the method from Fig. 4 (and optionally the method from Fig. 6).Sensor system 115 can detect various user inputs made by the user, in this example the occupant, in the vehicle or from within the vehicle. Such user inputs can be executed, in particular, using one or more free-space gestures, in particular pointing gestures, or through one or more targeted glances (eye tracking) and detected by sensor system 115.

[0065] In the example shown in Fig. 1, the vehicle occupant as user performs a pointing gesture 140 as a contactless free space gesture. The pointing direction corresponds to a direction of the user input, which can thus be referred to as a "directed" user input because it is assigned a specific detected direction overall or time-dependently, i.e. for specific points in time during its execution. It is illustrated in Fig. 1 using a virtual directional ray 145. The vehicle 105 can be assigned its own reference system, in particular a coordinate system, 120. This coordinate system 120 can optionally already be defined as the first reference system. In particular, it can be a Cartesian, in particular right-handed, coordinate system with coordinate axes x, y and z, as illustrated.However, a different variant is described below in which a coordinate system 120a shifted along the Y-axis relative to the coordinate system 120 is selected as the “first” coordinate system.

[0066] The target system can, in particular, be a fixed infrastructure facility, such as a parking meter, a barrier system, an ordering kiosk at a drive-thru store or restaurant, or a car wash. Many other applications or target systems are also conceivable. In the example in Fig. 1, the target system has a display device 130 in the form of a screen with a flat display surface. A second reference system 135, in particular a coordinate system, is assigned to this display surface, so that two axes of the coordinate system run parallel to the display surface or even lie on it. This can also, in particular, be a Cartesian coordinate system with coordinate axes x', y', and z', as illustrated. The coordinates can, in particular, be standardized with respect to the screen size.

[0067] Since the relative position of the vehicle 105 to the target system and thus to the second reference system 135 is typically variable due to the mobility of the vehicle 105 and is also not known a priori, the relative position of the two reference systems 120 and 135 is also typically variable and not known in advance, ie before execution of a calibration method.

[0068] In order to be able to perform a transformation H of a user input 140 detected by the vehicle-supported sensor system 115, which is thus anchored in the reference system 120, into the second reference system of the target system using the acquisition system 110, the acquisition system 110 must be calibrated. This transformation is first determined before user inputs can then be reliably performed based on the calibration.

[0069] To determine the transformation H, a first virtual surface 125 anchored in the reference system 120 and a second virtual surface anchored in the second reference system 135, which in the present example coincides with the display area of ​​the screen 130, are defined. To simplify the calculation of the transformation, it is expedient if both virtual surfaces 125 and 130 are planar, i.e., virtual planes or limited sections thereof.

[0070] A second exemplary calibration and operating scenario 200, which is expanded in comparison, is illustrated in Fig. 2. This figure largely corresponds to Figure 1, but the first virtual surface is divided into two separate, planar partial surfaces 125a and 125b, which are angled to one another (first virtual plane and a different third virtual plane). In this way, the spatial area into which directional rays of the user input 140 can point, so that they strike the first virtual surface, and thus one of the two partial surfaces 125a and 125b at an intersection point 150, can be expanded, in particular beyond the half-space defined by a single planar partial surface.The second partial surface 125b is in turn assigned its own reference system, in particular a Cartesian or other coordinate system 155 with coordinates x", y", z", expediently in the case of a flat partial surface such that two axes of this reference system run parallel to or in the second partial surface 125b.

[0071] The first partial surface 125 can also have more than two partial surfaces and / or in particular can also be curved, for example into a spatially closed shape surrounding the vehicle or the user, for example in the shape of a sphere or a section thereof.

[0072] Fig. 3A shows a perspective view of a relative position of the two virtual planes 125 and 130 according to the example from Fig. 1. The user performs a moving user input, in which they contactlessly imitate a recognition pattern 310 displayed on the display device 130 in the form of a linear path (here, for example, approximately "Z"-shaped) with the corner points V1 to V4. The imitation can be carried out, in particular, by the user pointing sequentially in discrete steps individually at the corner points V1 to V4, or by continuously "following" the path.

[0073] Due to the generally different position and orientation of the two virtual planes 125 and 130, a distortion regularly occurs between the recognition pattern 310 and a corresponding input pattern 305, which is defined in the first virtual surface (plane) 125 by the discrete or continuous sequence of intersection points of the directional rays 145 (only two of which are shown as examples) during the execution of the user input. The relative position of the vertices V'1 to V'4 corresponding to the corner points V1 to V4 of the recognition pattern thus generally differs from the relative position of the corner points V1 to V4 due to the distortion.

[0074] In Fig. 3B, the recognition pattern 310 and the corresponding input pattern 305 are again transferred to a common representation level in the sense of a comparison 301, so that the effect of the distortion is clearly visible.

[0075] In order to ensure correct user inputs with respect to a user interface Ul of the target system defined in relation to the display area 130, it is therefore necessary, as part of the calibration, to determine, in particular to calculate or estimate, the above-mentioned transformation H in such a way that it compensates for this distortion at least to a good approximation.

[0076] A suitable calibration method 400 is illustrated in Fig. 4 as an exemplary embodiment. The method 400, which is explained below with additional exemplary reference to Figs. 1, 3, and 5, comprises a process 405 in which the detection system 110 receives sensor data from the sensor system 115. If, however, the sensor system 115 is counted as a component of the detection system 110, the process 405 corresponds to the generation of the sensor data by the detection system 110, including its sensor system 115 itself.

[0077] The sensor data are related to the reference system 120 or already the reference system 120a and describe an input pattern 305 detected by the sensor system 115, which the user generates using the contactless, directed user input 140. This is done, as illustrated in Fig. 3, by imitating a linear path, in particular a line-shaped path (a chain of straight segments), displayed on the display device 130 as a recognition pattern 310, with their user input 140, in particular by following it or by sequentially pointing to the corner points in an order determined by the course of the path. In this process, the intersection points of directional rays of the user input with the first virtual plane 125 are determined, which in turn define the input pattern 305 as a line or path on the first virtual plane 125.

[0078] Typically, the input pattern 305 will only approximately correspond to the recognition pattern 310, in particular because contactless directed user inputs, such as free space gestures or eye movements that can be detected by eye tracking, cannot be precisely controlled by the user and thus naturally exhibit fluctuations or are subject to tolerance.

[0079] In Fig. 5, as part of an illustration 500 for determining partial patterns, an exemplary course of a path-shaped input pattern 305 is illustrated, such as a user could enter according to their user input 140 when tracing a path-shaped recognition pattern 310 with the corner points V1 to V4 displayed on a screen 130 of the target system. The input pattern 305 is similar to the line-shaped recognition pattern 310 constructed from straight-line segments, but does not exactly match it. In order to improve the quality of the further calibration to be determined, the input pattern 305 is therefore straightened in a further process 410 within the method 400 (cf. Fig. 4). This can be done in particular by means of a so-called Hough transformation, as is known from the field of image processing.As a result of this straightening, a line-shaped path formed from straight, linked segments results as a straightened input pattern 305a on the virtual layer 125.

[0080] Starting from the straightened input pattern 305a, in a further process 415 of the method 400 N different sub-patterns P i with i = 1 , ...,N, with N > 1 , in this example specifically path segments. Each of the path segments P i represents a subset of the points of the input pattern 305a, a part of the input pattern 305a, and can, in particular, comprise several interlinked, rectilinear segments of the input pattern 305a. The lower part of Fig. 5 illustrates an exemplary result of the process 415 for the case N = 3, where each path section has the same number M of vertices as the recognition pattern 310. In the present example, M = 4.

[0081] Then, in the further course of the method 400, the defined path sections P are processed i , which can be done, for example, within the framework of a loop-like processing process, which is initialized in a process 420 by setting a loop index i with i :=1. Within the framework of each loop run, on the basis of the recognition pattern 310, which is the same for all loop runs, and the respective subpattern or path section P i in a process 425 a mapping rule, in particular a homography, H i between the first coordinate system 120 and the second coordinate system 135 is calculated or estimated. One way to calculate H i will now be explained in more detail using a simple example, with particular reference to Figures 1 and 3A and 3B:

[0082] At a given time, the vehicle 105 is positioned such that the screen 130 is located to the front right of the user (e.g., driver). In the second coordinate system 135 of the screen 130, the corner points V1 to V4 of the recognition pattern 310 assume, for example, the following coordinates (x', y', z'): corner point V1 = (0, 0, 1); corner point V2 = (1, 0, 1), corner point V3 = (0, 0, 0), and corner point V4 = (1, 0, 0). Recall that this information is unknown to the detection system 110, which is why calibration is required. Since all corner points V1 to V4 lie on the screen plane, they all have the y-coordinate "0" in the coordinate system 135 of the screen 130.

[0083] The detection system 110 uses as virtual plane {(x, 1 , z): x, z ∈ IR} a plane that is orthogonal to the y-axis and, for example, placed in front of the origin in the y-direction by a unit of length “1” (e.g. 1 m) (cf. Fig. 1 ).

[0084] This plane is chosen here as the first virtual plane 125 and can be provided with its own two-dimensional coordinate system 120a, which is defined by (x,z) E := (x, 1 , z) F where the suffix "F" indicates the vehicle-based coordinates of coordinate system 120 and the suffix "E" indicates the coordinates of virtual plane 125. In this case, the coordinates (X,Z)E of plane 125 are simply the x and z components of the vehicle-based coordinates.

[0085] Based on the dedicated directed user inputs regarding the corner points V1 to V4 of the recognition pattern 310, the detection system 110 now calculates the coordinates of the corner points V'1 to V'4 of the input pattern 305 corresponding to the corner points V1 to V4, referred to the coordinate system 120a, as the intersection points of the directional rays 145 corresponding to the user inputs with the first virtual plane 125 (see Fig. 3). In the present example, the coordinates of the corner points V'1 to V'4 thus determined are as follows: V'1: (0.5; 0.5) E ; V'2: (2; 1 ) E ; V'3: (0.5; 0)E; and V'4: (2; 0)E. This is illustrated in Fig. 3B.

[0086] The projection made in the example models the following assumption: When pointing at the screen or the second plane 130, the user, to put it figuratively, holds their arm outstretched and their shoulder lies at the coordinate origin of the coordinate system 120. As a result, all pointing rays start at the coordinate origin, resulting in a perspective projection onto the virtual plane 125, which is obtained by projecting every point in 3D to which the user points in the direction of the coordinate origin onto the virtual plane 125. The fact that the shoulder is precisely at the origin here and not at another point is, from a mathematical point of view, only an arbitrary choice, but it does simplify the calculations in this example somewhat.

[0087] Now, a mapping rule applies, here specifically a homography H i, so that within a certain tolerable accuracy, which can be determined in particular by the measurement accuracy of the sensor system 1 15 and / or a quality of the user input 140, the following applies: (1 )

[0088] V k = (X k; Y k; Z k ); with k = 1 , ...,4 the coordinates of the vertices V k in the second reference system 135 and V' k = (X' k; Y' k; Z' k ) the coordinates of the corresponding vertices V' k in the reference system 120. The tilde in the equations means "multiples of each other." Accordingly, an equation v ~ w (for vectors v and w) states: "there is an x ​​in , such that v = x · w or x · v = w. (Allowing x on both sides is only necessary in the case that one of the vectors is 0.) The factor “x” should not be confused with the coordinate “x” of the coordinate system 120 introduced above. The homography H i is ultimately a mapping between two-dimensional spaces, i.e. from to or, more vividly speaking, from the virtual level 125 to the screen plane 130 (i.e., in Fig. 3B from the left side to the right side, so that the left, distorted, quadrilateral is correctly mapped onto the right quadrilateral). The homography H i is uniquely determined and can be calculated (or estimated) if at least four vertices are present on each side (= requirements for the recognition pattern 310). Consequently, based on the aforementioned coordinates for the vertices V1 to V4 and V'1 to V'4, the homography H ican be determined by solving the following system of equations: (2) This system of linear equations can be solved in the usual way using standard matrix calculation methods, resulting in the homography H i representing matrix (H i ) results in:

[0089] The following should be noted: a) While the homography H i itself (ie the mapping rule) between the levels is uniquely determined by the four point correspondences, the homography matrix (H i ) is only determined up to one scale. Therefore, any scalar non-zero multiple of the above solution for (H i ) function equally and have the same image H i b) In this example, measurement errors, noise, etc. were ignored. Therefore, there is an exact solution for H i or specifically (H i) that satisfies the boundary conditions according to relations (2). However, in a real-world scenario, one would usually only be able to find a suitable or even optimal approximate solution taking the projection error into account, rather than an exact solution, for example by estimation.

[0090] Referring again to Fig. 4, the process 425 is followed by a further process 430, in which an aberration Ei between the (unstraightened) current path section P i and the recognition pattern 310. This can be determined in particular as follows on the basis of the ordered sequence of the corner points V1 to V M of the recognition pattern 310 on the one hand and the correspondingly ordered sequence of M along the course of the path section P i consecutive vertices c'i to C'M of the path segment P i on the other hand:

[0091] This is H i ( c q) the matrix product of the coordinate vector to c' q with (H i ). The further illustration denotes a homogeneous projection: to Generation of homogeneous coordinates (here normalized with respect to the Z-direction). Then, in step 435, the index i is incremented, and subsequently, in process 440, a check is made as to whether another loop iteration is required. If this is the case (440 - no), another loop iteration is performed with processes 425-440. Otherwise (440 - yes), the path section Pj is selected from among the path sections that has the smallest imaging error Ej among all path sections P. i Finally, in a further process 450, the mapping rule or homography H:= Hj corresponding to the selected path section Pj is defined as a calibration variable for calibrating the acquisition system 110 for subsequent user inputs regarding the Ul of the target system.

[0092] An exemplary method 600 for capturing such user inputs is illustrated in Fig. 6. Method 600 can be performed after calibration method 400 has been completed. In a process 605, comparable to the receiving or generating sensor data in process 405 of method 400, input data is received or generated using sensor system 115. The input data represents, in the reference system 120 or already transmitted to the first reference system 120a, any contactless, directed user input captured by the capturing system, which the user performs with respect to the UI of the target system.

[0093] In a further process 610, these input data are transformed into the second reference system 135 using the calibration variable (mapping rule H) originating from the calibration method 400 in order to obtain corresponding output data which represent the user input in the second reference system or coordinate system 135 of the target system.

[0094] In a yet further process 620, these output data are transmitted to the target system, in particular to an interface unit thereof, in order to control, operate or configure the target system depending on the user input.

[0095] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide one skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents.

[0096] LIST OF REFERENCE SYMBOLS V1- V4Corner points of the recognition pattern V'1- V'4Corner points of the input pattern

[0097] 100 first example calibration and operation scenario

[0098] 105 Vehicle, especially motor vehicle

[0099] 110 Recording system

[0100] 115 Sensors for detecting contactless directed user inputs

[0101] 120 vehicle-related reference system or coordinate system

[0102] 120a first reference system or coordinate system for plane 125

[0103] 125 first virtual surface, especially virtual plane

[0104] 125a, b (flat) partial surfaces of the first virtual surface

[0105] 130 second virtual level, also real screen of a target system

[0106] 130a second partial area of ​​the second virtual plane, also the real screen of a target system

[0107] 135 second reference system or coordinate system

[0108] 140 User input, especially free space gesture, e.g. pointing gesture

[0109] 145 (virtual) directional ray(s) of the directed user input 140

[0110] 150 Intersection of the directional ray 145 with the first plane 125

[0111] 155 third reference system or coordinate system

[0112] 160 second partial surface of the second virtual plane, also projection surface of a projector of the target system

[0113] 200 second example calibration and operating scenario

[0114] 300 spatial relationship between input pattern and recognition pattern

[0115] 301 Comparison of input pattern and recognition pattern

[0116] 305 Input pattern, path-shaped

[0117] 310 recognition patterns, path-shaped

[0118] 400 calibration procedures

[0119] 405 - 450 Subprocesses of the calibration procedure

[0120] 500 Determination of partial patterns

[0121] 600 methods for capturing user input

[0122] 605-615 Subprocesses of procedure 600 P i Partial pattern or path sections of the input pattern 225 x, y, z coordinates of the first reference system 120 or the first partial area

[0123] 125a x', y', z' coordinates of the second reference system 135 x", y", z" coordinates of the second partial area 125b

Claims

CLAIMS 1. Calibration method (400) for calibrating a sensing system (110) for sensing non-contact directed user inputs (140) of a user with respect to a user interface, Ul, of a target system, wherein the method comprises: Receiving or sensor-based generation (405) of sensor data which, with respect to a first reference system (120a) associated with the detection system (110), represents a sensorially detected input pattern (305) which a user inputs by means of at least one contactless directed user input (140) with respect to the Ul, in order to imitate, by means of a spatial progression of the user input (140), a structure of a predefined recognition pattern (310) that is pictorially represented by a display device of the Ul, wherein a second reference system (135) is or is assigned to the display device; Determining (415) of two or more different subpatterns (P i ) of the input pattern (305); For each of the subpatterns (P i ): Determine (425) one of the subpatterns (P i ) associated mathematical mapping rule between the two reference systems (120a, 135) and mapping of the subpattern (P i ) or the recognition pattern (310) according to the mapping instruction such that, according to the mapping, the subpattern (P i ) and the recognition pattern (310) is in a common reference system (120a or 135); Determine (430) a projection error that is a measure of a deviation between the subpattern (P i ) and the recognition pattern (310) in the common reference system (120a or 135); Select (445) one of the subpatterns (P i ) depending on a comparison of its assigned imaging error with the respective imaging error of at least one other subpattern (P i ); and specifying (450) the selected subpattern (P i) assigned mapping rule as calibration parameter for calibrating the acquisition system (110) for subsequent use in transforming contactless directed user inputs (140) acquired with reference to the first reference system (120a) into corresponding inputs on the second reference system (135) related user inputs (140) with respect to the Ul of the target system.

2. Calibration method (400) according to claim 1, wherein the calibration method (400) is configured or used to calibrate a detection system (1 10) arranged in or on a vehicle (105) with respect to a target system external to that vehicle (105).

3. Calibration method (400) according to one of the preceding claims, wherein the sensor-based generation of sensor data is carried out by sensorial detection of a free space gesture of the user or by head or eye tracking.

4. Calibration method (400) according to one of the preceding claims, wherein: the input pattern (305) represents a sensorially detected path that a user enters by means of at least one contactless directed user input (140) with respect to the Ul in order to imitate a structure in the form of a course of a predefined linear recognition pattern (310) that is graphically represented by a display device of the user interface by means of a spatial progression of the user input (140); and at least two of the subpatterns (P i ) of the input pattern (305) each represent a different path segment of the entire path.

5. Calibration method (400) according to claim 4, further comprising: smoothing (410) the path before determining the path segments as partial patterns (P i ).

6. Calibration method (400) according to claim 5, wherein the smoothing of the path comprises a section-by-section straightening of the path, in which the path is approximated by means of a plurality of straight segments.

7. Calibration method (400) according to claim 6, wherein the section-by-section straightening of the path is carried out using a Hough transform.

8. Calibration method (400) according to one of the preceding claims, wherein: the first reference system (120a) is defined as a first virtual two-dimensional surface (125) or such a surface (125) is defined or is defined in it; the input pattern (305) on the first virtual surface (125) is defined by a linear-ray projection of the user input (140) onto the first virtual surface (125) that follows the current direction of the directed user input (140) along its spatial path; the second reference system (135) is defined as a second virtual surface (130; 160); and for each subpattern (P i) the associated mapping rule defines a mapping between the two virtual surfaces (120a; 130, 160), wherein the surface (120a; 130, 160) that serves as the target surface of the mapping rule is or is defined as the common reference system (120a or 135).

9. Calibration method (400) according to claim 8, wherein the first reference system (120a) is defined as a function of one or more previously detected directed user inputs (140) such that the user inputs (140) detected by these one or more previously detected inputs correspond to points and / or lines on the first virtual surface (125).

10. Calibration method (400) according to claim 8 or 9, wherein at least one of the two surfaces (120a; 130, 160) is or is defined as having a virtual plane or a bounded surface section thereof. 11.Calibration method (400) according to claim 10, wherein the first virtual surface (125) is defined such that it comprises a virtual plane or a bounded surface section thereof that is orthogonal to the direction of a previously detected directed user input (140).

12. Calibration method (400) according to any one of claims 8 to 11, wherein the first virtual surface (125) is defined such that it satisfies at least one of the following conditions: - it forms a boundary surface that encloses a three-dimensional spatial area on at least two sides; - it has at least two separate, at least partially non-parallel sub-areas (125a, 125b); - Every linear-ray projection of a directed user input (140) along its current direction through the first virtual surface (125) intersects the first virtual surface (125) at exactly one point.

13. Calibration method (400) according to one of the preceding claims, wherein the partial samples (P i ) are determined such that they are each represented at least partially by a sequence of discrete points on the input pattern (305).

14. Calibration method (400) according to one of the preceding claims, wherein to determine the respective imaging error of a partial sample (P i ) a comparison of the recognition pattern (310) with the subpattern (P i ) in the common reference system (120a or 135) based on homogeneous coordinates to describe the recognition pattern (310) on the one hand and the subpattern (P) on the other. i ) is used on the other hand.

15. Method (600) for capturing contactless directed user inputs (140) of a user with respect to a user interface, Ul, of a target system by means of a capture system (1 10), wherein the method comprises: Calibrating (605) the detection system (1 10) according to the calibration method (400) according to one of the preceding claims; Receiving or sensor-based generation (610) of input data which in the first reference frame (120a) represent a sensor-detected, contactless directed user input (140) of a user with respect to the Ul; Transforming (615) the input data into output data representing an input at the Ul corresponding to the user input (140) transformed into the second reference system (135), using the mapping rule or its inverse mapping rule determined within the framework of the calibration procedure (400); and Transmit (620) the output data to the target system.

16. Acquisition system (110) for acquiring contactless directed user inputs (140) of a user with respect to a user interface, Ul, of a target system, wherein the acquisition system (110) is configured to perform the calibration method (400) according to any one of claims 1 to 14 or the method according to claim 15.

17. Detection system (110) according to claim 16, further comprising a sensor device for sensorially detecting the contactless user inputs (140) and for generating sensor data and / or input- Data representing these user inputs (140) in the first reference frame (120a). Vehicle (105) comprising a detection system (110) according to claim 17. Computer program or computer program product comprising instructions which, when executed on one or more processors of a detection system (110) according to claim 16 or 17, cause it to execute the calibration method (400) according to any one of claims 1 to 14 or the method according to claim 15.

Images