Sensing system and method for sensing contactless directional user input and method for calibrating sensing system

Abstract

A calibration method for calibrating a sensing system for sensing a contactless directional user input of a user relative to a user interface (UI) of a target system, the calibration method comprising: receiving or generating sensor data based on a sensor, the sensor data represents a sensor-detected input pattern referenced to a first frame of reference associated with the sensing system, the user inputting the input pattern by means of at least one contactless directional user input with respect to the UI, in order to replicate a structure of a predefined recognition pattern depicted by a display device of the UI by means of the spatial route entered by the user. The second frame of reference is or is associated with a display device. Two or more different local patterns of the input pattern are determined, for each local pattern: determining an associated mathematical mapping rule between the two reference frames, based on which the local pattern or the recognition pattern is mapped such that the local pattern and the recognition pattern are subsequently in a common reference frame. One of the local patterns is selected according to a mapping error indicative of a measure of deviation between the local pattern and the identifying pattern, and a mapping rule associated with the local pattern is defined as a calibration variable for calibrating the sensing system for subsequent use.

Description

Technical Field

[0001] The present invention relates to a sensing system and method for sensing contactless directional user input relative to a user interface of a target system, a method for calibrating a sensing system, and a computer program or computer program product for performing at least one of the above methods. Background Art

[0002] Contactless interaction with a user interface (UI), especially with a computer's graphical user interface (GUI) or on a computer display screen through hand gestures (e.g., pointing, pulling, grabbing gestures, etc. relative to one or more GUI objects) and eye actions (e.g., looking at GUI objects, blinking to make a selection, etc.), is a powerful approach to computer UI concepts, especially for environments where traditional interaction (via keyboard or other touch devices) is not possible.

[0003] The concept of this interaction is based on the fact that when a user points a finger at a display screen, the pointing beam (the beam originating from the hand and extending in the pointing direction) virtually intersects the display screen at a specific point. If this intersection is interpreted as a touch on a touch screen, the basis for a touchless display screen control UI is created.

[0004] It is also possible to use viewing direction instead of pointing direction as the basis for an eye-controlled sensing system. Viewing and gesture can even be combined to achieve more powerful user input.

[0005] However, in order to be able to reliably convert a sensing-detected viewing direction or gesture into a correct user input in practice, it is necessary to calibrate the sensor used for detection or for any data processing of the raw data provided by the sensor relative to the display screen, in particular with respect to its relative orientation relative to the sensor. In permanently installed sensing systems, a single calibration is usually sufficient for this purpose, since the relative orientation usually does not change anymore thereafter. In contrast, in variable sensing systems, in which the relative orientation of the sensor relative to the display screen or another display surface is designed to be variable, repeated or even continuous calibrations are usually necessary, so that subsequent user inputs can be correctly sensed again even if the orientation has changed. Summary of the invention

[0006] The object of the present invention is to further improve the calibration of a sensing system for sensing contactless directional user input relative to a user interface of a target system, and in particular to enable reliably correct sensing even if the relative orientation between the sensing system and the target system or its user interface is variable.

[0007] This object is achieved according to the teaching of the independent claim. Various embodiments and developments of the invention are subject matter of the dependent claims.

[0008] A first aspect of the solution proposed here relates to a calibration method, in particular computer-implemented, for calibrating a sensing system for sensing a non-contact directional user input of a user relative to a user interface (UI) of a target system, in particular for control or configuration thereof. The method comprises:

[0009] (i) receiving or generating sensor data based on a sensor, the sensor data representing a sensed input pattern relative to a first reference frame assigned to the sensing system, the input pattern being input by a user based on at least one contactless directional user input relative to the UI, so as to replicate the structure of a predefined recognition pattern visually represented by a display device (e.g. a display screen) of the UI by means of a spatial route of the user input. Here, a second reference frame is assigned to the display device (usually, the first reference frame and the second reference frame differ from each other at least in their orientation (i.e., their position and / or orientation)). The replication of the recognition pattern can in particular be performed by continuous travel along the recognition pattern or point-by-point contactless sampling (e.g. in the sense of sampling);

[0010] (ii) determining two or more different local patterns of the input pattern;

[0011] (iii) For each local pattern:

[0012] - determining, in particular estimating, a mathematical mapping rule, in particular a homography, between two reference systems assigned to the local pattern and mapping the local pattern or the recognition pattern based on the mapping rule such that after the mapping the local pattern and the recognition pattern are present in a common reference system. Thus, the mapping rule can define a mapping of a first reference system into a second reference system or, alternatively, a mapping in an opposite mapping direction such that the reference system into which the mapping takes place is used simultaneously with the common reference system. The recognition pattern can in particular be known in advance to the sensing system or can be received dynamically by the sensing system in the context of a calibration method, for example via a direct or indirect, in particular at least partially wireless, data connection between the target system and the sensing system;

[0013] - determining a mapping error indicating a measure of the deviation between the local pattern and the recognition pattern in a common reference frame; and

[0014] - selecting one of the local patterns, the selection being based on a comparison of the assigned mapping error of the one local pattern with a corresponding mapping error of at least one other local pattern; and

[0015] (iv) Define the mapping rule assigned to the selected local pattern as a calibration variable for calibrating the sensing system for subsequently converting a non-contact directional user input sensed relative to a first reference frame into a corresponding user input relative to the UI of the target system and related to a second reference frame.

[0016] The sensing system can in particular be configured as a vehicle-based sensing system (e.g., as a control unit of a vehicle) for sensing user input in a vehicle, while the target system can in particular be configured as a system outside the vehicle. The sensing system can in particular be or include a program-controlled computer system.

[0017] The term "reference frame" as used herein should be particularly understood as an imaginary spatial or spatio-temporal structure required for clearly and completely describing the behavior of orientation-related variables. The reference frame, particularly its spatial aspect, can in particular be described based on a defined coordinate system. In particular, a first reference frame can be defined relative to the vehicle. The second reference frame assigned to the display device can in particular be defined such that the plane spanned by two spatial dimensions of the second reference frame coincides with the plane of the display device, in particular with the display surface or the projection surface of the display device.

[0018] The term "local pattern" as used herein should be particularly understood as a pattern that simultaneously forms a subset of a higher-order pattern (in particular, the recognition pattern here). For example, in a grid-like pattern composed of a family of parallel horizontal lines and a family of vertical lines, each of these families can be individually understood as a local pattern of the grid pattern. According to another example, in a broken line composed of N linked, particularly straight segments as a plurality, in particular one (or even each) subset of M < N particularly linked segments can be understood as a local pattern of the entire broken line.

[0019] The term "calibration variable" as used herein should be understood as information, in particular a mapping rule, which at least proportionally defines a specific calibration of the sensing system.

[0020] The terms "comprises", "contains", "includes", "has", "having" or any other variants thereof that may be used herein are intended to cover non-exclusive inclusion. For example, a method or device that includes or has a series of elements does not necessarily have to be limited to these elements, but can include other elements not explicitly listed or inherent in such a method or such a device.

[0021] In addition, unless expressly stated otherwise, "or" refers to an inclusive "or" rather than an exclusive "or". For example, condition A or B is satisfied by one of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0022] The terms “a” and “an”, as used herein, are defined as “one or more.” The terms “another” and “another” and any other variations thereof should be construed to mean “at least one other.”

[0023] The term "plurality" as may be used herein should be understood to mean "two or more".

[0024] The term "configured" or "designed" to realize a certain function (and any variations thereof) should be understood to mean for the purposes of the present invention that the corresponding device already exists in a configuration or setting in which it is able to perform the function, or that the device is at least adjustable (i.e. configurable) such that it is able to perform the function after appropriate adjustments. The configuration may be applied, for example, by appropriate setting of parameters of a process sequence or a switch, etc., for activating or deactivating a function or setting. In particular, the device may include a plurality of predetermined configurations or operating modes, such that the configuration can be performed by selecting one of these configurations or operating modes.

[0025] The above method according to the first aspect enables a proper calibration of the sensing system with respect to the user input relative to the UI of the target system, in particular in the case where the relative spatial orientation of the sensor for sensing the sensor data and the display device or its display area - and therefore the relative orientation of the first reference system and the second reference system with respect to each other - is variable, in particular unknown a priori. Even in such variable situations, reliable operability of the target system by means of non-contact directional user input relative to the UI can thereby be achieved.

[0026] This is particularly likely to occur if the sensing system is mobile, such as vehicle-based, and the target system is stationary.

[0027] Exemplary embodiments of the method will now be described below, each of which can be combined with each other and with further described aspects of the invention as desired, unless explicitly excluded or technically impossible.

[0028] In some embodiments, a calibration method is configured or used to calibrate a sensing system arranged in or on a vehicle relative to a target system external to the vehicle associated with the vehicle. Many different types of systems, in particular infrastructure systems, are considered here as target systems, which are suitable or intended for contactless operation by a user from a vehicle. These include, in particular, parking management systems, such as parking meters, barrier systems or other access control systems, drive-through terminals or kiosks, garage doors, or electronic information panels with UIs. Since the relative orientation of the vehicle relative to the UI of such a target system is naturally often variable or not explicitly known in advance, the calibration method can be used particularly advantageously here so that before the desired interaction with the UI, the calibration is performed taking into account the relative orientation of the vehicle or the user performing the interaction relative to the target system, so that subsequent interactions can be correctly and reliably sensed.

[0029] In some embodiments, sensor-based generation of sensor data is performed by means of sensory detection of the user's free-space gestures, in particular pointing gestures (e.g., via arms and / or hands), or by means of head or eye tracking. In this way, contactless directional user input (in particular based on measured or estimated viewing or pointing directions) that can be performed by most people with very high target accuracy can serve as the basis for a calibration method and be used to sense subsequent interactions of the user with the UI of the target system by means of the calibrated sensing system.

[0030] In some embodiments, the input pattern represents a path detected by sensing, which is input by the user based on at least one contactless directional user input relative to the UI, so as to replicate the structure of the route form of a predefined recognition pattern in the form of a polyline visually represented by a display device of the user interface (e.g., a display screen or a projection device for projecting an image on a projection surface) by means of the spatial route of the user input, in particular a "pointing trajectory" or a "viewing trajectory" defined by the user input. In this case, at least two of the partial patterns of the input pattern each represent a different path segment of the entire path. It is advantageous to select a linear path as a recognition pattern or input pattern, in particular because it can be particularly easily reproduced, even by an unskilled user in the case of a contactless directional user input of the input pattern, and is particularly easy to handle within the scope of data processing functions. Such a path can be defined in particular as a link (polyline) of multiple straight path segments, which in particular enables it to be efficiently represented and can be easily processed and stored only by the start and end points of the segments, i.e. the endpoints and corner points (vertices) of the polyline.

[0031] In some of these embodiments, the calibration method also includes smoothing the path before determining the path segment as a local pattern. In this way, in particular, any disturbances of the expected spatial route of the user input that occur during the user input ("unstable" input) can be corrected so as to reduce or even avoid completely erroneous understanding or unnecessary large amounts of data for representing and processing the path sensed as the input pattern. Therefore, based on the smoothest path, a more accurate or optimized calibration can be achieved, which ultimately facilitates a particularly reliable recognition of the user input at the UI of the target system by means of the subsequently calibrated sensing system.

[0032] According to some embodiments, the smoothing of the path may in particular comprise straightening the path segment by segment, wherein the path is approximated by means of a plurality of segments, each segment being a straight line. In particular, the above-mentioned broken line with the associated advantages can thus be achieved. In particular, the segment-by-segment straightening of the path may be performed using a Hough transform, by means of which a good approximation of the broken line to the input pattern may be achieved in a particularly robust and reliable manner.

[0033] In some embodiments, the first reference system is defined as a first virtual two-dimensional surface, or such a surface is defined in the first reference system. Furthermore, an input pattern is defined on the first virtual surface by projections of user inputs in the form of linear beams, which follow the respective current direction of the directional user input along their spatial course on the first virtual surface. The first virtual surface can in particular be an inner surface of a virtual conical surface or a conical partial surface, or can be located thereon. It is also conceivable that the first virtual surface is cumulatively composed of a plurality of individual partial surfaces.

[0034] The second reference system is defined as a second virtual surface. The associated mapping rules define, for each local pattern, a mapping between two virtual surfaces, wherein one of the surfaces serving as a target surface for the mapping rule is defined as a common reference system. In each case, two-dimensional surfaces are used, between which the mapping rules define a transformation, with the advantage, inter alia, of reduced complexity with respect to the general three-dimensional case. As a result, the calculation of the mapping rules and their application are also simplified, which can also be used to increase efficiency and, given computing power, to enable faster, in particular real-time, execution of the calibration method and subsequent sensing of user inputs relative to the UI in the context of interaction with the target system.

[0035] In some of these embodiments, the first reference frame is defined based on one or more previously detected directional user inputs such that these one or more previously detected user inputs correspond to points and / or lines on the first virtual surface. In other words, this means that the first virtual surface is defined, for example, such that these one or more previously detected user inputs "impact" the first virtual surface and do not impact beyond it. In this way, such a first virtual plane can be positioned in a simple manner in three-dimensional space based on one or more user inputs, in particular such that the first virtual plane is reachable as an impact surface for directional user inputs that can be performed by the user if, for example, the user is sitting in a vehicle seat inside a vehicle with a seatbelt on and is therefore restricted in their freedom of movement and possibly also with respect to a restricted field of view in an area outside the vehicle that is not visible to them.

[0036] In some embodiments, at least one of the two surfaces is defined such that it comprises a virtual plane or a finite surface segment of the virtual surface. This simplifies, among other things, the determination of the mapping rule and the implementation of the mapping between the reference frames according to the mapping rule.

[0037] In particular, according to some embodiments, the first virtual surface may be defined such that it comprises a virtual plane or a finite surface segment of the virtual surface, which is orthogonal to the direction of the previously detected directional user input. A particularly simple and efficient user-controlled definition of the virtual surface, in particular the spatial orientation of the surface in space, is thereby enabled.

[0038] In some embodiments, the first virtual surface is defined such that it satisfies at least one of the following conditions:

[0039] (i) it forms a boundary surface which encloses the three-dimensional spatial region at least on two sides;

[0040] (ii) it comprises at least two separate partial surfaces which are non-parallel at least in some parts;

[0041] (iii) each projection in the form of a linear beam of the directional user input along its respective current direction and through the first virtual surface intersects the first virtual surface at exactly one point.

[0042] Option (i) is particularly advantageous, since the solid angle range in which the user can successfully perform their directional user inputs (so that they are sensed) can be extended, in particular over more than half a space (in the case of a single space such as a first virtual surface in the form of a single virtual plane). Option (ii) can also be used to extend the solid angle range available for user inputs, in particular if the user inputs are assumed to be able to occur in solid angle ranges that are separated from each other (e.g. through a vehicle's windshield and through a side window separated from the windshield by a spar). Option (iii) is particularly useful for effectively avoiding ambiguities in the inputs from the outset.

[0043] In some embodiments, the local patterns are defined such that each of them is represented at least proportionally by a sequence of discrete points on the input pattern, in particular in the case of a path in the input pattern, along the course of a corresponding path segment of the path. This allows a particularly efficient and fast implementation, in particular with a low computational effort to save data, since the processing of the input pattern can be reduced to calculations on discrete points, in particular vertices.

[0044] In some embodiments, in order to determine the corresponding mapping error of the local pattern, a comparison of the recognition pattern and the local pattern in a common reference system based on homogeneous coordinates (in the sense of projective geometry) is used to describe the recognition pattern on the one hand and the local pattern on the other hand. Therefore, due to the use of homogeneous coordinates, the spatial dimension, that is, along the direction of user input (especially the pointing direction or the viewing direction), can be in fact "compressed" so that the multiples of the coordinate values ​​along the spatial dimension coincide with each other when converted to homogeneous coordinates, thereby reducing the complexity. Therefore, the advantage of homogeneous coordinates lies in the unified representation of the elements of the projection space, in which distant elements (especially multiples) no longer play a special role. In addition, due to the use of homogeneous coordinates, all direct projections and therefore parallel offsets can be uniformly described by linear mappings, and can therefore be effectively and well processed by matrices.

[0045] A second aspect of the solution proposed here relates to a method for sensing a contactless directional user input of a user relative to a user interface (UI) of a target system by means of a sensing system. The method comprises:

[0046] (i) calibrating the sensing system according to the calibration method of the first aspect;

[0047] (ii) receiving or generating input data based on a sensor, the input data representing a sensory detected, contactless user input of a user relative to the UI in a first reference frame, which may be defined in particular in a vehicle-based manner;

[0048] (iii) using a mapping rule defined in the scope of the calibration method or a mapping rule inverse to the mapping rule, converting the input data into output data representing the input at the UI corresponding to the user input and converted into the second reference frame; and

[0049] (iv) transmitting the output data to a target system, in particular external to the vehicle, in particular for the operation, control or configuration of the target system.

[0050] Thus, the method according to the second aspect may be used to enable operation, control or configuration of user input controls of a target system following calibration according to process (i).

[0051] A third aspect of the solution proposed here relates to a sensing system for sensing contactless directional user input of a user relative to a user interface (UI) of a target system, wherein the sensing system is configured to perform a calibration method according to the first aspect or a method according to the second aspect.

[0052] In some embodiments, the sensing system further comprises a sensor device for sensingly detecting contactless user inputs and for generating sensor data and / or input data representing these user inputs in the first reference frame. Thus, the sensor for detecting the user inputs is itself already part of the sensing system itself, for example in the form of being integrated in the housing of the sensing system.

[0053] A fourth aspect of the solution proposed here relates to a vehicle comprising the sensing system according to the third aspect.

[0054] A fifth aspect of the solution proposed here relates to a computer program or a computer program product comprising instructions which, when executed on one or more processors of a sensing system according to the third aspect, cause the computer program or the computer program product to perform a calibration method according to the first aspect or a method according to the second aspect.

[0055] In particular, the computer program can be stored in a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. It may be advantageous if such a computer program will be processed independently of the processor platform on which one or more programs will run. In another embodiment, the computer program can exist as a file on a data processing unit, in particular on a server, and can be downloaded via a data link, such as the Internet or a dedicated data link, such as a proprietary or local network. In addition, the computer program can have a plurality of separate interactive program modules. In particular, the module can be configured to be used, or in any case can be used in such a way, in the sense of distributed computing on different devices (computers or processor units) that are geographically far away and connected to each other via a data network.

[0056] The sensing system according to the third aspect may accordingly have a program memory storing the computer program. Alternatively, the sensing system may also be configured to access an externally available computer program, such as a computer program on one or more servers or other data processing units, via a communication link, in particular in order to exchange data used during the execution of the method or computer program, or to represent the output of the computer program.

[0057] Features and advantages explained with respect to the first and second aspects of the solution proposed here also apply correspondingly to the other aspects of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Further advantages, features and application possibilities of the present invention can be found in the following detailed description in conjunction with the accompanying drawings.

[0059] In the attached picture:

[0060] Figure 1 schematically illustrates a first exemplary calibration and operation scenario with a first virtual plane;

[0061] Figure 2 schematically illustrates a second exemplary calibration and operating scenario having a first virtual plane and a third virtual plane different from the first virtual plane;

[0062] Figure 3A A perspective view schematically illustrating an exemplary spatial relationship of an input pattern and a recognition pattern;

[0063] Figure 3B Again, it is shown that Figure 3A The spatial relationship between the input pattern and the recognition pattern, however, for better comparability, here the two patterns are transferred to a common representation plane;

[0064] Figure 4A flow chart illustrating an exemplary embodiment of a calibration method according to the first aspect is shown;

[0065] Figure 5 Schematically shows the Figure 4 An illustration of an exemplary local pattern determined in the context of a calibration method; and

[0066] Figure 6 A flow chart illustrating an exemplary embodiment of a method for sensing contactless directional user input according to the second aspect is shown. DETAILED DESCRIPTION

[0067] In the accompanying drawings, the same reference numerals represent the same, similar or mutually corresponding elements. The elements in the illustrated drawings are not necessarily shown to scale. On the contrary, the different elements shown in the drawings are reproduced in such a way that their functions and general purposes are understandable to those skilled in the art. Unless otherwise explicitly stated, the connections and couplings between the functional units and elements shown in the drawings may also be implemented as indirect connections or couplings. The functional units, in particular the sensing system and its components, may be specifically implemented as hardware, software or a combination of hardware and software.

[0068] Figure 1 A first exemplary calibration and operation scenario 100 is schematically shown. Operation of the target system via its user interface UI will take place from a vehicle 105, wherein an occupant of the vehicle makes user inputs with respect to the UI in a contactless manner, i.e. without physical contact with the target system. For this purpose, the vehicle 105 comprises a sensing system 110 having a sensor 115 for transmitting signals coupled thereto, in particular an image sensor (e.g. a camera, in particular a video camera). The sensing system 110 may in particular be a program-controlled data processing device which is configured and programmed to receive sensor data provided by the sensor 115 and to process the sensor data in a manner such that the sensor data is transmitted to the vehicle 10 ... Figure 4 methods (and possible Figure 6 By means of the sensor 115, various user inputs by a user (in this example, an occupant) inside or outside the vehicle can be detected. Such user inputs can in particular be performed via one or more free-space gestures (in particular pointing gestures) or via one or more target sight lines (eye tracking) and detected by the sensor 115.

[0069] exist Figure 1In the example shown, a vehicle occupant as user performs a pointing gesture 140 as a contactless free-space gesture. The pointing direction corresponds here to the direction of the user input and can therefore be referred to as a “directional” user input, since a specific detected direction is assigned to the user input as a whole or in a time-dependent manner, i.e. in each case for a specific point in time during its execution. Figure 1 145 is shown in FIG. 146 based on a virtual directional beam 145. A separate reference system can be assigned to the vehicle 105, in particular a coordinate system 120. This coordinate system 120 can optionally already be defined as a first reference system. As shown in the figure, it can in particular be a Cartesian coordinate system with coordinate axes x, y and z, in particular a right-handed coordinate system. However, another variant is described below, in which a coordinate system 120a offset along the Y axis relative to the coordinate system 120 is selected as the "first" coordinate system.

[0070] The target system can be, in particular, a stationary infrastructure device, such as a parking meter, a barrier system, an ordering kiosk for a drive-through store or restaurant, or a car wash facility. Many other applications or target systems are also conceivable. Figure 1 In the example of , the target system has a display device 130 in the form of a display screen with a planar display surface. The display surface is assigned a second reference system 135, in particular a coordinate system, so that the two axes of the coordinate system extend parallel to the display surface or even lie on it. As shown in the figure, this can also be a Cartesian coordinate system with coordinate axes x', y' and z'. The coordinates can be scaled in particular relative to the size of the display screen.

[0071] Since the relative orientation of the vehicle 105 relative to the target system and therefore relative to the second reference system 135 is typically variable and not known a priori due to the mobility of the vehicle 105, the relative orientation of the two reference systems 120 and 135 is typically also variable and not known in advance (i.e. before performing the calibration method).

[0072] In order to be able to perform a transformation H of the user input 140 detected by the vehicle-based sensor 115 (which is therefore anchored in the reference frame 120) into the second reference frame of the target system with the help of the sensing system 110, the sensing system 110 must be calibrated. Therefore, before the user input can be reliably performed based on the calibration, the transformation is initially determined.

[0073] In order to determine the transformation H, a first virtual surface 125 anchored in the reference system 120 and a second virtual surface anchored in the second reference system 135 are defined, which in this example coincides with the display surface of the display screen 130. For simpler computability of the transformation, it is advantageous in this case if both virtual surfaces 125 and 130 are planar, i.e. both are virtual planes or delimited segments thereof.

[0074] Figure 2 A second exemplary calibration and operation scenario 200 is shown with respect to its extension. This figure corresponds largely to Figure 1 , however, wherein the first virtual surface is divided into two separate local surfaces 125a and 125b, each of which is planar and at an angle relative to each other (a first virtual plane and a third virtual plane different therefrom). In this way, the spatial area that the directional beam of the user input 140 can hit (so that the directional beam hits the first virtual surface and therefore hits one of the two local surfaces 125a and 125b in the intersection 150) can be extended, in particular beyond the half-space defined by the single planar local surface. The second local surface 125b is in turn assigned a separate reference system, in particular a Cartesian or other coordinate system 155, with coordinates x", y", z", conveniently in the case of a planar local surface, so that the two axes of this reference system extend parallel to the second local surface 125b or in the second local surface 125b.

[0075] The first partial surface 125 may also comprise more than two partial surfaces and / or in particular may also be curved, for example curved into a spatially enclosed form around the vehicle or the user, for example a sphere or a part thereof.

[0076] Figure 3A By way of example, the perspective view according to Figure 1 The user performs a moving user input here, during which they contactlessly copy the identification pattern 310 displayed on the display device 130, the identification pattern 310 being a plane with vertices V 1 To V 4 The copying can especially take place so that the user sequentially points to the vertex V in discrete steps. 1 To V 4 , or "travel along a path" continuously.

[0077] Due to the generally different positions and orientations of the two virtual planes 125 and 130, a distortion generally occurs between the recognition pattern 310 and the corresponding input pattern 305, which is defined in the first virtual surface (plane) 125 by a discrete or continuous sequence of intersections of the directional beams 145 (only two of which are shown exemplarily) during the user input. 1 To V 4 The distortion of the relative orientation of each other, and the vertex V of the recognition pattern 1 To V 4 The corresponding vertex V' 1 To V' 4 The relative orientation of the

[0078] Figure 3B Again the identification pattern 310 and the input pattern 305 corresponding thereto on a common representation plane shifted as comparison 301 are shown, so that the effect of the distortion is clearly visible.

[0079] In order to be able to ensure correct user input relative to the defined user interface UI of the target system, relative to the display surface 130, it is therefore necessary to determine the above-mentioned transformation H in the context of calibration, in particular to calculate or estimate it such that it compensates this distortion at least to a good approximation.

[0080] A calibration method 400 suitable for this purpose is Figure 4 Method 400 is hereinafter additionally exemplarily referred to as Figure 1 , 3 and 5, the method 400 includes a process 405 in which the sensing system 110 receives sensor data from the sensor 115. In contrast, if the sensor 115 is counted as a component of the sensing system 110, the process 405 thus corresponds to the generation of sensor data by the sensing system 110 including its sensor 115 itself.

[0081] The sensor data are related to the reference frame 120 or already to the reference frame 120a and describe therein an input pattern 305 detected by the sensor 115, which input pattern 305 is generated by the user by means of a contactless directional user input 140. As shown in FIG3 , this occurs when they copy with their user input 140 a linear, in particular zigzag-shaped (chain of straight segments) path displayed as a recognition pattern 310 on the display device 130, in particular travel along this path or trace this path by successively pointing to the vertices in the order defined by the course of this path. The intersection of the directional beam of the user input with the first virtual plane 125 is determined, which in turn defines the input pattern 305 as a line or path on the first virtual plane 125.

[0082] Typically, the input pattern 305 will only approximately correspond to the recognition pattern 310 here, especially since contactless directional user input, such as free space gestures or especially eye movements that can be sensed via eye tracking, cannot be precisely controlled by the user himself and therefore naturally has variations or tolerances.

[0083] Figure 5 An exemplary path of an input pattern 305 of a path shape is shown in the context of diagram 500 for determining a local pattern, as a user follows a path having a vertex V displayed on display screen 130 of a target system. 1 To V 4 When the recognition pattern 310 of the shape of the path of the user 305 is followed, the user can make an input according to their user input 140. The input pattern 305 is similar here to the recognition pattern 310 of the shape of a broken line consisting of straight segments, but does not completely correspond to the recognition pattern 310. In order to improve the quality of the further calibration to be determined, in a further process 410 within the scope of the method 400 (see Figure 4 ), the input pattern 305 is thus straightened. This can be achieved in particular by means of a so-called Hough transform, which is known in the field of image processing. As a result of this straightening, the zigzag-shaped path formed by the straight linked segments produces a straightened input pattern 305a on the virtual plane 125.

[0084] Starting from the straightened input pattern 305a, in a further process 415 of the method 400, N different local patterns P are now determined. i , where i=1,...,N, where N>1, in this example, especially the path segment. Each path segment P i represents a portion of the input pattern 305a in the sense of a subset of its points and may in particular comprise a plurality of interconnected straight segments of the input pattern 305a. Figure 5 The lower part of 414 shows an exemplary result of the process 415 for the case where N=3, where each path segment has the same number M of vertices as the recognition pattern 310. Thus, in this example, M=4 applies.

[0085] In the further process of the method 400, the defined path segment P is then i The processing is performed, which can be performed, for example, in the form of a loop processing procedure, which is initialized in the process 420 by setting the loop index i with i:=1. In the context of each loop, based on the recognition pattern 310 (the recognition pattern 310 is here the same for all loops) and the corresponding local pattern or path segment P i In process 425, a mapping rule between the first coordinate system 120 and the second coordinate system 135, in particular a homography H is calculated or estimated. i .

[0086] The calculation of H will now be explained in more detail based on a simple example. i One possibility, with particular reference to Figure 1 and 3A And 3B.

[0087] At a particular point in time, the vehicle 105 is positioned so that the display screen 130 is located to the right in front of the user (eg, the driver). In the second coordinate system 135 of the display screen 130, as an example, the vertex V of the recognition pattern 310 is 1 To V 4 Occupies the following coordinates (x',y',z'): Vertex V 1 =(0,0,1); vertex V 2 =(1,0,1), vertex V 3 =(0,0,0), vertex V 4 =(1,0,0). It should be noted that this information is unknown to the sensing system 110 and thus requires calibration. 1 To V 4 Both lie on the plane of the display screen, so they both have a y coordinate of “0” in the coordinate system 135 of the display screen 130 .

[0088] The sensing system 110 uses a plane orthogonal to the y-axis as a virtual plane Here, as an example, it is placed one length unit "1" (e.g., 1 m) ahead of the origin in the y direction (see Figure 1 ).

[0089] This plane is here selected as the first virtual plane 125 and can be provided with its own two-dimensional coordinate system 120a, which is composed of (x, z) E :=(x,1,z) F Given as , where the suffix "F" represents the vehicle-based coordinates of the coordinate system 120 and the suffix "E" represents the coordinates of the virtual plane 125. In this case, the coordinates (x, z) of the plane 125 are E It is therefore simply the x and z components of the vehicle's coordinates.

[0090] The sensing system 110 now recognizes the vertex V of the pattern 310 based on 1 To V 4 The dedicated directional user input calculates the input pattern 305 corresponding to the vertex V 1 To V 4 Vertex V' 1 To V' 4 With respect to the coordinates of the coordinate system 120a, the vertex V' of the input pattern 305 is 1 To V' 4As the intersection of the directional beam 145 corresponding to the user input and the first virtual plane 125 (see FIG. 3 ). In this example, the vertex V′ thus determined is 1 To V' 4 The coordinates of V' are as follows: 1 :(0.5;0.5) E ; V' 2 :(2;1) E ; V' 3 :(0.5;0) E ; and V' 4 :(2;0) E This is Figure 3B Shown in.

[0091] The projections made in the example simulate the following assumption: when pointing at the display screen or the second plane 130, visually the user keeps their arms extended and their shoulders are located at the coordinate origin of the coordinate system 120. Therefore, all pointing beams start from the coordinate origin and produce a perspective projection on the virtual plane 125, which is obtained by projecting each point in 3D pointed by the user in the direction of the coordinate origin on the virtual plane 125. The shoulder here is specifically at the origin, rather than at another point, which is just an arbitrary choice mathematically speaking, but it facilitates the calculations in this example to a certain extent.

[0092] Now we need to determine the mapping rule, which is specifically the homography H i , so that within a specific range of permissible accuracy (which range may be determined in particular by the measurement accuracy of the sensor 115 and / or the quality of the user input 140 ), the following applies:

[0093] For all k.(1)

[0094] In this case, V k =(X k； Y k； Z k ); where k = 1,...,4, is the vertex V k coordinates in the second reference system 135, and V' k =(X' k； Y' k； Z' k ) is the corresponding vertex V' k Coordinates in reference system 120 .

[0095] The tilde "~" in an equation means "multiples of each other". Therefore, the equation v~w (for vectors v and w) is expressed as follows: There exists x in , such that v = x·w or x·v = w (x needs to be left on both sides only if one of the vectors is 0). The factor "x" here should not be confused with the coordinate "x" of the coordinate system 120 introduced above.

[0096] Therefore, the homography H i Ultimately it is a mapping between two-dimensional spaces, so from arrive Or visually speaking, from the virtual plane 125 to the display screen plane 130 (hence Figure 3B , from left to right, so that the deformed quad on the left is mapped to the quad on the right with the correct perspective).

[0097] Homography H i is uniquely determined, and when there are at least four vertices on each side, the homography H can be calculated (or estimated) i (=requirement for recognition pattern 310). Therefore, based on vertex V 1 To V 4 and V' 1 To V' 4 The above coordinates can be determined by solving the following system of equations: i :

[0098]

[0099] This system of linear equations can be solved in the usual way using standard methods of matrix computation, yielding the homography H i The matrix (H i )as follows:

[0100]

[0101] In this regard, the following points should also be noted:

[0102] a) Although the homography H between the planes i The mapping rule itself is uniquely determined by the correspondence of the four points, but the homography matrix (H i ) is determined only to a ratio. Therefore, the above i ) will act similarly and lead to the same map H i .

[0103] b) In this example, measurement errors, noise, etc. are not considered. Therefore, there is H that satisfies the boundary conditions according to equation (2) i or especially (H i). However, in practical situations, taking into account the projection error, usually only a suitable or even optimal approximate solution can be found instead of an exact solution, e.g. by estimation.

[0104] Reference again Figure 4 , process 425 is followed by a further process 430 in which the (unstraightened) current path segment P is determined i The mapping error E between the recognition pattern 310 i On the one hand, this can be based on identifying the vertex V of the pattern 310 1 To V M On the other hand, the ordered sequence of path segments P i The M vertices c' 1 to c' M Along the path segment P i The routes are ordered one after another, as follows:

[0105]

[0106] H i (c q ) is to c′ q The coordinate vector of (H i ). Further mapping A uniform projection is specified: Used to generate homogeneous (here scaled relative to the Z direction) coordinates.

[0107] Then in step 435, the index i is incremented and then in process 440 a check is made as to whether further loops are required. If this is the case (440 No), further loops are run using processes 425-440. Otherwise (440 Yes), the path segment having all path segments P is selected among the path segments. i The minimum mapping error E in j The path segment P j Finally, in a further process 450, corresponding to the selected path segment P j The mapping rule or homography H: =H J is defined as a calibration variable used to calibrate the sensing system 110 for subsequent user input relative to the UI of the target system.

[0108] Figure 6An exemplary method 600 for sensing such user input is shown. The method 600 may be performed when the calibration method 400 has been previously completed. In this case, input data is received or generated by means of the sensor 115 in process 605, comparable to the reception or generation of sensor data in process 405 of the method 400. In this case, the input data in the reference frame 120 or that has been transmitted to the first reference frame 120a represents any non-contact directional user input sensed by the sensing system, which the user performs relative to the UI of the target system.

[0109] In a further process 610 these input data are transformed into the second reference system 135 using calibration variables (mapping rule H) originating from the calibration method 400 in order to obtain corresponding output data representing the user input in the second reference system or coordinate system 135 of the target system.

[0110] In yet another process 620, these output data are transmitted to the target system, in particular to its interface unit, in order to control, operate or configure the target system accordingly according to the user input.

[0111] Although at least one exemplary embodiment has been described above, it must be noted that there are a large number of variations in this regard. It should also be noted here that the described exemplary embodiments constitute only non-limiting examples, and they are not intended to limit the scope, applicability or configuration of the apparatus and methods described herein. On the contrary, the above description will provide those skilled in the art with an indication of the implementation of at least one exemplary embodiment, wherein it should be understood that various changes may be made to the arrangement of the functional devices and elements described in the exemplary embodiments without departing from the subject matter defined in the attached claims or their legal equivalents, respectively.

[0112] Reference numerals list

[0113] V 1 -V 4 Identify the vertices of a pattern

[0114] V' 1 -V' 4 Enter the vertices of the pattern

[0115] 100 First Exemplary Calibration and Operation Scenario

[0116] 105 Vehicles, especially motor vehicles

[0117] 110 Sensing system

[0118] 115 Sensor for detecting contactless directional user input

[0119] 120 Vehicle-related reference or coordinate system

[0120] 120a A first reference system or coordinate system of plane 125

[0121] 125 First virtual surface, especially virtual plane

[0122] 125a, 125b (plane) local surface of the first virtual surface

[0123] 130 The second virtual plane is also the real display screen of the target system

[0124] 130a The second local surface of the second virtual plane is also the real display screen of the target system

[0125] 135 Second reference or coordinate system

[0126] 140 User input, especially free-space gestures such as pointing gestures

[0127] 145 (virtual) directional beam of directional user input 140

[0128] 150 Intersection point of directional beam 145 and first plane 125

[0129] 155 Third reference or coordinate system

[0130] 160 The second local surface of the second virtual plane is also the projection surface of the projector of the target system

[0131] 200 Second Exemplary Calibration and Operation Scenario

[0132] 300 Spatial relationship between input pattern and recognition pattern

[0133] 301 Comparison of input pattern and recognition pattern

[0134] 305 Input pattern, path shape

[0135] 310 Recognize patterns and path shapes

[0136] 400 Calibration Method

[0137] 405-450 Partial Procedure of Calibration Method

[0138] 500 Determination of local pattern

[0139] 600 Method for sensing user input

[0140] 605-615 Partial process of method 600

[0141] P i A partial pattern or path segment of the input pattern 225

[0142] x, y, z Coordinates of the first reference system 120 or the first partial surface 125a

[0143] x',y',z' Coordinates of the second reference system 135

[0144] x", y", z" coordinates of the second partial surface 125b

Claims

1. A calibration method (400) for calibrating a sensing system (110) for sensing a non-contact directional user input (140) of a user relative to a user interface (UI) of a target system, wherein: The method comprises: receiving or generating (405) sensor data based on a sensor, the sensor data representing a sensed input pattern (305) relative to a first reference frame (120a) assigned to the sensing system (110), the input pattern being input by a user by means of at least one contactless directional user input (140) relative to the UI, so as to replicate the structure of a predefined recognition pattern (310) visually represented by a display device of the UI by means of a spatial route of the user input (140), wherein the display device is assigned a second reference frame (135); Determine (415) two or more different local patterns (P i ); For each local pattern (P i ): Determine (425) the number of cells assigned to the local pattern (P i ) between the two reference systems (120a, 135), and mapping the local pattern (P i ) or the identification pattern (310), so that after mapping, the local pattern (P i ) and the identification pattern (310) exist in a common reference system (120a or 135); Determine (430) a mapping error, the mapping error indicating the local pattern (P i ) and the measurement of the deviation between the recognition pattern (310); Select (445) the local pattern (P i ), the selection is based on the mapping error of the distribution of the one local pattern and at least one other local pattern (P i )’s corresponding mapping errors; and Assign to the selected local pattern (P i ) is defined (450) as a calibration variable for calibrating the sensing system (110) for subsequently converting the user's contactless directional user input (140) sensed relative to the first reference system (120a) into a corresponding user input (140) relative to the UI of the target system and with respect to the second reference system (135).

2. The calibration method (400) according to claim 1, wherein: The calibration method (400) is configured or used to calibrate a sensing system (110) arranged in or on a vehicle (105) relative to a target system external to the vehicle and associated with the vehicle (105).

3. The calibration method (400) according to any one of the preceding claims, wherein: The sensor-based generation of sensor data is performed by means of sensory detection of the user's free-space gesture or by means of head or eye tracking.

4. The calibration method (400) according to any one of the preceding claims, wherein: The input pattern (305) represents a path detected by sensing, which is input by a user by means of the at least one contactless directional user input (140) relative to the UI, so as to replicate the structure of the route form of the recognition pattern (310) of the predefined foldline shape visually displayed by the display device of the user interface by means of the spatial route of the user input (140); and The local pattern (P) of the input pattern (305) i ) each represent a different path segment of the entire path.

5. The calibration method (400) according to claim 4, further comprising: In determining the path segment as a local pattern (P i ) before smoothing (410) the path.

6. The calibration method (400) according to claim 5, wherein: Smoothing the path includes straightening the path segment by segment, wherein the path is approximated by a plurality of respective straight segments.

7. The calibration method (400) according to claim 6, wherein: The segment-wise straightening of the path is performed using the Hough transform.

8. The calibration method (400) according to any one of the preceding claims, wherein: The first reference system (120a) is defined as a first virtual two-dimensional surface (125), or such a surface (125) is defined in the first reference system (120a); The input pattern (305) on the first virtual surface (125) is defined by a linear beam shape projection of the user input (140) on the first virtual surface (125), the linear beam shape projection following a respective current direction of the directional user input (140) along a spatial route of the directional user input (140); The second reference system (135) is defined as a second virtual surface (130; 160); and For each local pattern (P i ), the associated mapping rule defines a mapping between the two virtual surfaces (120a; 130, 160), wherein one of the surfaces (120a; 130, 160) serving as a target surface of the mapping rule is defined as the common reference system (120a or 135).

9. The calibration method (400) according to claim 8, wherein: The first reference system (120a) is defined based on one or more previously detected directional user inputs (140) such that these one or more previously detected user inputs (140) correspond to points and / or lines on the first virtual surface (125).

10. The calibration method (400) according to claim 8 or 9, wherein: At least one of the two surfaces (120a; 130, 160) is defined such that it comprises a virtual plane or a delimited surface section of the virtual plane.

11. The calibration method (400) according to claim 10, wherein: The first virtual surface (125) is defined such that it comprises a virtual plane or a bounded surface segment of the virtual plane, the virtual plane or the bounded surface segment of the virtual plane being orthogonal to the direction of the previously detected directional user input (140).

12. The calibration method (400) according to any one of claims 8 to 11, wherein: The first virtual surface (125) is defined as satisfying at least one of the following conditions: It forms a boundary surface which encloses the three-dimensional spatial region at least on two sides; It comprises at least two separate partial surfaces (125a, 125b) which are non-parallel at least in some parts; Each linear beam shape projection of a directional user input (140) through the first virtual surface (125) intersects the first virtual surface (125) at exactly one point along the respective current direction of the projection.

13. The calibration method (400) according to any one of the preceding claims, wherein: Determine the local pattern (P i ) such that each of them is at least proportionally represented by a sequence of discrete points on the input pattern (305).

14. The calibration method (400) according to any one of the preceding claims, wherein: In order to determine the local pattern (P i ), the identification pattern (310) and the local pattern (P) are mapped in the common reference system (120a or 135) based on homogeneous coordinates. i ) to describe the recognition pattern (310) on the one hand and the local pattern (P i ).

15. A method (600) for sensing a contactless directional user input (140) of a user relative to a user interface (UI) of a target system by means of a sensing system (110), wherein: The method comprises: calibrating (605) the sensing system (110) according to a calibration method (400) as claimed in any one of the preceding claims; receiving or generating (610) input data based on a sensor, the input data representing a sensed non-contact directional user input (140) of a user relative to the UI in the first reference frame (120a); converting (615) the input data into output data, using a mapping rule determined in the context of the calibration method (400) or a mapping rule inverse thereto, the output data representing an input at the UI corresponding to the user input (140) and converted into the second reference frame (135); and The output data is transmitted (620) to the target system.

16. A sensing system (110) for sensing a contactless directional user input (140) of a user relative to a user interface (UI) of a target system, wherein: The sensing system (110) is configured to perform the calibration method (400) according to any one of claims 1 to 14 or the method according to claim 15.

17. The sensing system (110) according to claim 16 further includes a sensor device, which is used to sense and detect the non-contact user input (140) and to generate sensor data and / or input data, which represent these user inputs (140) in the first reference frame (120a).

18. A vehicle (105) comprising the sensing system (110) according to claim 17.

19. A computer program or a computer program product comprising instructions which, when executed on one or more processors of a sensing system (110) according to claim 16 or 17, cause the computer program or the computer program product to perform a calibration method (400) according to any one of claims 1 to 14 or a method according to claim 15.

Images