Method, device and computer program for estimating directional information conveyed by a free-space gesture to determine user input at a human-machine interface

Abstract

Method for estimating directional information conveyed by a free-space gesture for determining user input at a human-machine interface (1), in particular of a vehicle, wherein the method comprises: Receiving sensor-acquired measurement data (MD) that represents a free-space gesture performed by a user with respect to the human-machine interface as a spatially multidimensional inhomogeneous point cloud in a correspondingly multidimensional space; Determine, based on the measurement data (MD), density data (PD) that represent a density distribution corresponding to the point cloud, in which density information is assigned to each point of the point cloud, which characterizes a local point density of the point cloud in the vicinity of the respective point; Applying a dimension-reducing approximation method to the density data (PD) to determine estimated data (ED) that represent an estimate of a one-dimensional direction curve characterizing at least one directional aspect of the density distribution; and Generating and providing direction data (RD) which represents direction information determined by the spatial progression of the direction curve represented by the estimation data (ED) in the higher-order multidimensional space.

Description

[0001] The present invention relates to a method and a device for estimating directional information conveyed by a free-space gesture for determining a user input at a human-machine interface, in particular a vehicle. Furthermore, the invention relates to a computer program configured to execute the method.

[0002] While traditional human-machine interfaces (abbreviated to "HMI" or more commonly "HMI") are usually based on a contact-based interaction between a user and a corresponding control element, such as a physical switch or a touch-sensitive surface or display device, HMIs are now known in which a static or dynamic gesture performed by a human user in free space without physical contact with an HMI device is sensorily recorded. This sensory recording of such gestures, also known as "free space gestures," can be achieved, for example, using a suitable camera. Mathematical methods, in particular image recognition methods, are then used to draw conclusions from the sensorily recorded data about the user input intended by the recorded gesture.

[0003] Particularly in the context of vehicle-related applications, such MIIs capable of recognizing free-space gestures can be used to provide inputs for operating the vehicle or a subsystem thereof. Furthermore, it is also possible for the free-space gestures to relate to the vehicle's surroundings, such as pointing at an interesting object external to the vehicle that is visible from the vehicle. The recognition of such a free-space gesture serves to provide input to a system intended to provide information related to the object or to prompt the vehicle to react to it.

[0004] Gesture recognition systems are known for implementing such contactless MMIs. These systems use calibrated 2D sensors to detect and track points associated with specific features along the contour of a user's hand, particularly from above and from the side. Based on the detected points, a pointing direction associated with the gesture performed in three-dimensional space can then be estimated. Such systems depend on the recognition of clearly distinguishable features (points) of the human hand and require fixed and calibrated hardware with at least two 2D sensors.

[0005] Furthermore, methods are known that use body-related information to infer a pointing direction expressed by a gesture. In most cases, a head position and a hand position are detected, and a pointing direction is estimated based on these. Orientations or even movements of the fingers of the hand are not detected or taken into account.

[0006] Furthermore, US 2005 / 0094879 A1 and the following documents are known from the state of the art, which also deal with the recognition of free space gestures: - GE, Liuhao [et al.]: Robust 3D Hand Pose Estimation From Single Depth Images Using Multi-View CNNs. In: IEEE Transactions on Image Processing, Vol. 27, 2018, No. 9, pp. 4422-4436. - ISSN 1941-0042; - DOMINIO, Fabio; DONADEO, Mauro; ZANUTTIGH, Pietro: Combining multiple depth-based descriptors for hand gesture recognition. In: Pattern Recognition Letters, Vol. 50, 2014, pp. 101-111. - ISSN 0167-8655

[0007] The present invention is based on the object of further improving the recognition of free space gestures and in particular the determination of directional information conveyed thereby.

[0008] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0009] A first aspect of the invention relates to a method, in particular a computer-implemented method, for estimating directional information conveyed by a free-space gesture for determining a user input at a human-machine interface, in particular of a vehicle. The method comprises: (i) receiving sensor-detected measurement data that describe a free-space gesture performed by a user with respect to the human-machine interface, in particular by means of an arm, hand and / or finger, as a spatially multidimensional, ieRepresent a 2-dimensional or 3-dimensional, inhomogeneous point cloud in a correspondingly multidimensional space; (ii) Determining, on the basis of the measurement data, density data which represent a density distribution corresponding to the point cloud, in which density information is assigned to each individual point of the point cloud, which characterizes a local point density of the point cloud in the vicinity of the respective point; (iii) Applying a dimension-reducing approximation method to the density data to determine estimated data which represent an estimate of a one-dimensional directional curve characterizing at least one directional aspect of the density distribution; and (iv) Generating and providing directional data which represent directional information determined by the spatial course of the directional curve represented by the estimated data in the higher-level multidimensional space.

[0010] "Directional information" within the meaning of the invention is understood, in particular, to mean information that indicates at least one specific, selected direction in space. It can optionally also indicate its temporal dependence, if applicable.

[0011] A "point cloud" within the meaning of the invention is understood to mean, in particular, a set of points in a vector space, in particular three-dimensional space or a two-dimensional subspace thereof, which has an unorganized spatial structure ("cloud"). A point cloud is described by the points it contains, each of which can be captured, in particular, by their spatial coordinates in a coordinate system associated with the vector space.

[0012] A "directional curve" within the meaning of the invention is understood to mean, in particular, a one-dimensional curve, or in particular, a one-dimensional directional curve in a higher-dimensional space, which has at least one directional aspect characterized by it. The directional curve, in particular in the simplest case as a one-dimensional directional curve, can represent a straight line whose direction simultaneously indicates the directional aspect. However, in the general case, it can also have any curvature, including multiple curvatures. Preferably, the directional curve is continuous and differentiable, at least for the most part, ideally everywhere.

[0013] In the solution according to the invention, the desired directional information is therefore not determined directly from the point distribution of the acquired point cloud. Instead, a density distribution representing a probability distribution is first derived from the point distribution. On this basis, a one-dimensional directional curve characterizing the desired directional information is determined using the aforementioned dimension-reducing approximation method. This curve characterizes the desired directional information. In this way, the achievable reliability and accuracy in the estimation-based determination of directional information from point clouds can be increased. Furthermore, the method requires relatively low computing resources, making it particularly suitable for real-time applications.In addition, the method is independent of a specific orientation of the sensor used to collect the measurement data, for example a camera, which makes it particularly well suited for determining pointing gestures, especially finger-based ones, within a vehicle, since the sensor can be arranged in a wide variety of positions there.

[0014] Preferred embodiments of the method are described below, which, unless expressly excluded or technically impossible, can be combined with each other as well as with the other aspects of the invention described or used as corresponding embodiments of the latter.

[0015] In some embodiments, the sensory multidimensional capture of the free-space gesture performed by a user with respect to the human-machine interface is carried out using a TOF camera system. A "TOF camera system" within the meaning of the invention is understood to be a 3D camera system that measures distances based on a time-of-flight (TOF) method. The measuring principle is based on the fact that the scene to be recorded is illuminated using a light pulse, and the camera measures the time it takes for the light to reach the object and back for each pixel. Due to the constancy of the speed of light, this required time is directly proportional to the distance. The camera thus provides the distance to the object imaged on it for each pixel.The use of a TOF camera system represents a particularly effective and high-resolution implementation option, requiring at least one sensor for sensory detection of the free space gesture.

[0016] In some embodiments, the multidimensional space is three-dimensional and the measurement data represents a point cloud spatially depicting the detected free space gesture in three dimensions, 3D.

[0017] In a first variant, a corresponding density distribution is determined directly for the 3D point cloud by assigning density information to each individual point of the 3D point cloud, which characterizes a local density of the 3D point cloud in the vicinity of the respective point. Associated density data representing the resulting 3D density distribution is generated for the 3D point clouds. A dimension-reducing mapping of the 3D point cloud to one or, in particular, several point clouds of lower dimensions is not required in this variant, thus eliminating the need for corresponding process steps.

[0018] In a second, alternative variant, determining the density data comprises: (i) dimension-reducing mapping of the 3D point cloud onto a plurality N, where N > 1, of spatially two-dimensional, 2D, point clouds, each defined in one of N associated 2D subspaces of the multidimensional space, of which at least two are not parallel to one another in the superordinate 3D space; and (ii) determining, for each of the N 2D point clouds, a corresponding density distribution, in which the individual points of the respective 2D point cloud are each assigned density information which characterizes a local density of the respective 2D point cloud in the vicinity of the respective point, wherein associated density data representing the respective resulting 2D density distribution are generated for each of the 2D point clouds.Furthermore, the approximation method is applied separately for each of the 2D density distributions represented by the respective density data to estimate a respective one-dimensional, 1D, directional curve characterized by it, wherein associated estimated data representing the respective 1D directional curve are generated for each of the 2D density distributions. Furthermore, the directional information is determined based on a, in particular weighted or unweighted, averaging of at least two, preferably all, of the individual 1D directional curves represented by the respective estimated data in the higher-level 3D space in order to obtain directional information related to the 3D space.

[0019] The latter can be achieved, in particular, by determining the gradient with respect to a 1D directional curve resulting from the averaging. This, as well as any further embodiments based on it, have the advantage that the complexity of the subsequent calculations can be significantly reduced due to the dimensional reduction, since less data needs to be processed overall, and only data related to two dimensions. This allows, in particular, performance to be increased and thus the real-time capability of the method to be achieved or improved.

[0020] In some of these embodiments according to the second variant, the dimension-reducing mapping comprises a projection mapping along one of the three dimensions of the 3D point cloud represented by the measurement data onto a corresponding 2D point cloud in an associated one of the N 2D subspaces. The dimension reduction to two dimensions is thus carried out by means of projection. For example, when using a Cartesian coordinate system, the projection can be carried out in particular along one of the axes of this coordinate system, so that, for example, all points of the point cloud in the three-dimensional space spanned by the coordinate axes X, Y, and Z are projected along the Z axis onto the X / Y plane. Of course, this is only an example and should not be understood as a limitation.In particular, the projection can be performed along any projection direction and instead of a Cartesian coordinate system, another coordinate system of the same dimension can be used to represent the coordinates of the points.

[0021] In some embodiments, determining the density data further comprises: (i) spatially quantizing the multidimensional point cloud represented by the measurement data or, if applicable, each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing mapping, and (ii) determining the density data based on a resulting density distribution of the points of the corresponding quantized point cloud. Instead of a continuous mapping or a mapping determined depending on the discrete resolution of the sensor system generating the measurement data, the point cloud is now mapped onto a predefined spatial grid and thus quantized. The individual cells of the grid can be referred to as "voxels" in the three-dimensional case and as "pixels" in the two-dimensional case, particularly when using a rectangular or even cubic grid. Quantization, in turn, simplifies the complexity of the subsequent calculations and, in particular, enables the use of simple indexing for the voxels or pixels instead of coordinate specifications.In particular, the complexity reduction can include replacing floating-point arithmetic with simpler and less complex integer arithmetic. This simplifies and accelerates implementation, especially in embedded systems. Furthermore, the type of quantization can be used to adjust the accuracy of the system, and in particular, to increase it, since the type of quantization allows for an adjustable trade-off between accuracy and speed (coarse grid = fast but inaccurate, fine grid = slower but more accurate).

[0022] In some developments of these embodiments, the quantization can additionally relate to a value assigned to the respective voxel or pixel, which is determined on the basis of the points of the point cloud present therein. The quantization can in particular be carried out in binary form, so that during the quantization the value of a voxel or pixel, if its number of points is below a predetermined threshold, is set to a first of the two binary values, for example to "0", whereas if the number of points is above the threshold this value is set to the second of the two binary values, for example "1". If the number of points is equal to the threshold, the value can be set in such a way that it is set to one of the binary values ​​in a predefined manner. In this way, the number of points to be processed can be reduced even further by means of this.Pixels with the second value are taken into account in the further process, which further increases the performance of the process.

[0023] In some of these embodiments, in the case of spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing mapping, the density distribution for each point of the resulting 2D point cloud is determined based on the number of points of the multidimensional point cloud represented by the measurement data that are mapped to the same quantized point of the 2D point cloud according to the projection mapping. Thus, if, during dimension-reducing mapping, many points are mapped to the same pixel of the resulting 2D point cloud by means of the projection, this pixel receives a high value corresponding to or derived from the number of these points. Conversely, if only a few points are mapped to the pixel, this value is correspondingly lower.In this way, the original point cloud can be converted into a two-dimensional density distribution in an effective, easy-to-implement and thus high-performance manner, which characterizes the resulting 2D point cloud.

[0024] In some embodiments, determining the density data further comprises filtering the respective point cloud, from which the respective density distribution represented by the respective density data is determined by means of, preferably multiple, convolution of the density distribution with a respective filter kernel, which may in particular have a low-pass property and may in particular be a smoothing kernel. In one variant, the filter kernel may in particular be a Gaussian filter kernel. With the aid of this filtering, the original density distribution, as it results in particular from the quantization, is smoothed, which can in particular improve the applicability of the subsequent method steps and the quality of the resulting estimation results for the directional information.Without filtering, the information between the points would not be interpolated, resulting in significantly less information being available for subsequent process steps, particularly for a PCA (described below), which could in some cases lead to less reliable results. A further advantage of filtering is that even individual "false" points (e.g., those caused by measurement noise or incorrect measurements) are smoothed out, thus having less impact on the subsequent process, particularly, for example, a PCA. The noise typically associated with TOF sensors, in particular, is thus suppressed by filtering, and the quality of the results can be improved.

[0025] In some embodiments, a principal component analysis, PCA, or a regression method is used as a dimension-reducing approximation method to determine the respective estimated data.

[0026] PCA represents a particularly suitable and high-performance approximation method within the scope of the method according to the invention. In particular, PCA can be weighted so that smaller resulting principal components are neglected for the resulting determination of the directional data. In particular, in the simplest case, PCA can be aborted immediately after determination of the first principal component, so that determination of the further principal components is no longer necessary. The directional information represented by the directional data is then determined based only on the largest principal component determined by PCA (i.e., the component with the largest eigenvalue determined in the PCA). This enables a particularly high-performance and simple implementation.

[0027] In some embodiments where a PCA is applied to a quantized and filtered two-dimensional density distribution F(i,j), the covariance matrix, M, of the PCA receives the following output values: M=[σccσcrσrcσrr] with σcc=∑i∑jF(i,j)(i−μc)2 σcr=σrc=∑i∑jF(i,j)(i−μc)(j−μr) σrr=∑i∑jF(i,j)(j−μr)2 and μc=1∑i∑jF(i,j)∑i∑jF(i,j)⋅i μr=1∑i∑jF(i,j)∑i∑jF(i,j)⋅j where F(i,j)=∑n∑mP(i+n,j+m)⋅K(n,m), and where (i,j) are each index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values ​​P(i,j) or density values ​​F(i,j) filtered by convolution with the kernel K, and the indexing of the kernel K is done with the index pairs (n,m).

[0028] In some embodiments, the method further comprises: (i) segmenting the respective point cloud intended for determining the associated density data therefrom into a plurality of different segments based on a detection of various body parts of the user depicted by the point cloud; (ii) selecting a true subset of the segments using a selection criterion defined as a function of the one or more body parts intended for executing the free-space gesture; and (iii) determining the associated density data based only on the part of the point cloud represented by the selected subset of segments. In this way, it is possible to determine the desired directional information solely on the basis of a section of the sensor-detected point cloud.This is particularly advantageous when the point cloud represents one or more additional body parts or sections of the user beyond a body part relevant for determining the directional information. Segmentation also allows, on the one hand, to avoid unnecessary processing of parts of the point cloud that are irrelevant in this sense, and, on the other hand, to increase the required reliability and accuracy of the results, since these cannot be distorted by irrelevant parts of the point cloud.

[0029] Thus, in some embodiments, the point cloud represented in the measurement data can correspond in particular to a free-space gesture performed with at least part of the upper limbs of a user. In this case, the segmentation contains at least one selection segment that corresponds only to one or more of the user's fingers, and the selection criterion is defined such that the subset of segments determined thereby contains at least, preferably exclusively, this selection segment. Thus, despite sensory detection of a larger part of the upper limbs, a focus can be achieved by means of the segmentation solely on aspects of the free-space gesture performed by the one or more of the user's fingers. In particular, a pointing direction expressed with the finger(s) can be recognized reliably, efficiently, and / or with high accuracy.

[0030] In some embodiments, the measurement data represent a temporal development of the free-space gesture as a spatially multidimensional and simultaneously time-dependent point cloud. Furthermore, the method-based determination of directional information conveyed by the free-space gesture occurs multiple times, each time for a different point in time with respect to the time-dependent point cloud. The directional information is generated and provided in such a way that it represents the temporal development of the directional information resulting from this multiple determination. In this way, a temporal development of the detected free-space gesture can also be tracked, for example, a temporal change in directional information expressed by the free-space gesture, such as a pointing direction.

[0031] In some embodiments (i) the measurement data in M+1 dimensions represent a temporal development of the free-space gesture as a spatially M-dimensional and at the same time correspondingly time-dependent point cloud, with M>1, (ii) the determination of the density data is preceded by a dimension-reducing mapping of the point cloud represented by the measurement data in M ​​spatial and an additional temporal dimension, with the temporal dimension omitted, onto an at least two-dimensional space in order to obtain a representation of a 2D or 3D spatial region corresponding to the free-space gesture; and (iii) the directional data are generated and provided such that the directional information represented by them, which is determined by the spatial course of the directional curve represented by the estimation data in the higher-level multi-dimensional space, represents a 2D or 3D spatial region defined by the directional curve and at least partially enclosed by it.3D spatial area. The dimension-reducing mapping of the point cloud represented by the measurement data in M ​​spatial and an additional temporal dimension can be carried out in particular by means of projection along the temporal dimension and, in the case of M=3, optionally additionally along one of the spatial dimensions. In this way, the method can be extended or designed such that it can be used to estimate directional information conveyed by a free-space gesture for determining a user input at a human-machine interface, wherein the directional information, instead of or in addition to a pointing direction, identifies a spatial area defined by the free-space gesture and thus describes in particular a region, i.e. a 2D or 3D spatial area.

[0032] This can be explained using a short example: Suppose the user performs a three-dimensional pointing gesture (M=3) in which they move their hand or arm in a circular motion, such that the pointing direction follows a temporal progression, describing a cone. The cone tip is (i) the base of the finger (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement originating from it or (iii) the shoulder joint (in the case of a movement of the entire arm originating from it), and the region desired by the user is located in the interior of the cone. The point cloud can then initially be described in M ​​+ 1 = 4 dimensions with three spatial and one time dimension, and from this, a dimension-reducing projection along the time dimension and optionally one of the spatial dimensions onto a three- or two-dimensional space can be made. The resulting point cloud in this reduced space could then (e.g.For example, the gesture (for the above gesture) must have a shape or behavior that at least partially encloses a spatial region, e.g., by exhibiting an at least largely closed ring shape. Based on this, the method can then generate corresponding density data, and from this, directional information that describes the region outlined by the free-space gesture. Thus, the applicability of the method can be extended from the estimation of pure pointing gestures to the estimation of spatial gestures.

[0033] In some embodiments, determining directional information conveyed by the free-space gesture based on the estimation data involves determining at least one directional vector lying tangential to the directional curve represented by the estimation data. This can be done, in particular, by means of a derivation in the sense of mathematical differentiation or gradient formation with respect to the directional curve. In this way, characterizing directional information can be easily obtained from the directional curve, which simultaneously corresponds to an estimate of the directional information conveyed by the free-space gesture.

[0034] A second aspect of the invention relates to a device for estimating directional information conveyed by a free-space gesture for determining a user input at a human-machine interface, in particular of a vehicle, wherein the device is configured to carry out the method according to the first aspect of the invention. The device can, in particular, be a control unit for a vehicle, in particular a motor vehicle, wherein the control unit itself comprises the said human-machine interface or is configured to be signal-connected to such an interface for receiving the measurement data therefrom.

[0035] A third aspect of the invention relates to a computer program configured to carry out the method of the first aspect of the invention. The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules.

[0036] The device according to the second aspect of the invention can accordingly comprise a program memory in which the computer program is stored. Alternatively, the device can also be configured to access an external computer program, for example, available on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0037] The features and advantages explained above with respect to the first aspect of the invention also apply accordingly to the further aspects of the invention.

[0038] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the figures.

[0039] This shows Fig. 1 schematically shows a system for estimating directional information including an estimator, according to various embodiments of the invention; Fig. 2 a flow chart illustrating a preferred embodiment of the method according to the invention; Fig. 3 for the purpose of illustrating the segmentation process S1 from Fig. 2 shows a two-dimensional representation of an exemplary point cloud represented by corresponding measurement data obtained by recording a human arm, wherein a segment corresponding to the fingers of the arm is marked; Fig. 4 graphic representations to illustrate during the procedure according to Fig. 2 occurring data and derived from the measured data; and Fig. 5 is an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information defining a 2D surface area selected by the free space gesture, according to embodiments of the invention.

[0040] Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.

[0041] Fig. 1 schematically shows a system 1 for estimating directional information according to various embodiments of the invention. The system 1 comprises an estimation device 2 and a sensor 3 for the three-dimensional detection of a free-space gesture performed by a user of the system. The estimation device 2 can, in particular, be a computer, which can be equipped in particular with a processor platform 2a and a program and data memory 2b as well as a data output 2c for outputting output data determined by the computer, in particular in the form of directional information. The sensor 3 can, in particular, comprise a TOF (time of flight) camera system. Other sensor types, in particular sensors based on ultrasound or radar measurement, can also be used instead or in combination therewith.A computer program can be stored in the program memory 2b, which can consist of one or more program modules and which is configured, when running on the processor platform 2a, to cause the estimation device 2 to carry out the method according to the invention, for example as described below with reference to FIG. Fig. 2 to 5.

[0042] The Fig. The flowchart shown in Figure 2 serves to illustrate a preferred embodiment of the method according to the invention for estimating directional information conveyed by a free space gesture for determining a user input at a human-machine interface, which in particular corresponds to or can have system 1. The actual estimation method, which here comprises, for example, the processes S1 to S8, is preceded by a measuring process S0, in which sensory, in particular with a measuring sensor 3 according to Fig. 1, measurement data MD are generated, which represent a spatial three-dimensional point cloud, which in turn represents a free space gesture recorded during the measurement and performed by a user. Fig. 3 shows a two-dimensional representation of an exemplary 3D point cloud obtained in this way, representing parts of the upper arm, the forearm and the hand of an arm of a user performing the free space gesture.

[0043] In the embodiment of the estimation method described here, the three-dimensional point cloud represented by the measurement data MD is first segmented within the scope of preprocessing in a segmentation process S1 using one or more known methods, for example, a “random forest” classification method, in order to obtain segment data SD that correspond to the section of the measurement data MD relevant for the further process flow. Fig. In Figure 3, such a section, corresponding to the fingers of the hand, is marked by a black box.

[0044] The point cloud represented by the segment data SD is then mapped into N=2 two-dimensional point clouds in a projection process S2. This is advantageously done by projecting along one of the three spatial dimensions, here as an example - without this being understood as a limitation - using a Cartesian coordinate system, on the one hand along the Z direction, resulting in a two-dimensional projection of the point cloud onto the XY plane, and on the other hand along the Y direction, resulting in a second projection onto the XZ plane. Fig. 4(a) and 4(e) graphically show these two-dimensional projections resulting from the segment data SD, which are each represented by corresponding projection data BD xy or BD xz be represented.

[0045] This is followed by a quantization process S3 in which each of the projection data BD xy or BD xz represented 2D point clouds are quantized according to a predefined spatial grid, so that now only predefined spatial positions are defined for the points (pixels). The two quantized 2D point clouds, which are shown as examples in the Fig. 4(b) and 4(f) respectively, are represented by corresponding image data ID xy or ID xz The image data can thus be represented by corresponding data fields as follows: Ixy(i,j) and ixy(i,j) with the 2D pixel index i=1,...,k and j=1,...,l

[0046] The quantization process S3 completes a preprocessing section of the method, which precedes the actual estimation of the directional information.

[0047] This estimation is carried out in an estimation section of the procedure, in which ID xy or ID xz in a probability estimation process S4 corresponding density data PD xy or PD xz To do this, each pixel of the two quantized 2D point clouds is assigned a density value equal to the number of points mapped to the corresponding pixel according to the image data using the preceding processes S2 and S3. The density data can thus be represented by corresponding data fields as follows: Pxy(i,j) and Pxz(i,j) with i=1,...,k and j=1,...,l

[0048] The Fig. Figures 4(c) and 4(g) graphically represent examples of such quantized 2D data fields, where the value of a pixel is represented by a corresponding gray value for illustrative purposes, with a darker color corresponding to a higher probability. It is also conceivable to choose another value dependent on or derived from the number of points instead of the number of points, which is suitable for at least indirectly characterizing this number of points.

[0049] This is followed by a filtering process S5, in which the density data PD xy or PD xz by convolution using a kernel K, which is in particular a Gaussian 2D kernel, e.g. of the form: K(n,m,σ)=a⋅e−n2+m22σ2 can be filtered as follows: F(i,j)=∑n∑mP(i+n,j+m)⋅K(n,m).

[0050] The kernel K is indexed with the index pairs (n,m). The kernel K has the variance σ, which can be selected as a parameter, as well as a normalization factor a, which is chosen such that it normalizes the sum of all kernel elements to 1. The filtering, which preferably represents a smoothing, results in corresponding filter data FD xy or FD xz which are exemplified in the Fig. 4(d) and 4(h) as corresponding filtered pixel clouds F xy (i,j) or F xz (i,j) are shown graphically.

[0051] In a subsequent approximation process S6, which is carried out using a dimension-reducing approximation method, which in this example is chosen as principal component analysis (PCA), corresponding estimated data ED are then obtained from the filter data, each of which represents a directional curve that is Fig. 4(d) and 4(h) respectively, is represented by the corresponding white arrow. In the simplest case, only the largest principal component determined by PCA is considered, which defines the direction and position of the corresponding white arrow.

[0052] In the case of using a PCA with respect to the quantized and filtered two-dimensional density distribution F(i,j), the covariance matrix, M, of the PCA can in particular receive the following output values: M=[σccσcrσrcσrr] with σcc=∑i∑jF(i,j)(i−μc)2 σcr=σrc=∑i∑jF(i,j)(i−μc)(j−μr) σrr=∑i∑jF(i,j)(j−μr)2 and μc=1∑i∑jF(i,j)∑i∑jF(i,j)⋅i μr=1∑i∑jF(i,j)∑i∑jF(i,j)⋅j where F(i,j)=∑n∑mP(i+n,j+m)⋅K(n,m), and where (i,j) are each index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values ​​P(i,j) or density values ​​F(i,j) filtered by convolution with the kernel K, and the indexing of the kernel K is done with the index pairs (n,m).

[0053] Finally, in a direction determination process S8, the desired direction information can be derived from the estimated data ED and represented using corresponding direction data RD. This can be done, in particular, by converting the one-dimensional direction curves (white arrows) represented by the estimated data ED, which are defined in the respective 2D space, into a common three-dimensional space by extending the two 2D spaces by the missing third dimension, and then averaging the estimated data ED in the common 3D space (this corresponds to a communication of the two white arrows) to obtain a resulting direction curve.In the more general case, the estimated data ED each represent a curved directional curve, so that after averaging the two directional curves, a resulting directional curve, which is usually also curved, is created in 3D space, from which a desired directional information can be determined, for example by means of gradient formation at a desired point of the resulting directional curve or by means of linear regression.

[0054] The above with reference to the Fig. The method described by way of example in Figures 2 to 4 can in particular also be carried out for different points in time during the execution of the free space gesture, so that time-dependent directional information is obtained and thus the temporal dynamics of a free space gesture can also be recorded.

[0055] Fig. 5 finally shows an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information defining a 2D surface area selected by the free space gesture, according to embodiments of the invention.

[0056] Suppose the user performs a three-dimensional pointing gesture (M=3) in which he moves the hand or arm in a circular motion such that the pointing direction follows a temporal progression (here characterized by the five points in time t0 to t4) in which it describes a cone C. The cone tip T is (i) the base of the finger (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement originating from it) or (iii) the shoulder joint (in the case of a movement of the entire arm originating from it), and the region desired by the user is located in the interior of the cone, optionally also in its projection along the cone axis towards even greater distances from T.The point cloud can then initially be described in M ​​+ 1 = 4 dimensions with three spatial and one time dimension, and from this, a dimension-reducing projection along the time dimension and optionally one of the spatial dimensions onto a three- or two-dimensional space can be made. The resulting point cloud in this reduced space could then (e.g., for the gesture above) exhibit a shape or behavior that at least partially encloses a spatial region, e.g., the hatched two-dimensional area or region A, e.g., by, as in . Fig.5 has an at least largely closed ring shape. Based on this, corresponding density data FD and, in turn, directional information or corresponding directional data RD can be generated according to the method, which describes or represents the region A outlined by the free-space gesture. In particular, this can also be used to select an object X located in the region A using the free-space gesture. The object can, in particular, correspond to a control element of a human-machine interface or another element of vehicle equipment. Objects outside the vehicle are also conceivable. Thus, the applicability of the method can be extended from the estimation of pure pointing gestures to the estimation of spatial gestures.

[0057] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide one skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents. List of reference symbols 1 System for estimating directional information 2 Estimating device 2a processor platform 2b Program and data memory 2c Data output 3 TOF camera system MD measurement data SD segment or segment data representing it BD projection data ID image data PD density data FD filter data ED estimated data RD direction data A 2D area selected by a free space gesture C cone T cone tip X object enclosed by the selected 2D area t0-t4 Sequence of different times corresponding to the execution of the free space gesture

Claims

[1] Method for estimating directional information conveyed by a free space gesture for determining a user input at a human-machine interface (1), in particular of a vehicle, the method comprising: Receiving sensory-detected measurement data (MD) representing a free-space gesture performed by a user with respect to the human-machine interface as a spatially multidimensional inhomogeneous point cloud in a correspondingly multidimensional space; Determining, on the basis of the measurement data (MD), density data (PD) which represent a density distribution corresponding to the point cloud, in which the individual points of the point cloud are each assigned density information which characterizes a local point density of the point cloud in the vicinity of the respective point; Applying a dimension-reducing approximation method to the density data (PD) to determine estimated data (ED) representing an estimate of a one-dimensional directional curve characterizing at least one directional aspect of the density distribution; and Generating and providing directional data (RD) representing directional information determined by the spatial course of the directional curve represented by the estimation data (ED) in the higher-level multidimensional space. [2] The method of claim 1, wherein: the multidimensional space is three-dimensional, and the measurement data (MD) represent a point cloud spatially depicting the captured free space gesture in three dimensions, 3D; determining the density data (PD): - Dimensionally reducing mapping of the 3D point cloud to a plurality N, with N > 1, of spatially two-dimensional, 2D, point clouds (BD), each defined in one of N associated 2D subspaces of the multidimensional space, of which at least two are not parallel to each other in the superordinate 3D space; and - determining for each of the N 2D point clouds a density distribution corresponding thereto, in which density information is assigned to the individual points of the respective 2D point cloud, which density information characterizes a local density of the respective 2D point cloud in the vicinity of the respective point, wherein for each of the 2D point clouds, assigned density data (PD) representing the respective resulting 2D density distribution are generated; the approximation method is applied separately for each of the 2D density distributions represented by the respective density data (PD) to estimate a respective one-dimensional, 1D, directional curve characterized by it, wherein for each of the 2D density distributions, associated estimated data (ED) representing the respective 1D directional curve are generated; and the determination of the directional information is carried out on the basis of an averaging of at least two of the individual 1D directional curves represented by the respective estimation data (ED) in the higher-level 3D space in order to obtain directional information related to the 3D space. [3] The method of claim 2, wherein the dimension-reducing mapping comprises a projection mapping along one of the three dimensions of the 3D point cloud represented by the measurement data (MD) onto a corresponding 2D point cloud (BD) in an associated one of the N 2D subspaces. [4] A method according to any one of the preceding claims, wherein determining the density data (PD) further comprises: Spatially quantizing the multidimensional point cloud represented by the measurement data (MD) or, if applicable, each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing mapping, and determining the density data (PD) on the basis of a resulting density distribution of the points of the corresponding quantized point cloud (ID). [5] Method according to claim 4, wherein in the case of spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing mapping, the density distribution for each point of the resulting 2D point cloud is determined on the basis of the number of those points of the multidimensional point cloud represented by the measurement data (MD) which are mapped to the same quantized point of the 2D point cloud according to the projection mapping. [6] A method according to any one of the preceding claims, wherein determining the density data (PD) further comprises: Filtering the respective point cloud, from which the respective density distribution represented by the respective density data (PD) is determined, by convolving the density distribution with a respective filter kernel having a low-pass property. [7] Method according to 6, wherein the filter kernel is a Gaussian filter kernel. [8] Method according to one of the preceding claims, wherein a principal component analysis, PCA, or a regression method is used as the dimension-reducing approximation method for determining the respective estimated data (ED). [9] Method according to claim 8, wherein, in the case of using a PCA, the directional information represented by the directional data (RD) is determined on the basis of only the largest principal component determined by means of the PCA. [10] Method according to claim 8 or 9, wherein in the case of using a PCA with respect to a quantized and filtered two-dimensional density distribution F(i,j), the covariance matrix, M, of the PCA receives the following output values: M=[σccσcrσrcσrr] with σcc=∑i∑jF(i,j)(i−μc)2 σcr=σrc=∑i∑jF(i,j)(i−μc)(j−μr) σrr=∑i∑jF(i,j)(i−μr)2 and μc=1∑i∑jF(i,j)∑i∑jF(i,j)⋅i μr=1∑i∑jF(i,j)∑i∑jF(i,j)⋅j where F(i,j)=∑n∑mP(i+n,j+m)⋅K(n,m), and where (i,j) are each index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values ​​P(i,j) or density values ​​F(i,j) filtered by convolution with the kernel K, and the indexing of the kernel K is done with the index pairs (n,m). [11] Method according to one of the preceding claims, further comprising: Segmenting the respective point cloud intended for determining the associated density data (PD) therefrom into several different segments (SD) based on a recognition of different body parts of the user depicted by the point cloud; Selecting a proper subset of the segments (SD) using a selection criterion defined depending on the one or more body parts designated to execute the free space gesture; and Determining the corresponding density data (PD) based only on the part (SD) of the point cloud represented by the selected subset of segments (SD). [12] Method according to claim 11, wherein the point cloud represented in the measurement data (MD) corresponds to a free space gesture performed with at least part of the upper limbs of a user; the segmentation contains at least one selection segment (SD) which corresponds only to one or more fingers of the user, and the selection criterion is defined such that the subset of the segments (SD) determined thereby contains at least this selection segment (SD). [13] Method according to one of the preceding claims, wherein the measurement data (MD) represent a temporal development of the free space gesture as a spatially multidimensional and at the same time correspondingly time-dependent point cloud; and the procedural determination of a directional information conveyed by the free space gesture is carried out several times, each time for a different point in time with respect to the time-dependent point cloud; and the direction data (RD) are generated and provided in such a way that they represent the temporal development of the direction information resulting from this multiple determination. [14] A method according to any one of the preceding claims, wherein: the measurement data (MD) in M+1 dimensions represent a temporal development (t0-t4) of the free space gesture as a spatially M-dimensional and at the same time correspondingly time-dependent point cloud, with M>1; the determination of the density data (PD) is preceded by a dimension-reducing mapping of the point cloud represented by the measurement data (MD) in M ​​spatial and an additional temporal dimension, with the temporal dimension omitted, to an at least two-dimensional space in order to obtain a representation of a 2D or 3D spatial region (A) corresponding to the free space gesture; and the direction data (RD) are generated and provided in such a way that the direction information represented by them, which is determined by the spatial course of the direction curve represented by the estimation data (ED) in the higher-level multi-dimensional space, characterizes a spatial area defined by the direction curve and at least partially enclosed by it. [15] Method according to one of the preceding claims, wherein determining a directional information conveyed by the free space gesture on the basis of the estimation data (ED) comprises determining at least one directional vector lying tangentially to the directional curve represented by the estimation data (ED). [16] Device (1) for estimating directional information conveyed by a free space gesture for determining a user input at a human-machine interface, in particular of a vehicle, wherein the device is configured to carry out the method according to one of the preceding claims. [17] A computer program configured to carry out the method according to any one of claims 1 to 15.

Images