Method for detecting a hand gesture

Abstract

The invention relates to a method for detecting a hand gesture. Image data are captured, the image data representing a two-dimensional image of a hand (1) of a user. Feature points (2, 3, 4, 5, 6) are determined in the two-dimensional image of the hand (1), the feature points (2, 3, 4, 5, 6) comprising a wrist point (2) and finger joint points (3, 4, 5) and a fingertip point (6) for the fingers of the hand (1), the finger joint points (3, 4, 5) each comprising an MCP joint point (3), a PIP joint point (4) and a DIP joint point (5) for a finger. A hand posture is determined on the basis of relative positional relationships of the feature points (2, 3, 4, 5, 6) to one another in the two-dimensional image of the hand (1).

Description

[0001] 2024PF01412

[0002] 1 / 20

[0003] METHOD FOR DETECTING A HAND GESTURE

[0004] The present invention relates to a method for recognizing a hand gesture. In particular, a hand position can be recognized based on a two-dimensional image of the hand. Hand gestures can be used, in particular, for contactless gesture control.

[0005] 5 of a user interface in a vehicle.

[0006] Modern vehicles typically have at least one control unit that allows the driver or other passengers to control various functions. These can include various vehicle or comfort features, such as navigation system settings, climate control, seat adjustments, lighting settings, and the like. It can also control various functions of an infotainment system, such as playing music, making phone calls, and so on.

[0007] It is known that certain functions in vehicles can be controlled contactlessly through gestures, especially hand gestures. For this, a user performs certain predefined actions.

[0008] 15 gestures, for example, in a specific area of ​​the vehicle cabin above the center console or in front of the dashboard, are detected by a corresponding sensor or camera. Gesture control enables intuitive human-machine interaction.

[0009] Other electronic devices, such as computer monitors or smart TVs, are also affected.

[0010] 20. These devices can have corresponding user interfaces through which a user can perform various actions, such as selecting channels, changing settings, choosing streaming services, or the like. For such types of electronic devices, at least one input device, such as a computer mouse or a remote control, is necessary, with which a cursor is controlled and various actions are performed by clicking or...

[0011] Up to 25 keystrokes can be selected. A touchscreen, operated by touch, can also be provided. However, touchless input via gesture control can also be advantageous here, as it eliminates the need for a separate input device.

[0012] The recognition of gestures, especially hand gestures, usually requires a

[0013] 30 Three-dimensional capture of the hand. A 3D sensor, such as a time-of-flight (ToF) camera, can be used as the capture device. This allows for very 2024PF01412

[0014] 2 / 20 precise data, especially regarding the position and posture of the hand in three-dimensional space, can be obtained, enabling highly accurate recognition of hand gestures. However, the 3D sensor is relatively expensive. Furthermore, processing the three-dimensional data requires significant computing power. For evaluation,

[0015] 5 typically complex trained machine learning models are required.

[0016] Therefore, methods are also known that initially use two-dimensional image data, such as simple camera images. Specific points of the hand can be extracted from these images. However, to recognize hand gestures, the two-dimensional data is often transformed into three-dimensional space. This transformation then allows for the recognition of hand gestures using appropriately trained machine learning models or by comparing the data with stored gesture data, as mentioned above. This process is computationally intensive, and the transformations can be error-prone, especially since two-dimensional image data relating to hand positions can already contain inaccuracies that can be further amplified by the transformations.

[0017] 15. Transformation into three-dimensional data can sometimes be amplified. Therefore, it is often only possible to generate a correct three-dimensional model of a hand to a limited extent from two-dimensional data.

[0018] The present invention aims to simplify the recognition of hand gestures. In particular, the recognition of hand gestures is intended to be based on...

[0019] 20 two-dimensional image data will be used.

[0020] The solution to this problem is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0021] A first aspect of the invention relates to a, in particular computer-implemented,

[0022] 25. Method for recognizing a hand gesture. In this method, image data is acquired, the image data representing a two-dimensional image of a user's hand. Feature points are determined in the two-dimensional image of the hand, wherein the feature points include a wrist point and, for each finger of the hand, finger joint points and a fingertip point, wherein the

[0023] 30 finger joint points, each comprising one MCP joint point, one PIP joint point, and one DIP joint point. This then results in a hand position 2024PF01412.

[0024] 3 / 20 determined based on the relative positional relationships of the feature points to each other in the two-dimensional image of the hand.

[0025] The aforementioned procedure according to the first aspect is therefore based in particular on the fact that a hand position or hand gesture can be determined (only) on the basis of relative positional relationships of

[0026] The method described in the first aspect directly uses these feature points in the two-dimensional image of the hand to determine a hand position, thus eliminating the need for complex models, for example, for transformations into three-dimensional space. Since no 3D information is required, and hand position recognition relies solely on the 2D points, hand positions can be determined more quickly and accurately.

[0027] To recognize hand posture, distinctive 2D positions of the hand, the so-called keypoints, are determined, so that, as will be explained in more detail below, both bent and extended fingers can be recognized.

[0028] 15 which in turn enables the recognition of a variety of hand gestures, such as an open hand, a pointing pose, or a pose for “two” (index and middle fingers extended, other fingers curled). It should be noted that numerous different methods for determining the feature points of a hand are known, for example, using machine learning methods. Therefore, the recognition or determination of these

[0029] 20 points are assumed to be a known prerequisite for the present invention. These points are then further processed according to the method described in the first aspect of the invention in order to recognize hand gestures.

[0030] The term "hand position" or "hand pose" used here refers in particular to the position of the fingers of a hand in relation to each other, for example whether individual fingers are bent.

[0031] 25 or are extended. Examples of hand positions include a flat hand, a clenched fist, a pointing pose in which only the index finger is extended, whether individual fingers are touching, especially the thumb touching one of the other four fingers, and the like. A hand position can be called a "hand gesture," especially if the hand gesture corresponds exactly to a hand position. The term "hand gesture" can also be used in particular as

[0032] 30 (temporal) sequence of hand positions can be understood.

[0033] The term "keypoints" used here refers specifically to prominent points on a hand. Keypoints for each finger can include: 2024PF01412

[0034] 4 / 20 points are defined, particularly at the fingertips and finger joints, especially at the distal interphalangeal joint, proximal interphalangeal joint, and metacarpophalangeal joint. The metacarpophalangeal joint (articulationes metacarpophalangeales, or "MCP joint" for short) is the articulating connection between the metacarpal bones (ossa metacarpalia) of the

[0035] 5. The metacarpus and the proximal phalanges (finger joints) are located near the midline of the body. The proximal interphalangeal joint (PIP joint) and the distal interphalangeal joint (DIP joint) are the joints between the phalanges. The phalanges can correspond (especially in a skeletal model of the hand) to bone vectors that connect adjacent joints of a finger (or the fingertip of a finger to the corresponding DIP joint). An additional landmark can be identified in the wrist area as the wrist point.

[0036] The term "detection device" used here refers in particular to a device that detects objects in three-dimensional space without contact and

[0037] 15. The corresponding data, such as image data, can be acquired. In particular, the acquisition device can acquire a user's hand. For example, optical methods can be used to acquire a user's hand in space. Within the scope of the invention, the acquisition device can in particular be an image acquisition device. The "acquisition" of image data can thereby

[0038] 20 refers in particular to the recording of image data using the recording device, but alternatively also to, for example, the procurement, such as the loading of existing image data.

[0039] The term "image capture device" used here refers specifically to a camera, particularly a digital camera. The camera can capture still images (photos) or moving images (videos), especially in the visible light (RGB) range.

[0040] 25 or in the infrared (IR) range, especially in the near-infrared (NIR) range. An IR image is well suited for monitoring, for example, a vehicle interior, as it is robust against changing lighting conditions, such as strong sunlight. Recordings are also possible in darkness. It goes without saying that a suitable IR light source must be present.

[0041] 30. An image capture device can capture or record such images and output corresponding image data. An image sequence consisting of several frames constitutes a moving image, in particular a video. 2024PF01412

[0042] 5 / 20

[0043] The term "vehicle" used here refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as sedans, vans, buses, trucks, delivery vans and the like.

[0044] 5. Any terms used herein, such as “comprises”, “includes”, “includes”, “has”, “with”, or any other variant thereof, are intended to cover non-exclusive inclusion. For example, a method or apparatus comprising or comprising a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent in such method or apparatus.

[0045] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive or and not an exclusive "or". For example, a condition A or B is satisfied by one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or

[0046] 15 are present), and both A and B are true (or present).

[0047] The terms "ein" or "eine," as used here, are defined as "one or more." The terms "ein anderer" and "ein Weitere," as well as any other variant thereof, are to be understood as "at least one more."

[0048] The term “plural” or “several”, as it may be used here, is in

[0049] Understanding 20 senses of "two or more".

[0050] The terms “configured” or “set up” to fulfill a specific function (and respective variations thereof) are to be understood within the meaning of the invention as meaning that the corresponding device is already in a configuration or setting in which it can perform the function or at least is adjustable – i.e., configurable – in such a way.

[0051] 25. The fact is that, after appropriate configuration, it can perform the function. This configuration can be achieved, for example, by adjusting parameters of a process flow or by using switches or similar devices to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that

[0052] 30. Configuration can be done by selecting one of these configurations or operating modes. 2024PF01412

[0053] 6 / 20

[0054] Preferred embodiments of the method are described below, which, unless expressly excluded or technically impossible, can be combined with each other and with the other described aspects of the invention as desired.

[0055] 5 In some embodiments, determining the hand position includes at least one of the following: An angle between two finger joints at at least one finger joint point can be determined. A distance between adjacent finger joint points and / or the fingertip point and the DIP joint point of a respective finger can be determined. A distance between the wrist point and at least one fingertip point and / or finger joint point can be determined. It is understood that, in particular, several angles and / or distances can be determined, up to and including all possible or relevant angles and / or distances that can be determined from the feature points. In particular, the relative positional relationships of the feature points can thus be angles and / or

[0056] Fifteen distances are determined. From these relative positions of the feature points, it can be deduced whether a finger (or several fingers) is bent or extended, from which a hand position can be inferred, as will be explained in more detail below.

[0057] In some embodiments, determining the hand position includes determining

[0058] 20 of an angle between two finger joints at at least one finger joint point and a comparison of the determined angle with a threshold value, wherein a finger is determined to be not extended if the angle is smaller than the at least one threshold value. In particular, several angles, in particular up to three angles, can be determined for each finger, in particular at the MCP joint point, at the PIP joint point.

[0059] 25. Joint point and / or at the DIP joint point. A threshold value can be less than 180 degrees, for example, in the range of 150 to 160 degrees. This angle criterion is particularly meaningful in a lateral view of the hand, because if the angle at a finger joint is less than the threshold value, it is easy to see that the finger is not extended. For the

[0060] The same threshold value can be assumed for 30 angles at the PIP joint and the DIP joint, since the individual finger joints cannot usually be moved independently for anatomical reasons anyway. If a finger cannot be clearly identified as not extended, the following additional criteria can be checked. 2024PF01412

[0061] 7 / 20

[0062] In some embodiments, determining the hand position includes determining a wrist distance between the wrist point and the MCP joint point of at least one finger. This wrist distance can, in a sense, be used as a reference value for normalizing other specific distances.

[0063] 5 so that the provisions explained below can be carried out regardless of the orientation of the hand in space and / or the distance of the hand to the camera.

[0064] In some embodiments, determining the hand position includes determining a distance between adjacent finger joint points and / or the fingertip point and the DIP joint point of a respective finger, wherein the

[0065] The distance is related to the specified wrist distance of the corresponding finger to obtain a corresponding distance relationship. It should first be noted that this "distance criterion" can also be used to determine whether a finger is extended or not. When viewing the hand from the front, the previously determined angles may be approximately 180 degrees, even though a finger is not extended. The distances between the feature points may then appear "shortened." As explained above, the relationship to the wrist distance allows the distances to be considered "relative" or "normalized." It should be noted that this determination can only be used for a finger if it has not already been possible to rule out, based on the angles, that a finger is extended.

[0066] It is stretched to 20.

[0067] In some embodiments, the obtained distance relation is further compared with a corresponding distance relation, which is obtained from a known standard distance between adjacent finger joint points and / or the fingertip point and the DIP joint point of a respective finger.

[0068] 25 which is related to a known standard wrist distance of the corresponding finger, where a finger is determined to be unstretched if comparing the distance relations reveals a difference (i.e., if the distance relations from the 2D image of the hand and the standard reference data differ). It may not yet be possible to determine the absolute distances.

[0069] 30. It must be determined whether a finger is extended or not, i.e., whether a distance is actually shorter due to flexion of the finger than with an extended finger, or merely due to perspective. Therefore, a comparison is made with standard distances, i.e., known average bone lengths (or finger joint lengths), which always refer to an extended hand. If a difference is found, 2024PF01412

[0070] 8 / 20 means one finger is therefore very likely not extended. It is understood that, due to measurement tolerances and for technical reasons, it may be stipulated that a difference will only be recognized as such if it exceeds a predetermined threshold.

[0071] 5 In some embodiments, determining the hand position includes at least determining whether at least one of the fingers of the hand is extended, whereby the finger is considered extended if the angle is greater than at least one threshold value and comparing the distances reveals no difference (or if the difference is less than a predetermined threshold value). In other words, if both an "angle criterion" and a "distance criterion" are met, a finger can be considered extended. It is understood that, in order to determine the hand position, it can be determined in particular which finger(s) of the hand is / are extended.

[0072] In some embodiments, determining the hand position includes determining,

[0073] 15. Whether the thumb is flexed or extended, whereby a relative position of at least one of the fingertip point, the DIP joint point, and the PIP joint point of the thumb is determined relative to a line connecting the wrist point and the MCP joint point. The thumb can be considered separately due to its mobility compared to the other fingers; in particular, the

[0074] The thumb, when bent, should be placed across the palm of the hand. This can be determined by observing the thumb's knuckle points relative to a line connecting it to the index finger. A dot product can then be calculated, which will be positive or negative depending on the thumb's position.

[0075] In some embodiments, determining the hand position includes determining

[0076] 25 of a distance between the wrist point and a fingertip point of a finger, and furthermore, a determination of whether the finger of the hand is flexed, wherein the finger is determined to be flexed if a distance of the fingertip point from the wrist point is less than a distance of the corresponding MCP joint point from the wrist point. In other words, by this determination and comparison of the

[0077] Based on the aforementioned distances, it can be determined whether the fingertip is located "on the palm," i.e., between the MCP joint and the wrist. From this, it can be concluded with a high degree of probability that the finger is bent. 2024PF01412

[0078] 9 / 20

[0079] In some embodiments, determining the hand position further includes determining a distance between the wrist point and a DIP joint point of a finger, wherein the finger is only determined to be bent if the distance of the DIP joint point from the wrist point is also less than a

[0080] 5. Distance of the corresponding PIP joint point from the wrist point. In other words, a finger is only recognized as (actually) bent when the DIP joint is closer to the wrist than the PIP joint.

[0081] While the previously mentioned criterion regarding the position of the fingertip may already be sufficient to determine whether a finger is bent, the second criterion just mentioned can increase the reliability in determining whether a finger is actually bent. For example, it can prevent (or at least reduce the risk of) unintentionally slightly bent (or not fully extended) fingers being interpreted as bent, thus misinterpreting a hand gesture and inadvertently and unintentionally controlling a vehicle function. Such a hand position can

[0082] 15. This can occur, for example, when gesturing, without a user intending a defined hand gesture.

[0083] It is understood that determining hand posture can be achieved in particular by identifying which finger(s) of the hand is / are bent. This is especially relevant when combined with information about which finger(s) of the hand is / are extended.

[0084] If there are 20, a corresponding hand position can be determined.

[0085] In some embodiments, a further distance is determined between the tip of the thumb and the tip of one of the other fingers, particularly the index finger. For example, this can determine whether the thumb tip and index finger tip approach or touch, which is important for recognition.

[0086] 25 other hand gestures can be useful.

[0087] In some embodiments, a first hand position is determined at at least a first time point and a second hand position at at least a second time point point, whereby a hand gesture is determined from the sequence of the first and second hand positions. While a single hand position can already represent a hand gesture,

[0088] 30 further hand gestures can be defined by a sequence of several hand positions. In other words, a change in a hand position can be determined as a hand gesture by considering hand positions at a plurality of successive points in time. 2024PF01412

[0089] 10 / 20 can be determined. A sufficiently high temporal resolution, determined by the sampling rate of the capture device, is advantageous for capturing smooth motion. This can be, for example, 30 fps (frames per second).

[0090] For example, a single hand gesture can be a single pointing gesture.

[0091] A hand gesture can also be defined, for example, by opening and closing a fist. By defining "static" hand gestures (individual hand positions) and "dynamic" hand gestures (sequences of hand gestures), diverse scenarios can be created, for example, to control a user interface in a vehicle with a range of vehicle functions.

[0092] A second aspect of the invention relates to a system for recognizing a hand gesture, comprising at least one processor configured to perform the method according to the first aspect. The system can, in particular, be configured to be installed in a vehicle. In this way, as described above, vehicle functions can be controlled by hand gestures.

[0093] 15 In some embodiments, the system further comprises a detection device configured to capture a two-dimensional image of a user's hand. The detection device may, in particular, be an image capture device, such as a camera, especially an infrared camera. As explained above, an infrared camera is particularly suitable because it captures image data in

[0094] 20. Essentially, it can detect objects independently of external lighting conditions, e.g., even in the dark.

[0095] A third aspect of the invention relates to a computer program with instructions which, when executed on a system according to the second aspect, cause the system to execute the method according to the first aspect.

[0096] 25 The computer program can, in particular, be stored on a non-volatile data carrier. Preferably, this is a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation

[0097] 30 The computer program can exist as a file on a data processing unit, in particular on a server, and via a data connection, for example 2024PF01412

[0098] 1 1 / 20 The program can be downloaded via the internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program may contain multiple interacting individual program modules.

[0099] The system according to the second aspect can accordingly have a program memory.

[0100] 5. The system may contain the computer program. Alternatively, the system may also be configured to access an external computer program, for example on one or more servers or other data processing units, via a communication link, in particular to exchange data with it that is used during the execution of the procedure or computer program or represents outputs of the computer program.

[0101] The features and advantages explained in relation to the first aspect of the invention also apply accordingly to the other aspects of the invention.

[0102] Further advantages, features and applications of the present invention will become apparent from the following detailed description in connection with the

[0103] 15 drawings.

[0104] This shows:

[0105] Fig. 1 schematically shows a flat hand with feature points;

[0106] Fig. 2 shows a hand in a pointing pose with feature points, using the index finger as an example;

[0107] Fig. 3 shows a decision tree to determine whether a finger is extended; and

[0108] 20 Fig. 4 to Fig. 7 each show a two-dimensional image of a hand in different hand positions, possibly from different perspectives.

[0109] The same reference numerals are used throughout the figures for the same or corresponding elements of the invention.

[0110] Fig. 1 shows a hand 1 with corresponding feature points 2, 3, 4, 5, 6 (“keypoints”).

[0111] 25 There are numerous different methods for determining these points, particularly using machine learning, which are assumed to be known. The 2024PF01412

[0112] 12 / 20

[0113] Feature points 2, 3, 4, 5, 6 include in particular a wrist point 2 and, for each finger, corresponding finger joint points 3, 4, 5 and a fingertip point 6. The finger joint points 3, 4, 5 include in particular an MCP joint point 3 at the metacarpophalangeal joint (“MCP”), a PIP joint point 4 at the proximal interphalangeal joint (“PIP”) and

[0114] 5. A DIP joint point 5 at the fingertip joint (“DIP”). It is understood that these points can be determined equally for all fingers, i.e. thumb, index finger, middle finger, ring finger and little finger.

[0115] Fig. 2 shows a user's hand 1 in a pointing pose with feature points 2, 3, 4, 5, 6 marked, using the index finger as an example. The index finger is extended, while all other fingers are bent. As will be explained below, a hand position or hand gesture can be recognized solely based on the two-dimensional relative positions of the feature points 2, 3, 4, 5, 6 of hand 1, which includes, in particular, angles and distances. This can be used as user input for gesture control in a vehicle or in other application scenarios.

[0116] 15 In the following, two different methods are described in particular for determining, firstly, whether a finger is extended and, secondly, whether a finger is bent. To recognize specific gesture features, these methods can now be applied to the different fingers. For example, an open hand can be recognized by five extended fingers. If a positive response is given for all fingers regarding

[0117] 20. If an extension is obtained, an open hand can be assumed, as shown in Fig. 1. Another example is the representation of a pointing position with an extended index finger and three bent fingers (middle, ring, and little finger), as shown in Fig. 2.

[0118] Essentially, the methodology for recognizing whether a finger is extended follows a

[0119] 25 Decision tree 10, in which certain conditions 12, 13, 14 are checked, as shown in Fig. 3. The individual conditions 12, 13, 14 are described in more detail below. The feature points 2, 3, 4, 5, 6 are assumed as input data 11 or determined from a two-dimensional image of the hand 1. Fig. 4 shows a two-dimensional image of a flat hand with extended fingers, where the

[0120] 30 feature points and corresponding connecting lines (i.e., distances between the feature points) are marked. The image could, in particular, be an IR image of the hand. 2024PF01412

[0121] 13 / 20

[0122] A first (necessary) condition 12 for an extended finger is that the angles between adjacent bones or finger joints are greater than a certain threshold. Therefore, it is checked whether the angles a, β, y at the finger joint points 3, 4, 5 are greater than certain thresholds, i.e., whether a > T 1;

[0123] 5 ß > T2, Y > with T i e [°, 180], T2 e [°< 180]. If one of the three angles is smaller, extension can already be ruled out (Result 16). In both Fig. 5 and Fig. 6, the index finger is bent, but the perspective is different. If one views the user's hand 1 from the side, as shown in Fig. 5, this criterion is sufficient to decide that an extended finger is present if all three

[0124] The angles are large enough. It becomes more difficult when the hand is positioned directly in front of the camera. Due to the 2D projection, the angles can be 180 degrees, even if the fingers are curled (see Fig. 6). Therefore, further conditions are checked to demonstrate the presence of an extended finger (result 15).

[0125] If the hand is oriented frontally to the camera, a relative distance criterion 13 is used to determine whether a finger is flexed or extended. In Fig. 6, it can be seen that the distances between the finger joint points 3, 4, 5 (including the fingertip point 6) of the index finger are shortened because it is flexed, while the angles are all 180 degrees. For this purpose, it is first assumed that average bone lengths of humans are known in order to make a comparison.

[0126] 20. In two dimensions, the Euclidean distance between two feature points in the case of an outstretched finger depends firstly on the actual distance between the points and secondly on both the distance of the hand to the camera and the orientation of the hand. To be as independent as possible from the latter two conditions, the Euclidean distances of the individual finger segments are expressed in relation to the distance.

[0127] 25 between wrist point 2 and MCP joint point 3. This ratio is then compared with the same ratio of the known average bone lengths. If the two ratios ("distance ratios") differ, the presence of an extended finger can again be ruled out (result 16). If the two ratios are close to each other,

[0128] If their difference is less than a threshold value (30 dh), a final special case must be examined.

[0129] A further condition, 14, is intended to examine questions specifically relating to the thumb in camera directions where the angles separately satisfy the angle condition, even if the thumb is not necessarily fully extended. 2024PF01412

[0130] 14 / 20

[0131] To avoid this problem, a line is first defined at the MCP joint, given by the normal vector from MCP joint point 3 to wrist point 2. In the case of an extended thumb, the PIP joint point 4, the DIP joint point 5, and the fingertip point 6 must be on the negative side of the line at the MCP joint.

[0132] The 5th joint must lie on the positive side of the line, meaning the dot product of the vector from the MCP joint to the corresponding feature point 4, 5, 6 with the normal vector must be less than zero. If any of the points 4, 5, 6 lies on the positive side of the line, the thumb is not extended, as shown in Fig. 7.

[0133] Another important characteristic of gestures is the property of a finger being fully bent, as is the case, for example, in a fist. In most cases, it is sufficient to check whether the fingertip lies between the MCP joint and the wrist. This can be checked, for example, using Euclidean distances; that is, the distance from fingertip point 6 to wrist point 2 must be less than the distance from MCP joint point 3 to wrist point 2. Further checks can be carried out analogously.

[0134] 15. Check whether the DIP joint lies between the PIP joint and the wrist. If both conditions are met, the finger is fully bent.

[0135] The two methods for determining extended or bent fingers allow for the identification of diverse hand positions that can be part of a multitude of gestures. Furthermore, additional characteristics can be determined to…

[0136] to be able to recognize 20 additional hand gestures. For example, an "okay" pose (thumb touching index finger, other fingers are extended) can also be recognized by applying the extended finger test to the middle, ring, and little fingers and adding a test of the Euclidean distance between the tip of the thumb and the tip of the index finger.

[0137] 25 The hand gesture recognition method can be used, in particular, in a vehicle to enable contactless interaction with a user interface. In this case, an IR sensor (an IR camera) can be used to capture data from the driver or another passenger, and any machine learning method can be used to determine the feature points (especially their 2D positions).

[0138] 30 of the hand can be determined. From this, various hand gestures can then be recognized as explained. 2024PF01412

[0139] 15 / 20

[0140] For example, if the thumb is detected as extended and the other fingers as bent, a thumb gesture can be recognized. Since the camera's position in the vehicle is known, the direction of the thumb can also be determined, and a distinction can be made between left and right. This gesture can be used, for example, to select a radio station.

[0141] 5. Shift forward or backward.

[0142] A hand position can also be a pointing pose, in which the index finger of the detected hand is extended. It should be noted that "extended" does not necessarily mean a fully extended finger, but rather that the extension exceeds a predefined value; the finger may still be slightly bent, but is essentially fully extended. A pointing pose is recognized, in particular, when at least the middle, ring, and little fingers of the hand are bent simultaneously. The thumb can be in any position. A pointing pose can indicate that the user wishes to enter information on the user interface.

[0143] Furthermore, certain sequences can also be achieved using the described method.

[0144] Fifteen hand gestures can be recognized. A double-five gesture, for example, consists of opening the hand twice. Since the fist can be recognized as a hand gesture (pose) (all fingers bent), as can the open hand (all fingers extended), a sequence of images can now be examined to recognize the desired hand gesture for interaction with the user interface.

[0145] 20 While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the exemplary embodiments described are only non-limiting examples, and it is not intended to thereby limit the scope, applicability, or configuration of the devices described herein and

[0146] 25. To limit the method. Rather, the preceding description will provide the person skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes can be made to the function and arrangement of the elements described in an exemplary embodiment without deviating from the one described in the

[0147] The subject matter and its legal equivalents specified in each of the 30 attached claims are not affected. 2024PF01412

[0148] 16 / 20

[0149] REFERENCE MARK LIST

[0150] 1 hand

[0151] 2 Wrist point

[0152] 3rd Finger Joint Point (MCP)

[0153] 5 4 Finger joint point (PIP)

[0154] 5 Finger joint point (DIP)

[0155] 6 Fingertip point

[0156] 10 Methodology for determining whether a finger is extended

[0157] 11-16 steps / processes / data / results within the framework of methodology 10

Claims

2024PF01412 17 / 20 REQUIREMENTS 1. Method for recognizing a hand gesture, wherein the method comprises: - Capturing image data, wherein the image data represents a two-dimensional image of a user's hand (1 ); 5 - Determining feature points (2, 3, 4, 5, 6) in the two-dimensional image of the hand (1), wherein the feature points (2, 3, 4, 5, 6) comprise a wrist point (2) and, for each finger of the hand (1), finger joint points (3, 4, 5) and a fingertip point (6), wherein the finger joint points (3, 4, 5) for each finger comprise an MCP joint point (3) and a PIP joint point. 10 (4) and a DIP pivot point (5); - Determining a hand position based on the relative positional relationships of the feature points (2, 3, 4, 5, 6) to each other in the two-dimensional image of the hand 12).

2. The method of claim 1, wherein determining the hand position comprises at least one of the following: - Determining an angle between two finger joints at at least one finger joint point (3, 4, 5), - Determining the distance between adjacent finger joint points (3, 4, 20 5) and / or the fingertip point (6) and the DIP joint point (5) of a respective finger, and - Determining a distance between the wrist point (2) and at least one fingertip point (6) and / or finger joint point (3, 4, 5).

3. Method according to claim 1 or 2, wherein determining the hand position is a 25 Determining an angle between two finger joints at at least one finger joint point (3, 4, 5) and comparing the determined angle with a threshold value, wherein a finger is determined to be not extended if the angle is smaller than the at least one threshold value.

4. Method according to any one of the preceding claims, wherein the determination of the 30 Hand position includes determining a wrist distance between the wrist point (2) and the MCP joint point (3) of at least one finger. 2024PF01412 18 / 20 5. The method of claim 4, wherein determining the hand position comprises determining a distance between adjacent finger joint points (3, 4, 5) and / or the fingertip point (6) and the DIP joint point (5) of a respective finger, wherein the distance to the determined wrist distance of the The 5 corresponding fingers are put in relation to each other in order to obtain a corresponding distance relation.

6. The method of claim 5, further comprising comparing the obtained distance relation with a corresponding distance relation obtained from a known standard distance between adjacent finger joint points (3, 4, 5) and / or the fingertip point (6) and the DIP joint point (5) of a respective finger, which is related to a known standard wrist distance of the corresponding finger, wherein a finger is determined to be not extended if the comparison of the distance relations reveals a difference. 15 7. Method according to claims 3 and 6, wherein determining the hand position includes at least determining whether at least one of the fingers of the hand (1) is extended, wherein the finger is determined to be extended if the angle is greater than the at least one threshold value and comparing the distance relations yields no difference. 20 8. Method according to one of the preceding claims, wherein determining the hand position comprises determining whether the thumb is flexed or extended, wherein for this purpose a relative position of at least one of the fingertip point (6), the DIP joint point (5) and the PIP joint point (4) of the thumb relative to a connecting line between the wrist point (2) and the MCP- 25 Joint point (3) is determined.

9. Method according to any of the preceding claims, wherein determining the hand position comprises determining a distance between the wrist point (2) and a fingertip point (6) of a finger and further comprising determining whether the finger of the hand (1) is bent, wherein the finger is considered to be bent 30 is determined if the distance of the fingertip point (6) from the wrist point (2) is less than the distance of the corresponding MCP joint point (3) from the wrist point (2). 2024PF01412 19 / 20 10. Method according to claim 9, wherein determining the hand position further comprises determining a distance between the wrist point (2) and a DIP joint point (5) of a finger, wherein the finger is determined to be bent only when a distance of the DIP joint point (5) from the 5 wrist point (2) is smaller than a distance of the corresponding PIP joint point (4) from the wrist point (2).

11. Method according to one of the preceding claims, further comprising determining a distance between the fingertip point (6) of the thumb and the fingertip point (6) of one of the other fingers, in particular the index finger.

12. A method according to any one of the preceding claims, further comprising: - Determining an initial hand position at at least one initial point in time; - Determining a second hand position at at least one second time; and 15 - Determining a hand gesture from the sequence of the first and second hand positions.

13. System for recognizing a hand gesture, comprising at least one processor configured to perform the method according to any of the preceding claims.

14. System according to claim 12, further comprising a detection device, 20 which is set up to capture a two-dimensional image of a user's hand (1 ).

15. Computer program with instructions which, when executed on a system according to claim 13 or 14, cause the system to execute the method according to any one of claims 1 to 12.

Images