Gesture recognition device and gesture recognition method

Abstract

An electronic device has circuitry for gesture recognition. The circuitry is configured to obtain gesture sensor data of a hand, recognize a finger pointing pattern based on the obtained gesture sensor data, realize the recognized finger pointing pattern as an object pointing in a virtual space, recognize a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data, and realize the recognized transition as a confirmation of the object pointing in the virtual space.

Description

[0001] GESTURE RECOGNITION DEVICE AND GESTURE RECOGNITION METHOD

[0002] The present disclosure generally pertains to the field of gesture recognition, in particular to devices and methods for gesture recognition for GUI use as well as virtual or augmented reality systems.

[0003] TECHNICAL BACKGROUND

[0004] To provide an effective graphical user interface (GUI) interaction and a realistic and effective virtual or augmented reality user experience, including, for example, painting in virtual reality or touching different objects, such as menus, in virtual or augmented reality, user gestures have to be represented accurately within the virtual space. In particular, pointing gestures need to be accurately and precisely realized in virtual space for a comfortable user experience.

[0005] Although there exist techniques for realizing pointing gestures of users in virtual space, it is generally desirable to improve existing techniques.

[0006] SUMMARY

[0007] According to a first aspect, the present disclosure provides an electronic device for gesture recognition configured to: obtain gesture sensor data of a hand; recognize a finger pointing pattern based on the obtained gesture sensor data; realize the recognized finger pointing pattern as an object pointing in a virtual space; recognize a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data; and realize the recognized transition as a confirmation of the object pointing in the virtual space.

[0008] According to a second aspect, the present disclosure provides a method for gesture recognition comprising: obtaining gesture sensor data of a hand; recognizing a finger pointing pattern based on the obtained gesture sensor data; realizing the recognized finger pointing pattern as an object pointing in a virtual space; recognizing a transition between the recognized finger pointing pattern and the recognized finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data; and realizing the recognized transition as a confirmation of the object pointing in the virtual space.

[0009] Further aspects of the present disclosure are set forth in the dependent claims, the drawings and the following description.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0012] Fig. la schematically illustrates an embodiment of gesture recognition of a finger pointing pattern based on user pointing at a virtual long-distance menu;

[0013] Fig. lb schematically illustrates an embodiment of gesture recognition of a finger pointing pattern based on user pointing at a virtual long-distance menu;

[0014] Fig. 1c schematically illustrates an embodiment of gesture recognition of a finger pointing pattern with thumb-extension pattern as a realized pointing confirmation of Fig. lb;

[0015] Fig. 2a schematically illustrates an embodiment of a virtual space and a recognized and realized gesture pattern in virtual space;

[0016] Fig. 2b schematically illustrates an embodiment of gesture recognition of a starting gesture for activating a virtual sliding bar;

[0017] Fig. 2c schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual sliding bar operation;

[0018] Fig. 2d schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual sliding bar operation;

[0019] Fig. 2e schematically illustrates an embodiment of gesture recognition for a confirmation of a virtual sliding bar selection.

[0020] Fig. 2f schematically illustrates an embodiment of gesture recognition for a confirmation of a virtual sliding bar selection. Fig. 2g schematically illustrates an embodiment of gesture recognition of a closing gesture of a virtual sliding bar after confirmation of a virtual sliding bar selection;

[0021] Fig. 2h schematically illustrates an embodiment of gesture recognition of a reset gesture for a virtual sliding bar application;

[0022] Fig. 3a schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual painting application;

[0023] Fig. 3b illustrates an embodiment of gesture recognition of a selection confirmation for a virtual painting application;

[0024] Fig. 4a schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for user pointing at a close distance menu;

[0025] Fig. 4b schematically illustrates an embodiment of gesture recognition of a finger pointing pattern with thumb -extension pattern as a realized pointing confirmation of Fig. 4a;

[0026] Fig. 5a schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space;

[0027] Fig. 5b schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition;

[0028] Fig. 5c schematically illustrates an embodiment of a recognized and realized finger pointing pattern in virtual space.

[0029] Fig. 5d schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for finger pointing pattern recognition;

[0030] Fig. 5e schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space;

[0031] Fig. 5f schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition;

[0032] Fig. 5g schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space;

[0033] Fig. 5h schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition;

[0034] Fig. 5i schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space; Fig. 5j schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition;

[0035] Fig. 5k schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb extension pattern in virtual space.

[0036] Fig. 51 schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for finger pointing pattern with thumb extension pattern recognition;

[0037] Fig. 6 shows a block diagram depicting an embodiment of recognition and realization of a finger pointing pattern and recognition and realization of a transition from the finger pointing pattern to a finger pointing pattern with thumb extension pattern;

[0038] Fig. 7a schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb extension pattern at wide angle;

[0039] Fig. 7b schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb extension pattern at medium angle;

[0040] Fig. 7c schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb extension pattern at narrow angle;

[0041] Fig. 8a schematically illustrates an embodiment of a user performing a finger pointing gesture in a car;

[0042] Fig. 8b schematically illustrates an embodiment of a side-view of a user performing a finger pointing gesture in a car;

[0043] Fig. 9 illustrates an embodiment of a method for gesture recognition; and

[0044] Fig. 10 shows a block diagram depicting an embodiment of an electronic device that implements gesture recognition.

[0045] DETAILED DESCRIPTION OF EMBODIMENTS

[0046] Before a detailed description of the embodiments under reference of Fig. la is given, general explanations are made.

[0047] It has been recognized that extended reality applications, such as virtual reality (VR) or augmented reality (AR) applications, require stable and reliable user pointing mechanisms for enhancing the user interaction within the virtual space. For example, user interaction with virtual menus would profit from accurate and precise representation of pointing. Additionally, it has been recognized that during user interaction with a menu the selection of a menu item depends on accurate representation of the user pointing within the virtual reality, that is, an accurate pointing gesture realization within the virtual space.

[0048] Hence, some embodiments pertain to an electronic device comprising circuitry for gesture recognition configured to obtain gesture sensor data of a hand, recognize a finger pointing pattern based on the obtained gesture sensor data, realize the recognized finger pointing pattern as an object pointing in a virtual space, recognize a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data, and realize the recognized transition as a confirmation of the object pointing in the virtual space.

[0049] The electronic device may for example be a PC, or a digital signage device or another outdoor interface device, for example any outdoor interface device, wherein the user may give feedback, or it may be a mobile device such as a smartphone, tablet, laptop, smart glasses, a head-mounted display (HMD), earphones, or the like. The HMD may be any type of augmented or virtual reality HMD.

[0050] The circuitry may include a processor, a memory (RAM, ROM or the like), a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g., liquid crystal, (organic) light emitting diode, etc.), a (wireless) interface, etc., as it is generally known for electronic devices (smartphones, tablet computers etc.). Moreover, it may include sensors for sensing still image or video image data (image sensor, camera sensor, video sensor, etc.), etc. The sensor of the electronic device may be a camera as described in the following.

[0051] The gesture sensor data may be based on sensor information captured by one or more cameras, which may be the sensor of the electronic device, for example, by a RGB camera, an infrared camera, a depth sensing camera, such as a time of flight (ToF) camera, or an event-based sensor. The time of flight camera may produce depth and / or confidence and / or intensity images. That is, the time of flight camera may produce depth and / or confidence and / or intensity data. The time of flight camera may be an indirect time of flight (iToF) camera or a direct time of flight camera (dTof). Therefore, the gesture sensor data may be based on any one or more of a grayscale image, a color image, such as an RGB image, a depth image, a confidence image, an intensity image, an infrared image. In other words, the gesture sensor data may be based on a 2D image and / or a 3D image. That is, the gesture sensor data may be based on any one or more of grayscale data, RGB data, depth data, confidence data, intensity data and infrared data. The resolution of the images on which the gesture sensor data may be based may differ based on the use case. For example, the depth image may be of low resolution and the confidence image, intensity image and / or RGB image may be of high resolution.

[0052] The above-described camera(s) may be mounted on the electronic device, e.g., mounted on the HMD, such as camera 106 of electronic device 100 of Fig. 10 or they may be external cameras as described, for example, in regard to Fig. 10. For example, in case the electronic device digital is a digital signage or other outdoor interface device the camera may be mounted on the digital signage or outdoor interface itself or it may be mounted on any post. The camera may be located any position around the user, in front, to the side, behind the user. The user may stand in a predefined position, for example, the user may be instructed explicitly and / or implicitly to move to a predefined position, within the scope of the camera, for example, the camera may be located behind the user’s head. Also, the user may be instructed explicitly and / or implicitly to move to a predefined position which gives the best camera view or which gives the best view of the user interface (UI).

[0053] In some embodiments the electronic device may include a cubicle, e.g., similar to a passport photo booth, wherein the camera(s) may be located within the cubicle. Also, the electronic device may include a vehicle, for example, a car, wherein the camera(s) may be located within the vehicle, and the user may be sitting in the vehicle. The camera may provide a first -person or third person view of a user’s gestures.

[0054] Some images, on which the gesture sensor data may be based, may require more computational resources, such as, the depth information of a depth image, which are therefore called, cost channels. By contrast, some images, on which the gesture sensor data may be based, may require less computational resources, such as an intensity image, a confidence image, an RGB image, a grayscale image and an infrared images, which are therefore called, cheap channels. Images based on cheap channels may therefore have a higher resolution than images based on cost channels. That is, gesture sensor data may be based on one or more images of cheap channels and / or cost channels, wherein the cheap channels may have higher resolution than the cost channels.

[0055] Additionally, to the gesture sensor data of a hand, which may be the user’s hand, also gesture sensor data of a second hand, for example, the second hand of a user, may be used as input. The user may be a human user, therefore the hand may be a human hand. Alternatively, the hand may be an emulated hand, such as a robotic hand. Also, the gesture sensor data may include data of the wrist of the user’s hand. Additionally, also gesture sensor data of a user’s shoulder and arm may be used. Furthermore, eye-tracking data for determining the direction of a pointing gesture may also be used as input.

[0056] That is, recognizing a finger pointing pattern may include determining the pointing direction or pointing line. Determining the pointing direction may be based on the target. For example, if the pointing target is planar, such as a virtual menu, the pointing direction may be based on the plan normal with respect to the target menu. The pointing direction may also be based on the finger direction, for example, the index finger direction, of the user. Moreover, the pointing direction may be based on one or more anchor points, preferably on two anchor points. That is, the pointing direction may be determined from the direction of a first anchor point to a second anchor point. An anchor point may be the palm center or the hand tip, for example, any fingertip, preferably, the index fingertip, or any other point on the body. That is, in case the gesture sensor data may include one or more of data of the wrist, shoulder, and / or in case eye direction data of the user is included as input, the pointing direction may be determined based on the direction of the palm-center and / or wrist and / or shoulder and / or eye as anchor points (for example, as the first or second anchor point) to the hand tip as anchor point (for example, second or first anchor point), the reverse direction may also be possible. Alternatively, the pointing direction may be determined based on any anchor point (anchor point 1) to any hand-feature-point (anchor point 2), preferably, to the hand tip or index fingertip.

[0057] Alternatively, the anchor point may be any pre-defined point in a current virtual scene, which may be configurated by a user before the scene start, or which may be dynamically reset to a current pointer-tip, e.g., the direction of the finger pointing pattern, via a recognized gesture event performed by the user, for example, via the thumb-extension performed by the user and recognized as a thumb-extension pattern. This may be used in a free movement virtual space application, such as, a medical application or education application. That is, if a medical professional as a user may want to study a medical image, such as an x-ray image, the user may select one anchor point in the current image and then freely move the fingertip to form a line direction. In other words, in medical imaging, when the image, for example an image of a tooth root scan or a polyp, is shown in the user interface, the user can select a portion of the image by pointing, select a part of the image by extending the thumb, then move the pointing finger around to rotate the portion of the image whilst the thumb is still extended. If the image segment is rendered in 3D then the image portion can be viewed from an alternative angle.

[0058] The finger pointing pattern may be based on a finger pointing gesture of a user and may, for example, be an index-finger pointing pattern, that is based on a user pointing with an index finger. Alternatively, any other finger beside the thumb may be used for the finger pointing gesture. The finger pointing pattern may also include an extended thumb. The transition may include a transition from the recognized finger pointing pattern to the recognized finger pointing pattern together with a thumb-extension pattern, or it may include a transition from the recognized finger pointing pattern together with a thumb-extension pattern to the recognized finger pointing pattern (reverse transition). That is, the transition may always includes a change in gesture pattern.

[0059] As a finger pointing gesture, in particular, an index-finger pointing gesture is a natural pose and extending a thumb, which may be the basis for the transition, i.e., the confirmation, is equally a natural pose, hand jitter which may negatively impact gesture recognition and the selection of virtual objects in particular may be reduced. This is of particular advantage for accurate virtual menu interaction or precise virtual sliding bar interaction. The natural thumb-extension which may trigger a selection or change confirmation, may therefore, keep a previous selection stable. This advantage is in contrast to alternative gestures, which have been proposed, such as pointing and then touching the index fingertip against the thumb, the “OK-gesture”, as a confirmation action. The described “OK-gesture” has the disadvantage that the thumb moves while performing the gesture and a stable pointing position is compromised, that is, the pointing finger also moves, and therefore a stable pointing pose cannot be maintained, which introduces jitter.

[0060] Based on the obtained gesture sensor data the electronic device may determine that a finger pointing gesture has been executed by a user. Following the finger pointing gesture the user may extend their thumb. The transition from the finger pointing only gesture, that is, pointing without the thumb, which may be recognized as a finger-pointing pattern, to a finger pointing gesture with thumb, which may be recognized as a finger pointing with thumb pattern may indicate to the electronic device that a selection has been performed by the user. That is, it may indicate the confirmation of a selection. Thus, the last pointing target of the user, which may be based on the recognized finger pointing gesture before the thumb extension of the user, for example, determined based on the pointing direction, may be selected. In other words, the transition from finger pointing pattern alone to finger pointing pattern and thumb extension may be recognized as a selection. Therefore, during the pointing the user may freely select an object of interest, that is the pointing direction may change, after the selection the user may show the thumb to confirm the selection without changing the pointing direction of the finger, that is without changing the finger pointing position. The thumb may be extended horizontally with respect to the virtual objects and the user. Thus, the thumb-extension may be a horizontal thumb extension. Alternatively, the thumb may also be extended vertically or in any other direction. The thumb may extend at a narrow angle, for example an angle close to the index finger of less than 45 degrees, in particular the thumb may be touching the index finger while extending, the thumb may extend at a medium angle, of 45 to less than 80 degrees to the index finger, particularly at 45 to 60 degrees, or the thumb may extend at a wide angle from 80 to over 90 degree. As the thumb extension does not affect the finger pointing gesture, that is, as the thumb extension pattern adds to the finger pointing pattern, but does not change the finger pointing pattern in terms of pointing direction, the selection, is stable and can be reliably used for selection of a pointing target within the virtual space. Furthermore, the usual jitter, which is a problem for recognizing pointing gestures within a virtual space is reduced. Thus, the thumb extension in virtual or augmented reality may be considered in the same vein as “clicking” in terms of a user using a computer mouse. The thumb extension together with pointing is also a natural hand movement comfortable for a user, which, therefore, reduces the stress of the user using unfamiliar or unnatural hand gestures, for selection or confirmation of selection, which may affect the jitter of hand movement and therefore, the gesture recognition precision.

[0061] The virtual space may refer to extended reality, that is, virtual or augmented reality. In other words, the pointing targets of the finger pointing pattern, which may be based on a finger pointing gesture of the user, may be objects in the virtual space, that is, may be virtual objects. Thus, the user may point to a virtual object in virtual space and the electronic device may recognize a gesture of the user as either a pointing gesture within the virtual space, that is, as an object pointing, which may be a pointing gesture towards a pointing target, in case of the finger pointing pattern, or as a selection within the virtual space, for example, in case of the finger pointing with thumb extension pattern.

[0062] Alternatively, virtual space may include a graphical user interface (GUI).

[0063] The recognition of the finger pointing pattern may include a detection of a finger pointing pattern. Furthermore, the recognition of the finger pointing pattern may refer to a detection of the finger pointing pattern or it may refer to a detection of the finger pointing pattern and a recognition as a finger pointing pattern among other hand gesture patterns.

[0064] The recognition of the transition may include a detection of the transition from pattern A (finger pointing pattern) to pattern A+B (finger pointing pattern + thumb-extension pattern) or a detection of the reverse transition, that is the transition from pattern A+B to pattern A. Alternatively or additionally, the recognition of the transition may refer to the recognition as a transition from finger pointing pattern (A) to the finger pointing pattern together with a thumbextension pattern (A+B), or as a reverse transition (A+B -> A), that is, from finger pointing pattern together with a thumb-extension pattern (A+B) to finger pointing pattern (A), as a transition among other gesture events.

[0065] The realization in virtual space may, for example, in case of the recognized finger pointing pattern, refer to the recognition of a pattern as a pointing gesture, i.e., recognition of a pattern as an object pointing or as belonging to an object pointing in virtual space. That is, the realization in virtual space may refer to the recognition of a gesture event occurring in the context of the virtual space. In other words, realization may refer to a pattern or gesture event being recognized as a pattern or gesture event affecting the virtual space in some way. Therefore, for example, in case of the realization of the recognized transition, realization may refer to a recognition as a confirmation of the object pointing, for example, a confirmation as a selection target. The realization may include a visualization in virtual space, for example, a virtual hand avatar may visualize the recognized gestures in virtual space or virtual objects or changes in virtual objects, for example affected by the gestures, may be visualized. Visualization may for example, include a plotting, such as a plotting of coordinates, e.g., 3D coordinates, it may include modelling or rendering. However, the realization may also not include any visualization and may only include a recognition.

[0066] As an example, a user may want to select one or a number of objects (virtual objects), for example, of a GUI. The virtual objects may be words, within the GUI. Thus, the user may point to an object, e.g., a word, on the GUI, wherein the GUI may be displayed on a display of the electronic device. The pointing gesture may be recognized as a finger-pointing pattern by the electronic device. Furthermore, the finger pointing pattern may be realized on the GUI, for example, by visualizing an icon representing the position of a finger pointing target on the GUI.

[0067] Following the pointing gesture, the user may extend their thumb and move their hand pose while the thumb is still extended to define a bounding box, which may be rectangular or oval or any other shape. The thumb extension and dragging motion may be recognized as a finger pointing pattern and thumb extension by the electronic device, which may be realized as a selection confirmation of the object or word, by starting a bounding box within the GUI. Additionally, the movement of the user while pointing and extending their thumb may be recognized and realized as an extension of the started bounding box within the GUI. Afterwards the user may tuck their thumb back in to select the contents of the bounding box, which may be recognized as a final selection of the contents of the bounding box and which may be realized as a final selection, as a stop of the bounding box extension, or it may be realized as a colored marking of the final selection within the bounding box. Thus, in general, a finger pointing pattern and finger pointing and thumb-extension pattern may be combined with movement or even other gesture events.

[0068] In the following, various other combinations of recognized and realized gesture patterns are explained for the example of a GUI, although other types of virtual space may be possible. The following examples are explained in terms of the user gesture and realization within the GUI. All gestures may be recognized as patterns and realized in the GUI.

[0069] For example, a user may extend their thumb to select multiple object one by one. Furthermore, if a user repeats a thumb-extension on already selected object it may cancel the current selection.

[0070] Alternatively, in a second example, the user may extend their thumb (which may correspond to a mouse click “down” of a computer mouse) to start a region of interest (bounding box) and continue with the thumb extension and movement pointer (which corresponds to dragging a computer mouse while continuing to click / press “down”) to end the region of interest (bounding box) and then finish with the thumb extension by tucking their thumb back in (which corresponds to a mouse click “up”) to confirm the selection.

[0071] Additionally, to the second example explained above, a user may extend their thumb during movement to draw a closed curve as a freestyle curve region (bounding box) selection.

[0072] Alternatively, in a third example, a user may extend their thumb to select one or more control points and connect those points to form a freedom selection (bounding box).

[0073] Alternatively, other forms of bounding box selection may be performed by a combination of pointing finger pattern and thumb extension pattern recognition and realization.

[0074] In summary, to simulate a traditional mouse click “down” and “up” while visualizing the “mouse” pointer within a GUI and closing the “mouse” pointer visualization gestures that are recognized as gesture events, for example using a combination of finger pointing and thumbextension, recognized as finger pointing pattern with thumb extension pattern, and also movement, e.g., movement of the finger pointing with thumb-extension gesture, may be used. Furthermore, tucking the thumb back in, i.e., a reverse of the thumb-extension pattern, may be recognized and realized as, for example, a final selection confirmation, or also as a cancel gesture enabling a user to start again. Other gesture patterns, such as a waving gesture may also be used in combination to signal a cancellation.

[0075] In particular, to simulate a mouse click “down” and “drag” may be performed by showing (extending) the thumb and movement of the hand, i.e., moving the pointing hand with thumb extension. Furthermore, the transition from finger pointing pattern to finger pointing pattern and thumb-extension pattern to finger pointing pattern and thumb-closing pattern (tucking thumb back in) may indicate either a final selection or a cancel gesture. That is, the transition from thumb-extension to thumb-closing to thumb-extension may either indicate a final selection confirmation or a reverse selection or a cancel and repeat gesture.

[0076] In some embodiments the circuitry may be further configured to recognize the finger pointing pattern and / or the transition based on a hand analysis based on the gesture sensor data. Furthermore, the circuitry may be further configured to perform the hand analysis based on a detection of a hand discrete pose based on the gesture sensor data.

[0077] That is, the hand analysis may include a detection of a hand discrete pose. In other words, based on the gesture sensor data the circuitry may compute a hand discrete pose. A hand discrete pose may refer to the stable and / or discrete position a hand is in. That is, a finger pointing gesture, for example a single finger pointing gesture, such as an index finger pointing gesture, may be a discrete pose. Alternatively, multiple fingers extended may be another discrete pose. Also, for example, the index finger pointing and thumb extension gesture may be a discrete pose.

[0078] Therefore, the discrete hand pose may refer to a discrete hand pose a user’s hand is in when interacting with the virtual space.

[0079] The circuitry may be further configured to detect the hand discrete pose based on a small neural network. The small neural network may be a convolutional neural network. A small neural network is a machine learning model. In some embodiments other machine learning models may be used to detect the hand discrete pose. A machine learning model may be, for example, based on a support vector machine (SVM) or random forrest, but may also be based on an artificial neural network, e.g., a convolutional neural network (CNN) or multimodal Al model, or the like (see as a hand pose estimation algorithm, for example, Al Koutayni, M. R., Rybalkin, V., Malik, J., Elhayek, A., Weis, C., Reis, G., ... & Stricker, D. (2020). “Real-time energy efficient hand pose estimation: A case study”. Sensors, 20(10), 2828.). The small neural network or the other machine learning model may be trained in advance. For example, data on different hand discrete poses may be used to train the small neural network or the other machine learning model.

[0080] One of the advantages of using a small neural network is that it is very computationally efficient, which reduces latency of detecting the hand discrete pose, and therefore reduces latency of the hand analysis, which, in turn, positively, affects the latency of the gesture recognition as a whole.

[0081] In some embodiments, the circuitry is further configured to perform the hand analysis based on a detection of a hand key point based on the gesture sensor data. That is, the hand analysis may include a detection of a hand key point. In other words, based on the gesture sensor data the circuitry may compute one or more hand key points. A hand key point may be a fingertip, a hand tip and / or a hand center point. The hand tip may refer to the fingertip of the tallest extended finger. The hand center point may refer to the palm center.

[0082] The hand key point may be detected based on a computer vision technique, which may be a pure traditional computer vision technique, which may include (but may not be limited to) any one or more of the group of smoothing, contouring (see for example, Papari, G., & Petkov, N. (2011). “Edge and line oriented contour detection: State of the art.” Image and Vision Computing, 29(2- 3), 79-103.), feature description, meanshift, triangulation and tracking. Real time, efficient gesture recognition, for example using RGB as described herein may for example be based on techniques described in: Zhang, F., Bazarevsky,V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C., Grundmann, M. “MediaPipe Hands: On-device Real-time Hand Tracking”, CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA, 2020.

[0083] One of the advantages of using a pure traditional computer vision technique as described above is that it is very computationally efficient, which reduces latency of detecting the hand key point, and therefore reduces latency of the hand analysis, which, in turn, positively, affects the latency of the gesture recognition as a whole.

[0084] In some embodiments a machine learning model may be used to detect the hand key point. A machine learning model may be, for example, based on a support vector machine (SVM) or based on random forrest, but may also be based on an artificial neural network, e.g., a convolutional neural network (CNN) or multimodal Al model, or the like.

[0085] In some embodiments the circuitry may be further configured to perform the hand analysis based on a detection of a hand mesh based on the gesture sensor data. That is, the hand analysis may include a detection of a hand mesh. In other words, based on the gesture sensor data the circuitry may compute a hand mesh. The detection of a hand mesh may include the detection of a hand contour. Alternatively, the hand analysis may be based on a hand contour without the hand mesh.

[0086] The hand mesh may be detected based on a computer vision technique, which may be a pure traditional computer vision technique, which may include (but may not be limited to) any one or more of the group of smoothing, contouring (see for example, Papari, G., & Petkov, N. (2011). “Edge and line oriented contour detection: State of the art.” Image and Vision Computing, 29(2- 3), 79-103.), feature description, meanshift, triangulation and tracking.

[0087] One of the advantages of using a pure traditional computer vision technique as described above is that it is very computationally efficient, which reduces latency of detecting the hand mesh and / or hand contour, and therefore reduces latency of the hand analysis, which, in turn, positively, affects the latency of the gesture recognition as a whole.

[0088] In some embodiments a machine learning model may be used to detect the hand mesh. A machine learning model may be, for example, based on a support vector machine (SVM) or random forrest, but may also be based on an artificial neural network, e.g., a convolutional neural network (CNN) or multimodal Al model, or the like.

[0089] The circuitry may be further configured to perform the computer vision technique based on temporal filtering. That is, the circuitry may be configured to perform the computer vision technique for detecting the hand key point and / or the computer vision technique for performing the hand mesh based on temporal filtering.

[0090] The temporal filtering may be based on a pre-determined threshold parameter. For example, a temporal filtering may include a median filter, gaussian filtering, exponential filtering or weighted averaging. The threshold parameter may be pre-determined to control smoothing and / or latency.

[0091] Temporal filtering may also reduce jitter of the detected hand key point and / or the detected hand mesh.

[0092] To further reduce the jitter of the detected hand key point and / or the detected hand mesh, the circuitry may be further configured to perform the computer vision technique based on a spatial filtering. That is, the circuitry may be further configured to perform the computer vision technique for detecting the hand key point and / or the computer vision technique for performing the hand mesh based on spatial filtering, which would in turn reduce any hand jitter. Hand jitter, that is jitter of a hand key point and / or the hand mesh, may occur due to small movements occurring when a user performs hand gestures, as even in a comparatively still pose of natural movement, the user is never frozen, i.e., complete reduction of movement cannot occur during natural user interaction.

[0093] Furthermore, reducing the jitter also improves smoothness of tracking, for example, smoothness of tracking of a point in virtual space, for example, on a GUI, is improved. That is, a point on the graphical user interface may be a realized key point of a gesture pattern. In other words, the detected key points, which may be detected via tracking, may be realized in the virtual space, which may include a GUI, as a visualized key point, e.g., a rendered point. Thus, the tracked key point may be visualized as movement of the rendered point and improved smoothness of the tracking may be realized as improved smoothness of the visualized point movement rendered in virtual space, e.g. for example presented on the GUI.

[0094] That is, trajectory noise including sensor noise may be reduced, furthermore, any negative effect of hand shaking without user intention, slowing of hand movement or slow hand movement, hand jitter during pose changes, for example, during the transition from a finger pointing gesture to a finger pointing and thumb extension gesture, or any other unwanted noise in the data may be reduced or smoothed via temporal and / or spatial filtering. At the same time, user intentional movements are not smoothed by the temporal and spatial filtering. Thus, to balance the noise reduction, on the one hand, while keeping all relevant information, i.e., intentional user information, a parameterized spatial and temporal filtering is used to make a good and configurable adaptation.

[0095] In some embodiments the hand analysis may comprise any one or more of the three detection types, hand discrete pose detection, hand key point(s) detection and hand mesh detection. Also, in some embodiments the hand analysis for recognizing the finger pointing pattern may differ from the hand analysis for recognizing the transition. That is, the hand analysis detection types for recognizing the finger pointing pattern may differ from the hand analysis detection types for recognizing the transition, i.e., the confirmation.

[0096] However, a confirmation based on a hand discrete pose detection and / or a hand key point detection has the particular advantage of being accurate and of low latency. In particular, the combination of traditional computer vision techniques, for example, for detection of the hand key point(s), hand mesh, hand contour, and a small neural network, in particular, a convolutional neural network, for detection of the hand discrete pose has the advantage of low latency. In some embodiments the circuitry is further configured to recognize the transition based on recognizing the transition of the recognized finger pointing pattern at a first timepoint to the recognized finger pointing pattern together with a thumb-extension pattern at a second timepoint later than the first timepoint, wherein the finger pointing pattern together with the thumbextension pattern at a second timepoint is based on the recognized finger pointing pattern at a first timepoint.

[0097] The thumb-extension may be considered as a confirmation of the pointing, that is, for example, the pointing towards a target object, determined based on the pointing direction. The pointing may be recognized, i.e., realized, as an object pointing based on the recognized finger pointing pattern. The confirmation, which may be considered a confirmation of a pointing target selection, may be based on detecting the start of the confirmation, that is the start of the transition from pattern A (finger pointing pattern at first timepoint) to pattern A+B (finger pointing pattern at second timepoint and thumb extension pattern). Thus, based on the detected start of the transition, the hand gesture may be visualized in virtual space, i.e., realizing of the transition may occur. That is, the initial pointing (pattern A, finger pointing pattern at first timepoint) may be used as a basis for visualizing, i.e., realizing, the pattern A+B (finger pointing pattern and thumb extension) in virtual space. That is instead of visualizing pattern A+B (finger pointing pattern at a second time pattern and thumb extension) directly, the visualization would occur based on the finger pointing pattern at a first time point and the thumb extension pattern. Thus, any parasitic jitter induced by the confirmation, i.e., the movement of the thumb extension influencing the finger pointing pattern, may be reduced.

[0098] To further improve the hand tip jitter, that is, to further decrease a possible jitter, during thumb - extension, the final confirmation position may be either fixed as the last frame(s) of pointing (finger pointing pattern) of the first timepoint, or an especially strong filtering may be applied during the thumb movement (i.e., during the thumb-extension) to make it smoother.

[0099] The circuitry may be further configured to recognize the finger pointing pattern and / or the transition based on a hand detection, in particular, a detection of a hand region of interest, based on the gesture sensor data. In other words, based on the gesture sensor data the circuitry may compute a hand region of interest (hand ROI).

[0100] The hand analysis, that is, the detection of the hand discrete pose and / or the hand key point(s) and / or the hand mesh, may be based on the hand detection. In particular the hand analysis, the detection of the hand discrete pose and / or the hand key point(s) and / or the hand mesh may be based on the detection of the hand region of interest. That is the hand analysis, the detection of the hand discrete pose and / or the hand key point(s) and / or the hand mesh may occur within the boundaries of the hand region of interest.

[0101] Furthermore, the confirmation, that is, the recognition of the transition from finger pointing pattern alone to finger pointing pattern together with thumb extension pattern, may be triggered by the detection of a hand discrete pose, such as the finger pointing pattern with thumb-extension pattern, within the hand region of interest. The thumb-extension may be detected as a thumb essentially perpendicular (with a small angle tolerance) to a finger, for example, an index finger.

[0102] The circuitry may be further configured to recognize the finger pointing pattern and / or the transition based on gesture classification. The gesture classification may include an event-based messenger. For example, if a gesture indicates the start-up of a virtual event, such as the pop-up of a virtual menu or a virtual slider bar. The gesture classification may also include a continuous controller. For example, if a gesture indicates a continuous activity in virtual space, such as the pointing gesture being used to move a sliding bar.

[0103] The circuitry may be further configured to perform gesture classification based on any one or a combination of hand discrete pose vocabulary, movement vocabulary and transition of hand discrete pose vocabulary.

[0104] In particular, gesture classification may, for example, be based on a hand discrete pose vocabulary and a movement vocabulary or a transition between different hand discrete poses together with a movement vocabulary.

[0105] The hand discrete pose vocabulary may include an any hand discrete pose, such as for example, an index finger pointing pose, an index finger and thumb-extension pointing pose, a two finger pointing pose, such as an index finger and middle finger extension pose, a five finger extension pose, a pose of index finger and thumb touching, and the like.

[0106] The movement vocabulary may for example include a 3D linear movement, a click movement (if a finger moves down or down and up), a wave movement (e.g., if the movement goes back and forth multiple times), a circle movement, being still (no movement), and the like, or any combination of thereof.

[0107] The transition of the hand discrete pose vocabulary may include any transition from one hand discrete pose to another hand discrete pose.

[0108] Thus, any type of gesture classification can be performed with a combination of different vocabularies, which is of advantage as gestures are uniformly treated, that is, gestures may be configured uniformly, which may be of particular advantage for gesture applications. Furthermore, gestures may be user configurable and may allow for a flexible gesture classification with a vast variety of classes, wherein not every gesture event need to be predefined as an individual pattern.

[0109] Furthermore, gesture events and therefore, gesture patterns, may be configurably (e.g., user configurable) linked to a variety of effects in the virtual space. For example, when a gesture is made by the user to trigger the activation or appearance of a sliding bar, or any other user interface (UI) feature (which refers to any feature of the virtual space), a user may signal and confirm that that was their intention via a gesture event which is recognized as a gesture pattern, for example, by forming an "OK" symbol with their hand, that is, by pinching thumb and index finger together. If the wrong UI feature (feature of virtual space) is displayed, the user may cancel it or "escape" by waving their hand, or swiping it away with the back of their hand. Any suitable gesture vocabulary can be assigned to the cancel or escape function. It may be tiered so that one gesture goes back one step in the UI and another or double gesture goes back to the beginning.

[0110] In some embodiments the gesture recognition may therefore include three stages, the hand detection, the hand analysis and the gesture classification. Additionally, an extended reality demonstration, for example, a virtual object interaction, such as menu item selection or sliding bar dragging, or virtual painting or virtual map navigation, or the like, may be performed, i.e., visualized to the user.

[0111] That is based on a received input signal from a sensing system, i.e., the gesture sensor data, the gesture recognition may be performed by the circuitry. Thus, the circuitry may be configured to receive an input signal from a sensing system, perform hand detection in order to select a hand ROI, perform hand analysis within the hand ROI in order to detect a hand discrete pose, a hand key point and / or a hand mesh, classify a hand gesture based on the hand discrete pose, movement and / or transition of different hand discrete poses, and realize a user interaction within the virtual space, e.g., with a menu or a gesture application.

[0112] The circuitry may be further configured to recognize one or more gesture patterns based on the obtained gesture sensor data and realize the one or more gesture patterns in virtual space. That is, additionally, one or more gesture patterns different from the finger pointing pattern or the finger pointing pattern with thumb-extension pattern may be recognized and realized in virtual space. However, the recognition and realization of gesture patterns may align to the recognition and realization as explained above. Thus, the above explained hand detection, hand analysis and / or gesture classification may also be performed with respect to the gesture patterns. In particular, any one or a combination of the hand discrete pose vocabulary, the movement vocabulary and / or the transition of different hand discrete poses may be used to recognize and realize the gesture patterns. Thus, a vast and flexible variety of combinations of different user hand gestures may be recognized as gesture patterns without predefining every user gesture as a pattern in advance.

[0113] The circuitry may be further configured to realize the object pointing in virtual space based on a virtual close-distance menu, a virtual long-distance menu, a virtual sliding bar, a virtual painting application or a virtual object manipulation application.

[0114] That is the user may interact with hand gestures with a virtual menu, which may be a longdistance menu, if the menu is out of a user’s arm’s reach, a close-distance menu if the menu is within arm’s reach, for example, 30 cm to 90 cm away. In other words, the user may point to items of a long-distance menu or a close-distance menu. A long-distance menu may be a menu starting from a distance of 50 to 60 cm up to a distance of 15 meters, but an over 100 m distance menu or an over 1000 m distance menu may also be possible if the menu is large (width and height) enough. The user may interact with hand gestures with a slider bar by pointing at the slider bar and dragging it across the virtual space, that is, within the constraints of the slider bar menu, for example, a horizontal dragging motion may be performed. Also, virtual painting or floating painting may be performed by the user via hand gestures. Other applications may include automotive applications, wherein, for example, passenger pointing may be performed. Thus, the present embodiments may be applicable to any extended reality application, wherein a user interacts with their hand, in particular, wherein a user interacts via pointing, within the extended reality space.

[0115] The type of virtual object or virtual gesture application, e.g., virtual menu (close vs far distance), virtual slider bar, virtual painting, etc., a user can interact with in extended reality with a gesture may also limit or inform the gesture recognition of the present embodiments. For example, the slider bar may only be dragged horizontally. Thus, if the slider bar application is active, any movements of the user in the vertical direction may not be recognized within the gesture recognition as they are not useful for the currently active extended reality application.

[0116] Additionally, depending on the application a context aware gesture may be performed as warmup, for example a pointing gesture and being still for a period of time may be recognized by the circuitry of the electronic device as a warm-up. The sliding bar may be a horizontal (x-direction) or vertical (y-direction) sliding bar or it may be a sliding bar movable in the front and back-direction relative to the user (z-direction). The sliding bar may be used to control any component or parameter of a virtual object or scene, such as, the size, color or orientation of an object as well as next frame selection or map navigation. Therein, the present embodiments allow for a stable, smooth and continuous sliding bar interaction which lead to stable adjustments of virtual object components and virtual scene parameters.

[0117] A selection or a change based on any gesture interaction of the user may be effective essentially immediately from the point of view of the user interface. However, the selection or change may be confirmed based only in combination with the confirmation, that is, the transition including the thumb-extension. In this way, the use of the traditional computer mouse point and click may be replaced by a more natural and non-attached user experience. That is, the point may be replaced by showing a pointing finger, such as an index finger, and the click may be replaced by the pointing finger with thumb extension. Thus, the click of the present embodiments may mean both a static pose of showing the finger, e.g., the index finger, with the thumb and it may also mean the gesture of changing from the pointing pose to the clicking pose. Furthermore, the disclosure may provide for the emulation of the use of a computer mouse where the pointing and thumb extension is "click and hold”. Thus, there may be a first thumb-extension position, for example at approximately 45 degrees or with a bent thumb and a second thumb-extension at 80- 90 degrees indicating a double click (corresponding to the double click of a computer mouse). Also, a double click may be indicated by two distinct thumb-extension movements within a predetermined period of time. By using and detecting the pointing pose, i.e., by recognizing the finger pointing pattern, and two different degrees of thumb-extension, more generally a tri-state or three state control of the UI (virtual space) may be performed. The three states can depend on the configuration of the UI and use case. Additionally, a reverse transition may indicate a confirmation. That is, a transition from finger pointing alone to finger pointing with thumb extension (finger pointing pattern with thumb-extension pattern) as a confirmation gesture may be replaced by the reverted transition from finger pointing with thumb extension (finger pointing pattern with thumb-extension pattern) to finger pointing (finger pointing pattern). Alternatively, a computer mouse click and release event, or any other mouse click even, may be simulated by any of the abovementioned gesture pattern transitions (e.g., tri-state) in the forward (as described) order or in the reverse order. All the above-mentioned extended reality applications have in common that a user may perform a hand gesture, for example a pointing gesture, to interact with the virtual object of the application and that at some point in the user interaction a confirmation of a pointing target may be needed. However, depending on the type of extended reality applications and therefore depending on the type of virtual objects interaction possibilities may be constraint.

[0118] Also, in case of a close-distance menu accurate pointing gesture recognition, for example, a 0.3 mm to 1 mm accuracy, may be needed to accurately select different menu items. Similarly, for a long-distance menu interaction a high precision is needed. A hand jitter would therefore negatively impact the accuracy and precision of selection.

[0119] However, as the present embodiments may include natural hand poses and a confirmation via recognition of a transition from a finger pointing pattern to a finger pointing pattern with thumbextension pattern, hand jitter is reduced. Generally speaking, human physiology allows a user to extend their thumb at least to a certain degree without substantially moving their (index) finger from a pointing pose.

[0120] Concerning adjacent menu items within a virtual menu (long-distance or close-distance), the border between adjacent menu items may include a predetermined tolerance, in a way that pointing gestures close to the border are recognized as pointing gestures towards the middle of a menu item. In the same way, a resistance that a physical button may have in the real world may be simulated as a constraint within the virtual space of a virtual menu application.

[0121] Some embodiments also pertain to a method for gesture recognition comprising obtaining gesture sensor data of a hand, recognizing a finger pointing pattern based on the obtained gesture sensor data, realizing the recognized finger pointing pattern as an obj ect pointing in a virtual space, recognizing a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data and realizing the recognized transition as a confirmation of the object pointing in the virtual space.

[0122] The method may also implement any one or more of all the processes described above.

[0123] Some embodiments pertain to a computer program comprising instructions which, when executed by a processor, performs the method.

[0124] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, 1 when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0125] Returning to Fig. la, an embodiment of gesture recognition of a finger pointing pattern based on user pointing at a virtual long-distance menu is schematically illustrated. Virtual space 1 includes long-distance menu 2, a virtual hand showing finger pointing pattern 3a and including fingertip hand key point 5a which indicates the hand tip. The virtual hand indicates that a user is interacting with the virtual space by performing a pointing gesture towards long-distance menu 2. The user’s pointing gesture is recognized as finger pointing pattern 3a and realized in virtual space as a virtual pointing at a long-distance menu by the virtual hand performing finger pointing pattern 3a. In other words, the visualized virtual hand performing finger pointing pattern 3a may indicate that a user gesture has been performed which is recognized as finger pointing pattern 3a and realized as finger pointing pattern 3 a according to some of the present embodiments.

[0126] The virtual hand performing finger pointing pattern 3 a includes fingertip hand key point 5 a indicating that hand analysis, in particular hand key point detection has been performed to recognize the gesture. The virtual hand performing finger pointing pattern 3 a points in the direction of long-distance menu 2 indicated by pointing line 6, which starts from fingertip hand key point 5a to current projection 7, which indicates the current remote hand pointer. That is current projection 7 indicates the pointing target of a user interacting with the virtual space by pointing at long-distance menu 2.

[0127] Current selection 8 indicates the menu item selected by the user. There is a difference between current projection 7 and current selection 8 as there is a border tolerance between adjacent menu items indicated by the menu grid. Thus, even though current selection 8 falls partly on the border of two adjacent menu items indicated by the grid line, current selection 8 is located in the center of the menu item most of current projection 7 falls into. This may indicate to a user which selection target, i.e., which menu item, has been selected. Final selection indicator 4 is positioned outside long-distance menu 2 indicating that a final selection has not been performed. The virtual hand showing finger pointing pattern 3a may indicate the pointing gesture of a user performing pointing within an extended reality set-up. Furthermore, the finger pointing pattern 3a may be a recognized finger pointing pattern recognized and realized according to some of the present embodiments.

[0128] Fig. lb schematically illustrates an embodiment of gesture recognition of a finger pointing pattern based on user pointing at a virtual long-distance menu. Fig. lb is a continuation of Fig. la including all the above-mentioned objects. However, in Fig. lb compared to Fig. la pointing line 6 current projection 7 and current selection 8 fall within another location on the menu indicating a different selected menu item. Thus, Fig. la and Fig. lb illustrate how a user may freely select different menu items of a virtual long-distance menu via pointing gestures that are recognized as finger pointing patterns and realized as object pointing, i.e., pointing towards longdistance menu 2, in virtual space 1. Thus, finger pointing pattern 3a of Fig. 3b may be a recognized finger pointing pattern realized according to some of the present embodiments.

[0129] Fig. 1c schematically illustrates an embodiment of gesture recognition of a finger pointing pattern with thumb-extension pattern as a realized pointing confirmation of Fig. lb. Fig. 1c is a continuation of Fig. lb following Fig. la. Pointing direction 6, current projection 7 and current selection 8 fall on the same location on the menu as in Fig. lb indicating the same selected menu item as in Fig. lb. However, the virtual hand of Fig. 1c shows finger pointing pattern with thumb extension pattern 3b. This indicates that the selected menu item indicated by current selection 8 is confirmed. Furthermore, it is illustrated that the target selection of a menu item performed in Fig. lb is confirmed in Fig. 1c. A user may therefore, have pointed at a virtual selection target, a menu item of long-distance menu 2, in Fig. lb and confirmed the selection by extending the thumb, which may have been recognized and realized according to some of the present embodiments. Therefore, Figs, lb and 1c illustrate gesture recognition according to some of the present embodiments. In other words, the transition from finger pointing pattern 3a of Fig. lb to finger pointing pattern and thumb-extension pattern 3b of Fig. 1c may have been recognized as a selection confirmation according to some of the present embodiments and realized in virtual space 1 as the virtual hand performing the recognized gesture. This is also indicated by final selection indicator 4, which overlaps the same menu item indicated by current selection 8. Thus, final selection indicator 4 may also be a realization of the recognized transition from finger pointing pattern 3a to finger pointing pattern with thumb-extension pattern 3b.

[0130] The gesture recognition according to some of the present embodiments as described above regarding Figs, la and lb, that is, the recognition and realization of finger pointing pattern 3a of Figs, la and lb in virtual space 1 and the recognition and realization of the transition from finger pointing pattern 3a of Fig. lb to finger pointing pattern with thumb extension 3b in virtual space 1 may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0131] Fig. 2a schematically illustrates an embodiment of a virtual space and a recognized and realized gesture pattern in virtual space. Virtual space 1 includes virtual ball 9 and selection pointer 12. Selection pointer 12 currently floats in virtual space indicating that no selection has been performed. Furthermore, a virtual hand performing gesture pattern 3c is located within the virtual space indicating that a user hand has been recognized and realized to be interacting with the virtual space.

[0132] Fig. 2b schematically illustrates an embodiment of gesture recognition of a starting gesture for activating a virtual sliding bar. Fig.2b is a continuation of Fig. 2a showing that the virtual hand performs gesture pattern 3d, thus indicating that a user has performed a gesture indicated by gesture pattern 3d. The gesture pattern 3d indicates activation of a sliding bar, which is visualized in Fig. 2b by the appearance of sliding bar 11 and horizontal marker 10 indicating the constraints of sliding bar 11 as a horizontally movable sliding bar. Thus, it is indicated to a user that the sliding bar may only be moved (selected) by performing horizontal hand movements. Other motions such as vertical hand movements (up-down movements) would not be recognized as a sliding bar selection movement. Alternatively, to the gesture pattern 3d, the starting gesture for activating the sliding bar may be any gesture in the configurable gesture vocabulary (hand discrete pose vocabulary, movement vocabulary, transition of hand discrete pose vocabulary).

[0133] Fig. 2c schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual sliding bar operation. Fig.2c is a continuation of Fig. 2b showing that the virtual hand performs finger pointing pattern 3a. Furthermore, pointing selection pointer 12 is located at the fingertip of the virtual hand and essentially overlaps sliding bar 11, from the perspective of the virtual hand indicating a user’s position. Thus, it is indicated that the user touch selected the sliding bar 11 via pointing and dragged it horizontally along the horizontal marker 10. In other words, by performing the finger pointing gesture as indicated by finger pointing pattern 3a at the sliding bar 11 position, a finger pointing pattern is recognized and realized in virtual space 1 as object pointing, i.e., sliding bar pointing and dragging operation. At the same time, the sliding bar 11 dragging towards the right by the user affects virtual ball 9. That is, the size of virtual ball 9 became smaller in Fig. 2c with the user operation of dragging sliding bar 11 towards the right. Also, the sliding bar may move if the hand tip touches the sliding bar 11 or is merely pointing to the sliding bar 11. If a touch of or pointing to sliding bar 11 occurs, sliding bar 11 may change to another color, for example, green color, to indicate to a user that the user can start the sliding bar movement to continuously adjust property, for example, the size of the, virtual ball 9, via a left / right 3D movement. Alternatively, any directions in 3D space may be supported not merely the left / right movement. Fig. 2d schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual sliding bar operation. Fig.2d is a continuation of Fig. 2c showing that the virtual hand still performs finger pointing pattern 3a and as selection pointer 12 indicating the fingertip of the pointing finger still overlaps sliding bar 11, the sliding bar operation of dragging the sliding bar 11 is still active. However, in contrast to Fig. 2b the sliding bar 11 is dragged towards the left by the user as indicated by the virtual hand performing finger pointing pattern 3a towards the left of the horizontal marker 10. Therefore, the size of virtual ball 9 is increased. In other words, the user’s finger pointing selection gesture is recognized as a finger pointing pattern and realized according to the constraints and context of virtual space 1, i.e., the active sliding bar application, as object pointing, i.e., as dragging sliding bar 11 along the horizontal marker 10 via pointing, which in turn affects the size of virtual ball 9 visualized in virtual space 1.

[0134] Therefore, to stop the current selection of Fig. 2d the user may show their thumb (thumbextension) as illustrated in Fig. 2e. To restart the sliding again, the user may also need to touch or point to the last location of the sliding bar 11 as illustrated in Fig. 2d. In that case, the previous selection may not be affected by accidently touching the horizontal marker 10. However, if the sliding bar is deactivated as illustrated in Fig. 2g, the previous selections may not change.

[0135] Fig. 2e schematically illustrates an embodiment of gesture recognition for a confirmation of a virtual sliding bar selection. Fig.2e is a continuation of Fig. 2d showing that the virtual hand performs finger pointing pattern with thumb-extension pattern 3b. This indicates that a user confirmed the last sliding bar selection of Fig. 2d. That is, the user performed the thumbextension after dragging the sliding bar 11 in the position to the left of the horizontal marker as illustrated in Fig. 2d. In turn, a finger pointing pattern and thumb-extension is recognized based on the user gestures and realized in virtual space 1 as the virtual hand performing finger pointing pattern with thumb-extension pattern 3b. Therefore, a transition from the finger pointing pattern 3a of Fig. 2d to the finger pointing pattern with thumb extension 3b of Fig. 2e is recognized, and realized as a confirmation of the last selection of Fig. 2d.

[0136] In other words, the size of virtual ball 9, which was freely changed as illustrated and explained in regard to Figs. 2c to Fig. 2d, was finally selected in Fig. 2d and the selection is confirmed in Fig. 2e.

[0137] Fig. 2f schematically illustrates an embodiment of gesture recognition for a confirmation of a virtual sliding bar selection. Fig.2e is a continuation of Fig. 2d showing that the virtual hand still performs finger pointing pattern with thumb-extension pattern 3b. However, now the selection pointer 12 is not overlapping the sliding bar 11. Together with the visualization of the virtual hand, this also indicates that the recognized transition from gesture pattern as illustrate in Fig. 2d to the gesture pattern as indicated by Figs. 2e and 2f, is realized as a confirmation of the selection as indicated by Fig. 2d. In other words, the size of virtual ball 9, which was freely changed as illustrated and explained in regard to Figs. 2c to Fig. 2d, was finally selected in Fig. 2d and the selection is confirmed as illustrated Fig. 2e and Fig. 2f.

[0138] Fig. 2g schematically illustrates an embodiment of gesture recognition of a closing gesture of a virtual sliding bar after confirmation of a virtual sliding bar selection. Fig.2g is a continuation of Fig. 2f showing that the virtual hand now performs gesture pattern 3d. After the selection of the virtual ball size 9 of Fig. 2d was confirmed in Figs. 2e and 2f, the user now closes the sliding bar application by performing a gesture according to gesture pattern 3d, which is recognized and realized in virtual space. In consequence the sliding bar 11 and horizontal marker 10 disappear. Virtual ball 9 remains the size selected and confirmed in Figs. 2d to 2f. Alternatively, to the gesture pattern 3d, the closing gesture for deactivating the sliding bar application may be any gesture in the configurable gesture vocabulary (hand discrete pose vocabulary, movement vocabulary, transition of hand discrete pose vocabulary).

[0139] Fig. 2h schematically illustrates an embodiment of gesture recognition of a reset gesture for a virtual sliding bar application. Fig.2h is a continuation of Fig. 2g showing that the virtual hand now performs gesture pattern 3e indicating a reset of the processes as shown in Figs. 2b to 2g. After closing the sliding bar application in Fig. 2g, the user now performs a hand waving gesture, as indicated by the arrows, which is recognized as gesture pattern 3e and realized in virtual space 1 as the virtual hand performing gesture pattern 3e. As a consequence of the user’s hand waving which is recognized as gesture pattern 3e, the gesture pattern 3e is realized in virtual space 1 as a visualized gesture pattern 3e and as a reset of ball size of virtual ball 9. The size of virtual ball 9 has increased to its starting size as indicated in Figs. 2a and 2b before the user performed the sliding bar operation in Figs. 2c to 2f. Alternatively, to the hand waving gesture pattern other gesture events may be set by a user as gesture patterns that would lead to a reset or initializing of the application. The same is true for gesture patterns of, for example, Figs. 2b and 2g. Alternatively, to the gesture pattern 3e, the reset gesture for resetting the sliding bar may be any gesture in the configurable gesture vocabulary (hand discrete pose vocabulary, movement vocabulary, transition of hand discrete pose vocabulary).

[0140] The gesture recognition according to some of the present embodiments as described above regarding Figs. 2a to 2h, in particular, the recognition and realization of finger pointing pattern 3a of Figs. 2c and 2d in virtual space 1 and the recognition and realization of the transition from finger pointing pattern 3 a of Fig. 2d to finger pointing pattern with thumb extension 3b in virtual space 1 of Figs. 2e and 2f may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0141] Fig. 3a schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for a virtual painting application. That is, a user points with their finger in virtual space by using the finger point as a brush to paint with in 3D space of the virtual space 1. The finger pointing gesture of the user is recognized as a finger pointing pattern and within the constraints of the virtual painting application the recognized finger pointing pattern is realized as the virtual hand performing the finger pointing pattern 3a and painting a line 13 comprising many 3D- voxels. For that purpose, fingertip hand key point 5a is detected and visualized in virtual space 1 as the fingertip of the virtual hand indicating to the user the finger point used as a brush for painting line 13.

[0142] Fig. 3b illustrates an embodiment of gesture recognition of a selection confirmation for a virtual painting application. Fig.3b is a continuation of Fig. 2a showing that the virtual hand now performs finger pointing pattern with thumb-extension pattern 3b. That is, a user who has been painting line 13 in the 3D space of virtual space 1 by pointing with their finger and using the fingertip as a brush, now confirms the painted line 13 by extending the thumb. The thumb extension following the finger pointing is recognized as a transition from a finger pointing pattern to the finger pointing pattern with thumb extension pattern and realized in virtual space 1 as a confirmation of the previously drawn line 13.

[0143] The gesture recognition according to some of the present embodiments as described above regarding Figs. 3a and 3b, in particular, the recognition and realization of finger pointing pattern 3a of Fig. 3a in virtual space 1 and the recognition and realization of the transition from finger pointing pattern to finger pointing pattern with thumb extension 3b in virtual space 1 of Fig. 3b as a confirmation of painted line 13 may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0144] Fig. 4a schematically illustrates an embodiment of gesture recognition of a finger pointing pattern for user pointing at a close distance menu. Virtual space 1 includes close-distance menu 14, a virtual hand showing finger pointing pattern 3 a and including fingertip hand key point 5 a which indicates a detected key point, namely the hand tip. The virtual hand indicates that a user is interacting with the virtual space by performing a pointing gesture towards close-distance menu 14. The user’s pointing gesture is recognized as finger pointing pattern 3a and realized in virtual space as a virtual pointing at close-distance menu 14. In other words, the visualized virtual hand performing finger pointing pattern 3a may indicate that a user gesture has been performed is recognized as finger pointing pattern 3a and realized as finger pointing pattern 3a according to some of the present embodiments.

[0145] The virtual hand performing finger pointing pattern 3 a includes fingertip hand key point 5 a indicating that hand analysis, in particular hand key point detection has been performed to recognize the gesture. The virtual hand performing finger pointing pattern 3 a points in the direction of close-distance menu 14 indicated by pointing line 6, which starts from fingertip hand key point 5a to current projection 7, which indicates the current remote hand pointer. That is current projection 7 indicates the pointing target of a user interacting with the virtual space by pointing at close-distance menu 14.

[0146] Current selection 8 indicates the menu item selected by the user. There is a minimal difference between current projection 7 and current selection 8, as current selection 8 always falls in the center of a menu item and current projection 7 is located close to the center. Thus, current selection 8 may indicate to a user which selection target, i.e., which menu item, has been selected. Final selection indicator 4 is positioned at another location on the close-distance menu indicating a previous final selection and that a current final selection has not been performed yet.

[0147] Fig. 4b schematically illustrates an embodiment of gesture recognition of a finger pointing pattern with thumb-extension pattern as a realized pointing confirmation of Fig. 4a. Fig. 4b is a continuation of Fig. 4a. Pointing line 6, current projection 7 and current selection 8 fall on the same location on the menu as in Fig. 4a indicating the same selected menu item as in Fig. 4a. However, the virtual hand of Fig. 4a shows finger pointing pattern with thumb extension pattern 3b. This indicates that the selected menu item indicated by current selection 8 is confirmed. Furthermore, this is realized by the final selection indicator 4 falling on the same location as the current selection 8. Thus, it is illustrated that the target selection of a menu item performed in Fig. 4a is confirmed in Fig. 4b. A user may therefore, have pointed at a virtual selection target, a menu item of close-distance menu 14, in Fig. 4a and confirmed the selection by extending the thumb, which may have been recognized and realized according to some of the present embodiments. Therefore, Figs. 4a and 4b illustrate gesture recognition according to some of the present embodiments. In other words, the transition from finger pointing pattern 3a of Fig. 4a to finger pointing pattern and thumb-extension pattern 3b of Fig. 4b may have been recognized as a selection confirmation according to some of the present embodiments and realized in virtual space 1 as the virtual hand performing the recognized gesture. This is also indicated by final selection indicator 4, which overlaps the same menu item indicated by current selection 8. Thus, final selection indicator 4 may also be a realization of the recognized transition from finger pointing pattern 3a to finger pointing pattern with thumb-extension pattern 3b.

[0148] The gesture recognition according to some of the present embodiments as described above regarding Figs. 4a and 4b, that is, the recognition and realization of finger pointing pattern 3a of Fig. 4a in virtual space 1 and the recognition and realization of the transition from finger pointing pattern 3a of Fig. 4b to finger pointing pattern with thumb extension 3b in virtual space 1 may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0149] In summary concerning the close-distance menu 14 application, either a clicking by touching the close-distance menu 14 or a clicking by thumb-extension (Fig. 4b) may be performed. That is, if the menu is in a close-distance range, for example, a 40 cm range, a hand floating over the menu, for example, 10 cm above, may be visualized and a projection (orthogonal projection from hand tip to the menu) may be used as a current selection. Then the selection may be confirmed by showing the thumb-extension gesture. This may solve the problem of a user easily clicking in a wrong direction in virtual space when a close-distance menu has a small menu size or small menu items.

[0150] Concerning the above examples (Figs, la to 4b) of realized gesture events, depending on the context of the currently active virtual space 1, the realization of the gesture events, such as, for example, finger pointing pattern 3a and finger pointing pattern with thumb-extension 3b, may differ. Realization may include that the gesture patterns are visualized as a virtual hand with key points 5a and 5b, e.g., aa rendered polygon hand. However, it may be that the gesture pattern is not visualized as a hand, but as another object, or it may be that the gesture pattern is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 are visualized. Alternatively, no visualization may be conducted at all and the effect of the gesture event may be simply recognized as an effect within the virtual space 1.

[0151] Furthermore, concerning the above examples, (Figs, la to 4b) the virtual objects, such as, longdistance menu 2, close-distance menu 14, sliding bar 11, virtual ball 9 etc, may be displayed on a 2D display, for example, a screen such as a LCD or OLED screen. The virtual objects may be projected onto a surface, for example from a front position or from behind. That is, the objects may be visualized from the front or from behind. Also, the virtual objects may be in a 3D Spatial reality display such as the Sony ELF-SR1 or ELF-SR2.Fig. 5a schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space. Virtual space 1 includes a realized virtual hand performing gesture pattern 3e of a waving hand in line with gesture sensor data of a user’s hand performing a hand waving gesture as illustrated in Fig. 5b. The realized virtual hand performing gesture pattern 3e includes fingertip hand key point 5a and center key point 5b. Depending on the context of the virtual space 1, the realization may include virtual object interactions as described with regard to Fig. 2h.

[0152] The realization of gesture pattern 3e may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0153] Fig. 5b schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. The user’s hand performing the hand waving gesture 21e in real space 20 is illustrated in Fig. 5b. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a of the tallest finger indicating the hand tip and of center hand key point 5b is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest 22, the detected hand key points 5a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as a hand waving gesture pattern (3e of Fig. 5a). Then, based on the detected hand region of interest, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture, the hand waving gesture pattern is realized as gesture pattern 3e in virtual space 1 of Fig. 5a including hand key points 5a and 5b. Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5a may also be based on the detected hand discrete pose.

[0154] Depending on the context of the currently active virtual space, the realization of the gesture pattern 3e may differ. The pattern may be visualized as a virtual hand with key points 5a and 5b, e.g., a rendered polygon hand, as illustrated in Fig. 5a. However, the gesture pattern may not be visualized as a hand, but as another object, or it may be that the gesture pattern itself is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 may be visualized, as illustrated, for example, in regard to closing / resetting of the sliding bar application in Figs. 2h. Alternatively, no visualization may be conducted and the effect of the gesture event may simply be recognized as an effect within the virtual space 1.

[0155] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0156] Fig. 5c schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. Virtual space 1 includes a realized virtual hand performing finger pointing pattern 3a in line with gesture sensor data of a user’s hand performing a finger pointing gesture as illustrated in Fig. 5d. The realized virtual hand performing finger pointing pattern 3 a includes fingertip hand key point 5 a and center key point 5b. Depending on the context of the virtual space 1, the realization may include virtual object interactions as described with regard to Figs, la, lb, 2c, 2d, 3a and 4.

[0157] The realization of the finger pointing pattern 3 a may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0158] Fig. 5d schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for finger pointing pattern recognition. A user’s hand performing the finger pointing gesture 21a in real space 20 is illustrated in Fig. 5d. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a indicating the hand tip and of center hand key point 5b is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest 22, the detected hand key points 5a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as a finger pointing pattern (3a of Fig. 5c). Then, based on the detected hand region of interest 22, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture the finger pointing pattern is realized as finger pointing pattern 3a in virtual space 1 of Fig. 5c including the hand key points 5a and 5b. Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5c may also be based on the detected hand discrete pose.

[0159] Depending on the context of the currently active virtual space, the realization of the finger pointing pattern 3a may differ. The pattern may be visualized as a virtual hand with key points 5a and 5b, e.g., a rendered polygon hand, as illustrated in Fig. 5e. However, the pattern may not be visualized as a hand, but as another object, or it may be that the hand is not visualized at all and instead only the effect of the finger pointing pattern on other objects within visual space 1 is visualized. Alternatively, no visualization may be conducted and the effect of the gesture event may as a finger pointer may be simply recognized as an effect within the virtual space 1.

[0160] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0161] Fig. 5e schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space. Virtual space 1 includes a realized virtual hand performing gesture pattern 3f in line with gesture sensor data of a user’s hand performing a hand gesture as illustrated in Fig. 5f. The realized virtual hand performing gesture pattern 3f includes fingertip hand key point 5a and center key point 5b.

[0162] The realization of the gesture pattern 3f may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0163] Fig. 5f schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. A user’s hand performing gesture 21f in real space 20 is illustrated in Fig. 5f. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a of the tallest finger indicating the hand tip and of center hand key point 5b is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest, the detected hand key points 5 a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as a gesture pattern (e.g., as gesture pattern 21f of Fig. 53). Then, based on the detected hand region of interest, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture, the gesture pattern is realized as gesture pattern 3f in virtual space 1 of Fig. 5e including hand key points 5a and 5b.

[0164] Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5e may also be based on the detected hand discrete pose.

[0165] Depending on the context of the currently active virtual space, the realization of the gesture pattern 3f may differ. The pattern may be visualized as a virtual hand with key points 5a and 5b, e.g., a rendered polygon hand, as illustrated in Fig. 5e. However, it may be that the hand is not visualized as a hand, but as another object, or it may be that the hand is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 are visualized. Alternatively, no visualization may be conducted and the effect of the gesture event may be simply recognized as an effect within the virtual space 1.

[0166] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0167] Fig. 5g schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space. Virtual space 1 includes a realized virtual hand performing gesture pattern 3g, a thumb pointing to the left, in line with gesture sensor data of a user’s hand performing a hand gesture as illustrated in Fig. 5h. The realized virtual hand performing gesture pattern 3g includes fingertip hand key point 5a and center key point 5b. Alternatively, the user hand of Fig. 5h may also perform a gesture, wherein the thumb points to the right, which may then be reflected in gesture pattern 3g as a thumb pointing to the right.

[0168] The realization of the gesture pattern 3g may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0169] Fig. 5h schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. A user’s hand performing gesture 21g in real space 20 is illustrated in Fig. 5f. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a and of center hand key point 5b is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest 22, the detected hand key points 5a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as a gesture pattern (e.g., gesture pattern 3g of Fig. 5g). Then, based on the detected hand region of interest, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture, the gesture pattern is realized as gesture pattern 3g in virtual space 1 of Fig. 5g including hand key points 5a and 5b.

[0170] Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5g may also be based on the detected hand discrete pose.

[0171] Depending on the context of the currently active virtual space, the realization of the gesture pattern 3g may differ. The pattern may be visualized as a virtual hand with key points 5a and 5b, e.g., a rendered polygon hand, as illustrated in Fig. 5g. However, it may be that the hand is not visualized as a hand, but as another object, or it may be that the hand is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 are visualized. Alternatively, no visualization may be conducted and the effect of the gesture event may be simply recognized as an effect within the virtual space 1.

[0172] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10. Fig. 5i schematically illustrates an embodiment of a recognized and realized gesture pattern in virtual space. Virtual space 1 includes a realized virtual hand performing gesture pattern 3d in line with gesture sensor data of a user’s hand performing a hand gesture as illustrated in Fig. 5h. The realized virtual hand performing gesture pattern 3d includes fingertip hand key point 5a and center key point 5b (not visible). Depending on the context of virtual space 1, the realization may include virtual object interactions as described with regard to Fig. 2b.

[0173] The realization of the gesture pattern 3d may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0174] Fig. 5j schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. A user’s hand performing gesture 21d in real space 20 is illustrated in Fig. 5j. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a of the tallest finger indicating the hand tip and of center hand key point 5b is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest 22, the detected hand key points 5a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as gesture pattern 3d of Fig. 5i. Then, based on the detected hand region of interest, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture, the gesture pattern is realized as gesture pattern 3d in virtual space 1 of Fig. 5g including hand key points 5a and 5b.

[0175] Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5i may also be based on the detected hand discrete pose.

[0176] Depending on the context of the currently active virtual space, the realization of the gesture pattern 3d may differ. The pattern may be visualized as a virtual hand with key points 5a and / or 5b, e.g., a rendered polygon hand, as illustrated in Fig. 5i. However, it may be that the hand is not visualized as a hand, but as another object, or it may be that the hand is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 may be visualized, as illustrated, for example, in regard to activation and closing of the sliding bar 11 in Figs. 2b and 2g. Alternatively, no visualization may be conducted and the effect of the gesture event may be simply recognized as an effect within the virtual space 1.

[0177] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0178] Fig. 5k schematically illustrates an embodiment of hand detection, hand analysis and gesture classification for gesture pattern recognition. Virtual space 1 includes a realized virtual hand performing finger pointing pattern and thumb extension pattern 3b in line with gesture sensor data of a user’s hand performing a finger pointing gesture and thumb extension as illustrated in Fig. 51. The realized virtual hand performing finger pointing pattern with thumb-extension pattern 3b includes fingertip hand key point 5a of the index finger and center key point 5b as well as fingertip hand key point 5 a of the thumb. Depending on the context of the virtual space 1, the realization may include virtual object interactions as described with regard to Figs. 1c, 2e, 2f, 3b and 4b.

[0179] The realization of the finger pointing pattern 3 a may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0180] Fig. 51 schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb extension pattern in virtual space. A user’s hand performing the finger pointing gesture 21b in real space 20 is illustrated in Fig. 51. A sensor (not seen) captures gesture sensor data of the user’s hand on which basis the hand detection and the hand analysis are performed. The hand detection includes detection of hand region of interest 22. Within the hand region of interest 22 the hand analysis is performed. That is, within the hand region of interest 22 the hand key point detection of fingertip hand key point 5a indicating the hand tip and of center hand key point 5b as well as the fingertip hand key point 5a of the thumb is performed. Furthermore, hand contour 23 is detected within the hand region of interest 22. Based on the detected hand region of interest 22, the detected hand key points 5a, 5b and the detected hand contour 23 the gesture of the user’s hand is classified as a finger pointing pattern with thumb extension pattern (3b of Fig. 5k). Then, based on the detected hand region of interest 22, the detected hand key points 5a, 5b, the detected hand contour 23 and the classified gesture the finger pointing pattern is realized as finger pointing pattern 3b in virtual space 1 of Fig. 5k including the hand key points 5a and 5b.

[0181] Additionally, hand discrete pose detection may be performed within hand region of interest 22 as part of the hand analysis to detect the hand discrete pose. In that case the gesture classification as well as the realization of Fig. 5k may also be based on the detected hand discrete pose.

[0182] Depending on the context of the currently active virtual space, the realization of the finger pointing pattern with thumb extension pattern 3b may differ. The pattern may be visualized as a virtual hand with key points 5a and 5b, e.g., a rendered polygon hand, as illustrated in Fig. 51. However, it may be that the pattern is not visualized as a hand, but as another object, or it may be that the hand is not visualized at all and instead only the effect of the gesture event on other objects within visual space 1 is visualized, as illustrated, for example, in regard to the size of the virtual ball 9 of Figs. 2a to 2h. Alternatively, no visualization may be conducted and the effect of the gesture event as a confirmation may simply be recognized as an effect within the virtual space 1.

[0183] The sensor for capturing the gesture sensor data may be camera 109 of electronic device 100 of Fig. 10. The hand detection, including hand region of interest detection, the hand analysis, including hand key point detection and hand contour detection, and the gesture classification may be executed by a circuitry of an electronic device (see 100, Fig. 10), in particular, it may be implemented in one or more processors of the circuitry, e.g., processors such as computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 of Fig. 10.

[0184] Concerning the Figs. 5a to 51, these are examples of possible gestures and gesture events based on a user’s right hand that may be recognized and realized in virtual space 1. However, other gestures and gesture events with the other hand (e.g., opposite hand, the left hand) or the same hand (left hand) which may then be recognized and realized as gesture patterns or other gesture events are possible. In particular, the same gesture events as illustrated in Figs. 5a to 51 but performed based on a left hand are possible and also the reverse gesture events (e.g.., if the hand flips around) of the right or left hand as illustrated in Figs. 5a and 51 may be possible.

[0185] Fig. 6 shows a block diagram depicting an embodiment of recognition and realization of a finger pointing pattern and recognition and realization of a transition from the finger pointing pattern to a finger pointing pattern with thumb extension pattern. At 30 the gesture sensor data is obtained. The gesture sensor data may for example be capture by a camera 109 of electronic device 100 of Fig. 10. At 31 hand detection is performed based on the obtained gesture sensor data 30. The hand detection 31 may include detection of a hand region of interest (22, Fig. 5a- 5j). At 32 hand analysis is performed based on the obtained gesture sensor data and the output of the hand detection, for example, based on the detected hand region of interest (22, Fig. 5a-5j ). Hand analysis 32 may include hand key point detection (e.g., 5a, 5b of Fig. 5a -5j), hand discrete pose detection and / or hand mesh detection, wherein hand mesh detection may include detection of the hand contour (23, Fig. 5a-5j ). The hand analysis 32, that is, the hand key point detection (e.g., 5a, 5b of Fig. 5a -5j), the hand discrete pose detection and / or the hand mesh detection may be performed within the detected hand region of interest (22, Fig. 5a- 5j). At 33 gesture classification is performed based on the output of the hand detection and the hand analysis. That is the gesture classification at 33 may be performed based on the detected hand region of interest and based on the detected hand key point, the detected hand discrete pose and / or the detected hand mesh, which may include the detected hand contour (23, Fig. 5a-5j). The gesture classification at 33 may include an event-based messenger or a continuous controller. The gesture classification at 33 may result in classified gesture events, which may include a finger pointing pattern (see 3a, Figs, la, lb, 2c, 2d, 3a, and 5c) or a transition to a finger pointing pattern with thumb extension pattern (see 3b, Fig. 1c, Figs. 2e-f, Fig. 3b) or another gesture pattern (see 3c-3g, Figs. 2a-b, Figs. 5a, 5e, 5g, 5i and 5j). At 34 a realization of the gesture event in virtual space (e.g. 1 of Figs. 1-4) is performed. The realization at 34 may include a realization as an interaction of the gesture event with an extended reality demonstration, such as a virtual sliding bar application, a virtual painting application, a virtual long-distance or close-distance menu application. That is, depending on the context of the virtual space, the realization may include virtual object interactions as described with regard to Figs. la-4.

[0186] The realization at 34 may be based on the classified gesture events and may also be based on the detected hand region of interest, the detected hand discrete pose, the detected hand key point and / or the detected hand mesh.

[0187] The detected hand key point may be a detected hand tip key point (5a, Figs. 5a-5j ) and / or a detected hand center point (5b, Figs. 5a- 5j).

[0188] In other words, the recognition of gesture events, such as the recognition of a finger pointing pattern or a transition from finger pointing pattern to a finger pointing pattern with thumbextension pattern may include the hand detection at 31, the hand analysis at 32and the gesture classification at 33, and the realization of a gesture event may include the realization at 34. The hand detection at 31 may also include an object detection and the gesture classification at 33 may also include an activity classification.

[0189] Fig. 7a schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb-extension pattern at wide angle. Fig. 7a shows an index finger pointing pattern with a thumb-extension pattern 3b, wherein the thumb is extended at an angle of approx. 80 to 90 degrees from the index finger.

[0190] Fig. 7b schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb-extension pattern at a medium angle Fig. 7b shows an index finger pointing pattern with a thumb-extension pattern 3b, wherein the thumb is extended at an angle of approx. 45 to 60 degrees from the index finger. The medium thumb extension angle may also be below 45 degree to the index finger, but without touching the thumb to the index finger as may be possible in case of a narrow thumb-extension angle (for example, Fig. 7c).

[0191] Fig. 7c schematically illustrates an embodiment of a recognized and realized finger pointing pattern with thumb-extension pattern at a narrow angle. Fig. 7c shows an index finger pointing pattern with a thumb-extension pattern 3b, wherein the thumb is located close to the index finger. The thumb may also be touching the index finger or may be tucked in and touching any part of the closed hand.

[0192] The thumb-extension as illustrated in Figs. 1c, 2e, 2f, 3b and 4b may be the result of a movement between a user's natural pointing pose in which the thumb is nearly parallel to the index finger (see Fig. 7c) to around 45degrees or even 60 degrees from the index finger as illustrated in Fig. 7b or to a clearly extended thumb as illustrated in Fig. 7c. That is, depending on a user's physiology, the thumb extension may cause the thumb to move to 80 or 90 degrees from the index finger (Fig. 7c).

[0193] Some users’ natural pointing pose might be detected as a thumb already in its extended position. In that case, sensor feedback might cause the UI (virtual space) to ask the user to tuck in their thumb when the initial gesture or the gesture that should be a finger pointing gesture, is detected as a thumb-extension, that is, a finger pointing pattern with thumb extension pattern, instead of as a finger pointing alone (i.e., finger pointing pattern alone).

[0194] In other words, a thumb extension may transition from a thumb extension of Fig. 7c at a narrow angle to a thumb-extension of Fig. 7b at a medium angle or to thumb-extension at a wide angle of Fig. 7c or any combination thereof. Furthermore, when an electronic device or a method for performing gesture recognition of some embodiments as described above has been started and has been running for an extended period of time, the device or method may measure how often a user performed any of the thumb-extension events of Figs. 7a to Fig. 7b during the finger pointing (finger pointing pattern with thumb-extension). Thus, the system may smartly learn and adapt the clicking / thumb-extension gesture in a way that a user is most likely to perform the gesture. Furthermore, information to teach and / or guide the user to perform a gesture better, in a way that the gesture pattern is more easily recognized, may be presented to the user, for example as a pop-up on a display.

[0195] Fig. 8a schematically illustrates an embodiment of a user performing a finger pointing gesture in a vehicle. In Fig. 8a the electronic device (which may correspond to electronic device 100 of Fig. 10) for gesture recognition is implemented as a vehicle 50, which may be a car. Alternatively, the electronic device for gesture recognition may be included in the vehicle 50. Vehicle 50 includes display 51, seats 52 roof cameras 106a and wide field of view camera(s) 106b (The cameras 106a and 106b may correspond to camera 106 of Fig. 10). Cameras 106a and 106b may be 3D cameras. Sensing Area 54 illustrates the sensing area covered by cameras 106a and 106b. Within the sensing area 54 images on which sensor gesture data may be based may be captured. A virtual space (1, Figs, la to 51) may be visualized to the user on display 51. User 60 is a vehicle 50 occupant sitting within vehicle 50 and performs pointing gesture 61 towards a virtual object presented on display 51. Fig. 8a may illustrate an AR gaming experience for vehicle occupants 60 using interior sensing cameras 106a and 106 b with or without exterior ADAS sensors (not visible). For interacting with the in car user experience interface (UX HMI) or a Gaming application interior occupant 60 sensing 3D cameras 106a and 106b may be used. An interior camera 106a, 106b may detect the hand position, hand pose and other body key points of the occupants. Furthermore, exterior ADAS sensors may be used to create an AR map of exterior environment of the vehicle. Passengers may interact with the simulated outside environment using hand or finger direction, and events such as a click motion (finger pointing pattern with thumb extension pattern). For that purpose, the location and orientation of the controller (in this case user 60 hand) may be precisely correlated to the surrounding interaction environment and elements; i.e. create a pointer. Furthermore, the transition from a pointer highlighting an item to selecting or confirming it in an intuitive way without moving the pointer itself may be achieved by using 3D interior camera-based algorithms and may be achieved by a hand pose and click function of exposing or rolling the thumb (finger pointing pattern with thumb-extension pattern). This point and click motion may be used to create gaming sequences. Fig. 8b schematically illustrates an embodiment of a side-view of a user performing a finger pointing gesture in a car. Fig 8a shows a side-view of vehicle 50 of Fig. 8a. User 60 sits in the passenger seat 52 and performs the pointing gesture 62 at the virtual space presented on the display (not seen). Gesture recognition may be performed based on the gesture of user 60 as explained in detail in regard to Figs, la to 8a.

[0196] Fig. 9 illustrates an embodiment of a method for gesture recognition. At 70 gesture sensor data of a hand is obtained. At 71 a finger pointing pattern is recognized based on the obtained gesture sensor data. At 72 the recognized finger pointing pattern is realized as an object pointing in virtual space. At 73 a transition between the finger pointing pattern and the finger pointing pattern together with a thumb - extension pattern is recognized based on the obtained gesture sensor data. At 74 the recognized transition is realized as a confirmation of the object pointing in the virtual space.

[0197] Fig. 10 shows a block diagram depicting an embodiment of an electronic device that implements gesture recognition. The electronic device 100 comprises a computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 as processor. The electronic device 100 further comprises camera(s) 206, microphone(s) 107 and loudspeaker(s) 108 that are connected to the processor 101. The processor 101 may for example implement an extended reality (virtual reality or augmented reality) application, a hand detection a hand analysis and / or a gesture classification according to the processes described with regard to Figs, la to 6 in more detail. Loudspeaker 108 may be headphones, e.g., on-ear, in-ear, over-ear, wireless headphones and the like, or may consist of one or more loudspeakers that are distributed over a predefined space and is configured to render any kind of audio, such as 3D audio. The microphone 107 may be configured to receive any kind of audio signal. The camera 106 may be one or more cameras, such as an RGB camera, and IR camera, a ToF camera, for example, an iToF or dTof, an event-based camera or the like.

[0198] The electronic device 100 further comprises a user interface 109 that is connected to the processor 101. This user interface 109 acts as a man-machine interface and enables a dialogue between a user and the electronic device. For example, a user may make configurations to the system using this user interface 109. The electronic device 100 further comprises a Bluetooth interface 104, and a WLAN interface 105. These units 104, 105 act as I / O interfaces for data communication with external devices. For example, additional loudspeakers, microphones, and cameras, e.g., a ToF camera, RGB camera or an event-based camera with WLAN or Bluetooth connection may be coupled to the processor 101 via these interfaces 104 and 105.

[0199] The electronic device 100 further comprises a data storage 102 and a data memory 103 (here a RAM). The data memory 103 is arranged to temporarily store or cache data or computer instructions for processing by the processor 101. The data storage 102 is arranged as a long-term storage, e.g., gesture sensor data, such as image data, and / or hand discrete pose data, hand key point data hand mesh data, hand contour data and / or data on gesture events obtained from the computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 that may implement capture of the gesture sensor data, the hand detection, hand analysis and / or gesture classification.

[0200] The connection between the computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 and the camera 106 may include a camera serial interface (CSI). The CSI is an interface between a camera 106 and a host processor 101. Thus, control signals and data from the computation hardware (e.g., CPU, GPU, TPU, DSP etc.) 101 to the camera 106 as well as from the camera 106 to the processor 101 may be sent.

[0201] Some further embodiments of the disclosure may describe:

[0202] A device comprising circuitry configured to:

[0203] • detect a user hand pose including a pointing gesture, the pointing gesture defining a first position in space;

[0204] • communicate the first position in space to an interface circuitry of a Graphical User Interface (GUI);

[0205] • detect a first change in the first position in space of the pointing gesture to a second position in space;

[0206] • convey the second position in space to the interface circuitry of the GUI;

[0207] • detect a second change, the second change being a change in the user hand pose comprising a displacement of a thumb; and

[0208] • communicate the second change to the interface circuitry of the GUI (to indicate selection of an object at the second position).

[0209] A User Interface running on an apparatus comprising circuitry configured to

[0210] • receive information representing initiation of a user hand pose including a pointing gesture, the pointing gesture defining a first position in space, for example, activated upon detection of the pointing pose;

[0211] • receive information representing the first position in space;

[0212] • receive information representing a second position in space corresponding to a first change in the first position in space of the pointing gesture;

[0213] • receive information representing a second change, the second change being a change in the user hand pose comprising a displacement of a thumb;

[0214] • map the initiation of a user hand pose at the first position in space and the second position in space to the User Interface; and

[0215] • allow manipulation of an object on the User interface at the second position when the information representing the second change is received. In other words, the finger pointing pose may move around and the thumb-extension may trigger the selection. Thus, object edition / manipulation may be conducted via user gestures. Therefore, the position of the pointing gesture and the thumb-extension gesture may be linked to a different mapping of operations. For example, one sample-like pointing gesture to an object of interest together with a thumb-extension may start a state of rotation of the object of interest or may increase the size of the object of interest to study object of interest in detail. This may be achieved via linkage to a state pattern. That is, pointing (pointing gesture) to a position of interest may be a fist state, showing the thumb (e.g., thumb-extension) may correspond to a change from the first state to a second state, moving the pointing gesture with the extended thumb may be a third state and closing the thumb again may achieve a return to the first state. Each state system (e.g., state pattern, state) may predefine some operation vocabulary. Furthermore, there may exist a user support for a user to add more states and / or more operation vocabulary.

[0216] The circuitry may detect one or more additional positions between the first and second position as the pointing gesture moves from first to second position. These additional positions may be conveyed to the interface circuitry of the GUI. The number of additional positions detected may be reduced or capped for example to maintain smoothness of tracking on the GUI and / or reduce the amount of processing and / or data transmission. This may be achieved by filter circuitry. In some embodiments, the positions may be sampled and therefore detected at discrete time intervals, or spatially sampled such that for example only a certain number of intermediate locations are conveyed to the GUI. Temporal or spatial filtering may be applied. In some embodiments an apparatus running the GUI may perform the filtering. Manipulation may include selection or marking for example to provide feedback, initiating any other function associated with the object or processing of the object. The objects may be rectangular or may be any shape. Corresponding methods and non-transitory storage media comprising code components which when executed on computer cause the computer to perform the methods are within scope of the disclosure.

[0217] In some embodiments, the plan of the UI may not be perpendicular to the pointing direction of the user. The UI may be displayed with a virtual tilt such as 45 degrees even if a display screen is perpendicular to the user. By displaying a marker on the UI corresponding to the position pointed to, for example using a virtual ball, a user can recognise the position pointed to and so does not have to start making unnatural hand gestures such as point around angle or a comer. The marker may implicitly guide a user to make a straight pointing pose, even if the UI is angled.

[0218] In some embodiments, the state of the thumb extension allows the user to emulate a click and hold and hold function on the UI. Therefore, an object can be pointed to by the pointing pose, selected by the thumb-extension and further dragged or moved around the interface by maintaining the extended thumb position. It should be noted that the description above is only an example configuration. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces, or the like.

[0219] The electronic device 100 may be a PC or a digital signage device or any other outdoor interface or a vehicle or included in a vehicle (e.g. vehicle 50 of Figs. 8a and 8b) or it may be a mobile device or any other kind of portable or wearable device, for example, for augmented reality applications, such as, for example, smart glasses, head mounted displays (HMDs) earphones or other types of smart wearable devices, or the like.

[0220] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. For example, the ordering of method steps 71, 72 73 and 74 in the embodiment of Fig. 9 may be exchanged, for example, in the order of 73, 74, 71 and 72 or in the order of 73, 71, 72 and 74 or in the order of 71, 73, 72 and 74 or in the order of 73, 71, 74, 72 or in any other order. Similarly, the order of 31 to 34 of the embodiment of Fig. 6 may be exchanged. Changes of the ordering of method steps may be apparent to the skilled person.

[0221] The embodiments discussed above, in particular, with regard to Figs, la to 8b, may also be implemented as a method, in particular, a method for controlling an electronic device, such as electronic device 100, may be implemented.

[0222] The method can also be implemented as a computer program causing a computer and / or a processor, such as processor 101 discussed above, to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.

[0223] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0224] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0225] Note that the present technology can also be configured as described below.

[0226] (1) An electronic device comprising circuitry for gesture recognition configured to: obtain gesture sensor data of a hand; recognize a finger pointing pattern (3 a) based on the obtained gesture sensor data; realize the recognized finger pointing pattern (3a) as an object pointing in a virtual space (1); recognize a transition between the finger pointing pattern (3a) and the finger pointing pattern (3a) together with a thumb-extension pattern (3b) based on the obtained gesture sensor data; and realize the recognized transition as a confirmation of the object pointing in the virtual space (1).

[0227] (2) The electronic device of (1), wherein the circuitry is further configured to recognize the finger pointing pattern (3a) and / or the transition based on a hand analysis based on the gesture sensor data.

[0228] (3) The electronic device of (2), wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand discrete pose based on the gesture sensor data.

[0229] (4) The electronic device of (3), wherein the circuitry is further configured to detect the hand discrete pose based on a small neural network.

[0230] (5) The electronic device of any one of (2) to (4), wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand key point based on the gesture sensor data.

[0231] (6) The electronic device of (5), wherein the circuitry is further configured to detect the hand key point based on a computer vision technique. (7) The electronic device of any one of (2) to (6), wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand mesh based on the gesture sensor data.

[0232] (8) The electronic device of (7), wherein the circuitry is further configured to detect the hand mesh based on a computer vision technique.

[0233] (9) The electronic device of (6) or (8), wherein the circuitry is further configured to perform the computer vision technique based on temporal filtering.

[0234] (10) The electronic device of (9), wherein the circuitry is further configured to perform temporal filtering based on a pre-determined threshold parameter.

[0235] (11) The electronic device of (6) or (8), wherein the circuitry is further configured to perform the computer vision technique based on a spatial filtering.

[0236] (12) The electronic device of any one of (1) to (11), wherein the circuitry is further configured to recognize the transition based on recognizing the transition of the recognized finger pointing pattern at a first timepoint to the recognized finger pointing pattern together with a thumb - extension pattern at a second timepoint later than the first timepoint, wherein the finger pointing pattern together with the thumb-extension pattern at a second timepoint is based on the recognized finger pointing pattern at a first timepoint.

[0237] (13) The electronic device of any one of (1) to (12), wherein the circuitry is further configured to recognize the finger pointing pattern and / or the transition based on a hand detection based on the gesture sensor data.

[0238] (14) The electronic device of (13), wherein the hand detection includes a detection of a hand region of interest.

[0239] (15) The electronic device of any one of (1) to (14), wherein the circuitry is further configured to recognize the finger pointing pattern and / or the transition based on gesture classification.

[0240] (16) The electronic device of (15), wherein the circuitry is further configured to perform gesture classification based on any one or a combination of hand discrete pose vocabulary, movement vocabulary and transition of hand discrete pose vocabulary.

[0241] (17) The electronic device of any one of (1) to (16), wherein the circuitry is further configured to recognize one or more gesture patterns based on the obtained gesture sensor data and realize the one or more gesture patterns in virtual space. (18) The electronic device of any one of (1) to (17), wherein the circuitry is further configured to realize the object pointing in virtual space based on a virtual close-distance menu, a virtual long-distance menu, a virtual sliding bar, a virtual painting application or a virtual object manipulation.

[0242] (19) A method for gesture recognition comprising: obtaining gesture sensor data of a hand; recognizing a finger pointing pattern based on the obtained gesture sensor data; realizing the recognized finger pointing pattern as an object pointing in a virtual space; recognizing a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data; and realizing the recognized transition as a confirmation of the object pointing in the virtual space.

[0243] (20) The method of (19), wherein the method further comprises recognizing the finger pointing pattern (3a) and / or the transition based on a hand analysis based on the gesture sensor data.

[0244] (21) The method of (20), wherein the method further comprises performing the hand analysis based on a detection of a hand discrete pose based on the gesture sensor data.

[0245] (22) The method of (21), wherein the method further comprises detecting the hand discrete pose based on a small neural network.

[0246] (23) The method of any one of (20) to (22), wherein the method further comprises performing the hand analysis based on a detection of a hand key point based on the gesture sensor data.

[0247] (24) The method of (23), wherein the method further comprises detecting the hand key point based on a computer vision technique.

[0248] (25) The method of any one of (20) to (24), wherein the method further comprises performing the hand analysis based on a detection of a hand mesh based on the gesture sensor data.

[0249] (26) The method of (25), wherein the method further comprises detecting the hand mesh based on a computer vision technique.

[0250] (27) The method of (24) or (26), wherein the method further comprises performing the computer vision technique based on temporal filtering. (28) The method of (27), wherein the method further comprises performing temporal filtering based on a pre-determined threshold parameter.

[0251] (29) The method of (24) or (26), wherein the method further comprises performing the computer vision technique based on a spatial filtering.

[0252] (30) The method of any one of (19) to (29), wherein the method further comprises recognizing the transition based on recognizing the transition of the recognized finger pointing pattern at a first timepoint to the recognized finger pointing pattern together with a thumb-extension pattern at a second timepoint later than the first timepoint, wherein the finger pointing pattern together with the thumb-extension pattern at a second timepoint is based on the recognized finger pointing pattern at a first timepoint.

[0253] (31) The method of any one of (19) to (30), wherein the method further comprises recognizing the finger pointing pattern and / or the transition based on a hand detection based on the gesture sensor data.

[0254] (32) The method of (31), wherein the hand detection includes a detection of a hand region of interest.

[0255] (33) The method of any one of (19) to (32), wherein the method further comprises recognizing the finger pointing pattern and / or the transition based on gesture classification.

[0256] (34) The method of (33), wherein the method further comprises performing gesture classification based on any one or a combination of hand discrete pose vocabulary, movement vocabulary and transition of hand discrete pose vocabulary.

[0257] (35) The method of any one of (19) to (34), wherein the method further comprises recognizing one or more gesture patterns based on the obtained gesture sensor data and realize the one or more gesture patterns in virtual space.

[0258] (36) The method of any one of (19) to (35), wherein the method further comprises realizing the object pointing in virtual space based on a virtual close-distance menu, a virtual longdistance menu, a virtual sliding bar, a virtual painting application or a virtual object manipulation.

[0259] (37) A computer program comprising program code causing a computer to perform the method according to anyone of (19) to (36), when being carried out on a computer. (38) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (19) to (37) to be performed.

Claims

CLAIMS1. An electronic device comprising circuitry for gesture recognition configured to: obtain gesture sensor data of a hand; recognize a finger pointing pattern based on the obtained gesture sensor data; realize the recognized finger pointing pattern as an object pointing in a virtual space; recognize a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data; and realize the recognized transition as a confirmation of the object pointing in the virtual space.

2. The electronic device of claim 1, wherein the circuitry is further configured to recognize the finger pointing pattern and / or the transition based on a hand analysis based on the gesture sensor data.

3. The electronic device of claim 2, wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand discrete pose based on the gesture sensor data.

4. The electronic device of claim 3, wherein the circuitry is further configured to detect the hand discrete pose based on a small neural network.

5. The electronic device of claim 2, wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand key point based on the gesture sensor data.

6. The electronic device of claim 5, wherein the circuitry is further configured to detect the hand key point based on a computer vision technique.

7. The electronic device of claim 2, wherein the circuitry is further configured to perform the hand analysis based on a detection of a hand mesh based on the gesture sensor data.

8. The electronic device of claim 7, wherein the circuitry is further configured to detect the hand mesh based on a computer vision technique.

9. The electronic device of claim 6, wherein the circuitry is further configured to perform the computer vision technique based on temporal filtering.

10. The electronic device of claim 8, wherein the circuitry is further configured to perform the computer vision technique based on temporal filtering.

11. The electronic device of claim 9, wherein the circuitry is further configured to perform temporal filtering based on a pre-determined threshold parameter.

12. The electronic device of claim 10, wherein the circuitry is further configured to perform temporal filtering based on a pre-determined threshold parameter.

13. The electronic device of claim 6, wherein the circuitry is further configured to perform the computer vision technique based on a spatial filtering.

14. The electronic device of claim 8, wherein the circuitry is further configured to perform the computer vision technique based on a spatial filtering.

15. The electronic device of claim 1, wherein the circuitry is further configured to recognize the transition based on recognizing the transition of the recognized finger pointing pattern at a first timepoint to the recognized finger pointing pattern together with a thumb-extension pattern at a second timepoint later than the first timepoint, wherein the finger pointing pattern together with the thumb-extension pattern at a second timepoint is based on the recognized finger pointing pattern at a first timepoint.

16. The electronic device of claim 1, wherein the circuitry is further configured to recognize the finger pointing pattern and / or the transition based on a hand detection based on the gesture sensor data.

17. The electronic device of claim 16, wherein the hand detection includes a detection of a hand region of interest.

18. The electronic device of claim 1, wherein the circuitry is further configured to recognize the finger pointing pattern and / or the transition based on gesture classification.

19. The electronic device of claim 18, wherein the circuitry is further configured to perform gesture classification based on any one or a combination of hand discrete pose vocabulary, movement vocabulary and transition of hand discrete pose vocabulary.

20. The electronic device of claim 1, wherein the circuitry is further configured to recognize one or more gesture patterns based on the obtained gesture sensor data and realize the one or more gesture patterns in virtual space.

21. The electronic device of claim 1, wherein the circuitry is further configured to realize the object pointing in virtual space based on a virtual close-distance menu, a virtual long-distancemenu, a virtual sliding bar, a virtual painting application or a virtual object manipulation application.

22. A method for gesture recognition comprising: obtaining gesture sensor data of a hand; recognizing a finger pointing pattern based on the obtained gesture sensor data; realizing the recognized finger pointing pattern as an object pointing in a virtual space; recognizing a transition between the finger pointing pattern and the finger pointing pattern together with a thumb-extension pattern based on the obtained gesture sensor data; and realizing the recognized transition as a confirmation of the object pointing in the virtual space.

Images