Hand gesture detection using discrete values of gesture language
Through the combination of discrete value architecture and SSVM and FSM, hand posture detection is simplified, the problems of high complexity and computational resources are solved in the existing technology, and efficient and accurate hand posture detection is achieved, suitable for multimodal users Interaction.
Patent Information
- Application Number
- CN202110375375.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2015-12-31
- Filing Date
- 2016-12-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2036-12-22
AI Technical Summary
The existing hand posture detection technology has high complexity, intensive computing resources and insufficient accuracy, making it difficult to widely use in human-computer interfaces.
Using a discrete value architecture, the mobile image of the user's hand is analyzed by predefined discrete hand values of the hand posture, and the optimization process is performed using structured support vector machine (SSVM) and finite state machine (FSM) to simplify hand posture detection. .
It significantly reduces the running time calculation load, reduces the need for hand bone modeling, improves detection efficiency and accuracy, and is suitable for multimodal user interaction detection.
Smart Images

Figure CN113093909B_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with international application number PCT / US2016 / 068160, international application date December 22, 2016, entering the Chinese national stage on June 29, 2018, Chinese national application number 201680077619.X, and invention name “Detection of hand gestures using discrete values of gesture language”. Background Art
[0002] With the development of computerized environments, the use of human-machine interfaces (HMIs) has increased significantly. There is an increasing need for more natural human-machine user interface (NUI) methods (such as, for example, voice and / or gaze) and more specifically hand gesture interactions to replace and / or supplement traditional HMIs, such as, for example, keyboards, pointing devices, and / or touch interfaces. This can be used to, for example, eliminate and / or reduce the need for intermediary devices (such as keyboards and / or pointing devices), support hands-free interaction, improve accessibility for (multiple) disabled people, and / or provide a multimodal interactive environment. There may be current solutions for identifying and / or recognizing (multiple) hand gestures, but most of them are immature, inaccurate, and / or highly complex, and require a high degree of computing resources for extensive computer vision processing and / or machine learning. Such technology may rely on entire hand skeletal joints and / or complex machine learning algorithms to detect and / or classify hand gestures, which may make such implementations expensive and unattractive for integration, thereby hindering their adoption for a wide range of uses. Summary of the invention
[0003] According to some embodiments of the present disclosure, a system and method for detecting a user's hand gestures is provided by analyzing one or more images depicting the movement of a user's (multiple) hands with respect to a plurality of predefined hand gestures to estimate a runtime sequence of one or more runtime hand data sets to identify the runtime sequence as a valid hand gesture. The predefined hand gestures mentioned below throughout the present disclosure refer to predefined hand gesture representations that simulate the corresponding hand gestures of (multiple) hands. In the same manner, the definition, creation, construction and / or generation of hand gestures, hand postures and / or hand movements mentioned below throughout the present disclosure refer to the definition, creation, construction and / or generation of representations of hand gestures, hand postures and hand movements that respectively simulate the corresponding hand gestures, postures and movements of (multiple) hands. The predefined hand gestures are constructed using a discrete value architecture, wherein each predefined hand gesture may include one or more hand gestures and / or hand movements. Each of the hand poses and hand motions is represented as a predefined hand feature record defined by discrete hand values, each discrete hand value indicating the state of a corresponding hand feature (characteristic) of the depicted hand, such as, for example, various finger and / or hand states and / or actions. Similarly, each of the one or more runtime hand data sets may include one or more hand poses and / or hand motions each defined by the runtime hand data set, the runtime hand data set including one or more discrete hand part value scores, each discrete hand part value score indicating the current state of the corresponding hand feature. By quantizing the continuous values of the one or more hand features, the continuous values may be represented by the discrete hand values to support the discrete architecture of the hand gesture detection process. The one or more discrete hand part value scores are inferred by analyzing one or more images depicting the user's hand. Each runtime hand data set is submitted to one or more structured support vector machine (SSVM) functions together with the one or more predefined hand feature records to generate multiple estimation items for the runtime hand data set with respect to each predefined hand feature record. The runtime hand data set can then be estimated to conform to one of the predefined hand feature records by parsing the weighted calculation using an estimate term having a discrete hand value score and a discrete hand value. Each predefined hand gesture is optionally defined by one or more sequential logic models and can be represented by a finite state machine (FSM) that records transitions between (multiple) hand postures and / or (multiple) hand movements. An optimization process for identifying the best predefined hand gesture that best describes the described runtime sequence can be accomplished by dynamic programming that uses, for example, Viterbi decoding to enhance the sequential logic model of the predefined hand gesture FSM using one or more scoring functions on one or more sequences within the FSM.Optionally, hand gesture detection is not limited to specific predefined hand gestures, but can detect an entire class of hand gestures consisting of hand gestures and / or hand movements, where hand gestures and / or hand movements are constructed using a discrete hand value architecture.
[0004] According to some embodiments of the present disclosure, a system and method for detecting multimodal user interaction actions are provided, and the multimodal user interaction actions may include one or more actions, such as hand gestures, head gestures, body gestures, facial expressions, gaze directions, text, voice and / or tactile input. The detection of multimodal actions can be performed for the user using one or more computerized devices attached to one or more capture devices. The captured multimodal user interactions can be processed to identify one or more multimodal actions and optionally initiate one or more application functions, actions, operations and / or commands that can be associated with the detected (multiple) multimodal actions.
[0005] Detecting one or more hand gestures of a user as depicted in (multiple) images by estimating scores of discrete hand values of a runtime hand dataset compared to discrete hand values recorded by predefined hand gesture features can significantly reduce the computational load required at runtime. The detection process is simplified to an optimization process over multiple discrete hand values of the runtime hand dataset and those predefined hand gestures where the optimization is performed on binary representations of the discrete hand values. In addition, due to the discrete nature of the detection process dealing with discrete hand values, for each hand feature, a limited number of possible values (e.g., 5, 10, and / or 20) can be effective, avoiding the need to fully represent the hand skeleton, thereby further reducing computer vision and modeling resources.
[0006] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the art. Although methods and materials similar or equivalent to the methods and materials described herein can be used in the practice or testing of the embodiments of the present disclosure, exemplary methods and / or materials are described below. If a conflict occurs, the patent specification including definitions will control. In addition, these materials, methods and examples are illustrative only and are not intended to be limiting. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Some embodiments of the present disclosure are described herein with reference to the accompanying drawings only by way of example. Now with specific reference to the accompanying drawings, it is emphasized that the details shown are by way of example and for the purpose of illustrative discussion of the embodiments of the present disclosure. In this regard, the description in conjunction with the accompanying drawings makes it clear to those skilled in the art how to practice the embodiments of the present disclosure.
[0008] In the attached picture:
[0009] Figure 1 is a flow chart of an exemplary process for detecting a hand gesture depicted by image(s) according to some embodiments of the present disclosure;
[0010] Figure 2 is a block diagram of the building blocks of exemplary hand gestures according to some embodiments of the present disclosure;
[0011] Figure 3 is a schematic diagram of an exemplary FSM representing a hand gesture sequential logic model as a sequence of hand motions and hand postures according to some embodiments of the present disclosure;
[0012] Figure 4 is a schematic diagram of an exemplary discrete representation of hand gestures according to some embodiments of the present disclosure;
[0013] Figure 5 is a schematic diagram of an exemplary pinch hand gesture discrete representation according to some embodiments of the present disclosure;
[0014] Figure 6 is a schematic diagram of an exemplary discrete representation of hand motion according to some embodiments of the present disclosure;
[0015] Figure 7 is a schematic diagram of an exemplary semicircular basic hand motion discrete representation according to some embodiments of the present disclosure;
[0016] Figure 8 is a block diagram of building blocks of an exemplary slingshot hand gesture according to some embodiments of the present disclosure;
[0017] Fig. 9 is a schematic diagram of an exemplary system for detecting a hand gesture depicted by image(s) according to some embodiments of the present disclosure;
[0018] Fig.10 is a block diagram of building blocks of exemplary multimodal actions according to some embodiments of the present disclosure;
[0019] Fig.11 is a schematic diagram of a multimodal FSM defining a multimodal action according to some embodiments of the present disclosure; and
[0020] Fig.12 is a schematic diagram of exemplary multimodal actions according to some embodiments of the present disclosure. DETAILED DESCRIPTION
[0021] According to some embodiments of the present disclosure, a system and method for detecting a user's hand gestures by analyzing one or more images depicting the movement of a user's (multiple) hands with respect to a plurality of predefined hand gestures to estimate a runtime sequence of one or more runtime hand data sets to identify the runtime sequence as a valid hand gesture is provided. The detection of a hand gesture performed by a user and depicted in one or more images is based on a discrete architecture, in which discrete hand values are used as building blocks to define hand gestures and / or hand motions that constitute a hand gesture. Each hand gesture may include one or more hand gestures and / or hand motions, each of which is represented as a predefined hand feature record that may be a hand gesture feature record or a hand motion feature record, respectively. Each hand feature record is defined by a unique set of discrete hand values, each of which indicates the state of a corresponding one of a plurality of hand features (characteristics) of the depicted hand (such as, for example, various finger and / or hand states and / or motions). The hand features include hand gesture features and hand motion features that are each associated with one or more discrete gesture values and discrete motion values, respectively. Each predefined hand gesture can be represented as a sequential logic model including one or more hand gestures and / or hand motions, each hand gesture and / or hand motion is represented by a hand feature record, which is in turn defined by a plurality of discrete hand values. In the same manner, each of the one or more images of one or more runtime hand data sets that capture a runtime sequence of hand movements of the user is analyzed to identify it as a hand gesture or hand motion, and further infer one or more discrete hand values used to construct each runtime hand data set. The discrete hand values identified for each runtime hand data set are referred to as discrete hand value scores, and may include discrete gesture values or discrete motion values, each discrete gesture value or discrete motion value indicating the current (runtime) value of a corresponding one of a plurality of hand features. By quantizing the continuous values of one or more hand features, the continuous values may be represented by discrete hand values to support the discrete architecture of the hand gesture detection process. Since the predefined hand poses (representations) and / or hand motions (representations) and the runtime hand data set are defined by multiple discrete hand values, estimating the runtime sequence as one of the predefined hand poses is essentially an optimization problem, where the best predefined hand pose that best describes the runtime sequence is selected. Each of the one or more runtime hand data sets is submitted to one or more SSVM functions together with one or more predefined hand feature records to generate multiple estimation terms for the runtime hand data set with respect to each predefined hand feature record. The estimation terms include singleton terms and paired terms. The singleton terms define the correlation between each runtime hand data set and one predefined hand feature record. The paired terms define the correlation between each runtime hand data set and two (current and previous) predefined hand feature records.Then, the runtime sequence can be estimated to conform to one of the predefined hand gestures by parsing the best matching predefined hand feature records for each of the one or more runtime hand data sets. Singletons and / or paired terms can be generated by simulating discrete hand values of predefined hand postures and / or hand motions on discrete hand value scores of each runtime hand data set. The predefined hand feature records can be expressed as Boolean formulas, such as conjunctive normal form (CNF). One or more SSVM functions can apply one or more parameter functions to generate singletons and / or paired terms. One or more SSVM functions can be trained offline to identify the most accurate estimation term to be associated with each predefined hand feature record. Each of the one or more sequential logic models defining one or more hand gestures can be represented by a FSM, where each hand feature record (hand posture or hand motion) is a state and the FSM records the transitions between (multiple) hand postures and / or (multiple) hand motions. Before initiating the optimization process, the FSM representing one or more predefined hand gestures can be enhanced using one or more scoring functions on the sequence within the FSM to allow efficient and accurate optimization, each of the one or more sequences within the FSM representing a hand gesture. The optimization process can be performed by dynamic programming, which can utilize, for example, Viterbi decoding performed on one or more scoring functions using the generated singleton terms and / or paired terms. Once completed, the optimization process produces the best predefined hand gesture that best matches the runtime sequence of the movement of the user's hand depicted in one or more images. Optionally, one or more weights are assigned to each estimated term to improve the optimization process. The one or more weights can be calculated by, for example, one or more SSVM functions, which can be trained to select the best matching weight for each predefined hand feature record. Optionally, the runtime sequence of the runtime hand data set can be estimated as a sequential logic model of a hand gesture, which is not predefined, but may be constructed using a discrete architecture discrete hand value of each indicating the value of a corresponding hand feature. Optionally, one or more SSVM functions are specifically used to identify the runtime sequence as one of a plurality of registered hand gestures. One or more registered hand gestures may be registered based on the context of the user's activity, for example, one or more predefined hand gestures may be registered (associated) to a particular application. If the particular application is active during the detection of the user's hand movements depicted in the runtime sequence, the optimization process only considers the registered hand gestures to select the best hand gesture that best matches the runtime sequence.
[0022] Basing the detection of the user's hand pose on a discrete architecture can significantly reduce the runtime computational load because the need to model the entire hand skeleton is avoided by employing an optimization process over multiple discrete hand values (predefined) and scoring the discrete hand values that present a limited number of possible values (at runtime). In addition, the optimization is based on estimates generated using binary representations of hand posture and / or hand motion, which further reduces the computational complexity because binary computations require significantly fewer processing resources compared to a continuous representation of the skeletal hand. Specializing one or more SSVM functions can reduce further computational load at runtime because a limited number of predefined hand poses are considered during the optimization process, making the optimization process simpler and faster.
[0023] According to some embodiments of the present disclosure, a system and method for detecting multimodal user interaction actions are provided, wherein the multimodal user interaction actions include one or more actions, such as hand gestures, head gestures, body gestures, facial expressions, gaze directions, text, voice and / or tactile input. The detection of multimodal actions can be performed for a user using one or more computerized devices, such as computers, mobile devices, computerized machines and / or computerized devices equipped with and / or attached to one or more user interaction capture devices, such as imaging devices, recorders, tactile receiving devices, text input devices (such as, for example, keyboards, digital pens, touchpads, etc.), touch screens, digital surfaces and / or (multiple) indicating devices (such as, for example, mice, trackballs, game controllers, joysticks, etc.). The captured multimodal user interactions can be processed to identify one or more multimodal actions and optionally initiate one or more application functions, actions, operations and / or commands that can be associated with the detected (multiple) multimodal actions. Multimodal actions may be classified and / or identified during runtime by employing one or more detection techniques to identify one or more user actions that constitute the multimodal action (e.g., text input parsing, pointing device input analysis, speech analysis, tactile input analysis, and specifically hand gesture detection).
[0024] Before explaining at least one embodiment of the exemplary embodiments in detail, it should be understood that the present disclosure is not necessarily limited in its application to the details of construction and arrangement of components and / or methods given in the following description and / or illustrated in the drawings and / or examples. The present disclosure is capable of other embodiments or of being practiced or implemented in various ways.
[0025] Now referring to the accompanying drawings, Figure 11 is a flowchart of an exemplary process for detecting hand gestures depicted by (multiple) images according to some embodiments of the present disclosure. The exemplary hand gesture detection process 100 presents the detection process of the movement of the user's hand depicted in one or more images as a predefined hand gesture. As shown in 110, the exemplary process 100 begins by receiving a plurality of predefined hand gestures, each of which is represented as a sequential logic model. Each sequential logic model can map one or more hand gestures and / or hand motions, each represented by a unique one of a plurality of hand feature records. The hand feature records can be hand gesture feature records or hand motion feature records, each of which is associated with a hand gesture and a hand motion, respectively. Each hand feature record is defined by one or more of a plurality of discrete hand values, each of which indicates the state of a corresponding hand feature, such as a palm posture, a (multiple) finger posture, a palm motion, and / or a (multiple) finger motion. Similarly, discrete hand values may be discrete posture values and / or discrete motion values, each discrete posture value and / or discrete motion value indicating a state of a corresponding hand feature which may be a hand posture feature and / or a hand motion feature.
[0026] As shown at 120, a runtime sequence including one or more of a plurality of runtime hand data sets is received. Each runtime hand data set is defined by a plurality of discrete hand value scores, each discrete hand value score indicating a current state of a corresponding hand feature of a user's moving hand. The plurality of discrete hand value scores are inferred and / or calculated by analyzing one or more of a plurality of images depicting the user's moving hand.
[0027] Before further describing the hand gesture detection process 100, it is important to understand the discrete architecture used to define hand gestures, hand postures and / or hand motions. The discrete hand features that define the hand feature records and the discrete hand feature scores that define the runtime hand data set all refer to the same discrete hand values presented herein. The discrete hand values can be discrete posture values or discrete motion values. The continuous values can be represented by discrete hand values to support the discrete architecture of the hand gesture detection process by quantizing the continuous values of one or more hand features.
[0028] Reference now Figure 2 , Figure 2 200 is a block diagram of the building blocks of an exemplary hand gesture according to some embodiments of the present disclosure. Diagram 200 depicts several configurations of an exemplary hand gesture 210. Hand gesture 210 may be created by one or more possible configurations, such as:
[0029] (a) Hand gesture 210 may be composed of hand posture 250 .
[0030] (b) Hand gesture 210 may be a combination and / or sequence of two hand gestures 250 .
[0031] (c) Hand gesture 210 can be a combination and / or sequence of hand posture 250 and hand movement 260.
[0032] (d) The (first) hand gesture 210 may be a combination and / or sequence of the second hand gesture 210 and the hand gesture 250. The second hand gesture may be the same hand gesture as the first hand gesture, or it may be a different hand gesture 210.
[0033] (e) The (first) hand gesture 210 may be a combination and / or sequence of a second hand gesture 210 and a hand motion 260. The second hand gesture 210 may be the same hand gesture as the first hand gesture, or it may be a different hand gesture 210.
[0034] Hand gestures 210 can be created by multiple iterations of constructions (d) and / or (e) above. Each hand gesture 210 is constructed as a unique combination and / or sequence represented by a hand gesture sequence 201, which includes one or more of hand postures 250, hand movements 260, and / or hand gestures 210. Each hand gesture 210 begins and ends with an idle state 280, which can be a virtual state that identifies the beginning and / or end of a unique hand gesture sequence 201 of hand gestures 210. Hand gesture sequence 201 can be considered as a sequential logic model that describes hand gestures 210.
[0035] Reference now Figure 3 , Figure 3300 is a schematic diagram of an exemplary FSM that represents a sequential logic model of a hand gesture as a sequence of hand motions and hand gestures according to some embodiments of the present disclosure. Diagram 300 depicts a FSM 301 that can represent a sequential logic model of a hand gesture sequence (such as hand gesture sequence 201). FSM 301 begins at a start point 310 indicating a start state of FSM 301. FSM 301 may include one or more states 302A, 302B, 302C to 302N. First state 302A is a hand gesture (such as hand gesture 250) that can describe the beginning of a sequence representing a hand gesture (such as hand gesture 210). Each of subsequent states 302B, 302C to 302N may be a hand gesture 250 or a hand motion such as hand motion 260. FSM 301 ends at a FSM end point 320. Start point 310 and / or end point 320 may be defined as a virtual idle state such as idle state 280. During a detection process of hand gestures 210 (such as hand gesture detection process 100), FSM 301 can be enhanced using one or more scoring functions on one or more sequences within FSM 301 to allow processing of one or more runtime hand data sets included in a runtime sequence depicting a user's moving hand compared to each of the hand gestures 250 and / or hand motions 260 that include each sequence in FSM 301. Each hand gesture 250 and / or hand motion 260 is represented by a corresponding hand gesture feature record and hand motion feature record. Using FSM 301 to represent each of a plurality of hand gesture sequences 201 that are each associated with one of the hand gestures 210 allows for significantly simplified estimation of a runtime sequence of a runtime hand data set to detect the runtime sequence as one of the predefined hand gestures (such as hand gestures 210) at runtime using dynamic programming by applying, for example, Viterbi decoding on the (multiple) scoring functions of the enhanced FSM 301.
[0036] Reference now Figure 4 , Figure 4400 is a schematic diagram of an exemplary hand gesture discrete representation according to some embodiments of the present disclosure. Diagram 400 depicts an exemplary hand gesture 250 representation as a hand gesture feature record 251 that includes one or more hand gesture features 410, 420, 430, and 440. A hand gesture feature record 251, which is referenced within the general term of a hand feature record, may be represented by one or more of discrete gesture values 411, 421, 431, and / or 441, each of which indicates a state (value) of a corresponding hand gesture feature 410, 420, 430, and / or 440 of an associated hand gesture in hand gestures 250. A combination of one or more discrete gesture values 411, 421, 431, and / or 441 of corresponding hand gesture features 410, 420, 430, and 440 defined by a hand gesture feature record 251 identifies a specific gesture in hand gestures 250 that may be detected by analyzing one or more images depicting movement of a user's hand. The continuous values of one or more hand posture features 410, 420, 430, and / or 440 may be represented by discrete posture values 411, 421, 431, and / or 441 by quantizing the continuous values. The hand posture feature record 251 may be represented as, for example, a feature vector, a feature matrix, and / or a feature table. The hand posture feature record 251 may include values of one or more of the following exemplary hand posture features:
[0037] Palm posture features - One or more palm posture features 410 include, for example, hand selection, palm direction, palm rotation, and / or hand position. Hand selection can identify which hand is active and can include discrete posture values 411, such as, for example, right, left, both, and / or any. Palm direction can define the direction the palm of the active hand faces, and can include discrete posture values 411, such as, for example, left, right, up, down, forward, and / or backward. Palm rotation can define the rotation state of the palm of the active hand, and can include discrete posture values 411, such as, for example, left, right, up, down, forward, and / or backward. Hand position can identify the spatial position of the active hand in space, and can include discrete posture values 411, such as the center of the field of view (FOV), the right side of the FOV, the left side of the FOV, the top of the FOV, the bottom of the FOV, the front of the FOV, and / or the back of the FOV. For example, where the FOV is the visible space of an imaging device that monitors the movement of the user's hand. Optionally, the hand position is identified with respect to a fixed object (e.g., a keyboard and / or pointing device) present in the FOV such that the hand position can be defined by discrete posture values 111, such as, for example, above the keyboard, below the keyboard, to the right of the keyboard, and / or to the left of the keyboard.
[0038] Finger flex features - one or more finger flex features 420 defined for each finger. For example, a finger feature 420 may be a bend and / or curve state that may include discrete posture values 421, such as stretched, folded, and / or open, for example, represented by 0, 1, and 2. Each finger (thumb, index finger, middle finger, ring finger, and / or pinky) is assigned one or more specific finger features, such as {thumb, middle finger, ring finger, pinky} for the {folded} state and {index finger} for the {stretched} state.
[0039] Finger contact condition features - For each finger, one or more finger contact features 430 are defined. A contact feature may define a touch condition for any two or more fingers and / or touch types, and may include discrete posture values 431 such as, for example, no touch, fingertip, and / or full touch.
[0040] Finger relative position condition features - One or more finger relative position features 440 are defined for each finger. Each finger relative position condition feature 440 may define the relative position of one finger relative to another finger. The finger relative position features 440 may include discrete posture values 441, such as, for example, one or more fingers are located to the left, right, above, below, inward, outward, forward, and / or backward relative to another one or more fingers.
[0041] Each of the hand poses 250 is defined by a unique one of the hand pose feature records 251, which may be a combination and / or sequence of one or more discrete pose values 411, 421, 431, and / or 441, each of which indicates the value of the corresponding hand pose feature 410, 420, 430, and / or 440. The hand pose feature record 251 may include only some (but not all) of the discrete pose values 411, 421, 431, and / or 441, while the other discrete pose values 411, 421, 431, and / or 441 not included are free. For example, the hand pose feature record 251 may define a specific state of the fingers (e.g., discrete pose values 421, 431, and / or 441) while the orientation of the palm is unspecified (e.g., discrete pose value 411). In this case, the hand pose 250 is detected at runtime by identifying the finger states defined by the hand pose feature record 251 when the hand is facing any direction. Detection of one or more hand poses 250 is simplified because discrete pose values 411, 421, 431, and / or 441 can be easily identified because there is a finite, limited number of possible states for each of the hand pose features 410, 420, 430, and / or 440, avoiding the need for hand skeleton modeling, thereby reducing the level of computer vision processing. The discrete representation of the hand pose features 410, 420, 430, and / or 440 may not be limited to discrete values. The continuous values of one or more hand features 410, 420, 430, and / or 440 may be represented by discrete pose values 411, 421, 431, and / or 441, respectively, by quantizing the continuous values. For example, a palm rotation palm pose feature may be defined with eight discrete motion values 411 - 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° to quantize the entire rotation range of 0° to 360°.
[0042] Reference now Figure 5 , Figure 5 500 is a schematic diagram of an exemplary pinch hand gesture discrete representation according to some embodiments of the present disclosure. Diagram 500 depicts an exemplary pinch hand gesture 250A representation of a pinch gesture feature record 251A including discrete gesture values (such as discrete gesture values 411, 421, 431, and / or 441), each discrete gesture value indicating a value of a corresponding gesture feature (such as gesture features 410, 420, 430, and / or 440). The pinch hand gesture 250A visualized by image capture 501 is defined by some of the plurality of discrete gesture values 411, 421, 431, and 441 as follows:
[0043] • The hand selection feature 410A is assigned a discrete posture value 41 IA to indicate that the left hand is active.
[0044] • Palm orientation feature 410B is assigned discrete posture value 41 IB to indicate that the palm of the active hand is facing forward.
[0045] • Finger bend feature 420A is assigned discrete posture value 421A and discrete bend value 421B to indicate that the thumb and index finger are folded.
[0046] • Finger bend feature 420B is assigned discrete posture value 421C and discrete posture value 421D to indicate that the middle finger, ring finger, and pinky finger are open.
[0047] • The finger contact condition feature 430A is assigned a discrete posture value 431A to indicate that the thumb and index finger are touching at their fingertips.
[0048] • Finger relative position feature 440A is assigned discrete pose value 441A, discrete pose value 441B, and discrete pose value 441C to indicate that the index finger is located above the thumb.
[0049] As seen above, pinch hand gesture 250A is uniquely defined by pinch feature gesture feature record 251A including discrete gesture values 411A, 411B, 421A, 421B, 421C, 421D, 431A, 431B, 441A, 441B, and 441C corresponding to gesture features 410A, 410B, 420A, 420B, 430A, and 440A, respectively. Similarly, additional hand gestures 250 may be defined.
[0050] Reference now Figure 6 , Figure 6 600 is a schematic diagram of an exemplary hand motion discrete representation according to some embodiments of the present disclosure. Diagram 600 depicts each of a plurality of exemplary hand motion 260 representations as a hand motion feature record 261. The hand motion feature record 261, which is referenced in the general term of a hand feature record, may include one or more hand motion features 610 and 620. Each of the hand motion features 610 and 620 may be assigned one or more discrete motion values 611 and / or 621 indicating the state (value) of the corresponding hand motion feature 610 and / or 620 of the associated hand motion in the hand motion 260. By quantizing the continuous values, the continuous values of the one or more hand motion features 610 and / or 620 may be represented by the discrete motion values 611 and / or 621. The hand motion feature record 261 identifies a specific motion of a hand and / or (multiple) fingers that can be detected by analyzing one or more images depicting the movement of the user's hand. The hand motion feature record 261 may be represented as, for example, a feature vector, a feature matrix, and / or a feature table. The hand motion feature record 261 may include one or more of the following exemplary hand motion features:
[0051] ·Motion attribute features - One or more motion attribute features 610 may include, for example, motion size, motion speed, and / or motion location. The motion size may identify the size (range) of the motion and may include discrete motion values 611, such as, for example, small, normal, and / or large. The motion speed may define the speed of the motion and may include discrete motion values 611, such as, for example, slow, normal, fast, and / or sudden. The motion location may identify the spatial location where the motion is performed and may include discrete motion values 611, such as, for example, the center of the FOV, the right side of the FOV, the left side of the FOV, the top of the FOV, the bottom of the FOV, the front of the FOV, and / or the back of the FOV. Optionally, the hand position is identified relative to a fixed object present in the FOV, such as a keyboard and / or a pointing device, so that the hand position may include discrete motion values 611, such as, for example, above the keyboard, below the keyboard, to the right of the keyboard, and / or to the left of the keyboard.
[0052] ·Motion script features - One or more motion script features 620 can define the actual motion performed. The motion script values 620 can include, for example, a motion direction, a motion start point, a motion end point, and / or a predefined curve shape. The motion direction feature 620 can include discrete motion values 621, such as, for example, up, down, left to right, right to left, diagonal left up, diagonal right up, diagonal left down, diagonal right down, clockwise arc right up, clockwise arc right down, clockwise arc left up, clockwise arc left down, counterclockwise arc right up, counterclockwise arc right down, counterclockwise arc left up, and / or counterclockwise arc left down. Optionally, the motion curve shape can include, for example, an at sign (@), an infinity sign (∞), a numeric symbol, an alphabetic symbol, etc. Optionally, additional one or more curve shapes (e.g., a check mark, a bill request, etc.) can be defined by the hand motion feature record 261. Each motion script feature 620 is defined for a two-dimensional (2D) plane, however each motion script feature 620 may be transposed to depict another 2D plane, such as XY, XZ and / or YZ. Optionally, the motion script feature 620 defines three-dimensional (3D) motion and / or curves using a 3D image data representation format.
[0053] Each of the hand motions 260 is defined by a unique one of the hand motion feature records 261, which may be a combination and / or sequence of one or more discrete motion values 611 and / or 621, each discrete motion value 611 and / or 621 indicating the value of the corresponding hand motion feature 610 and / or 620. The use of discrete motion values 621 and / or 621 allows for simple detection of the hand motion 260, because there are a limited number of discrete motion values 611 and / or 621 to be analyzed and estimated, avoiding the need for modeling the entire hand skeleton, thereby reducing the level of computer vision processing. For example, the motion speed feature included in the hand motion attribute feature 610 may include up to four discrete motion values 611 - slow, normal, fast and sudden. Similarly, additional hand motions 260 may be defined. The discrete representation of the hand motion features 610 and / or 620 may not be limited to discrete values, and the continuous values of one or more hand motion features 610 and / or 620 may be represented by discrete motion values 611 and / or 621, respectively, by quantizing the continuous values. For example, the motion speed motion attribute feature can be defined by 6 discrete motion values 611 - 5m / s (meters / second), 10m / s, 15m / s, 20m / s, 25m / s and 30m / s to quantify the motion speed of a normal person's hand from 0m / s to 30m / s.
[0054] Reference now Figure 7 , Figure 7 700 depicts an exemplary upper left-to-right semicircle hand motion 260A construction of an upper left-to-right semicircle motion feature record 261A including discrete motion values (such as discrete motion values 611 and / or 621), each discrete motion value 611 and / or 621 indicating the value of a corresponding hand motion feature (such as hand motion features 610 and / or 620). The upper left-to-right semicircle hand motion 260A visualized by image captures 701A, 701B, and 701C is created using some of the plurality of discrete motion values 611 and 621 as follows:
[0055] • The motion magnitude feature 610A is assigned a discrete motion value 611A to indicate that the motion magnitude is normal.
[0056] • The motion speed feature 610B is assigned a discrete motion value 611B to indicate that the motion speed is normal.
[0057] • The motion position feature 610C is assigned a discrete motion value 611C to indicate that the motion is performed above the keyboard.
[0058] • The first motion script feature 620A is assigned a discrete motion value 621A to indicate the clockwise arc to the left and upward motion shape presented by the image capture 701B.
[0059] • The second motion script feature 620B is assigned a discrete motion value 621B to indicate the clockwise arc downward to the left motion shape presented by the image capture 701C.
[0060] As seen above, the left-to-right upper semicircle hand motion 260A is uniquely defined by the left-to-right upper semicircle motion feature record 261A including discrete motion values 611A, 611B, 611C, 621A, and 621B corresponding to hand motion features 610A, 610B, 610C, 620A, and 620B, respectively. Similarly, additional hand and / or finger(s) motions may be defined.
[0061] Reference now Figure 8 , Figure 8 800 depicts a discrete representation of an exemplary slingshot hand gesture 210A as a plurality of hand gestures such as hand gesture 250 and hand motions such as hand motion 260. The slingshot hand gesture 210A visualized by image captures 801A, 801B, 801C, and 801D is constructed from a combination and / or sequence of idle states such as virtual idle state 280, no pinch hand gesture 250B, pinch hand gestures such as pinch hand gesture 250A, retract hand motion 260B, and ending idle states such as virtual idle state 280. The sequence of the slingshot hand gesture 210A is as follows:
[0062] • Virtual idle state 280 defines the starting state and / or point of the sequence of slingshot hand gestures 210A.
[0063] • No-pinch hand gesture 250B, defined by hand gesture feature record 251B, represents a no-pinch action as depicted in image capture 801A.
[0064] • A pinch hand gesture 250A defined by a hand gesture feature record 251A, in which a pinch action is identified, as depicted in image capture 801B.
[0065] • A retracting hand movement 260B defined by hand movement feature record 261B, in which the hand moves backwards, as depicted in image capture 801C.
[0066] • The no-pinch hand gesture 250B defined by the hand gesture feature record 251B indicates that the pinch gesture is released and identified as a no-pinch action, as depicted in image capture 801D.
[0067] • Idle state 280 defines the ending state and / or point of the sequence of slingshot hand gesture 210A.
[0068] The sequence of slingshot hand gestures 210A as described above is represented by a unique slingshot hand gesture sequence 201A that can be considered as a sequential logic model associated with the slingshot hand gesture 210A. For each of the hand gestures 250A, 250B, and hand motion 260B, only relevant discrete gesture and / or motion values can be defined. For example, for the no-pinch gesture 250B, the no-pinch hand gesture feature record 251B is defined by the hand selection discrete gesture value 411 (left), the finger bending discrete gesture value 421 (stretch) and the finger contact discrete gesture value 431 (not touched). Other discrete gesture values that are not related to the distinction between the no-pinch hand gesture 250B and the pinch hand gesture 250A are free and unspecified. Specifying only relevant discrete gestures and / or motion values allows several degrees of freedom in the expression of hand gestures 250 and / or hand motions 260 performed by different one or more users at runtime. This means that each of the one or more users may perform hand gestures 250 and / or hand movements 260 slightly differently at runtime, yet they are still detected as being the same.
[0069] Reference again Figure 1 As shown at 130, one or more runtime hand data sets included in the received runtime sequence depicting the user's moving hands are submitted to one or more SSVM functions along with a plurality of predefined hand feature records (hand posture feature records and / or hand motion feature records). The one or more SSVM functions generate a plurality of estimation terms that are later used to estimate the runtime sequence as one of a plurality of hand gestures (such as hand gesture 210).
[0070] Conventions and Notation:
[0071] 1)x t ∈R d ;x 1:t =(x 1 ,.....,x t )
[0072] x t is defined as a set of discrete hand value scores at time t for a particular runtime hand dataset, where each of the d scores corresponds to a specific hand feature, e.g., the index finger is pointing forward, the middle finger is touching the thumb, the ring finger is folded, etc. For simplicity, we use the notation x 1:t To describe (x 1 ,.....,x t ).
[0073] 2)y t ∈Y t ;y 1:t =(y 1 ,.....,y t )
[0074] y t is defined as the set of predefined hand poses Y valid at time t t A set of predefined hand feature records (hand posture feature records and / or hand movement feature records) associated with a specific hand gesture 210 that is a part of the hand gesture.
[0075] For simplicity, we use the symbol y 1:t To describe (y 1 ,.....,y t ).
[0076] The estimation terms include singleton terms and paired terms. The singleton terms refer to estimation terms in which each runtime hand data set is simulated by a plurality of discrete hand values of valid predefined hand feature records (each predefined hand feature record defines one of the hand postures and / or hand motions included in the valid hand poses 210). The calculation of the singleton terms is represented in the following formula 1.
[0077] Formula 1:
[0078] S(x 1:T ,y t )= <w s , F s (x 1:t ,y t )>;
[0079] Where S(x 1:T ,y t ) is a set of singleton items, F s (x 1:t ,y t ) is a set of singleton features, and w s is a set of weights assigned to each singleton feature. The set of singleton items and the set of weights w s One or more SSVM functions may be used to select the singletons and weights created through a training session that provide the best accuracy for an associated set of predefined hand feature records.
[0080] The paired estimation term involves an estimation term in which each runtime hand data set is modeled by a plurality of discrete hand values of a current predefined hand feature record and a previous predefined hand feature record in valid predefined hand feature records (each valid predefined hand feature record defines one of the hand postures and / or hand motions included in the valid hand poses 210). The calculation of the paired term is represented in Formula 2 below.
[0081] Formula 2:
[0082] P(x 1:T ,yt-1 ,y t )= <w p , F p (x 1:t ,y t-1 ,y t )>;
[0083] Where P(x 1:T ,y t-1 ,y t ) is a set of paired terms, F p (x 1:t ,y t-1 ,y t ) is a set of paired features, and w p is a set of weights assigned to each pair of features. The set of pair terms and the set of weights w p One or more SSVM functions may be used to select pairs of terms and weights created through a training session that provide the best accuracy for an associated set of predefined hand feature records.
[0084] The set of singleton features and the set of paired features are created by simulating discrete hand values of hand feature records that define valid hand gestures 210 on discrete hand value scores of the runtime hand dataset. Discrete hand values can be represented in Boolean form, such as "(index finger up" or "index finger right)" and "ring finger touches thumb)". After this process, the entire hand feature record is represented by a Boolean expression. Boolean expressions can allow efficient and / or simple simulation. Boolean representations can take many forms, but the Boolean representation that presents the best results can be, for example, CNF. CNF is a Boolean expression in which each Boolean expression can be represented as an "and" operator on two or more "or" operators, for example, "(palm forward" or "palm up" or "palm left)" and "index finger touches thumb" or "middle finger touches thumb)". Simulating the discrete hand values of the hand feature records that define the runtime hand dataset on the discrete hand value scores is performed using one or more of a plurality of parameter functions in which one or more parameters are used to achieve the best simulation results. The generation of singleton features is given in Formula 3 below.
[0085] Formula 3:
[0086]
[0087] where ∝(x t ,y t; ρ) is a set of parameter functions from which the parameter function of the singleton feature that presents the best accuracy can be selected. The set parameter function is executed at runtime because the parameter ρ that provides the best accurate result for the hand gesture detection process 100 may not be known in advance. The execution of the parameter function of each singleton feature can be performed using one or more SSVM functions.
[0088] Similarly, the generation of pairwise features is given in Equation 4 below.
[0089] Formula 4:
[0090]
[0091] Where β(x t ,y t ; σ) is a set of parameter functions from which the parameter functions of the paired features that present the best accuracy are selected. The set parameter functions are executed at runtime because the parameters that provide the best accurate results for the hand gesture detection process 100 may not be known in advance. The execution of the parameter function of each paired feature can be performed using one or more SSVM functions.
[0092] As shown at 140, the hand gesture detection process 100 continues to perform optimization of one or more score functions that use the generated estimates (singleton terms and / or pair terms) to select the predefined hand gestures 210 that best fit the runtime sequence comprising the one or more runtime hand data sets. The score function is optimized by applying the score function to one or more sequences within a FSM such as FSM 301, where each of the one or more sequences corresponds to one of the hand gestures 210. The score function is represented in Formula 5 below.
[0093] Formula 5:
[0094]
[0095] Among them represents the hand pose 210 that best matches the runtime sequence depicting the user's moving hand at time t. The optimization may be performed by dynamic programming employing, for example, Viterbi decoding on a FSM (such as FSM 301) for each of one or more of the plurality of valid hand poses 210 after augmenting the FSM 301 with one or more scoring functions on one or more sequences in the FSM 301.
[0096] Optionally, the set of valid hand gestures 210 at time t includes one or more hand gestures 210 that are not predefined, but include one or more hand gestures (such as hand gesture 250) and / or hand movements (such as hand movement 260) that can be represented using discrete hand values.
[0097] Optionally, one or more SSVM functions are specialized to include only one or more registered hand gestures 210 by selecting the set of valid predefined hand gestures 210 at time t. One or more registered hand gestures 210 may be considered valid with respect to the context of the user's runtime environment. The context may describe one or more runtime execution parameters and / or conditions at time t, such as, for example, active applications at time t, user interaction states, and / or limitations on the hand gestures 210 available to the user. Specializing one or more SSVM functions may further speed up the optimization process to allow for a faster hand gesture detection process 100.
[0098] Reference now Fig. 9 , Fig. 99 is a schematic diagram of an exemplary system for detecting hand gestures depicted by images according to some embodiments of the present disclosure. The exemplary system 900 includes an imaging unit 910 for capturing one or more images of a moving hand of a user 950, one or more hardware processors 920 for detecting hand gestures (such as hand gestures 210 performed by the user 950 depicted by the one or more captured images), and a storage medium 930 for storing one or more predefined hand gesture sequences 201 (each representing a corresponding hand gesture such as hand gesture 210) and / or one or more trained SSVM functions. The imaging unit 910 may include one or more imaging devices, such as a camera sensor, a stereo camera, a depth camera, and / or an IR sensor. The imaging unit 910 that monitors the moving hand of the user 950 may transmit one or more captured images that are timed and synchronized to the processor 920. The one or more timed captured images are analyzed to create a runtime sequence of one or more runtime hand data sets, each runtime hand data set being defined by one or more of a plurality of discrete hand part value scores inferred from the one or more timed captured images. The processor 920 may employ a hand gesture detection process such as the hand gesture detection process 100 to estimate a best one of a plurality of predefined hand gestures 210 (each predefined hand gesture 210 represented by one of the hand gesture sequences 201) that best matches the sequence of runtime gestures performed by the user 950. Optionally, the system 900 includes a SSVM hardware unit 925 that may execute SSVM functions on one or more hardware circuits optimized for SSVM execution. The SSVM hardware unit 925 may efficiently and quickly execute one or more SSVM functions of the hand gesture detection process 100, thereby reducing the computational load of the processor 920. Optionally, the SSVM hardware unit 925 is integrated within one or more processors 920.
[0099] According to some embodiments of the present disclosure, a system and method for detecting multimodal user interaction actions are provided. Each multimodal action can include one or more user actions, for example, hand gestures such as hand gestures 210, head gestures, body gestures, facial expressions, gaze directions, text, voice and / or tactile input. The detection of multimodal actions can be performed for a user using one or more computerized devices, such as computers, mobile devices, computerized machines and / or computerized devices equipped with and / or attached to one or more user interaction capture devices, such as imaging devices, recorders, tactile receiving devices, text input devices (such as, for example, keyboards, digital pens, touch pads, etc.), touch screens, digital surfaces and / or indicating devices (such as, for example, mice, trackballs, game controllers, joysticks, etc.).
[0100] The captured user interactions may be processed to identify one or more multimodal actions. Once one or more multimodal actions are identified, one or more associated functions, actions, operations, and / or commands may be initiated. Multimodal actions may be classified and / or identified during runtime by employing one or more detection techniques to identify one or more user actions that constitute a multimodal action using a hand gesture detection process such as hand gesture detection process 100 (e.g., text input parsing, pointing device input analysis, speech analysis, tactile input analysis, and particularly hand gesture detection).
[0101] Reference now Fig.10 , Fig.10 1000 is a block diagram of building blocks of an exemplary multimodal action according to some embodiments of the present disclosure. Diagram 1000 depicts several configurations of an exemplary multimodal action 1010. The multimodal action 1010 can be configured into one or more possible configurations, for example:
[0102] (a) A multimodal action 1010 can be a combination and / or sequence of hand gestures 210 and voice input 1020.
[0103] (b) The multimodal action 1010 can be a combination and / or sequence of voice input 1020 and hand gesture 210.
[0104] (c) Multimodal action 1010 can be a combination and / or sequence of hand gesture 210 and text input 1030.
[0105] (d) The multimodal action 1010 can be a combination and / or sequence of text input 1030 and hand gesture 210.
[0106] (e) Multimodal action 1010 can be a combination and / or sequence of hand gesture 210 and visual element selection 1040 .
[0107] (f) The multimodal action 1010 can be a combination and / or sequence of visual element selection 1040 and hand gesture 210.
[0108] (g) The multimodal action 1010 can be a combination and / or sequence of hand gestures 210 and tactile inputs 1050 .
[0109] (h) The multimodal action 1010 can be a combination and / or sequence of tactile input 1050 and hand gesture 210.
[0110] (i) A multimodal action 1010 may be a combination and / or sequence of another multimodal action 1010 and one of the hand gestures 210 .
[0111] (j) A multimodal action 1010 may be a combination and / or sequence of another multimodal action 1010 and a non-gestural input 1060 (eg, voice input 1020, text input 1030, visual element selection 1040, and / or tactile input 1050).
[0112] The multimodal actions 1010 may include multiple iterations of the above constructions (i) and / or (j). Each multimodal action 1010 is defined by a unique combination and / or sequence represented by a multimodal sequence 1011, which includes one or more multimodal actions, such as hand gestures 210, voice input 1020, text input 1030, visual element selection 1040, tactile input 1050, and / or multimodal actions 1010. Each multimodal action 1010 begins and ends with an idle state 1080, which is a virtual state that identifies the beginning and / or end of a unique multimodal sequence 1011. One or more multimodal actions that make up the multimodal sequence 1011 can be defined to occur sequentially and / or in parallel with each other. One or more hand gestures 210 and one or more non-gesture inputs 1060 may support each other in context, thereby detecting and / or classifying multimodal actions 1010 because there are fewer possible hand gesture 210 and / or non-gesture input 1060 candidates that effectively match the runtime multimodal sequence of one or more user actions. For example, where multimodal actions 1010 include voice input (such as voice input 1020) specifying a text-related application function, only hand gestures 210 related to text manipulation may be considered during detection and / or classification.
[0113] Reference now Fig.11 , Fig.1111 is a schematic diagram of a multimodal FSM defining a multimodal action according to some embodiments of the present disclosure. Diagram 1100 depicts a multimodal FSM 1101 that can represent a multimodal sequence (such as multimodal sequence 1011) that defines a multimodal action (such as multimodal action 1010). Multimodal FSM 1101 can represent one or more multimodal actions, such as hand gestures (such as hand gesture 210) and / or non-gesture inputs (such as non-gesture input 1060). Multimodal FSM 1101 begins at start point 1110, which can be a virtual state indicating an idle state of multimodal FSM 1101 and can include one or more states 1102A, 1102B, 1102C to 1102N. Each of states 1102A to 1102N can represent one of hand gestures 210 and / or non-gesture input 1060. The multimodal FSM 1101 ends at a FSM end point 1120, which marks the completion of the multimodal sequence 1011, at which point the detection and / or classification process provides an estimate of the detected multimodal action. The starting point 1110 and / or the end point 1120 can be defined by a virtual idle state such as the idle state 1080. Optionally, any transition between one or more of the multimodal FSM states 1102A to 1102N can be detected and recorded by the system and / or operating system (OS) and used for one or more of a number of purposes, such as use by the currently active application, use by one or more other applications, and / or use by the OS controlling the execution environment. Optionally, the multimodal FSM 1101 can be constructed as a complex multimodal action, such as the multimodal action 1010, by including repetitions of one or more of the states 1102A to 1102N, dividing into several parallel and / or sequential paths, and / or combining two or more multimodal FSMs 1101 and / or portions thereof. The multimodal sequence 1011 represented as a multimodal FSM 1101 can improve the detection and / or classification of one or more multimodal actions 1010. More specifically, the detection of one or more hand gestures 210 included in the multimodal action 1010 can be highly improved because each hand gesture 210 is constructed using a discrete architecture. Each of the t hand gestures 210 is defined by a hand gesture sequence (such as hand gesture sequence 201) represented by a hand feature record (such as a hand feature record). Each hand feature record is defined by a discrete hand value score each indicating a finite number of states of a corresponding hand feature that can be easily identified. Detection, identification and / or classification can be simplified because a user application is not required to perform hand skeleton modeling to perform recognition, identification and / or classification of the hand gesture 210. In addition, computer learning and / or computer vision processing can be significantly reduced due to the discrete architecture performed by the hand gesture detection process 100.
[0114] Reference now Fig.12 , Fig.12 12 is a schematic diagram of exemplary multimodal actions according to some embodiments of the present disclosure. Diagram 1200 depicts three multimodal actions 1010A, 1010B, and 1010C, such as multimodal action 1010. Each of multimodal actions 1010A, 1010B, and 1010C is defined by a multimodal sequence (e.g., multimodal sequence 1011), where multimodal sequence 1011A defines multimodal action 1010A, multimodal sequence 1011B defines multimodal action 1010B, and multimodal sequence 1011C defines multimodal action 1010C. Each of multimodal sequences 1011A, 1011B, and 1011C begins and ends in an idle state, such as idle state 1080, that respectively identifies the beginning and end of multimodal actions 1010A, 1010B, and 1010C.
[0115] For example, in a drawing tool application, a multimodal action 1010A is associated with an item fill application function. As shown, a multimodal sequence 1011A includes three multimodal actions: item visual element selection 1040A, a brush hand gesture 210B, and a color voice input 1020A. Item visual element selection 1040A identifies an item in a drawing area of a drawing tool application. For example, item visual element selection 1040A can be detected, for example, as a selection of an item using, for example, a touch screen, a digital surface, and / or an indicating device. Brush hand gesture 210B identifies a brush hand gesture, such as swinging an open hand from left to right, which is associated with an item fill application function related to an item selected by item visual element selection 1040A. Color voice input 1020A identifies a color, such as the blue color that will be used to fill the item selected by item visual element selection 1040A. The result of the multimodal action 1010A is that the item selected by item visual element selection 1040A is filled with blue.
[0116] Multimodal action 1010B is associated with a text editing application function, such as in a text editor application. As shown, multimodal sequence 1011B includes 2 multimodal actions: selection hand gesture 210C and text input 1030A. Selection hand gesture 210C identifies a selection hand gesture, for example, by, for example, a text editing application moving an indicating index finger on text presented on a screen to select a text location that needs to be edited. Text input 1030A inserts new and / or modified text at the location identified by selection hand gesture 210C. The result of multimodal action 1010B is that the text identified by selection hand gesture 210C is updated using the text received by text input 1030A.
[0117] Multimodal action 1010C is associated with the text message transmission project application function using, for example, a text message transceiving application. As shown, multimodal sequence 1011C includes 4 multimodal actions: text message hand gesture 210D, contact voice input 1020B, text input 1030B, and sending hand gesture 210E. Text message hand gesture 210D (e.g., performing a graffiti action and pushing the hand away) is detected and initiates the text message transceiving application. Contact voice input 1020B identifies a contact from the contact list of the text message transceiving application. Text input 1030B inserts the message text into the message. Sending hand gesture 210E (e.g., pushing the hand to the right) is detected and initiates the transmission operation of a message to the contact identified by the contact voice input 1020B.
[0118] It is expected that during the life of the patent, from this application, many related DFEs, HMIs and / or NUIs will be developed, and the scope of the terms DFE, HMI and / or NUI is intended to include all such new technologies in advance.
[0119] The terms "comprises," "comprising," "includes," "including," "having" and their cognates mean "including but not limited to."
[0120] The term "consisting of" means "including and limited to."
[0121] The term "consisting essentially of" means that the composition, method or structure may include additional ingredients, steps and / or parts, but only if the additional ingredients, steps and / or parts do not materially change the basic and novel characteristics of the claimed composition, method or structure.
[0122] As used herein, the singular forms "a," "an," and "the" include the plural forms unless the context clearly dictates otherwise. For example, the term "hand gesture" or "at least a hand gesture" may include a single hand gesture and / or a double hand gesture.
[0123] As used herein, the term "method" refers to ways, means, techniques and procedures for accomplishing a given task, including but not limited to those ways, means, techniques and procedures that are known or that can be readily developed from known ways, means, techniques and procedures by practitioners in the fields of chemistry, pharmacology, biology, biochemistry and medicine.
[0124] According to some embodiments of the present disclosure, there is provided a computer-implemented method for detecting a user's hand gesture, comprising:
[0125] a) receiving a plurality of sequential logic models, each of the plurality of sequential logic models representing a hand gesture in a plurality of hand gestures. The sequential logic model maps one or more of a plurality of predefined hand gestures and predefined hand motions. Each of the plurality of predefined hand gestures and hand motions is represented by a predefined hand feature record in a plurality of predefined hand feature records, the plurality of predefined hand feature records being defined by one or more discrete hand values in a plurality of discrete hand values, each of the plurality of discrete hand values indicating a state of a hand feature in a plurality of hand features of a reference hand.
[0126] b) receiving a runtime sequence of a plurality of runtime hand data sets, each of the plurality of runtime hand data sets being defined by a plurality of discrete hand value scores, each of the plurality of discrete hand value scores indicating a current state of a corresponding one of the plurality of hand features of a moving hand of a user, the plurality of discrete hand value scores being inferred by analysis of a plurality of timed images visually depicting the moving hand.
[0127] c) submitting the plurality of runtime hand data sets and the plurality of predefined hand feature records in one or more SSVM functions to generate a plurality of estimation terms for each runtime hand data set in the plurality of runtime hand data sets with respect to the plurality of predefined hand feature records.
[0128] d) estimating which hand pose of the plurality of hand poses best matches a runtime sequence depicted in the plurality of timed images by optimizing one or more scoring functions using an estimated term for each of the plurality of runtime hand datasets in the runtime sequence.
[0129] Each discrete hand value of the plurality of discrete hand values is represented by a Boolean formula, which is defined in conjunctive normal form (CNF).
[0130] The one or more SSVM functions generate the plurality of estimation terms by applying a plurality of parameter functions to each of the plurality of runtime hand data sets to identify one or more parameter functions that provide a highest detection accuracy for each of the plurality of runtime hand data sets. Each of the plurality of parameter functions models each of the plurality of discrete hand values at each of the plurality of discrete hand value scores.
[0131] The one or more SSVM functions include a plurality of sub-functions, each of the plurality of sub-functions receiving as input a runtime hand dataset of the plurality of runtime hand datasets and a hand feature record of the plurality of hand feature records. Each of the plurality of sub-functions is assigned a weight, the weight being learned by the SSVM function during a training session.
[0132] The one or more SSVM functions include a plurality of sub-functions, each of the plurality of sub-functions receiving as input one of the plurality of runtime hand data sets and two of the plurality of hand feature records, the two hand feature records representing a current term and a previous term of the plurality of hand gestures and / or hand motions within the sequential logic model. Each of the plurality of sub-functions is assigned a weight, the weight being learned by the SSVM function during a training session.
[0133] Optionally, the sequential logic model is represented as a FSM. Each state of the FSM is associated with a corresponding one of the plurality of predefined hand feature records. The FSM is enhanced on one or more sequences within the FSM using the one or more scoring functions before the optimization.
[0134] Optionally, the one or more SSVM functions are specialized by selecting one or more sequential logic models from the plurality of sequential logic models that represent context-registered hand gestures from the plurality of hand gestures.
[0135] Optionally, the plurality of runtime hand data sets are estimated as one of a plurality of hand postures and / or hand motions that are not predefined.
[0136] Optionally, the detection comprises detection of one or more of a plurality of multimodal actions. The one or more multimodal actions comprise one or more of the plurality of hand gestures and one or more non-gesture user interactions. The non-gesture user interaction is a member of the group comprising: text input, visual element selection, tactile input, and / or voice input.
[0137] According to some embodiments of the present disclosure, a system for detecting a user's hand gesture is provided, the system comprising a storage device storing a plurality of predefined hand feature records; a memory storing a code, and one or more processors coupled to the storage device and the memory for executing the stored code. The code comprises:
[0138] a) code instructions for receiving a plurality of sequential logic models, each of the plurality of sequential logic models representing a hand gesture of a plurality of hand gestures. The sequential logic model maps one or more of a plurality of predefined hand gestures and predefined hand motions. Each of the plurality of predefined hand gestures and hand motions is represented by a predefined hand feature record of a plurality of predefined hand feature records, the plurality of predefined hand feature records being defined by one or more discrete hand values of a plurality of discrete hand values, each of the plurality of discrete hand values indicating a state of a hand feature of a plurality of hand features of a reference hand.
[0139] b) code instructions for receiving a runtime sequence of a plurality of runtime hand data sets, each of the plurality of runtime hand data sets being defined by a plurality of discrete hand value scores, each of the plurality of discrete hand value scores indicating a current state of a corresponding one of the plurality of hand features of a moving hand of a user, the plurality of discrete hand value scores being inferred by analysis of a plurality of timed images visually depicting the moving hand.
[0140] c) code instructions for submitting the plurality of runtime hand data sets and the plurality of predefined hand feature records in one or more SSVM functions to generate a plurality of estimation terms for each runtime hand data set in the plurality of runtime hand data sets with respect to the plurality of predefined hand feature records.
[0141] d) code instructions for estimating which of the plurality of hand gestures best matches a runtime sequence depicted in the plurality of timed images by optimizing one or more scoring functions using an estimated term for each of the plurality of runtime hand data sets in the runtime sequence.
[0142] According to some embodiments of the present disclosure, a software program product for detecting a user's hand gesture is provided, comprising a non-transitory computer-readable storage medium and program instructions:
[0143] a) First program instructions for receiving a plurality of sequential logic models, each of the plurality of sequential logic models representing a hand gesture in a plurality of hand gestures. The sequential logic model maps one or more of a plurality of predefined hand gestures and predefined hand motions. Each of the plurality of predefined hand gestures and hand motions is represented by a predefined hand feature record in a plurality of predefined hand feature records, the plurality of predefined hand feature records being defined by one or more discrete hand values in a plurality of discrete hand values, each of the plurality of discrete hand values indicating a state of a hand feature in a plurality of hand features of a reference hand.
[0144] b) second program instructions for receiving a runtime sequence of a plurality of runtime hand data sets, each of the plurality of runtime hand data sets being defined by a plurality of discrete hand value scores, each of the plurality of discrete hand value scores indicating a current state of a corresponding one of the plurality of hand features of a moving hand of a user, the plurality of discrete hand value scores being inferred by analysis of a plurality of timed images visually depicting the moving hand.
[0145] c) third program instructions for submitting the plurality of runtime hand data sets and the plurality of predefined hand feature records in one or more SSVM functions to generate a plurality of estimation terms for each runtime hand data set in the plurality of runtime hand data sets with respect to the plurality of predefined hand feature records.
[0146] d) fourth program instructions for estimating which of the plurality of hand gestures best matches a runtime sequence depicted in the plurality of timed images by optimizing one or more scoring functions using an estimated term for each of the plurality of runtime hand data sets in the runtime sequence.
[0147] The first, second, third, and fourth program instructions are executed by one or more computerized processors from the non-transitory computer readable storage medium.
[0148] Optionally, the detection of the software program product comprises detection of one or more of a plurality of multimodal actions. The one or more multimodal actions comprise one or more of the plurality of hand gestures and one or more non-gesture user interactions. The non-gesture user interaction is a member of the group comprising: text input, visual element selection, tactile input, and / or voice input.
[0149] For the sake of clarity, certain features of the examples described herein that are described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity, various features of the examples described herein that are described in the context of a single embodiment may also be provided individually or in any suitable sub-combination, or appropriately provided in any other described embodiment of the present disclosure. Certain features described in the context of various embodiments are not considered essential features of those embodiments, unless the embodiment does not function without those elements.
Claims
1. A computer-implemented method for detecting a user's hand gestures, include: receiving a plurality of logical models representing a plurality of hand gestures, each of the logical models being associated with at least one predefined hand gesture of a plurality of predefined hand gestures and at least one predefined hand motion of a plurality of predefined hand motions; receiving a runtime sequence for a moving hand, the runtime sequence comprising a plurality of runtime hand data sets, the plurality of runtime hand data sets defining a plurality of discrete motion values, the plurality of discrete motion values indicating a plurality of motion attribute features or motion script features associated with a plurality of parts of the moving hand, wherein the plurality of discrete motion values indicate a size or direction for the plurality of parts of the moving hand; generating a plurality of estimation terms based on the plurality of discrete motion values of the motion attribute feature or the motion script feature of the moving hand, wherein the plurality of estimation terms include singleton terms and pairwise terms determined from simulation of the runtime hand dataset; as well as The generated plurality of estimates are used to estimate, using a computerized processor, which of the plurality of hand gestures identifies the moving hand.
2. The computer-implemented method of claim 1 , further comprising detecting at least one of a plurality of multimodal actions, the at least one multimodal action comprising at least one of a plurality of hand movements and at least one non-gestural user interaction. 3 . The computer-implemented method of claim 1 , further comprising estimating each of the plurality of runtime hand data sets as one of a plurality of hand poses or hand motions that are not predefined.
4. The computer-implemented method of claim 1 , further comprising implementing a structured support vector machine (SSVM) function to generate a plurality of estimation terms by identifying at least one parameter function for each of the plurality of runtime hand data sets by applying a plurality of parameter functions to each of the plurality of runtime hand data sets, each of the plurality of parameter functions simulating each of the plurality of discrete motion values by each of the plurality of discrete hand part value scores.
5. A computer-implemented method according to claim 1, wherein using the generated multiple estimation terms to estimate which hand gesture of the multiple hand gestures identifies the moving hand is performed at least in part by analyzing one or more pairs of terms identified through a training session.
6. The computer-implemented method of claim 1 , further comprising implementing a structured support vector machine (SSVM) function, the SSVM function comprising a plurality of sub-functions, each sub-function receiving as input one of the plurality of runtime hand data sets.
7. The computer-implemented method of claim 1 , wherein at least one logical model is represented as a finite state machine (FSM) having states associated with a corresponding one of a plurality of predefined hand feature records, the FSM being configured to be augmented over at least one sequence using a scoring function. 8 . The computer-implemented method of claim 1 , further comprising implementing a structured support vector machine (SSVM) to select a context-registered hand gesture among the plurality of hand gestures.
9. The computer-implemented method of claim 1, wherein each discrete motion value of the plurality of discrete motion values is represented by a Boolean formula, the Boolean formula being defined in conjunctive normal form (CNF).
10. A computer-implemented method according to claim 1, wherein using the generated multiple estimation items to estimate which hand posture of the multiple hand postures identifies the moving hand includes submitting the hand data set together with one or more predefined hand feature records to one or more structured support vector machine (SSVM) functions.
11. A system for detecting a user's hand gesture, include: a storage device for storing a plurality of predefined hand feature records; Memory for storing code; at least one processor, coupled to the storage device and the memory, configured to execute the stored code, the code comprising: code instructions for receiving a plurality of logical models representing a plurality of hand gestures, each of the logical models being associated with at least one predefined hand gesture of a plurality of predefined hand gestures and at least one predefined hand motion of a plurality of predefined hand motions; code instructions for receiving a runtime sequence for a moving hand, the runtime sequence comprising a plurality of runtime hand data sets, the plurality of runtime hand data sets defining a plurality of discrete motion values, the plurality of discrete motion values being a plurality of motion attribute features or motion script features associated with a plurality of parts of the moving hand, wherein the plurality of discrete motion values indicate magnitudes or directions for the plurality of parts of the moving hand; code instructions for generating a plurality of estimation terms based on the plurality of discrete motion values of the motion attribute feature or the motion script feature of the moving hand, wherein the plurality of estimation terms include singleton terms and paired terms determined from a simulation of the runtime hand data set; and Code instructions for using the generated plurality of estimates to estimate which of the plurality of hand gestures identifies the moving hand.
12. The system of claim 11, wherein at least one logical model is represented as a finite state machine (FSM) having states associated with a corresponding one of a plurality of predefined hand feature records, the FSM being configured to be enhanced on at least one sequence using a scoring function.
13. The system of claim 11, further comprising code implementing a structured support vector machine (SSVM) function to select a context-registered hand gesture from among the plurality of hand gestures.
14. The system of claim 11, wherein each discrete motion value of the plurality of discrete motion values is represented by a Boolean formula, the Boolean formula being defined in conjunctive normal form (CNF).
15. The system of claim 11, wherein using the generated plurality of estimates to estimate which of the plurality of hand poses identifies the moving hand comprises submitting the hand data set to one or more structured support vector machine (SSVM) functions.
16. The system according to claim 11 further includes the logical model being represented as a FSM, each state of the FSM is associated with a corresponding one of the multiple predefined hand feature records, and the FSM is enhanced on at least one sequence within the FSM using the at least one scoring function.
17. The system of claim 11, further comprising code implementing a structured support vector machine (SSVM) function to generate a plurality of estimation terms by identifying at least one parameter function for each of the plurality of runtime hand data sets by applying a plurality of parameter functions to each of the plurality of runtime hand data sets, each of the plurality of parameter functions simulating each of the plurality of discrete motion values by each of the plurality of discrete hand part value scores.
18. The system of claim 11, further comprising code instructions for detecting at least one of a plurality of multimodal actions, the at least one multimodal action comprising at least one of the plurality of hand gestures and at least one non-gesture user interaction, the non-gesture user interaction being a member of a group comprising: text input, visual element selection, tactile input, and voice input.
19. One or more computer storage memory devices embodied with executable instructions configured to detect a user's hand gestures, include: a non-transitory computer-readable storage medium; first program instructions for receiving a plurality of logical models representing a plurality of hand gestures, the logical models associated with at least one predefined hand gesture of a plurality of predefined hand gestures and at least one predefined hand motion of a plurality of predefined hand motions; second program instructions for receiving a runtime sequence for a moving hand, the runtime sequence comprising a plurality of runtime hand data sets defining a plurality of discrete motion values for a plurality of motion attribute features or motion script features associated with a plurality of parts of the moving hand; third program instructions for generating a plurality of estimation terms based on the plurality of discrete motion values of the motion attribute feature or the motion script feature of the moving hand, wherein the plurality of estimation terms include singleton terms and pairwise terms determined from simulation of the runtime hand data set; as well as fourth program instructions for estimating which of the plurality of hand gestures identifies the moving hand using the generated plurality of estimates, and Wherein said first, second, third and fourth program instructions are executed from said computer readable storage medium by at least one computerized processor.
20. One or more computer storage memory devices according to claim 19, wherein the plurality of hand gestures comprises a plurality of finger characteristics selected from a group consisting of: a finger bend state, a finger curve state, a finger touch condition, a finger touch type, or a relative position of one finger relative to another finger.
Citation Information
Patent Citations
Vision system for vehicle
US20160068160A1
Gesture detection and recognition
CN102693007A
Dynamic gesture based short-range human-machine interaction
CN103946863A
Action recognition method of posture sequence finite-state machine
CN104317386A
