Three-dimensional gesture detection apparatus and three-dimensional gesture detection method

Abstract

A three-dimensional gesture detection device and a three-dimensional gesture detection method. The three-dimensional gesture detection device includes a node detection unit, a gesture recognition model, and a gesture trajectory detection unit. The node detection unit obtains a plurality of nodes according to each hand image of a continuous hand image. The gesture recognition model obtains a plurality of gesture class confidences. The gesture trajectory detection unit includes a weight analyzer, a gesture analyzer, a key point classifier, and a trajectory analyzer. The weight analyzer obtains a plurality of weights of the plurality of gesture classes according to a user interface. The gesture analyzer performs weighted calculation on the plurality of gesture class confidences to analyze a gesture of each image. The key point classifier classifies a plurality of key points from the plurality of nodes. The trajectory analyzer obtains an inertia trajectory of a gesture according to the plurality of key points.

Description

[Technical Field]

[0001] This invention relates to a detection device and a detection method, and more particularly to a three-dimensional gesture detection device and a three-dimensional gesture detection method. [Background Technology]

[0002] With advancements in image recognition technology, a gesture detection technology has been developed. This technology allows users to control electronic devices more intuitively through gestures.

[0003] However, current technology does not detect gestures very consistently; sometimes the gestures are misplaced, sometimes they are missed, and occasionally the detection is intermittent. Gesture control technology still faces significant limitations. Researchers are working to improve the detection stability. [Summary of the Invention]

[0004] This invention relates to a three-dimensional gesture detection device and a three-dimensional gesture detection method, which are adaptively adjusted for the user interface, making gesture analysis more stable. The detection of inertial trajectory does not simply rely on gestures to determine the inertial trajectory, but further considers node analysis to ensure the inertial trajectory is continuous and smooth.

[0005] According to one aspect of the present invention, a three-dimensional gesture detection device is provided. The three-dimensional gesture detection device includes a node detection unit, a gesture recognition model, and a gesture trajectory detection unit. The node detection unit receives a continuous image of a hand. The continuous image of the hand has several hand frames. The node detection unit obtains several nodes based on each hand frame. The gesture recognition model obtains several gesture category confidence scores for several gesture categories based on the multiple nodes. The gesture trajectory detection unit includes a weight analyzer, a gesture analyzer, a keypoint classifier, and a trajectory analyzer. The weight analyzer obtains several weights for the multiple gesture categories based on a user interface. The gesture analyzer performs a weighted calculation of the multiple gesture category confidence scores based on the multiple weights to analyze a gesture in each frame. The keypoint classifier classifies several key points from the multiple nodes based on the gesture. The trajectory analyzer obtains an inertial trajectory of the gesture based on the changes of the multiple key points in the multiple hand frames.

[0006] According to another aspect of the present invention, a three-dimensional gesture detection method is proposed. The three-dimensional gesture detection method includes the following steps: Obtaining a continuous image of a hand. The continuous image of a hand has several hand frames. Obtaining several nodes based on each hand frame. Obtaining several gesture category confidence scores for several gesture categories based on the nodes. Obtaining several weights for the multiple gesture categories based on a user interface. Calculating a weighted average of the multiple gesture category confidence scores based on the multiple weights to analyze a gesture in each frame. Classifying several key points from the multiple nodes based on the gesture. Obtaining an inertial trajectory of the gesture based on the changes of the multiple key points in the multiple hand frames.

[0007] To provide a better understanding of the above and other aspects of the present invention, specific embodiments are described below in conjunction with the accompanying drawings: [Attached Image Description]

[0008] Figure 1 A schematic diagram of a three-dimensional gesture detection device according to one embodiment is shown. Figure 2 Examples illustrate the actions corresponding to the "tap" gesture.

[0009] Figure 3 The image shows multiple points in time.

[0010] Figure 4 The drawing detected Figure 3 The corresponding operations for the gestures.

[0011] Figure 5 A block diagram of a three-dimensional gesture detection device according to one embodiment is shown. Figure 6 A flowchart illustrating a three-dimensional gesture detection method according to one embodiment is shown.

[0012] Figure 7 Example illustration of step S110.

[0013] Figure 8 Example illustration of step S120.

[0014] Figure 9 The following is a schematic diagram of step S130.

[0015] Figure 10 Example illustration of step S140.

[0016] Figure 11 Example illustration of step S150.

[0017] Figure 12 Example illustration of step S160.

[0018] Figure 13 Example illustration of step S170.

[0019] Figures 14A-14B The example illustrates another variation in key points.

[0020] Figures 15A-15B The example illustrates another variation in key points.

[0021] Figure 16 The example illustrates another instance of step S160.

[0022] [Symbol Explanation]

[0023] 100: Three-dimensional gesture detection device

[0024] 110: Node Detection Unit

[0025] 120: Gesture Recognition Model

[0026] 130: Gesture trajectory detection unit

[0027] 131: Weight Analyzer

[0028] 132: Gesture Analyzer

[0029] 133: Key Point Classifier

[0030] 134: Trajectory Analyzer

[0031] 300: Notebook computer

[0032] 800: Image Capture Unit

[0033] CF, CF*: Confidence level of gesture category

[0034] CG: Gesture Category

[0035] CT: Center of Gravity

[0036] FM: screen

[0037] GT: Gestures

[0038] ND: Node

[0039] ND*: Key Points

[0040] P1: Point

[0041] S110, S120, S130, S140, S150, S160, S170: Steps

[0042] t2, t3, t4, t5, t6, t7, t8, t9: Time points

[0043] TR: Inertial trajectory

[0044] UI: User Interface

[0045] VD: Hand Continuous Footage

[0046] VT: Longest radial

[0047] WT: Weight

Detailed Implementation Methods

[0048] Please refer to Figure 1 The diagram illustrates a three-dimensional gesture detection device 100 according to one embodiment. The three-dimensional gesture detection device 100 is, for example, an external electronic device that can be connected to a laptop computer 300 via USB. Alternatively, the three-dimensional gesture detection device may also be built into the laptop computer 300. Figure 1 As shown, users can make gestures above the 3D gesture detection device 100, allowing the 3D gesture detection device 100 to function as an intuitive input device for the notebook computer 300.

[0049] For example, please refer to Figure 2 The example illustrates the action corresponding to the "tap" gesture. The notebook computer 300 can define that when a "tap" gesture is detected, the left mouse button is pressed. Furthermore, the notebook computer 300 can further define that when the "tap" gesture moves in the air, a dragging action is triggered.

[0050] However, for the dragging motion to be smooth, the "click" gesture must be detected on every screen. Please refer to... Figure 3 The image shows the screen at time points t2 to t9. When detecting gestures, due to factors such as the user's palm being relaxed or changes in ambient brightness, the "click" gesture may only be detected at time points t2 to t4, t8, and t9, while the "open palm" gesture may be detected at time points t5 to t7.

[0051] Please refer to Figure 4 Its drawing detected Figure 3 The corresponding operation for the gesture. Only when the "click" gesture is detected on the screen at time points t2-t4, t8, and t9, the cursor trajectory of the notebook computer 300 will stay at point P1 for a period of time (i.e., time points t5-t7), making the dragging action unsmooth.

[0052] Please refer to Figure 5 The diagram illustrates a block diagram of a three-dimensional gesture detection device 100 according to an embodiment. The three-dimensional gesture detection device 100 includes a node detection unit 110, a gesture recognition model 120, and a gesture trajectory detection unit 130. The gesture trajectory detection unit 130 includes a weight analyzer 131, a gesture analyzer 132, a key point classifier 133, and a trajectory analyzer 134.

[0053] Node detection unit 110 is used to detect nodes (ND), and gesture recognition model 120 is used to analyze gesture category confidence level (CF). Both node ND and gesture category confidence level (CF) are output to gesture trajectory detection unit 130 for inertial trajectory (TR) analysis. Node detection unit 110, gesture recognition model 120, and gesture trajectory detection unit 130 can be, for example, a circuit, a chip, a circuit board, program code, or a storage device for storing program code. In this embodiment, the gesture GT analysis is adaptively adjusted for the user interface (UI), making the gesture GT analysis more stable. Furthermore, the gesture trajectory detection unit 130 does not simply use the gesture GT to determine the inertial trajectory (TR), but further refers to node ND for analysis, making the inertial trajectory (TR) continuous and smooth. A flowchart is provided below to explain the operation of each of the above components in detail.

[0054] Please refer to Figure 5 and Figure 6 , Figure 6 A flowchart illustrating a three-dimensional gesture detection method according to an embodiment is provided. The three-dimensional gesture detection method includes steps S110 to S170. Please refer to... Figure 7 The example illustrates step S110. In step S110, an image capturing unit 800 acquires a continuous hand image VD. The continuous hand image VD has several hand frames FM (illustrated in...). Figure 7 In one embodiment, the image capturing unit 800 may be built into the 3D gesture detection device 100. The image capturing unit 800 may be, for example, a color camera or an infrared camera.

[0055] Next, please refer to Figure 8 The example illustrates step S120. In step S120, the node detection unit 110 receives a continuous hand image VD and obtains several nodes ND based on each hand image FM. In this embodiment, these multiple nodes ND are input to the gesture recognition model 120 and the gesture trajectory detection unit 130.

[0056] Then, please refer to Figure 9 The diagram illustrates step S130. In step S130, the gesture recognition model 120 obtains several gesture category confidence scores (CFs) for several gesture categories (CGs) based on the multiple nodes (ND). After the multiple nodes (ND) of each frame (FM) are input into the gesture recognition model 120, the corresponding gesture category confidence score (CF) can be output for various preset gestures. For example, for gesture categories (CGs) such as "tap", "pinch", "open palm", "clenched fist", and "upright palm", their gesture category confidence scores (CFs) are 0.9, 0.1, 0.7, 0.5, and 0.1, respectively. "Tap" is the gesture category (CG) with the highest category confidence score (CF).

[0057] In this embodiment, after obtaining the multiple gesture category confidence scores (CFs), the gesture recognition model 120 does not directly use the one with the highest category confidence score (CF) as the analysis result of the gesture GT, but instead inputs the multiple gesture category confidence scores (CFs) to the gesture trajectory detection unit 130.

[0058] Next, please refer to Figure 10 The example illustrates step S140. In step S140, the weight analyzer 131 obtains several weights WT for the multiple gesture categories CG based on a user interface (UI). For example, in Figure 10 The most likely user interface (UI) to use is the "click" gesture category CG. Gesture categories such as "pick up", "open palm", "clench fist", and "open palm" are rarely used. Therefore, for gesture categories such as "click", "pick up", "open palm", "clench fist", and "open palm", their weights WT can be 0.9, 0.1, 0.1, 0.1, 0.1 or 0.9, 0.0, 0.0, 0.0, 0.0.

[0059] Then, please refer to Figure 11 The example illustrates step S150. In step S150, the gesture analyzer 132 performs a weighted calculation on the multiple gesture category confidence scores CF based on the multiple weights WT, obtaining the gesture category confidence score CF*, to analyze the gesture GT of each frame FM. As described above, after weighted calculation, in Figure 10 In the user interface (UI), "click" will no longer be misinterpreted as "open palm".

[0060] Next, please refer to Figure 12 The example illustrates step S160. In step S160, the key point classifier 133 classifies several key points ND* from the multiple nodes ND based on the gesture GT. Taking the "click" gesture GT as an example, the user may relax the middle, ring, and little fingers, but the index and thumb usually do not relax. Therefore, the key point ND* can be set as the node ND of the index finger and thumb. By setting the key point ND*, the continuity of the inertial trajectory can be enhanced.

[0061] Then, please refer to Figure 13 The example illustrates step S170. In step S170, the trajectory analyzer 134 obtains an inertial trajectory TR of the gesture GT based on the changes of the multiple key points ND* on the multiple hand images FM.

[0062] In one embodiment, the changes of the multiple key points ND* in the multiple hand images FM include a change in the center of gravity CT. In this embodiment, as long as the key points ND* are obtained in the image FM, this image FM can be added to the analysis of the inertial trajectory TR. In this way, the inertial trajectory TR can be detected smoothly.

[0063] In one embodiment, the trajectory analyzer 134 performs inertial trajectory TR analysis, for example, using average, single-exponential, double-exponential, or Kalman filters.

[0064] Please refer to Figures 14A-14B The example illustrates another variation in keypoint ND*. In another embodiment, the variation of keypoint ND* in the plurality of hand images FM includes a variation in vector length. Figures 14A-14B As shown, the vector length can be analyzed based on the longest radial distance VT of the key point ND*. The trajectory analyzer 134 can analyze the inertial trajectory TR based on the change in the vector length. For example, when the vector length increases, it indicates that the user is sliding their index finger upward.

[0065] Please refer to Figures 15A-15B The example illustrates another type of keypoint ND* variation. In another embodiment, the variation of the keypoint ND* with respect to the plurality of hand images FM includes a change in a vector angle. For example... Figures 15A-15B As shown, the vector angle can be analyzed based on the longest radial distance VT of the key point ND*. The trajectory analyzer 134 can analyze the inertial trajectory TR based on the change in the vector angle. For example, when the vector angle changes from a negative value to a positive value, it indicates that the user is sliding their index finger to the right.

[0066] According to the above embodiments, the gesture GT analysis has been adaptively adjusted for the user interface UI, making the gesture GT analysis more stable. Furthermore, the gesture trajectory detection unit 130 does not simply use the gesture GT to determine the inertial trajectory, but further refers to the node ND for analysis, ensuring that the inertial trajectory TR is continuous and smooth.

[0067] In addition, please refer to Figure 16 The example illustrates another instance of step S160. In step S160, the key point classifier 133 classifies several key points ND* from the multiple nodes ND based on the gesture GT. Taking the gesture GT of "taking" as an example, the user may usually relax the middle, ring, and little fingers, but the fingertips of the index finger and thumb usually do not relax. Therefore, the key point ND* can be set as the node of the fingertip of the index finger and the fingertip of the thumb. By setting the key points ND*, the continuity of the inertial trajectory can be enhanced.

[0068] In summary, although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A three-dimensional gesture detection apparatus, characterized by, Comprising: a node detection unit to receive a hand continuous video, the hand continuous video having a plurality of hand frames, the node detection unit to obtain a plurality of nodes from each of the hand frames; a gesture recognition model to obtain a plurality of gesture class confidences of a plurality of gesture classes from the plurality of nodes; and a gesture trajectory detection unit comprising: a weight analyzer to obtain a plurality of weights of the plurality of gesture classes from a user interface; a gesture analyzer to weight the plurality of gesture class confidences by the plurality of weights to analyze a gesture from each of the frames; a key point categorizer to categorize a plurality of key points from the plurality of nodes according to the gesture; and a trajectory analyzer to obtain an inertia trajectory of the gesture from changes of the plurality of key points in the plurality of hand frames.

2. The three-dimensional gesture detection apparatus of claim 1, wherein, If the gesture is a tap, the plurality of key points are nodes of an index finger and a thumb.

3. The three-dimensional gesture detection apparatus of claim 1, wherein If the gesture is a pinch or a pick, the plurality of key points are nodes of an index finger tip and a thumb tip.

4. The three-dimensional gesture detection apparatus of claim 1, wherein, The changes of the plurality of key points in the plurality of hand frames include a change of a center of gravity.

5. The three-dimensional gesture detection apparatus of claim 1, wherein, The changes of the plurality of key points in the plurality of hand frames include a change of a vector angle and a change of a vector length.

6. A method of three-dimensional gesture detection, the method comprising: Comprising: obtaining a hand continuous video, the hand continuous video having a plurality of hand frames; obtaining a plurality of nodes from each of the hand frames; obtaining a plurality of gesture class confidences of a plurality of gesture classes from the plurality of nodes; obtaining a plurality of weights of the plurality of gesture classes from a user interface; weighting the plurality of gesture class confidences by the plurality of weights to analyze a gesture from each of the frames; categorizing a plurality of key points from the plurality of nodes according to the gesture; and obtaining an inertia trajectory of the gesture from changes of the plurality of key points in the plurality of hand frames. If the gesture is a tap, the plurality of key points are nodes of an index finger and a thumb.

7. The three-dimensional gesture detection method of claim 6, wherein, If the gesture is a pinch or a pick, the plurality of key points are nodes of an index finger tip and a thumb tip.

8. The three-dimensional gesture detection method of claim 6, wherein, The changes of the plurality of key points in the plurality of hand frames include a change of a center of gravity.

9. The three-dimensional gesture detection method of claim 6, wherein, The changes of the plurality of key points in the plurality of hand frames include a change of a vector angle and a change of a vector length.

10. The three-dimensional gesture detection method of claim 6, wherein, ​

Images