Gesture recognition apparatus, head-mounted display apparatus, gesture recognition method, program, and storage medium

Abstract

To provide a technology capable of recognizing gestures with high accuracy under various situations.SOLUTION: A gesture recognition apparatus includes: first detection means configured to detect from a captured image a first portion making a gesture; second detection means configured to detect from the captured image a second portion making the gesture in the first portion detected by the first detection means; and recognition means configured to recognize the gesture on the basis of a motion of the first portion detected by the first detection means and a motion of the second portion detected by the second detection means. In a case where a detection result satisfying a predetermined condition is not obtained by the second detection means, the recognition means recognizes the gesture using a detection result satisfying the predetermined condition obtained in the past by the second detection means.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a technique for recognizing gestures.

Background Art

[0002] Techniques have been proposed for recognizing gestures made by a user's hand or finger and performing processing according to the recognized gesture. According to such a technique, the user can operate the electronic device (input device of the electronic device) without touching the electronic device by using a gesture.

[0003] Patent Document 1 discloses extracting a hand, which is a part where a gesture is made, from an image obtained by imaging a user's hand or finger with a camera, specifying the shape of the extracted hand, and recognizing the gesture.

[0004] Patent Document 2 discloses using deep learning for gesture recognition.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when recognizing gestures using both hands, depending on the situation, the gesture may not be recognized with high accuracy (the recognition of the gesture becomes unstable). For example, when the right hand and the left hand overlap, it may not be possible to accurately separate the right hand and the left hand, detect joints, etc. (key points for gesture recognition), or specify (recognize) the shape of the hand, etc. (the part where the gesture is made) with high accuracy (become unstable).

[0007] The present invention aims to provide a technology that can recognize gestures with high accuracy under various circumstances. [Means for solving the problem]

[0008] A first aspect of the present invention is a gesture recognition device comprising: a first detection means for detecting a first part in which a gesture is performed from an captured image; a second detection means for detecting a second part in which the gesture is performed at the first part detected by the first detection means from the captured image; and a recognition means for recognizing the gesture based on the movement of the first part detected by the first detection means and the movement of the second part detected by the second detection means, wherein if the second detection means does not obtain a detection result that satisfies predetermined conditions, the recognition means recognizes the gesture using a detection result previously obtained by the second detection means that satisfies the predetermined conditions.

[0009] A second aspect of the present invention includes: a first detection step of detecting a first part that performs a gesture from an captured image; a second detection step of detecting a second part that performs the gesture at the first part detected in the first detection step from the captured image; and a recognition step of recognizing the gesture based on the movement of the first part detected in the first detection step and the movement of the second part detected in the second detection step. Furthermore, if a detection result satisfying predetermined conditions is not obtained in the second detection step, the recognition step is characterized in that the gesture is recognized using the detection result that satisfies the predetermined conditions obtained in the previous second detection step.

[0010] A third aspect of the present invention is a head-mounted display device characterized by having a display control means that controls the display based on the recognition result of the gesture recognition device. A fourth aspect of the present invention is a program for causing a computer to function as each of the means of the gesture recognition device. A fifth aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as each of the means of the gesture recognition device. [Effects of the Invention]

[0011] According to the present invention, gestures can be recognized with high accuracy in various environments. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example configuration of a gesture recognition device. [Figure 2] This figure shows an example of hand and joint detection. [Figure 3] This is a flowchart illustrating an example of gesture recognition processing. [Figure 4] This diagram illustrates an example of gesture recognition processing. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will now be described. Figure 1 is a block diagram showing an example of the configuration of the gesture recognition device 101 in this embodiment. The gesture recognition device 101 includes an imaging unit 102, a system control unit 103, a non-volatile memory 104, a system memory 105, and a gesture recognition unit 110.

[0014] The imaging unit 102 has an optical system with multiple lenses and an image sensor (e.g., a CCD or CMOS sensor), which are arranged so as to face the direction in which user gestures can be captured. In this embodiment, an image sensor is used, but any sensor capable of recognizing gestures is acceptable, and a distance measuring sensor (e.g., a laser-based sensor such as LiDAR (Light Detection and Ranging)) may also be used.

[0015] The non-volatile memory 104 is an electrically erasable and recordable memory, such as Flash-ROM. The non-volatile memory 104 stores constants and programs for the operation of the system control unit 103. The program referred to here is, for example, a program for executing each process in the flowchart described later in this embodiment.

[0016] System memory 105 is, for example, RAM, and stores variables for the operation of the system control unit 103, as well as constants and programs read from non-volatile memory 104.

[0017] The system control unit 103 is a control unit consisting of at least one processor or circuit, and controls the entire gesture recognition device 101. The system control unit 103 executes the program stored in the non-volatile memory 104 described above to realize each process in the flowchart described later. The system control unit 103 generates a detection image by applying image processing such as noise reduction and resizing to the captured image obtained from the imaging unit 102, and records the detection image in the system memory 105. The system control unit 103 also executes processing according to the gesture recognized by the gesture recognition unit 110. For example, the system control unit 103 generates a control signal corresponding to the recognized gesture and uses this control signal to control each part of the gesture recognition device 101.

[0018] The gesture recognition unit 110 includes a target part detection unit 111, a detailed part detection unit 112, a detailed part determination unit 113, and a gesture detection unit 114, and recognizes gestures based on the captured image obtained from the imaging unit 102 (specifically, the above-described detection image).

[0019] The gesture recognition unit 110 will be described in detail with reference to FIG. 2. FIG. 2 is a diagram showing a detection example of a hand and joints in the present embodiment.

[0020] The image 200 in FIG. 2 is a detection image obtained by performing image processing on the captured image obtained from the imaging unit 102. In the detection image 200, the left hand 211 and the right hand 212 of the user performing a gesture (gesture operation) are shown. The image 201 in FIG. 2 is an image obtained by superimposing the detection results of the target part detection unit 111 and the detailed part detection unit 112 on the detection image 200.

[0021] The target part detection unit 111 detects (extracts) the hand, which is the target part for performing a gesture, from the detection image 200. In FIG. 2, for each of the left hand 211 and the right hand 212, a hand area (area of the hand) is detected. The hand area is, for example, a rectangular area surrounding the hand with the center position of the hand as the center. For the left hand 211, the hand area 221 is detected, and for the right hand 212, the hand area 222 is detected.

[0022] The detailed area detection unit 112 detects (extracts) the joints, which are the detailed areas where gestures are performed, from the detected image 200 in the target area detected by the target area detection unit 111. In Figure 2, 21 joint positions (joint locations) from the wrist to each fingertip are detected for both the left hand 211 and the right hand 212. The target area detection unit 111 obtains joint information based on the 21 joint positions. Joint information 231 is obtained for the left hand 211, and joint information 232 is obtained for the right hand 212. Joint information 231 and 232 show the 21 joint positions and multiple line segments connecting the 21 joint positions to form a skeleton. The joint positions shown by the joint information are, for example, relative to the hand region. The joint positions shown by joint information 231 are relative to the hand region 221, and the joint positions shown by joint information 232 are relative to the hand region 222. In this embodiment, the user performs gestures with their hands and fingers, but gestures may also be performed with the body, arms, legs, face, eyes, mouth, etc. The target area may be the body, and the detailed area may be the arm or arm joint, etc. The target area may be the face, and the detailed area may be the eye, etc. Furthermore, if the detected image is large (the number of pixels in the detected image is large), it will take a long time to detect the detailed area. If a smaller detected image (a detected image with low resolution (pixel density)) is obtained by resizing, the time required to detect the detailed area will be shortened, but the accuracy of the detection will decrease. For this reason, the region of the target area (hand region 221 or hand region 222) may be cut out from the detected image, and the detailed area may be detected from the image of the cut-out region (cut-out image). The resolution of the cut-out image is the same as the detected image, but the cut-out image is smaller than the detected image (the number of pixels in the cut-out image is less than that of the detected image). Therefore, by detecting the detailed area from the cut-out image, it becomes possible to detect the detailed area in a short time and with high accuracy.

[0023] The detailed part determination unit 113 determines (calculates) the reliability of the detection results (joint information) obtained by the detailed part detection unit 112. Based on whether the reliability is above the threshold TH, the detailed part determination unit 113 determines the joint information to be used by the gesture detection unit 114. Details (specific examples) of how to determine the reliability and how to determine the joint information to be used by the gesture detection unit 114 will be described later.

[0024] The gesture detection unit 114 detects (recognizes) gestures based on the movement of the target area (hand) detected by the target area detection unit 111 and the movement of the detailed area (joint) detected by the detailed area detection unit 112. In this embodiment, the gesture detection unit 114 detects (recognizes) the target area The gesture detection unit 114 detects gestures using the detection result (hand region) from the body part detection unit 111 and the determination result (joint information) from the detailed body part determination unit 113. The gesture detection unit 114 detects gestures by comparing the detection result (hand region) from the target body part detection unit 111 and the determination result (joint information) from the detailed body part determination unit 113 with a gesture model that has been stored in the non-volatile memory 104 in advance. The gesture detection method is not limited to this, and gestures may be detected using a detector trained by deep learning or the like. When using deep learning, by using an RNN (recurrent neural network), it becomes possible to detect gestures that cannot be detected from data of a single frame (such as a gesture of drawing a circle with the fingers) from time-series data of joint information. The gesture detection unit 114 is not limited to detecting only one gesture, and may be capable of detecting multiple gestures.

[0025] The gesture recognition process in this embodiment will now be described. Figure 3 is a flowchart illustrating an example of the gesture recognition process in this embodiment. This process is achieved by the system control unit 103 loading a program stored in the non-volatile memory 104 into the system memory 105 and executing it. For example, when the gesture recognition device 101 is started, the process in Figure 3 begins. Figure 4 is a diagram illustrating an example of the gesture recognition process in this embodiment. In Figure 4, the detected images 401 to 403 are arranged in chronological order. Detected image 401 is the oldest, and detected image 403 is the newest. Detected images 401 to 403 show the left hand 405 and the right hand 411. The process for the right hand 411 will be described below, but the process for the right hand 411 and the process for the left hand 405 may be performed in parallel or sequentially so that gestures of multiple hands can be recognized. The process for the left hand 405 is the same as the process for the right hand 411.

[0026] In step S301, the system control unit 103 acquires an image from the imaging unit 102 and generates (acquires) a detection image 401 by performing image processing on the image. Then, the system control unit 103 uses the target area detection unit 111 to detect the hand region 412 from the generated detection image 401.

[0027] In step S302, the system control unit 103 determines whether or not the hand region 412 was detected in step S301. If the hand region 412 was detected, the process proceeds to step S303; otherwise, it returns to step S301. Here, we assume that the hand region 412 was detected and the process proceeds to step S303.

[0028] In step S303, the system control unit 103 uses the detailed part detection unit 112 to acquire joint information 413 from the hand region 412 detected in step S301.

[0029] In step S304, the system control unit 103 uses the detailed part determination unit 113 to determine (calculate) the reliability of the joint information 413 (detection result of the detailed part detection unit 112) acquired in step S303. Based on the time-series data of the joint information, if the amount of joint movement (movement speed) in the detected image is determined to be greater than threshold TH1, for example, if it is determined that the joint moved at an unexpected size (speed), there is a high possibility that incorrect joint information has been acquired. If the interval between multiple joints in the detected image is determined to be longer than threshold TH2, for example, if the interval between joints is determined to be an unexpected length, there is also a high possibility that incorrect joint information has been acquired. If the proportion of the hand that is visible in the detected image (for example, the ratio of the number of pixels of the right hand 411 to the number of pixels of the hand region 412) is smaller than threshold TH3, for example, if a large part of the hand is hidden by another hand, there is also a high possibility that incorrect joint information has been acquired. Therefore, in these cases, the reliability may be calculated to be lower. Furthermore, the detailed part detection unit 112 may output a score indicating its reliability along with the detection result, similar to a detector trained using deep learning.

[0030] In step S305, the system control unit 103 uses the detailed part determination unit 113 to determine whether the confidence level determined (calculated) in step S304 is equal to or greater than the threshold TH. If the confidence level is equal to or greater than the threshold TH, the system control unit 103 decides to use the joint information 413 acquired in step S303 as the joint information to be used by the gesture detection unit 114 and stores it in the system memory 105. Then, the process proceeds to step S306. If the confidence level is less than the threshold TH, the system control unit 103 decides to use the joint information stored in the system memory 105, that is, the joint information with a confidence level equal to or greater than the threshold TH that was previously obtained by the detailed part detection unit 112, as the joint information to be used by the gesture detection unit 114. Then, the process proceeds to step S307. For example, the confidence level will be lower than the threshold TH if the amount of joint movement in the detected image is greater than the threshold TH1, if the distance between multiple joints in the detected image is longer than the threshold TH2, or if the proportion of the hand that is visible in the detected image is smaller than the threshold TH3. Although the condition that the confidence level is equal to or greater than the threshold TH was used, other predetermined conditions may be used if the condition is met when joint information can be detected with high accuracy (accurately) and not met otherwise. By switching the joint information used by the gesture detection unit 114 depending on whether the predetermined condition is met or not, the use of incorrect joint information by the gesture detection unit 114 can be suppressed, and consequently, false detection of gestures can be suppressed. Here, we assume that the confidence level is equal to or greater than the threshold TH and proceed to step S306.

[0031] In step S306, the system control unit 103 uses the gesture detection unit 114 to detect (recognize) a gesture from the hand region 412 detected in step S301 and the joint information 413 acquired in step S303.

[0032] In step S309, the system control unit 103 determines whether or not to terminate the gesture recognition process. If it decides to terminate the gesture recognition process, it terminates the process; otherwise, it returns to step S301. Here, we assume it returns to step S301. In step S301, the system control unit 103 acquires a new detection image 402 and detects the hand region 422. In step S302, the system control unit 103 determines whether or not the hand region 422 was detected. Here, we assume that the hand region 422 was detected and proceed to step S303. In step S303, the system control unit 103 acquires joint information 423 and determines (calculates) the reliability of the joint information 423 (detection result of the detailed part detection unit 112) in step S304. Then, in step S305, the system control unit 103 determines whether or not the reliability of the joint information 423 is greater than or equal to the threshold TH. Here, we assume that the reliability is less than the threshold TH and proceed to step S307.

[0033] In step S307, the system control unit 103 retrieves past joint information (joint information with a confidence level of threshold TH or higher) from the system memory 105. Then, it proceeds to step S308. If no joint information is stored in the system memory 105 (i.e., joint information with a confidence level of threshold TH or higher has not been obtained in the past), it returns to step S301 so that the gesture is not recognized. Here, it is assumed that joint information 413 is stored in the system memory 105, and the system retrieves joint information 413 from the system memory 105 and proceeds to step S308. If multiple joint information is stored in the system memory 105, the latest joint information may be retrieved. Even if joint information is stored in the system memory 105, if there is a high possibility that the gesture cannot be recognized with high accuracy, the system may return to step S301 so that the gesture is not recognized (detected). For example, if only joint information older than a predetermined time prior to the present is stored (i.e., joint information with a confidence level of threshold TH or higher has not been obtained after a predetermined time prior to the present), the system may return to step S301. If the amount of movement of the right hand 411 in the detected image (for example, the amount of movement of the center position of the right hand 411) is greater than the threshold TH4, the process may return to step S301.

[0034] In step S308, the system control unit 103 uses the gesture detection unit 114 to: Gestures are detected (recognized) from the hand region 422 detected in step S301 and the past joint information 413 acquired in step S307.

[0035] In step S309, the system control unit 103 determines whether or not to terminate the gesture recognition process. If it decides to terminate the gesture recognition process, it terminates the process; otherwise, it returns to step S301. Here, we assume it returns to step S301. In step S301, the system control unit 103 acquires a new detection image 403 and detects the hand region 432. In step S302, the system control unit 103 determines whether or not the hand region 432 was detected. Here, we assume that the hand region 432 was detected and proceed to step S303. In step S303, the system control unit 103 acquires joint information 433, and in step S304, it determines (calculates) the reliability of the joint information 433 (detection result of the detailed part detection unit 112).

[0036] As described above, according to this embodiment, if the detailed part detection unit 112 does not obtain a detection result that satisfies the predetermined conditions, the gesture is recognized using a detection result that satisfies the predetermined conditions previously obtained by the detailed part detection unit 112. In this way, gestures can be recognized with high accuracy in various environments.

[0037] It should be noted that the above embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configuration of the above embodiments within the scope of the gist of the present invention are also included in the present invention. For example, the gesture recognition device 101 may be provided in an electronic device (head-mounted display device) such as smart glasses that support AR (augmented reality). In this case, the display control unit of the head-mounted display device controls the display based on the recognition result of the gesture recognition device 101. The gesture recognition device 101 (gesture recognition unit 110) may be an electronic device separate from the head-mounted display device, for example, a computer (server) on the cloud.

[0038] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0039] 101: Gesture recognition device 103: System control unit 110: Gesture recognition unit 111: Target area detection unit 112: Detailed area detection unit 113: Detailed area determination unit 114: Gesture detection unit

Claims

1. A first detection means for detecting a first portion from a captured image; a second detection means for detecting a second part in the first part detected by the first detection means from the captured image; a recognition means for recognizing a gesture based on the movement of the first part detected by the first detection means and the movement of the second part detected by the second detection means; having When the second detection means does not detect a second part satisfying a predetermined condition from the captured image at the predetermined time, the recognition means recognizes a gesture using the second part satisfying the predetermined condition detected by the second detection means from the captured image before the predetermined time and the first part detected by the first detection means from the captured image at the predetermined time. A gesture recognition device comprising:

2. When the amount of movement of the second part in the captured image at the predetermined time point is greater than a threshold value, the recognition means uses the second part that satisfies the predetermined condition and is detected by the second detection means from the captured image before the predetermined time point. The gesture recognition device according to claim 1 .

3. the second detection means detects a plurality of second sites; When an interval between the plurality of second parts in the captured image at the predetermined time point is longer than a threshold value, the recognition means uses the second part that satisfies the predetermined condition and was detected by the second detection means before the predetermined time point.

3. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

4. The first detection means detects a plurality of first regions; When a ratio of a portion of a predetermined first portion that is hidden by overlapping with another first portion to an area of ​​the predetermined first portion is greater than a threshold value, the recognition means uses a second portion that satisfies the predetermined condition and was detected by the second detection means before the predetermined time point.

4. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

5. The method further includes a determination unit for determining the reliability of the second portion detected by the second detection unit, When the reliability determined by the determination means is lower than a threshold value, the recognition means uses a second portion having a reliability higher than the threshold value detected by the second detection means from an image captured before the predetermined time point.

5. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

6. The second detection means cuts out a region of the first part detected by the first detection means from the captured image, and detects the second part from an image of the cut-out region. The gesture recognition device according to any one of claims 1 to 5.

7. a generating means for generating a control signal corresponding to the gesture recognized by the recognition means. The gesture recognition device according to any one of claims 1 to 6.

8. When the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time, the recognition means uses the latest second part that satisfies the predetermined condition and is detected by the second detection means from an image captured before the predetermined time. The gesture recognition device according to any one of claims 1 to 7.

9. Even if the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time, if the second detection means has not previously detected a second part that satisfies the predetermined condition, the recognition means does not recognize the gesture. The gesture recognition device according to any one of claims 1 to 8.

10. Even if the second detection means does not detect a second part satisfying the predetermined condition from the captured image at the predetermined time point, if the second detection means does not obtain a second part satisfying the predetermined condition after a timing that is a predetermined time before the predetermined time point, the recognition means does not recognize the gesture. The gesture recognition device according to any one of claims 1 to 9.

11. Even if the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time, if the amount of movement of the first part in the captured image at the predetermined time is greater than a threshold value, the recognition means does not recognize the gesture. The gesture recognition device according to any one of claims 1 to 10.

12. The first part is a hand and the second part is a joint. The gesture recognition device according to any one of claims 1 to 11.

13. The method further comprises recording means for recording the detection result of the second part detected by the second detection means in a recording unit, The recognition means acquires from the recording unit a detection result of a second portion that satisfies the predetermined condition and is detected from an image captured before the predetermined time point. The gesture recognition device according to any one of claims 1 to 12.

14. When the first detection means does not detect the first portion, the recognition means Does not recognize the source The gesture recognition device according to any one of claims 1 to 13.

15. When the second detection means does not detect a second part satisfying the predetermined condition from the image captured at the predetermined time, the recognition means uses a second part that satisfies the predetermined condition and is detected by the second detection means from an image captured prior to the predetermined time as the second part in the first part detected from the image captured at the predetermined time. The gesture recognition device according to any one of claims 1 to 14.

16. The first detection means detects a rectangular area including the first part from the captured image, The second detection means detects a point corresponding to the second portion from the captured image. The gesture recognition device according to any one of claims 1 to 15.

17. The first portion includes a plurality of the second portions. The gesture recognition device according to any one of claims 1 to 16.

18. The first portion and the second portion have different movements. The gesture recognition device according to any one of claims 1 to 17.

19. The recognition means detects a movement of a first part based on a first part detected from a plurality of captured images, and detects a movement of a second part based on a second part detected from the plurality of captured images. The gesture recognition device according to any one of claims 1 to 18.

20. A display control means for controlling display based on the recognition result of the gesture recognition device according to any one of claims 1 to 19. A head-mounted display device comprising:

21. a first detection step of detecting a first portion from the captured image; a second detection step of detecting a second part in the first part detected in the first detection step from the captured image; a recognition step of recognizing a gesture based on the movement of the first part detected in the first detection step and the movement of the second part detected in the second detection step; having When a second part satisfying a predetermined condition is not detected from the captured image at the predetermined time in the second detection step, the recognition step recognizes a gesture using the second part satisfying the predetermined condition detected from the captured image before the predetermined time in the second detection step and the first part detected from the captured image at the predetermined time in the first detection step. A gesture recognition method comprising:

22. A program for causing a computer to function as each of the means of the gesture recognition device according to any one of claims 1 to 19.

23. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the gesture recognition device according to any one of claims 1 to 19.

Images