Gesture recognition device, head-mounted display device, gesture recognition method, program, and storage medium

Abstract

To provide a technology capable of recognizing gestures with high accuracy under various situations.SOLUTION: A gesture recognition apparatus includes: first detection means configured to detect from a captured image a first portion making a gesture; second detection means configured to detect from the captured image a second portion making the gesture in the first portion detected by the first detection means; and recognition means configured to recognize the gesture on the basis of a motion of the first portion detected by the first detection means and a motion of the second portion detected by the second detection means. In a case where a detection result satisfying a predetermined condition is not obtained by the second detection means, the recognition means recognizes the gesture using a detection result satisfying the predetermined condition obtained in the past by the second detection means.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for recognizing gestures. [Background technology]

[0002] A technology has been proposed that recognizes gestures made by a user's hands or fingers and performs processing according to the recognized gestures. With such a technology, a user can operate an electronic device (an input device of the electronic device) by using gestures without touching the electronic device.

[0003] Patent Document 1 discloses that the hand, which is the part making the gesture, is extracted from an image of the user's hand, fingers, etc. captured by a camera, and the shape of the extracted hand is identified to recognize the gesture.

[0004] Patent Document 2 discloses the use of deep learning for gesture recognition. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-172887 [Patent Document 2] Japanese Patent Application Publication No. 2019-71048 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when recognizing gestures using both hands, depending on the situation, the gesture may not be recognized with high accuracy (gesture recognition may become unstable). For example, when the right and left hands are overlapping, it may not be possible to separate the right and left hands, detect joints (key points for gesture recognition), or identify (recognize) the shape of the hand (the part making the gesture) with high accuracy (become unstable).

[0007] An object of the present invention is to provide a technology that can recognize gestures with high accuracy under various circumstances. [Means for solving the problem]

[0008] The first aspect of the present invention is to , th a first detecting means for detecting a first part; and a second detecting means for detecting a first part from the captured image. , th a second detection means for detecting two regions, and a second detection means for detecting the movement of the first region based on the movement of the first region detected by the first detection means and the movement of the second region detected by the second detection means; , Ji recognition means for recognizing a gesture; From the captured image at a predetermined time, The second detection means detects whether a predetermined condition is met. The second part detection difference If the detection result is not received, the recognition means , from the predetermined time past Detected from the captured image The above-mentioned predetermined conditions are satisfied. a second region; and a first region detected by the first detection means from the captured image at the predetermined time point. Using , Ji The gesture recognition device is characterized by recognizing gestures.

[0009] The second aspect of the present invention is to , th a first detection step of detecting a first part; and detecting from the captured image the first part detected in the first detection step. , th a second detection step of detecting two regions, and a second detection step of detecting the movement of the first region based on the movement of the first region detected in the first detection step and the movement of the second region detected in the second detection step. , Ji a recognition step of recognizing a gesture, From the captured image at a predetermined time, In the second detection step, a predetermined condition is satisfied. The second part detection difference If not, the recognition step , th In the second detection step , detected from the captured image from the previous time point. The above-mentioned predetermined conditions are satisfied. a second region; and a first region detected from the captured image at the predetermined time point in the first detection step. Using , Ji A gesture recognition method is characterized by recognizing a gesture.

[0010] A third aspect of the present invention is a head-mounted display device having display control means for controlling display based on the recognition result of the gesture recognition device. A fourth aspect of the present invention is a program for causing a computer to function as each means of the gesture recognition device. A fifth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the gesture recognition device. [Effects of the Invention]

[0011] According to the present invention, gestures can be recognized with high accuracy in various environments. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a gesture recognition device. [Figure 2] FIG. 10 is a diagram illustrating an example of detection of hands and joints. [Figure 3] 10 is a flowchart illustrating an example of a gesture recognition process. [Figure 4] FIG. 10 is a diagram illustrating an example of gesture recognition processing. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of the present invention will be described. Fig. 1 is a block diagram showing an example of the configuration of a gesture recognition device 101 according to this embodiment. The gesture recognition device 101 includes an imaging unit 102, a system control unit 103, a nonvolatile memory 104, a system memory 105, and a gesture recognition unit 110.

[0014] The imaging unit 102 has an optical system with multiple lenses and an image sensor (for example, a CCD or CMOS sensor), and is arranged so as to face a direction in which it can capture an image of a gesture by a user. In this embodiment, an image sensor is used, but any sensor that can be used to recognize a gesture may be used, and a ranging sensor (for example, a sensor that uses a laser such as LiDAR (Light Detection and Ranging)) may also be used.

[0015] The nonvolatile memory 104 is an electrically erasable and recordable memory, such as a Flash-ROM. The nonvolatile memory 104 stores constants and programs for the operation of the system control unit 103. The programs referred to here are, for example, programs for executing the processes of the flowcharts described later in this embodiment.

[0016] The system memory 105 is, for example, a RAM, and stores variables for the operation of the system control unit 103, constants and programs read from the nonvolatile memory 104, and the like.

[0017] The system control unit 103 is a control unit including at least one processor or circuit, and controls the entire gesture recognition device 101. The system control unit 103 executes programs stored in the nonvolatile memory 104 described above to realize each process of the flowchart described below. The system control unit 103 generates a detected image by performing image processing such as noise removal and resizing on the captured image obtained from the imaging unit 102, and records the detected image in the system memory 105. The system control unit 103 also executes processing according to the gesture recognized by the gesture recognition unit 110. For example, the system control unit 103 generates a control signal corresponding to the recognized gesture and controls each unit of the gesture recognition device 101 using the control signal.

[0018] The gesture recognition unit 110 has a target part detection unit 111, a detailed part detection unit 112, a detailed part judgment unit 113, and a gesture detection unit 114, and recognizes gestures based on the captured image (specifically, the detected image described above) obtained from the imaging unit 102.

[0019] The gesture recognition unit 110 will be described in detail with reference to Fig. 2. Fig. 2 is a diagram showing an example of detection of hands and joints in this embodiment.

[0020] 2 is a detected image obtained by performing image processing on a captured image obtained from the imaging unit 102. The detected image 200 shows a left hand 211 and a right hand 212 of a user making a gesture (gesture operation). The image 201 in FIG. 2 is an image obtained by superimposing the detection results of the target part detection unit 111 and the detailed part detection unit 112 on the detected image 200.

[0021] The target part detection unit 111 detects (extracts) the hand, which is the target part for making a gesture, from the detected image 200. In FIG. 2, a hand region (hand area) is detected for each of the left hand 211 and the right hand 212. The hand region is, for example, a rectangular region that surrounds the hand with the center position of the hand as the center. A hand region 221 is detected for the left hand 211, and a hand region 222 is detected for the right hand 212.

[0022] The detailed part detection unit 112 detects (extracts) joints, which are detailed parts for making a gesture, from the target part detected by the target part detection unit 111, from the detected image 200. In FIG. 2, 21 joint positions (joint positions) from the wrist to each fingertip are detected for each of the left hand 211 and the right hand 212. The target part detection unit 111 obtains joint information based on the 21 joint positions. Joint information 231 is obtained for the left hand 211, and joint information 232 is obtained for the right hand 212. The joint information 231 and 232 indicate the 21 joint positions and a plurality of line segments connecting the 21 joint positions so as to form a skeleton. The joint positions indicated by the joint information are, for example, relative positions with respect to the hand region. The joint positions indicated by the joint information 231 are relative positions with respect to the hand region 221, and the joint positions indicated by the joint information 232 are relative positions with respect to the hand region 222. In this embodiment, the user makes gestures using their hands or fingers. However, gestures may also be made using the body, arms, legs, face, eyes, mouth, etc. The target part may be the body, and the detailed part may be the arm or arm joint. The target part may be the face, and the detailed part may be the eye. Furthermore, when the detection image is large (the number of pixels in the detection image is large), it takes a long time to detect the detailed part. If a small detection image (a detection image with low resolution (pixel density)) is obtained by resizing, the time required to detect the detailed part is shortened, but the accuracy of the detection decreases. Therefore, the region of the target part (hand region 221 or hand region 222) may be cut out from the detection image, and the detailed part may be detected from an image of the cut-out region (cut-out image). The resolution of the cut-out image is the same as that of the detection image, but the cut-out image is smaller than the detection image (the number of pixels in the cut-out image is smaller than that of the detection image). Therefore, detecting the detailed part from the cut-out image enables the detailed part to be detected in a short time with high accuracy.

[0023] The detailed part determination unit 113 determines (calculates) the reliability of the detection result (joint information) obtained by the detailed part detection unit 112. Then, the detailed part determination unit 113 determines the joint information to be used in the gesture detection unit 114 based on whether the reliability is equal to or greater than a threshold value TH. Details (specific examples) of a method for determining the reliability and a method for determining the joint information to be used in the gesture detection unit 114 will be described later.

[0024] The gesture detection unit 114 detects (recognizes) a gesture based on the movement of the target part (hand) detected by the target part detection unit 111 and the movement of the detailed part (joint) detected by the detailed part detection unit 112. In this embodiment, the gesture detection unit 114 detects (recognizes) a gesture based on the movement of the target part (hand) detected by the target part detection unit 111 and the movement of the detailed part (joint) detected by the detailed part detection unit 112. A gesture is detected using the detection result (hand region) of the part detection unit 111 and the determination result (joint information) of the detailed part determination unit 113. The gesture detection unit 114 detects a gesture by, for example, comparing the detection result (hand region) of the target part detection unit 111 and the determination result (joint information) of the detailed part determination unit 113 with a gesture model previously stored in the non-volatile memory 104. Note that the gesture detection method is not limited to this, and gestures may be detected using a detector trained by deep learning or the like. When deep learning is used, a recurrent neural network (RNN) may be used to detect gestures that cannot be detected from one frame of data (such as a gesture of drawing a circle with a finger) from the time-series data of joint information. The gesture detection unit 114 is not limited to being able to detect one gesture, and may be able to detect multiple gestures.

[0025] The gesture recognition process according to this embodiment will be described. FIG. 3 is a flowchart showing an example of the gesture recognition process according to this embodiment. This process is realized by the system control unit 103 loading a program stored in the nonvolatile memory 104 into the system memory 105 and executing the program. For example, when the gesture recognition device 101 is started, the process of FIG. 3 starts. FIG. 4 is a diagram showing an example of the gesture recognition process according to this embodiment. In FIG. 4, detected images 401 to 403 are arranged in chronological order. The detected image 401 is the oldest, and the detected image 403 is the newest. The detected images 401 to 403 show a left hand 405 and a right hand 411. The process for the right hand 411 will be described below, but the process for the right hand 411 and the process for the left hand 405 may be performed in parallel or sequentially so that gestures of multiple hands can be recognized. The process for the left hand 405 is the same as the process for the right hand 411.

[0026] In step S301, the system control unit 103 acquires a captured image from the imaging unit 102 and performs image processing on the captured image to generate (acquire) a detected image 401. Then, the system control unit 103 uses the target part detection unit 111 to detect a hand region 412 from the generated detected image 401.

[0027] In step S302, the system control unit 103 determines whether or not the hand region 412 was detected in step S301. If the hand region 412 was detected, the process proceeds to step S303; otherwise, the process returns to step S301. Here, it is assumed that the hand region 412 was detected, and the process proceeds to step S303.

[0028] In step S303, the system control unit 103 uses the detailed part detection unit 112 to acquire joint information 413 from the hand region 412 detected in step S301.

[0029] In step S304, the system control unit 103 uses the detailed part determination unit 113 to determine (calculate) the reliability of the joint information 413 (detection result of the detailed part detection unit 112) acquired in step S303. If it is determined based on the time-series data of the joint information that the movement amount (movement speed) of the joint in the detected image is greater than a threshold value TH1, for example, if it is determined that the joint has moved at an unexpected magnitude (speed), it is highly likely that erroneous joint information has been acquired. If it is determined that the interval between multiple joints in the detected image is longer than a threshold value TH2, for example, if it is determined that the interval between the joints is an unexpected length, it is also highly likely that erroneous joint information has been acquired. If the proportion of the part of the hand that appears in the detected image (for example, the proportion of the number of pixels of the right hand 411 to the number of pixels of the hand region 412) is smaller than a threshold value TH3, for example, if most of the hand is hidden by another hand, it is also highly likely that erroneous joint information has been acquired. Therefore, in these cases, the reliability may be calculated to be low. The detailed part detection unit 112 may output a score indicating the reliability of the detection result together with the detection result, like a detector trained by deep learning or the like.

[0030] In step S305, the system control unit 103, using the detailed part determination unit 113, determines whether the reliability determined (calculated) in step S304 is equal to or greater than a threshold value TH. If the reliability is equal to or greater than the threshold value TH, the system control unit 103 determines the joint information 413 acquired in step S303 as the joint information to be used by the gesture detection unit 114 and stores it in the system memory 105. Then, the process proceeds to step S306. If the reliability is less than the threshold value TH, the system control unit 103 determines the joint information stored in the system memory 105, that is, the joint information previously obtained by the detailed part detection unit 112 and having a reliability equal to or greater than the threshold value TH, as the joint information to be used by the gesture detection unit 114. Then, the process proceeds to step S307. For example, if the movement amount of the joint in the detected image is greater than a threshold value TH1, if the interval between multiple joints in the detected image is greater than a threshold value TH2, or if the proportion of the part of the hand that appears in the detected image is smaller than a threshold value TH3, the reliability is lower than the threshold value TH. Although the condition that the reliability is equal to or greater than the threshold value TH is used, another predetermined condition may be used as long as the condition is satisfied when the joint information is detected with high accuracy (accuracy) and is not satisfied otherwise. By switching the joint information used by the gesture detection unit 114 depending on whether the predetermined condition is satisfied, it is possible to prevent the gesture detection unit 114 from using incorrect joint information, and ultimately to prevent erroneous detection of a gesture. Here, it is assumed that the reliability is equal to or greater than the threshold value TH, and the process proceeds to step S306.

[0031] In step S306, the system control unit 103 uses the gesture detection unit 114 to detect (recognize) a gesture from the hand region 412 detected in step S301 and the joint information 413 acquired in step S303.

[0032] In step S309, the system control unit 103 determines whether or not to end the gesture recognition process. If the gesture recognition process is to be ended, the gesture recognition process is ended; if not, the process returns to step S301. Here, it is assumed that the process returns to step S301. In step S301, the system control unit 103 acquires a new detected image 402 and detects a hand region 422. In step S302, the system control unit 103 determines whether or not the hand region 422 has been detected. Here, it is assumed that the hand region 422 has been detected, and the process proceeds to step S303. In step S303, the system control unit 103 acquires joint information 423, and determines (calculates) the reliability of the joint information 423 (the detection result of the detailed part detection unit 112) in step S304. Then, in step S305, the system control unit 103 determines whether or not the reliability of the joint information 423 is equal to or greater than a threshold value TH. Here, it is assumed that the reliability is less than the threshold value TH, and the process proceeds to step S307.

[0033] In step S307, the system control unit 103 acquires past joint information (joint information with a reliability equal to or greater than the threshold value TH) from the system memory 105. Then, the process proceeds to step S308. If no joint information is stored in the system memory 105 (if no joint information with a reliability equal to or greater than the threshold value TH has been obtained in the past), the process returns to step S301 so as not to recognize the gesture. Here, it is assumed that joint information 413 is stored in the system memory 105, and the joint information 413 is acquired from the system memory 105, and the process proceeds to step S308. Note that if multiple pieces of joint information are stored in the system memory 105, the latest joint information may be acquired. Even if joint information is stored in the system memory 105, if it is highly likely that the gesture cannot be recognized with high accuracy, the process may return to step S301 so as not to recognize (detect) the gesture. For example, if only joint information older than a timing a predetermined time before the present is stored (if no joint information with a reliability equal to or greater than the threshold value TH has been obtained since a timing a predetermined time before), the process may return to step S301. If the amount of movement of the right hand 411 in the detected image (for example, the amount of movement of the center position of the right hand 411) is greater than the threshold value TH4, the process may also return to step S301.

[0034] In step S308, the system control unit 103 uses the gesture detection unit 114 to A gesture is detected (recognized) from the hand region 422 detected in step S301 and the past joint information 413 acquired in step S307.

[0035] In step S309, the system control unit 103 determines whether or not to end the gesture recognition process. If the gesture recognition process is to be ended, the gesture recognition process is ended; if not, the process returns to step S301. Here, it is assumed that the process returns to step S301. In step S301, the system control unit 103 acquires a new detected image 403 and detects a hand region 432. In step S302, the system control unit 103 determines whether or not the hand region 432 has been detected. Here, it is assumed that the hand region 432 has been detected, and the process proceeds to step S303. In step S303, the system control unit 103 acquires joint information 433, and in step S304, determines (calculates) the reliability of the joint information 433 (the detection result of the detailed part detection unit 112).

[0036] As described above, according to this embodiment, when the detailed part detection section 112 does not obtain a detection result that satisfies a predetermined condition, a gesture is recognized using a detection result that satisfies the predetermined condition and that was obtained in the past by the detailed part detection section 112. In this way, gestures can be recognized with high accuracy in various environments.

[0037] Note that the above-described embodiment is merely an example, and configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention are also included in the present invention. For example, the gesture recognition device 101 may be provided in an electronic device (head-mounted display device) such as smart glasses compatible with AR (augmented reality). In this case, a display control unit of the head-mounted display device controls the display based on the recognition result of the gesture recognition device 101. The gesture recognition device 101 (gesture recognition unit 110) may be an electronic device separate from the head-mounted display device, and may be, for example, a computer (server) on the cloud.

[0038] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0039] 101: Gesture recognition device 103: System control unit 110: Gesture recognition unit 111: Target part detection unit 112: Detailed part detection unit 113: Detailed part determination unit 114: Gesture detection unit

Claims

1. a first detection means for detecting a first region from the captured image; a second detection means for detecting a second part in the first part detected by the first detection means from the captured image; recognition means for recognizing a gesture based on the movement of the first part detected by the first detection means and the movement of the second part detected by the second detection means; and When the second detection means does not detect a second part that satisfies a predetermined condition from the captured image at the predetermined time, the recognition means recognizes the gesture using the second part that satisfies the predetermined condition and that is detected by the second detection means from the captured image before the predetermined time and the first part that is detected by the first detection means from the captured image at the predetermined time. A gesture recognition device characterized by:

2. When the movement amount of the second part in the captured image at the predetermined time point is larger than a threshold value, the recognition means uses the second part that satisfies the predetermined condition and that is detected by the second detection means from the captured image before the predetermined time point. The gesture recognition device according to claim 1 .

3. the second detection means detects a plurality of second regions; When the interval between the plurality of second regions in the captured image at the predetermined time point is longer than a threshold value, the recognition means uses the second region that was detected by the second detection means before the predetermined time point and satisfies the predetermined condition.

3. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

4. The first detection means detects a plurality of first regions, When a ratio of a portion of a predetermined first portion that is hidden by overlapping with another first portion to an area of ​​the predetermined first portion is greater than a threshold, the recognition means uses a second portion that was detected by the second detection means in the past before the predetermined time point and satisfies the predetermined condition.

4. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

5. The method further includes a determination unit that determines the reliability of the second region detected by the second detection unit, When the reliability determined by the determining means is lower than a threshold, the recognizing means uses a second portion having a reliability higher than the threshold, which is detected by the second detecting means from an image taken before the predetermined time point.

5. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

6. The second detection means cuts out the region of the first part detected by the first detection means from the captured image, and detects the second part from the image of the cut-out region.

6. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

7. The present invention further includes a generating means for generating a control signal corresponding to the gesture recognized by the recognizing means.

7. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

8. When the second detection means does not detect a second part that satisfies the predetermined condition from the image captured at the predetermined time, the recognition means uses the latest second part that satisfies the predetermined condition and that is detected by the second detection means from an image captured before the predetermined time.

8. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

9. Even if the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time, if the second detection means has not previously detected a second part that satisfies the predetermined condition, the recognition means does not recognize the gesture.

9. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

10. Even if the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time point, if the second detection means does not obtain a second part that satisfies the predetermined condition after a timing that is a predetermined time before the predetermined time point, the recognition means does not recognize the gesture.

10. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

11. Even if the second detection means does not detect a second part that satisfies the predetermined condition from the captured image at the predetermined time, if the amount of movement of the first part in the captured image at the predetermined time is greater than a threshold, the recognition means does not recognize the gesture. The gesture recognition device according to any one of claims 1 to 10.

12. The first part is a hand, and the second part is a joint. The gesture recognition device according to any one of claims 1 to 11.

13. The device further includes a recording means for recording the detection result of the second part detected by the second detection means in a recording unit, The recognition means acquires from the recording unit a detection result of a second portion that satisfies the predetermined condition and that is detected from an image captured before the predetermined time point. The gesture recognition device according to any one of claims 1 to 12.

14. When the first detection means does not detect the first part, the recognition means Does not recognize the scha The gesture recognition device according to any one of claims 1 to 13.

15. When the second detection means does not detect a second part that satisfies the predetermined condition from the image captured at the predetermined time, the recognition means uses a second part that satisfies the predetermined condition and that is detected by the second detection means from an image captured before the predetermined time as the second part of the first part detected from the image captured at the predetermined time. The gesture recognition device according to any one of claims 1 to 14.

16. The first detection means detects a rectangular area including the first part from the captured image, The second detecting means detects a point corresponding to the second part from the captured image. The gesture recognition device according to any one of claims 1 to 15.

17. The first portion includes a plurality of the second portions. The gesture recognition device according to any one of claims 1 to 16.

18. The first portion and the second portion move differently. The gesture recognition device according to any one of claims 1 to 17.

19. The recognition means detects movement of a first part based on a first part detected from a plurality of captured images, and detects movement of a second part based on a second part detected from the plurality of captured images. The gesture recognition device according to any one of claims 1 to 18.

20. A display control means for controlling display based on the recognition result of the gesture recognition device according to any one of claims 1 to 19 is provided. A head-mounted display device characterized by:

21. a first detection step of detecting a first portion from the captured image; a second detection step of detecting a second part in the first part detected in the first detection step from the captured image; a recognition step of recognizing a gesture based on the movement of the first part detected in the first detection step and the movement of the second part detected in the second detection step; and When a second part satisfying a predetermined condition is not detected in the second detection step from the captured image at the predetermined time point, the recognition step recognizes a gesture using the second part satisfying the predetermined condition detected in the second detection step from the captured image before the predetermined time point and the first part detected in the first detection step from the captured image at the predetermined time point. A gesture recognition method comprising:

22. A program for causing a computer to function as each means of the gesture recognition device according to any one of claims 1 to 19.

23. A computer-readable storage medium storing a program for causing a computer to function as each means of the gesture recognition device according to any one of claims 1 to 19.

Images