Gesture recognition device, head-mounted display device, gesture recognition method, program, and storage medium

Abstract

To provide a technology capable of recognizing gestures with high accuracy under various situations.SOLUTION: A gesture recognition apparatus includes: portion detection means configured to detect from a captured image a portion making a gesture; recognition means configured to recognize the gesture on a basis of motion of the portion detected by the portion detection means; region detection means configured to detect from the portion a region to be used for the gesture; and motion detection means configured to detect a motion of the region detected by the region detection means. The recognition means recognizes the gesture on a basis of the motion detected by the motion detection means.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for recognizing gestures. [Background technology]

[0002] A technology has been proposed that recognizes gestures made by a user's hands or fingers and performs processing according to the recognized gestures. With such a technology, a user can operate an electronic device (an input device of the electronic device) by using gestures without touching the electronic device.

[0003] Patent Document 1 discloses that the hand, which is the part making the gesture, is extracted from an image of the user's hand, fingers, etc. captured by a camera, and the shape of the extracted hand is identified to recognize the gesture.

[0004] Patent Document 2 discloses the use of deep learning for gesture recognition. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-172887 [Patent Document 2] Japanese Patent Application Publication No. 2019-71048 Summary of the Invention [Problem to be solved by the invention]

[0006] However, electronic devices such as smart glasses that support AR (Augmented Reality) are portable and are used in a variety of situations. For example, when smart glasses are used to assist with work, the work site may be dark or the worker wearing the smart glasses may be wearing gloves. Therefore, when recognizing gestures with such electronic devices, depending on the usage situation of the electronic device, the gesture may not be recognized with high accuracy (gesture recognition may become unstable). For example, the detection of joints (key points for gesture recognition) or the identification (recognition) of the shape of the hand (the part of the body that performs the gesture) may not be performed with high accuracy (become unstable).

[0007] An object of the present invention is to provide a technology that can recognize gestures with high accuracy under various circumstances. [Means for solving the problem]

[0008] The first aspect of the present invention is to perform a gesture from a captured image. User's a part detection means for detecting a part; and a recognition means for recognizing the gesture based on the movement of the part detected by the part detection means. , Ji an area detection means for detecting an area used for a gesture; before If the part of the user is not detected by the part detection means, a motion detection unit that detects the motion of the area detected by the area detection unit, and the recognition unit When the movement is detected by the movement detection means, Based on the movement detected by the movement detection means Teji The gesture recognition device is characterized by recognizing gestures.

[0009] A second aspect of the present invention is a method for detecting a gesture from a captured image. User's a part detection step of detecting a part; and a recognition step of recognizing the gesture based on the movement of the part detected in the part detection step. , Ji an area detection step of detecting an area used for gestures; If no part of the user is detected in the part detection step,a motion detection step of detecting a motion of the region detected in the region detection step, wherein the recognition step If the motion is detected in the motion detection step, Based on the movement detected in the movement detection step Teji A gesture recognition method is characterized by recognizing a gesture.

[0010] A third aspect of the present invention is a head-mounted display device having display control means for controlling display based on the recognition result of the gesture recognition device. A fourth aspect of the present invention is a program for causing a computer to function as each means of the gesture recognition device. A fifth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the gesture recognition device. [Effects of the Invention]

[0011] According to the present invention, gestures can be recognized with high accuracy in various environments. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a gesture recognition device. [Figure 2] FIG. 10 is a diagram illustrating an example of detection of hands and joints. [Figure 3] FIG. 10 is a diagram illustrating an example of an important area according to a gesture. [Figure 4] 10A and 10B are diagrams illustrating an example of a detection result of a motion detection unit. [Figure 5] 10 is a flowchart illustrating an example of a gesture recognition process. [Figure 6] FIG. 10 is a diagram illustrating an example of gesture recognition processing. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of the present invention will be described. Fig. 1 is a block diagram showing an example of the configuration of a gesture recognition device 101 according to this embodiment. The gesture recognition device 101 includes an imaging unit 102, a system control unit 103, a nonvolatile memory 104, a system memory 105, and a gesture recognition unit 110.

[0014] The imaging unit 102 has an optical system with multiple lenses and an image sensor (for example, a CCD or CMOS sensor), and is arranged so as to face a direction in which it can capture an image of a gesture by a user. In this embodiment, an image sensor is used, but any sensor that can be used to recognize a gesture may be used, and a ranging sensor (for example, a sensor that uses a laser such as LiDAR (Light Detection and Ranging)) may also be used.

[0015] The nonvolatile memory 104 is an electrically erasable and recordable memory, such as a Flash-ROM. The nonvolatile memory 104 stores constants and programs for the operation of the system control unit 103. The programs referred to here are, for example, programs for executing the processes of the flowcharts described later in this embodiment.

[0016] The system memory 105 is, for example, a RAM, and stores variables for the operation of the system control unit 103, constants and programs read from the nonvolatile memory 104, and the like.

[0017] The system control unit 103 is a control unit consisting of at least one processor or circuit, and controls the entire gesture recognition device 101. The system control unit 103 executes the programs stored in the nonvolatile memory 104 described above to realize each process of the flowchart described below. The system control unit 103 generates a detected image by performing image processing such as noise removal and resizing on the captured image obtained from the imaging unit 102, and records the detected image in the system memory 105. The system control unit 103 also executes processing according to the gesture recognized by the gesture recognition unit 110. For example, the system control unit 103 generates a control signal corresponding to the recognized gesture, and uses the control signal to perform the gesture. It controls each part of the texture recognition device 101.

[0018] The gesture recognition unit 110 has a target part detection unit 111, an important area detection unit 112, a movement detection unit 113, and a gesture detection unit 114, and recognizes gestures based on the captured image (specifically, the detected image described above) obtained from the imaging unit 102.

[0019] The gesture recognition unit 110 will be described in detail with reference to Figs. 2 to 4. Fig. 2 is a diagram showing an example of detection of hands and joints in this embodiment. Figs. 3(A) and 3(B) are diagrams explaining an example of an important area according to a gesture in this embodiment. Fig. 4 is a diagram explaining an example of a detection result by the movement detection unit 113 in this embodiment.

[0020] 2 is a detected image obtained by performing image processing on a captured image obtained from the imaging unit 102. The detected image 200 shows a left hand 211 and a right hand 212 of a user making a gesture (gesture operation). The image 201 in FIG. 2 is an image obtained by superimposing the detection result of the target part detection unit 111 on the detected image 200.

[0021] The target part detection unit 111 detects (extracts) the hands and joints, which are the parts (target parts) that make a gesture, from the detected image 200. In FIG. 2, a total of 22 positions are detected for each of the left hand 211 and the right hand 212, including the hand position (hand position) and 21 joint positions (joint positions) from the wrist to each fingertip. The hand position is, for example, the center position of the hand. The target part detection unit 111 obtains joint information based on the 21 joint positions. For the left hand 211, a hand position 221 and joint information 231 are obtained, and for the right hand 212, a hand position 222 and joint information 232 are obtained. The joint information 231 and 232 indicate the 21 joint positions and a plurality of line segments connecting the 21 joint positions so as to form a skeleton. Note that in this embodiment, the user makes gestures with their hands and fingers, but gestures may also be made with their arms, legs, eyes, mouth, etc.

[0022] The important region detection unit 112 detects an important region, which is a region (mainly) used in a gesture, from the target regions (hands and joints) detected by the target region detection unit 111. For example, the important region detection unit 112 selects an important region, which is a region (mainly) used in a gesture, from the multiple target regions detected by the target region detection unit 111, and determines the region of the selected important region as the important region. The important region differs depending on the type of gesture. FIGS. 3(A) and 3(B) show examples of gestures, where FIG. 3(A) shows a pinch motion (a motion of pinching with the thumb and index finger), and FIG. 3(B) shows a drag motion (movement motion) of the index finger. A pinch motion is mainly performed by moving the tips of the thumb and index finger. Therefore, the important region detection unit 112 detects the region of the tips of the thumb and index finger as the important region during a pinch motion. A drag motion (a drag motion of the index finger) is mainly performed by moving the tip of the index finger. Therefore, when a dragging motion is performed, the important region detection unit 112 detects the region of the tip of the index finger as the important region. Note that the gesture is not limited to a pinch motion or a dragging motion. By predetermining information on the correspondence between gestures and important regions and storing it in the nonvolatile memory 104, the important region detection unit 112 can detect the important region corresponding to the gesture being performed using the information on the correspondence.

[0023] The motion detection unit 113 detects the motion of the important region detected by the important region detection unit 112 based on the detection result of the important region detection unit 112. In FIG. 4, a region 421 of the tip of the index finger of a hand 411 was previously detected as an important region. For example, the motion detection unit 113 detects a moving region 422 corresponding to the important region 421 from the current detected image using various methods such as block matching, and detects motion 430 from the important region 421 to the moving region 422. The region of the tip of the index finger of a hand 412 (the same hand as the hand 411) is detected as the moving region 422. The motion detection unit 113 detects the moving region and the motion of the important region by calculating the motion vector of feature points (for example, joint positions) in the important region. It is possible.

[0024] The gesture detection unit 114 detects (recognizes) a gesture based on the movement of the target part (hand and joint) detected by the target part detection unit 111. In this embodiment, the gesture detection unit 114 detects a gesture using the detection result (hand position and joint information) of the target part detection unit 111 or the detection result (movement of an important area) of the movement detection unit 113. The gesture detection unit 114 detects a gesture, for example, by comparing the detection result of the target part detection unit 111 or the movement detection unit 113 with a gesture model stored in advance in the non-volatile memory 104. Note that the gesture detection method is not limited to this, and gestures may be detected using a detector trained by deep learning or the like. When deep learning is used, by using an RNN (recurrent neural network), it becomes possible to detect gestures (such as a gesture of drawing a circle with fingers) from time-series data of joint information that cannot be detected from one frame of data. The gesture that the gesture detection unit 114 can detect is not limited to one, and the gesture detection unit 114 may be capable of detecting a plurality of gestures.

[0025] The gesture recognition process in this embodiment will be described. FIG. 5 is a flowchart showing an example of the gesture recognition process in this embodiment. This process is realized by the system control unit 103 expanding a program stored in the nonvolatile memory 104 into the system memory 105 and executing the program. For example, when the gesture recognition device 101 is started, the process in FIG. 5 starts. FIG. 6 is a diagram showing an example of the gesture recognition process in this embodiment. In FIG. 6, detected images 601 to 603 are arranged in chronological order. Detected image 601 is the oldest, and detected image 603 is the newest. Detected image 601 contains a hand 611, detected image 602 contains a hand 621, and detected image 603 contains a hand 631. The hands 611, 621, and 631 are the same hand.

[0026] In step S501, the system control unit 103 acquires a captured image from the imaging unit 102 and generates (acquires) a detected image 601 by performing image processing on the captured image.

[0027] In step S502, the system control unit 103 uses the target part detection unit 111 to acquire the hand position 612 and joint information 613 from the detected image 601 generated in step S501.

[0028] In step S503, the system control unit 103 determines whether or not the hand position 612 and joint information 613 were acquired in step S502. If the hand position 612 and joint information 613 were acquired, the process proceeds to step S504; if not, the process proceeds to step S506. If a part of the target body part (for example, some joints) was not detected, incomplete joint information was obtained, and the process proceeds to step S506. Here, it is assumed that the hand position 612 and joint information 613 were acquired, and the process proceeds to step S504.

[0029] In step S504, the system control unit 103 detects (recognizes) a gesture from the hand position 612 and joint information 613 acquired in step S502, using the gesture detection unit 114. Note that the gesture may be detected using only the hand position and joint information acquired from one detected image, or the gesture may be detected using the hand position and joint information acquired from each of the multiple detected images up to the present.

[0030] In step S505, the system control unit 103 detects important regions 614 and 615 from the hand 611 using the important region detection unit 112, and stores data of the important regions 614 and 615 in the system memory 105. The important region data stored in the system memory 105 may be data of an image of the important region cut out from the detected image, or data of the image of the important region ( In step S505, an important region corresponding to the gesture detected in step S504 is detected. If no gesture is detected in step S504, an important region corresponding to a gesture that can be performed at hand position 612 is detected. Therefore, the important region to be detected differs depending on the hand position. If multiple gestures can be performed at hand position 612, the region of hand 611 may be detected as the important region, or the region of a target region that is most frequently used for gestures among multiple target regions (for example, multiple joints) may be detected as the important region.

[0031] In step S510, the system control unit 103 determines whether or not to end the gesture recognition process. If the gesture recognition process is to be ended, the process proceeds to step S511. In step S511, the system control unit 103 deletes the data of the important area saved in the system memory 105. Then, the gesture recognition process ends. If the gesture recognition process is not to be ended, the process returns to step S501. Here, it is assumed that the process returns to step S501. The system control unit 103 acquires a new detected image 602 in step S501, and acquires a hand position 622 and joint information 623 in step S502. Then, the process proceeds to step S503.

[0032] In step S503, the system control unit 103 determines whether or not the hand position 622 and the joint information 623 were acquired in step S502. If the hand position 622 and the joint information 623 were acquired, the process proceeds to step S504, and if not, the process proceeds to step S506. 2 2 and joint information 6 2 Assume that part of 3) could not be acquired, and the process proceeds to step S506.

[0033] In step S506, system control unit 103 determines whether data of the important regions is stored in system memory 105. If data of the important regions is stored, the process proceeds to step S507; if not, the process returns to step S501. Here, it is assumed that data of important regions 614 and 615 is stored, and the process proceeds to step S507.

[0034] In step S507, the system control unit 103 uses the data of the important regions 614, 615 acquired in step S506 to detect two moving regions from the detected image 602 corresponding to the important regions 614, 615, respectively, and detects the movement of the important regions 614, 615. The moving regions may be detected from the entire region of the detected image 602, but the region in which the moving regions are detected may be limited to a specific region 626 that is part of the detected image 602, for example, based on the positions of the important regions 614, 615, etc.

[0035] In step S508, the system control unit 103 determines whether or not movement of the important regions 614, 615 was detected in step S507. If movement of the important regions 614, 615 was detected, the process proceeds to step S509; if not, the process returns to step S501. If the parts corresponding to the important regions 614 and 615 are outside the detected image 602 or are hidden by another subject, movement of the important regions 614 and 615 cannot be detected, and the process returns to step S501. Here, it is assumed that movement of the important regions 614 and 615 was detected, and the process proceeds to step S509.

[0036] In step S509, the system control unit 103 detects a gesture from the movements of the important areas 614 and 615 detected in step S507 using the gesture detection unit 114. At this time, the system control unit 103 may further use the hand position 612 and joint information 613 acquired in the past.

[0037] In step S510, the system control unit 103 determines whether or not to end the gesture recognition process. If the gesture recognition process is to be ended, the process proceeds to step S511, and if not, the process returns to step S501. In this example, the system control unit 103 acquires a new detected image 603 in step S501, and then executes the process in step S502. Then, the hand position 632 and joint information 633 are acquired. Then, the process proceeds to step S503.

[0038] As described above, according to the present embodiment, a gesture is recognized based on the movement of the area used for the gesture among the body parts used for the gesture, thereby enabling highly accurate recognition of gestures in various environments.

[0039] Note that the above-described embodiment is merely an example, and configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention are also included in the present invention. For example, the gesture recognition device 101 may be provided in an electronic device (head-mounted display device) such as smart glasses compatible with AR (augmented reality). In this case, a display control unit of the head-mounted display device controls the display based on the recognition result of the gesture recognition device 101. The gesture recognition device 101 (gesture recognition unit 110) may be an electronic device separate from the head-mounted display device, and may be, for example, a computer (server) on the cloud.

[0040] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0041] 101: Gesture recognition device 103: System control unit 110: Gesture recognition unit 111: Target part detection unit 112: Important area detection unit 113: Movement detection unit 114: Gesture detection unit

Claims

1. a part detection means for detecting a part of a user making a gesture from a captured image; recognition means for recognizing the gesture based on the movement of the body part detected by the body part detection means; an area detection means for detecting an area used for a gesture; a movement detection means for detecting a movement of the area detected by the area detection means when the area detection means does not detect the user's area; and When the movement is detected by the movement detection means, the recognition means recognizes a gesture based on the movement detected by the movement detection means. A gesture recognition device characterized by:

2. The recognition means recognizes a gesture based on the movement detected by the movement detection means when the part of the body part is not detected by the body part detection means. The gesture recognition device according to claim 1 .

3. The area detection means changes the area to be detected depending on the type of the gesture recognized by the recognition means.

3. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

4. the part detection means detects a hand from the captured image; When the gesture recognized by the recognition means is a pinch motion, the area detection means detects an area of ​​the tip of the thumb and an area of ​​the tip of the index finger. The gesture recognition device according to claim 3 .

5. the part detection means detects a hand from the captured image; When the gesture recognized by the recognition means is a drag action, the area detection means detects the area of ​​the tip of the index finger.

5. The gesture recognition device according to claim 3, wherein the gesture recognition device is a device for recognizing a gesture.

6. The area detection means changes the area to be detected depending on the position of the part detected by the part detection means.

3. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

7. When the part detection means detects a plurality of parts, the area detection means detects an area of ​​the part that is most frequently used for the gesture among the plurality of parts.

7. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

8. The present invention further includes a generating means for generating a control signal corresponding to the gesture recognized by the recognizing means.

8. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

9. When the area detection means has successfully detected the user's body part, the area detection means detects the area used for the gesture based on a correspondence relationship between the gesture and the area used for the gesture.

9. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

10. The motion detection means detects the motion of the region by acquiring a motion vector of a feature point within the region.

10. The gesture recognition device according to claim 1, wherein the gesture recognition device is a device for recognizing a gesture.

11. A display control means for controlling a display based on the recognition result of the gesture recognition device according to any one of claims 1 to 10 is provided. A head-mounted display device characterized by:

12. a part detection step of detecting a part of the user making a gesture from the captured image; a recognition step of recognizing the gesture based on the movement of the part detected in the part detection step; an area detection step of detecting an area used for a gesture; a motion detection step of detecting a motion of the area detected in the area detection step if the user's area is not detected in the area detection step; and In the recognition step, if the movement is detected in the movement detection step, a gesture is recognized based on the movement detected in the movement detection step. A gesture recognition method comprising:

13. A program for causing a computer to function as each means of the gesture recognition device according to any one of claims 1 to 10.

14. A computer-readable storage medium storing a program for causing a computer to function as each means of the gesture recognition device according to any one of claims 1 to 10.

Images