Gesture recognition device and method, head-mounted display device, and computer-readable medium

Abstract

A posture recognition device and method, a head-mounted display device, and a computer-readable medium are provided. The posture recognition device according to the present invention includes a first detection unit configured to detect a first part making a posture from a captured image, a second detection unit configured to detect a second part making the posture from the first part detected by the first detection unit from the captured image, and a recognition unit configured to recognize the posture based on a motion of the first part detected by the first detection unit and a motion of the second part detected by the second detection unit, wherein, in a case where the second detection unit does not obtain a detection result satisfying a predetermined condition, the recognition unit recognizes the posture using a detection result satisfying the predetermined condition obtained by the second detection unit in the past.

Description

Technical Field

[0001] This invention relates to a technique for recognizing postures. Background Technology

[0002] A technique has been proposed for recognizing and processing gestures made by a user's hand or fingers. This technique allows users to operate an electronic device (the input device of the electronic device) by making gestures without touching it.

[0003] Japanese Patent Application Publication No. 2015-172887 discloses a method for extracting the hand used to make a gesture from an image obtained by taking a picture of a user's hand or fingers with a camera, and specifying the shape of the extracted hand to identify the gesture.

[0004] Japanese Patent Application Publication No. 2019-71048 discloses the use of deep learning to identify poses.

[0005] However, when recognizing gestures using both hands, depending on the situation, there is a possibility that the gesture cannot be recognized with high accuracy (gesture recognition becomes unstable). For example, when the right and left hands overlap, there is a possibility that the separation of the right and left hands, the detection of joints (key points used to recognize gestures), or the identification (recognition) of the shape of the hand (the part making the gesture) cannot be performed with high accuracy (it becomes unstable). Summary of the Invention

[0006] This invention provides a technique that can identify postures with high accuracy in various situations.

[0007] The present invention provides, in a first aspect, a pose recognition device comprising: a first detection unit configured to detect a first part of a body making a pose from a captured image; a second detection unit configured to detect a second part of the first part of the body detected by the first detection unit making the pose from the captured image; and a recognition unit configured to recognize the pose based on the movement of the first part detected by the first detection unit and the movement of the second part detected by the second detection unit, wherein, if the second detection unit does not obtain a detection result that satisfies a predetermined condition, the recognition unit uses a detection result that satisfies the predetermined condition previously obtained by the second detection unit to recognize the pose.

[0008] In a second aspect, the present invention provides a head-mounted display device comprising: a display control unit configured to control the display based on the recognition result of the posture recognition device described above.

[0009] The present invention provides a pose recognition method in a third aspect, comprising: a first detection step for detecting a first part of a body making a pose from a captured image; a second detection step for detecting a second part of the first part of the body making the pose from the captured image; and an identification step for identifying the pose based on the movement of the first part of the body detected in the first detection step and the movement of the second part of the body detected in the second detection step, wherein, in the case that no detection result satisfying a predetermined condition is obtained in the second detection step, the pose is identified in the identification step using a detection result satisfying the predetermined condition obtained in a previous second detection step.

[0010] In its fourth aspect, the present invention provides a computer-readable medium storing a program for enabling a computer to function as various units of the aforementioned gesture recognition device.

[0011] Other features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example configuration of a gesture recognition device;

[0013] Figure 2 This is a diagram showing an example of hand and joint detection;

[0014] Figure 3 This is a flowchart illustrating an example of pose recognition processing; and

[0015] Figure 4 This is a diagram used to illustrate an example of pose recognition processing. Detailed Implementation

[0016] Embodiments of the present invention will now be described. Figure 1 This is a block diagram illustrating a configuration example of the gesture recognition device 101 in this embodiment. The gesture recognition device 101 includes a camera unit 102, a system control unit 103, a non-volatile memory 104, a system memory 105, and a gesture recognition unit 110.

[0017] The camera unit 102 has an optical system including multiple lenses and an image sensor (e.g., a CCD or CMOS sensor), and is arranged to face an direction capable of capturing an image of the user's posture. An image sensor is used in this embodiment, but any sensor suitable for posture recognition can be used. A ranging sensor (e.g., a sensor using a laser such as LiDAR (light detection and ranging)) can also be used.

[0018] The non-volatile memory 104 is an electrically erasable / recordable memory, and is, for example, a Flash-ROM. The non-volatile memory 104 stores operating constants or programs of the system control unit 103. Here, "program" refers to, for example, a program used to perform the various processes described in the flowcharts later in this embodiment.

[0019] The system memory 105 is, for example, RAM, and includes the operating variables of the system control unit 103 or constants or programs read from the non-volatile memory 104.

[0020] The system control unit 103 is a control unit comprised of at least one processor or circuit, and controls the entire posture recognition device 101. The system control unit 103 implements the various processes described in the flowchart below by running the program stored in the non-volatile memory 104. The system control unit 103 generates a detection image by applying image processing, such as noise reduction and resizing, to the captured image obtained from the camera unit 102, and records the detection image in the system memory 105. Furthermore, the system control unit 103 processes the postures identified by the posture recognition unit 110. For example, the system control unit 103 generates a control signal corresponding to the identified posture and uses this control signal to control the various units of the posture recognition device 101.

[0021] The pose recognition unit 110 includes an object part detection unit 111, a detailed part detection unit 112, a detailed part judgment unit 113, and a pose detection unit 114, and recognizes poses based on the captured images (specifically the detection images mentioned above) obtained from the camera unit 102.

[0022] Will use Figure 2 The gesture recognition unit 110 is described in detail. Figure 2 This is a diagram illustrating an example of hand and joint detection in this embodiment.

[0023] Figure 2 The image 200 shown is a detection image obtained by applying image processing to a captured image obtained from the camera unit 102. In the detection image 200, the left hand 211 and right hand 212 of the user making a gesture (gesture operation) are reflected. Figure 2 The image 201 shown is obtained by superimposing the detection results of the object part detection unit 111 and the detailed part detection unit 112 onto the detection image 200.

[0024] The object part detection unit 111 detects (extracts) the hand as the object part making a pose from the detection image 200. Figure 2In this process, for each of the left hand 211 and the right hand 212, a hand region (the area of ​​the hand) is detected. The hand region is, for example, a rectangular area surrounding the hand with the center position of the hand as the center. Hand region 221 is detected for the left hand 211, and hand region 222 is detected for the right hand 212.

[0025] The detailed part detection unit 112 detects (extracts) from the detection image 200 the joints in the object part detected by the object part detection unit 111 that are detailed parts for posing. Figure 2 In this embodiment, for each of the left hand 211 and the right hand 212, the joint positions (joint positions) of 21 points from the wrist to the fingertips are detected. The object part detection unit 111 obtains joint information based on the joint positions of the 21 points. Joint information 231 is obtained for the left hand 211, and joint information 232 is obtained for the right hand 212. Joint information 231 and 232 show the joint positions of the 21 points and multiple line segments connecting the joint positions of the 21 points to form a skeletal framework. The joint positions shown in the joint information are, for example, relative positions relative to the hand region. The joint position shown in joint information 231 is a relative position relative to the hand region 221, and the joint position shown in joint information 232 is a relative position relative to the hand region 222. Note that in this embodiment, the user makes gestures with his / her hands or fingers, but can make gestures with his / her body, arms, feet, face, eyes, or mouth, etc. The object part can be the body, and a specific part can be the arm or the joints of the arm, etc. The object part can be the face, and a specific part can be the eyes, etc. Furthermore, when the detection image (the number of pixels in the detection image) is large, it takes a long time to detect detailed areas. If a smaller detection image (a detection image with low resolution (pixel density)) is obtained by resizing, it takes less time to detect detailed areas. However, the detection accuracy decreases. Therefore, a region of the object part (hand region 221 or hand region 222) can be cut out from the detection image, and detailed areas can be detected from the image of the cut-out region (the cut-out image). The resolution of the cut-out image is the same as that of the detection image, but the cut-out image is smaller than the detection image (the number of pixels in the cut-out image is less than the number of pixels in the detection image). Therefore, if detailed areas are detected from the cut-out image, they can be detected in a short time and with high accuracy.

[0026] The detailed part determination unit 113 determines (calculates) the reliability of the detection results (joint information) obtained by the detailed part detection unit 112. Then, the detailed part determination unit 113 determines the joint information to be used in the posture detection unit 114 based on whether the reliability is above the threshold TH. Details (specific examples) related to the method for determining reliability or the method for determining the joint information to be used in the posture detection unit 114 will be described later.

[0027] The pose detection unit 114 detects (identifies) poses based on the movement of the object part (hand) detected by the object part detection unit 111 and the movement of the detailed parts (joints) detected by the detailed part detection unit 112. In this embodiment, the pose detection unit 114 uses the detection result (hand region) of the object part detection unit 111 and the judgment result (joint information) of the detailed part judgment unit 113 to detect poses. The pose detection unit 114 detects poses, for example, by comparing the detection result (hand region) of the object part detection unit 111 and the judgment result (joint information) of the detailed part judgment unit 113 with a pose model pre-stored in the non-volatile memory 104. Note that the method for detecting poses is not limited to this, and a detector that has been learned through deep learning or the like can be used to detect poses. In the case of using deep learning, by using an RNN (recurrent neural network), poses that cannot be detected from a single frame of data (such as the pose of drawing a circle with a finger) can be detected from the time-series data of joint information. The pose detection unit 114 is not limited to one pose, but can detect multiple poses.

[0028] The gesture recognition process in this embodiment will be described. Figure 3 This is a flowchart illustrating an example of the gesture recognition processing in this embodiment. This processing is implemented when the system control unit 103 expands the program stored in the non-volatile memory 104 into the system memory 105 and runs the program. For example, when the gesture recognition device 101 is started, Figure 3 The processing begins. Figure 4 This is a diagram used to illustrate an example of pose recognition processing in this embodiment. Figure 4 In the image, detection images 401 to 403 are arranged in chronological order. Detection image 401 is the oldest, and detection image 403 is the most recent. Detection images 401 to 403 reflect the left hand 405 and the right hand 411. The processing related to the right hand 411 will be described below. However, in order to enable the recognition of multiple hand gestures, the processing related to the right hand 411 and the processing related to the left hand 405 can be performed in parallel or sequentially. The processing related to the left hand 405 is the same as the processing related to the right hand 411.

[0029] In step S301, the system control unit 103 acquires a captured image from the camera unit 102 and applies image processing to the captured image to generate (acquire) a detection image 401. Then, the system control unit 103 uses the object part detection unit 111 to detect the hand region 412 from the generated detection image 401.

[0030] In step S302, the system control unit 103 determines whether the hand region 412 has been successfully detected in step S301. If the hand region 412 has been successfully detected, the process proceeds to step S303. Otherwise, the process returns to step S301. Here, it is assumed that the hand region 412 has been successfully detected and the process has proceeded to step S303.

[0031] In step S303, the system control unit 103 uses the detailed part detection unit 112 to obtain joint information 413 from the hand region 412 detected in step S301.

[0032] In step S304, the system control unit 103 uses the detailed part determination unit 113 to determine (calculate) the reliability of the joint information 413 (the detection result of the detailed part detection unit 112) acquired in step S303. When it is determined, based on the time-series data of the joint information, that the amount of movement (movement speed) of a joint in the detection image is greater than a threshold TH1, for example, when it is determined that the joint has moved at an unexpected degree (speed), it is very likely that incorrect joint information has been acquired. Furthermore, when it is determined that the interval between multiple joints in the detection image is longer than a threshold TH2, for example, when it is determined that the interval between joints is an unexpected length, it is very likely that incorrect joint information has been acquired. Additionally, when the ratio of the portion reflected in the hand detection image (e.g., the ratio of the number of pixels in the right hand 411 to the number of pixels in the hand region 412) is less than a threshold TH3, for example, when most of the hand is hidden behind other hands, it is very likely that incorrect joint information has been acquired. Therefore, in these cases, the reliability of the joint information can be calculated to be low. Note that the detailed part detection unit 112 can output a score showing the reliability of the detection result along with the detection result, just like a detector that has been learned through deep learning.

[0033] In step S305, the system control unit 103 uses the detailed part determination unit 113 to determine whether the reliability determined (calculated) in step S304 is above the threshold TH. When the reliability is above the threshold TH, the system control unit 103 determines the joint information 413 obtained in step S303 as the joint information to be used in the posture detection unit 114 and stores the joint information in the system memory 105. Then, the process proceeds to step S306. When the reliability is less than the threshold TH, the system control unit 103 determines the joint information stored in the system memory 105 (i.e., the joint information previously obtained by the detailed part detection unit 112 with a reliability of above the threshold TH) as the joint information to be used in the posture detection unit 114. Then, the process proceeds to step S307. For example, when the amount of joint movement in the detection image is greater than the threshold TH1, when the interval between multiple joints in the detection image is longer than the threshold TH2, or when the ratio of the portion reflected in the hand detection image is less than the threshold TH3, the reliability becomes lower than the threshold TH. Note that a reliability threshold TH or higher is used, but other predetermined conditions can be used, wherein the other predetermined conditions are met when joint information can be detected with high precision (accuracy) and not met when joint information cannot be detected with high precision (accuracy). By switching the joint information to be used in the pose detection unit 114 depending on whether the predetermined conditions are met, the pose detection unit 114 can be prevented from using incorrect joint information, and ultimately, incorrect pose detection can be prevented. Here, it is assumed that the reliability threshold TH or higher is used, and the process proceeds to step S306.

[0034] In step S306, the system control unit 103 uses the posture detection unit 114 to detect (recognize) the posture from the hand area 412 detected in step S301 and the joint information 413 obtained in step S303.

[0035] In step S309, the system control unit 103 determines whether to end the posture recognition process. The system control unit 103 ends the posture recognition process, or the process returns to step S301. Here, it is assumed that the process returns to step S301. In step S301, the system control unit 103 acquires a new detection image 402 to detect the hand region 422. In step S302, the system control unit 103 determines whether the hand region 422 has been successfully detected. Here, it is assumed that the hand region 422 has been successfully detected, and the process proceeds to step S303. In step S303, the system control unit 103 acquires joint information 423, and in step S304, it determines (calculates) the reliability of the joint information 423 (the detection result of the detailed part detection unit 112). Then, in step S305, the system control unit 103 determines whether the reliability of the joint information 423 is above the threshold TH. Here, it is assumed that the reliability is less than the threshold TH, and the process proceeds to step S307.

[0036] In step S307, the system control unit 103 retrieves past joint information (joint information with a reliability of threshold TH or higher) from the system memory 105. Then, the process proceeds to step S308. If no joint information is stored in the system memory 105 (when no joint information with a reliability of threshold TH or higher has been obtained in the past), the process returns to step S301 to not recognize the pose. Here, it is assumed that joint information 413 is stored in the system memory 105, and the system control unit 103 retrieves joint information 413 from the system memory 105 to allow the process to proceed to step S308. Note that when multiple joint information items are stored in the system memory 105, the system control unit 103 can retrieve the latest joint information. When joint information is stored in the system memory 105 but pose recognition with high accuracy is unlikely, the process can return to step S301 to not recognize (detect) the pose. For example, when only joint information prior to a predetermined time earlier than the current time is stored (when joint information with a reliability of more than the threshold TH has not been obtained after a predetermined time earlier than the current time), the process can return to step S301. Furthermore, when the amount of movement of the right hand 411 in the detected image (e.g., the amount of movement of the center position of the right hand 411) is greater than the threshold TH4, the process can return to step S301.

[0037] In step S308, the system control unit 103 uses the posture detection unit 114 to detect (identify) postures from the hand region 422 detected in step S301 and the past joint information 413 obtained in step S307.

[0038] In step S309, the system control unit 103 determines whether to end the posture recognition process. The system control unit 103 ends the posture recognition process, or the process returns to step S301. Here, it is assumed that the process returns to step S301. In step S301, the system control unit 103 acquires a new detection image 403 to detect the hand region 432. In step S302, the system control unit 103 determines whether the hand region 432 has been successfully detected. Here, it is assumed that the hand region 432 has been successfully detected, and the process proceeds to step S303. In step S303, the system control unit 103 acquires joint information 433, and in step S304, it determines (calculates) the reliability of the joint information 433 (the detection result of the detailed part detection unit 112).

[0039] As described above, according to this embodiment, when the detailed part detection unit 112 does not obtain a detection result that meets the predetermined conditions, the posture is identified using a detection result that meets the predetermined conditions previously obtained by the detailed part detection unit 112. In this way, posture can be identified with high accuracy in various environments.

[0040] Note that the above embodiments are merely examples, and configurations obtained by making appropriate variations or modifications to the above configurations within the scope of the present invention are also included in the present invention. For example, the gesture recognition device 101 can be disposed in an electronic device (head-mounted display device) such as smart glasses for AR (augmented reality). In this case, the display control unit disposed in the head-mounted display device controls the display based on the recognition result of the gesture recognition device 101. The gesture recognition device 101 (gesture recognition unit 110) can be an electronic device separate from the head-mounted display device, and can be, for example, a computer (server) in the cloud.

[0041] According to the present invention, posture can be identified with high accuracy in various environments.

[0042] Other embodiments

[0043] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0044] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation so as to cover all such modifications and equivalent structures and functions.

Claims

1. A posture recognition device, comprising: The first detection unit is configured to detect a first part of the body making a pose from the captured image; The second detection unit is configured to detect from the captured image a second part of the first part that is making the gesture, which was detected by the first detection unit. as well as The recognition unit is configured to recognize the posture based on the movement of the first part detected by the first detection unit and the movement of the second part detected by the second detection unit. Wherein, if the second detection unit does not detect a second part that meets the predetermined conditions from the image captured at the predetermined time, the recognition unit uses the second part that meets the predetermined conditions detected by the second detection unit from the image captured at a time prior to the predetermined time and the first part detected by the first detection unit from the image captured at the predetermined time to recognize the posture.

2. The posture recognition device according to claim 1, wherein, If the movement of the second part in the image captured at the predetermined time exceeds a threshold, the recognition unit uses the second part that meets the predetermined condition, which was detected by the second detection unit from the image captured at a time prior to the predetermined time.

3. The posture recognition device according to claim 1 or 2, wherein, The second detection unit detects multiple second locations, and If the interval between multiple second parts in the image captured at the predetermined time exceeds a threshold, the recognition unit uses multiple second parts that meet the predetermined conditions, detected by the second detection unit from the image captured at a time prior to the predetermined time.

4. The posture recognition device according to claim 1 or 2, wherein, The first detection unit detects multiple first locations, and If the ratio of the portion of a first part hidden by the overlap of another first part to the first part at the predetermined time is greater than a threshold, the identification unit uses a second part that meets the predetermined condition, detected by the second detection unit from an image captured at a time prior to the predetermined time, as the second part of the first part.

5. The posture recognition device according to claim 1 or 2 further includes a judgment unit, the judgment unit being configured to judge the reliability of the second part detected by the second detection unit. in, If the reliability of the second part at the predetermined time is lower than a threshold, the identification unit uses the second part that has a reliability higher than the threshold, which was detected by the second detection unit from an image taken at a time prior to the predetermined time.

6. The posture recognition device according to claim 1 or 2, wherein, The second detection unit cuts out the region of the first part detected by the first detection unit from the captured image, and detects the second part from the image of the cut-out region.

7. The posture recognition device according to claim 1 or 2 further includes a generation unit configured to generate a control signal corresponding to the posture recognized by the recognition unit.

8. The posture recognition device according to claim 1 or 2, wherein, If the second detection unit does not detect a second part that meets the predetermined condition from the image captured at the predetermined time, the identification unit uses the latest second part that meets the predetermined condition, which was detected by the second detection unit from the image captured at a time prior to the predetermined time.

9. The posture recognition device according to claim 1 or 2, wherein, If the second detection unit does not detect a second part that meets the predetermined condition from the image captured at the predetermined time, then the recognition unit does not recognize the posture if the second detection unit does not detect a second part that meets the predetermined condition before the predetermined time.

10. The posture recognition device according to claim 1 or 2, wherein, If the second detection unit does not detect a second part that meets the predetermined condition from the image captured at the predetermined time, and if the second detection unit does not detect a second part that meets the predetermined condition during a predetermined time period up to the predetermined time, then the recognition unit does not recognize the posture.

11. The posture recognition device according to claim 1 or 2, wherein, If the second detection unit does not detect a second part that meets the predetermined condition from the image captured at the predetermined time, and if the movement of the first part in the image captured at the predetermined time is greater than a threshold, then the recognition unit does not recognize the posture.

12. The posture recognition device according to claim 1 or 2, wherein, The first part is the hand, and the second part is the joint.

13. The posture recognition device according to claim 1 or 2, further comprising a recording unit configured to record on a storage device the detection result of the second part detected by the second detection unit. in, The identification unit obtains from the storage device the detection result of the second part that meets the predetermined condition, which was detected from an image taken at a time prior to the predetermined time.

14. The posture recognition device according to claim 1 or 2, wherein, If the first detection unit does not detect the first part, the recognition unit does not recognize the posture.

15. The posture recognition device according to claim 1 or 2, wherein, If the second detection unit does not detect a second part that meets the predetermined condition from the image captured at the predetermined time, the recognition unit uses the second part that meets the predetermined condition detected by the second detection unit from the image captured at a time prior to the predetermined time as the second part among the first parts detected from the image captured at the predetermined time.

16. The posture recognition device according to claim 1 or 2, wherein, The first detection unit detects a rectangular region surrounding the first location from the captured image, and The second detection unit detects points in the captured image that correspond to the second part.

17. The posture recognition device according to claim 1 or 2, wherein, The first part includes multiple second parts.

18. The posture recognition device according to claim 1 or 2, wherein, The movement of the first part is different from the movement of the second part.

19. The posture recognition device according to claim 1 or 2, wherein, The recognition unit detects the motion of the first part based on a first part detected from multiple captured images, and detects the motion of the second part based on a second part detected from the multiple captured images.

20. A head-mounted display device, comprising: At least one memory and at least one processor, wherein the at least one processor serves as a display control unit, the display control unit being configured to control the display based on the recognition result of the gesture recognition device according to any one of claims 1 to 19.

21. A pose recognition method, comprising: The first detection step is used to detect the first part of the body that is posing from the captured image; The second detection step is used to detect from the captured image a second part of the first part that is making the gesture, which was detected in the first detection step; as well as The identification step is used to identify the posture based on the movement of the first part detected in the first detection step and the movement of the second part detected in the second detection step. In the second detection step, if no second part satisfying the predetermined condition is detected from the image captured at a predetermined time, the posture is identified in the recognition step using the second part satisfying the predetermined condition detected in the second detection step from the image captured at a time prior to the predetermined time, and the first part detected in the first detection step from the image captured at the predetermined time.

22. A computer-readable medium storing a program, wherein, When the program is executed by a computer, it implements the steps of the method according to claim 21.

23. A computer program product comprising a program that, when executed by a computer, implements the steps of the method according to claim 21.

Images