Information processing device, information processing system, information processing program, and information processing method

Abstract

Depending on the orientation of the hand, the fingertips may not appear in the image used to recognize the hand gesture, making it difficult to accurately calculate the orientation of the CG object grasped by the hand gesture. [Solution] The information processing device has a storage means for storing information based on a first posture in a storage unit when a first hand included in a first captured image has a first posture indicating a specific gesture, and an estimation means for estimating the posture of the second hand when indicating the specific gesture based on the information and the posture of the specific part in a second captured image when a second hand included in the second captured image does not indicate the specific gesture, even if the second captured image is taken after the first captured image and the second captured image includes the specific part of the second hand.

Description

[Technical Field]

[0001] The present invention relates to an information processing device that recognizes hand gestures. [Background technology]

[0002] Mixed reality (MR) and augmented reality (AR) are technologies that fuse the real world with computer-generated CG (computer graphics) in real time. Mixed reality and augmented reality use a head-mounted display (HMD), a device worn on the user's head, to present a composite image of the real world and CG to the user, enabling interaction between the user and the CG, thereby providing an immersive experience. Hand gesture operation is one of the operation methods that allows users to interact with CG. Various sensors, such as a camera, mounted on the head-mounted display detect the user's hands, and when the user's hands are making a specific gesture, CG is displayed in accordance with the detected hands, making it appear as if the user is grasping an object. For example, Patent Document 1 discloses a technology that recognizes hand gestures from multiple images that include the user's hands. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-71048 A Summary of the Invention [Problem to be solved by the invention]

[0004] To make it appear as if a CG object is being grasped using hand gestures, it is important to display the CG image in a way that accurately matches the orientation of the actual hand. However, depending on the orientation of the hand, parts of the hand, such as the fingertips, may not appear in the image used to recognize the hand gesture, making it difficult to accurately calculate the orientation of the CG object grasped by the hand gesture (hereafter referred to as the gesture posture).

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to make it possible to calculate the gesture posture when grasping a CG with a hand gesture, even in cases where it is difficult to calculate the gesture posture depending on the orientation of the hand. [Means for solving the problem]

[0006] In order to achieve the above object, the information processing device of the present invention is characterized by having: a storage means for storing information based on a first posture in a storage unit when a first hand included in a first captured image has a first posture indicating a specific gesture; and an estimation means for estimating the posture of the second hand when indicating the specific gesture based on the information and the posture of the specific part when a second hand included in a second captured image captured after the first captured image does not indicate the specific gesture but the second captured image includes the specific part of the second hand. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that can calculate the gesture posture when grasping a CG with a hand gesture, even in cases where it is difficult to calculate the gesture posture depending on the orientation of the hand. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Figure 2] 1 is a diagram illustrating the internal configuration of an HMD according to a first embodiment. [Figure 3] 1A to 1C are diagrams illustrating gesture postures in a pinch gesture and a grasp gesture, which are hand gestures, according to the first embodiment. [Figure 4] 10 is a flowchart illustrating a process of calculating a gesture posture and generating a display image according to the first embodiment. [Figure 5] 3A and 3B are diagrams illustrating a gesture posture and a reference posture according to the first embodiment. [Figure 6] 10 is a flowchart illustrating a method for calculating a gesture posture in a captured image using the amount of change from a stored reference posture to the reference posture in the captured image according to the first embodiment. [Figure 7] 10 is a flowchart illustrating a method for calculating a gesture posture in a captured image from a reference posture in the captured image using an amount of change from a stored reference posture to a stored gesture posture according to the first embodiment. [Figure 8] 2A to 2C are diagrams illustrating a captured image acquired by the HMD 100 and a generated display image according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be assigned the same reference numerals, and duplicate descriptions will be omitted where appropriate. In addition, some of the components, members, and processes will be omitted in each drawing.

[0010] (Embodiment 1) <System configuration> An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes a head-mounted display (HMD) 100 and a PC (personal computer) 110.

[0011] The HMD100 is a head-mounted display device (electronic device) that can be worn on the user's head. The HMD100 includes a camera for imaging the area in front of the user and a display for displaying images to the user. On the display of the HMD100, a composite image is displayed, which is a composite of the captured image of the area in front of the user captured by the HMD100 and content such as CG (computer graphics) in a form corresponding to the posture of the HMD100. Thereby, the user can experience virtual reality with the user's eyes. Also, in the system 1, the hand of the user is detected from the image captured by the HMD100, and by acquiring information related to the position and orientation of the hand as the hand posture, it has a function of causing the movement of the hand to act on a virtual object. Thereby, the user can intuitively operate on the virtual object using the user's hand.

[0012] The PC110 controls the HMD100. The PC110 is connected to the HMD100 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC100 and the HMD101 communicate with each other by wireless or wired communication and can transmit and receive images and other necessary information to and from each other. The PC110 generates a composite image by synthesizing the image captured by the HMD100 and the CG generated by the PC110, and transmits the composite image to the HMD100. Here, although a PC is described as an example of the information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each configuration of the PC110 may be possessed by the HMD100.

[0013] <Internal Configuration of HMD> Referring to FIG. 2, the internal configuration of the HMD100 will be described. The HMD100 has an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, and a working memory 206.

[0014] The HMD control unit 201 controls each component of the HMD 100. The HMD control unit 201 has at least one CPU that executes programs stored in a nonvolatile memory 205, and at least one other circuit. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0015] The imaging unit 202 includes two cameras (imaging devices). The two cameras are disposed near the positions of the user's left and right eyes when the HMD 100 is worn on the user's head. Therefore, the two cameras can capture a space similar to the space seen by the user wearing the HMD 100. The images captured by the imaging unit 202 are output to the HMD control unit 201, which transmits the images from the imaging unit 202 to the PC 110. As described below, the PC 110 combines the captured images transmitted from the HMD 100 with CG to generate a composite image. Furthermore, the imaging unit 202 simultaneously captures a first image having a parallax with respect to each other and a second image different from the first image using the two cameras. Therefore, information on the distance from the HMD 100 to a subject (distance information) can be acquired using the images from the two cameras in the imaging unit 202. The imaging unit 202 may also capture and output a video.

[0016] As described below, when a composite image is transmitted from the PC 110, the image display unit 203 displays the composite image transmitted from the PC 110. The image display unit 203 has a display such as a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, a display such as an organic EL panel is disposed in front of each of the user's eyes. Note that a device using a semi-transparent half mirror may also be used for the image display unit 203. In this case, for example, the image display unit 203 may use a technology generally called AR (Augmented Reality) to display an image so that CG appears to be directly superimposed on the real space visible through the half mirror. Furthermore, the image display unit 203 may use a technology generally called VR (Virtual Reality) to display an image of a completely virtual space without using captured images.

[0017] The orientation sensor unit 204 acquires orientation (and position) information of the HMD 100. Then, the orientation sensor unit 204 acquires orientation information of the user (the user wearing the HMD 100) that corresponds to the orientation (and position) of the HMD 100. The orientation sensor unit 204 has an inertial measurement unit (IMU) that is configured from an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. When the user wears the HMD 100, the orientation sensor unit 204 acquires information about the user's orientation (orientation information). The HMD control unit 201 outputs the information about the user's orientation (orientation information) detected by the orientation sensor 204 to the PC 110.

[0018] The nonvolatile memory 205 is an electrically erasable and recordable nonvolatile memory, and stores programs executed by the HMD control unit 201 and the like.

[0019] The volatile memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a work area for the HMD control unit 201, and the like.

[0020] <Internal Structure of the PC> Referring to FIG. 2, the internal structure of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, and a working memory 213.

[0021] The control unit 211 is a CPU composed of at least one processor or circuit. By executing the program stored in the non-volatile memory 212, the control unit 211 realizes each process of the flowchart described later. Note that instead of the control unit 211 controlling the entire device, a plurality of hardware may share the processing to control the entire device. The control unit 211 receives from the HMD 100 the image (captured image) acquired by the imaging unit 202 and the posture information acquired by the posture sensor unit 204. Based on the received information, the control unit 211 synthesizes the captured image and an arbitrary CG to generate a synthesized image. The control unit 211 transmits the synthesized image to the HMD control unit 201 in the HMD 100.

[0022] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores programs described later executed by the control unit 211 and information such as CG. Note that the control unit 211 can switch the CG read from the non-volatile memory 212 (that is, the CG used for generating the synthesized image).

[0023] The working memory 213 is a storage unit used as a working area of the control unit 211, such as a buffer memory that temporarily holds the image data captured by the imaging unit 202.

[0024] <Explanation of the Gesture Posture of the Pinch Gesture> In the information processing system 1 of the present embodiment, when a hand gesture is detected from the captured image by the HMD 100, it has a function of synthesizing a virtual object corresponding to the detected hand gesture into the captured image.

[0025] FIG. 3(a) and FIG. 3(b) are diagrams for explaining the gesture posture of the pinch gesture as a hand gesture in the present embodiment.

[0026] FIG. 3(a) is a diagram illustrating gesture postures in a pinch gesture.

[0027] The user's hand 301 has its index finger and thumb touching, forming a hand shape as if pinching something. A gesture in which the hand is in this state is called a pinch gesture. When the user's hand 311 is making a pinch gesture, it is assumed that the user is grasping an object between the touching index finger and thumb. The direction in which the object is grasped at this time is called the gesture posture. The gesture posture 302 of a pinch gesture may be set to the direction of the arrow shown in FIG. 3(a), for example. Here, the arrow indicating the gesture posture 302 is composed of three orthogonal three-dimensional vectors.

[0028] By using the three-dimensional positions 303 of the feature points of the user's hand 301 acquired by the control unit 211, it is possible to determine whether the user's hand 301 is making a pinch gesture. In this case, by calculating the distance between the feature points of the index finger and thumb among the three-dimensional positions 303 of the feature points, it is possible to determine that the user's hand 301 is making a pinch gesture if the distance is equal to or less than a predetermined threshold. Furthermore, it is also possible to determine whether the user's hand 301 is making a pinch gesture by using an image recognition technology such as machine learning. In this case, the control unit 211 can use an image recognition model that outputs whether a hand included in an image is making a pinch gesture. The image recognition model is created, for example, by learning images including a hand in the shape of a pinch gesture. Furthermore, such an image recognition model can perform a classification process to determine whether the user's hand 301 is making a pinch gesture. Note that as a method of determining whether the user's hand 301 is making a pinch gesture, a method using the three-dimensional positions 303 of the feature points of the user's hand 301 acquired by the control unit 211 and a method using an image recognition technology such as machine learning can be used together.

[0029] In a machine learning model, for example, a captured image is input into a CNN (convolutional neural network). The CNN outputs features used to identify the type of subject or the type of scene. The features output from the CNN are then used to identify the type of subject or scene. Examples of machine learning models that can be used to detect and classify target objects or people include YOLO, MobileNet, VGG16, and SSD.

[0030] Next, a possible method for determining gesture posture 302 for a pinch gesture is to use three-dimensional positions 303 of feature points of the hand calculated by control unit 211. Gesture posture 302 can be determined by calculating a vector from a plurality of feature points among the three-dimensional positions 303 of feature points. For example, gesture posture 302 can be calculated from feature points corresponding to the tip of the thumb, the base of the thumb, and the base of the index finger. By accurately calculating gesture posture 302 in accordance with the orientation of the hand, when an object is displayed in accordance with the orientation of pinch gesture posture 302, it is possible to display the object as if it is being grasped naturally.

[0031] FIG. 3(b) shows an example of displaying an object according to the gesture posture of a pinch gesture.

[0032] Object 314 is a CG (virtual object) shaped like a pen, and when the user's hand 311 is in a pinch gesture state, the pen-shaped object 314 is displayed along the vector of gesture posture 312. By displaying it in this manner, the user can experience as if they were actually pinching object 314. Furthermore, when the orientation of the user's hand 311 changes, the gesture posture 312 and the posture of object 314 are changed accordingly, enabling a more realistic experience.

[0033] <Explanation of the gesture posture for the Grasp gesture> FIG. 3C is a diagram illustrating a gesture posture in a grasp gesture as a hand gesture in this embodiment.

[0034] The user's hand 321 has all of its fingers bent, forming a hand shape as if grasping an object. A gesture in which the hand is in this state is called a grasp gesture. When the user's hand 321 is making a grasp gesture, it is assumed that the object is grasped by wrapping all of the fingers and palm around it. Therefore, the gesture posture 322 of the grasp gesture may be set in a direction such as that shown in FIG. 3(c), for example. Here, the arrow indicating the gesture posture 322 is composed of three orthogonal three-dimensional vectors.

[0035] By using the three-dimensional positions 323 of the feature points of the user's hand 321 acquired by the control unit 211, it is possible to determine whether the user's hand 321 is making a grasp gesture. In this case, among the three-dimensional positions 323 of the feature points, a specific feature point of a specific finger is used as the origin, and the angle between two vectors is calculated using vectors extending to two adjacent feature points of the same finger. If the angle between the two vectors is equal to or less than a specific threshold, it is possible to determine that the user's hand 321 is making a grasp gesture. In addition, it is possible to determine whether the user's hand 321 is making a grasp gesture by using image recognition technology such as machine learning. In this case, the control unit 211 can use an image recognition model that outputs whether a hand included in an image is making a grasp gesture. The image recognition model is created, for example, by learning images of hands in the shape of a grasp gesture. Furthermore, such an image recognition model can perform a classification process to determine whether the user's hand 321 is making a grasp gesture. As a method for determining whether or not the user's hand 321 is making a grasp gesture, it is possible to use both a method using the three-dimensional position 323 of the feature point of the user's hand 321 acquired by the control unit 211 and a method using image recognition technology such as machine learning.

[0036] One possible method for determining gesture posture 322 is to use three-dimensional positions 323 of feature points of the hand calculated by control unit 211. Gesture posture 322 can be determined by calculating a vector from a plurality of feature points among the three-dimensional positions 323 of feature points. For example, gesture posture 322 can be calculated from feature points corresponding to the wrist, the base of the index finger, the base of the middle finger, and the base of the little finger. By accurately calculating gesture posture 322 in accordance with the orientation of the hand, it becomes possible to display an object as if it is being grasped naturally when displayed in accordance with the orientation of gesture posture 322.

[0037] <How to find gesture posture and reference posture> 5(a) and 5(b) are diagrams illustrating gesture postures and reference postures according to this embodiment.

[0038] The captured image 500 in FIG. 5(a) is an image of one frame among multiple frames in a video captured by the imaging unit 202. The captured image 500 includes a user's hand 501, and the control unit 211 acquires the three-dimensional position of each feature point of the user's hand 501 included in the captured image 500. Furthermore, the control unit 211 calculates a reference posture 502 and a gesture posture 503 of the user's hand 501 based on the acquired three-dimensional positions of each feature point. The reference posture 502 can be calculated using the posture of the palm or the back of the hand. Even when the fingertips are hidden, the palm or the back of the hand can be detected with high accuracy. The posture of the palm or the back of the hand can be calculated using feature points such as the base of the index finger, the base of the little finger, and the wrist. Although FIG. 5(a) illustrates the gesture posture 503 of a pinch gesture, it may also be a grasp gesture or another gesture. For example, a gesture of two fingers together or a gesture of one or more fingers extended may be used. Note that the gesture posture and the reference posture may each include information on the position and information on the orientation, or may include only information on the orientation.

[0039] Captured image 504 in FIG. 5(b) is an image of one frame in a moving image captured by imaging unit 202, and is an image of a frame temporally later than captured image 500. Captured image 504 includes user's hand 501, and control unit 211 acquires the three-dimensional position of each feature point of user's hand 501 included in captured image 504. Furthermore, based on the acquired three-dimensional position of each feature point, reference posture 505 of user's hand 501 is calculated. Here, reference posture 505 is the posture of the same part as reference posture 502, and here the posture of the palm or back of the hand is taken as reference posture 502.

[0040] A gesture posture 506 in the second captured image 504 is calculated based on the reference posture 502 and gesture posture 503 calculated in the captured image 500 and the reference posture 505 calculated in the second captured image 504. A specific method for calculating the gesture posture 506 in the second captured image 504 will be a calculation method described later.

[0041] <Explanation of Flowchart of Processing for Obtaining Gesture Posture and Generating Display Image> FIG. 4 is a flowchart of a process for determining a gesture posture and generating a display image in this embodiment. This process is realized by the control unit 211 loading a program stored in the non-volatile memory 212 into the work memory 213 and executing it. Note that the timing of execution of this flowchart is not limited to the timing when a virtual object is displayed in the mixed reality space. For example, it may be the timing when the user starts the HMD 100 or the timing when the user starts a predetermined application of the HMD 100. The predetermined application may be, for example, an application that the user selects on the home screen (home space) after starting the HMD, such as an app that allows the user to interact with a virtual object using hand gestures. The following process may also be performed in a virtual space, not just in the mixed reality space. The process of FIG. 4 is executed each time one frame of a moving image is acquired from the imaging unit 202. In addition, the process of FIG. 4 will be described as detecting a pinch gesture shown in FIG. 4 as a specific user hand gesture and displaying a virtual object corresponding to the pinch gesture.

[0042] In step S401, the control unit 211 acquires the captured image captured by the imaging unit 202, and the process proceeds to step S402.

[0043] In step S402, the control unit 211 detects three-dimensional positions of feature points of the user's hand based on the captured image acquired in step S401, and proceeds to step S403. As described above, the process of S402 is performed for one frame of the moving image captured by the imaging unit 202. Note that the feature points of the user's hand include joint points, which are points that estimate the positions of at least one of the joints and fingertips of the hand. For example, 21 points may be acquired as joint points, including 20 points at the fingertips, first joints, second joints, and bases of the fingers, and one point at the wrist.

[0044] In step S403, the control unit 211 determines whether or not the three-dimensional positions of the feature points (joint points) of the user's hand have been detected. If the control unit 211 has been able to detect the three-dimensional positions of the feature points (joint points) of the user's hand, the process proceeds to step S404, and if the control unit 211 has not been able to detect the three-dimensional positions of the feature points (joint points) of the user's hand, the process proceeds to step S409. For example, if the hand is not captured in the captured image, the three-dimensional positions of the feature points (joint points) of the user's hand cannot be detected, and the process proceeds to step S409.

[0045] In step S404, the control unit 211 determines whether the user's hand is making a specific gesture. If the control unit 211 determines that the hand is making a specific gesture, the process proceeds to step S405. If the control unit 211 determines that the hand is not making a specific gesture, the process proceeds to step S407. That is, if the hand included in the captured image does not make a specific gesture, or if the hand included in the captured image is making a gesture different from the specific gesture, the hand is determined not to be making a specific gesture, and the process proceeds to step S407.

[0046] In step S405, the control unit 211 calculates the reference posture and gesture posture in the captured image, and the process proceeds to step S406.

[0047] In step S406, control unit 211 calculates correction information for the gesture posture from the reference posture and gesture posture in the captured image calculated in step S405, stores the correction information in working memory 213, and proceeds to step S409. Note that the reference posture and gesture posture may be stored in working memory 213 without calculating the correction information, or only the gesture posture may be stored in working memory 213. Here, the correction information may be, for example, information on the difference between the rotation amount and position of the reference posture and gesture posture in the captured image calculated in step S405. Using such difference and the reference posture, it is possible to calculate (estimate) a gesture posture that could not be estimated from the image alone, using a method described below.

[0048] In step S407, control unit 211 reads (acquires) correction information from work memory 213, and proceeds to step S408. If correction information cannot be acquired or if correction information is not stored in work memory 213, control unit 211 proceeds to step S408 without acquiring correction information in step S407. Here, if correction information is stored in work memory 213, control unit 211 reads the correction information from work memory 213. If a reference posture and a gesture posture are stored in work memory 213, control unit 211 reads the reference posture and gesture posture from work memory 213 and calculates correction information. If a gesture posture is stored in work memory 213 but a reference posture is not stored, control unit 211 reads the gesture posture from work memory 213, extracts a reference posture from the gesture posture, and calculates correction information from the reference posture and gesture posture. If neither the reference posture nor the gesture posture is stored in the working memory 213, the process proceeds to step S408 without reading (acquiring) the correction information.

[0049] In step S408, control unit 211 calculates a gesture posture based on the correction information acquired in step S407 and a reference posture, such as the palm or the back of the hand, detected from the captured image. The method of calculating the gesture posture in step S408 will be described later. Furthermore, even if correction information is acquired in step S407, if the gesture posture can be detected from the captured image without using the correction information, the gesture posture is detected without using the correction information.

[0050] In step S409, control unit 211 draws a virtual object and proceeds to step S410. Here, the virtual object is drawn as if the user were holding the virtual object with their thumb and index finger using a pinch gesture, which is a specific gesture. Therefore, control unit 211 calculates the orientation of the virtual object when the user is holding the virtual object with their thumb and index finger in a pinch gesture, based on the calculated gesture posture. Then, the virtual object is drawn in the calculated orientation. Furthermore, if there is a virtual object to be placed in the mixed reality space, the virtual object is drawn in step S409. Furthermore, if the gesture posture has not been calculated, the virtual object placed in the mixed reality space is drawn.

[0051] In step S410, the control unit 211 generates a display image to be displayed on the image display unit 203 of the HMD 100 using the virtual object drawn in step S409 (performs image generation), and proceeds to step S411. Note that after generating the display image, the control unit 211 may further transmit the display image to the HMD 100 and perform display control to display the display image on the image display unit 203 of the HMD 100.

[0052] In step S411, control unit 211 determines whether or not to end this process. If control unit 211 determines to end this process, it proceeds to step S412, and if control unit 211 does not determine to end this process, it proceeds to step S401.

[0053] In step S412, if the correction information is stored in the work memory 213, the control unit 211 deletes the correction information and ends this process. Note that the correction information may be deleted when the HMD 100 is turned off, and may not be deleted when a predetermined application is ended.

[0054] According to the above flow, for example, if the captured image includes a hand with the index finger and thumb separated, it is determined that the hand is not making a specific gesture, and the process proceeds to step S407. Furthermore, in step S407, if correction information is stored, the correction information is acquired. However, in step S408, a gesture posture is detected from the hand with the index finger and thumb separated without using the correction information. Furthermore, if an image including a hand with the index finger and thumb separated is acquired following an image of a hand making a specific gesture, a virtual object is drawn so as to represent that the virtual object is not being moved. In other words, the virtual object is drawn so as to represent that the hand has not moved from the position of the virtual object drawn when the image of the hand making the specific gesture was acquired.

[0055] Here, in step S406, if correction information is calculated from the reference posture and gesture posture in the captured image and stored in working memory 213, there is an advantage that the calculation load in step S407 is reduced compared to when correction information is not stored. If importance is placed on real-time video, it is preferable to calculate the reference posture and gesture posture in the captured image and store them in working memory 213 in advance.

[0056] Furthermore, if the gesture posture in the captured image is stored in step S406 without storing the reference posture, the reference posture is calculated from the stored gesture posture in step S407. Storing the gesture posture in the captured image in step S406 in this way has the effect of reducing the amount of data stored in working memory 213 compared to storing the gesture posture and the reference posture without calculating correction information. If the amount of data in working memory 213 in this system is tight, it is preferable to store the gesture posture in the captured image and not store the reference posture.

[0057] 4, correction information is stored each time a specific gesture is performed, but this is not limiting and correction information may be stored only for the first frame in which the gesture is detected. Furthermore, when multiple pieces of correction information are stored, the average of the multiple pieces of correction information may be used to calculate the gesture posture when part of the hand is hidden. Furthermore, when a gesture is detected multiple times, correction information may be overwritten each time and stored in working memory 213.

[0058] Alternatively, correction information may be acquired from the first frame when a first gesture is detected and stored in working memory 213, but not stored until the end of the first gesture. Then, when a second gesture is detected, correction information acquired from the first frame when the second gesture is detected may be overwritten and stored in working memory 213. In other words, correction information may be acquired and stored in working memory 213 only for the first detected frame of a series of gestures, and the correction information may be used only within that series of gestures.

[0059] As described above, according to FIG. 4, by calculating the gesture posture of the captured image based on a plurality of image frames, it is possible to obtain the gesture posture more stably than by calculating the gesture posture from information of only the second captured image.

[0060] <First calculation method of gesture posture when reference posture and gesture posture are stored> 6, a flowchart will be described illustrating a method for calculating a gesture posture in a captured image using a rotation amount θ from the orientation of a stored reference posture to the orientation of the reference posture detected from the captured image. This process is executed in step S407 when, for example, the gesture posture and the reference posture are stored in working memory 213 in step S406 of FIG.

[0061] In step S601, the control unit 211 calculates a rotation amount θ1, which is the difference between the orientation of the reference orientation stored in the working memory 213 and the orientation of the reference orientation detected from the captured image, and proceeds to step S602. Note that the orientation of each reference orientation may be expressed by a 3×3 rotation matrix or quaternion, and the rotation amount θ1 may be calculated.

[0062] In step S602, the control unit 211 calculates the direction of the gesture posture in the captured image by rotating the gesture posture stored in the working memory 213 by the rotation amount θ1 calculated in step S601.

[0063] In step S603, control unit 211 calculates the amount of change in position (Δx1, Δy1, Δz1) that is the difference between the position of the reference posture stored in working memory 213 and the position of the reference posture detected from the captured image, and proceeds to step S604. Note that each of the reference postures and the amount of change in position may be expressed in a three-dimensional coordinate system or in a predefined two-dimensional coordinate system.

[0064] In step S604, the control unit 211 calculates the position of the gesture posture in the captured image by moving the gesture posture stored in the working memory 213 by the amount of change in position (Δx1, Δy1, Δz1) calculated in step S603.

[0065] As described above, according to FIG. 6, even if a part of the user's hand used for calculating the gesture posture is hidden in the captured image, the gesture posture can be stably calculated.

[0066] <Second method for calculating gesture posture when reference posture and gesture posture are stored> 7, a flowchart will be described illustrating a method for calculating a gesture posture in a captured image using the relationship between the position and orientation from a stored reference posture to the stored gesture posture. This process is executed in step S407 when the gesture posture and the reference posture are stored in working memory 213 in step S406 of FIG. 4, for example. Note that the process in step S701 below may be executed in step S406 of FIG. 4 to calculate correction information and store it in the working memory, and the process in step S702 below may be executed in step S407 of FIG. 4 to calculate the gesture posture.

[0067] In step S701, control unit 211 calculates a rotation amount θ2, which is the difference between the orientation of the reference orientation stored in working memory 213 and the orientation of the gesture orientation stored in working memory 213, and proceeds to step S702. Note that the orientation of each reference orientation may be expressed by a 3×3 rotation matrix or quaternion, and the rotation amount θ2 may be calculated. Note that at this time, if there are multiple combinations of reference orientations and gesture orientations stored in working memory 213, the rotation amount from each reference orientation to the gesture orientation may be calculated, and then the average value thereof may be used.

[0068] In step S702, the control unit 211 calculates the direction of the gesture posture in the captured image by rotating the reference posture in the captured image by the rotation amount θ2 calculated in step S701.

[0069] In step S703, control unit 211 calculates a position change amount (Δx2, Δy2, Δz2) that is a difference between the position of the reference posture stored in working memory 213 and the position of the gesture posture stored in working memory 213, and proceeds to step S704. Note that the reference posture, gesture posture, and position change amount may be expressed in a three-dimensional coordinate system or a predefined two-dimensional coordinate system. Note that at this time, if there are multiple combinations of reference postures and gesture postures stored in working memory 213, the position change amount from each reference posture to the gesture posture may be calculated, and then the average value thereof may be used.

[0070] In step S704, the control unit 211 calculates the position of the gesture posture in the captured image by moving the reference posture in the captured image by the amount of change in position (Δx2, Δy2, Δz2) calculated in step S703.

[0071] As described above, according to FIG. 7, even if a part of the user's hand used for calculating the gesture posture is hidden in the captured image, the gesture posture can be stably calculated.

[0072] <Relationship between captured images and flow> With reference to Figures 8(a), 8(b), 8(c), 8(d), 8(e), 8(f), and 8(g), the relationship between the captured image acquired by HMD100 and the processing performed in the flow of Figure 4 will be explained.

[0073] 8(a) is a diagram illustrating a scene in which a hand 801 making a specific gesture is included in a captured image 800. When an image like the captured image 800 is acquired in step S401 of FIG. 4, the control unit 211 determines in step S404 that the hand is making a specific gesture, and proceeds to step S405.

[0074] 8(b) is a diagram showing the reference posture 802 calculated in step S405 and the gesture posture 803. When such reference posture 802 and gesture posture 803 are acquired, in step S406, correction information acquired by calculating the difference between the gesture posture 803 and the reference posture 802 is stored in the working memory 213.

[0075] 8(c) is a diagram showing a display image 820 generated in step S410 from the captured image 800. In the display image 820, a virtual object 804 drawn based on the gesture posture 803 in step S409 is superimposed.

[0076] FIG. 8(d) illustrates a frame in which captured image 840 includes (shows) a hand 841 that is not making a specific gesture. It is assumed that captured image 840 is an image acquired after captured image 800. In captured image 840, it is assumed that the hand is actually moving while maintaining the same shape as hand 801 in captured image 800, but the orientation of the hand is changing. That is, it is assumed that the user is still making a pinch pose. However, in captured image 840, the back of the hand is visible, but the thumb and index finger are not. Therefore, the positions of the fingers cannot be confirmed in the image, and it is impossible to determine whether all the fingers are folded or whether they are making a pinch pose similar to hand 801. Thus, if an image like captured image 840 is acquired in step S401 of FIG. 4, control unit 211 determines in step S404 that the hand is not making a specific gesture, and proceeds to step S407.

[0077] 8(e) is a diagram showing the reference orientation 852 calculated in step S407 or step S408 in the captured image 840. Although the position of the finger can be estimated using only this image information, it is not possible to confirm whether the estimation result is truly correct.

[0078] 8(f) is a diagram showing a gesture posture 863 calculated in step S408. Here, the gesture posture 863 is calculated from the correction information calculated based on the captured image 800 and stored in the working memory 213, and the reference posture 852 acquired from the captured image 840. When the captured image 840 is acquired, in step S409, the orientation for displaying the virtual object is calculated based on the gesture posture 863, and the virtual object is drawn.

[0079] 8(g) is a diagram showing a display image 870 generated in step S410 by superimposing the virtual object 874 drawn in step S409 on the captured image 840. Here, control unit 211 acquires depth information of the hand and expresses the depth relationship between the virtual object and hand 841, and therefore assumes a scene in which the part of virtual object 874 hidden behind the hand is not superimposed on the captured image 840.

[0080] In this way, even if a specific gesture is detected and a state in which a virtual object is grasped is detected, and then a specific part of the user's hand is hidden and the specific gesture is no longer being made, the posture (shape) of the specific gesture is estimated if the specific part is included. Then, by drawing a virtual object based on the estimated posture (shape) of the specific gesture and generating an image for display, it is possible to express a state in which the virtual object is being grasped even if a part of the hand is hidden. In other words, although the specific gesture is not made (it is not clear that a specific gesture is being made) from the captured image 840 alone, the posture of the hand 841 is estimated assuming that the hand 841 makes a specific gesture. In other words, the posture of the hand 841 is estimated based on information acquired from the captured image 800, assuming that the hand 841 in the captured image 840 makes a specific gesture.

[0081] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0082] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0083] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0084] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0085] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0086] The disclosure of this embodiment includes the following configuration, method, and program.

[0087] [Configuration 1] a storage means for storing information based on a first posture in a storage unit when a first hand included in the first captured image has a first posture indicating a specific gesture; and an estimation means for estimating a posture of the second hand when the specific gesture is being performed based on the information and a posture of the specific portion when the second hand included in the second captured image does not perform the specific gesture in the second captured image captured after the first captured image, if the second captured image includes a specific portion of the second hand. 1. An information processing device comprising:

[0088] [Configuration 2] the storage means stores the first attitude as the information; The estimation means estimates the posture of the second hand showing the specific gesture based on the first posture and the posture of the specific part of the second hand when the specific part is included in the second captured image, even if the second hand does not show the specific gesture in the second captured image. 2. The information processing device according to configuration 1,

[0089] [Configuration 3] the storage means further stores, as the information, a posture of the specific part of the first hand; When the second captured image includes the specific part even if the second hand does not make the specific gesture in the second captured image, the estimation means estimates the posture of the second hand making the specific gesture based on the first posture, the posture of the specific part of the first hand, and the posture of the specific part of the second hand. 3. The information processing device according to configuration 1 or 2.

[0090] [Configuration 4] the storage means stores, as the information, a difference in position and orientation between the first posture and the posture of the specific part of the first hand; The estimation means estimates the posture of the second hand showing the specific gesture based on the difference and the posture of the specific part of the second hand when the specific part is included in the second captured image even if the second hand does not show the specific gesture in the second captured image. 4. The information processing device according to any one of configurations 1 to 3.

[0091] [Configuration 5] The method further includes a determination unit for determining whether or not a hand included in a captured image is making the specific gesture, the storage means stores the information in the storage unit when the determination means determines that the first hand makes the specific gesture; The estimation means estimates the posture of the second hand showing the specific gesture based on the information and the posture of the specific part when the specific part is included in the second captured image, even if the determination means does not determine that the second hand shows the specific gesture because a part of the hand is hidden in the second captured image. 5. The information processing device according to any one of configurations 1 to 4.

[0092] [Configuration 6] The determination means determines whether the hand is making the specific gesture based on the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints and fingertips of the hand. 6. The information processing device according to configuration 5.

[0093] [Configuration 7] The determining means determines whether the hand is making the specific gesture by classifying the hand. 7. The information processing device according to configuration 5 or 6.

[0094] [Configuration 8] The present invention further includes a detection means for detecting the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints of the hand and the fingertips, from the captured image; The storage means stores the information in the storage unit based on the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints of the hand and the fingertips. 8. The information processing device according to any one of configurations 1 to 7.

[0095] [Configuration 9] The specific part is a part of the hand that can be detected with high accuracy. 9. The information processing device according to any one of configurations 1 to 8.

[0096] [Configuration 10] The specific gesture is a pinch gesture in which the thumb and index finger are brought together, or a grasp gesture in which the hand is clenched. 10. The information processing device according to any one of configurations 1 to 9.

[0097] [Configuration 11] a storage means for storing information based on a first posture in a storage unit when a first hand included in the first captured image has a first posture indicating a specific gesture; a display control means for controlling a display unit to display a first image on which a virtual object having an orientation corresponding to the first posture is superimposed when the first hand is in the first posture in the first captured image, The display control means controls the display unit to display a second image, in which the virtual object is superimposed in an orientation corresponding to the posture of a specific part of the second hand, when the second captured image includes a specific part of the second hand even if the second hand included in the second captured image does not make the specific gesture. 1. An information processing device comprising:

[0098] [Configuration 12] the storage means stores, as the information, a difference in position and orientation between the first posture and the posture of the specific part of the first hand; The display control means controls the display unit to display the second image generated based on the difference and the posture of the specific part. 12. The information processing device according to configuration 11.

[0099] [Configuration 13] The apparatus further includes an estimation unit for estimating the posture of the second hand when the specific gesture is performed based on the information and the posture of the specific part in the second captured image. 13. The information processing device according to configuration 11 or 12.

[0100] [Configuration 14] The present invention further includes a generating means for generating a virtual object based on the posture of the second hand estimated by the estimating means. 14. The information processing device according to configuration 13.

[0101] [Configuration 15] The image generating device further includes an image generating unit that generates the first image by combining the virtual object in an orientation corresponding to the first posture with the first captured image, and generates the second image by combining the virtual object in an orientation corresponding to the posture of the specific part with the second captured image. 15. The information processing device according to any one of configurations 11 to 14.

[0102] [program] 16. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 15.

[0103] [method] a storage step of storing information based on a first posture in a storage unit when a first hand included in the first captured image has a first posture indicating a specific gesture; and an estimation step of estimating a posture of the second hand when the specific gesture is performed based on the information and a posture of the specific portion when the second hand included in the second captured image captured after the first captured image is not performing the specific gesture, if the second captured image includes the specific portion of the second hand. 2. A method for controlling an information processing apparatus comprising:

[0104] [system] a storage device that stores information based on a first posture of a first hand included in a first captured image when the first hand is in a first posture that indicates a specific gesture; and an estimation device that, even if a second hand included in a second captured image captured after the first captured image does not make the specific gesture, if a specific part of the second hand is included in the second captured image, estimates the posture of the second hand when making the specific gesture based on the information and the posture of the specific part. An information processing system comprising:

[0105] [method] a storage step of storing information based on a first posture in a storage unit when a first hand included in the first captured image has a first posture indicating a specific gesture; a display control step of controlling, when the first hand is in the first posture in the first captured image, to display on a display unit a first image in which a virtual object having an orientation corresponding to the first posture is superimposed; In the display control step, even if a second hand included in a second captured image captured after the first captured image does not make the specific gesture, if a specific part of the second hand is included in the second captured image, control is performed to display on the display unit a second image in which the virtual object is superimposed in an orientation corresponding to the posture of the specific part. 2. A method for controlling an information processing apparatus comprising:

[0106] [system] A display device; a storage device that stores information based on a first posture of a first hand included in a first captured image when the first hand is in a first posture that indicates a specific gesture; a display control device that controls the display device to display, when the first hand is in the first posture in the first captured image, a first image in which a virtual object having an orientation corresponding to the first posture is superimposed; The display control device controls the display device to display a second image on which the virtual object is superimposed in an orientation corresponding to the posture of a specific part of the second hand when the second captured image includes a specific part of the second hand even if the second captured image does not include the specific gesture. An information processing system comprising:

Claims

1. a storage unit configured to store information based on a first posture of a first hand included in a first captured image, the first posture indicating a specific gesture, in a storage unit; and an estimation means for estimating a posture of the second hand when the specific gesture is being performed based on the information and a posture of the specific portion of the second hand when the specific gesture is being performed, in a case where the second captured image includes a specific part of the second hand even if the second hand included in the second captured image does not perform the specific gesture in the second captured image captured after the first captured image.

1. An information processing device comprising:

2. the storage means stores the first attitude as the information; The estimation means estimates the posture of the second hand showing the specific gesture based on the first posture and the posture of the specific part of the second hand when the specific part is included in the second captured image even if the second hand does not show the specific gesture in the second captured image.

2. The information processing apparatus according to claim 1, wherein:

3. the storage means further stores, as the information, a posture of the specific part of the first hand; The estimation means estimates the posture of the second hand showing the specific gesture based on the first posture, the posture of the specific part of the first hand, and the posture of the specific part of the second hand, when the specific part is included in the second captured image, even if the second hand does not show the specific gesture in the second captured image.

2. The information processing apparatus according to claim 1, wherein:

4. the storage means stores, as the information, a difference in position and orientation between the first posture and the posture of the specific part of the first hand; The estimation means estimates the posture of the second hand showing the specific gesture based on the difference and the posture of the specific part of the second hand when the specific part is included in the second captured image, even if the second hand does not show the specific gesture in the second captured image.

2. The information processing apparatus according to claim 1, wherein:

5. The method further includes a determination unit for determining whether or not a hand included in a captured image is making the specific gesture, the storage means stores the information in the storage unit when the determination means determines that the first hand makes the specific gesture; The estimation means estimates the posture of the second hand showing the specific gesture based on the information and the posture of the specific part when the specific part is included in the second captured image, even if the determination means does not determine that the second hand shows the specific gesture because a part of the hand is hidden in the second captured image.

2. The information processing apparatus according to claim 1, wherein:

6. The determination means determines whether the hand is making the specific gesture based on the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints and fingertips of the hand.

6. The information processing apparatus according to claim 5,

7. The determining means determines whether the hand is making the specific gesture by classifying the hand.

6. The information processing apparatus according to claim 5,

8. The present invention further includes a detection means for detecting the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints of the hand and the fingertips, from the captured image; The storage means stores the information in the storage unit based on the positions of a plurality of joint points, which are points that estimate the positions of at least one of the joints of the hand and the fingertips.

2. The information processing apparatus according to claim 1, wherein:

9. The specific part is a part of the hand that can be detected with high accuracy.

2. The information processing apparatus according to claim 1, wherein:

10. The specific gesture is a pinch gesture in which the thumb and index finger are brought together, or a grasp gesture in which the hand is clenched.

2. The information processing apparatus according to claim 1, wherein:

11. a storage unit configured to store information based on a first posture of a first hand included in a first captured image, the first posture indicating a specific gesture, in a storage unit; a display control means for controlling, when the first hand is in the first posture in the first captured image, to display on a display unit a first image onto which a virtual object having an orientation corresponding to the first posture is superimposed; The display control means controls the display unit to display a second image, on which the virtual object is superimposed in an orientation corresponding to the posture of a specific part of the second hand, when the second captured image includes a specific part of the second hand even if the second hand included in the second captured image does not make the specific gesture.

1. An information processing device comprising:

12. the storage means stores, as the information, a difference in position and orientation between the first posture and the posture of the specific part of the first hand; The display control means controls the display unit to display the second image generated based on the difference and the posture of the specific part.

12. The information processing apparatus according to claim 11,

13. The apparatus further includes an estimation unit for estimating the posture of the second hand when the specific gesture is performed based on the information and the posture of the specific part in the second captured image.

12. The information processing apparatus according to claim 11,

14. The present invention further includes a generating means for generating a virtual object based on the posture of the second hand estimated by the estimating means.

14. The information processing apparatus according to claim 13,

15. The image generating device further includes an image generating unit that generates the first image by combining the virtual object in an orientation corresponding to the first posture with the first captured image, and generates the second image by combining the virtual object in an orientation corresponding to the posture of the specific part with the second captured image.

12. The information processing apparatus according to claim 11,

16. 12. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 11.

17. a storage step of storing information based on a first posture of a first hand included in a first captured image, the first hand having the first posture indicating a specific gesture, in a storage unit; and an estimation step of estimating a posture of the second hand when the specific gesture is performed based on the information and a posture of the specific portion when the second hand included in the second captured image captured after the first captured image is not performing the specific gesture but the specific portion of the second hand is included in the second captured image.

2. A method for controlling an information processing apparatus comprising:

18. a storage device that stores information based on a first posture of a first hand included in a first captured image when the first hand has the first posture indicating a specific gesture; and an estimation device that, even if a second hand included in a second captured image captured after the first captured image does not make the specific gesture, if a specific part of the second hand is included in the second captured image, estimates the posture of the second hand when making the specific gesture based on the information and the posture of the specific part. An information processing system comprising:

19. a storage step of storing information based on a first posture of a first hand included in a first captured image, the first hand having the first posture indicating a specific gesture, in a storage unit; a display control step of controlling, when the first hand is in the first posture in the first captured image, to display on a display unit a first image in which a virtual object having an orientation corresponding to the first posture is superimposed; In the display control step, even if a second hand included in a second captured image captured after the first captured image does not make the specific gesture, if a specific part of the second hand is included in the second captured image, a second image in which the virtual object is superimposed in an orientation corresponding to the posture of the specific part is controlled to be displayed on the display unit.

2. A method for controlling an information processing apparatus comprising:

20. a display device; a storage device that stores information based on a first posture of a first hand included in a first captured image when the first hand has the first posture indicating a specific gesture; a display control device that controls the display device to display, when the first hand is in the first posture in the first captured image, a first image in which a virtual object having an orientation corresponding to the first posture is superimposed; The display control device controls the display device to display a second image on which the virtual object is superimposed in an orientation corresponding to the posture of a specific part of the second hand when the second captured image includes a specific part of the second hand even if the second captured image does not include the specific gesture. An information processing system comprising:

Images