Information processing device, information processing system, information processing program, and information processing method

Abstract

If the determination that a virtual object has been grabbed is made earlier than expected, the user may feel uncomfortable interacting with the virtual object, resulting in a reduced sense of control. [Solution] The information processing device has an acquisition means for acquiring an image, an estimation means for estimating whether the user's hand in the image acquired by the acquisition means has a specific shape based on class classification, a detection means for detecting the positions of a plurality of joint points from the image, which are points that estimate the positions of at least one of the joints and fingertips of the user's hand in the image, and a determination means for, when the estimation means estimates that the user's hand in the image has the specific shape, determining whether the user's hand in the image has the specific shape based on the positions of the plurality of joint points detected by the detection means.

Description

Technical Field

[0004] , , , , , ,

[0003] , ,

[0001] The present invention relates to an information processing apparatus that determines whether or not a user is making a hand pose.

Background Art

[0002] As technologies for fusing the real world and computer-generated CG (Computer Graphics) in real time, there are technologies such as mixed reality (MR) and augmented reality (AR). Mixed reality and augmented reality use a device called a head-mounted display (HMD) that is worn on the user's head to present a composite image of the real world and CG to the user, and enable interaction between the user and the CG to provide an immersive experience. As one of the operation means for the user to interact with the CG, there is a hand gesture operation. In order to realize the hand gesture operation, a technique for determining the hand pose made by the user using a machine learning model is used. For example, in Patent Document 1, a technique is disclosed in which a user's hand is photographed with a camera mounted on an HMD, and an image taken is analyzed using a machine learning model or the like and classified into a pre-defined hand pose class to determine the hand pose. By using such a technique, for example, a virtual object can be grasped by making a hand pose of grasping the virtual object near the virtual object to be operated, and the virtual object can be moved in accordance with the movement of the user's hand.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the technology disclosed in Patent Document 1, it can be difficult to match a specific hand pose to the feel of each user. For example, in a hand pose called a pinch, where a virtual object is grasped with the thumb and index finger, it is necessary to make a pose as if grasping space in real space, so the distance between the thumb and index finger that makes a person feel like they are grasping a virtual object will differ from person to person. For example, as shown in Figure 7, when grasping a virtual object 701 with a pinch hand pose, it is assumed that some people will make a pinch hand pose 603 where they bring their fingers together near the virtual object 701, as shown in Figure 7(a). In addition, it is assumed that some people will make a pinch hand pose 604 where they grasp the handle part of the virtual object 701 with their thumb and index finger, as shown in Figure 7(b). Therefore, if the feel of grasping a virtual object is made to be such that a person who tries to grasp a virtual object with the hand shape in Figure 7(a) can grasp the virtual object with the hand shape in Figure 7(b), the judgment of grasping the virtual object will be made earlier than intended, which may increase the likelihood of errors. In this way, users may feel uncomfortable interacting with virtual objects, which can reduce the overall user experience.

[0005] Therefore, the present invention aims to reduce the discrepancy between the hand posture that a user perceives as performing a specific hand pose and the hand pose in which interaction with a virtual object actually occurs, thereby improving the user's sense of control. [Means for solving the problem]

[0006] To achieve the above objective, the information processing apparatus of the present invention includes: acquisition means for acquiring an image; estimation means for estimating whether the user's hand in the image acquired by the acquisition means has a specific shape based on class classification; detection means for detecting the positions of a plurality of joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand in the image; determination means for determining whether the user's hand in the image has the specific shape based on the positions of the plurality of joint points detected by the detection means, when the estimation means estimates that the user's hand in the image has the specific shape; and control means for controlling the processing to be performed according to the specific shape when the estimation means estimates that the user's hand in the image has the specific shape based on class classification and the determination means determines that the user's hand has the specific shape. The control means is characterized in that, even if the estimation means estimates that the user's hand in the image has the specific shape based on class classification, if the determination means does not determine that the user's hand has the specific shape, the control means does not perform the processing. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that reduces the discrepancy between the hand posture that a user perceives as performing a specific hand pose and the hand pose in which interaction with a virtual object actually occurs, thereby improving the user's sense of control. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the information processing system according to Embodiment 1. [Figure 2] This is a diagram illustrating the internal configuration of the HMD according to Embodiment 1. [Figure 3] This is a flowchart of the hand pose detection process according to Embodiment 1. [Figure 4] This is a flowchart of the hand pose detection process according to Embodiment 2. [Figure 5] This is a flowchart of the virtual object manipulation process according to Embodiment 1. [Figure 6] This figure illustrates an example of a hand pose according to Embodiment 1. [Figure 7] This is a diagram illustrating an example of a pinch pose according to Embodiment 1. [Figure 8] This figure illustrates an example of a composite image drawn according to Embodiment 1. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted from the drawings.

[0010] (Embodiment 1) <System Configuration> Referring to Figure 1, the information processing system 1 according to Embodiment 1 will be described. The information processing system 1 includes a head-mounted display (HMD) 100 and a PC (personal computer) 110.

[0011] The HMD100 is a head-mounted display device (electronic device) that can be worn on the user's head. The HMD100 is equipped with a camera for capturing images in front of the user and a display for showing images to the user. The HMD100's display shows a composite image, which is a combination of the image captured by the HMD100 in front of the user and content such as CG (computer graphics) in a form corresponding to the HMD100's posture. This allows the user to experience virtual reality with their own eyes. Furthermore, System 1 has a function that detects the user's hands from the images captured by the HMD100 and acquires information related to the position and orientation of the hands as hand posture, thereby allowing the hand movements to affect virtual objects. This allows the user to intuitively manipulate virtual objects using their hands.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 100 and the HMD 101 can communicate with each other through wireless or wired communication and transmit and receive images and other necessary information. The PC 110 generates a composite image by synthesizing the image captured by the HMD 100 and the CG generated by the PC 110, and transmits the composite image to the HMD 100. Here, a PC is described as an example of the information processing device, but the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be possessed by the HMD 100.

[0013] <Internal Structure of HMD> Referring to FIG. 2, the internal structure of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, and a working memory 206.

[0014] The HMD control unit 201 controls each component of the HMD 100. The HMD control unit 201 includes at least one CPU that executes a program stored in the non-volatile memory 205 and at least one other circuit. When the HMD control unit 201 acquires a composite image (an image in which the captured image of the space in front of the user captured by the imaging unit 202 and the CG are synthesized) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Instead of the HMD control unit 201 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0015] The imaging unit 202 includes two cameras (imaging devices). The two cameras are arranged near the positions of the user's left and right eyes when the HMD 100 is worn on the user's head. Therefore, the two cameras can capture a space similar to the space that the user wearing the HMD 100 is looking at. The images captured by the imaging unit 202 are output to the HMD control unit 201, and the HMD control unit 201 transmits the images from the imaging unit 202 to the PC 110. As will be described later, the PC 110 synthesizes the captured image transmitted from the HMD 100 and the CG to generate a synthesized image. Also, in the imaging unit 202, the two cameras simultaneously acquire a first image having a parallax with each other and a second image different from the first image. Therefore, using the images from the two cameras in the imaging unit 202, information on the distance (distance information) from the HMD 100 to the subject can be obtained. Note that the imaging unit 202 may capture and output a moving image.

[0016] As will be described later, when the synthesized image is transmitted from the PC 110, the image display unit 203 displays the synthesized image transmitted from the PC 110. The image display unit 203 has a display such as a liquid crystal panel or an organic EL panel. In a state where the user is wearing the HMD 100, displays such as an organic EL panel are arranged in front of each of the user's eyes. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image such that CG is directly superimposed on the real space visible through the half mirror by a technique generally called AR (Augmented Reality). Also, the image display unit 203 may display an image of a complete virtual space without using the captured image by a technique generally called VR (Virtual Reality).

[0017] The posture sensor unit 204 acquires the posture (and position) information of the HMD 100. Then, the posture sensor unit 204 acquires the posture information of the user (the user wearing the HMD 100) corresponding to the posture (and position) of the HMD 100. The posture sensor unit 204 includes an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. When the user is wearing the HMD 100, the posture sensor unit 204 acquires the information (posture information) of the user's posture. The HMD control unit 201 outputs the information (posture information) of the user's posture detected by the posture sensor 204 to the PC 110.

[0018] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs and the like executed by the HMD control unit 201.

[0019] The volatile memory 206 is used as a buffer memory for temporarily holding the image data captured by the imaging unit 202, an image display memory for the image display unit 203, a work area of the HMD control unit 201, and the like.

[0020] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, and a working memory 213.

[0021] The control unit 211 is a CPU composed of at least one processor or circuit. By executing the programs stored in the non-volatile memory 212, the control unit 211 realizes each process of the flowchart described later. Note that instead of the control unit 211 controlling the entire device, a plurality of hardware may share the processing to control the entire device. The control unit 211 receives, from the HMD 100, the image (captured image) acquired by the imaging unit 202 and the posture information acquired by the posture sensor unit 204. The control unit 211 synthesizes the captured image and an arbitrary CG based on the received information to generate a synthesized image. The control unit 211 transmits the synthesized image to the HMD control unit 201 in the HMD 100.

[0022] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory that stores information such as the program executed by the control unit 211 (described later) and computer graphics (CG). The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate the composite image).

[0023] The working memory 213 is a storage unit used as a working area for the control unit 211, such as a buffer memory that temporarily holds image data captured by the imaging unit 202.

[0024] <Flowchart explaining the process for determining hand pose> The process for determining hand poses will be explained with reference to the flowchart in Figure 3. The process in the flowchart in Figure 3 is executed from the moment the user launches an application on the HMD and is executed each time the imaging unit 202 acquires an image (each time a picture is taken). An application is, for example, an application that the user selects on the home screen (home space) after starting the HMD, and includes apps that allow the user to interact with virtual objects using hand gestures. Note that the timing of execution of this flowchart is not limited to the moment the user launches an application on the HMD. For example, it may be executed when the user starts the HMD or when a virtual object is displayed in the mixed reality space.

[0025] In step S301, the control unit 211 acquires the image captured by the imaging unit 202 and proceeds to step S302.

[0026] In step S302, the control unit 211 estimates the hand pose by class classification based on the captured image acquired from the imaging unit 202. Examples of hand pose classes include grip pose, pinch pose, and no pose. A grip pose is a hand that makes a fist, and a pinch pose is a hand that brings the tips of the thumb and index finger together to pinch something.

[0027] In this embodiment, the grip pose and pinch pose are used as hand poses when grasping a virtual object. "No pose" indicates a hand shape that does not fall under the grip pose or pinch pose. For example, a deep learning model trained with these hand poses as classes is used to perform classification processing. Note that the pinch pose is not limited to bringing the tips of the thumb and index finger together, but may also be an action that changes multiple fingers from a separated state to a state of bringing them together, such as bringing the thumb and middle finger together, or bringing the thumb, index finger, and middle finger together.

[0028] In deep learning models, for example, an image is input to a Convolutional Neural Network (CNN). The CNN outputs features used to identify the type of subject or the type of scene being photographed. These features are then used to identify the type of subject or scene being photographed. Examples of deep learning models used as object or person detection and classification systems include YOLO, MobileNet, VGG16, and SSD.

[0029] The control unit 211 uses a deep learning model to estimate which hand pose class the subject in the image captured by the imaging unit 202 belongs to, and records the assigned hand pose in the working memory 213 as the hand pose performed by the user. In other words, when an image is input to the deep learning model, it detects the region in the image that contains a hand. If a region containing a hand is detected in the image, it estimates (classifies) whether the shape of the subject is a specific shape. In this class classification process, if an image that does not contain a hand is input, nothing is detected from the input image. An example of such a hand pose will be explained with reference to Figure 6.

[0030] Figure 6(a) shows an example of a grip pose. Hand 601 is gripping with all fingers visible. Figure 6(b) shows an example of a grip pose. Hand 602 has only the thumb and index finger visible, with the other fingers hidden. Figure 6(c) shows an example of a pinch pose. Hand 603 has the tips of the thumb and index finger touching. Figure 6(d) shows an example of a pinch pose. Hand 604 does not have the tips of the thumb and index finger touching.

[0031] In step S303, the control unit 211 estimates the three-dimensional position of the joint points of the hand based on the image captured by the imaging unit 202. Here, "joint points" refer to the points of the joints of each finger, points indicating predetermined positions on the wrist, and points at the tips of each finger. In step S303, the position of at least one of the joints and fingertips of the user's hand is estimated as a joint point. For example, a deep learning model is used to estimate the three-dimensional position of the joint points of the hand.

[0032] The control unit 211 estimates the 3D position of each joint point in the image space of the image captured by the imaging unit 202, and the depth position of the wrist, with the wrist depth set as the reference value 0, along an axis perpendicular to the camera of the imaging unit 202. The estimated values ​​are recorded in the working memory 213.

[0033] An example of estimating the three-dimensional position of the joint points of the hand will be explained with reference to Figure 6. Figure 6(e) is an example of joint points when the three-dimensional position of the joint points of hand 603 in Figure 6(c) is estimated. A total of 21 joint points 605 are estimated, four points for each finger and one point at the wrist. Note that the deep learning model used to estimate the hand pose and the deep learning model used to estimate the three-dimensional position of the joint points of the hand may be the same model or different models.

[0034] In step S304, the control unit 211 determines whether the hand pose determined by the classification in step S302 is a grip pose or a pinch pose. If the hand pose performed by the user is a grip pose or a pinch pose, the process proceeds to step S305; otherwise, it proceeds to step S307. For example, if the hand is not visible in the captured image, or if the hand is visible in the captured image but it is presumed that the user is not performing a specific hand pose (hand gesture) such as a grip pose or a pinch pose, the process proceeds to step S307.

[0035] In step S305, the control unit 211 determines whether the user is in a grip pose or a pinch pose using the joint points estimated in step S303. If the user's hand pose is estimated to be a grip pose in step S302, the control unit 211 determines whether a grip pose is being performed based on the angles of the finger joint points. The angles of the finger joint points represent the angles formed by three adjacent joint points. If the angles of the finger joint points are smaller than a predetermined angle, the control unit determines that a grip pose has been performed and sets the determination result to TRUE.

[0036] The angle of the finger joint used for the determination can be any joint of the finger. Furthermore, one joint or multiple joints may be used for the determination. One finger or multiple fingers may be used for the determination.

[0037] If the user's hand pose is estimated to be a pinch pose in step S302, a determination is made as to whether a pinch pose is being performed based on the distance between the joint points at the tips of the thumb and index finger. If this distance is smaller than a predetermined distance (threshold), it is determined that a pinch pose has been performed, and the determination result is set to TRUE. If the determination result of the hand pose based on the joint points is TRUE, the process proceeds to step S306; otherwise, the process proceeds to step S307.

[0038] Furthermore, when determining a hand pose based on joint points after estimating it using a deep learning model, the determination may be performed using only the characteristic parts, depending on the estimated hand pose. For example, when determining whether a pinch pose is being performed based on the distance between the joint points at the tips of the thumb and index finger, only the joint points at the tips of the thumb and index finger may be extracted, and the determination of whether a pinch pose is being performed may be made. In this way, estimating the hand pose by class classification first has the effect of reducing the processing load of determining the hand pose based on joint points.

[0039] In step S306, if the grip pose determination result in step S305 is TRUE, the control unit 211 records TRUE in the working memory 213, indicating that a grip pose is being performed. Also, if the pinch pose determination result in step S305 is TRUE, the control unit 211 records TRUE in the working memory 213, indicating that a pinch pose is being performed, and proceeds to step S308.

[0040] In step S307, the control unit 211 records FALSE in the working memory 213, indicating that grip pose and pinch pose are not being performed, and proceeds to step S308.

[0041] In step S308, the control unit 211 performs virtual object manipulation processing based on the determination result recorded in the working memory 213 in step S306 or step S307, and proceeds to step S309. The virtual object manipulation processing performed in step S308 will be described later with reference to Figure 5.

[0042] In step S309, the control unit 211 determines whether the application has terminated. If the application has terminated, the process proceeds to termination; otherwise, the process from step S301 is repeated.

[0043] In the flowchart of Figure 3, the order of steps S302 and S303 may be reversed, or they may be performed simultaneously. For example, one deep learning model may perform both hand pose estimation by classification and hand pose determination by joint points. Also, the processing in steps S302 and S303 may be performed on both the right eye image and the left eye image, or on only one of the right eye image or the left eye image.

[0044] According to Embodiment 1, the control unit 211 can determine the hand pose by estimating the hand pose through class classification and determining the hand pose through joint points, thereby discriminating even the fine details of the fingers. This improves the user's feel for manipulating the virtual object.

[0045] <Flowchart for manipulating virtual objects> Referring to the flowchart in Figure 5, the virtual object manipulation process performed in step S308 of Figure 3 will be explained.

[0046] In step S501, the control unit 211 determines whether or not a virtual object exists near the hand. If the control unit 211 determines that a virtual object exists near the hand, it proceeds to step S502. If it does not determine that a virtual object exists near the hand, it terminates the virtual object manipulation process flow.

[0047] In step S502, the control unit 211 determines whether the user's hand pose determination result is set to TRUE. If the control unit 211 determines that the hand pose determination result is set to TRUE, it proceeds to step S503. If it determines that the hand pose determination result is set to FALSE, it terminates the virtual object manipulation process flow.

[0048] In step S503, the control unit 211 moves the virtual object based on the hand position, superimposes it onto the captured image, and ends the virtual object manipulation process.

[0049] Thus, if a virtual object is located near the hand and the hand pose detection result is set to TRUE, the virtual object is rendered to follow the hand's position, assuming the user has selected and moved the virtual object.

[0050] Furthermore, if the system does not determine that a virtual object is located near the hand, or if the hand pose determination is set to FALSE, the virtual object will not be moved based on the hand's position.

[0051] <Explanation of the scene where you stop manipulating a virtual object> Referring to Figure 8, an example of a scene where manipulation of a virtual object is stopped is explained. In this scene, the first captured image 801 shows a hand 603 in a pinch position, and the virtual object 701 is positioned to match the hand's position. At this point, the user has selected (grabbed) the virtual object. In addition, the composite image 802 generated after the captured image 801 shows the hand 803 in a non-pinch position, and the virtual object 701 is not selected.

[0052] Here, we assume that in the first captured image, the user is performing a pinch pose to select a virtual object, and in the second captured image, which is acquired after the first captured image, the user has released the pinch pose. In this case, in the flow when acquiring the first captured image, the process in step S503 is executed, and composite image 801 is generated. Also, in the flow when acquiring the second captured image, the hand pose determination result is determined to be set to FALSE, and the virtual object manipulation process is terminated with the virtual object in the position of the virtual object from the flow at the first time, and composite image 802 is generated. In other words, at the second time, image 802 in the mixed reality space is generated while maintaining the position of the virtual object drawn at the first time. Note that if the position of the field of view of the captured image changes, such as when the user moves their head, the position of the virtual object is adjusted according to that change.

[0053] (Embodiment 2) Because hand pose estimation using class classification can lead to misestimations, it may be incorrectly estimated that a hand pose is not being performed even when it is. Since the hand pose from when a virtual object was grasped is maintained until the virtual object is released, this misestimation may lead to the virtual object being released because it is judged that no hand pose is being performed, potentially reducing operability. Embodiment 2 describes a case where only the joint points are used to determine the hand pose after grasping a virtual object in order to mitigate such a reduction in operability.

[0054] Referring to Figure 4, the process after the hand pose is performed will be described. The process that is the same as in Figure 3 of Embodiment 1 will not be described, and steps S404, S409, and S410 will be described.

[0055] Steps S401 to S403 are the same processes as steps S301 to S303 in Figure 3, so their explanation is omitted. After performing the process in step S403, the control unit 211 proceeds to step S404.

[0056] In step S404, the control unit 211 determines whether a grip pose or pinch pose is in progress based on the setting information of the grip pose or pinch pose recorded in the working memory 213. If a grip pose or pinch pose is in progress, the process proceeds to step S409; otherwise, the process proceeds to step S405.

[0057] Steps S405 to S408 are the same processes as steps S305 to S309 in Figure 3, so their explanation is omitted. After performing the process in step S407, the control unit 211 proceeds to step S411. Also, after performing the process in step S408, the control unit 211 proceeds to step S411.

[0058] In step S409, the control unit 211 performs the same processing as in S305 in Figure 3. If the result of the grip pose or pinch pose determination is TRUE, proceed to step S411; otherwise, proceed to step S410.

[0059] In step S410, the control unit 211 records FALSE in the working memory 213, indicating that grip pose and pinch pose are not being performed, and proceeds to step S411.

[0060] Steps S411 to S412 are the same process as steps S308 to S309 in Figure 3, so their explanation is omitted.

[0061] Thus, in Embodiment 2, as long as the user's hand is determined to be of a specific shape, the determination of whether the user's hand is of a specific shape is made based on the estimation results of the joint points of the hand, rather than on the estimation results of the hand pose by class classification. Alternatively, the determination of whether the user's hand is of a specific shape may be made based on the estimation results of the joint points of the hand, rather than on the estimation results of the hand pose by class classification, until the user's hand is no longer determined to be of a specific shape.

[0062] Furthermore, in the flow chart of Figure 4, the hand pose estimation by classification was performed regardless of whether or not it was determined whether the user's hand was of a specific shape, using the results of the hand pose estimation by classification. However, if it is not determined whether or not the user's hand is of a specific shape using the results of the hand pose estimation by classification, then it is not necessary to perform the hand pose estimation by classification. In other words, as long as the user's hand is determined to be of a specific shape, it is possible to determine whether or not the user's hand is of a specific shape from the results of the hand joint point estimation without performing the hand pose estimation by classification. Also, until the user's hand is no longer determined to be of a specific shape, it is possible to determine whether or not the user's hand is of a specific shape from the results of the hand joint point estimation without performing the hand pose estimation by classification.

[0063] As described above, according to Embodiment 2, when a grip pose or pinch pose is performed, the control unit 211 determines the hand pose from the joint points only, thereby preventing a decrease in operability due to misestimation of class classification. Therefore, the user's feel for manipulating the virtual object can be improved.

[0064] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0065] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0066] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0067] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0068] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0069] This embodiment includes the following configurations, methods, and programs.

[0070] [Configuration 1] An acquisition means for acquiring captured images, Based on the classification, an estimation means estimates whether the user's hand in the captured image acquired by the acquisition means has a specific shape. A detection means for detecting the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand in the captured image, If the estimation means estimates that the user's hand in the captured image has the specific shape, the system includes a determination means that determines whether the user's hand in the captured image has the specific shape based on the positions of the plurality of joint points detected by the detection means. An information processing device characterized by the following:

[0071] [Configuration 2] The determination means determines that the user's hand is the specific shape if the shape of the hand formed from the positions of the plurality of joint points is the specific shape. The information processing device according to configuration 1, characterized by the above.

[0072] [Configuration 3] The determination means does not determine that the user's hand is the specific shape if the shape of the hand formed by the positions of the multiple joint points is not the specific shape. The information processing apparatus according to configuration 2, characterized in that...

[0073] [Structure 4] The system further includes a control means that controls the system to perform processing corresponding to the specific shape of the user's hand if the determination means determines that the user's hand is of the specific shape. An information processing apparatus according to configuration 2 or 3, characterized by the above.

[0074] [Composition 5] If the estimation means does not estimate that the user's hand is of the specific shape, the determination means does not determine whether the user's hand is of the specific shape. An information processing apparatus according to any one of configurations 1 to 4, characterized by the above.

[0075] [Composition 6] The determination means determines whether the user's hand has the specific shape based on the angle formed by three adjacent joint points among the plurality of joint points. An information processing device according to any one of configurations 1 to 5, characterized by the above.

[0076] [Composition 7] If the estimation means estimates that the user's hand is of the specific shape, and the determination means determines that the user's hand is of the specific shape, the determination means will determine whether the user's hand is of the specific shape, regardless of the estimation result of the estimation means, until the determination means no longer determines that the user's hand is of the specific shape. An information processing device according to any one of configurations 1 to 6.

[0077] [Structure 8] If the estimation means estimates that the user's hand is of the specific shape, and the determination means determines that the user's hand is of the specific shape, the determination means will determine whether the user's hand is of the specific shape without the estimation means estimating whether the user's hand is of the specific shape until the determination means no longer determines that the user's hand is of the specific shape. An information processing device according to any one of configurations 1 to 7, characterized by the above.

[0078] [Composition 9] The aforementioned specific shape is a hand shape in which the distance between the thumb and index finger is less than a threshold. An information processing device according to any one of configurations 1 to 8.

[0079] [Configuration 10] The aforementioned specific shape is the shape when grasping a virtual object with your hand, or the shape when gripping a virtual object with your hand. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above.

[0080] [Composition 11] The acquisition means acquires a first image having parallax with respect to the first image and a second image that is different from the first image. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above.

[0081] [Composition 12] The estimation means estimates whether the user's hand, as seen in the first image and the second image acquired by the acquisition means, has a specific shape. The information processing apparatus according to configuration 11, characterized by the features described above.

[0082] [Control Method] The acquisition step involves obtaining the captured image, Based on the classification, an estimation step is performed to estimate whether the user's hand captured in the image acquired in the acquisition step is of a specific shape. A detection step involves detecting the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand, from the captured image obtained in the acquisition step. If the estimation step estimates that the user's hand has the specific shape, the method includes a determination step that determines whether the user's hand has the specific shape based on the positions of the plurality of joint points detected in the detection step. A control method for an information processing device characterized by the following features.

[0083] [program] A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 12.

[0084] [system] An acquisition device for acquiring captured images, An estimation device that estimates whether the user's hand, as seen in the image captured by the acquisition device, has a specific shape, based on the classification of the class, A detection device that detects the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand, from the captured image acquired by the acquisition device, The system includes, if the estimation device estimates that the user's hand has the specific shape, a determination device that determines whether the user's hand has the specific shape based on the positions of the plurality of joint points detected by the detection device. An information processing system characterized by the following:

Claims

1. An acquisition means for acquiring captured images, Based on the classification, an estimation means estimates whether the user's hand in the captured image acquired by the acquisition means has a specific shape. A detection means for detecting the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand in the captured image, If the estimation means estimates that the user's hand in the captured image has the specific shape, the determination means determines whether the user's hand in the captured image has the specific shape based on the positions of the plurality of joint points detected by the detection means. The system includes a control means which controls the system to perform processing corresponding to the specific shape when the estimation means estimates that the user's hand in the captured image is of the specific shape based on the class classification, and the determination means determines that the user's hand is of the specific shape. The control means controls the system so as not to perform the process if, even if the estimation means estimates that the user's hand in the captured image has the specific shape based on the class classification, the determination means does not determine that the user's hand has the specific shape. An information processing device characterized by the following:

2. The determination means determines that the user's hand is the specific shape if the shape of the hand formed from the positions of the plurality of joint points is the specific shape. The information processing apparatus according to feature 1.

3. If the estimation means does not estimate that the user's hand is of the specific shape, the determination means does not determine whether the user's hand is of the specific shape. The information processing apparatus according to feature 1.

4. The determination means determines whether the user's hand has the specific shape based on the angle formed by three adjacent joint points among the plurality of joint points. The information processing apparatus according to feature 1.

5. If the estimation means estimates that the user's hand is of the specific shape, and the determination means determines that the user's hand is of the specific shape, the determination means will determine whether the user's hand is of the specific shape, regardless of the estimation result of the estimation means, until the determination means no longer determines that the user's hand is of the specific shape. The information processing apparatus according to feature 1.

6. If the estimation means estimates that the user's hand is of the specific shape, and the determination means determines that the user's hand is of the specific shape, the determination means will determine whether the user's hand is of the specific shape without the estimation means estimating whether the user's hand is of the specific shape until the determination means no longer determines that the user's hand is of the specific shape. The information processing apparatus according to feature 1.

7. The aforementioned specific shape is a hand shape in which the distance between the thumb and index finger is less than a threshold. The information processing apparatus according to feature 1.

8. The aforementioned specific shape is the shape when grasping a virtual object with your hand, or the shape when gripping a virtual object with your hand. The information processing apparatus according to feature 1.

9. The acquisition means acquires a first image having parallax with respect to the first image and a second image that is different from the first image. The information processing apparatus according to feature 1.

10. The estimation means estimates whether the user's hand, as seen in the first image and the second image acquired by the acquisition means, has the specific shape. The information processing apparatus according to feature 9.

11. The determination means determines that the user's hand in the captured image has the specific shape if, based on the classification by the estimation means, the user's hand in the captured image has the specific shape, and the distance between the first joint point and the second joint point among the plurality of joint points is less than a threshold. The information processing apparatus according to feature 1.

12. The acquisition step involves obtaining the captured image, Based on the classification, an estimation step is performed to estimate whether the user's hand captured in the image acquired in the acquisition step is of a specific shape. A detection step involves detecting the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand, from the captured image obtained in the acquisition step. If the estimation step estimates that the user's hand has the specific shape, the determination step determines whether the user's hand has the specific shape based on the positions of the multiple joint points detected in the detection step. The estimation step estimates that the user's hand in the captured image has the specific shape based on the class classification, and the determination step determines that the user's hand has the specific shape, and the control step controls the system to perform processing corresponding to the specific shape. In the control step, even if the estimation step estimates that the user's hand in the captured image has the specific shape based on the class classification, if the determination step does not determine that the user's hand has the specific shape, the control is configured not to perform the process. A control method for an information processing device characterized by the following features.

13. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

14. An acquisition device for acquiring captured images, An estimation device that estimates whether the user's hand, as seen in the image captured by the acquisition device, has a specific shape, based on the classification of the class, A detection device that detects the positions of multiple joint points, which are points that estimate the position of at least one of the joints and fingertips of the user's hand, from the captured image acquired by the acquisition device, If the estimation device estimates that the user's hand has the specific shape, a determination device determines whether the user's hand has the specific shape based on the positions of the multiple joint points detected by the detection device. The estimation device estimates, based on the class classification, that the user's hand in the captured image is of the specific shape, and the determination device determines that the user's hand is of the specific shape, and the control device controls the device to perform processing corresponding to the specific shape. The control device is controlled not to perform the process if, even if the estimation device estimates that the user's hand in the captured image has the specific shape based on the class classification, the determination device does not determine that the user's hand has the specific shape. An information processing system characterized by the following:

Images