Information processing apparatus, information processing system, information processing method, and program

Abstract

To allow accurate acquisition (estimation) of the attitude of a user's hand based on information from a controller even if the controller held by the user's hand is of a small size.SOLUTION: An information processing apparatus estimates the attitude of the hand of a user who holds a controller by hand, and the information processing apparatus has: acquisition means that acquires inertial information from an inertial sensor included in the controller; determination means that determines whether a specific portion of the user's hand is detected in a picked-up image acquired by imaging means through imaging; and estimation means that, when the specific portion is detected in the picked-up image, estimates the attitude of the user's hand based on the picked-up image and the inertial information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004]

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, in a cross-reality (XR) system that allows a user to experience virtual reality, a hand controller is utilized to convert hand movements into operations in a virtual space when controlling the display of a head-mounted display (HMD). The HMD is a glasses-type device equipped with a small display that the user wears on the head.

[0003] In Patent Document 1, a hand controller has been proposed that can detect the position and posture of a hand by emitting a plurality of infrared lights (IR lights) from the hand controller and having a camera mounted on the HMD receive the infrared lights. Also, in Patent Document 2, a device has been proposed that reflects the position and posture of a user in a virtual space by comparing the body parts of the user shown in the captured image of a camera mounted on the HMD with a skeleton model stored in a memory.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technology disclosed in Patent Document 1 requires the implementation of multiple light-emitting diodes in the hand controller to detect its position and orientation, making it difficult to miniaturize the hand controller. Furthermore, the technology disclosed in Patent Document 2 has the problem that detection accuracy decreases depending on the orientation of the user's hand.

[0006] Therefore, the present invention aims to provide a technology that enables accurate acquisition (inference) of the user's hand posture based on information from the controller, even if the controller held by the user's hand is small. [Means for solving the problem]

[0007] One aspect of the present invention is, An information processing device that estimates the hand posture of a user holding a controller in their hand, The controller has an acquisition means for acquiring inertial information from an inertial sensor, A determination means for determining whether a specific part of the user's hand is detected in the image captured by the imaging means, If the specific part is detected in the captured image, estimation means for estimating the user's hand posture based on the captured image and the inertial information, This is an information processing device characterized by having [a certain feature].

[0008] One aspect of the present invention is, An information processing method for estimating the hand posture of a user holding a controller in their hand, The acquisition step involves acquiring inertial information from an inertial sensor provided by the controller, A determination step of determining whether a specific part of the user's hand is detected in the image captured by the imaging means, If the specific part is detected in the captured image, the estimation step involves estimating the user's hand posture based on the captured image and the inertial information. This is an information processing method characterized by having [a certain feature]. [Effects of the Invention]

[0009] According to the present invention, even if the controller held by the user's hand is small, the posture of the user's hand can be accurately acquired (estimated) based on the information of the controller.

Brief Description of Drawings

[0010] [Figure 1] It is a configuration diagram of a controller system according to Embodiment 1. [Figure 2] It is a diagram for explaining the controller according to Embodiment 1. [Figure 3] It is a diagram for explaining the acquisition of the imaging posture angle according to Embodiment 1. [Figure 4] It is a flowchart of the estimation process according to Embodiment 1. [Figure 5] It is a configuration diagram of a controller system according to Embodiment 2. [Figure 6] It is a diagram for explaining the reliability of the imaging posture angle according to Embodiment 2. [Figure 7] It is a flowchart of the estimation process according to Embodiment 2. [Figure 8] It is a configuration diagram of a controller system according to Embodiment 3. [Figure 9] It is a flowchart of the estimation process according to Embodiment 3. [Figure 10] It is a diagram for explaining the CG drawing according to Embodiment 3. [Figure 11] It is a configuration diagram of a controller system according to Embodiment 4 [Figure 12] It is a flowchart of the estimation process according to Embodiment 4. [Figure 13] It is a diagram for explaining the generation of the posture angle relative data according to Embodiment 4. [Figure 14] It is a diagram for explaining a plurality of imaging posture angles according to Embodiment 4. [[ID=--]]

Modes for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings.

[0012] <Embodiment 1> FIG. 1 is a configuration diagram showing a configuration example of a controller system (information processing system) according to Embodiment 1. The controller system includes a controller 110 and an HMD 120.

[0013] The controller 110 is a hand controller (control device) for controlling the display of the HMD 120. The controller 110 is held by the user's finger. For example, the controller 110 is a ring-shaped hand controller as shown in FIG. 2A and can be worn on the user's finger. In Embodiment 1, the shape of the controller 110 will be described as a ring shape, but it may be a glove shape or the like. The controller 110 includes a communication unit 111, an inertial sensor 112, and a bus 113. Also, the communication unit 111 and the inertial sensor 112 are connected to each other via the bus 113. Note that it is not necessary to mount a plurality of light-emitting diodes (large sensors, etc.) as shown in Patent Document 1 on the controller 120. Therefore, the controller 120 can be miniaturized.

[0014] The communication unit 111 transmits the inertial information (information on angular velocity and acceleration) acquired by the inertial sensor 112 to the HMD 120 by wireless communication.

[0015] The inertial sensor 112 is an inertial measurement unit (IMU). The inertial sensor 112 acquires information such as angular velocity and acceleration as inertial information. The inertial sensor 112 includes an angular velocity sensor and an acceleration sensor. In the present embodiment, the inertial sensor 112 may further include a geomagnetic sensor and a plurality of angular velocity sensors.

[0016] As shown in Figure 2B, the HMD120 is a glasses-type information processing device worn by the user on their head. The HMD120 includes a small display unit. The HMD120 has a communication unit 121, an imaging unit 122, a camera posture detection unit 123, a sensor posture acquisition unit 124, a wrist joint detection unit 125, a hand posture acquisition unit 126, an estimation unit 127, and a bus 140. All components of the HMD120 except the bus 140 are connected to each other via the bus 140. Note that the HMD120 may have only the communication unit 121 and the imaging unit 122, and the information processing device (control device) such as a computer that controls the HMD120 may have the other components (hand posture acquisition unit 126 and estimation unit 127).

[0017] The communication unit 121 acquires inertial information from the controller 110 via wireless communication.

[0018] The imaging unit 122 images the space in front of the HMD 120. The imaging unit 122 is, for example, a stereo camera. The imaging unit 122 may also be an infrared distance camera.

[0019] The camera attitude detection unit 123 detects the position and attitude of the imaging unit 122 (HMD120). The position and attitude detected by the camera attitude detection unit 123 are the position and attitude of the imaging unit 122 in a world coordinate system representing real space. Known techniques can be used to detect the position and attitude. Known techniques include, for example, calculations using Visual SLAM (Simultaneous Localization and Mapping). Visual SLAM is a technique that can simultaneously estimate the self-position of the imaging unit 122 and create environmental map coordinates in an unknown environment.

[0020] The sensor attitude acquisition unit 124 acquires inertial information obtained by the inertial sensor 112 via the communication unit 121. Then, the sensor attitude acquisition unit 124 calculates the attitude angle of the controller 110 (hereinafter referred to as the "controller attitude angle") based on the angular velocity and acceleration indicated by the inertial information. Known techniques, such as utilizing an extended Kalman filter, can be used for calculating the attitude angle of the controller 110.

[0021] Here, the attitude angle is an indicator of an object's posture, determined by how much the object is tilted in the front-to-back, up-and-down, and left-to-right directions relative to the "reference state." The attitude angle is, for example, a combination of the yaw angle, pitch angle, and roll angle. For example, the roll angle is the rotation angle in the direction along the circumference of the controller 110. The "reference state" is, for example, the state of the controller 110 and the user's hand when the hand with the controller 110 attached is previously imaged by the imaging unit 122. Therefore, when the user has the controller 110 attached to their fingers, if the shape of the user's hand is substantially fixed, the actual attitude angle of the controller 110 and the actual attitude angle of each finger will substantially coincide, even if the posture of the user's hand changes.

[0022] The wrist joint detection unit 125 detects the position of the user's wrist joints (joint points of the fingers) from the image (captured image) captured by the imaging unit 122 of the user's hand. Here, the position of the user's wrist joints in the captured image is represented by the coordinate system of the captured image (hereinafter referred to as the "camera coordinate system"). Known hand tracking techniques can be used to detect the position of the wrist joints. Known hand tracking techniques include, for example, machine learning-based detection of wrist joints. Alternatively, known hand tracking techniques can also be used, which calculate the distance from the imaging unit 122 to the wrist joints by disparity estimation using stereo matching and triangulation.

[0023] The hand posture acquisition unit 126 determines the position of the wrist joint point in the camera coordinate system detected by the wrist joint detection unit 125. The image is converted to a position in a world coordinate system representing real space based on the position and orientation of the imaging unit 122. The hand posture acquisition unit 126 then acquires the user's hand posture angle (hereinafter referred to as the "imaging posture angle") estimated based on the position of the wrist joints. In Embodiment 1, the user's hand posture angle is the thumb posture angle. For example, as shown in Figure 3, the hand posture acquisition unit 126 acquires the imaging posture angle based on the direction 1110 (slope) of the straight line connecting the three thumb joint points 1101, 1102, and 1103. In other words, the hand posture acquisition unit 126 acquires the hand posture angle assuming that the hand is facing in the direction 1110. Note that the hand posture angle is not limited to the thumb posture angle, but may be the posture angle of any finger, such as the index finger.

[0024] The estimation unit 127 estimates (acquires) the user's hand posture angle based on the controller posture angle and the imaging posture angle. For example, if the difference between the controller posture angle and the imaging posture angle is less than a threshold, the estimation unit 127 acquires the average of the two posture angles as the user's hand posture angle. If the difference between the controller posture angle and the imaging posture angle is greater than a predetermined value, the estimation unit 127 selects the value of the controller posture angle or the imaging posture angle that has changed less from the previous value, and acquires the selected value as the user's hand posture angle.

[0025] (Regarding estimation processing) Referring to the flowchart in Figure 4, the process for estimating the user's hand posture (posture angle) in Embodiment 1 (estimation process) will be explained. Each process in the flowchart of Figure 4 is realized by the HMD120's processor operating as each component, such as the communication unit 121 and the estimation unit 127. At this time, the processor operates as each component by executing a program stored in the HMD120's storage medium.

[0026] In step S401, the communication unit 121 acquires inertial information from the inertial sensor 112 of the controller 110.

[0027] In step S402, the sensor attitude acquisition unit 124 acquires the controller attitude angle (attitude angle of the controller 110) based on the inertial information (acceleration and angular velocity acquired by the inertial sensor 112) acquired in step S401.

[0028] In step S403, the imaging unit 122 captures images of the space in front of the imaging unit 122 using a stereo camera.

[0029] In step S404, the camera attitude detection unit 123 detects the position and attitude of the imaging unit 122 based on the image (captured image) of the space in front of the imaging unit 122 captured in step S403.

[0030] In step S405, the wrist joint detection unit 125 detects the user's wrist joint point from the captured image.

[0031] In step S406, the wrist joint detection unit 125 determines whether a specific wrist joint point has been detected in the captured image based on the wrist joint point detection result in step S405. Here, a specific wrist joint point is a wrist joint point necessary for obtaining the imaging posture angle, and in Embodiment 1, it is the three joint points of the thumb (the thumb joint points 1101, 1102, and 1103 as shown in Figure 3). If it is determined that a specific wrist joint point has been detected, the process proceeds to step S407. If it is determined that a specific wrist joint point has not been detected, the process proceeds to step S409. Alternatively, instead of a specific wrist joint point, it may be determined whether a specific part that can be used to obtain the hand posture angle (for example, the fingernail or bone of a specific finger) has been detected.

[0032] In step S407, the hand posture acquisition unit 126 converts the position of a specific wrist joint point in the camera coordinate system in the captured image to a position in the world coordinate system representing real space, based on the position and orientation of the imaging unit 122. Then, the hand posture acquisition unit 126 acquires the imaging posture angle (the posture angle of the user's hand) based on the position of the specific wrist joint point in the world coordinate system. For example, the hand posture acquisition unit 126 acquires the imaging posture angle according to the direction indicated by the line connecting the three joint points of the thumb.

[0033] In step S408, the estimation unit 127 estimates (acquires) the user's hand posture angle (position) based on the controller posture angle and the imaging posture angle. For example, the estimation unit 127 acquires the average value of the controller posture angle and the imaging posture angle as the user's hand posture angle (position).

[0034] In step S409, the estimation unit 127 determines whether or not it has received a termination command from the user. If it is determined that no termination command has been received, the process proceeds to step S401. If it is determined that a termination command has been received, the process in this flowchart ends.

[0035] If it is determined in step S406 that a specific wrist joint point has not been detected (i.e., NO in step S406), the estimation unit 127 may acquire the controller posture angle as the hand posture angle.

[0036] In Embodiment 1, the controller system estimates the user's hand posture by using both the posture angle based on the inertial information of the controller's inertial sensor (controller posture angle) and the hand posture angle based on the imaging of the HMD's imaging unit (imaging posture angle). Therefore, the user's hand posture can be estimated (detected) with high accuracy. Furthermore, since the controller does not need to be equipped with sensors other than the inertial sensor, the controller can be miniaturized.

[0037] <Embodiment 2> Embodiment 2 describes a controller system (information processing system) that varies the method of estimating the hand posture angle depending on the reliability of the imaging posture angle (detection of the wrist joint in the captured image).

[0038] Figure 5 is a configuration diagram of the controller system according to Embodiment 2. Note that in Figure 5, the explanation of the configurations that are the same as those in Embodiment 1 is omitted. The HMD220 has a reliability determination unit 228 in addition to the configuration of the HMD120 according to Embodiment 1.

[0039] The reliability determination unit 228 determines the reliability of the imaging posture angle (i.e., the reliability of the detection of a specific wrist joint by the wrist joint detection unit 125) based on the imaging posture angle acquired by the hand posture acquisition unit 126. Here, the larger the angle between the user's hand orientation and the ground (horizontal plane), the easier it is for the imaging unit 122 to see the wrist joint. And, since the wrist joint is detected from the captured image, the more easily a specific wrist joint is visible, the higher the reliability of the imaging posture angle. For this reason, the reliability determination unit 228 determines that the larger the angle (inferior angle) between the user's hand orientation and the ground (horizontal plane) indicated by the imaging posture angle, as shown in Figure 6A, the higher the reliability. For example, the reliability may be the angle between the user's hand orientation and the ground (horizontal plane) indicated by the imaging posture angle itself. Also, for example, since the posture angle of the thumb can be estimated from the state of the other fingers, the reliability may be a higher value the more wrist joint points are captured in the image.

[0040] The reliability determination unit 228 may determine the reliability of the imaging posture angle, for example, according to the size of the area of ​​a specific finger that is captured in the captured image. In this case, for example, in the example of the captured image in Figure 6B, the reliability is low because part of the thumb is hidden, and in the example of the captured image in Figure 6C, The image is highly reliable because the thumb is visible. In the example image in Figure 6D, an even larger area of ​​the thumb is visible, making it the most reliable image.

[0041] Referring to the flowchart in Figure 7, the estimation process for estimating the user's hand posture in Embodiment 2 will be explained. Note that steps that are the same as in Embodiment 1 will be omitted from this flowchart.

[0042] In step S710, the reliability determination unit 228 determines the reliability of the imaging posture angle (the reliability of detecting a specific wrist joint) based on the value of the imaging posture angle.

[0043] In step S711, the reliability determination unit 228 determines whether the reliability of the imaging attitude angle is higher than a specific threshold (i.e., whether the angle between the user's hand orientation indicated by the imaging attitude angle and the ground is greater than a predetermined angle). If it is determined that the reliability of the imaging attitude angle is higher than the specific threshold, the process proceeds to step S408. If it is determined that the reliability of the imaging attitude angle is below a specific threshold (i.e., whether the angle between the user's hand orientation indicated by the imaging attitude angle and the ground is less than or equal to a predetermined angle), the process proceeds to step S712.

[0044] In step S712, the estimation unit 127 estimates the hand's attitude angle based on the controller attitude angle acquired by the sensor attitude acquisition unit 124, rather than on the imaging attitude angle, because the reliability of the imaging attitude angle is low. In other words, the estimation unit 127 acquires the controller attitude angle as the hand's attitude angle.

[0045] As described above, according to Embodiment 2, the controller system determines the reliability of the imaging pose angle based on the images taken by the HMD's imaging unit, and if the reliability is low, it does not use the imaging pose angle to estimate the hand pose. This reduces the possibility of using low-accuracy information, and thus enables more accurate estimation (detection) of the user's hand pose.

[0046] <Embodiment 3> Embodiment 3 describes a controller system (information processing system) that further estimates the position of the hand based on the detection of the wrist joint from the captured image.

[0047] Figure 8 is a configuration diagram showing an example of the configuration of the controller system according to Embodiment 3. Note that in Figure 8, the explanation of the configuration which is the same as that of Embodiment 1 is omitted. The HMD320 according to Embodiment 3 has a position estimation unit 329 in addition to the configuration of the HMD120 according to Embodiment 1.

[0048] The position estimation unit 329 converts the position of the wrist joint point in the camera coordinate system detected by the wrist joint detection unit 125 into a world coordinate system representing real space to estimate the position of the hand. In Embodiment 3, the position of the hand is assumed to be the position of the tip 1103 of the thumb as shown in Figure 3. The position of the hand estimated by the position estimation unit 329 may also be the position of the joint point of other fingers. Furthermore, this position of the hand may be the centroid of all the wrist joint points detected by the wrist joint detection unit 125, or it may be the centroid of some of the detected wrist joint points.

[0049] Referring to the flowchart in Figure 9, the estimation process for estimating the user's hand posture (posture angle) in Embodiment 3 will be explained. Note that the steps in the flowchart in Figure 9 that are the same as in Embodiment 1 will be omitted from the explanation.

[0050] In step S910, the position estimation unit 329 estimates the position of the hand based on the position of the user's wrist joint detected in step S405.

[0051] Next, using Figure 10, we will demonstrate the CG rendering that utilizes the hand position and posture in Embodiment 3. Let's explain an example. Composite image 1000 is an image representing a virtual space (virtual space) displayed on the HMD 110's screen. In composite image 1000, a ray 1002 (CG ray) is drawn as a UI (user interface; display item) for selecting a CG object 1001 located at a distance. The direction in which the ray 1002 extends is determined by the control unit of the HMD 100 based on the hand's pose angle estimated by the estimation unit 127. The starting point of the ray 1002 is determined by the control unit based on the hand's position estimated by the position estimation unit 329. Alternatively, a pointer displayed at a position based on the hand's pose and position may be used instead of a ray.

[0052] The estimated hand position and orientation may be used for purposes other than controlling the display of the ray. For example, the control unit of the HMD320 may determine the object pointed to by the user's hand based on the estimated hand position and orientation, and adjust the focus of the imaging unit 122 to that object. Alternatively, the control unit of the HMD320 may determine the object pointed to by the user's hand based on the estimated hand position and orientation, and display the distance from the HMD320 to that object on the display.

[0053] As described above, according to Embodiment 3, the hand posture can be estimated based on the controller posture angle and the imaging posture angle, and the hand position can be estimated further, thereby enabling accurate detection of the user's hand position and posture.

[0054] <Embodiment 4> Embodiment 4 describes a controller system (information processing system) that further includes a calibration unit for calibrating an inertial sensor. The output characteristics of an inertial sensor change due to temperature, atmospheric pressure, external vibration, or shock. Therefore, the bias value (inertial information) output by the inertial sensor may change depending on environmental information (external environment). In Embodiment 4, the calibration unit is used to maintain the accuracy of the inertial sensor.

[0055] Figure 11 is a configuration diagram of the controller system in Embodiment 4. Note that in Figure 11, explanations of configurations that are the same as those in Embodiments 1 and 2 are omitted. The controller 410 according to Embodiment 4 has a calibration unit 414 in addition to the configuration of the controller 110 according to Embodiment 1. The HMD 420 according to Embodiment 4 has a calibration information generation unit 430 in addition to the configuration of the HMD 220 according to Embodiment 2.

[0056] The calibration unit 414 acquires calibration information generated by the calibration information generation unit 430 via the communication unit 121 and the communication unit 111. Based on the calibration information, the calibration unit 414 calibrates the inertial sensor 112. After calibration, the inertial sensor 112 outputs inertial information that is corrected according to the calibration information compared to the inertial information acquired by the inertial sensor 112 before calibration.

[0057] The calibration information generation unit 430 stores the relative relationship (attitude angle relative data) between the controller attitude angle and the imaging attitude angle when the controller 410 is stationary (when the controller 410 is stationary). In this embodiment, the attitude angle relative data is described as the value obtained by subtracting the imaging attitude angle from the controller attitude angle when the controller 410 is stationary (the difference between the two attitude angles), but it can be any information that shows the relative relationship between the two attitude angles.

[0058] Furthermore, the calibration information generation unit 430 estimates the controller attitude angle based on attitude angle relative data from the imaging attitude angle acquired during the operation of the HMD 420 and the controller 410. Here, the estimated controller attitude angle is called the "estimated attitude angle". For example, the calibration information generation unit 430 estimates the estimated attitude angle by adding the imaging attitude angle acquired during the operation of the HMD 420 and the attitude angle relative data. Then, the calibration information generation unit 430 adds the estimated attitude angle and the controller attitude angle acquired by the sensor attitude acquisition unit 124 (current controller The difference between the estimated attitude angle and the current controller attitude angle is acquired as calibration information for the inertial sensor 112. The calibration information is information for calibrating the inertial sensor 112, and can be any information as long as it corresponds to the difference between the estimated attitude angle and the current controller attitude angle.

[0059] Referring to the flowchart in Figure 12, the estimation process for estimating the user's hand posture (posture angle) in Embodiment 4 will be explained. Note that the steps in the flowchart in Figure 12 that are the same as those in Embodiments 1 and 2 will be omitted from the explanation.

[0060] In step S1215, the calibration information generation unit 430 generates (acquires) the relative relationship between the controller attitude angle and the imaging attitude angle (attitude angle relative data) while the controller 410, HMD 420, and hand are all stationary.

[0061] Referring to Figure 13, the specific process for generating relative attitude angle data will be explained. First, the user stops their hand in front of the HMD420, and the imaging unit 122 mounted on the HMD420 captures an image of the hand. Then, the hand attitude acquisition unit 126 acquires the imaging attitude angle based on the direction 1304 of the line connecting the joint points 1301, 1302, and 1303 of the thumb in the captured image. At the same time, the sensor attitude acquisition unit 124 acquires the controller attitude angle based on the inertial information output from the inertial sensor 112 of the controller 410. Finally, the calibration information generation unit 430 generates relative attitude angle data showing the relationship between the imaging attitude angle and the controller attitude angle.

[0062] In step S1216, the calibration information generation unit 430 determines whether a predetermined time has elapsed since the time the calibration information was last generated. In order to periodically generate (update) the calibration information, if a predetermined time has elapsed since the time the calibration information was last generated, the process proceeds to step S1217. If a predetermined time has not elapsed since that time, the process proceeds to step S409.

[0063] In step S1217, the calibration information generation unit 430 obtains an estimated attitude angle (the angle estimated for the controller attitude angle) based on the attitude angle relative data generated in step S1215 and the imaging attitude angle acquired in step S407. The calibration information generation unit 430 generates calibration information based on the difference between the controller attitude angle acquired in step S402 and the estimated attitude angle. The calibration information generation unit 430 then transmits the newly generated calibration information to the calibration unit 414 via the communication unit 121 and the communication unit 111. The calibration unit 414, having received the calibration information, calibrates the calibration sensor based on the calibration information.

[0064] Furthermore, since the processing in step S1217 is executed only if the confidence level of the imaging posture angle (detection of a specific wrist joint) is determined to be higher than the threshold in step S711, calibration information can be generated using the imaging posture angle with high confidence.

[0065] As described above, according to Embodiment 4, the inertial sensor of the controller can be calibrated periodically based on a highly reliable imaging attitude angle, thereby improving the output accuracy of the inertial sensor. This allows for even more accurate detection of the user's hand posture.

[0066] (Variation 1) As a modification of Embodiment 4, the calibration information generation unit 430 may further include a table generation unit that generates a calibration table for storing posture angle relative data for each state of the wrist joint. In this case, the controller system can select (switch) posture angle relative data according to the hand posture and calibrate the inertial sensor 112. Here, the calibration table is stored, for example, in a storage unit of the table generation unit.

[0067] For example, in step S1215 of Figure 12, the calibration information generation unit 430 generates a set of multiple attitude angles (imaging attitude angles) of the controller 410 when it is stationary (for example, Figures 14A to 14A). The calibration information generation unit 430 generates relative attitude angle data for each of the multiple attitude angles corresponding to the hands shown in C. The calibration information generation unit 430 then stores the multiple relative attitude angle data in a calibration table.

[0068] In step S1217, the calibration information generation unit 430 obtains attitude angle relative data from the calibration table (storage unit) that corresponds to the attitude angle closest to the imaging attitude angle (current imaging attitude angle) acquired by the hand attitude acquisition unit 126. Then, the calibration information generation unit 430 calculates an estimated attitude angle based on the acquired attitude angle relative data and the imaging attitude angle. The calibration information generation unit 430 generates calibration information based on the difference between the estimated attitude angle and the controller attitude angle. Subsequently, the calibration information generation unit 430 transmits the calibration information to the calibration unit 414. The calibration unit 414 calibrates the inertial sensor 112 based on the calibration information.

[0069] In the modified example 1 described above, the controller system has multiple relative attitude angle data corresponding to multiple hand attitude angles. The controller system then selects the appropriate relative attitude angle data according to the acquired hand attitude angles and calibrates the inertial sensor. This further stabilizes the output accuracy of the inertial sensor and enables more accurate detection of the user's hand attitude.

[0070] Furthermore, in the above, the statement "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" can be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, the statement "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" can be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, the expression "greater than or equal to A" can be replaced with "A or greater than (higher; longer; more)" or rephrased as "greater than (higher; longer; more)." On the other hand, the expression "less than or equal to A" can be replaced with "A or less than (lower; shorter; fewer)" or rephrased as "less than (lower; shorter; fewer)." Furthermore, "larger than A (higher; longer; more)" can be rephrased as "greater than or equal to A," and "smaller than A (lower; shorter; fewer)" can be rephrased as "less than or equal to A."

[0071] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0072] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0073] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0074] The above-disclosed embodiments include the following configurations, methods, systems, and programs. [Configuration 1] An information processing device that estimates the hand posture of a user holding a controller in their hand, The controller has an acquisition means for acquiring inertial information from an inertial sensor, A determination means for determining whether a specific part of the user's hand is detected in the image captured by the imaging means, If the specific part is detected in the captured image, estimation means for estimating the user's hand posture based on the captured image and the inertial information, An information processing device characterized by having the following features. [Configuration 2] The captured image has a detection means for detecting the part of the user's hand, If the reliability of the detection of the specific part by the detection means is lower than a threshold, the estimation means estimates the user's hand posture based on the inertial information, without relying on the captured image. The information processing device according to configuration 1, characterized by the above. [Configuration 3] The case in which the confidence level is lower than the threshold is when the angle between the orientation of the user's hand and the horizontal plane is smaller than a predetermined threshold. The information processing apparatus according to configuration 2, characterized in that... [Structure 4] When the estimation means detects the specific part in the captured image, it estimates the hand's posture based on a first posture, which is the posture of the hand obtained based on the captured image, and a second posture, which is the posture of the controller obtained based on the inertial information. The first attitude described above is an attitude acquired without relying on the inertial information, The second posture described above is a posture acquired without relying on the captured image. An information processing device according to any one of configurations 1 to 3. [Composition 5] The aforementioned inertial sensor is calibrated based on calibration information. The information processing device further includes a generation means for generating the calibration information based on the captured image and the inertial information. The information processing apparatus according to configuration 4, characterized by the features described above. [Composition 6] The generation means 1) acquires relative data showing the relationship between the first and second attitudes of the controller when it is stationary, and 2) generates the calibration information based on the difference between the attitude of the controller estimated based on the first attitude at the present time and the relative data, and the second attitude at the present time. The information processing apparatus according to configuration 5, characterized by the features described herein. [Composition 7] It has storage means for storing the relative data for each of multiple hand poses, The generation means obtains relative data of the posture closest to the first posture at the current time from the storage means. The information processing device according to configuration 6, characterized by the features described therein. [Structure 8] A position estimation means for estimating the position of the user's hand based on the captured image, A control means that performs processing based on the position of the user's hand estimated by the position estimation means and the posture of the user's hand estimated by the estimation means, An information processing device according to any one of configurations 1 to 7, further comprising [Composition 9] The control means determines the position of the user's hand estimated by the position estimation means and the estimation means Control the display means to display items based on the estimated hand posture of the user. The information processing apparatus according to configuration 8, characterized by the above. [Configuration 10] The information processing device according to configuration 9, characterized in that the display item is a ray of light extending from a position based on the position of the user's hand estimated by the position estimation means, in a direction based on the posture of the user's hand estimated by the estimation means. [Composition 11] The aforementioned specific area is multiple joints of the user's specific finger, The user's hand posture is the posture of the specific fingers, If the multiple joints are detected in the captured image, the posture of the specific finger is estimated based on the orientation of the specific finger determined by the multiple joints in the captured image and the inertial information. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above. [Composition 12] The controller is a ring-shaped controller that can be worn on the user's finger. An information processing device according to any one of configurations 1 to 11, characterized by the features described herein. [Composition 13] The aforementioned information processing device is a head-mounted display, The head-mounted display has the imaging means and the display means, An information processing device according to any one of configurations 1 to 12, characterized by the above. [system] A controller having the inertial sensor that acquires the inertial information, An information processing device described in any one of items 1 to 13, An information processing system characterized by having the following features. [method] An information processing method for estimating the hand posture of a user holding a controller in their hand, The acquisition step involves acquiring inertial information from an inertial sensor provided by the controller, A determination step of determining whether a specific part of the user's hand is detected in the image captured by the imaging means, If the specific part is detected in the captured image, the estimation step involves estimating the user's hand posture based on the captured image and the inertial information. An information processing method characterized by having the following features. [program] A program for causing a computer to function as one of the information processing devices described in any one of the configurations 1 to 13. [Explanation of Symbols]

[0075] 110: Controller, 120: HMD 112: Inertial sensor, 122: Imaging unit, 124: Sensor attitude acquisition unit, 125: Wrist joint detection unit, 126: Hand posture acquisition unit, 127: Estimation part

Claims

1. An information processing device that estimates a hand posture of a user holding a controller, an acquisition means for acquiring inertial information from an inertial sensor included in the controller; a determination means for determining whether or not a specific part of the user's hand is detected in a captured image acquired by the imaging means; an estimation means for estimating a posture of the user's hand based on the inertial information when the acquisition means acquires the inertial information, the estimation means, when it is determined by the determination means that the specific part has been detected in the captured image, estimates a posture of the user's hand based on the captured image and the inertial information, and, when it is not determined by the determination means that the specific part has been detected in the captured image, estimates a posture of the user's hand based on the inertial information.

23. An information processing apparatus comprising:

2. a detection means for detecting a part of the user's hand in the captured image; When a reliability of detection of the specific part by the detection means is lower than a threshold, the estimation means estimates the posture of the user's hand based on the inertial information, not based on the captured image.

2. The information processing apparatus according to claim 1,

3. The case where the reliability is lower than the threshold value is a case where the angle between the orientation of the user's hand and a horizontal plane is smaller than a predetermined threshold value.

3. The information processing apparatus according to claim 2.

4. the estimation means, when the specific part is detected in the captured image, estimates a posture of the hand based on a first posture acquired based on the captured image and a second posture acquired based on the inertial information, the first attitude is an attitude obtained without being based on the inertial information, The second orientation is an orientation acquired without being based on the captured image.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. the inertial sensor is calibrated based on calibration information; The information processing device further includes a generating unit configured to generate the calibration information based on the captured image and the inertial information.

5. The information processing apparatus according to claim 4.

6. the generation means 1) acquires relative data indicating a relationship between the first orientation and the second orientation when the controller is stationary, and 2) generates the calibration information based on a difference between an orientation of the controller estimated based on the first orientation at a current time and the relative data, and the second orientation at the current time.

6. The information processing apparatus according to claim 5,

7. a storage means for storing said relative data for each of a plurality of hand postures; the generating means acquires from the storing means relative data of a posture closest to the first posture at the current time; 7. The information processing apparatus according to claim 6,

8. a position estimation means for estimating a position of the user's hand based on the captured image; a control means for executing a process based on the position of the user's hand estimated by the position estimation means and the posture of the user's hand estimated by the estimation means; 4. The information processing apparatus according to claim 1, further comprising:

9. the control means controls the display means to display a display item based on the position of the user's hand estimated by the position estimation means and the posture of the user's hand estimated by the estimation means; 9. The information processing apparatus according to claim 8,

10. 10. The information processing device according to claim 9, wherein the display item is a ray extending from a position based on the position of the user's hand estimated by the position estimation means, in a direction based on the posture of the user's hand estimated by the estimation means.

11. When the determination means determines that the specific part has been detected in the captured image, the estimation means estimates each of a posture of the user's hand based on the captured image and a posture of the user's hand based on the inertial information.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

12. The estimation means determines, when the determination means determines that the specific part has been detected in the captured image and a difference between the user's hand posture based on the captured image and the user's hand posture based on the inertial information is less than a predetermined value, an average of the user's hand posture based on the captured image and the user's hand posture based on the inertial information as the user's hand posture.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

13. The estimation means, when it is determined by the determination means that the specific part has been detected in the captured image, and the difference between the posture of the user's hand based on the captured image and the posture of the user's hand based on the inertial information is greater than a predetermined value, determines the posture of the user's hand based on the captured image or the posture of the user's hand based on the inertial information, whichever has changed less from the previous time, as the posture of the user's hand; 4. The information processing device according to claim 1, wherein the information processing device is a computer.

14. the specific part is a plurality of joints of a specific finger of the user, the user's hand posture is a posture of the particular finger, When the plurality of joints are detected in the captured image, the estimation means estimates a posture of the specific finger based on an orientation of the specific finger determined by the plurality of joints in the captured image and the inertial information.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

15. The controller is a ring-type controller that can be worn on a user's finger.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

16. the information processing device is a head mounted display, The head mounted display has the imaging means and a display means.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

17. a controller having the inertial sensor for acquiring the inertial information; An information processing device according to any one of claims 1 to 3; An information processing system comprising:

18. An information processing method for estimating a hand posture of a user holding a controller, comprising: an acquisition step of acquiring inertial information from an inertial sensor included in the controller; a determination step of determining whether or not a specific part of the user's hand is detected in a captured image acquired by the imaging means; an estimation step of estimating a posture of the user's hand based on the inertial information when the inertial information is acquired in the acquisition step, In the estimation step, if it is determined in the determination step that the specific part has been detected in the captured image, a posture of the user's hand is estimated based on the captured image and the inertial information, and if it is not determined in the determination step that the specific part has been detected in the captured image, a posture of the user's hand is estimated based on the inertial information.

23. An information processing method comprising:

19. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 3.

Images