Information processing apparatus, information processing system, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and a program that accurately acquire the position of a user's hand based on information from a controller even if the controller held by the user's hand is of a small size.SOLUTION: In an information processing system 1, an image processing apparatus 110 being an information processing apparatus functions as: acquisition means that acquires inertial information from an inertial sensor included in a motion detection unit of a ring-shape controller 120 in a finger ring shape that can be worn on a user's finger; detection means that detects the controller from a picked-up image acquired by an imaging unit of a HMD 100 through imaging; and control means that, when the controller is not detected from the picked-up image, acquires the position of the controller based on the position of the controller acquired in the past and the inertial information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] In cross reality (XR) systems, which allow users to experience virtual reality, hand controllers have been used to convert hand movements into actions in the virtual space when controlling the display on a head-mounted display (HMD). An HMD is a glasses-type device equipped with a small display that the user wears on their head.

[0003] Patent Document 1 proposes a hand controller that can detect the position and posture of the hand by emitting a plurality of infrared light beams (IR light beams) from the hand controller and receiving the infrared light beams with a camera mounted on an HMD. Patent Document 2 also proposes a device that reflects the position and posture of a user in a virtual space by comparing the user's body parts captured in an image captured by a camera mounted on an HMD with a skeletal model stored in memory. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-519992 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-514652 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 requires that multiple light-emitting diodes be mounted on the hand controller to detect the position and orientation of the hand controller, making it difficult to miniaturize the hand controller. Also, the technology disclosed in Patent Document 2 has the problem that detection accuracy decreases depending on the orientation of the user's hand.

[0006] Therefore, an object of the present invention is to provide a technique that makes it possible to accurately obtain the position of a user's hand based on information from the controller, even if the controller held in the user's hand is small. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. an acquisition means for acquiring inertial information from an inertial sensor included in the controller; a detection means for detecting the controller from an image captured by the imaging means; a control means for acquiring a position of the controller based on a previously acquired position of the controller and the inertia information when the controller is not detected from the captured image; The information processing device is characterized by having:

[0008] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. an acquisition step of acquiring inertial information from an inertial sensor included in the controller; a detection step of detecting the controller from a captured image acquired by imaging means; If the controller is not detected from the captured image, the controller a control step of acquiring a position of the controller based on a position of the controller and the inertial information; The information processing method is characterized by having the following. [Effects of the Invention]

[0009] According to the present invention, even if the controller held by the user's hand is small, it is possible to accurately obtain the position of the user's hand based on information from the controller. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Figure 2] 1 is a diagram illustrating the internal configuration of an HMD according to a first embodiment. [Figure 3] 10 is a flowchart of a ray display process according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing one display example of an HMD according to Modification 1. [Figure 8] FIG. 10 is a diagram showing one display example of an HMD according to Modification 2. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0012] <Embodiment 1> An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, an image processing device 110, and a controller 120.

[0013] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0014] The image processing device 110 is a control device (information processing device; electronic device) that controls the HMD 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a PC (personal computer). The image processing device 110 is connected to the HMD 100 wirelessly or via a wired connection. The image processing device 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that each component of the image processing device 110 may be included in the HMD 100.

[0015] The controller 120 is a device for performing various controls of the HMD 100. When the image processing device 110 is in a specific control mode, if a user operates the controller 120, the HMD 100 is controlled in accordance with the user's operation. The controller 120 is, for example, ring-shaped so that it can be worn on the user's finger, as shown in FIG. 1. If the controller 120 can be worn on the user's finger, the user can move their hand freely while holding the controller 120. The controller 120 also has a button with a built-in optical trackpad (hereinafter referred to as "OTP") that can detect the amount of planar movement. The controller 120 communicates wirelessly with the image processing device 110 via Bluetooth.

[0016] For example, the user may press and hold the OTP button to display a menu including a pointer on the HMD 100. After that, the user may place their finger on the OTP and rub it in any direction to move the pointer to a desired item. Then, the user may press the OTP button. By pressing the controller 120 in, a confirmation action can be performed to confirm the selection of the item. Note that although the controller 120 has been described as having a ring-like shape, the shape is not limited to this. For example, the controller 120 may have a shape that can be worn on the hand, such as a glove-like shape. In this way, the controller 120 is preferably in a form that can be held in the user's hand or worn on the hand, so that it is easy for the user to use.

[0017] (About the internal structure of the HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, and an orientation sensor unit 204.

[0018] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the image processing device 110, the HMD control unit 201 displays the composite image on the image display unit 203. Therefore, by wearing the HMD 100, the user can view the composite image displayed on the image display unit 203. The user can experience various mixed realities, such as CG blended into real space.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are arranged near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images (captured images) captured by the two cameras of a subject (the range in front of the user) are output to the image processing device 110. Furthermore, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances using a stereo camera.

[0020] The image display unit 203 displays the composite image. The image display unit 203 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, an organic EL panel is placed in front of each of the user's eyes.

[0021] It is also possible to use a device using a semi-transparent half mirror for the image display unit 203. In this case, for example, the image display unit 203 may use a technology generally called AR to display an image in such a way that CG appears to be directly superimposed on the real space visible through the half mirror. Also, the image display unit 203 may use a technology generally called VR to display an image of a completely virtual space without using a captured image.

[0022] The orientation sensor unit 204 detects the orientation and position of the HMD 100. The orientation sensor unit 204 has an inertial measurement unit (IMU). The orientation sensor unit 204 outputs information on the orientation of the HMD 100 to the image processing device 110 as orientation information.

[0023] (Internal configuration of image processing device) The internal configuration of the image processing device 110 will be described with reference to Fig. 2. The image processing device 110 includes a control unit 211, a content DB 212, and a communication unit 213.

[0024] The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. The control unit 211 then composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0025] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger the closer the distance between the specific object and the imaging unit 202. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a composite image in which a CG object that is not located in real space appears to be located in real space.

[0026] The content DB 212 is a storage unit that stores information such as CG, etc. The control unit 211 can switch the CG read from the content DB 212 (that is, the CG used to generate a composite image).

[0027] (Internal structure of the controller) The internal configuration of controller 120 will be described with reference to Fig. 2. Controller 120 has a controller control unit 221, an operation unit 222, a communication unit 223, and a motion detection unit 224. Here, it is not necessary for controller 120 to be equipped with a plurality of light-emitting diodes (such as large sensors) as shown in Patent Document 1. This allows controller 120 to be miniaturized.

[0028] The controller control unit 221 controls each component of the controller 120 .

[0029] The operation unit 222 includes a button with an OTP built in. Information (operation information) about pressing or sliding a finger on the OTP on the operation unit 222 is transmitted to the image processing device 110 via the communication unit 223.

[0030] For example, the user can move a pointer displayed on the HMD 100 to a predetermined position by sliding a finger on the OTP. Also, the user can instruct the HMD 100 or the image processing device 110 to perform a specific process by pressing a button on the OTP. In this way, the user can control the HMD 100 by combining sliding a finger on the OTP and pressing a button. More specifically, the user can control the HMD 100 by, for example, pressing and holding the OTP button to display a menu on the HMD 100, and then pressing the OTP button after aligning the pointer with a desired location to select that location.

[0031] The operation unit 222 may include any operation member instead of the OTP, as long as the user can perform operations by physical contact. For example, the operation unit 222 may include any of a touchpad, a touch panel, a cross key, a joystick, and a trackpad device instead of the OTP.

[0032] The communication unit 223 performs wireless communication with the image processing device 110 (communication unit 213).

[0033] The motion detection unit 224 has an IMU (inertial measurement unit; inertial sensor) that can detect inertial information (spatial movement amount and angle) of the controller 120. Note that the motion detection unit 224 may have any device that does not hinder the miniaturization of the controller 120 and that can detect inertial information (information such as positional displacement, speed, or acceleration).

[0034] (About ray display processing) The process of displaying a ray, which is a display item of a ray pointing to a position, will be described with reference to the flowchart of FIG. 3. For example, in FIG. 4, a ray 403 is a hand in the composite image. Rays 402 extend from the window 402 toward the multiple windows 401. Ray 403 indicates the position of window C.

[0035] It is assumed that before the processing of this flowchart starts, the operation mode of the image processing device 110 is set to a mode (ray control mode) in which the HMD 100 controls a ray. Then, the processing of this flowchart starts when the controller 120 is included in the imaging range (detection range) of the imaging unit 202 and the imaging unit 202 captures an image of the controller 120. "The controller 120 is included in the imaging range (detection range) of the imaging unit 202" means that the controller 120 can be detected from the captured image (at least one of the two captured images acquired by imaging with the two cameras).

[0036] In step S1001, the control unit 211 detects the controller 120 from a captured image of the controller 120 and acquires the position of the controller 120 by analyzing the captured image. Specifically, the control unit 211 converts the position of the controller 120 in the camera coordinate system in the captured image into a position in the spatial coordinate system in real space, based on the attitude and position of the HMD 100. Note that any known technology can be used as a method for acquiring the position of the controller 120 based on the captured image.

[0037] In step S1002, the control unit 211 acquires the orientation of the controller 120 based on the inertial information (for example, information about the tilt of the controller 120) acquired by the motion detection unit 224. Then, the control unit 211 displays a ray in the composite image on the HMD 100 (image display unit 203) based on the position and orientation of the controller 120 acquired in step S1001.

[0038] In step S1003, control unit 211 determines whether or not controller 120 is included in the imaging range of imaging unit 202 (whether controller 120 can be detected from the captured image). Here, for example, if the process has proceeded from step S1002 to step S1003, control unit 211 determines whether or not controller 120 is included in the imaging range after a predetermined time (a short time of about 1 / 60 seconds) has elapsed since the end of the process of step S1002. If it is determined that controller 120 is included in the imaging range of imaging unit 202, the process proceeds to step S1004. If it is determined that controller 120 is not included in the imaging range of imaging unit 202, the process proceeds to step S1005.

[0039] Note that the case where "the controller 120 is not included in the imaging range of the imaging unit 202" also includes a case where another object exists between the controller 120 and the imaging unit 202 and the controller 120 is not visible from the imaging unit 202. For example, it can also be said that "the controller 120 is not included in the imaging range of the imaging unit 202" when the controller 120 cannot be detected from the captured image because the controller 120 is covered by a hand or hidden behind the user's clothing.

[0040] Because there is a possibility that the controller 120 may move out of the imaging range over time, the control unit 211 determines in step S1003 whether the controller 120 is included in the imaging range. For example, the user moves the controller 120 while controlling a ray. Therefore, even if the controller 120 is present within the imaging range of the imaging unit 202 at a certain time, the controller 120 may later move out of the imaging range. Furthermore, even if the controller 120 has not moved, if the user wearing the HMD 100 changes the direction of their head, the position or orientation of the imaging unit 202 may change, and the controller 120 may end up outside the imaging range of the imaging unit 202.

[0041] In step S1004, the control unit 211 detects the controller 120 from the captured image. Then, the control unit 211 acquires the position of the controller 120 based on the detection result from the captured image and the inertial information (detection result of the motion detection unit 224). The control unit 211 can acquire the position of the controller 120 using only the captured image, as in step S1001. However, if the image capture unit 202 is out of focus or if there is insufficient light for the image capture unit 202 to capture an image, the accuracy of detecting the position of the controller 120 based on the captured image may decrease. In anticipation of such a case, the control unit 211 improves the accuracy of detecting the position of the controller 120 by additionally using the inertial information. For example, the control unit 211 weights and combines the coordinates of the position of the controller 120 acquired based on the captured image, as in step S1001, and the coordinates of the position of the controller 120 acquired based on the inertial information, as in step S1005 (described later). The control unit 211 then acquires the position indicated by the combined coordinates as the position of the controller 120. Here, the control unit 211 may determine the weight for synthesis depending on the state of the captured image (such as the degree of blurring of the captured image or the brightness of the captured image.) This allows the control unit 211 to appropriately control the extent to which the position of the controller 120 acquired based on the inertial information is taken into consideration depending on the detection accuracy of the position of the controller 120 based on the captured image.

[0042] In step S1005, since the controller 120 is not detected from the captured image, the control unit 211 detects the position of the controller 120 based on previously detected positions of the controller 120 and the inertial information (result of the motion detection unit 224). Detailed processing of step S1005 will be described later.

[0043] In step S1006, similarly to step S1002, the control unit 211 causes the image display unit 203 (HMD 100) to move (display) the ray based on the position and orientation of the controller 120. In other words, the control unit 211 updates the display of the ray.

[0044] In step S1007, the control unit 211 determines whether or not to end the ray display mode. The control unit 211 determines whether or not to end the ray display mode, for example, depending on whether or not the user has performed an operation to change the display mode. If it is determined that the ray display mode should be ended, the process proceeds to step S1008. If it is determined that the ray display mode should not be ended, the process returns to step S1003.

[0045] In step S1008, the control unit 211 removes the display of the ray from the composite image (hides the ray from view).

[0046] (Details of step S1005) The process in step S1005 in which control unit 211 detects the position of controller 120 based on inertial information (detection result of motion detection unit 224) will be described in more detail.

[0047] Here, the time when the controller 120 is no longer included in the imaging range (the imaging unit 202 can no longer detect the controller 120) is defined as time t1, and the position of the controller 120 at time t1 is defined as position A1. The time immediately before time t1 is defined as time t0, and the position of the controller 120 at time t0 is defined as position A0. Here, the immediately previous time t0 is the time when the controller 120 was included in the imaging range, for example, the time when the processing of step S1004 was most recently performed.

[0048] In this case, at time t1, the control unit 211 has already acquired the position A0 of the controller 120 at time t0, and therefore acquires position A1 based on the information on position A0. Specifically, the control unit 211 acquires the amount of movement of the controller 120 between time t0 and time t1 from the motion detection unit 224 (controller 120). Then, the control unit 211 calculates the position moved by that amount from coordinate A0 as position A1 of the controller 120. Get it as 1.

[0049] If the controller 120 is not included in the imaging range at the next time t2, the control unit 211 acquires the position of the controller 120 at time t2 based on the amount of movement of the controller 120 between time t1 and time t2. This process is repeated during the period when the controller 120 is not included in the imaging range.

[0050] Furthermore, the motion detection unit 224 may be able to directly detect the amount of movement of the position of the controller 120, or may be able to directly detect only the speed of movement of the position of the controller 120 or the acceleration of the movement. When the motion detection unit 224 acquires the speed of movement, it can calculate the amount of movement by integrating the speed. When the motion detection unit 224 acquires the acceleration of movement, it can calculate the amount of movement by integrating the acceleration twice.

[0051] (HMD display) 4 to 6, examples of the display of the HMD 100 according to the first embodiment will be described. Fig. 4 shows a first display example of the HMD 100. Fig. 4 shows a display example when the imaging unit 202 is able to image (detect) the controller 120 (when the controller 120 appears in the captured image). The controller 120 is worn on the fingers of the user's hand 402.

[0052] The controller 120 and the user's hand 402 are included within the imaging range of the imaging unit 202. Therefore, the controller 120 and the user's hand 402 are displayed in a display area 404 of the image display unit 203 (composite image).

[0053] In FIG. 4, multiple windows 401 are displayed in a composite image (display area 404) as if floating in space. The multiple windows 401 include three windows: window A, window B, and window C. For example, when window A is selected, an internet browser runs and specific search results are displayed. When window B is selected, an application that can create documents runs. When window C is selected, a video or music player application runs, allowing the user to listen to music, etc.

[0054] Additionally, a ray 403 extending from the index finger is displayed as CG in the direction the index finger is pointing. The user can control the direction of the ray by moving the controller 120. Specifically, after the motion detection unit 224 detects the orientation of the controller 120, the control unit 211 controls the direction of the CG ray 403 based on the detection result.

[0055] The ray 403 extends from the controller 120 or the tip of the finger wearing the controller 120 as a reference point (irradiation source). The control unit 211 detects the position of the controller 120 based on the captured image and inertial information, and sets the reference point of the ray 403. The user can select a window by moving the tip of the ray 403. For example, the user can select window A by keeping the tip of the ray 403 overlapping window A for a predetermined period of time.

[0056] Fig. 5 shows a second display example of the HMD 100. Fig. 5 shows a display example in which the controller 120 is present outside the imaging range of the imaging unit 220 of the HMD 100. In this case, the controller 120 and the user's hand 402 are not displayed in the display area 404 of the HMD 100. Fig. 5 shows that the controller 120 and the user's hand 402 are present outside (below) the display area 404. Even in this case, the position of the controller 120 can be acquired in step S1005, so the ray 403 extends from the reference point outside the display area 404 toward the window C.

[0057] FIG. 6 shows a third display example of the HMD 100. FIG. 6 is another example of the case where the controller 120 is present outside the imaging range of the imaging unit 220 of the HMD 100. Here, the controller 120 has moved spatially to the left from the state in FIG. 5, and a ray 403 indicates window B. The controller 120 and the user's hand 402 are present outside (below) the display area 404, and the controller 120 and the user's hand 402 have moved to the left from the state in FIG. 5. Although the imaging unit 220 has not captured (detected) the controller 120, the control unit 211 has detected, via the motion detection unit 224, that the controller 120 has moved to the left. Therefore, the control unit 211 is able to reflect the movement of the controller 120 in the display of the ray.

[0058] According to the first embodiment, even if the controller 120 is outside the imaging range of the imaging unit 202 of the HMD 100, the position of the controller 120 can be detected (acquired) with high accuracy by using the motion detection unit 224. This allows the HMD 100 to display the ray 403 at a more accurate position. Furthermore, since the controller 120 does not need to include sensors other than the inertial sensor, the controller 120 can be made smaller.

[0059] <Variation 1> In the first modification, the operation of the information processing system 1 will be described assuming that the motion detection unit 224 is a device with a large detection error in the inertial information.

[0060] If the motion detection unit 224 is, for example, an IMU, slight noise occurs in the detected inertial information even when the controller 120 is completely stationary. As a result, the inertial information erroneously conveys to the control unit 122 that the controller 120 is moving in response to the noise. This noise is negligible for a short period of time, but accumulates over a long period of time, increasing to a level that cannot be ignored. As a result, for example, as shown in FIG. 7, the actual position of the controller 120 deviates from the reference position of the ray 403 controlled based on the inertial information.

[0061] 7 is a fourth display example of the HMD 100 in Modification Example 1. This figure shows the positional relationship between the display area 404 and the controller 120 after a threshold time has elapsed while the controller 120 remains outside the imaging range of the imaging unit 202 of the HMD 100.

[0062] To prevent such a discrepancy, in Modification 1, once a threshold time has elapsed since the last time the process of step S1004 was performed, control unit 211 stops acquiring the position of controller 120 based on the previously acquired position of controller 120 and inertia information. Specifically, if controller 120 is not included in the imaging range (if controller 120 is not detected in the captured image), control unit 211 determines whether the threshold time has elapsed since the time when controller 120 was no longer included in the imaging range. Then, once the threshold time has elapsed from that time, control unit 211 stops acquiring the position of controller 120 based on the previously acquired position and inertia information of controller 120 (stops position acquisition using inertia information).

[0063] After stopping position acquisition using inertial information, in step S1005, the control unit 211 acquires the position of the controller 120 acquired at the time when position acquisition using inertial information was stopped (when the threshold time has elapsed) as the current position of the controller 120. Then, the control unit 211 displays the ray 403 on the image display unit 203 based on the acquired current position of the controller 120. Alternatively, the control unit 211 acquires a predetermined position (predetermined position) as the current position of the controller 120, and displays the ray 403 or the pointer (position-specified display) on the image display unit 203 based on the acquired position.

[0064] If the controller 120 remains outside the imaging range of the imaging unit 202 for a long period of time (longer than the threshold time), it is unlikely that the controller 120 is constantly moving, and it is more likely that the controller 120 is staying at a fixed position outside the imaging range. This is because if the controller 120 is constantly moving, it is more likely that the controller 120 will return to the imaging range. If the controller 120 remains outside the imaging range, it is possible that the controller 120 is being operated in a pocket of the user's clothes, or that the user's hand is relaxed and the controller 120 is located under the user's arm. Therefore, in this case, keeping the reference position of the ray at a fixed position reduces the error between the reference position and the actual position of the hand, rather than moving the reference position of the ray based on inertial information acquired by the motion detection unit 224. Furthermore, reducing the error between the reference position and the actual position of the hand can reduce the sense of discomfort felt by the user when operating the controller. As described above, this fixed position may be not only the position of the controller 120 at the time position acquisition based on inertial information is stopped, but also a predetermined position. For example, the user may register in advance a position where the user can hold the controller 120 comfortably as a certain location.

[0065] The control unit 211 may change the threshold time in accordance with the temperature of the motion detection unit 224 (IMU) at the time when the controller 120 is no longer included in the detection range (the time when the controller 120 is no longer detected). The amount of noise generated by the motion detection unit 224 varies depending on the temperature of the motion detection unit 224. For this reason, the control unit 211 measures the amount of noise generated for each temperature of the motion detection unit 224 in advance, and shortens the threshold time if the current temperature of the motion detection unit 224 is one at which noise is likely to be generated. This reduces the error between the reference position and the actual position of the hand. The control unit 211 may control the threshold time in accordance with the air pressure at the time when the controller 120 is no longer included in the detection range, rather than the temperature of the motion detection unit 224 (IMU). In other words, the control unit 211 may control the threshold time in accordance with any environmental information that may affect the inertial information.

[0066] <Variation 2> Hereinafter, as an example different from the first modification, the operation of the information processing system 1 will be described assuming that the motion detection unit 224 is a device that has a large detection error in the inertial information.

[0067] Fig. 8 shows a display pattern in the fifth case in Modification Example 2. As with Fig. 7, this is a diagram showing a display after a threshold time has elapsed while controller 120 continues to be outside the imaging range of imaging section 202.

[0068] 8, unlike Fig. 7, in a display 801 of a portion of the ray 403 close to the controller 120, the ray 403 is displayed partially faded or blurred to blend into the surroundings. Also, the display 801 of the portion of the ray 403 close to the controller 120 does not need to be displayed at all.

[0069] In FIG. 7 of Modification 1, the ray is clearly displayed up to the edge of the display area, so the difference between the position of the controller 120 and the reference point of the ray is noticeable. Therefore, in Modification 2, as shown in FIG. 8, the control unit 211 changes (modifies) the display of the portion close to the reference point of the ray so that it becomes less visible after a threshold time has elapsed. This has the effect of making the difference between the position of the controller 120 and the reference point of the ray less noticeable, even if there is a difference. This makes it possible to suppress the sense of discomfort felt by the user when controlling the ray. Note that the control unit 211 may change the display of the ray according to the time that the controller 120 continues to exist outside the imaging range of the imaging unit 202. For example, the control unit 211 increases the transparency of the ray as the time that the controller 120 continues to exist outside the imaging range continues. Then, the control unit 211: Once the specified time has passed, the ray becomes completely invisible (the ray's transparency becomes 100%).

[0070] In the above embodiment and modified examples, an example has been described in which a ray is displayed based on the position and orientation of the controller 120, but other display items may also be displayed. For example, the information processing system 1 may estimate the position at which the user's finger is pointing based on the position and orientation of the controller 120, and display a pointer or a specific icon at the estimated position.

[0071] Furthermore, in the above, "If A is equal to or greater than B, proceed to step S1; if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1; if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1; if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1; if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction arises, the expression "equal to or greater than A" may be read as "A or greater than A (high; long; many)" or "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller than A (low; short; few)" or "smaller than A (low; short; few)." Furthermore, "bigger than A (higher; longer; more)" can be read as "A or more," and "smaller than A (lower; shorter; fewer)" can be read as "A or less."

[0072] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0073] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0074] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The disclosure of the above embodiments includes the following configurations, methods, systems, and programs. [Configuration 1] an acquisition means for acquiring inertial information from an inertial sensor included in the controller; a detection means for detecting the controller from an image captured by the imaging means; a control means for acquiring a position of the controller based on a previously acquired position of the controller and the inertia information when the controller is not detected from the captured image; An information processing device comprising: [Configuration 2] when the controller is detected from the captured image, the control means acquires the position of the controller based on the inertial information and the captured image; 2. The information processing device according to configuration 1, [Configuration 3] the control means controls the display means to display a display item at a position based on the acquired position of the controller. 3. The information processing device according to configuration 1 or 2. [Configuration 4] the display item is a light beam extending from a position based on the position of the controller acquired by the control means; 4. The information processing device according to configuration 3. [Configuration 5] the display means and the imaging means are provided in a head-mounted display that can be worn on the head of a user; 5. The information processing device according to configuration 3 or 4. [Configuration 6] the control means controls the display means to change the display of a part of the display items when a state in which the controller is not detected from the captured image continues for a period longer than a threshold time. 6. The information processing device according to any one of configurations 3 to 5. [Configuration 7] when a state in which the controller is not detected from the captured image continues for a period longer than a threshold time, the control means stops acquiring the position of the controller based on previously acquired positions of the controller and the inertia information. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the control means acquires a predetermined position as the current position of the controller. 8. The information processing device according to configuration 7. [Configuration 9] when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the control means acquires the position of the controller acquired when the threshold time has elapsed as the current position of the controller. 8. The information processing device according to configuration 7. [Configuration 10] the control means controls the threshold time in accordance with the temperature of the controller when the controller is not detected from the captured image. 10. The information processing device according to any one of configurations 6 to 9. [Configuration 11] The controller is a ring-type controller that can be worn on a user's finger. 11. The information processing device according to any one of configurations 1 to 10. [system] An information processing device according to any one of configurations 1 to 11; a controller having the inertial sensor that acquires the inertial information; a head mounted display having the imaging means; An information processing system comprising: [method] an acquisition step of acquiring inertial information from an inertial sensor included in the controller; a detection step of detecting the controller from a captured image acquired by imaging means; a control step of acquiring a position of the controller based on a previously acquired position of the controller and the inertia information when the controller is not detected from the captured image; An information processing method comprising: [program] 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0076] 1: information processing system, 110: information processing device, 120: controller, 211: control unit, 213: communication unit, 224: motion detection unit

Claims

1. An acquisition means for acquiring inertial information from an inertial sensor included in the controller; a detection means for detecting the controller from an image captured by the imaging means; a control means for acquiring a position of the controller based on a previously acquired position of the controller and the inertia information when the controller is not detected from the captured image; having the control means controls the display unit to display a display item at a position based on the acquired position of the controller, and when a state in which the controller is not detected from the captured image continues for a period longer than a threshold time, stops reflecting the position of the controller, which is based on a previously acquired position of the controller and the inertia information, in the position of the display item.

23. An information processing apparatus comprising:

2. when the controller is detected from the captured image, the control means acquires a position of the controller based on the inertial information and the captured image.

2. The information processing apparatus according to claim 1,

3. the display item is a light beam extending from a position based on the position of the controller acquired by the control means; 2. The information processing apparatus according to claim 1,

4. The display unit and the imaging means are provided in a head mounted display that can be worn on a user's head.

2. The information processing apparatus according to claim 1,

5. the control means controls the display unit to change a display of a part of the display items when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time.

2. The information processing apparatus according to claim 1,

6. when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the control means stops acquiring the position of the controller based on previously acquired positions of the controller and the inertia information.

2. The information processing apparatus according to claim 1,

7. the control means, when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, acquires a predetermined position as a current position of the controller.

7. The information processing apparatus according to claim 6,

8. when a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the control means acquires the position of the controller acquired at the time when the threshold time has elapsed as the current position of the controller.

7. The information processing apparatus according to claim 6,

9. When a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the display item is displayed at a position based on a predetermined position and the inertial information.

2. The information processing apparatus according to claim 1,

10. When a state in which the controller is not detected from the captured image continues for a period longer than the threshold time, the display item is displayed at a position based on the inertial information while keeping a reference position of the display item at a predetermined position.

2. The information processing apparatus according to claim 1,

11. the control means controls the threshold time in accordance with a temperature of the controller when the controller is not detected from the captured image.

11. The information processing apparatus according to claim 1,

12. The controller is a ring-type controller that can be worn on a user's finger.

11. The information processing apparatus according to claim 1,

13. An information processing device according to any one of claims 1 to 10; a controller having the inertial sensor for acquiring the inertial information; A head mounted display having the imaging means; An information processing system comprising:

14. An acquisition step of acquiring inertial information from an inertial sensor included in the controller; a detection step of detecting the controller from an image captured by an imaging means; a control step of acquiring a position of the controller based on a previously acquired position of the controller and the inertia information when the controller is not detected from the captured image; having In the control step, a display unit is controlled so as to display a display item at a position based on the acquired position of the controller, and when a state in which the controller is not detected from the captured image continues for a period longer than a threshold time, a position of the controller based on a previously acquired position of the controller and the inertia information is reflected in the position of the display item. Stop 23. An information processing method comprising:

15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images