Position input system, controller, and attachment

Abstract

The present invention balances productions costs with ensuring necessary tracking precision. This position input system comprises: an attachment (110) that has a plurality of light-emitting parts (81) which are for detecting the position of the attachment, and an engaging part (89); and a controller (100) that includes an engagement-receiving part (79) to which the engaging part of the attachment can be attached, and a device which is for detecting the position of the controller.

Description

Technical Field

[0001] The present disclosure relates to a position input system, a controller, and an attachment.

Background Art

[0002] For VR and the like, tracking technologies for measuring the three-dimensional position of an input device have been developed. In tracking technologies, a plurality of light-emitting tracking points are provided on the outer surface of the input device, the tracking points are detected through a camera, and the position and orientation of the input device are detected by further combining the output of an inertial sensor.

[0003] Patent Document 1 discloses an input device for game operation provided with a spherical light-emitting portion.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to accurately track the position of the input device and the like, an inertial sensor and a large number of light-emitting portions are required. Therefore, there has been a trade-off relationship between improvement in accuracy and manufacturing cost.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a technology capable of achieving both manufacturing cost and ensuring required tracking accuracy.

Means for Solving the Problems

[0007] The position input system according to the present invention includes a plurality of light emitting units for detecting its own position, an attachment having an engaging portion, an engaged portion to which the engaging portion of the attachment can be attached, and a controller including a device for detecting its own position.

[0008] The controller according to the present invention includes an engaged portion to which an attachment having a plurality of light emitting units for detecting its own position and an engaging portion can be attached, and a device for detecting its own position.

[0009] The attachment according to the present invention includes an engaging portion that can be attached to a controller including a device for detecting its own position, and a plurality of light emitting units for detecting its own position.

[0010] According to the present invention, it is possible to achieve both low manufacturing costs and the required tracking accuracy.

[0011] In one embodiment of the present invention, the device may include a sensor that detects acceleration and angular changes.

[0012] In one embodiment of the present invention, the position input system may further include a position detection unit that detects the position and orientation of the controller to which the attachment is attached based on the light output from any one of the plurality of light emitting units and the output of the sensor.

[0013] In one embodiment of the present invention, the device may include one or more light emitting units.

[0014] In one embodiment of the present invention, the position input system may further include a position detection unit that detects the position and orientation of the controller to which the attachment is attached based on the light output from any one of the plurality of light emitting units included in the attachment and the light output from any one of the plurality of light emitting units included in the device.

[0015] In one aspect of the present invention, the position detection unit may detect the position of the attachment in a state where the attachment is not attached to the controller.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Mode for Carrying Out the Invention

[0017] Hereinafter, the position input system according to the embodiment of the present invention will be described with reference to the drawings. Hereinafter, among the components that appear, those having the same function are denoted by the same reference numerals, and duplicate explanations are omitted.

[0018] [First Embodiment] FIG. 1 is a diagram showing an example of a position input system according to a first embodiment of the present invention. The position input system includes a head-mounted display (HMD) 2, a controller 100, and an attachment 110. A plurality of cameras 3 for photographing the front are attached to the head-mounted display 2. The user wears the head-mounted display 2 on the head and holds the controller 100 with at least one of the right hand and the left hand. An attachment 110 is attached to the controller 100.

[0019] FIG. 2 is a block diagram showing the configuration of the position input system, and FIG. 3 is a diagram showing an example of the controller 100 and the attachment 110.

[0020] The controller 100 includes an inertial sensor 72, an operation unit 73, a transmission unit 75, a battery 76, an engaged portion 79, and a control unit (not shown). The inertial sensor 72, called an IMU, includes an acceleration sensor that measures triaxial acceleration and a gyro sensor that measures triaxial rotational speed. The operation unit 73 acquires operations by the user's finger. The operation unit 73 is, for example, an operation button, an operation stick, or a touch sensor. The transmission unit 75 transmits the user input acquired by the operation unit 73 and the output of the inertial sensor 72 to the head-mounted display 2 by a predetermined communication method such as Bluetooth (registered trademark). The control unit includes a processor and a memory, and controls the inertial sensor 72, the operation unit 73, and the transmission unit 75. The battery 76 supplies power to the control unit, the inertial sensor 72, the transmission unit 75, and the like. The attachment 110 may not include the battery 76. In this case, power may be supplied from the controller 100 to the attachment 110.

[0021] The attachment 110 includes a plurality of light emitting units 81, a battery 86, an engaging portion 89, and a control unit (not shown). The plurality of light emitting units 81 are indicated by black circles in FIG. 3, and are similarly shown in subsequent figures. The plurality of light emitting units 81 include a plurality of light emitting diodes (LEDs). The light emitting unit 81 emits light toward the outside of the attachment 110. The camera 3 captures the light emitted from the light emitting unit 81, and the position and orientation of the attachment 110 are detected based on the position of the light of the captured light emitting unit 81. Any plurality of the light emitting units 81 may be configured by one light emitting diode and a light guide tube. The battery 86 supplies power to the control unit, the light emitting unit 81, and the like. The attachment 110 may have the inertial sensor 72 and the transmission unit 75.

[0022] The head-mounted display 2 includes a display unit 24 (see FIG. 13). An image (e.g., a game image) generated based on the position and orientation of the controller 100 and the operations performed on the operation unit 73 is displayed on the display unit 24.

[0023] In FIG. 2, the functions implemented in the head-mounted display 2 are described inside the box of the head-mounted display 2. Functionally, the head-mounted display 2 includes a feature point extraction unit 51, an attitude estimation unit 52, a reception unit 53, an integration unit 54, and an application unit 58. The feature point extraction unit 51 extracts the position of the light from the light-emitting unit 81 from the image captured by the camera 3. The attitude estimation unit 52 estimates the position and attitude of the attachment 110 and the controller 100 based on the position of the extracted light. The reception unit 53 receives the output of the inertial sensor 72 and the user's operation on the operation unit 73 from the transmission unit 75. The integration unit 54 detects the position and attitude of the attachment 110 and the controller 100 based on the estimated position and attitude and the output of the inertial sensor 72. The detailed processing of these functions will be described later. The feature point extraction unit 51, the attitude estimation unit 52, the reception unit 53, and the integration unit 54 constitute a position detection unit 50. The position detection unit 50 detects (tracks) the position and attitude of the controller 100 to which the attachment 110 is attached based on the light output from any one of the plurality of light-emitting units 81 and the output of the inertial sensor 72.

[0024] In FIG. 2, these functions are realized by the head-mounted display 2, but may also be realized by other external information processing devices (for example, a game device separate from the head-mounted display 2 or a personal computer). Generation of the image output to the display unit 24 of the head-mounted display 2 may be performed by an external information processing device. In this case, information on the image captured by the camera 3 is transmitted wirelessly or wired to the external information processing device, and the information processing device generates an image and transmits the information on the generated image wirelessly or wired to the head-mounted display 2. Note that the position input system may not include the head-mounted display 2. In this case, the camera 3 may be attached to an external display device (for example, a monitor of a television or a personal computer) that displays an image generated by an external information processing device.

[0025] FIGS. 4 to 7 are respectively a front view, a plan view, a side view, and a rear view of the controller 100 and the attachment 110. In these figures, the attachment 110 is attached to the controller 100. The controller 100 shown in these figures is for operation with the right hand. There may be a left-handed controller 100 having a bilaterally symmetric shape.

[0026] The controller 100 includes a grip 77 and operation units 73a, 73b, and 73c. The grip 77 extends downward of the operation units 73a, 73b, and 73c. The stick of the operation unit 73a and the button of the operation unit 73b are operated, for example, by the thumb in a state of holding the grip 77, and the operation unit 73c is operated, for example, by the index finger. The grip 77 is held, for example, by the thenar, the middle finger, the ring finger, and the little finger. The operation unit 73 disposed on the controller 100 may not be as described above. For example, the controller 100 may have a touch sensor, a trigger button, or a button with a touch sensor as the operation unit 73.

[0027] The attachment 110 is circular in plan view, has a larger radius near the center in the vertical direction, and has a smooth curved surface on the sides of the attachment 110. On the surface of the attachment 110, light-emitting parts 81 are provided at random. The light-emitting parts 81 are arranged on the surface of the attachment 110 such that three or more light-emitting parts 81 are captured regardless of the direction from which the image is captured.

[0028] The engaging part 89 of the attachment 110 is attached to the engaged part 79 of the controller 100. FIG. 8 is a cross-sectional view taken along the cutting line VIII-VIII of FIG. 4. As shown in FIGS. 8 and 4 to 7, the engaged part 79 of the controller 100 and the engaging part 89 of the attachment 110 are engaged with each other.

[0029] FIGS. 9 to 12 are a perspective view, a plan view, a front view, and a cross-sectional view of the engaged part 79, respectively. The engaged part 79 is attached to the upper end of the controller 100 and has a recess 91 that is open on the front side and the upper side. The upper side of the recess 91 is narrow, and the engaging part 89 has a shape corresponding to the recess 91. This enables the engaging part 89 to be attached to the engaged part 79 from the front side. Terminals 92 are provided on the engaged part 79. When the engaging part 89 is attached to the engaged part 79, the terminals 92 are electrically connected to the terminals on the engaging part 89 side. Communication may be performed between the controller 100 and the attachment 110 through the terminals 92, or power may be supplied from the controller 100 to the attachment 110.

[0030] Here, not all the light-emitting parts 81 always emit light. The pattern of the light-emitting parts 81 that emit light and those that do not emit light may change at regular intervals, or the light-emitting parts 81 may emit light in synchronization with the shooting timing of the camera 3. The synchronization of the light-emitting pattern and timing of the light-emitting parts 81 may be controlled by the control unit of the controller 100.

[0031] Next, the processing related to the detection of the position and orientation will be described. FIG. 13 is a diagram showing an example of the hardware configuration of the head-mounted display 2. The head-mounted display 2 includes a processor 21, a memory 22, a communication unit 23, a display unit 24, and a camera 3.

[0032] The processor 21 executes a program stored in the memory 22 and controls the communication unit 23 and the display unit 24. The memory 22 is a storage element such as a RAM or a flash memory, and stores programs and calculation results of the processor 21. The program may be stored in the flash memory in advance, or may be provided via a communication line such as the Internet.

[0033] The communication unit 23 is an integrated circuit and an antenna or a terminal that realizes an interface for communicating with an external device, and includes, for example, a wireless LAN module and a Bluetooth (registered trademark) module. The communication unit 23 also corresponds to the receiving unit 53.

[0034] The display unit 24 is a display such as a liquid crystal display or an organic EL display disposed on the front side of the head-mounted display 2. The camera 3 has an element that captures the light of the light emitting unit 81. When the light emitting unit 81 outputs infrared rays, the camera 3 captures infrared rays, and when the light emitting unit 81 outputs visible light, the camera 3 captures visible light.

[0035] FIG. 14 is a diagram showing an example of the processing for detecting the position. The processor 21 executes the processing shown in FIG. 14 by executing a program stored in the memory 22. The feature point extraction unit 51, the attitude estimation unit 52, the integration unit 54, and the application unit 58 are realized by the processor 21 executing a program stored in the memory 22.

[0036] First, the feature point extraction unit 51 acquires the image captured by the camera 3 (step S201). The head-mounted display 2 may have four cameras 3, and the feature point extraction unit 51 may acquire four images respectively captured by the four cameras 3.

[0037] Next, the feature point extraction unit 51 extracts the light points from the light source 81 from the acquired image (step S202). More specifically, the feature point extraction unit 51 obtains the centroid coordinates and the area of the region from the light region of the light source 81. The centroid coordinates correspond to the position of the light point. Then, based on the plurality of two-dimensional points of the plurality of light sources 81 extracted and the three-dimensional model of the light source 81 of the attachment 110, the attitude estimation unit 52 estimates the attitudes of the controller 100 and the attachment 110 (step S203).

[0038] More specifically, the attitude estimation unit 52 randomly samples a combination of three out of the plurality of extracted points and three out of the plurality of light sources 81 included in the three-dimensional model of the controller 100, and obtains a combination that minimizes the error between the points obtained by reprojection of the sampled light source 81. The attitude estimation unit 52 calculates the three-dimensional positions and angles (attitudes) of the attachment 110 and the controller 100 based on the positions and angles of the three-dimensional models of the attachment 110 and the controller 100 in the combination with the minimum error, and obtains the calculated positions and angles as the estimated positions and attitudes of the controller.

[0039] In the sampling of the light source 81 from the three-dimensional model, more specifically, the attitude estimation unit 52 samples three out of the light sources 81 that are emitting light at the time when the camera 3 takes a picture. Also, the method for obtaining the error, position, and attitude for the combination of the sampled light points on the plane and the light source 81 of the three-dimensional model is known as a solution to the P3P problem using RANSAC, which is a kind of robust estimation method, so the description is omitted. Note that the number of sampled points and the number of lights of the light source 81 do not have to be 3, and may be 2 or 4, for example.

[0040] On the one hand, the receiving unit 53 acquires the angular velocity and angular velocity detected by the inertial sensor 72 of the controller 100 (step S204). Then, the integrating unit 54 determines the positions and postures of the controller 100 and the attachment 110 based on the acquired acceleration and angular velocity, and the estimated position and posture (step S205). This process is a process of improving accuracy using the output of the inertial sensor 72. More specifically, the integrating unit 54 inputs the information acquired from the inertial sensor 72 and the estimated position and posture into the Kalman filter, and determines the output result as the positions and postures of the controller 100 and the attachment 110. Further, when performing the process of step S204, the integrating unit 54 converts both or one of the coordinate spaces so that the coordinate space of the inertial sensor 72 and the coordinate space of the position and posture calculated by the posture estimation unit 52 are matched, and uses the value of the inertial sensor 72 in the converted coordinate space and the estimated position and posture to execute the Kalman filter process.

[0041] The determined positions and postures of the controller 100 and the attachment 110 are output to the application unit 58 as the detected positions and postures, and the application unit 58 executes processing based on the positions and postures, and controls the display unit 24 and the like so that an image corresponding to the positions and postures is output.

[0042] When estimating the position and posture only by the light of the light emitting unit 81, there is a possibility that the position and posture may move larger than the actual due to errors. In the present embodiment, by using the estimation of the position and posture based on the light from the light emitting unit 81 and the acceleration and angular velocity detected by the inertial sensor 72, the detection accuracy of the positions and postures of the controller 100 and the attachment 110 can be improved.

[0043] In this embodiment, while using a plurality of light-emitting units 81 provided in the attachment 110, the inertial sensor 72 built in the controller 100 is combined. When the inertial sensor 72 is originally built in the controller 100, it is not necessary to build the inertial sensor 72 on the attachment 110 side, so it becomes easy to achieve both low cost and high accuracy.

[0044] The attachment 110 alone may be attached to the arm or leg. In this case, the position and orientation may be detected based only on the light from the light-emitting unit 81 of the attachment 110. The position detection unit 50 may output the position and orientation estimated by the orientation estimation unit 52 to the application unit 58 as the detected position and orientation. However, since the detection result of the inertial sensor 72 is not used, the accuracy of the detected position and orientation decreases.

[0045] Note that an inertial sensor with lower accuracy and lower cost than the controller 100 may be built in the attachment 110, and the acceleration and angular velocity detected by this inertial sensor 72 may be used for detecting the position and orientation in the case of the attachment alone. In this case as well, it is possible to achieve both low cost and high accuracy.

[0046] The controller 100 to which the attachment 110 is attached is not limited to those described so far. FIGS. 15 and 16 are a front view and a side view showing another example of the controller 100 and the attachment 110. FIGS. 15 and 16 show a controller 100 that is held with both hands. The operation units 73a and 73b are operated by the fingers of the right hand, and the operation units 73e and 73f are operated by the fingers of the left hand. The engaging portion 89 of the attachment 110 is attached to the engaged portion 79 of the controller 100.

[0047] The shape of the attachment 110 may be different from that described so far. FIG. 17 is a perspective view showing another example of the controller 100 and the attachment 110. The attachment 110 shown in FIG. 17 has a main part 115 with a shape where a part of the ring is cut, and an arc-shaped and plate-shaped arc part 117 arranged on the side of the center of the ring of the main part 115. A plurality of light-emitting parts 81 are provided at random on the surface of the main part 115. The light-emitting parts 81 are arranged on the surface of the attachment 110 so that the plurality of light-emitting parts 81 are photographed even when photographed from many directions. The controller 100 shown in FIG. 17 is held with both hands. An engaging part 89 (not shown) is arranged at the end of the main part 115, and an engaged part 79 (not shown) is arranged below the center of the controller 100.

[0048] Even with the attachment 110 and the controller 100 having the shapes shown in FIGS. 15, 16, and 17, the same effects can be obtained.

[0049] [Second Embodiment] In the first embodiment, the accuracy is improved by combining the light-emitting part 81 of the attachment 110 and the inertial sensor 72 of the controller 100. In the second embodiment, other combinations are used to improve the accuracy. Below, the differences mainly from the first embodiment will be described.

[0050] FIGS. 18 and 19 are a front view and a side view showing an example of the controller 100 and the attachment 110 according to the second embodiment. FIG. 20 is a perspective view of the controller 100 and the attachment 110. In FIGS. 18 and 19, the engaging part 89 of the attachment 110 is attached to the engaged part 79 of the controller 100, and in FIG. 20, the attachment 110 and the controller 100 are shown separated from each other.

[0051] In the examples of FIGS. 18 to 20, the shape of the attachment 110 is the same as that shown in FIG. 17. The attachment 110 is attached to the controller 100 such that the main part 115 bends to the right when viewed from the front.

[0052] The controller 100 in the examples of FIGS. 18 to 20 includes a plurality of light emitting parts 71, operation parts 73a, 73b, 73c, a grip 77, and an engaged part 79. The controller 100 shown in FIGS. 18 to 20 is held by the right hand, and the operation parts 73a, 73b are operated by the thumb of the right hand. The operation part 73c is operated by, for example, the index finger of the right hand. The grip 77 is held by, for example, the middle finger, ring finger, and little finger of the right hand.

[0053] The plurality of light emitting parts 71 are arranged in a region on the surface of the controller 100 where there is little possibility of being hidden by the user's hand. The plurality of light emitting parts 71 are arranged in that region so that the plurality of light emitting parts 81 are photographed from as many directions as possible and separately. More specifically, the plurality of light emitting parts 71 are mainly provided on the left side of the operation part 73a on the left side when viewed from the front and under the grip 77.

[0054] In the examples of FIGS. 18 to 20, not only the light emitting part 81 of the attachment 110 but also the light emitting part 71 of the controller 100 are used for detecting the positions of the controller 100 and the attachment 110. Thereby, even when the controller 100 includes the inertial sensor 72 and the position and orientation are detected by the same processing as in the first embodiment, the position and orientation of the controller 100 detected by the position detection unit 50 can be detected with higher accuracy.

[0055] Figs. 21 and 22 are diagrams showing the light emission of the controller 100 held by the user. Figs. 22 and 23 are diagrams showing the light emission of the controller 100 and the attachment 110 held by the user. In Figs. 21 and 22, the attachment 110 is not attached to the controller 100, and in Figs. 23 and 24, the attachment 110 is attached to the controller 100. Figs. 21 and 23 are views seen from the right side when looking at the front of the controller 100, and Figs. 22 and 24 are views seen from the upper side of the controller 100.

[0056] As can be seen from Figs. 21 and 22, with only the light of the light-emitting part 71 of the controller 100, depending on the shooting direction, the number of light-emitting parts 71 photographed by the camera 3 decreases. On the other hand, in Figs. 23 and 24, the light of the light-emitting part 81 of the attachment 110 is also photographed by the camera 3. For this reason, the attitude estimation unit 52 can estimate the position and attitude with higher accuracy based on the light of the light-emitting parts 71 and 81 photographed by the camera 3.

[0057] In the examples of Figs. 18 to 20, when viewed from the front, the light-emitting part 81 is mainly arranged on the upper side and the right side, and the light-emitting part 71 is mainly arranged on the left side and the lower side. For this reason, even if the orientation of the controller 100 changes, the change in the number of the light-emitting parts 71 and 81 photographed by the camera 3 can be suppressed, and it is also possible to disperse the positions of the light-emitting parts 71 and 81 in the photographed image. Therefore, it is possible to prevent the accuracy of the position and attitude estimated due to the change in the orientation of the controller 100 from decreasing.

[0058] Note that even when the inertial sensor 72 is not mounted on the controller 100, by combining the light-emitting part 81 of the attachment 110 and the light-emitting part 71 of the controller 100, the position and attitude can be detected with higher accuracy than when only using the light-emitting part 71 of the controller 100 alone.

Claims

1. A plurality of light emitting units for detecting its own position, and an attachment having an engaging portion, an engaged portion to which the engaging portion of the attachment can be attached, and a controller including a device for detecting its own position, a position detection unit for detecting the position and orientation of the controller to which the attachment is attached, comprising the device includes one or more light emitting units, the position detection unit is based on the light output from any one of the plurality of light emitting units included in the attachment and the light output from any one of the plurality of light emitting units included in the device, and detects the position and orientation of the controller to which the attachment is attached. A position input system.

2. In the position input system according to claim 1, the device includes a sensor for detecting acceleration and angular change, A position input system.

3. In the position input system according to claim 2, the position detection unit is based on the light output from any one of the plurality of light emitting units included in the attachment, the light output from any one of the plurality of light emitting units included in the device, and the output of the sensor, and detects the position and orientation of the controller to which the attachment is attached. A position input system.

4. In the position input system according to any one of claims 1 to 3, the position detection unit detects the position of the attachment in a state where the attachment is not attached to the controller. A position input system.

5. An engaged part capable of attaching an attachment having a plurality of light-emitting parts and engaging parts for detecting its own position, A device for detecting its own position, the device including one or more light-emitting parts, A control part for controlling the light emission pattern and light emission timing of the plurality of light-emitting parts included in the attachment attached to the engaged part, including, The position and orientation of the controller to which the attachment is attached are detected based on the light output by any one of the plurality of light-emitting parts included in the attachment and the light output by any one of the plurality of light-emitting parts included in the device, Controller.

6. An engaging part capable of attaching to a controller including a device for detecting its own position, the device including one or more light-emitting parts, A plurality of light-emitting parts for detecting its own position, the plurality of light-emitting parts being controlled in light emission pattern and light emission timing by the controller attached to the engaging part, An attachment including, The position and orientation of the controller attached to the engaging part are detected based on the light output by any one of the plurality of light-emitting parts included in the attachment and the light output by any one of the plurality of light-emitting parts included in the device, Attachment.

Images