Devices with multiple markers

Abstract

To provide a device with a plurality of markers in which the number of light sources is made to be smaller than the number of markers.SOLUTION: An input device 16 comprises: a case body; and a plurality of markers 30 which emit light outwardly from the case body. A first light source 32 is a specialized light source for lighting one of the markers 30 and a second light source 34 is a shared light source for lighting the plurality of markers 30. The second light source 34, connected to the plurality of the markers 30 via a light guide tube 36, supplies the light to the plurality of the markers 30 via the light guide tube 36.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a device having a plurality of markers. [Background technology]

[0002] Patent document 1 discloses a game device that acquires a frame image taken in front of the game device, estimates position information and orientation information of the game controller in real space from the position of the LED image of the game controller in the frame image, and reflects the estimated position information and / or orientation information in the processing of a game application. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-296248 A Summary of the Invention [Problem to be solved by the invention]

[0004] A head-mounted display (HMD) is worn on the user's head and provides the user with a virtual reality (VR) image space. The user wearing the HMD can perform various inputs into the image space by operating the operation buttons of the input device.

[0005] In recent years, technology that tracks the position and orientation of a device and reflects it in a 3D model in a VR space has become widespread. By linking the movement of the player character or game object in the game space to the change in the position and orientation of the device being tracked, users can intuitively operate the device. Multiple illuminated markers are used for device tracking, and the position and orientation of the device in real space are estimated by analyzing an image of the multiple markers and identifying the position of each marker image in the image.

[0006] In order to estimate the position and orientation of a device with high accuracy, it is necessary to accurately identify the position of each marker image in the image of the device. The more marker images are captured, the higher the accuracy of estimating the position and orientation of the device. However, since the markers are illuminated by light sources such as light-emitting elements, an increase in the number of light sources increases the manufacturing cost.

[0007] Therefore, an object of the present invention is to make the number of light sources less than the number of markers in a device equipped with multiple markers. The device may be an input device having an operation button, or it may be a device that does not have an operation member and is simply a tracking target. [Means for solving the problem]

[0008] In order to solve the above problems, a device according to an embodiment of the present invention includes a case body, a plurality of markers that emit light to the outside of the case body, a first light source for lighting one marker, and a second light source for lighting the plurality of markers. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the external shape of an HMD. [Diagram 3] FIG. 2 is a diagram showing functional blocks of an HMD. [Figure 4] FIG. 2 is a diagram showing the external shape of an input device. [Diagram 5] 11A and 11B are diagrams illustrating examples of light emission patterns for synchronization processing that specifies the timing of photographing. [Figure 6] FIG. 2 is a diagram illustrating functional blocks of an input device. [Figure 7] FIG. 13 is a diagram showing an example of a portion of an image captured using an input device. [Figure 8] 11 is a diagram showing a lighting period of a marker set within an exposure period of an imaging device. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 shows an example of the configuration of an information processing system 1 in an embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head mounted display (HMD) 100, an input device 16 operated by a user's fingers, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing device 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or by a known wireless communication protocol.

[0011] The recording device 11 records applications such as system software and game software. The information processing device 10 may download game software from a content server to the recording device 11 via the network 2. The information processing device 10 executes the game software and supplies image data and audio data of the game to the HMD 100. The information processing device 10 and the HMD 100 may be connected by a known wireless communication protocol or by a cable.

[0012] The HMD 100 is a display device that is worn on the user's head and displays images on a display panel located in front of the user's eyes. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, achieving stereoscopic vision. Since the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data in which optical distortion caused by the lenses has been corrected.

[0013] The output device 15 is not necessary for a user wearing the HMD 100, but by providing the output device 15, another user can view the image displayed on the output device 15. The information processing device 10 may display on the output device 15 the same image as the image viewed by the user wearing the HMD 100, or may display a different image. For example, when a user wearing an HMD and another user play a game together, the output device 15 may display a game image from the viewpoint of the character of the other user.

[0014] The information processing device 10 and the input device 16 may be connected by a known wireless communication protocol or by a cable. The input device 16 has a plurality of operation members such as operation buttons, and a user operates the operation members with his / her fingers while holding the input device 16. When the information processing device 10 executes a game, the input device 16 is used as a game controller. The input device 16 has an attitude sensor including a three-axis acceleration sensor and a three-axis gyro sensor, and transmits sensor data to the information processing device 10 at a predetermined cycle (for example, 1600 Hz).

[0015] In the game of the embodiment, not only the operation information of the operation members of the input device 16 but also the position, attitude, movement, etc. of the input device 16 are treated as operation information and reflected in the movement of the player character in the virtual three-dimensional space. For example, the operation information of the operation members may be used as information for moving the player character, and the operation information such as the position, attitude, movement, etc. of the input device 16 may be used as information for moving the arms of the player character. In a battle scene in the game, the movement of the input device 16 is reflected in the movement of the player character holding a weapon, thereby realizing intuitive operation by the user and enhancing the sense of immersion in the game.

[0016] In order to track the position and orientation of the input device 16, the input device 16 is provided with a plurality of markers (light emitting units) that can be photographed by an imaging device 14 mounted on the HMD 100. The information processing device 10 analyzes an image of the input device 16, estimates position information and orientation information of the input device 16 in real space, and provides the estimated position information and orientation information to the game.

[0017] The HMD 100 is equipped with a plurality of imaging devices 14. The plurality of imaging devices 14 are attached in different positions and in different orientations on the front surface of the HMD 100 so that the total imaging range obtained by adding up the imaging ranges of the respective devices includes the entire visual field of the user. The imaging device 14 may be an image sensor capable of acquiring images of a plurality of markers on the input device 16. For example, when the markers emit visible light, the imaging device 14 has a visible light sensor used in general digital video cameras, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. When the markers emit invisible light, the imaging device 14 has an invisible light sensor. The plurality of imaging devices 14 capture images of the area in front of the user at a predetermined cycle (for example, 60 frames / second) in a synchronized manner, and transmit image data of the input device 16 to the information processing device 10.

[0018] The information processing device 10 identifies the positions of multiple marker images of the input device 16 included in the captured image. Although one input device 16 may be captured by multiple image capture devices 14 at the same time, the mounting positions and mounting orientations of the image capture devices 14 are known, so the information processing device 10 combines the multiple captured images to identify the positions of the marker images.

[0019] The three-dimensional shape of the input device 16 and the position coordinates of multiple markers arranged on its surface are known, and the information processing device 10 estimates the position coordinates and attitude of the input device 16 based on the distribution of the marker images in the captured image. The position coordinates of the input device 16 may be position coordinates in a three-dimensional space with a reference position as the origin, and the reference position may be position coordinates (latitude, longitude) set before the start of the game.

[0020] The information processing device 10 can also estimate the position coordinates and the orientation of the input device 16 by using sensor data detected by an orientation sensor of the input device 16. Therefore, the information processing device 10 of the embodiment performs tracking processing of the input device 16 with high accuracy by using an estimation result based on an image captured by the imaging device 14 and an estimation result based on the sensor data.

[0021] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism 102 and a wearing mechanism 104. The wearing mechanism 104 includes a wearing band 106 that goes around the head when worn by the user to secure the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the user's head circumference.

[0022] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of left and right optical lenses located between the display panel and the user's eyes to expand the user's viewing angle. The HMD 100 may further include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.

[0023] A plurality of imaging devices 14a, 14b, 14c, and 14d are provided on the front outer surface of the housing 108. With respect to the user's line of sight, the imaging device 14a is attached to the upper right corner of the front outer surface so that the camera optical axis faces diagonally upward to the right, the imaging device 14b is attached to the upper left corner of the front outer surface so that the camera optical axis faces diagonally upward to the left, the imaging device 14c is attached to the lower right corner of the front outer surface so that the camera optical axis faces diagonally downward to the right, and the imaging device 14d is attached to the lower left corner of the front outer surface so that the camera optical axis faces diagonally downward to the left. By installing a plurality of imaging devices 14 in this manner, the total imaging range obtained by adding up the respective imaging ranges includes the entire visual field of the user. This visual field of the user may be the visual field of the user in a three-dimensional virtual space.

[0024] The HMD 100 transmits sensor data detected by the orientation sensor and image data captured by the imaging device 14 to the information processing device 10, and also receives game image data and game audio data generated by the information processing device 10.

[0025] 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as commands. The storage unit 122 temporarily stores data and commands processed by the control unit 120. The orientation sensor 124 acquires sensor data related to the movement of the HMD 100. The orientation sensor 124 includes at least a three-axis acceleration sensor and a three-axis gyro sensor.

[0026] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0027] When the control unit 120 receives game image data and game audio data from the information processing device 10, it supplies them to the display panel 130 for display, and also supplies them to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit sensor data from the orientation sensor 124, audio data from the microphone 126, and captured image data from the imaging device 14 to the information processing device 10.

[0028] FIG. 4 shows the external shape of the input device 16. FIG. 4(a) shows the front shape of the input device 16, and FIG. 4(b) shows the rear shape of the input device 16. The input device 16 includes a case body 20, a plurality of operation members 22a, 22b, 22c, and 22d (hereinafter, referred to as "operation members 22" unless otherwise specified) operated by the user, and a plurality of markers 30a to 30t (hereinafter, referred to as "markers 30" unless otherwise specified) that emit light to the outside of the case body 20. The operation members 22 are disposed at the head of the case body 20, and include an analog stick that is operated by tilting, a push button, a trigger button that inputs the amount of pulling, and the like.

[0029] The case body 20 has a grip part 21 and a curved part 23 that connects the case body head and the case body bottom, and the user passes his / her fingers from the index finger to the little finger between the grip part 21 and the curved part 23 to grip the grip part 21. While gripping the grip part 21, the user operates the operation members 22a, 22b, and 22c with his / her thumb and operates the operation member 22d with his / her index finger. The markers 30h, 30i, and 30j are provided on the grip part 21, but are arranged in positions that are not hidden by the hand even when the user is gripping the grip part 21. By providing one or more markers 30 on the grip part 21, tracking accuracy can be improved.

[0030] The marker 30 is a light emitting part that emits light to the outside of the case body 20, and includes a resin part on the surface of the case body 20 that diffuses and emits light from a light source such as an LED (Light Emitting Diode) element to the outside. The marker 30 is photographed by the imaging device 14 and used in the estimation process of the position information and the attitude information of the input device 16. Since the imaging device 14 photographs the input device 16 at a predetermined period (for example, 60 frames / second), it is preferable that the marker 30 emits light in synchronization with the periodic photographing timing of the imaging device 14 and is turned off during the non-exposure period by the imaging device 14 to suppress unnecessary power consumption. The imaging device 14 and the input device 16 operate based on their respective clocks, and in the embodiment, the synchronization process of the exposure period by the imaging device 14 and the lighting period of the marker 30 is performed as follows.

[0031] FIG. 5 shows an example of a light emission pattern for synchronization processing that specifies the shooting timing of the imaging device 14. The horizontal length represents a shooting cycle (16.7 ms) for one frame, and lighting control of the marker 30 is performed in units of a time grid into which this shooting cycle is divided. In this example, the shooting cycle is divided into 32, and one time grid is 521 μs. In FIG. 5, the colored time grids indicate the lighting periods at the first luminance, and the uncolored time grids indicate the lighting periods at the second luminance. Note that the first luminance is different from the second luminance, and the first luminance may be higher than the second luminance. When the marker 30 is shot when emitting light at the first luminance, the shot image includes a marker image with high luminance, and when the marker 30 is shot when emitting light at the second luminance, the shot image includes a marker image with low luminance. The light emission pattern is determined so that the order of light emission at the first luminance and light emission at the second luminance differs for each time grid when six images are shot continuously in a shooting cycle for one frame.

[0032] In the synchronization process, one or more markers 30 captured by the imaging device 14 are controlled to be illuminated in the illumination pattern shown in Fig. 5. It is assumed that after the synchronization process starts, the illumination-controlled markers 30 are captured at "first luminance" in the first captured image (Frame 0), "second luminance" in the second captured image (Frame 1), "first luminance" in the third captured image (Frame 2), "second luminance" in the fourth captured image (Frame 3), "second luminance" in the fifth captured image (Frame 4), and "first luminance" in the sixth captured image (Frame 5). The combination of first luminance and second luminance in six consecutive captured images corresponds to the time grid with grid number 14. Therefore, the input device 16 can thereafter periodically light up the marker 30 at the timing of grid number 14, thereby synchronizing the lighting period of the marker 30 with the exposure period of the imaging device 14 and controlling the marker 30 not to light up during the non-exposure period of the imaging device 14.

[0033] 6 shows functional blocks of the input device 16. The control unit 50 receives operation information input to the operating member 22, and also receives sensor data acquired by the attitude sensor 52. The attitude sensor 52 acquires sensor data related to the movement of the input device 16, and includes at least a three-axis acceleration sensor and a three-axis gyro sensor. The control unit 50 supplies the received operation information and sensor data to the communication control unit 54. The communication control unit 54 transmits the operation information and sensor data output from the control unit 50 to the information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 54 also acquires a light emission pattern and / or a light emission instruction for synchronization processing from the information processing device 10.

[0034] The input device 16 includes a first light source 32 for illuminating one marker 30, and a second light source 34 for illuminating multiple markers 30. Both the first light source 32 and the second light source 34 may be LED elements.

[0035] The first light source 32 is a dedicated light source for lighting only one marker 30, and independently lights one marker 30. The marker 30 includes a resin part on the surface of the case body 20 that diffuses and emits light to the outside, but the resin part of the marker 30 that is lighted by the first light source 32 may be a sealing resin for an LED element. In this case, the marker 30 and the first light source 32 may have the form of a single LED device.

[0036] On the other hand, the second light source 34 is a shared light source for lighting two or more markers 30, and simultaneously lights up two or more markers 30. By allowing one second light source 34 to light up multiple markers 30, the number of light sources in the input device 16 can be made smaller than the number of markers 30, which contributes to reducing manufacturing costs.

[0037] The second light source 34 is connected to the multiple markers 30 via a light guide tube 36. The light guide tube 36 branches the output light of the second light source 34 into multiple pipes and transmits the light to the resin parts of the multiple markers 30. This allows one second light source 34 to light up multiple markers 30, and the number of light sources can be made smaller than the number of markers 30.

[0038] 7 shows an example of a portion of an image captured by the input device 16. As shown in the figure, the captured image includes an image of a marker 30 emitting light. In the HMD 100, the communication control unit 128 transmits image data captured by the imaging device 14 to the information processing device 10, and the information processing device 10 extracts the image of the marker 30 from the image data. In the synchronization process, the information processing device 10 can distinguish whether the marker 30 is emitting light at a first luminance or a second luminance.

[0039] The synchronization process between the exposure period of the imaging device 14 and the illumination period of the marker 30 is performed before the start of the game, but may also be performed while the game is being played. If synchronization is lost, the imaging device 14 will not be able to capture an image of the marker 30, so if synchronization is lost, synchronization process must be performed immediately.

[0040] In the synchronization process, the control unit 50 causes one or more first light sources 32 that can be turned on independently to emit light in a light emission pattern for synchronization process (see FIG. 5) provided by the information processing device 10. Note that all light sources may be turned on in the light emission pattern for synchronization process, but when the light sources are turned on in a second luminance lower than the first luminance, the accuracy of the position and orientation estimation process may decrease depending on the environment. For this reason, it is preferable to use as few light sources as possible for the synchronization process, and the control unit 50 may dynamically select the first light sources 32 of the two markers 30 facing the imaging device 14, for example, and cause them to emit light in the light emission pattern for synchronization process. Note that since the second light source 34 turns on a plurality of markers 30 with the same luminance, it is not preferable for the control unit 50 to use the second light source 34 when dynamically selecting and turning on two markers 30, and therefore, in the embodiment, the control unit 50 causes the first light source 32 to emit light in the light emission pattern for synchronization process.

[0041] 5, the light emission pattern defines a lighting period at a first luminance and a lighting period at a second luminance in a period of multiple frames. The information processing device 10 identifies a pattern of changes in the luminance value of the marker 30 included in multiple consecutive captured images, thereby identifying a time grid number included in an exposure period by the imaging device 14. The exposure period may be set to a length about twice that of the time grid, for example.

[0042] 8 shows the illumination period of the markers 30 set within the exposure period of the imaging device 14. When the information processing device 10 identifies the time grid number, it generates an illumination instruction at the timing of the time grid number and transmits it to the input device 16. In the input device 16, the control unit 50 periodically illuminates all the markers 30 at the time position of grid number 14 based on the illumination instruction. After synchronization is established, the control unit 50 turns on the light source for the length of one time grid (521 μsec) per frame period and turns off the light source for the rest of the period, thereby reducing unnecessary power consumption.

[0043] In the HMD 100, when the communication control unit 128 transmits image data captured by the imaging device 14 to the information processing device 10, the information processing device 10 extracts images of the markers 30 from the image data. Since the three-dimensional shape of the input device 16 and the position coordinates of the multiple markers 30 arranged on its surface are known, the information processing device 10 estimates the position and orientation of the input device 16 relative to the imaging device 14 by solving a PnP (Perspective n-Point) problem from the distribution of the images of the markers 30 in the captured image.

[0044] The present invention has been described above based on the embodiments. The above embodiments are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0045] In the embodiment, the arrangement of the multiple markers 30 in the input device 16 having the operation member 22 has been described, but the device to be tracked does not necessarily have to have the operation member 22. Also, in the embodiment, the imaging device 14 is attached to the HMD 100, but the imaging device 14 may be attached to a position other than the HMD 100 as long as it can capture a marker image. [Explanation of symbols]

[0046] 1 information processing system, 10 information processing device, 14 imaging device, 16 input device, 20 case body, 32 first light source, 34 second light source, 36 light guide tube, 50 control unit.

Claims

1. A device including a case body and a plurality of markers that emit light to the outside of the case body, a first light source for illuminating one marker; a second light source for illuminating the plurality of markers; A control unit that causes one or more of the first light sources to emit light in a predetermined light emission pattern, the predetermined light emission pattern defines a lighting period during which light is emitted at a first luminance and a lighting period during which light is emitted at a second luminance different from the first luminance in a plurality of frame periods; The frame period is the shooting period of the imaging device. A device characterized in that

2. The control unit does not cause the second light source to emit light in the predetermined light emission pattern.

2. The device of claim 1 .

3. The light emission of the first light source in the predetermined light emission pattern is performed to specify a timing of photographing by an imaging device.

2. The device of claim 1 .

4. The control unit causes the first light source of the marker facing the direction of the imaging device to emit light in the predetermined light emission pattern.

2. The device of claim 1 .

5. In the predetermined light emission pattern, a frame period is divided into a plurality of time grids, and the light emission luminance in each time grid is set to either a first luminance or a second luminance.

2. The device of claim 1 .

6. In the predetermined light emission pattern, an order of emission at the first luminance or emission at the second luminance in a plurality of frame periods of one time grid is different from an order of emission at the first luminance or emission at the second luminance in a plurality of frame periods of another time grid.

6. The device according to claim 5 .

Images