Information processing device and representative coordinate derivation method

Abstract

To provide a technique for appropriately detecting a marker image in a captured image.SOLUTION: A captured image acquisition unit 212 obtains image data that is read upside down from an image sensor. A first extraction processing unit 234 extracts a connected component of consecutive pixels, from the image data that is read upside down from the image sensor. A representative coordinate derivation unit 238 derives representative coordinates of a marker image, based on the pixels of the connected component extracted by the first extraction processing unit 234.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] This invention relates to a technique for detecting marker images contained in captured images. [Background technology]

[0002] Patent Document 1 discloses an information processing device that identifies representative coordinates of marker images from an image of a device equipped with multiple markers, and derives positional information and orientation information of the device using the representative coordinates of the marker images. The information processing device disclosed in Patent Document 1 identifies a first bounding box that surrounds a region in the captured image where pixels with a first brightness or higher are continuous, and identifies a second bounding box within the first bounding box that surrounds a region where pixels with a second brightness or higher, which is higher than the first brightness, are continuous, and derives representative coordinates of the marker images based on the pixels in the first bounding box or the second bounding box.

[0003] Patent Document 2 discloses an input device equipped with multiple light-emitting units and multiple operating members. The light-emitting units of the input device are photographed by a camera provided on a head-mounted device, and the position and orientation of the input device are calculated based on the detected position of the light-emitting units. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-181322 [Patent Document 2] International Publication No. 2021 / 240930 [Overview of the project] [Problems that the invention aims to solve]

[0005] In recent years, information processing technologies that track the position and orientation of devices and reflect them in 3D models in VR spaces have become widespread. By linking the movement of player characters and game objects in the game space to changes in the position and orientation of the tracked device, information processing devices enable intuitive operation by the user.

[0006] To estimate the position and orientation of a device, multiple illuminated markers are provided on the device. The information processing device identifies the representative coordinates of the multiple marker images contained in an image of the device and compares them with the 3D coordinates of the multiple markers in a 3D model of the device to estimate the position and orientation of the device in real space. In order to estimate the position and orientation of the device with high accuracy, it is necessary to be able to appropriately detect each marker image in the captured image.

[0007] Therefore, the present invention aims to provide a technology for appropriately detecting marker images in captured images. The device may be an input device having an operating member, but it may also be simply a device that is the target of tracking and does not have an operating member. [Means for solving the problem]

[0008] To solve the above problems, an information processing device according to one aspect of the present invention includes an image acquisition unit that acquires an image of a device equipped with multiple markers, and an estimation processing unit that estimates the position information and orientation information of the device based on the marker images in the acquired image. The estimation processing unit includes a marker image coordinate identification unit that identifies representative coordinates of the marker images from the acquired image, and a position and orientation derivation unit that derives the position information and orientation information of the device using the representative coordinates of the marker images. The image acquisition unit acquires image data read out from an image sensor inverted vertically, and the marker image coordinate identification unit includes an extraction processing unit that extracts connected components of pixels from the image data read out from the image sensor inverted vertically, and a representative coordinate derivation unit that derives representative coordinates of the marker images based on the pixels of the connected components extracted by the extraction processing unit.

[0009] Another aspect of the present invention is a method for deriving representative coordinates, comprising the steps of: acquiring image data read out inverted from an image sensor that has captured a device equipped with a plurality of markers; extracting connected components in which pixels are continuous from the image data; and deriving representative coordinates of the marker image based on the pixels of the extracted connected components.

[0010] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of an information processing system in an embodiment. [Figure 2] This figure shows an example of the external shape of an HMD (Head-Mounted Display). [Figure 3] This diagram shows the functional blocks of the HMD (Head-Mounted Display). [Figure 4] This diagram shows the shape of the input device. [Figure 5] This diagram shows the shape of the input device. [Figure 6] This figure shows some examples of images taken from an input device. [Figure 7] This diagram shows the functional blocks of an input device. [Figure 8] This diagram shows the functional blocks of an information processing device. [Figure 9] This is a flowchart showing the position and orientation estimation process. [Figure 10] This flowchart shows the process of extracting connected components of 8 neighboring pixels from a captured image. [Figure 11] This figure shows an example of a captured frame image. [Figure 12] This diagram illustrates the order in which line data from an image is read. [Figure 13]This is a diagram for explaining pixel connectivity. [Figure 14] This is a diagram showing a plurality of pixels in a captured image. [Figure 15] This is a diagram showing a boundary box surrounding the first connected component. [Figure 16] This is a diagram showing a boundary box surrounding another first connected component. [Figure 17] This is a diagram showing an example of a boundary box extracted from a captured image. [Figure 18] This is a diagram showing an example of erroneously extracting two marker images as one first connected component. [Figure 19] This is a flowchart showing a process of extracting a plurality of second connected components from the first connected component. [Figure 20] This is a diagram showing an example of a captured image including the region of the boundary box. [Figure 21] This is a diagram showing a target region for extracting the second connected component. [Figure 22] This is a diagram showing a boundary box surrounding the second connected component. [Figure 23] This is a diagram showing a flowchart showing a process of deriving representative coordinates. [Figure 24] This is a diagram showing an example of a boundary box extracted from a captured image.

Embodiment for Carrying Out the Invention

[0012] FIG. 1 shows a configuration example of an information processing system 1 in an embodiment. The information processing system 1 includes an information processing apparatus 10, a recording apparatus 11, a head-mounted display (HMD) 100, an input device 16 operated by a user's finger, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing apparatus 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has functions of a wireless access point and a router, and the information processing apparatus 10 may be connected to the AP 17 by a cable or may be connected by a known wireless communication protocol.

[0013] The recording device 11 records system software and applications such as game software. The information processing device 10 may download game software from the content server to the recording device 11 via the network 2. The information processing device 10 executes the game software and supplies game image data and audio data to the HMD 100. The information processing device 10 and the HMD 100 may be connected by a known wireless communication protocol or by a cable.

[0014] The HMD100 is a display device that, when worn on the user's head, displays images on a display panel positioned in front of the user's eyes. The HMD100 separately displays the left eye image on the left eye display panel and the right eye image on the right eye display panel. These images constitute parallax images viewed from the left and right viewpoints, achieving stereoscopic vision. Since the user views the display panel through optical lenses, the information processing device 10 supplies the HMD100 with parallax image data corrected for optical distortion caused by the lenses.

[0015] The user wearing the HMD 100 does not need the output device 15, but by providing the output device 15, another user can view the image displayed on the output device 15. The information processing device 10 may display the same image on the output device 15 as the image seen by the user wearing the HMD 100, or it may display a different image. For example, if the user wearing the HMD and another user are playing a game together, the output device 15 may display the game image from the perspective of the other user's character.

[0016] The information processing device 10 and the input device 16 may be connected by a known wireless communication protocol or by a cable. The input device 16 has multiple operating elements such as operation buttons, and the user operates the operating elements with their fingers while holding the input device 16. When the information processing device 10 runs a game, the input device 16 is used as a game controller. The input device 16 has an inertial measurement unit (IMU) which includes a 3-axis accelerometer and a 3-axis gyroscope, and transmits sensor data to the information processing device 10 at a predetermined period (for example, 800 Hz).

[0017] The game in this embodiment handles not only the operation information of the control elements of the input device 16, but also the position, orientation, and movement of the input device 16 as operation information, and reflects this in the movement of the player character in the virtual 3D space. For example, the operation information of the control elements may be used as information for moving the player character, while the operation information such as the position, orientation, and movement of the input device 16 may be used as information for moving the player character's arms. In combat scenes within the game, the movement of the input device 16 is reflected in the movement of the player character holding a weapon, enabling intuitive operation for the user and enhancing immersion in the game.

[0018] To track the position and orientation of the input device 16, the input device 16 is equipped with multiple markers (light-emitting parts) that can be captured by the imaging device 14 mounted on the HMD 100. The information processing device 10 analyzes the images captured of the input device 16 to estimate the position and orientation information of the input device 16 in real space, and provides the estimated position and orientation information to the game.

[0019] The HMD100 is equipped with multiple imaging devices 14. The multiple imaging devices 14 are mounted in different positions and orientations on the front of the HMD100 so that the combined imaging range of each device covers the entire field of view of the user. The imaging devices 14 are equipped with image sensors capable of acquiring images of multiple markers on the input device 16. For example, if the markers emit visible light, the imaging device 14 has a visible light sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, which are commonly used in digital video cameras. If the markers emit invisible light, the imaging device 14 has an invisible light sensor. The multiple imaging devices 14 capture images in front of the user at a predetermined interval (for example, 120 frames / second) in synchronous timing and transmit the image data captured by the input device 16 to the information processing device 10.

[0020] The information processing device 10 identifies the positions of multiple marker images of the input device 16 included in the captured image. Although one input device 16 may be captured by multiple imaging devices 14 at the same time, since the mounting position and orientation of the imaging devices 14 are known, the information processing device 10 may identify the positions of the marker images by combining the multiple captured images.

[0021] The three-dimensional shape of the input device 16 and the position coordinates of multiple markers placed on its surface are known, and the information processing device 10 estimates the position coordinates and orientation of the input device 16 based on the distribution of marker images in the captured image. The position coordinates of the input device 16 may be position coordinates in a three-dimensional space with a reference position as the origin, and the reference position may be position coordinates (latitude, longitude) set before the start of the game.

[0022] The information processing device 10 in this embodiment has a function to estimate the position coordinates and orientation of the input device 16 using sensor data detected by the orientation sensor of the input device 16. Therefore, the information processing device 10 in this embodiment may perform high-precision tracking of the input device 16 using the estimation results based on the captured image taken by the imaging device 14 and the estimation results based on the sensor data. In this case, the information processing device 10 may apply a state estimation technique using a Kalman filter to integrate the estimation results based on the captured image and the estimation results based on the sensor data, thereby determining the position coordinates and orientation of the input device 16 at the current time with high precision.

[0023] Figure 2 shows an example of the external shape of the HMD100. The HMD100 consists of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that wraps around the head when worn by the user to secure the HMD100 to the head. The mounting band 106 has a material or structure that allows its length to be adjusted to the user's head circumference.

[0024] The output mechanism 102 includes a housing 108 shaped to cover the left and right eyes when the HMD 100 is worn by the user, and inside is a display panel that faces the eyes when worn. The display panel may be an LCD panel or an organic EL panel. Inside the housing 108 is a pair of left and right optical lenses positioned between the display panel and the user's eyes to expand the user's field of view. The HMD 100 may also be equipped with speakers or earphones at positions corresponding to the user's ears, and may be configured to allow connection of external headphones.

[0025] Multiple imaging devices 14a, 14b, 14c, and 14d are provided on the front outer surface of the housing 108. Using the user's face as a reference, imaging device 14a is mounted in the upper right corner of the front outer surface with the camera optical axis pointing diagonally upward to the right; imaging device 14b is mounted in the upper left corner of the front outer surface with the camera optical axis pointing diagonally upward to the left; imaging device 14c is mounted in the lower right corner of the front outer surface with the camera optical axis pointing diagonally downward to the right; and imaging device 14d is mounted in the lower left corner of the front outer surface with the camera optical axis pointing diagonally downward to the left. By installing multiple imaging devices 14 in this manner, the combined imaging range of each device covers the entire user's field of view. This user's field of view may be the user's field of view in a three-dimensional virtual space.

[0026] The HMD100 transmits sensor data detected by the posture sensor and image data captured by the imaging device 14 to the information processing device 10, and also receives game image data and game audio data generated by the information processing device 10.

[0027] Figure 3 shows the functional blocks of the HMD100. The control unit 120 is the main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as commands. The memory unit 122 temporarily stores the data and commands processed by the control unit 120. The attitude sensor 124 acquires sensor data related to the movement of the HMD100. The attitude sensor 124 includes at least three-axis accelerometers and three-axis gyroscopes. The attitude sensor 124 detects the values ​​of each axis component (sensor data) at a predetermined period (for example, 800 Hz).

[0028] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 via a network adapter or antenna, either by wired or wireless communication. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0029] When the control unit 120 receives game image data and game audio data from the information processing device 10, it supplies them to the display panel 130 for display and also supplies them to the audio output unit 132 for audio output. The display panel 130 consists of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images are displayed on each display panel. The control unit 120 also transmits sensor data from the attitude sensor 124, audio data from the microphone 126, and captured image data from the imaging device 14 to the information processing device 10 via the communication control unit 128.

[0030] Figure 4(a) shows the shape of the left-handed input device 16a. The left-handed input device 16a comprises a case body 20, a plurality of operating members 22a, 22b, 22c, and 22d (hereinafter referred to as "operating members 22" unless otherwise specified) operated by the user, and a plurality of markers 30 that emit light to the outside of the case body 20. The markers 30 may have an emission surface with a circular cross-section. The operating members 22 may include an analog stick for tilting operation, a push-button, etc. The case body 20 has a gripping portion 21 and a curved portion 23 connecting the top and bottom of the case body, and the user places their left hand in the curved portion 23 and grips the gripping portion 21. While gripping the gripping portion 21, the user uses their left thumb to operate the operating members 22a, 22b, 22c, and 22d.

[0031] Figure 4(b) shows the shape of the right-handed input device 16b. The right-handed input device 16b comprises a case body 20, a plurality of operating members 22e, 22f, 22g, and 22h (hereinafter referred to as "operating members 22" unless otherwise specified) operated by the user, and a plurality of markers 30 that emit light to the outside of the case body 20. The operating members 22 may include an analog stick for tilting operation, a push-button, etc. The case body 20 has a gripping portion 21 and a curved portion 23 connecting the top and bottom of the case body, and the user places their right hand in the curved portion 23 and grips the gripping portion 21. While gripping the gripping portion 21, the user uses their right thumb to operate the operating members 22e, 22f, 22g, and 22h.

[0032] Figure 5 shows the shape of the input device 16b for the right hand. In addition to the operating members 22e, 22f, 22g, and 22h shown in Figure 4(b), the input device 16b has operating members 22i and 22j. The user holds the gripping part 21 and operates operating member 22i with the index finger of the right hand and operating member 22j with the middle finger. Hereafter, unless otherwise distinguished, the input device 16a and the input device 16b will be referred to as "input device 16".

[0033] The operating members 22 provided on the input device 16 are equipped with a touch-sense function that recognizes a finger simply by touching it, without the need to press it. Regarding the input device 16b for the right hand, the operating members 22f, 22g, and 22j may be equipped with capacitive touch sensors. While the touch sensors may be mounted on other operating members 22, it is preferable that they be mounted on operating members that do not come into contact with the surface on which the input device 16 is placed when it is placed on a table or the like.

[0034] The marker 30 is a light-emitting part that emits light to the outside of the case body 20, and includes a resin part on the surface of the case body 20 that diffuses and emits light to the outside from a light source such as an LED (Light Emitting Diode) element. The marker 30 is photographed by the imaging device 14 and used for estimating the position and orientation of the input device 16. Since the imaging device 14 photographs space at a predetermined period (for example, 120 frames / second), it is preferable that the marker 30 emits light in synchronization with the periodic shooting timing of the imaging device 14 and turns off during non-exposure periods by the imaging device 14 to suppress unnecessary power consumption.

[0035] In this embodiment, the images captured by the imaging device 14 are used for tracking processing of the input device 16 and for tracking processing (SLAM) of the HMD 100. Therefore, images captured at 60 frames per second are used for tracking processing of the input device 16, and other images captured at 60 frames per second may be used for processing that simultaneously performs self-localization and environmental mapping of the HMD 100.

[0036] Figure 6 shows an example of a portion of an image captured of the input device 16. This image is a capture of the input device 16b held in the right hand and includes images of multiple markers 30 that emit light. In the HMD 100, the communication control unit 128 transmits the image data captured by the imaging device 14 to the information processing device 10 at a predetermined interval.

[0037] Figure 7 shows the functional blocks of the input device 16. The control unit 50 receives operation information input to the operating member 22 and also receives sensor data acquired by the attitude sensor 52. The attitude sensor 52 acquires sensor data related to the movement of the input device 16 and includes at least a 3-axis accelerometer and a 3-axis gyroscope. The attitude sensor 52 detects the values ​​(sensor data) of each axis component at a predetermined period (e.g., 800 Hz). The control unit 50 supplies the received operation information and sensor data to the communication control unit 54. The communication control unit 54 transmits the operation information and sensor data output from the control unit 50 to the information processing device 10 via a network adapter or antenna, either by wired or wireless communication. The communication control unit 54 also receives light emission instructions from the information processing device 10.

[0038] The input device 16 includes multiple light sources 58 for illuminating multiple markers 30. The light sources 58 may be LED elements that emit light in a predetermined color. The control unit 50 illuminates the light sources 58 and lights up the markers 30 based on the light emission instructions obtained from the information processing device 10. In the example shown in Figure 7, one light source 58 is provided for each marker 30, but one light source 58 may light up multiple markers 30.

[0039] Figure 8 shows the functional blocks of the information processing device 10. The information processing device 10 comprises a processing unit 200 and a communication unit 202. The processing unit 200 comprises an acquisition unit 210, a game execution unit 220, an image signal processing unit 222, an estimation processing unit 230, and a marker information holding unit 250. The communication unit 202 receives operation information and sensor data of the operating member 22 transmitted from the input device 16 and supplies it to the acquisition unit 210. The communication unit 202 also receives captured image data and sensor data transmitted from the HMD 100 and supplies it to the acquisition unit 210.

[0040] The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216. The estimation processing unit 230 includes a marker image coordinate identification unit 232, a marker image coordinate extraction unit 240, and a position and orientation derivation unit 242. The marker image coordinate identification unit 232 includes a first extraction processing unit 234, a second extraction processing unit 236, and a representative coordinate derivation unit 238. The estimation processing unit 230 estimates the position and orientation information of the input device 16 based on the marker image contained in the captured image. In this embodiment, although not described, the estimation processing unit 230 may estimate the position and orientation information of the input device 16 with high accuracy by inputting the position and orientation information of the input device 16 estimated from the marker image contained in the captured image and the position and orientation information of the input device 16 estimated from the sensor data detected by the input device 16 into a Kalman filter. The estimation processing unit 230 supplies the estimated position information and orientation information of the input device 16 to the game execution unit 220.

[0041] The information processing device 10 includes a computer, and various functions shown in Figure 8 are realized by the computer executing a program. The computer includes, as hardware, a memory for loading the program, one or more processors for executing the loaded program, auxiliary storage devices, and other LSIs. The processor is composed of multiple electronic circuits, including semiconductor integrated circuits and LSIs, and these multiple electronic circuits may be mounted on one chip or on multiple chips. The functional blocks shown in Figure 8 are realized through the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof.

[0042] The image acquisition unit 212 acquires image data from the input device 16 equipped with multiple markers 30 and supplies it to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data and supplies the high-quality captured image data to the estimation processing unit 230.

[0043] The image acquisition unit 212 supplies horizontal line data of the image to the image signal processing unit 222 one line at a time. In this embodiment, the image signal processing unit 222 is hardware-based and stores several lines of image data in a line buffer. It performs image quality enhancement processing on the several lines of image data stored in the line buffer and supplies the enhanced line data to the estimation processing unit 230.

[0044] The sensor data acquisition unit 214 acquires sensor data transmitted from the input device 16 and the HMD 100 and supplies it to the estimation processing unit 230. The operation information acquisition unit 216 acquires operation information transmitted from the input device 16 and supplies it to the game execution unit 220. The game execution unit 220 proceeds with the game based on the operation information and the position and orientation information of the input device 16.

[0045] The marker image coordinate identification unit 232 identifies a two-dimensional coordinate (hereinafter also referred to as "marker image coordinate") that represents the image of the marker 30 included in the captured image. The marker image coordinate identification unit 232 may identify a region in which pixels with a brightness value of a predetermined value or higher are continuous, calculate the centroid coordinate of that pixel region, and use it as the representative coordinate of the marker image. The method for deriving the representative coordinate by the marker image coordinate identification unit 232 will be described later.

[0046] A method for estimating the position and orientation of an imaging device that captured an image of an object whose three-dimensional shape and size are known is known by solving the PNP (Perspective n-Point) problem. In this embodiment, the marker image coordinate extraction unit 240 extracts N (N is an integer of 3 or more) two-dimensional marker image coordinates from the captured image, and the position and orientation derivation unit 242 derives position and orientation information of the input device 16 from the N marker image coordinates extracted by the marker image coordinate extraction unit 240 and the three-dimensional coordinates of the N markers in the three-dimensional model of the input device 16. The position and orientation derivation unit 242 estimates the position and orientation of the imaging device 14 using the following (Equation 1), and derives position and orientation information of the input device 16 in three-dimensional space based on the estimation result.

number

[0047] Here, (u,v) are the marker image coordinates in the captured image, and (X,Y,Z) are the position coordinates of the marker 30 in 3D space when the 3D model of the input device 16 is in the reference position and reference orientation. The 3D model is a model that has the exact same shape and size as the input device 16, with the markers placed in the same position. The marker information holding unit 250 holds the 3D coordinates of each marker in the 3D model in the reference position and reference orientation, and the position orientation derivation unit 242 reads the 3D coordinates of each marker from the marker information holding unit 250 to obtain (X,Y,Z).

[0048] (f x ,f y ) is the focal length of the imaging device 14, (cx , c y ) is the principal point of the image, and both are internal parameters of the imaging device 14. r 11 ~r 33 , the matrix with elements t1~t3 is a rotation and translation matrix. In (Equation 1), (u, v), (f x , f y ), (c x , c y ), (X, Y, Z) are known, and the position and orientation derivation unit 242 obtains a common rotation and translation matrix for them by solving equations for N markers 30. The position and orientation derivation unit 242 derives the position information and orientation information of the input device 16 based on the angle and translation amount represented by this matrix. In the embodiment, the process of estimating the position and orientation of the input device 16 is implemented by solving the P3P problem. Therefore, the position and orientation derivation unit 242 uses the three marker image coordinates and the three three-dimensional marker coordinates in the three-dimensional model of the input device 16 to derive the position and orientation of the input device 16. The information processing device 10 generates the world coordinates of the three-dimensional real space by the SLAM technology. Therefore, the position and orientation derivation unit 242 derives the position and orientation of the input device 16 in the world coordinate system.

[0049] FIG. 9 is a flowchart showing the position and orientation estimation process by the estimation processing unit 230. The captured image acquisition unit 212 sequentially acquires the line data of the image captured by the input device 16 (S10) and supplies it to the image signal processing unit 222. In order to reduce the computational load of the position and orientation estimation process, the captured image acquisition unit 212 may perform binning processing (processing of combining 4 pixels into 1 pixel) on the two acquired line data and supply it to the image signal processing unit 222. The image signal processing unit 222 stores the line data for several lines in the line buffer and performs image signal processing such as noise reduction and optical correction (S12). The image signal processing unit 222 supplies the line data processed by the image signal to the marker image coordinate identification unit 232, and the marker image coordinate identification unit 232 identifies the representative coordinates of a plurality of marker images included in the captured image (S14). The line data processed by the image signal and the representative coordinates of the identified marker images are temporarily stored in a memory (not shown).

[0050] The marker image coordinate extraction unit 240 extracts any three marker image coordinates from among the multiple marker image coordinates identified by the marker image coordinate identification unit 232. The marker information holding unit 250 holds the three-dimensional coordinates of each marker in the three-dimensional model of the input device 16 at the reference position and reference orientation. The position and orientation derivation unit 242 reads the three-dimensional coordinates of the markers in the three-dimensional model from the marker information holding unit 250 and solves the P3P problem using (Equation 1). Once the position and orientation derivation unit 242 identifies the rotation-translation sequence common to the three extracted marker image coordinates, it calculates the reprojection error using the marker image coordinates of the input device 16 other than the three extracted marker image coordinates.

[0051] The marker image coordinate extraction unit 240 extracts a predetermined number of combinations of three marker image coordinates. The position and orientation derivation unit 242 identifies a rotation / translation sequence for each of the extracted combinations of three marker image coordinates and calculates the reprojection error for each. Then, the position and orientation derivation unit 242 identifies the rotation / translation sequence that results in the smallest reprojection error from the predetermined number of reprojection errors and derives the position and orientation information of the input device 16 (S16). The position and orientation derivation unit 242 supplies the derived position and orientation information of the input device 16 to the game execution unit 220.

[0052] The position and orientation estimation process is performed at the image acquisition cycle (60 frames / second) of the tracking image of the input device 16 (N in S18). When the game execution unit 220 ends the game, the position and orientation estimation process by the estimation processing unit 230 ends (Y in S18).

[0053] The following describes, using several flowcharts, the method by which the marker image coordinate identification unit 232 derives the representative coordinates of the marker image. The captured image in the embodiment is a grayscale image, and the brightness of each pixel is represented by 8 bits, taking brightness values ​​from 0 to 255. In the captured image, the marker image is captured as a high-brightness image, as shown in Figure 6.

[0054] Figure 10 is a flowchart showing the process by which the first extraction processing unit 234 extracts connected components of eight neighboring pixels from a captured image. The first extraction processing unit 234 obtains image signal processed line data from the image signal processing unit 222 (S20). The first extraction processing unit 234 performs the process of extracting connected components of eight neighboring pixels from the captured image (S22).

[0055] Figure 11 shows an example of a captured frame image. The high-luminosity object included in the lower part of the image is a light-emitting marker 30. The image signal processing unit 222 supplies the horizontal line data of the frame image to the first extraction processing unit 234 in order from the top vertically. The line data supplied from the image signal processing unit 222 may be sequentially stored in memory (not shown).

[0056] Figure 12 is a diagram illustrating the order in which line data from an image is read. The first extraction processing unit 234 receives the horizontal line data of the frame image from top to bottom and performs the process of extracting connected components of eight neighboring pixels.

[0057] Figure 13(a) is a diagram illustrating the 8-neighbor pixels. In the CCL (Connected-component labeling) algorithm, pixels surrounding a single pixel P (in the up, down, left, right, and four diagonal directions) are called "8-neighbor pixels." In a binary image, when two pixels with the same value are 8 neighbors of each other, these two pixels are said to be "8-adjacent," and in this embodiment, the set of multiple pixels connected by 8-adjacentness is called the "first connected component." The first extraction processing unit 234 is hardware-based and, when two or three line data are input from the image signal processing unit 222, performs the process of extracting the connected component of the 8-neighbor pixels.

[0058] On the other hand, as will be described later, the second extraction processing unit 236 of the embodiment performs a process to extract connected components of four neighboring pixels by software calculation. Figure 13(b) is a diagram illustrating 4-neighbor pixels. Pixels located above, below, left, and right of a single pixel P are called "4-neighbor pixels." 4-neighbor pixels do not include pixels located diagonally. In a binary image, when two pixels with the same value are 4 neighbors of each other, these two pixels are said to be "4-adjacent." In this embodiment, a set of multiple pixels connected by 4-adjacentness is called a "second connected component." The processing function of the second extraction processing unit 236 is realized by software calculations using a DSP. In this embodiment, the second extraction processing unit 236 extracts the connected components of 4-neighbor pixels from the connected components extracted by the first extraction processing unit 234.

[0059] When extracting connected components of 8 neighboring pixels and connected components of 4 neighboring pixels independently from the same frame image, the size of the connected components of 8 neighboring pixels will be greater than or equal to the size of the connected components of 4 neighboring pixels because the connected components of 8 neighboring pixels include pixels connected diagonally. Therefore, the number of connected components of 8 neighboring pixels extracted will be less than or equal to the number of connected components of 4 neighboring pixels extracted.

[0060] Returning to Figure 10, the extraction process (S22) of the first connected component of eight neighboring pixels by the first extraction processing unit 234 will be explained. The first extraction processing unit 234 searches for a region in the captured image where pixels with a first brightness or higher are connected in eight neighboring areas. For example, the first brightness may be a brightness value of 128. By the first extraction processing unit 234 extracting the connected component of eight neighboring pixels, the number of connected components to be extracted can be reduced compared to the case where the connected component of four neighboring pixels is extracted, thereby reducing the load on the subsequent marker image representative coordinate derivation process.

[0061] Figure 14 shows an example of multiple pixels in a captured image. In an actual captured grayscale image, pixels with the highest brightness value of 255 are represented as white, and pixels with the lowest brightness value of 0 are represented as black. However, in Figures 14 to 16 and 20 to 22 below, the brightness representation of each pixel is inverted (black and white are reversed) for the sake of readability. Therefore, in Figures 14 to 16 and 20 to 22, black represents a brightness value of 255 (highest brightness value), and white represents a brightness value of 0 (lowest brightness value). When the first extraction processing unit 234 finds a region where pixels with a brightness of 1 or higher are connected in a neighborhood of 8, it extracts it as the first connected component of the 8 neighboring pixels (S22), and identifies the bounding box surrounding the first connected component (S24).

[0062] Figure 15 shows the bounding box 80a surrounding the first connected component 78a of the extracted 8 neighboring pixels. The bounding box 80a is identified as the smallest rectangle surrounding the first connected component 78a of the 8 neighboring pixels. Note that the first extraction processing unit 234 performs the extraction of the first connected component for each line data of the image, so when it extracts the first connected component 78a, it is not aware of the existence of another first connected component shown below it. When the first extraction processing unit 234 identifies the bounding box 80a, it outputs the coordinate information (bounding box information) of the bounding box 80a to memory (not shown) and stores it s(S26).

[0063] Here, the first extraction processing unit 234 determines whether the number of extracted first connected components is within a predetermined upper limit (S28). For example, the upper limit may be set to 256. In this embodiment, the position and orientation estimation process is performed at the image acquisition cycle (60 frames / second) of the tracking image of the input device 16. If the number of extracted first connected components becomes enormous, it becomes difficult to complete the position and orientation estimation process within the imaging cycle. Therefore, an upper limit is set for the number of first connected components extracted by the first extraction processing unit 234, and if the number of extracted first connected components exceeds the upper limit (N in S28), the first extraction processing unit 234 forcibly terminates the extraction process of the first connected components.

[0064] If the number of extracted first linked components is within a predetermined upper limit (Y in S28), steps S20 to S26 are repeated until processing of one frame of the captured image is completed (N in S30). Figure 16 shows the bounding box 80b surrounding another first connected component 78b extracted in S22. The bounding box 80b is identified as the smallest rectangle surrounding the first connected component 78b of the 8 neighboring pixels. The first extraction processing unit 234 outputs the coordinate information of the bounding box 80b to memory. When processing of one frame of the captured image is completed (Y in S30), the first extraction processing unit 234 starts processing the next frame image.

[0065] Figure 17 shows an example of a bounding box extracted from a captured image. The first extraction processing unit 234 extracts multiple first connected components of eight neighboring pixels from the captured image and outputs information about the bounding box surrounding each of the multiple first connected components to memory for storage. In the example shown in Figure 17, the bounding box of the marker image is identified at the bottom of the captured image, and the bounding box of the light source image, such as illumination light, is identified at the top of the captured image.

[0066] In the example shown in Figure 17, the user is operating the input device 16 at a position close to the HMD 100, so a bounding box surrounding a large marker image is identified at the bottom of the captured image. However, if the user operates the input device 16 with their arm fully extended forward, for example, the distance between the input device 16 and the imaging device 14 increases, resulting in a smaller captured marker image. If multiple small marker images are close together, the first extraction processing unit 234 may mistakenly extract multiple marker images as a single first connected component.

[0067] Figure 18 shows an example where two marker images were mistakenly extracted as a single first connected component. In the example shown in Figure 18, the two small marker images are connected in the vicinity of 8, so the first extraction processing unit 234 extracts the two marker images as a single first connected component and identifies the bounding box surrounding the two marker images. The second extraction processing unit 236 in this embodiment has a function to separate the multiple marker images contained within the bounding box identified by the first extraction processing unit 234.

[0068] Figure 19 is a flowchart showing the process by which the second extraction processing unit 236 extracts the second connected components of multiple four-neighbor pixels from the first connected component contained in the bounding box. The second extraction processing unit 236 investigates whether the first connected component extracted by the first extraction processing unit 234 can be separated into the second connected components of multiple four-neighbor pixels. If it can be separated, the original first connected component is discarded and replaced with the separated multiple second connected components. If it cannot be separated, the original first connected component is retained.

[0069] The second extraction processing unit 236 obtains the bounding box information (coordinate information) identified by the first extraction processing unit 234 from memory (S40). At this time, the second extraction processing unit 236 also obtains the captured image data including the bounding box and its surroundings from the memory where the captured image data is stored (S42).

[0070] Figure 20 shows an example of a captured image that includes the region of the bounding box 80a. The width and height of the captured image region to be acquired are set to approximately twice the width and height of the bounding box 80a, and the center position of the image region is set to approximately coincide with the center position of the bounding box 80a. The second extraction processing unit 236 checks the contrast between the bounding box 80a identified by the first extraction processing unit 234 and its surroundings (S44). If the bounding box 80a contains a mark image, the average brightness inside the bounding box 80a will be high, while the average brightness outside the bounding box 80a will be relatively low. Therefore, the second extraction processing unit 236 calculates the average brightness inside the bounding box 80a and the average brightness in the region outside the bounding box 80a of the acquired image region, and determines the brightness ratio.

[0071] The second extraction processing unit 236 calculates the average brightness B1 of the pixels within the bounding box 80a and the average brightness B2 of the pixels in the image area outside the bounding box 80a. If the brightness ratio (B1 / B2) is less than a predetermined value (N in S44), the second extraction processing unit 236 determines that the first connected component included in the bounding box 80a is not to be separated and stops the separation process of the first connected component. The predetermined value may be, for example, 3. At this time, the second extraction processing unit 236 may determine that the bounding box 80a does not contain a marker image and discard the bounding box 80a.

[0072] If the luminance ratio is greater than or equal to a predetermined value (Y in S44), the second extraction processing unit 236 checks whether the size and shape of the bounding box 80a satisfy predetermined conditions (S46). Specifically, the second extraction processing unit 236 determines whether the number of pixels x in the horizontal direction and the number of pixels y in the vertical direction satisfy the following conditions 1 to 4. (Condition 1) Xmin ≦ x ≦ Xmax (Condition 2) Ymin ≦ y ≦ Ymax (Condition 3) x / y ≦ Aspect_Thresh (Condition 4) y / x ≦ Aspect_Thresh

[0073] Conditions 1 and 2 specify that the size of the bounding box 80a is within a predetermined range, that is, that the bounding box 80a is neither too large nor too small. When multiple marker images are mistakenly extracted as a single first connected component, each marker image is always small (if each marker image is large, multiple marker images will not be extracted as a single first connected component), so the bounding box 80a is investigated if the number of pixels x and y are less than or equal to Xmax and Ymax, respectively. Also, if the bounding box 80a is too small, the possibility of it containing a marker image is low, so the bounding box 80a is investigated if the number of pixels x and y are greater than or equal to Xmin and Ymin, respectively. Conditions 3 and 4 are conditions to exclude elongated bounding boxes 80a from the investigation. When the second extraction processing unit 236 determines that the size and shape of the bounding box 80a do not satisfy any of conditions 1 to 4 (N in S46), it determines that the first connected component contained in the bounding box 80a is not subject to separation and terminates the separation process for the first connected component.

[0074] When the second extraction processing unit 236 determines that the size and shape of the bounding box 80a satisfy all of conditions 1 to 4 (Y in S46), it performs a process to separate the first connected component contained in the bounding box 80a. Specifically, the second extraction processing unit 236 searches for regions connected by four neighbors from the first connected component and extracts the second connected component of the pixels in the four neighboring regions.

[0075] Figure 21 shows the target region from which the second connected component of four neighboring pixels is extracted. This target region is the area obtained by extending the bounding box 80a by one pixel on both the horizontal and vertical sides. In the second connected component extraction process, the second extraction processing unit 236 searches for a region in which pixels with a second brightness or higher are connected in four neighboring areas. The second brightness may be the same as the first brightness, but may be higher than the first brightness; for example, the second brightness may have a brightness value of 160.

[0076] The second extraction processing unit 236, upon finding a region where pixels with a brightness of 2 or higher are connected in a 4-neighborhood network, extracts it as the second connected component of the 4 neighboring pixels (S48) and identifies a bounding box surrounding the second connected component (S50). If the second extraction processing unit 236 does not extract multiple second connected components from the first connected component (N in S52), it determines that the first connected component contained in the bounding box 80a is not to be separated and cancels the separation process for the first connected component. On the other hand, if the second extraction processing unit 236 extracts multiple second connected components from the first connected component (Y in S52), it separates the first connected component 78a contained in the bounding box 80a into multiple second connected components (S54).

[0077] Figure 22 shows the bounding boxes surrounding the second connected components of the extracted four neighboring pixels. In this example, the second extraction processing unit 236 extracts three second connected components 82a, 82b, and 82c from the target region shown in Figure 21 and identifies the bounding boxes 84a, 84b, and 84c surrounding each second connected component. In Figure 22, the second extraction processing unit 236 assigns label value 1 to the second connected component 82a, label value 2 to the second connected component 82b, and label value 3 to the second connected component 82c according to the CCL algorithm. Here, the second connected component 82c, to which label value 3 is assigned, includes pixels outside the bounding box 80a. Therefore, the second extraction processing unit 236 recognizes that the second connected component 82c is not separated from the first connected component 78a and excludes it from processing.

[0078] In this example, the first connected component 78a, which was connected in 8 neighbors, is separated into the second connected component 82a and the second connected component 82b, which are connected in 4 neighbors. The second extraction processing unit 236 replaces the first connected component 78a extracted by the first extraction processing unit 234 with the second connected component 82a and the second connected component 82b if the second connected component 82a and the second connected component 82b satisfy predetermined conditions. Specifically, the second extraction processing unit 236 may discard the first connected component 78a and replace it with the second connected component 82a and the second connected component 82b, provided that the number of pixels in each of the second connected component 82a and the second connected component 82b is greater than or equal to a predetermined value. This process makes it possible to separate the two marker images that were mistakenly extracted as a single first connected component 78a. Furthermore, if the first linked component 78a is separated into a predetermined number (for example, 3 or 4) or more, the second extraction processing unit 236 may determine that the separation process is inappropriate and retain the first linked component 78a.

[0079] The second extraction processing unit 236 investigates whether all bounding boxes identified by the first extraction processing unit 234 contain a separable first connected component (N in S56). When the second extraction processing unit 236 has finished investigating all bounding boxes (Y in S56), the representative coordinate derivation unit 238 performs a process to derive representative coordinates of the marker image based on the pixels of the first connected component extracted by the first extraction processing unit 234 and / or the pixels of the second connected component extracted by the second extraction processing unit 236.

[0080] Figure 23 shows a flowchart illustrating the process of deriving representative coordinates. The representative coordinate derivation unit 238 derives representative coordinates of the marker image using the bounding box identified by the first extraction processing unit 234 and the bounding box identified by the second extraction processing unit 236. In this embodiment, the representative coordinate derivation unit 238 checks whether the marker image is included in the bounding boxes identified by the first extraction processing unit 234 and the second extraction processing unit 236 by comparing them against several criteria. First, the representative coordinate derivation unit 238 acquires bounding box information (S60) and checks whether the size of the bounding box is within a predetermined range (S62). If the bounding box is too large (N in S62), the first or second connected component included in the bounding box is not an image of the marker 30. Therefore, the representative coordinate derivation unit 238 discards the excessively large bounding box.

[0081] If the size of the bounding box is within a predetermined range (Y in S62), the second extraction processing unit 236 checks whether the shape of the connected component of the high-luminance pixels contained within the bounding box is elongated (S64). Since the marker 30 has an output surface with a circular cross-section, the marker image has a shape close to a circle and will not be elongated. If the shape of the connected component of the high-luminance pixels is elongated (Y in S64), the high-luminance illuminated object contained within the bounding box is not the marker 30, so the representative coordinate derivation unit 238 discards the elongated bounding box.

[0082] If the shape of the connected portion of the high-luminance pixels is not elongated (N in S64), the representative coordinate derivation unit 238 checks the contrast between the identified boundary box and its surroundings (S66). This contrast check process may be the same as the process shown in S44 of Figure 19, for example. If the ratio of the average luminance within the boundary box to the average luminance in a predetermined area outside the boundary box is less than a predetermined value (N in S66), the representative coordinate derivation unit 238 discards the boundary box.

[0083] If the luminance ratio is greater than or equal to a predetermined value (Y in S66), the representative coordinate derivation unit 238 recognizes that the marker image is contained within the boundary box and derives the representative coordinates of the marker image based on the pixels with a third luminance or higher within the boundary box (S68). These representative coordinates may be centroid coordinates. The third luminance is lower than the first luminance, for example, a luminance value of 64. The representative coordinate derivation unit 238 calculates the average luminance position in the X-axis and Y-axis directions and derives the representative coordinates (u,v). At this time, it is preferable that the representative coordinate derivation unit 238 takes into account the pixel values ​​of each pixel with a third luminance or higher to determine the luminance centroid position and derives the representative coordinates (u,v).

[0084] In the above-described embodiment, it was explained that, in relation to S28 in Figure 10, an upper limit is set on the number of first linked components that the first extraction processing unit 234 can extract. The first extraction processing unit 234 forcibly terminates the extraction process of first linked components when the number of extracted first linked components reaches the upper limit, but the second extraction processing unit 236 may perform the above-described separation process on the extracted number of first linked components up to the upper limit.

[0085] Figure 24 shows an example of a bounding box extracted by the first extraction processing unit 234 in the captured image. This image includes a blind that is installed on the inside of a window for purposes such as sun protection or privacy. The blind captured here is a Venetian blind, which has multiple horizontal slats arranged in the vertical direction and is a type of blind commonly used in offices.

[0086] The first extraction processing unit 234 in this embodiment is composed of hardware that sequentially acquires line data from an image and extracts the first connected component of eight neighboring pixels. The arrows in Figure 24 indicate the order in which line data from the image sensor of the imaging device 14 is read, and the first extraction processing unit 234 performs the extraction of the first connected component based on the read line data. In the example shown in Figure 24, the first extraction processing unit 234 performs the extraction of the first connected component sequentially from the top to the bottom of the captured image, but before processing all the image data is completed, the number of extracted first connected components reaches the upper limit (256), and the extraction of the first connected component is forcibly terminated. As shown in the captured image in Figure 24, the marker image of the marker 30 of the input device 16 is present in the lower left of the image, but the marker image is not extracted because the number of extracted first connected components has reached the upper limit.

[0087] As shown in Figure 17, the input device 16 is captured by the image sensor of the imaging device 14 mounted on the HMD 100. Therefore, when the user is playing a game normally, the input device 16 is captured on the lower side of the field of view. In this case, the control unit 120 in the HMD 100 may read the image data from the image sensor of the imaging device 14 inverted vertically, and transmit the read image data from the communication control unit 128 to the information processing device 10.

[0088] In the information processing device 10, the captured image acquisition unit 212 acquires image data read from the image sensor inverted vertically. Therefore, the captured image acquisition unit 212 acquires line data of the captured image sequentially from the bottom of the image and supplies it to the estimation processing unit 230 via the image signal processing unit 222. As a result, the first extraction processing unit 234 can extract a first connected component consisting of consecutive pixels with a predetermined brightness or higher from the image data read from the image sensor inverted vertically, and can increase the possibility of extracting a first connected component corresponding to a marker image located at the bottom of the captured image before the number of extracted first connected components reaches the upper limit.

[0089] The present invention has been described above based on examples. The above examples are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their respective components and processing processes, and that such modifications also fall within the scope of the present invention. In the examples, the estimation process was performed by the information processing device 10, but the functions of the information processing device 10 may be provided in the HMD 100, and the HMD 100 may perform the estimation process. In other words, the HMD 100 may be the information processing device 10.

[0090] In this embodiment, the arrangement of multiple markers 30 in an input device 16 equipped with an operating member 22 has been described, but the device to be tracked does not necessarily have to be equipped with an operating member 22. In this embodiment, the imaging device 14 is attached to the HMD 100, but the imaging device 14 only needs to be able to capture marker images and may be attached to a location other than the HMD 100. [Explanation of symbols]

[0091] 1... Information processing system, 10... Information processing device, 14... Imaging device, 16a, 16b... Input devices, 20... Case body, 21... Gripping part, 22... Operating member, 23... Curved part, 30... Marker, 50... Control unit, 52... Attitude sensor, 54... Communication control unit, 58... Light source, 100... HMD, 102... Output mechanism, 104... Mounting mechanism, 106... Mounting band, 108... Housing, 120... Control unit, 122... Memory unit, 124... Attitude sensor, 126... Microphone, 128... Communication Control unit, 130...Display panel, 132...Audio output unit, 200...Processing unit, 202...Communication unit, 210...Acquisition unit, 212...Captured image acquisition unit, 214...Sensor data acquisition unit, 216...Operation information acquisition unit, 220...Game execution unit, 222...Image signal processing unit, 230...Estimation processing unit, 232...Marker image coordinate identification unit, 234...First extraction processing unit, 236...Second extraction processing unit, 238...Representative coordinate derivation unit, 240...Marker image coordinate extraction unit, 242...Position and orientation derivation unit, 250...Marker information holding unit.

Claims

1. a captured image acquisition unit that acquires an image of a device having a plurality of markers; an estimation processing unit that estimates position information and orientation information of the device based on a marker image in a captured image, The estimation processing unit: a marker image coordinate identifying unit that identifies representative coordinates of a marker image from the captured image; a position and orientation derivation unit that derives position information and orientation information of the device by using representative coordinates of the marker image; the captured image acquisition unit acquires image data that is read out from an image sensor in a vertically inverted state; The marker image coordinate identifying unit is an extraction processing unit that extracts connected components of consecutive pixels within a range of a predetermined upper limit number from image data that has been read out from the image sensor after being upside down; a representative coordinate derivation unit that derives representative coordinates of a marker image based on pixels of the connected components extracted by the extraction processing unit, When the number of extracted connected components reaches a predetermined upper limit, the extraction processing unit terminates the extraction of connected components.

23. An information processing apparatus comprising:

2. the extraction processing unit is configured with hardware and extracts the connected components from line data of an image read out from the image sensor in a vertically inverted state; 2. The information processing apparatus according to claim 1,

3. The image sensor is mounted on a head mounted display.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. acquiring image data read out in a vertically inverted state from an image sensor that has captured an image of a device having a plurality of markers; extracting connected components of consecutive pixels within a range of a predetermined upper limit number from the image data; deriving representative coordinates of the marker image based on the pixels of the extracted connected components; terminating the extraction process of connected components when the number of extracted connected components reaches a predetermined upper limit; A representative coordinate deriving method comprising the steps of:

5. On the computer, A function for acquiring image data read out upside down from an image sensor that captures an image of a device equipped with multiple markers; A function of extracting connected components of consecutive pixels within a range of a predetermined upper limit number from the image data; A function of deriving representative coordinates of a marker image based on the pixels of the extracted connected components; a function of terminating the extraction process of connected components when the number of extracted connected components reaches a predetermined upper limit; A program to achieve this.

Images