Information processing device and representative coordinate derivation method

Abstract

To provide a technique for appropriately detecting a marker image in a captured image.SOLUTION: A captured image acquisition unit 212 obtains image data that is read upside down from an image sensor. A first extraction processing unit 234 extracts a connected component of consecutive pixels, from the image data that is read upside down from the image sensor. A representative coordinate derivation unit 238 derives representative coordinates of a marker image, based on the pixels of the connected component extracted by the first extraction processing unit 234.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting a marker image included in a captured image. [Background technology]

[0002] Patent Document 1 discloses an information processing device that identifies representative coordinates of marker images from a captured image of a device equipped with multiple markers and derives position information and orientation information of the device using the representative coordinates of the marker images. The information processing device disclosed in Patent Document 1 identifies a first bounding box that encloses an area in the captured image where pixels with a first brightness or higher are consecutive, and identifies a second bounding box that encloses an area within the first bounding box where pixels with a second brightness or higher that is higher than the first brightness are consecutive, and derives representative coordinates of the marker images based on the pixels in the first bounding box or the second bounding box.

[0003] Patent Document 2 discloses an input device equipped with multiple light-emitting units and multiple operating members. The light-emitting units of the input device are photographed by a camera mounted on a head-mounted device, and the position and orientation of the input device are calculated based on the detected positions of the light-emitting units. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-181322 [Patent Document 2] International Publication No. 2021 / 240930 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, information processing technology that tracks the position and orientation of a device and reflects it in a 3D model in a VR space has become widespread. By linking the movement of a player character or game object in a game space to changes in the position and orientation of the device being tracked, an intuitive operation by the user is realized.

[0006] To estimate the device's position and orientation, multiple illuminated markers are provided on the device, and the information processing device identifies the representative coordinates of the multiple marker images included in a captured image of the device and compares them with the 3D coordinates of the multiple markers in a 3D model of the device, thereby estimating the device's position and orientation in real space. In order to estimate the device's position and orientation with high accuracy, it is necessary to be able to properly detect each marker image in the captured image.

[0007] Therefore, an object of the present invention is to provide a technique for appropriately detecting a marker image in a captured image. The device may be an input device having an operation member, or may be a device that does not have an operation member and is simply a target for tracking. [Means for solving the problem]

[0008] To solve the above problem, an information processing device according to one aspect of the present invention includes a captured image acquisition unit that acquires a captured image of a device equipped with multiple markers, and an estimation processing unit that estimates position information and orientation information of the device based on the marker images in the captured image. The estimation processing unit includes a marker image coordinate identification unit that identifies representative coordinates of the marker images from the captured image, and a position and orientation derivation unit that derives position information and orientation information of the device using the representative coordinates of the marker images. The captured image acquisition unit acquires image data that has been read out from the image sensor after being flipped upside down, and the marker image coordinate identification unit includes an extraction processing unit that extracts connected components of consecutive pixels from the image data that has been read out from the image sensor after being flipped upside down, and a representative coordinate derivation unit that derives representative coordinates of the marker images based on the pixels of the connected components extracted by the extraction processing unit.

[0009] Another aspect of the present invention is a method for deriving representative coordinates, which includes the steps of acquiring image data that has been read upside down from an image sensor that has photographed a device equipped with multiple markers, extracting connected components of consecutive pixels from the image data, and deriving representative coordinates of the marker image based on the pixels of the extracted connected components.

[0010] In addition, any combination of the above components, and conversions of the present invention between methods, devices, systems, computer programs, recording media on which computer programs are readably recorded, data structures, etc. are also valid aspects of the present invention. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating examples of the external shape of an HMD. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of an HMD. [Figure 4] FIG. 2 is a diagram illustrating the shape of an input device. [Figure 5] FIG. 2 is a diagram illustrating the shape of an input device. [Figure 6] FIG. 10 is a diagram showing an example of a portion of an image captured using an input device. [Figure 7] FIG. 2 is a diagram illustrating functional blocks of an input device. [Figure 8] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 9] 10 is a flowchart illustrating a position and orientation estimation process. [Figure 10] 10 is a flowchart showing a process for extracting connected components of eight neighboring pixels from a captured image. [Figure 11] FIG. 10 is a diagram showing an example of a captured frame image. [Figure 12] FIG. 10 is a diagram for explaining the order in which line data of an image is read out; [Figure 13]FIG. 1 is a diagram for explaining pixel connectivity. [Figure 14] FIG. 2 is a diagram showing a plurality of pixels in a captured image. [Figure 15] FIG. 10 shows a bounding box surrounding a first connected component. [Figure 16] FIG. 10 shows a bounding box surrounding another first connected component. [Figure 17] FIG. 10 is a diagram showing an example of a bounding box extracted from a captured image. [Figure 18] FIG. 10 is a diagram showing an example in which two marker images are erroneously extracted as one first connected component. [Figure 19] 10 is a flowchart showing a process of extracting a plurality of second connected components from a first connected component. [Figure 20] FIG. 10 is a diagram illustrating an example of a captured image including a bounding box region. [Figure 21] FIG. 10 is a diagram showing a target region from which a second connected component is extracted. [Figure 22] FIG. 10 illustrates a bounding box surrounding a second connected component. [Figure 23] FIG. 10 is a flowchart showing a process of deriving representative coordinates. [Figure 24] FIG. 10 is a diagram showing an example of a bounding box extracted from a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1 shows an example of the configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head-mounted display (HMD) 100, an input device 16 operated by a user's fingers, and an output device 15 that outputs images and sounds. The output device 15 may be a television. The information processing device 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or by a known wireless communication protocol.

[0013] The recording device 11 records applications such as system software and game software. The information processing device 10 may download game software to the recording device 11 from a content server via the network 2. The information processing device 10 executes the game software and supplies game image data and audio data to the HMD 100. The information processing device 10 and HMD 100 may be connected using a known wireless communication protocol, or may be connected by cable.

[0014] The HMD 100 is a display device worn by a user on the head, which displays images on display panels positioned in front of the user's eyes. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, realizing stereoscopic vision. Because the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data that has been corrected for optical distortion caused by the lenses.

[0015] The output device 15 is not necessary for a user wearing the HMD 100, but providing the output device 15 allows another user to view the image displayed on the output device 15. The information processing device 10 may display on the output device 15 the same image as the image viewed by the user wearing the HMD 100, or may display a different image. For example, when a user wearing an HMD and another user play a game together, the output device 15 may display a game image from the viewpoint of the character of the other user.

[0016] The information processing device 10 and the input device 16 may be connected via a known wireless communication protocol or via a cable. The input device 16 has a plurality of operation members such as operation buttons, and a user operates the operation members with their fingers while holding the input device 16. When the information processing device 10 executes a game, the input device 16 is used as a game controller. The input device 16 has an attitude sensor (IMU: Inertial Measurement Unit) including a three-axis acceleration sensor and a three-axis gyro sensor, and transmits sensor data to the information processing device 10 at a predetermined cycle (for example, 800 Hz).

[0017] In the game of the embodiment, not only operation information of the operation members of the input device 16 but also the position, posture, and movement of the input device 16 are treated as operation information and reflected in the movement of the player character in the virtual three-dimensional space. For example, operation information of the operation members may be used as information for moving the player character, and operation information such as the position, posture, and movement of the input device 16 may be used as information for moving the arms of the player character. In battle scenes in the game, the movement of the input device 16 is reflected in the movement of the player character holding a weapon, thereby realizing intuitive operation by the user and increasing the sense of immersion in the game.

[0018] In order to track the position and orientation of the input device 16, the input device 16 is provided with a plurality of markers (light emitting units) that can be photographed by an imaging device 14 mounted on the HMD 100. The information processing device 10 analyzes the photographed image of the input device 16 to estimate position information and orientation information of the input device 16 in real space, and provides the estimated position information and orientation information to the game.

[0019] The HMD 100 is equipped with multiple image capture devices 14. The multiple image capture devices 14 are attached to the front of the HMD 100 at different positions and in different orientations so that the combined capture range of each captures the entire user's field of view. The image capture devices 14 are equipped with image sensors capable of capturing images of multiple markers on the input device 16. For example, if the markers emit visible light, the image capture devices 14 have visible light sensors used in general digital video cameras, such as CCD (Charge Coupled Device) sensors or CMOS (Complementary Metal Oxide Semiconductor) sensors. If the markers emit invisible light, the image capture devices 14 have invisible light sensors. The multiple image capture devices 14 capture images of the area in front of the user at a predetermined cycle (e.g., 120 frames per second) in a synchronized manner, and transmit image data of the input device 16 to the information processing device 10.

[0020] The information processing device 10 identifies the positions of multiple marker images of the input device 16 included in the captured image. Note that although one input device 16 may be captured by multiple image capture devices 14 at the same time, the information processing device 10 may combine the multiple captured images to identify the positions of the marker images because the mounting positions and mounting orientations of the image capture devices 14 are known.

[0021] The three-dimensional shape of the input device 16 and the position coordinates of multiple markers arranged on its surface are known, and the information processing device 10 estimates the position coordinates and attitude of the input device 16 based on the distribution of marker images in the captured image. The position coordinates of the input device 16 may be position coordinates in a three-dimensional space with a reference position as the origin, and the reference position may be position coordinates (latitude, longitude) set before the start of the game.

[0022] The information processing device 10 of the embodiment has a function of estimating the position coordinates and orientation of the input device 16 using sensor data detected by an orientation sensor of the input device 16. Therefore, the information processing device 10 of the embodiment may perform tracking processing of the input device 16 with high accuracy using an estimation result based on an image captured by the imaging device 14 and an estimation result based on the sensor data. In this case, the information processing device 10 may apply a state estimation technique using a Kalman filter to integrate the estimation result based on the captured image and the estimation result based on the sensor data, thereby identifying the position coordinates and orientation of the input device 16 at the current time with high accuracy.

[0023] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by the user, goes around the head and secures the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows its length to be adjusted to fit the user's head circumference.

[0024] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the HMD 100 is worn by the user, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of optical lenses, one on each side, that are positioned between the display panel and the user's eyes and expand the user's field of view. The HMD 100 may also include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.

[0025] The front outer surface of the housing 108 is provided with multiple image capture devices 14a, 14b, 14c, and 14d. With respect to the direction of the user's face, image capture device 14a is attached to the upper right corner of the front outer surface so that its optical axis faces diagonally upward to the right. Image capture device 14b is attached to the upper left corner of the front outer surface so that its optical axis faces diagonally upward to the left. Image capture device 14c is attached to the lower right corner of the front outer surface so that its optical axis faces diagonally downward to the right. Image capture device 14d is attached to the lower left corner of the front outer surface so that its optical axis faces diagonally downward to the left. By installing multiple image capture devices 14 in this manner, the total image capture range obtained by adding up the image capture ranges of each capture device encompasses the entire field of view of the user. This user's field of view may be the user's field of view in a three-dimensional virtual space.

[0026] The HMD 100 transmits sensor data detected by the orientation sensor and image data captured by the imaging device 14 to the information processing device 10, and also receives game image data and game audio data generated by the information processing device 10.

[0027] 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as commands. The storage unit 122 temporarily stores the data and commands processed by the control unit 120. The orientation sensor 124 acquires sensor data related to the movement of the HMD 100. The orientation sensor 124 includes at least a three-axis acceleration sensor and a three-axis gyro sensor. The orientation sensor 124 detects the values ​​of each axial component (sensor data) at a predetermined cycle (for example, 800 Hz).

[0028] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0029] When the control unit 120 receives game image data and game audio data from the information processing device 10, it supplies the data to the display panel 130 for display and to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit sensor data from the orientation sensor 124, audio data from the microphone 126, and captured image data from the imaging device 14 to the information processing device 10.

[0030] FIG. 4(a) shows the shape of an input device 16a for a left hand. The input device 16a for a left hand includes a case body 20, multiple operation members 22a, 22b, 22c, and 22d (hereinafter referred to as "operation members 22" unless otherwise specified) operated by a user, and multiple markers 30 that emit light to the outside of the case body 20. The markers 30 may have an emission surface with a circular cross section. The operation members 22 may include an analog stick that is operated by tilting, a push-button, or the like. The case body 20 includes a grip portion 21 and a curved portion 23 that connects the top and bottom of the case body. The user inserts their left hand into the curved portion 23 to grip the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22a, 22b, 22c, and 22d using the thumb of their left hand.

[0031] FIG. 4(b) shows the shape of a right-handed input device 16b. The right-handed input device 16b includes a case body 20, multiple operation members 22e, 22f, 22g, and 22h (hereinafter referred to as "operation members 22" unless otherwise specified) operated by the user, and multiple markers 30 that emit light to the outside of the case body 20. The operation members 22 may include an analog stick that is operated by tilting, a push-button, or the like. The case body 20 has a grip portion 21 and a curved portion 23 that connects the top and bottom of the case body. The user inserts their right hand into the curved portion 23 to grip the grip portion 21. While gripping the grip portion 21, the user operates the operation members 22e, 22f, 22g, and 22h with the thumb of their right hand.

[0032] FIG. 5 shows the shape of a right-handed input device 16b. In addition to the operation members 22e, 22f, 22g, and 22h shown in FIG. 4(b), the input device 16b has operation members 22i and 22j. While holding the grip portion 21, the user operates operation member 22i with the index finger of the right hand and operation member 22j with the middle finger. Hereinafter, when there is no particular distinction between the input device 16a and the input device 16b, they will be referred to as "input device 16."

[0033] The operation members 22 provided on the input device 16 are equipped with a touch sensing function that recognizes a finger simply by touching it, without the need for pressing. Regarding the right-hand input device 16b, the operation members 22f, 22g, and 22j may be equipped with a capacitance-type touch sensor. Although touch sensors may be mounted on other operation members 22, it is preferable to mount them on operation members that do not come into contact with the surface on which the input device 16 is placed, such as a table.

[0034] The marker 30 is a light-emitting unit that emits light to the outside of the case body 20, and includes a resin portion on the surface of the case body 20 that diffuses and emits light from a light source such as an LED (Light Emitting Diode) element to the outside. The marker 30 is photographed by the imaging device 14 and used in the process of estimating the position and orientation of the input device 16. Since the imaging device 14 photographs the space at a predetermined cycle (for example, 120 frames per second), it is preferable that the marker 30 emits light in synchronization with the periodic photographing timing of the imaging device 14 and is turned off during periods when the imaging device 14 is not exposing the image to light, thereby reducing unnecessary power consumption.

[0035] In the embodiment, images captured by the imaging device 14 are used for tracking processing of the input device 16 and tracking processing (SLAM) of the HMD 100. Therefore, images captured at 60 frames per second may be used for tracking processing of the input device 16, and other images captured at 60 frames per second may be used for processing of the HMD 100 to simultaneously estimate the self-position and create an environmental map.

[0036] 6 shows an example of a portion of an image captured of the input device 16. This image is a captured image of the input device 16b held in the right hand, and includes images of multiple markers 30 that emit light. In the HMD 100, the communication control unit 128 transmits image data captured by the imaging device 14 to the information processing device 10 at a predetermined interval.

[0037] FIG. 7 shows functional blocks of the input device 16. The control unit 50 receives operation information input to the operating member 22 and also receives sensor data acquired by the attitude sensor 52. The attitude sensor 52 acquires sensor data related to the movement of the input device 16 and includes at least a three-axis acceleration sensor and a three-axis gyro sensor. The attitude sensor 52 detects the values ​​(sensor data) of each axis component at a predetermined cycle (e.g., 800 Hz). The control unit 50 supplies the received operation information and sensor data to the communication control unit 54. The communication control unit 54 transmits the operation information and sensor data output from the control unit 50 to the information processing device 10 via a network adapter or an antenna, by wired or wireless communication. The communication control unit 54 also receives a light emission instruction from the information processing device 10.

[0038] The input device 16 includes a plurality of light sources 58 for lighting up a plurality of markers 30. The light sources 58 may be LED elements that emit light in a predetermined color. The control unit 50 causes the light sources 58 to emit light and light up the markers 30 based on a light emission instruction received from the information processing device 10. Note that in the example shown in FIG. 7, one light source 58 is provided for one marker 30, but one light source 58 may light up a plurality of markers 30.

[0039] 8 shows functional blocks of the information processing device 10. The information processing device 10 includes a processing unit 200 and a communication unit 202. The processing unit 200 includes an acquisition unit 210, a game execution unit 220, an image signal processing unit 222, an estimation processing unit 230, and a marker information storage unit 250. The communication unit 202 receives operation information and sensor data of the operation member 22 transmitted from the input device 16, and supplies the same to the acquisition unit 210. The communication unit 202 also receives captured image data and sensor data transmitted from the HMD 100, and supplies the same to the acquisition unit 210.

[0040] The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216. The estimation processing unit 230 includes a marker image coordinate identification unit 232, a marker image coordinate extraction unit 240, and a position and orientation derivation unit 242, and the marker image coordinate identification unit 232 includes a first extraction processing unit 234, a second extraction processing unit 236, and a representative coordinate derivation unit 238. The estimation processing unit 230 estimates position information and orientation information of the input device 16 based on marker images included in the captured image. Although not described in the embodiment, the estimation processing unit 230 may estimate the position information and orientation information of the input device 16 with high accuracy by inputting the position information and orientation information of the input device 16 estimated from the marker images included in the captured image and the position information and orientation information of the input device 16 estimated from sensor data detected by the input device 16 to a Kalman filter. The estimation processing unit 230 supplies the estimated position information and orientation information of the input device 16 to the game executing unit 220 .

[0041] The information processing device 10 includes a computer, which executes a program to realize various functions shown in FIG. 8. The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or multiple chips. The functional blocks shown in FIG. 8 are realized by cooperation between hardware and software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using hardware alone, software alone, or a combination thereof.

[0042] The captured image acquisition unit 212 acquires image data obtained by capturing an image of the input device 16 equipped with a plurality of markers 30, and supplies the image data to the image signal processing unit 222. The image signal processing unit 222 performs image signal processing such as noise reduction and optical correction (shading correction) on the image data, and supplies the captured image data with improved image quality to the estimation processing unit 230.

[0043] The captured image acquisition unit 212 supplies horizontal line data of the image, one line at a time, to the image signal processing unit 222. The image signal processing unit 222 of this embodiment is configured with hardware, stores several lines of image data in a line buffer, performs image quality improvement processing on the several lines of image data stored in the line buffer, and supplies the image quality improved line data to the estimation processing unit 230.

[0044] The sensor data acquisition unit 214 acquires sensor data transmitted from the input device 16 and the HMD 100 and supplies the data to the estimation processing unit 230. The operation information acquisition unit 216 acquires operation information transmitted from the input device 16 and supplies the operation information to the game execution unit 220. The game execution unit 220 progresses the game based on the operation information and the position and orientation information of the input device 16.

[0045] The marker image coordinate identifying unit 232 identifies two-dimensional coordinates (hereinafter also referred to as "marker image coordinates") that represent the image of the marker 30 included in the captured image. The marker image coordinate identifying unit 232 may identify an area where pixels having a luminance value equal to or greater than a predetermined value are consecutive, calculate the coordinates of the center of gravity of the pixel area, and use the calculated coordinates as the representative coordinates of the marker image. The method by which the marker image coordinate identifying unit 232 derives the representative coordinates will be described later.

[0046] A method for solving the PNP (Perspective n-Point) problem is known as a method for estimating the position and orientation of an image capturing device that captured an object whose three-dimensional shape and size are known from the captured image. In this embodiment, the marker image coordinate extraction unit 240 extracts N (N is an integer equal to or greater than 3) two-dimensional marker image coordinates from the captured image, and the position and orientation derivation unit 242 derives position information and orientation information of the input device 16 from the N marker image coordinates extracted by the marker image coordinate extraction unit 240 and the three-dimensional coordinates of the N markers in a three-dimensional model of the input device 16. The position and orientation derivation unit 242 estimates the position and orientation of the image capturing device 14 using the following (Equation 1), and derives the position information and orientation information of the input device 16 in three-dimensional space based on the estimation result.

number

[0047] Here, (u, v) are the marker image coordinates in the captured image, and (X, Y, Z) are the position coordinates of the marker 30 in three-dimensional space when the three-dimensional model of the input device 16 is in the reference position and reference orientation. The three-dimensional model has exactly the same shape and size as the input device 16, and the markers are arranged in the same positions. The marker information storage unit 250 stores the three-dimensional coordinates of each marker in the three-dimensional model in the reference position and reference orientation, and the position and orientation derivation unit 242 reads the three-dimensional coordinates of each marker from the marker information storage unit 250 to obtain (X, Y, Z).

[0048] (f x , f y ) is the focal length of the image capture device 14, (cx , c y ) is the image principal point, and both are internal parameters of the image capture device 14. 11 ~r 33 The matrix with elements t1 to t3 is a rotation / translation matrix. x , f y ), (c x , c y ), (X, Y, Z) are known, and the position and orientation derivation unit 242 solves equations for the N markers 30 to find a common rotation and translation matrix for them. The position and orientation derivation unit 242 derives position information and orientation information for the input device 16 based on the angles and translation amounts represented by this matrix. In this embodiment, the process of estimating the position and orientation of the input device 16 is performed by solving a P3P problem. Therefore, the position and orientation derivation unit 242 derives the position and orientation of the input device 16 using the three marker image coordinates and the three 3D marker coordinates in the 3D model of the input device 16. The information processing device 10 generates world coordinates in a three-dimensional real space using SLAM technology. Therefore, the position and orientation derivation unit 242 derives the position and orientation of the input device 16 in the world coordinate system.

[0049] 9 is a flowchart showing the position and orientation estimation processing by the estimation processing unit 230. The captured image acquisition unit 212 sequentially acquires line data of an image captured of the input device 16 (S10) and supplies it to the image signal processing unit 222. Note that in order to reduce the calculation load of the position and orientation estimation processing, the captured image acquisition unit 212 may perform binning processing (a process of combining four pixels into one pixel) on two acquired line data and supply it to the image signal processing unit 222. The image signal processing unit 222 stores several lines of line data in a line buffer and performs image signal processing such as noise reduction and optical correction (S12). The image signal processing unit 222 supplies the line data that has been subjected to image signal processing to the marker image coordinate identification unit 232, which then identifies representative coordinates of multiple marker images included in the captured image (S14). The line data that has been subjected to image signal processing and the identified representative coordinates of the marker images are temporarily stored in a memory (not shown).

[0050] The marker image coordinate extraction unit 240 extracts any three marker image coordinates from the multiple marker image coordinates identified by the marker image coordinate identification unit 232. The marker information storage unit 250 stores the three-dimensional coordinates of each marker in a three-dimensional model of the input device 16 at a reference position and reference orientation. The position and orientation derivation unit 242 reads the three-dimensional coordinates of the markers in the three-dimensional model from the marker information storage unit 250 and solves the P3P problem using Equation 1. After identifying a rotation and translation matrix common to the extracted three marker image coordinates, the position and orientation derivation unit 242 calculates a reprojection error using the marker image coordinates of the input device 16 other than the extracted three marker image coordinates.

[0051] The marker image coordinate extraction unit 240 extracts a predetermined number of combinations of three marker image coordinates. The position and orientation derivation unit 242 identifies a rotation and translation matrix for each combination of the extracted three marker image coordinates and calculates the respective reprojection errors. The position and orientation derivation unit 242 then identifies the rotation and translation matrix that results in the smallest reprojection error from the predetermined number of reprojection errors, and derives position information and orientation information of the input device 16 (S16). The position and orientation derivation unit 242 supplies the derived position information and orientation information of the input device 16 to the game execution unit 220.

[0052] The position and orientation estimation process is performed at the capturing period (60 frames / second) of the tracking image of the input device 16 (N in S18). When the game execution unit 220 ends the game, the position and orientation estimation process by the estimation processing unit 230 ends (Y in S18).

[0053] Below, a method in which the marker image coordinate specifying unit 232 derives the representative coordinates of a marker image will be described using several flowcharts. The captured image in this embodiment is a grayscale image, and the brightness of each pixel is expressed in 8 bits, taking brightness values ​​from 0 to 255. In the captured image, the marker image is captured as an image with high brightness, as shown in FIG.

[0054] 10 is a flowchart showing the process of extracting connected components of eight neighboring pixels from a captured image by the first extraction processing unit 234. The first extraction processing unit 234 acquires line data that has undergone image signal processing from the image signal processing unit 222 (S20). The first extraction processing unit 234 performs a process of extracting connected components of eight neighboring pixels from the captured image (S22).

[0055] 11 shows an example of a captured frame image. The bright object included in the bottom of the image is the illuminated marker 30. The image signal processing unit 222 supplies horizontal line data of the frame image to the first extraction processing unit 234, starting from the top in the vertical direction. The line data supplied from the image signal processing unit 222 may be stored sequentially in a memory (not shown).

[0056] 12 is a diagram for explaining the order in which line data of an image is read out. The first extraction processing unit 234 receives horizontal line data of a frame image in order from the top, and performs processing to extract connected components of eight neighboring pixels.

[0057] FIG. 13(a) is a diagram illustrating eight neighboring pixels. In the CCL (Connected-component labeling) algorithm, pixels surrounding a single pixel P (in the up, down, left, right, and four diagonal directions) are called "eight neighboring pixels." In a binary image, when two pixels with the same value are in eight-neighborhood with each other, the two pixels are said to be "eight-adjacent." In this embodiment, a set of multiple pixels connected by eight-adjacent neighbors is called a "first connected component." The first extraction processor 234 is configured by hardware, and when two or three line data are input from the image signal processor 222, it performs a process of extracting connected components of the eight neighboring pixels.

[0058] On the other hand, as will be described later, the second extraction processing unit 236 of the embodiment executes processing for extracting connected components of four neighboring pixels by software calculation. FIG. 13(b) is a diagram illustrating four-neighboring pixels. The pixels located above, below, left, and right of a single pixel P are called "four-neighboring pixels." The four-neighboring pixels do not include pixels located diagonally. In a binary image, when two pixels with the same value are located in four-neighborhood with each other, the two pixels are said to be "four-neighboring." A set of multiple pixels connected by four-neighborhood is called a "second connected component" in this embodiment. The processing function of the second extraction processor 236 is realized by software calculation using a DSP. In this embodiment, the second extraction processor 236 performs processing to extract connected components of four-neighboring pixels from the connected component extracted by the first extraction processor 234.

[0059] When connecting components of eight neighboring pixels and connecting components of four neighboring pixels are extracted independently from the same frame image, the size of the connecting components of the eight neighboring pixels will be equal to or larger than the size of the connecting components of the four neighboring pixels because the connecting components of the eight neighboring pixels also include pixels connected diagonally. Therefore, the number of connecting components extracted from the eight neighboring pixels will be equal to or smaller than the number of connecting components extracted from the four neighboring pixels.

[0060] Returning to Fig. 10, the process (S22) of extracting a first connected component of eight neighboring pixels by the first extraction processing unit 234 will be described. The first extraction processing unit 234 searches for an area in the captured image where eight neighboring pixels having a first luminance or higher are connected. For example, the first luminance may be a luminance value of 128. By extracting connected components of eight neighboring pixels by the first extraction processing unit 234, the number of connected components to be extracted can be reduced compared to when extracting connected components of four neighboring pixels, and the load on the subsequent process of deriving marker image representative coordinates can be reduced.

[0061] FIG. 14 shows an example of multiple pixels in a captured image. In an actually captured grayscale image, a pixel with the highest brightness value of 255 is represented as white, and a pixel with the lowest brightness value of 0 is represented as black. However, in the following FIGS. 14 to 16 and 20 to 22, the brightness representation of each pixel is inverted (black and white are inverted) to prioritize visibility. Therefore, in FIGS. 14 to 16 and 20 to 22, black represents a brightness value of 255 (the highest brightness value), and white represents a brightness value of 0 (the lowest brightness value). When the first extraction processing unit 234 finds an area where eight neighboring pixels with a first brightness or higher are connected, it extracts the area as a first connected component of the eight neighboring pixels (S22) and identifies a bounding box surrounding the first connected component (S24).

[0062] FIG. 15 shows a bounding box 80a that encloses the extracted first connected component 78a of the eight neighboring pixels. The bounding box 80a is identified as the smallest rectangle that encloses the first connected component 78a of the eight neighboring pixels. Note that the first extraction processor 234 performs the extraction process for the first connected component for each line of image data, and therefore, when the first connected component 78a is extracted, it is unaware of the existence of another first connected component that may be illustrated below it. After identifying the bounding box 80a, the first extraction processor 234 outputs and stores the coordinate information of the bounding box 80a (bounding box information) in memory (not shown) (S26).

[0063] Here, the first extraction processing unit 234 determines whether the number of extracted first connected components is within a predetermined upper limit (S28). For example, the upper limit may be set to 256. In this embodiment, the position and orientation estimation process is performed in accordance with the capture cycle (60 frames / second) of the tracking images of the input device 16. Therefore, if the number of extracted first connected components becomes enormous, it becomes difficult to complete the position and orientation estimation process within the capture cycle. Therefore, an upper limit is set for the number of first connected components that the first extraction processing unit 234 will extract, and if the number of extracted first connected components exceeds the upper limit (N in S28), the first extraction processing unit 234 forcibly terminates the extraction process of the first connected components.

[0064] If the number of extracted first connected components is within a predetermined upper limit (Y of S28), steps S20 to S26 are repeated until processing of one frame of the captured image is completed (N of S30). 16 shows a bounding box 80b that encloses another first connected component 78b extracted in S22. The bounding box 80b is identified as the smallest rectangle that encloses the first connected component 78b of eight neighboring pixels. The first extraction processor 234 outputs coordinate information of the bounding box 80b to memory. When processing of one frame of the captured image is completed (Y in S30), the first extraction processor 234 begins processing the next frame image.

[0065] 17 shows an example of a bounding box extracted from a captured image. The first extraction processing unit 234 extracts a plurality of first connected components of eight neighboring pixels from the captured image, and outputs and stores information about the bounding box surrounding each of the plurality of first connected components in memory. In the example shown in FIG. 17, a bounding box of a marker image is identified in the lower part of the captured image, and a bounding box of a light source image such as an illumination light is identified in the upper part of the captured image.

[0066] 17, the user operates the input device 16 at a position close to the HMD 100, and therefore a bounding box surrounding a large marker image is identified at the bottom of the captured image. However, for example, if the user operates the input device 16 with their hand fully extended forward, the distance between the input device 16 and the image capture device 14 increases, and the captured marker image becomes smaller. If multiple small marker images are close to each other, the first extraction processing unit 234 may erroneously extract the multiple marker images as one first connected component.

[0067] 18 shows an example in which two marker images are erroneously extracted as one first connected component. In the example shown in FIG. 18, two small marker images are connected at a neighborhood of eight, so the first extraction processor 234 extracts the two marker images as one first connected component and identifies a bounding box surrounding the two marker images. Therefore, the second extraction processor 236 of the embodiment has a function of separating and processing multiple marker images included in the bounding box identified by the first extraction processor 234.

[0068] 19 is a flowchart showing the process by which the second extraction processor 236 extracts second connected components of multiple four-neighboring pixels from a first connected component included in a bounding box. The second extraction processor 236 investigates whether the first connected component extracted by the first extraction processor 234 can be separated into second connected components of multiple four-neighboring pixels, and if separation is possible, it discards the original first connected component and replaces it with the multiple second connected components after separation; if separation is not possible, it maintains the original first connected component.

[0069] The second extraction processing unit 236 acquires from memory the bounding box information (coordinate information) identified by the first extraction processing unit 234 (S40). At this time, the second extraction processing unit 236 also acquires captured image data including the bounding box and its surroundings from the memory in which the captured image data is stored (S42).

[0070] FIG. 20 shows an example of a captured image including the bounding box 80a. The width and height of the captured image area to be acquired are approximately twice the width and height of the bounding box 80a, and the center position of the image area is set so as to approximately coincide with the center position of the bounding box 80a. The second extraction processing unit 236 checks the contrast between the bounding box 80a identified by the first extraction processing unit 234 and its surroundings (S44). If the bounding box 80a includes a mark image, the average brightness within the bounding box 80a will be high, while the average brightness outside the bounding box 80a will be relatively low. Therefore, the second extraction processing unit 236 calculates the average brightness within the bounding box 80a and the average brightness within the acquired image area outside the bounding box 80a to determine the brightness ratio.

[0071] The second extraction processing unit 236 calculates the average brightness B1 of the pixels within the bounding box 80a and the average brightness B2 of the pixels within the image region outside the bounding box 80a. If the brightness ratio (B1 / B2) is less than a predetermined value (N in S44), the second extraction processing unit 236 determines that the first connected component included in the bounding box 80a is not a target for separation and discontinues the separation process for that first connected component. The predetermined value may be, for example, 3. In this case, the second extraction processing unit 236 may determine that the bounding box 80a does not include a marker image and discard the bounding box 80a.

[0072] If the brightness ratio is equal to or greater than a predetermined value (Y in S44), the second extraction processor 236 checks whether the size and shape of the bounding box 80a satisfy predetermined conditions (S46). Specifically, the second extraction processor 236 determines whether the number of pixels x in the horizontal direction and the number of pixels y in the vertical direction satisfy the following conditions 1 to 4: (Condition 1) Xmin ≦ x ≦ Xmax (Condition 2) Ymin ≦ y ≦ Ymax (Condition 3) x / y ≦ Aspect_Thresh (Condition 4) y / x ≦ Aspect_Thresh

[0073] Conditions 1 and 2 require that the size of the bounding box 80a be within a predetermined range, i.e., that the bounding box 80a be neither too large nor too small. When multiple marker images are erroneously extracted as a single first connected component, each marker image is necessarily small (if each marker image is large, multiple marker images will not be extracted as a single first connected component). Therefore, bounding boxes 80a whose pixel counts x and y are less than or equal to Xmax and Ymax, respectively, are investigated. Furthermore, if the bounding box 80a is too small, it is unlikely that it contains a marker image. Therefore, bounding boxes 80a whose pixel counts x and y are greater than or equal to Xmin and Ymin, respectively, are investigated. Conditions 3 and 4 are conditions for excluding elongated bounding boxes 80a from the investigation. If the second extraction processing unit 236 determines that the size and shape of the bounding box 80a do not satisfy any of conditions 1 to 4 (N in S46), it determines that the first connected component contained in the bounding box 80a is not to be separated, and discontinues the separation process of the first connected component.

[0074] If the second extraction processing unit 236 determines that the size and shape of the bounding box 80a satisfy all of conditions 1 to 4 (Y in S46), it performs processing to separate the first connected component included in the bounding box 80a. Specifically, the second extraction processing unit 236 searches for a region connected in four neighborhoods from the first connected component, and extracts a second connected component of the four neighborhood pixels.

[0075] 21 shows a target region for extracting a second connected component of four neighboring pixels. This target region is obtained by expanding the bounding box 80a by one pixel on both sides of the horizontal direction and both sides of the vertical direction. In the process of extracting the second connected component, the second extraction processing unit 236 searches for a region where four neighboring pixels of a second luminance or higher are connected. The second luminance may be the same as the first luminance, or may be higher than the first luminance; for example, the second luminance may be a luminance value of 160.

[0076] When the second extraction processor 236 finds an area where pixels with a second luminance or higher are connected in four-neighborhood, it extracts the area as a second connected component of the four-neighborhood pixels (S48) and identifies a bounding box surrounding the second connected component (S50). If the second extraction processor 236 does not extract multiple second connected components from the first connected component (N in S52), it determines that the first connected component included in the bounding box 80a is not a target for separation and cancels the separation process for that first connected component. On the other hand, if the second extraction processor 236 extracts multiple second connected components from the first connected component (Y in S52), it separates the first connected component 78a included in the bounding box 80a into multiple second connected components (S54).

[0077] FIG. 22 shows bounding boxes enclosing the extracted second connected components of four neighboring pixels. In this example, the second extraction processor 236 extracts three second connected components 82a, 82b, and 82c from the target region shown in FIG. 21 and identifies bounding boxes 84a, 84b, and 84c enclosing each second connected component. Note that in FIG. 22, the second extraction processor 236 assigns a label value of 1 to the second connected component 82a, a label value of 2 to the second connected component 82b, and a label value of 3 to the second connected component 82c in accordance with the CCL algorithm. Here, the second connected component 82c, which is assigned a label value of 3, includes pixels outside the bounding box 80a. Therefore, the second extraction processor 236 recognizes that the second connected component 82c is not separated from the first connected component 78a and excludes it from processing.

[0078] In this example, the first connected component 78a, which is connected by eight neighbors, is separated into the second connected components 82a and 82b, which are connected by four neighbors. If the second connected component 82a and the second connected component 82b satisfy a predetermined condition, the second extraction processor 236 replaces the first connected component 78a extracted by the first extraction processor 234 with the second connected component 82a and the second connected component 82b. Specifically, the second extraction processor 236 may discard the first connected component 78a and replace it with the second connected component 82a and the second connected component 82b, provided that the number of pixels in each of the second connected component 82a and the second connected component 82b is equal to or greater than a predetermined value. This process makes it possible to separate two marker images that were erroneously extracted as a single first connected component 78a. If the first connected component 78a is separated into a predetermined number (for example, three or four) or more, the second extraction processing unit 236 may determine that the separation process is inappropriate and may maintain the first connected component 78a.

[0079] The second extraction processing unit 236 investigates whether or not a separable first connected component is included in all of the bounding boxes identified by the first extraction processing unit 234 (N in S56). When the second extraction processing unit 236 has finished investigating all of the bounding boxes (Y in S56), the representative coordinate derivation unit 238 performs processing to derive representative coordinates of the marker image based on the pixels of the first connected component extracted by the first extraction processing unit 234 and / or the pixels of the second connected component extracted by the second extraction processing unit 236.

[0080] FIG. 23 shows a flowchart illustrating the representative coordinate derivation process. The representative coordinate derivation unit 238 derives the representative coordinates of the marker image using the bounding box identified by the first extraction processing unit 234 and the bounding box identified by the second extraction processing unit 236. In this embodiment, the representative coordinate derivation unit 238 checks whether the marker image is included in the bounding boxes identified by the first extraction processing unit 234 and the second extraction processing unit 236 based on several criteria. First, the representative coordinate derivation unit 238 acquires bounding box information (S60) and checks whether the size of the bounding box is within a predetermined range (S62). If the bounding box is too large (N in S62), the first connected component or the second connected component included in the bounding box is not an image of the marker 30. Therefore, the representative coordinate derivation unit 238 discards the bounding box that is too large.

[0081] If the size of the bounding box is within a predetermined range (Y in S62), the second extraction processing unit 236 checks whether the shape of the connected components of high-luminance pixels contained within the bounding box is elongated (S64). Because the marker 30 has an exit surface with a circular cross section, the marker image has a shape close to round and will not be elongated. If the shape of the connected components of high-luminance pixels is elongated (Y in S64), the high-luminance lighting object contained in the bounding box is not a marker 30, so the representative coordinate derivation unit 238 discards the elongated bounding box.

[0082] If the shape of the connected portion of high-luminance pixels is not elongated (N in S64), the representative coordinate derivation unit 238 checks the contrast between the identified bounding box and its surroundings (S66). This contrast checking process may be the same as the process shown in S44 of Fig. 19, for example. If the ratio between the average luminance within the bounding box and the average luminance within a predetermined area outside the bounding box is less than a predetermined value (N in S66), the representative coordinate derivation unit 238 discards the bounding box.

[0083] If the brightness ratio is equal to or greater than a predetermined value (Y in S66), the representative coordinate derivation unit 238 recognizes that a marker image is included in the bounding box, and derives representative coordinates of the marker image based on pixels in the bounding box that have a third brightness or higher (S68). These representative coordinates may be centroid coordinates. The third brightness may be lower than the first brightness, for example, a brightness value of 64. The representative coordinate derivation unit 238 calculates the average brightness position in the X-axis direction and the Y-axis direction, and derives the representative coordinates (u, v). In this case, it is preferable that the representative coordinate derivation unit 238 calculates the brightness centroid position taking into account the pixel value of each pixel that has a third brightness or higher, and derives the representative coordinates (u, v).

[0084] In the above embodiment, in relation to S28 in Fig. 10, it has been explained that an upper limit is set on the number of first connected components that the first extraction processing unit 234 can extract. Note that when the number of extracted first connected components reaches the upper limit, the first extraction processing unit 234 forcibly terminates the extraction processing of first connected components, but the second extraction processing unit 236 may perform the above-mentioned separation processing on the extracted first connected components up to the upper limit number.

[0085] 24 shows an example of a bounding box extracted from a captured image by the first extraction processing unit 234. This captured image includes blinds that are attached to the inside of windows for purposes such as sun protection and privacy. The blinds captured in this image are Venetian blinds, which have multiple horizontal slats arranged in the vertical direction, and are a type of blind that is often used in offices and the like.

[0086] The first extraction processing unit 234 of the embodiment is configured with hardware that sequentially acquires line data of an image and extracts first connected components of eight neighboring pixels. The arrows in FIG. 24 indicate the order in which line data of the image is read from the image sensor of the imaging device 14, and the first extraction processing unit 234 performs extraction processing of first connected components based on the read line data. In the example shown in FIG. 24, the first extraction processing unit 234 performs extraction processing of first connected components sequentially from top to bottom of the captured image. As a result, the number of extracted first connected components reaches the upper limit (256) before processing all image data is completed, and the extraction processing of first connected components is forcibly terminated. As shown in the captured image in FIG. 24, a marker image of the marker 30 of the input device 16 is present in the lower left of the image, but the marker image is not extracted because the number of extracted first connected components has reached the upper limit.

[0087] 17 , the input device 16 is photographed by the image sensor of the imaging device 14 mounted on the HMD 100, and therefore, when the user is playing a game normally, the input device 16 is photographed at the bottom of the angle of view. Therefore, in the HMD 100, the control unit 120 may read image data from the image sensor of the imaging device 14 by inverting the image upside down, and transmit the read image data to the information processing device 10 from the communication control unit 128.

[0088] In the information processing device 10, the captured image acquisition unit 212 acquires image data that has been read out from the image sensor after being inverted upside down. Therefore, the captured image acquisition unit 212 acquires line data of the captured image in order from the bottom of the image, and supplies the data to the estimation processing unit 230 via the image signal processing unit 222. This allows the first extraction processing unit 234 to extract first connected components consisting of consecutive pixels with a predetermined luminance or higher from the image data that has been read out from the image sensor after being inverted upside down, and increases the likelihood of extracting a first connected component corresponding to a marker image present at the bottom of the captured image before the number of extracted first connected components reaches an upper limit.

[0089] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and those skilled in the art will understand that various modifications are possible in the combination of the respective components and processing processes, and that such modifications are also within the scope of the present invention. In the embodiments, the estimation process is performed by the information processing device 10, but the functions of the information processing device 10 may be provided in the HMD 100, and the HMD 100 may perform the estimation process. In other words, the HMD 100 may be the information processing device 10.

[0090] In the embodiment, the arrangement of the multiple markers 30 in the input device 16 having the operation member 22 has been described, but the device to be tracked does not necessarily have to have the operation member 22. In the embodiment, the imaging device 14 is attached to the HMD 100, but the imaging device 14 may be attached to a position other than the HMD 100 as long as it can capture marker images. [Explanation of symbols]

[0091] 1 Information processing system, 10 Information processing device, 14 Imaging device, 16a, 16b Input device, 20 Case body, 21 Grip portion, 22 Operation member, 23 Bending portion, 30 Marker, 50 Control unit, 52 Orientation sensor, 54 Communication control unit, 58 Light source, 100 HMD, 102 Output mechanism unit, 104 Wearing mechanism unit, 106 Wearing band, 108 Housing, 120 Control unit, 122 Memory unit, 124 Orientation sensor, 126 Microphone, 128 Communication Control unit, 130... display panel, 132... audio output unit, 200... processing unit, 202... communication unit, 210... acquisition unit, 212... captured image acquisition unit, 214... sensor data acquisition unit, 216... operation information acquisition unit, 220... game execution unit, 222... image signal processing unit, 230... estimation processing unit, 232... marker image coordinate identification unit, 234... first extraction processing unit, 236... second extraction processing unit, 238... representative coordinate derivation unit, 240... marker image coordinate extraction unit, 242... position and orientation derivation unit, 250... marker information storage unit.

Claims

1. a captured image acquisition unit that acquires an image of a device having a plurality of markers; an estimation processing unit that estimates position information and orientation information of the device based on marker images in the captured image, The estimation processing unit a marker image coordinate specifying unit that specifies representative coordinates of the marker image from the captured image; a position and orientation derivation unit that derives position information and orientation information of the device using representative coordinates of the marker images, the captured image acquisition unit acquires a plurality of line data of the captured image that has been read out from the image sensor in a vertically inverted manner such that lower line data among the plurality of line data of the captured image is acquired before upper line data; The marker image coordinate specifying unit an extraction processing unit that processes each line data in order from the lower line data to the upper line data, and extracts connected components of pixels from the captured image that is read out from the image sensor in a vertically inverted state; a representative coordinate derivation unit that derives representative coordinates of the marker image based on the pixels of the connected components extracted by the extraction processing unit, the extraction processing unit terminates the extraction process of connected components when the number of extracted connected components reaches a predetermined upper limit.

1. An information processing device comprising:

2. the extraction processing unit is configured with hardware and extracts the connected components from line data of the captured image that is read out from the image sensor after being inverted upside down; 2. The information processing apparatus according to claim 1, wherein:

3. The image sensor is mounted on a head-mounted display that is worn on the user's head, The device is held by a user.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. acquiring, from an image sensor that has captured an image of a device having a plurality of markers, a plurality of line data of the captured image that has been read out in a manner that the lower line data of the captured image is acquired before the upper line data; a step of processing each line data in order from the lower line data to the upper line data to extract connected components of pixels from the captured image; deriving representative coordinates of the marker image based on the pixels of the extracted connected components; terminating the extraction of connected components when the number of extracted connected components reaches a predetermined upper limit; A representative coordinate deriving method comprising:

5. On the computer, A function of acquiring a plurality of line data of a captured image from an image sensor that has captured an image of a device having a plurality of markers, the line data being read out in a manner that the lower line data of the captured image is acquired before the upper line data; A function of processing each line data in order from the lower line data to the upper line data and extracting connected components of pixels from the captured image; A function of deriving representative coordinates of the marker image based on the pixels of the extracted connected components; a function of terminating the extraction of connected components when the number of extracted connected components reaches a predetermined upper limit; A program to achieve this.

Images