Information processing device, display system, information processing method, and program

Abstract

The present invention provides an information processing device capable of uniquely identifying indicators in an image. [Solution] An information processing device (200) acquires an image taken from an imaging device (210) that photographs a real space in which an indicator is placed, and estimates the relative position and orientation of the indicator, comprising: detection means (201) that detects the indicator in the image taken and acquires image coordinates indicating the position of the indicator in the image taken; extraction means (202) that extracts peripheral information of the indicator from the image taken; first holding means (204) that holds the image coordinates of the indicator and the peripheral information in association; identification means (203) that identifies the indicator by comparing the peripheral information extracted by the extraction means with the peripheral information held by the holding means; and estimation means (206) that estimates three-dimensional geometric information of the indicator or imaging device based on the indicator identified by the identification means.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a display system, an information processing method, and a program.

Background Art

[0002] Conventionally, Mixed Reality (MR) technology has been known. MR technology is a technology that seamlessly fuses the real space and the virtual space created by a computer. Among MR technologies, Augmented Reality (AR, also called extended reality or enhanced reality), which overlays and displays the real space and the virtual space, has attracted particular attention. In a system that realizes AR, since it is necessary to accurately align the virtual space with the real space, it is required to accurately estimate the position and orientation of the camera in the real space.

[0003] Patent Document 1 discloses a method of taking a large number of images including a plurality of indicators, and correcting the position and orientation of the indicators by iterative calculation so as to minimize the projection error of the indicators using those images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] The method disclosed in Patent Document 1 is difficult to apply when there is insufficient space to place the indicators. If the indicators are made physically smaller, the pattern or design used to uniquely identify the indicators becomes smaller, which may make it difficult to properly recognize the indicators. On the other hand, if multiple indicators of the same shape are used, it becomes impossible to uniquely identify the target indicator from the captured image, and the indicators cannot be used to estimate the position and orientation of the camera.

[0007] Therefore, the present invention aims to provide an information processing device capable of uniquely identifying indicators in an image. [Means for solving the problem]

[0008] An information processing device as one aspect of the present invention is an information processing device that acquires a captured image from an imaging device that photographs a real space in which an indicator is placed, and estimates the relative position and orientation of the indicator, and is characterized by comprising: detection means for detecting the indicator in the captured image and acquiring image coordinates indicating the position of the indicator in the captured image; extraction means for extracting peripheral information of the indicator from the captured image; first holding means for holding the image coordinates of the indicator and the peripheral information in association; identification means for identifying the indicator by comparing the peripheral information extracted by the extraction means with the peripheral information held by the holding means; and estimation means for estimating three-dimensional geometric information of the indicator or the imaging device based on the indicator identified by the identification means.

[0009] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an information processing device that can uniquely identify indicators in an image. [Brief explanation of the drawing]

[0011] [Figure 1] These are conceptual diagrams of the MR system in each embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of the information processing device in each embodiment. [Figure 3] This is a hardware configuration diagram of the information processing device in each embodiment. [Figure 4] This is a flowchart showing the process performed by the information processing device in Example 1. [Figure 5] This is an illustrative diagram showing an example of an index in Example 1. [Figure 6] This is an illustrative diagram illustrating an example of extracting information surrounding the indicator in Example 1. [Figure 7] This is an illustrative diagram illustrating an example of identifying the indicator in Example 1. [Figure 8] This flowchart shows the process performed by the information processing device in Example 2. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [Examples]

[0013] First, referring to FIG. 1, an example of applying the method of Example 1 of the present invention to an MR system for realizing robot teaching will be described. FIG. 1 is an image diagram of an MR system (display system) for teaching (teaching) the operations that a real robot should perform on an object virtually displayed by MR.

[0014] 101 is a camera (imaging device), which is a camera mounted on a head mounted display (HMD). The information processing device (corresponding to the information processing device 200 in FIG. 2) generates and draws an image to be displayed on the head mounted display based on the image captured by the camera 101. The range captured by the camera 101 becomes the range displayed on the head mounted display, and the captured range in the real space and the CG of the virtual object are superimposed and drawn.

[0015] 102 is a CG displayed on a virtual display device (corresponding to the display device 220 in FIG. 2), and is displayed at a position directly above the indicator 103. 104 is a robot to be taught, and the user uses the head mounted display to teach the operation to be performed on the CG 102 displayed at an accurate position based on the indicator 103 in real time while viewing the CG 102 and the robot 104. At this time, although a plurality of indicators 105 having the same shape as the indicator 103 are arranged, by using the method of this embodiment, the image processing device can appropriately display the CG 102 above the indicator 103 using the display device.

[0016] In addition, in order to superimpose and display the real space and the CG of the virtual object, information on the position and orientation of the camera 101 in the real space is required. As a method for estimating the position and orientation of the camera 101, the SLAM (Simultaneous Localization And Mapping) technology is known. For example, in the method of Raul et al. (Non-Patent Document 1), the SLAM technology using a camera is disclosed.

[0017] In SLAM, the process of creating a 3D map and the process of estimating the self-position and orientation using the 3D map are performed in parallel. As information included in the 3D map, a group of images having 3D coordinate information (hereinafter, each of the images is referred to as a key frame) is created. When the self-position and orientation estimation fails, as a recovery method, a process called relocalization is performed. Relocalization is a method of comparing key frames included in the 3D map with a captured image and specifying the position based on the degree of coincidence. At this time, it is also possible to return to the position and orientation estimation from the relative positional relationship between the index detected from the captured image and the camera using an index whose 3D coordinates are defined in advance.

[0018] Here, when a plurality of indices are arranged, it is necessary to uniquely identify which index among the plurality of indices is the index (target index) included in the captured image. If the target index (one of the plurality of indices) has a shape that includes a pattern or pattern that can be uniquely identified alone, the index can be uniquely identified. On the other hand, when the plurality of indices are indices of the same shape, the target index cannot be uniquely identified. Therefore, in this embodiment, an example of uniquely identifying and returning the target index from among the plurality of indices of the same shape when performing relocalization will be described.

[0019] Next, referring to FIG. 2, a functional configuration example of the information processing apparatus 200 in this embodiment will be described. FIG. 2 is a block diagram showing a configuration example of the information processing apparatus 200. The information processing apparatus 200 is connected to an imaging apparatus 210 (corresponding to the camera 101 in FIG. 1) and a display apparatus 220. The information processing apparatus 200, the imaging apparatus 210, and the display apparatus 220 constitute a display system.

[0020] The information processing device 200 may be configured integrally with either or both of the imaging device 210 and the display device 220. Alternatively, the information processing device 200 may be provided as a separate device from each of the imaging device 210 and the display device 220. In this case, the information processing device 200 can send and receive information by communicating with the imaging device 210 or the display device 220. Furthermore, at least some of the components of the information processing device 200 may be implemented using cloud computing.

[0021] The information processing device 200 acquires images from an imaging device 210 that photographs the real space where the indicator is placed, and estimates the relative position and orientation of the indicator. The information processing device 200 includes an indicator detection unit (detection means) 201, an indicator surrounding information extraction unit (extraction means) 202, an indicator identification unit (identification means) 203, an indicator surrounding information holding unit (first holding means) 204, a map information holding unit (second holding means) 205, and an estimation unit (estimation means) 206.

[0022] The indicator detection unit 201 acquires an image (captured image) from the imaging device 210 and detects an indicator based on pre-registered information about the shape of the indicator. The indicator detection unit 201 outputs the two-dimensional coordinates (image coordinates) of the detected indicator on the image to the indicator surrounding information extraction unit 202. In this way, the indicator detection unit 201 detects an indicator in the captured image and acquires image coordinates indicating the position of the indicator in the captured image. In other words, the indicator detection unit 201 uses an image acquired by imaging in real space to detect an indicator placed in real space.

[0023] The indicator surrounding information extraction unit 202 extracts information (indicator surrounding information) that uniquely identifies the indicator from its surroundings, based on the image coordinates of the indicator detected by the indicator detection unit 201, and outputs the extracted information to the indicator identification unit 203. In other words, the indicator surrounding information extraction unit 202 is an extraction means for extracting indicator surrounding information from a captured image, or an acquisition means for acquiring indicator surrounding information using an image.

[0024] The index identification unit 203 identifies (specifies) index peripheral information that indicates the same index as the index peripheral information extracted by the index peripheral information extraction unit 202, from among the index peripheral information held by the index peripheral information holding unit 204. The index identification unit 203 outputs the identification result regarding whether or not the index peripheral information held by the index peripheral information holding unit 204 and the index peripheral information extracted by the index peripheral information extraction unit 202 have been identified to the estimation unit 206 and the display device 220. In this way, the index identification unit 203 identifies the index by comparing the index peripheral information extracted by the index peripheral information extraction unit 202 with the index peripheral information held by the index peripheral information holding unit 204.

[0025] The indicator surrounding information storage unit 204 stores indicator surrounding information and keyframes associated with the indicator. In other words, the indicator surrounding information storage unit 204 stores the image coordinates of the indicator and the indicator surrounding information in association. The map information storage unit 205 stores a 3D map including keyframes.

[0026] The estimation unit 206 estimates the position and orientation of the imaging device 210 based on the identification result by the indicator identification unit 203, the keyframes held by the indicator surrounding information holding unit 204, and the 3D map held by the map information holding unit 205, and outputs the estimation result to the display device 220. In other words, the estimation unit 206 estimates the 3D geometric information of the indicator or the imaging device 210 based on the indicator identified by the indicator identification unit 203. In this embodiment, the indicator identification unit 203, the indicator surrounding information holding unit 204, and the estimation unit 206 function as identification means that use the indicator surrounding information to identify the indicator in a way that distinguishes it from other indicators placed in real space.

[0027] Next, the hardware configuration of the information processing device 200 will be described with reference to Figure 3. Figure 3 is a hardware configuration diagram of the information processing device 200. The CPU (Central Processing Unit) H11, which acts as the control unit, controls various devices connected to the system bus H21. H12 is a ROM (Read Only Memory) that stores the BIOS (Basic Input / Output System) program and the boot program.

[0028] External memory H13 stores applications and programs processed by the information processing device 200, as well as various data and files. External memory H13 is, for example, memory such as a hard disk (HD) or solid state drive (SSD). RAM (Random Access Memory) H14 is used as the main memory of CPU H11. RAM H14 also functions as a work area. CPU H11 loads the program that executes the processing of this embodiment, stored in ROM H12 or external memory H13, into RAM H14 and executes it, and comprehensively controls each unit connected to the system bus H21.

[0029] The input unit H15 is an input device such as a keyboard, pointing device, or robot controller, and accepts input from the user. The display unit H16 has a display device such as a liquid crystal display and outputs the processing results of the information processing device 200 to the display device 220 according to commands from the CPU H11. The display device 220 may be a liquid crystal display, projector, LED indicator, or head-mounted display capable of VR (virtual reality) display. Furthermore, the input unit H15 and the display unit H16 may be configured as a touch panel. By associating input coordinates with display coordinates on the touch panel, a GUI can be configured that makes it appear as if the user can directly operate the screen displayed on the touch panel.

[0030] The communication interface H17 communicates information with external devices via the network. The communication interface H17 can be of any type, including Ethernet, USB, serial communication, or wireless communication. The network may consist of a communication network such as a LAN or WAN, a cellular network (e.g., LTE or 5G), a wireless network, or a combination of these. In other words, the network only needs to be configured to enable data transmission and reception, and any physical layer communication method may be used.

[0031] Furthermore, the flowcharts illustrating the processing steps in the following explanation are not limited to examples; any combination of steps, grouping of multiple processes, or subdivision of processes is possible as long as the results of each embodiment are satisfied. Each process can also be individually extracted and function as a standalone functional element, and can be used in combination with processes other than those shown.

[0032] Next, with reference to Figure 4, the processing (information processing method) performed by the information processing device 200 in this embodiment will be described. Figure 4 is a flowchart of the information processing method in this embodiment. The processing steps include initialization processing S101, indicator detection S102, detection success / failure determination S103, extraction of information surrounding the indicator S104, indicator identification S105, identification success / failure determination S106, relocalization using the indicator S107, normal relocalization S108, and termination determination S109. The processing in this flowchart starts when the user activates the head-mounted display in order to use the MR system.

[0033] First, in step S101, CPUH11 initializes the information processing device 200. Specifically, CPUH11 reads the program and information about the indicator to be detected, such as the shape and size of the indicator, from the external memory H13, and makes the information processing device 200 operational.

[0034] Next, in step S102, the indicator detection unit 201 acquires an image from the imaging device 210, detects indicators in the image based on the shape and size of the indicators read from the external memory H13, and determines the position (image coordinates) of the indicators in the image. The indicator detection unit 201 then outputs the image coordinates of the detected indicators to the indicator surrounding information extraction unit 202. If the image contains multiple indicators, the indicator detection unit 201 determines the image coordinates of all indicators.

[0035] Figure 5 is an illustrative diagram showing examples of indicators to be detected. 501 is a square-shaped planar indicator, and 502 is a circular point indicator. Generally, indicators used in MR contain specific patterns and designs within a square, and can be uniquely identified by matching these patterns with pre-registered shapes. In this embodiment, however, indicators that do not contain patterns and designs for unique identification are used. As a specific method for detecting the indicator, in the case of the planar indicator 501, since the shape of the indicator is a rectangle, a rectangle is detected from the image and the intersection of the diagonals of the rectangle is used as the image coordinate of the indicator. On the other hand, in the case of the point indicator 502, since the shape of the indicator is circular, a circle is detected from the image and the coordinates of the centroid of the circle are used as the image coordinate of the indicator.

[0036] Next, in step S103, the indicator surrounding information extraction unit 202 determines whether or not it succeeded in detecting the indicator in step S102. If it succeeds in detecting the indicator, it proceeds to step S104. On the other hand, if it fails to detect the indicator, that is, if no indicators are detected from the captured image, it proceeds to step S108.

[0037] In step S104, the indicator surrounding information extraction unit 202 extracts indicator surrounding information from the image based on the image coordinates of the indicator obtained from the indicator detection unit 201. The indicator surrounding information includes, for example, information about feature points detected from the image. In terms of processing flow, the indicator surrounding information extraction unit 202 performs a projection transformation on the captured image based on the image coordinates of the indicator, and detects feature points around the indicator from the image after the projection transformation. The indicator surrounding information extraction unit 202 then outputs the relative positional relationship between the detected feature points and the indicator as indicator surrounding information to the indicator identification unit 203.

[0038] Figure 6 is an illustrative image of the result when step S104 is applied. 600 is the image captured by the imaging device 210. 601 is the index detected in step S102. 602 is a triangular object located around index 601.

[0039] First, the indicator surrounding information extraction unit 202 calculates a projection transformation matrix based on the image coordinates of each vertex of the indicator 601. When the indicator surrounding information extraction unit 202 performs a projection transformation on image 600, an image 610 is obtained that is directly facing the indicator 601. Next, the indicator surrounding information extraction unit 202 detects feature points in the range where the distance from the indicator is below a threshold (below a predetermined distance). 611 indicates the range in which feature points are detected. Image 620 shows the result of feature point detection, and 621 are the detected feature points. Each vertex of the triangle is detected as a feature point 621. In this way, the indicator surrounding information extraction unit 202 outputs the relative positional relationship between the extracted feature points and the indicator as indicator surrounding information to the indicator identification unit 203.

[0040] Next, in step S105 of Figure 4, the index identification unit 203 compares the multiple index peripheral information held by the index peripheral information holding unit 204 with the index peripheral information extracted by the index peripheral information extraction unit 202. The index identification unit 203 then determines whether the index extracted by the index peripheral information extraction unit 202 is an index included in the captured image. In other words, the index identification unit 203 determines whether the identification was successful based on whether the degree of agreement obtained by comparing the multiple index peripheral information held by the index peripheral information holding unit 204 with the index peripheral information extracted by the index peripheral information extraction unit 202 is above a threshold.

[0041] Figure 7 is an illustrative diagram of the surrounding information of the comparison target indicator when step S105 is applied, visually showing the indicator and its surrounding information held by the indicator surrounding information holding unit 204. For example, the indicator surrounding information holding unit 204 holds three pieces of information, 701, 702, and 703, for each of the three indicators. Each piece of information about the indicator includes the positional relationship between the indicator and the feature points around it.

[0042] Meanwhile, the feature points 621 extracted by the index surrounding information extraction unit 202 in step S104 have the positional relationship shown in image 620 of Figure 6. The index identification unit 203 compares the positional relationship between these feature point groups and each of the three feature point groups in Figure 7. The index identification unit 203 compares each of the three index surrounding information in Figure 7 with the index surrounding information shown in image 620 of Figure 6. The index identification unit 203 then determines that the degree of agreement of the index surrounding information 702 in Figure 7 is the highest and is above the threshold, and identifies the index 601 included in the captured image as the index corresponding to the index surrounding information 702.

[0043] Next, in step S106 of Figure 4, the indicator identification unit 203 determines whether the identification in step S105 was successful or not. If the identification is successful, the process proceeds to step S107. On the other hand, if the identification is unsuccessful, the process proceeds to step S108.

[0044] In step S107, the estimation unit 206 estimates the position and orientation of the imaging device 210 based on the index surrounding information identified by the index identification unit 203. The estimation unit 206 performs relocalization based on the 3D map held by the map information holding unit 205 and the keyframes associated with the index surrounding information held by the index surrounding information holding unit 204. The estimation unit 206 estimates the position and orientation of the imaging device 210 on the 3D map from the keyframes associated with the identified index.

[0045] In step S108, the estimation unit 206 performs normal relocalization. If the detection of the indicator in step S102 fails, or if the identification of the indicator in step S105 fails, the estimation unit 206 performs relocalization without using the indicator information and estimates the position and orientation of the imaging device 210.

[0046] Next, in step S109, the estimation unit 206 determines whether all processing has been completed. If all processing is completed, the process is terminated. On the other hand, if the processing is not completed, the process returns to step S102 and continues.

[0047] According to this embodiment, even when multiple indicators of the same shape are arranged, the indicators included in the captured image can be uniquely identified and relocalized.

[0048] (Variation 1) In this embodiment, the index surrounding information extraction unit 202 used information about feature points as index surrounding information, but is not limited to this and may use other information as long as it can be extracted from the vicinity of the index. Index surrounding information may be, for example, feature quantities, edges, patterns, or object information. Feature quantities or edges can be extracted from an image by feature extraction or edge detection, and patterns can be extracted by contour extraction. Object information can be extracted by object recognition processing.

[0049] Alternatively, some of these extraction methods may be combined to extract information about the area surrounding the indicator. For example, object recognition and feature extraction may be combined, and the area recognized as a desk by object recognition may be assumed to be a plane, and feature extraction may be performed after projecting the area of ​​the desk. Furthermore, the extracted information about the area surrounding the indicator can be used by the indicator identification unit 203 to identify the indicator by storing the relative positional relationship with the detected indicator in the indicator surrounding information storage unit 204, similar to this embodiment.

[0050] In this embodiment, the indicator surrounding information extraction unit 202 detects feature points within a range where the distance from the indicator is less than or equal to a threshold (an extraction range where the distance from the indicator is less than or equal to a predetermined distance), but the extraction range is not limited to this. For example, indicator surrounding information may be extracted from the entire image without limiting the distance, or the extraction range may be dynamically expanded by setting distance thresholds in stages until indicator surrounding information can be extracted. Alternatively, indicator surrounding information may be extracted using images captured before and after the image in chronological order (multiple images captured at different times). By expanding the extraction range in this way, indicator surrounding information can be sufficiently extracted, and the accuracy of indicator identification can be improved.

[0051] (Modification 2) In this embodiment, the indicator surrounding information holding unit 204 holds keyframes in addition to the indicator and indicator surrounding information, but is not limited to this, and may hold other information as long as it is information that the estimation unit 206 can use for estimation processing. For example, it may hold both the 3D coordinates of the real space and information about objects around the indicator, or it may hold information that the indicator placed closest to the robot is the origin in the world coordinate system. This makes it possible to precisely specify the position of the indicator in the real space.

[0052] (Variation 3) In this embodiment, the indicator identification unit 203 identifies an indicator based on the indicator peripheral information output by the indicator peripheral information extraction unit 202 and the indicator peripheral information held by the indicator peripheral information holding unit 204. However, the indicator peripheral information held by the indicator peripheral information holding unit 204 may be updated during the identification process. That is, if the indicator peripheral information extracted by the indicator peripheral information extraction unit 202 and the indicator peripheral information held by the indicator peripheral information holding unit 204 differ with respect to the identified indicator, the indicator identification unit 203 may update the held indicator peripheral information with the extracted peripheral information.

[0053] For example, if an image contains a moving object, the feature points extracted by the indicator surrounding information extraction unit 202 will change over time. To limit the extraction to only feature points that can be reliably extracted, the changed feature points may be deleted. This improves the accuracy of indicator identification.

[0054] In this embodiment, the indicator identification unit 203 identifies the indicator only if it succeeds in detecting the indicator, but it may also identify the indicator if it fails to detect it. In that case as in this embodiment, the indicator surrounding information extracted by the indicator surrounding information extraction unit 202 is compared with the indicator surrounding information held by the indicator surrounding information holding unit 204, and the success or failure of the identification is determined according to the degree of agreement. That is, if the indicator detection unit 201 cannot detect the indicator from the captured image, the indicator identification unit 203 identifies the indicator by comparing the indicator surrounding information extracted by the indicator surrounding information extraction unit 202 with the indicator surrounding information held by the indicator surrounding information holding unit 204. This makes it possible to identify the indicator based only on the indicator surrounding information, even if the captured image does not contain the indicator.

[0055] (Modification 4) In this embodiment, the estimation unit 206 estimates the position and orientation of the imaging device 210 based on the index identification result by the index identification unit 203, but is not limited to this. The estimation target may be other targets as long as estimation is possible using the index identification result. For example, the estimation target may be the three-dimensional geometric information of the index, or the perspective projection matrix between the index or index surrounding information and the image may be estimated for use in camera calibration.

[0056] (Variation 5) In this embodiment, the process from when the information processing device 200 identifies an indicator to when the identification result is used for position and orientation estimation may be output to the display device 220. For example, at least one of the indicator identified by the indicator identification unit 203 and the information surrounding the indicator may be superimposed on the input image and displayed on the display device 220. Alternatively, the identification result of the indicator identification unit 203 and the position and orientation estimation result of the imaging device 210 of the estimation unit 206 may be used to display the detected indicator and the information surrounding the indicator in 3D on the display of the head-mounted display. [Examples]

[0057] Next, Embodiment 2 of the present invention will be described. Embodiment 1 described an example in which, when the estimation of the camera's position and orientation using SLAM technology fails, the estimation is relocalized using an index. If position and orientation estimation is continued continuously using SLAM technology, errors accumulate in the estimation results, and the error in the 3D map used for position and orientation estimation increases. This embodiment describes a method for reducing errors by correcting the 3D map using multiple indexes of the same shape. Note that the configuration in this embodiment is the same as that of the information processing device 200 described in Embodiment 1, as shown in Figure 2, so its explanation will be omitted.

[0058] Referring to Figure 8, the processing (information processing method) performed by the information processing device 200 in this embodiment will be described. Figure 8 is a flowchart of the information processing method in this embodiment. In Figure 8, steps S107 and S108 have been deleted from Figure 4, which was described in Embodiment 1, and steps S201 for registering information around the indicator and S202 for correcting the 3D map have been added. Here, only the steps that differ from Embodiment 1 will be described.

[0059] In step S201, if it is determined that the indicator detection was successful in step S103, but that the indicator identification failed in step S106, the estimation unit 206 determines that the indicator surrounding information storage unit 204 does not yet hold the indicator surrounding information for the detected indicator. Therefore, the estimation unit 206 registers the indicator surrounding information together with the current keyframe in the indicator surrounding information storage unit 204.

[0060] In step S202, the estimation unit 206 corrects the 3D map held by the map information holding unit 205 based on the index surrounding information held by the index surrounding information holding unit 204. Bundle adjustment, an optimization method, can be used as the method for correcting the 3D map. However, bundle adjustment is generally a method for minimizing reprojection errors. Therefore, estimation may be performed in combination with the method of Skrypnyk et al. (Non-Patent Literature 2), which estimates the position and orientation so that the sum of the reprojection error and the difference in position and orientation between keyframes is minimized. In this method, the targets of optimization are the positions of all feature points included in the 3D map and the positions and orientations of the keyframes.

[0061] On the other hand, the indicator surrounding information storage unit 204 maintains the relative positional relationships between feature points around the indicator, and also maintains keyframes from the time of indicator detection. Therefore, the 3D coordinates of the feature point group held by the indicator surrounding information storage unit 204 can be used as known information. For feature points included in the 3D map that correspond to the feature points held by the indicator surrounding information storage unit 204, the 3D coordinates are fixed and the 3D map is optimized.

[0062] Furthermore, since the optimization of the 3D map uses all the index surrounding information held by the index surrounding information holding unit 204, the index surrounding information registered in step S201, i.e., feature points, is also used. When new indexes are detected, the number of feature points fixed during the optimization of the 3D map increases, which further reduces the error of the 3D map.

[0063] According to this embodiment, when using multiple indicators of the same shape, it is possible to uniquely identify the indicators included in the image and reduce the error of the 3D map used for position and orientation estimation.

[0064] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0065] Each embodiment's disclosure includes the following configuration and method. (Composition 1) An information processing device that acquires images from an imaging device that photographs a real space in which indicators are placed, and estimates the relative position and orientation of the indicators, A detection means for detecting the indicator in the captured image and obtaining image coordinates indicating the position of the indicator in the captured image, An extraction means for extracting peripheral information of the indicator from the captured image, A first storage means that stores the image coordinates of an indicator in association with surrounding information, An identification means for identifying the index by comparing the peripheral information extracted by the extraction means with the peripheral information held by the holding means, An estimation means for estimating the index or the three-dimensional geometric information of the imaging device based on the index identified by the identification means, An information processing device characterized by comprising: (Configuration 2) The estimation means estimates the position and orientation of the imaging device. The information processing device according to configuration 1, characterized by the above. (Composition 3) The system further includes a second means for holding map information, The estimation means estimates the position and orientation of the imaging device using the image coordinates of the index identified by the identification means and the map information held by the second holding means. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The extraction means extracts at least one of feature points, feature quantities, edges, patterns, and object information from the captured image as peripheral information. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The extraction means extracts the surrounding information based on the relative positional relationship between the surrounding information and the index detected by the detection means, or the relative positional relationship between the extracted plurality of surrounding information. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The extraction means extracts the surrounding information by projecting the captured image based on the image coordinates of the index detected by the detection means. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) If the extraction means determines that it is not possible to obtain the surrounding information within an extraction range below a predetermined distance from the index detected by the detection means, it expands the extraction range or extracts the surrounding information using the captured images before and after in chronological order. An information processing device according to any one of configurations 1 to 6, characterized by the above. (Composition 8) If the peripheral information extracted by the extraction means and the peripheral information held by the first holding means differ with respect to the identified index, the identification means updates the peripheral information held by the first holding means with the peripheral information extracted by the extraction means. An information processing device according to any one of configurations 1 to 7, characterized by the above. (Composition 9) If the detection means cannot detect the index from the captured image, the identification means identifies the index by comparing the surrounding information extracted by the extraction means with the surrounding information held by the first holding means. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) At least one of the aforementioned indicator or the aforementioned peripheral information is output to the display device. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Composition 11) A detection means for detecting indicators placed in the real space using images acquired by imaging the real space, An acquisition means for acquiring peripheral information of the indicator using the aforementioned image, A means for identifying the indicator in such a way as to distinguish it from other indicators placed in the real space using the surrounding information, An information processing device characterized by comprising: (Composition 12) The information processing device described in any of configurations 1 to 11, the imaging device, and the display device, A display system characterized by comprising the following features. (Method 1) An information processing method for acquiring images from an imaging device that photographs a real space in which indicators are placed, and for estimating the relative position and orientation of the indicators, A detection step of detecting the indicator in the captured image and obtaining image coordinates indicating the position of the indicator in the captured image, An extraction step for extracting peripheral information of the indicator from the captured image, An identification step involves identifying the index by comparing the surrounding information extracted in the extraction step with the surrounding information held in the holding means in association with the image coordinates of the index. An estimation step is performed to estimate the three-dimensional geometric information of the index or the imaging device based on the index identified in the identification step, An information processing method characterized by comprising: (Method 2) A detection step that uses an image obtained by imaging the real space to detect an indicator placed in the real space, An acquisition step is to acquire information about the surrounding area of ​​the indicator using the aforementioned image, A process of identifying the indicator in such a way as to distinguish it from other indicators placed in the real space using the surrounding information, An information processing method characterized by comprising: (Composition 13) A program characterized by causing a computer to execute each step of the information processing method described in Method 1 or 2.

[0066] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0067] 200 Information Processing Devices 201 Indicator detection unit (detection means) 202 Indicator peripheral information extraction unit (extraction means) 203 Indicator Identification Unit (Identification Means) 204 Indicator peripheral information holding unit (first holding means) 206 Estimation section (estimation means) 210 Imaging device

Claims

1. An information processing device that acquires images from an imaging device that photographs a real space in which indicators are placed, and estimates the relative position and orientation of the indicators, A detection means for detecting the indicator in the captured image and obtaining image coordinates indicating the position of the indicator in the captured image, An extraction means for extracting peripheral information of the indicator from the captured image, A first storage means that stores the image coordinates of an indicator in association with surrounding information, An identification means for identifying the index by comparing the peripheral information extracted by the extraction means with the peripheral information held by the holding means, An estimation means for estimating the three-dimensional geometric information of the index or the imaging device based on the index identified by the identification means, An information processing device characterized by comprising:

2. The estimation means estimates the position and orientation of the imaging device. The information processing apparatus according to feature 1.

3. The system further comprises a second means for holding map information, The estimation means estimates the position and orientation of the imaging device using the image coordinates of the index identified by the identification means and the map information held by the second holding means. The information processing apparatus according to feature 2.

4. The extraction means extracts at least one of feature points, feature quantities, edges, patterns, and object information from the captured image as peripheral information. The information processing apparatus according to feature 1.

5. The extraction means extracts the surrounding information based on the relative positional relationship between the surrounding information and the index detected by the detection means, or the relative positional relationship between the extracted plurality of surrounding information. The information processing apparatus according to feature 1.

6. The extraction means extracts the surrounding information by projecting the captured image based on the image coordinates of the index detected by the detection means. The information processing apparatus according to feature 1.

7. If the extraction means determines that it is not possible to obtain the surrounding information within an extraction range below a predetermined distance from the index detected by the detection means, it expands the extraction range or extracts the surrounding information using the captured images before and after in chronological order. The information processing apparatus according to feature 1.

8. The identification means updates the peripheral information held by the first holding means with respect to the identified index if the peripheral information extracted by the extraction means differs from the peripheral information held by the first holding means. The information processing apparatus according to feature 1.

9. If the detection means cannot detect the index from the captured image, the identification means identifies the index by comparing the surrounding information extracted by the extraction means with the surrounding information held by the first holding means. The information processing apparatus according to feature 1.

10. At least one of the aforementioned indicator or the aforementioned peripheral information is output to the display device. The information processing apparatus according to feature 1.

11. A detection means for detecting indicators placed in the real space using images acquired by imaging the real space, An acquisition means for acquiring peripheral information of the indicator using the aforementioned image, A means for identifying the indicator in such a way as to distinguish it from other indicators placed in the real space using the surrounding information, An information processing device characterized by comprising:

12. An information processing device according to any one of claims 1 to 11, an imaging device, and a display device, A display system characterized by comprising the following features.

13. An information processing method for acquiring images from an imaging device that photographs a real space in which indicators are placed, and for estimating the relative position and orientation of the indicators, A detection step of detecting the indicator in the captured image and obtaining image coordinates indicating the position of the indicator in the captured image, An extraction step for extracting peripheral information of the indicator from the captured image, An identification step involves identifying the index by comparing the surrounding information extracted in the extraction step with the surrounding information held in the holding means in association with the image coordinates of the index. An estimation step is performed to estimate the three-dimensional geometric information of the index or the imaging device based on the index identified in the identification step, An information processing method characterized by comprising:

14. A detection step that uses an image obtained by imaging the real space to detect an indicator placed in the real space, An acquisition step is to acquire information about the surrounding area of ​​the indicator using the aforementioned image, A process of identifying the indicator in such a way as to distinguish it from other indicators placed in the real space using the surrounding information, An information processing method characterized by comprising:

15. A program characterized by causing a computer to execute each step of the information processing method described in claim 13 or 14.

Images