Information processing device, information processing method, and program

Abstract

To maintain the stability of position measurement even when natural feature points are scarce.SOLUTION: An information processing device comprises: derivation means for deriving the position of an imaging device using captured images and a three-dimensional map; evaluation means for evaluating the stability of the position on the basis of the information used by the derivation means to derive the position; creation means for creating the three-dimensional map on the basis of the captured images and the derived position; setting means for setting information indicative of the reliability of information indicative of the three-dimensional position of the features of the subject obtained on the basis of the captured images in an expansion region of a predetermined region in the three-dimensional map held by holding means on the basis of the evaluation results of the evaluation means and information indicative of the three-dimensional position of the features included in the predetermined region in the three-dimensional map; and correction means for correcting the three-dimensional map based on the information indicating reliability set by the setting means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Measurement of the position and orientation of an imaging device based on images is used for various purposes. One example of a purpose is alignment between a real space and a virtual object in mixed reality technology / augmented reality technology. Mixed reality is also called Mixed Reality (MR). Augmented reality is also called Augmented Reality (AR). Another example of a purpose is self-localization for autonomous movement of a robot or automatic driving of a car.

[0003] A known method for measuring the position and orientation of an image capturing device based on an image is to estimate the position and orientation from the correspondence between feature points detected from the image and a three-dimensional map that stores the three-dimensional coordinates of feature points in a scene.

[0004] In Patent Document 1, the reliability of feature points whose positions are known in advance (hereinafter referred to as "fixed feature points") is set high, and the contribution rate of these constraints is increased when correcting the position and orientation of the imaging device, thereby correcting the three-dimensional map, thereby attempting to achieve highly accurate position and orientation measurement. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-13560 [Non-patent literature]

[0006] [Non-Patent Document 1] Raul Mur-Artal et.al, ORB-SLAM:A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics [Non-patent document 2] C. Forster, M. Pizzoli, and D. Scaramuzza, SVO:fast semi-direct monocular visual odometry, Proc. 2014 IEEE International Conference on Robotics and Automation(ICRA), pp.15-22, 2014. [Non-patent document 3] Z.Zhang, A flexible new technique for camera calibration, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.2, no.11, pp.1330-1334, 2000. [Non-patent document 4] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana, Virtual object manipulation on a table-top AR environment, Proc. IEEE and ACM International Symposium on Augmented Reality 2000, pp.111-119, 2000. [Non-patent document 5] J.Engel, T.Schoeps, and D.Cremers, LSD-SLAM: Large-scale direct monocular SLAM, Proc. 14th European Conference on Computer Vision(ECCV), pp.834-849, 2014. Summary of the Invention [Problem to be solved by the invention]

[0007] In the method disclosed in Patent Document 1, when there are few feature points other than fixed feature points (hereinafter referred to as "natural feature points"), the stability of position measurement may decrease.

[0008] An object of the present invention is to maintain stability in position measurement even when there are few natural feature points. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present invention comprises an input means for inputting a captured image of a subject from an imaging device with a variable position and orientation; a storage means for storing a three-dimensional map including information indicating the three-dimensional positions of features possessed by the subject; a derivation means for deriving the position of the imaging device using the captured image and the three-dimensional map; an evaluation means for evaluating the stability of the position based on the information used by the derivation means to derive the position; a creation means for creating the three-dimensional map based on the captured image and the position; a setting means for setting, based on the results of the evaluation by the evaluation means, information indicating the reliability of information indicating the three-dimensional positions of features possessed by the subject, obtained based on the captured image in an extended area of a predetermined area in the three-dimensional map stored by the storage means, and information indicating the three-dimensional positions of features included in a predetermined area in the three-dimensional map; and a correction means for correcting the three-dimensional map based on the information indicating the reliability set by the setting means. [Effects of the Invention]

[0010] According to the present invention, it is possible to maintain stability in position measurement even when there are few natural feature points. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 2]1 is a block diagram showing a functional configuration of an information processing device according to a first embodiment of the present invention. [Figure 3] FIG. 1 is a diagram illustrating a three-dimensional map. [Figure 4] 3 is an example of a flowchart of an information processing method according to the first embodiment. [Figure 5] 10 is an example of a flowchart of a three-dimensional map correction process. [Figure 6] FIG. 10 is a conceptual diagram regarding correction of a three-dimensional map. [Figure 7] FIG. 10 is a diagram illustrating an auxiliary index. [Figure 8] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device 3 according to a third embodiment. [Figure 9] 10 is a flowchart illustrating an example of an information processing method according to a third embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a graphical user interface according to the fourth embodiment. [Figure 11] FIG. 1 is a diagram illustrating a problem to be solved by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following describes embodiments of the present invention with reference to the drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0013] [Embodiment 1] 1 is a block diagram showing the hardware configuration of an information processing device according to a first embodiment of the present invention. The information processing device 1 has a CPU 10, a ROM 20, a RAM 30, an input I / F 40, an output I / F 50, and a bus 60. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. I / F is an abbreviation for interface. The CPU 10 is an example of a computer that executes a computer program.

[0014] The CPU 10 controls each unit connected to the bus 60 via the bus 60. The input I / F 40 acquires an input signal from an external device (such as a display device or an operation device) in a format that can be processed by the information processing device 1. The output I / F 50 outputs an output signal to the external device (such as a display device) in a format that can be processed by the external device.

[0015] Programs for realizing the functions of the present invention are stored in a storage medium such as ROM 20, which is a read-only memory. ROM 20 also stores programs such as an OS and device drivers. OS is an abbreviation for Operating System. A random access memory such as RAM 30 temporarily stores these programs. CPU 10 then executes the programs stored in RAM 30, thereby performing the processes described below and realizing the functions of the present invention. Instead of software processing using CPU 10, the functions of the present invention can also be realized using hardware having a calculation unit or circuit corresponding to the processing of each functional unit.

[0016] The information processing device 1 acquires and processes images from an imaging device 180 (see FIG. 2). The imaging device 180 is assumed to be a monocular camera. The imaging device 180 is assumed to capture grayscale images. The type of imaging device 180 is not limited to these. The imaging device 180 has a variable position and orientation and acquires captured images of a subject. Hereinafter, a three-dimensional coordinate system with the optical center of the imaging device 180 as the origin, the optical axis direction as the Z axis, the horizontal direction of the image as the X axis, and the vertical direction of the image as the Y axis will be defined as the imaging device coordinate system or the imaging coordinate system. The position and orientation of the imaging device 180 refer to the position and orientation (e.g., the position of the origin and the direction of the Z axis) of the imaging coordinate system relative to a reference coordinate system (hereinafter referred to as the "world coordinate system") defined in the space (scene) where imaging is performed. The position and orientation of the imaging device have six degrees of freedom (three degrees of freedom for position and three degrees of freedom for orientation). In the following description, an object present in a scene will be referred to as the "subject." The type of subject is not particularly limited, and may be, for example, a building or a room.

[0017] The information processing device 1 matches feature points obtained from an image input from a camera, which is an imaging device 180, with feature points obtained from key frames of the three-dimensional map, calculates the position and orientation of the camera, and generates and modifies the three-dimensional map. An information processing device that implements so-called SLAM will be described. SLAM is an abbreviation for Simultaneous Localization and Mapping. For example, SLAM uses the method of Raul et al. (Non-Patent Document 1). The calculated position and orientation is used, for example, to control a moving object. The moving object may be, for example, an AMR or an AGV. AMR is an abbreviation for Autonomous Mobile Robot. AGV is an abbreviation for Automatic Guided Vehicle.

[0018] The problem to be solved by the present invention will now be described with reference to FIG. 11. FIG. 11 is a diagram for explaining the problem to be solved by the present invention. In FIG. 11, key frames E01 to E03 represent key frames before correction is performed, and key frame E03 is the most recent key frame. Key frames will be described later with reference to FIG. 3. Key frames E07 and E08 represent key frames after correction is performed. Note that key frame E07 is a key frame that was added when the imaging device observed an index. Feature points E04 and E05 represent feature points used for position measurement. Note that feature point E05 represents the feature point observed by key frame E07. When key frame E03 is added, map correction is performed on key frames E01 to E03.

[0019] If the reliability of index E06 is high, the position of key frame E07 is corrected to match index E06. Accordingly, the position of feature point E05 is also corrected to match the index. Meanwhile, the position of feature point E04 is also corrected, but because feature point E04 is not observed in key frame E07, it is not corrected to match the index, but rather is corrected to ensure consistency of observation information between key frames that observe feature point E04.

[0020] For example, key frames E01 to E03 have few feature points available for position measurement, and if the stability of the position measurement is low, errors accumulate in the key frames. In such a case, feature point E05 is corrected to match the index, and feature point E04 is corrected to maintain the consistency of the observation information from key frames E01 to E02 where errors have accumulated. As a result, consistency cannot be achieved between feature points E04 and E05, and there is a statistically high probability that one of the feature points will be an outlier when position measurement is performed. This reduces the number of feature points available for position measurement, and the stability of the position measurement decreases.

[0021] In contrast, in the first embodiment, the stability of the position and orientation of the image capture device 180 is evaluated. If the stability of the position and orientation of the image capture device 180 is high, the reliability of the portion of the 3D map of the scene that was created in advance is increased, and the reliability of the portion that was extended at runtime is decreased, and the 3D map is corrected. On the other hand, if the stability of the position and orientation of the image capture device 180 is low, the reliability of the portion of the 3D map of the scene that was created in advance is set lower than when the stability is high, and the 3D map is corrected.

[0022] 2 is a block diagram showing the functional configuration of an information processing device according to the first embodiment of the present invention. The information processing device 1 has a storage unit 110, an input unit 120, a derivation unit 130, an evaluation unit 140, a creation unit 150, a setting unit 160, and a correction unit 170. The input unit 120 is connected to an imaging device 180. The notification unit 190 notifies the operator in response to an instruction from the information processing device 1. The notification unit 190 is, for example, a display device.

[0023] The storage unit 110 stores a three-dimensional map of a scene that the derivation unit 130 uses to derive the position and orientation of the image capture device 180. In this embodiment, the three-dimensional map includes information indicating the three-dimensional positions of features of the subject. In one embodiment, the three-dimensional map also includes information indicating the observation results of the subject from each of a plurality of viewpoints. The storage unit 110 sequentially stores three-dimensional maps created in advance.

[0024] FIG. 3 is a diagram illustrating an example of a three-dimensional map. The three-dimensional map of a scene shown in FIG. 3 includes a collection of keyframes. A keyframe refers to an image captured by an imaging device at various locations in a scene. The keyframe contains information indicating the observation results of a subject from various viewpoints. For example, feature positions can be obtained from the keyframes through feature extraction processing, and color information at specific image positions can also be obtained.

[0025] The three-dimensional map in FIG. 3 also includes the position and orientation in the world coordinate system of the imaging device (i.e., the viewpoint) when capturing a key frame (hereinafter referred to as the "position and orientation of the key frame"). Furthermore, the three-dimensional map also includes the image coordinates (u, v) of features (feature points in this embodiment) on the key frame and a depth d (z coordinate in the imaging coordinate system of the key frame) based on the position and orientation of the key frame. These pieces of information represent information indicating the three-dimensional positions of the features of the subject. In this embodiment, the three-dimensional coordinates of the feature points in the imaging coordinate system are calculated from the image coordinates (u, v) and depth d of the feature points using a known method (for example, the method described in Non-Patent Document 2). The three-dimensional coordinates of the feature points calculated in this way are then used for measuring the position and orientation of the imaging device 180, which will be described later.

[0026] Returning to the description of FIG. 2, the storage unit 110 stores, as a three-dimensional map, not only the key frames but also the positions and orientations of the key frames and three-dimensional information on geometric features on the key frames. Such a three-dimensional map may be generated using known techniques. In this embodiment, a three-dimensional map of a scene is created in advance. The storage unit 110 acquires the three-dimensional map created in advance and stored in a storage unit (not shown). Hereinafter, the part of the three-dimensional map created in advance will be referred to as the "pre-map." That is, the pre-map includes information created in advance indicating the three-dimensional positions of features (sometimes referred to as first features). Meanwhile, information indicating the three-dimensional positions of features (sometimes referred to as second features) is also added to the three-dimensional map by the creation unit 150, which will be described later.

[0027] In this embodiment, the pre-map includes multiple key frames created in advance, the position and orientation of each key frame, and information about feature points in each key frame. Here, the feature point information includes the image coordinates (u, v) of the feature points on the key frame and the depth d of the feature points. On the other hand, in this embodiment, the three-dimensional map is expanded at runtime (when the information processing device 1 measures the position of the image capture device 180). For example, the three-dimensional map is expanded when, for example, there is a possibility that the image capture device 180 will move out of the range in which the position and orientation can be measured using the pre-map, depending on a predetermined condition according to the position and orientation of the image capture device 180. That is, a new key frame, the position and orientation of this key frame, and information about the feature points in this key frame are added to the three-dimensional map. This method allows the position and orientation of the image capture device 180 to be continuously measured. For example, if an obstacle is present in the area for which the pre-map was created and the image capture device 180 avoids this obstacle, the image capture device 180 may move out of the range in which the position and orientation can be measured. Furthermore, when the image capture device 180 moves outside the range in which the position and orientation can be measured using the prior map, the image capture device 180 may fall outside the range in which the position and orientation can be measured.

[0028] The input unit 120 acquires an image (hereinafter referred to as an "input image") captured by the imaging device 180. The input unit 120 can acquire moving images from the imaging device 180, and acquires time-series images at, for example, 30 frames per second. Note that a storage unit (not shown) of the information processing device 1 holds internal parameters of the imaging device 180 (focal length, image center position, lens distortion parameters, etc.). The internal parameters of the imaging device 180 are calibrated in advance using a known method (for example, Non-Patent Document 3).

[0029] The derivation unit 130 uses the input image and the three-dimensional map to derive the position and orientation of the image capturing device 180. The derivation unit 130 derives the position and orientation of the image capturing device 180 at the time of image capturing for each image input in time series from the input unit 120.

[0030] The evaluation unit 140 calculates the stability of the position and orientation of the image capture device 180 derived by the derivation unit 130 using the information used by the derivation unit 130 to derive the position and orientation.

[0031] The creation unit 150 adds information indicating the three-dimensional positions of additional features of the subject, which is obtained based on the input image and the position and orientation of the image capture device 180, to the three-dimensional map. In this way, the three-dimensional map is expanded. In this embodiment, the creation unit 150 expands the three-dimensional map by adding key frames and information indicating the three-dimensional positions of additional features included in the key frames (information on the position and orientation of the key frames and feature points). The creation unit 150 adds key frames, for example, when there is a possibility that the position of the image capture device 180 may be outside the range in which the position and orientation can be measured using the existing three-dimensional map.

[0032] The setting unit 160 sets reliability (information indicating reliability) for information indicating the three-dimensional position of a feature. In this embodiment, a higher reliability is assigned to information indicating the three-dimensional position of a feature (first feature) included in the pre-map than to information indicating the three-dimensional position of a feature (second feature) added by the creation unit 150. In this embodiment, the setting unit 160 sets the reliability of keyframes included in the three-dimensional map. That is, the setting unit 160 sets a high reliability for keyframes included in the pre-map and a low reliability for keyframes added by the creation unit 150 at runtime. On the other hand, the setting unit 160 changes the reliability of keyframes included in the pre-map based on the magnitude of the stability calculated by the evaluation unit 140. That is, when the stability is low, the setting unit 160 sets the reliability of keyframes included in the pre-map to be lower than when the stability is high.

[0033] The correction unit 170 corrects information indicating the three-dimensional positions of features included in the three-dimensional map based on the reliability. For example, the correction unit 170 updates information indicating the three-dimensional positions of features included in the three-dimensional map according to the reliability. By performing such processing, the correction unit 170 can improve the consistency of the three-dimensional map. This processing is known as optimizing the three-dimensional map (or pose graph). In this embodiment, the correction unit 170 updates the information indicating the three-dimensional positions of features by correcting the positions and orientations of keyframes so as to achieve consistency across the entire three-dimensional map. The correction unit 170 performs the correction based on the reliability of the keyframes set by the setting unit 160. In this embodiment, the correction unit 170 fixes the positions and orientations of keyframes included in the prior map. This makes it possible to expand the range in which the position and orientation of the image capture device 180 can be measured while maintaining accuracy.

[0034] Next, processing by the information processing device 1 according to this embodiment will be described with reference to the flowchart in FIG. 4. The information processing device 1 executes initialization processing at the start of a series of processing. As an example of this initialization processing, the storage unit 110 reads a pre-map from an external storage device (not shown). The method for creating the pre-map is not particularly limited. For example, the pre-map is created using a plurality of captured images (key frames) and the positions and orientations of the viewpoints of the captured images (positions and orientations of the key frames). The image coordinates (u, v) and depth d of the feature points are determined by extracting features from the captured images and matching between the captured images. An example of a method for creating the pre-map is SLAM technology. In this embodiment, the pre-map is created using the method of Raul et al. (Non-Patent Document 1).

[0035] The storage unit 110 also stores the reliability B of each key frame included in the pre-map. i (i=1, , Np) high, where Np is the number of keyframes contained in the premap.

[0036] In this embodiment, the reliability of a key frame is an index indicating whether the position and orientation of the key frame are reliable. If the reliability of a key frame is high, the contribution rate to the correction performed by the correction unit 170 is large. For a key frame included in the pre-map, the correction unit 170 does not correct the position and orientation of this key frame. The position and orientation of a key frame included in the pre-map is used as a constraint for correcting the position and orientation of a key frame generated at runtime.

[0037] The pre-map may be prepared over time. For example, the pre-map may be created in a static state where no moving objects, such as people or automobiles, are in the field of view. Alternatively, the pre-map may be created by performing an optimization process that would take too long to be performed in real time. In this manner, a highly accurate pre-map can be created. Therefore, in this embodiment, the reliability of keyframes included in the pre-map is set to be high.

[0038] In step S1010, the input unit 120 acquires an input image captured by the imaging device 180. The input unit 120 acquires one frame of the input image.

[0039] In step S1020, the derivation unit 130 derives the position and orientation of the imaging device 180 when the input image was captured, using the 3D map stored in the storage unit 110 and the input image. Various known methods can be used as the derivation method. For example, the position and orientation are repeatedly corrected so as to reduce the difference between the image positions of the feature points on the input image calculated based on the 3D positions of the feature points and the derived position and orientation, and the image positions of the feature points on the input image. In this way, the derivation unit 130 can derive the position and orientation of the imaging device 180. The derivation unit 130 may also derive the relative position and orientation between the key frame and the imaging device 180 so as to reduce the color difference (e.g., luminance difference) between the feature points of the key frame and the corresponding points on the input image that correspond to the feature points determined based on the 3D positions of the feature points. The derivation unit 130 acquires the position and orientation of the imaging device 180 from this relative position and orientation and the position and orientation of the key frame. Here, the three-dimensional positions of the feature points are determined from the position and orientation of the key frame and information on the feature points (image coordinates (u, v) and depth d). In this embodiment, the position and orientation derivation method disclosed in the method by Raul et al. (Non-Patent Document 1) is used.

[0040] In step S1030, the evaluation unit 140 calculates the stability depending on the number of feature points used by the derivation unit 130 to derive the position and orientation of the image capture device 180 in step S1020. For example, the evaluation unit 140 increases the stability the more feature points used to derive the position and orientation, and decreases the stability the fewer feature points used. As a specific calculation example, assuming that the stability is expressed from 0 to 1, stability 0 corresponds to M feature points and stability 1 corresponds to N feature points, the stability can be calculated using the following formula (1): Stability = (number of feature points - M) / (NM) Equation (1) However, if the stability is 1 or greater, it is set to 1, and if it is 0 or less, it is set to 0.

[0041] In step S1040, the creation unit 150 creates (expands) a three-dimensional map using the position and orientation of the image capture device 180 derived in step S1020. The creation unit 150 expands the three-dimensional map by adding key frames to the three-dimensional map. The creation unit 150 also expands the three-dimensional map when a predetermined condition is met. Details of the processing in step S1040 will be described later. The number of key frames in the three-dimensional map after the key frames are added in step S1040 is set to N. k Let's say.

[0042] In step S1050, the setting unit 150 calculates the reliability B of the key frame added to the three-dimensional map in step S1040. j (j=N k ) is set low. Here, the reliability of a keyframe refers to the reliability of information stored in the 3D map in relation to the keyframe (the position and orientation of the keyframe, or information on feature points in the keyframe). At runtime, it is difficult to control the situation of the scene and maintain a static state (for example, to prevent moving objects from entering the field of view), so the reliability of the keyframe is limited. For this reason, in this embodiment, the reliability of keyframes added at runtime is set low. On the other hand, the reliability of keyframes included in the pre-map is set as a new reliability value obtained by multiplying the stability calculated in step S1030 by the reliability already set for the keyframe in the pre-map.

[0043] In step S1060, correction unit 170 corrects the three-dimensional map. For example, correction unit 170 corrects the positions and orientations of key frames included in the three-dimensional map so that the entire three-dimensional map is consistent. While various known correction methods can be used, in this embodiment, correction is performed according to the method by Raul et al. (Non-Patent Document 1).

[0044] On the other hand, in this embodiment, the positions and orientations of keyframes included in the pre-map are not subject to correction, and only the positions and orientations of keyframes generated at runtime and information indicating the three-dimensional positions of features are updated.

[0045] In this way, the correction unit 170 can improve the consistency of the three-dimensional map by updating the information indicating the three-dimensional positions of features while fixing the information indicating the three-dimensional positions of features that have been created in advance but not created by the creation unit 150. This makes it possible to expand the measurable range of the position and orientation without changing the pre-map created with high accuracy. However, if the reliability B i The degree of constraint by the prior map changes depending on the size of the map. Specifically, if the stability of the position and orientation of the image capture device 180 is low, the reliability is low, so the influence of the prior map decreases, and the influence of the 3D map generated at runtime increases. As a result, although the accuracy of the position and orientation temporarily decreases, stability is improved. Details of the processing in step S1060 will be described later.

[0046] In step S1070, the derivation unit 130 determines whether to terminate position and orientation measurement. For example, the position and orientation measurement terminates when a user inputs an end instruction from an input device such as a mouse or a keyboard via the input I / F 40. If the measurement does not terminate, the process returns to step S1020, and the position and orientation measurement continues.

[0047] (Details of the process in step S1040) In step S1040, creation unit 150 first determines whether to expand the three-dimensional map, i.e., whether to add a new key frame to the three-dimensional map. Here, creation unit 150 makes the following determination based on the position and orientation of image capture device 180 derived in step S1030.

[0048] First, the creation unit 150 selects a keyframe (hereinafter referred to as a "nearest keyframe" or simply a "neighboring keyframe") in the three-dimensional map based on the derived position and orientation of the image capture device 180. For example, the creation unit 150 selects a neighboring keyframe according to a predetermined condition based on the derived position and line of sight direction of the image capture device 180 (the Z-axis direction of the image capture coordinate system in the world coordinate system) and the position and line of sight direction of the keyframe. The creation unit 150 selects a keyframe having a position and line of sight direction close to that of the image capture device 180 as a neighboring keyframe. As an example, the creation unit 150 selects a keyframe group from the three-dimensional map based on the visual axis direction of the image capture device 180. Here, the angular difference between the visual axis direction of the image capture device 180 and the visual axis direction of the selected keyframe group in the world coordinate system is within a threshold value Tv. Next, the creation unit 150 selects a neighboring keyframe from the keyframe group. Here, a neighboring keyframe is a keyframe included in the keyframe group whose position is closest to the position of the image capture device 180.

[0049] Next, whether or not to add a new key frame is determined based on the number of feature points of neighboring key frames included in the input image acquired in step S1010. For example, the creation unit 150 calculates the image coordinates of each feature point of the neighboring key frames on the input image acquired in step S1010. For example, in order to calculate the image coordinates, the creation unit 150 first calculates the three-dimensional coordinates X of the feature points in the image capture coordinate system of the neighboring key frames using the method described above. Key Next, the creation unit 150 calculates the three-dimensional coordinates X Key is expressed as the three-dimensional coordinate X Cam Finally, the creation unit 150 converts the three-dimensional coordinates X Cam into image coordinates (u, v) of the input image. In this way, the creation unit 150 calculates the ratio R of feature points whose calculated image coordinates are included in the input image. inc R incWhen R is small, there is little overlap between the nearest keyframe and the input image, so the image capture device may fall outside the position and orientation measurement range. inc is the threshold T inc If it is less than this, the creating unit 150 determines to add a new key frame.

[0050] When it is determined that a new key frame should be added, the creation unit 150 adds the input image as a new key frame using the method of Raul et al. (Non-Patent Document 1). When using the method of Raul et al. (Non-Patent Document 1), feature point information of the new key frame is created by projecting and propagating feature point information on the immediately preceding key frame (or on a nearby key frame) onto the input image. For example, the three-dimensional coordinates of the feature points are obtained from the feature point information of nearby key frames, and these are projected onto the input image to determine the image coordinates (u, v) and depth d of the feature points of the new key frame.

[0051] On the other hand, if it is not determined that a new key frame should be added, the creation unit 150 updates the feature point information (image coordinates (u, v) and depth d) of the key frame created immediately before. For example, the creation unit 150 can add new feature point information or update the depth d information by extracting features from captured images and matching between captured images. This process may be performed using, for example, the method of Raul et al. (Non-Patent Document 1).

[0052] (Details of the process in step S1060) FIG. 5 is a flowchart showing the processing procedure for 3D map correction in step S1060. In the optimization process, information indicating the 3D position of a feature is updated so as to reduce the error between the observation result observed at the viewpoint (position and orientation of the key frame) when the feature is located in a 3D position and the observation result actually observed at the viewpoint. In this embodiment, information indicating the 3D position of the feature and the position and orientation of the key frame are updated so as to reduce the error (reprojection error) between the detected position of the feature on the image and the position of the feature projected onto the image using the estimated position and orientation of the imaging device. Specifically, when a new key frame is added, the 3D position of the feature point and the position and orientation of the key frame are updated so as to minimize the reprojection error of the feature point for each key frame for a predetermined number of key frames immediately preceding the new key frame. These processes correspond to the local bundle adjustment described in the method of Raul et al. (Non-Patent Document 1).

[0053] After performing these processes in steps S1210 to S1220, the positions and orientations of the new and existing key frames and the three-dimensional positions of the features observed by each key frame are updated. The process of step S1060 will now be described in detail.

[0054] In step S1210, the correction unit 170 searches for a predetermined number of key frames from among the previously added key frames, whose positions and orientations are close to those of the key frame newly added in step S1040. Hereinafter, the key frame newly added in step S1040 will be referred to as a "new key frame." For example, the correction unit 170 searches for a new key frame when the angle difference in the visual axis direction in the world coordinate system (the Z axis of the image capture coordinate system) is equal to or greater than a threshold T Angle and the position difference is within the threshold T Dist Select a predetermined number of keyframes that are within the range.

[0055] In step S1220, the correction unit 170 performs local bundle adjustment described in the method of Raul et al. (Non-Patent Document 1) on the keyframe selected in step S1220. In step S1220, the correction unit 170 thereby updates the positions and orientations of keyframes including the new keyframe and the information on the three-dimensional positions of feature points. In this way, the correction unit 170 corrects the three-dimensional map. In step S1220, only keyframes generated at runtime are corrected, and the positions and orientations of keyframes included in the pre-map are not corrected.

[0056] To correct the position and orientation and the three-dimensional positions of the feature points, the error between the projected position of the feature point projected onto the image using the position and orientation in the world coordinate system that the key frame has as an attribute and the detected position of the feature point detected on the corresponding image is used. In this case, the sum of these reprojection errors is used as the evaluation function. In this way, the position and orientation of the key frame are updated so that the three-dimensional positions of the feature points included in the three-dimensional map are consistent with the position and orientation of the key frame.

[0057] The correction unit 170 performs correction so that the following formula (2) is minimized. Formula (2) is the sum of the key frames selected in step S1210 from all key frames included in the 3D map. Information about the key frames included in the pre-map is used to calculate the relative positions and orientations between key frames, which is necessary to correct the positions and orientations of key frames generated at runtime. Σ (reliability of keyframe) (reprojection error of feature points) Equation (2)

[0058] To minimize equation (2), for example, the Gauss-Newton method, which performs iterative calculations, is used. The position and orientation derived in step S1020 are used as the initial values of the position and orientation for the new key frame. The corrected position and orientation are saved again as attributes of the key frame in the 3D map, and are used as initial values the next time the 3D map is corrected. In addition, the correction unit 170 can change the contribution rate of each constraint condition by multiplying the error by the reliability of each key frame. That is, if the reliability is high, the contribution rate will be high, and if the reliability is low, the contribution rate will be low.

[0059] FIGS. 6A and 6B are conceptual diagrams illustrating correction of a 3D map. FIG. 6A illustrates correction according to a conventional technique. Assume that a pre-map is created as shown in the leftmost diagram in FIG. 6A. When the image capture device moves out of the range where position and orientation can be measured using the pre-map, map data is added as shown in the center diagram in FIG. 6A. No consistency processing is performed here. Next, as shown in the rightmost diagram in FIG. 6A, feature positions are adjusted based on the pre-map and the expanded map (expanded region) as a whole. Furthermore, because the pre-map contributes significantly to position calculation, keyframes added at runtime that share features with the pre-map are corrected to be consistent with the pre-map. On the other hand, if the total number of feature points is small, keyframes that do not share features with the pre-map are not corrected to be consistent with the pre-map because the number of shared feature points between keyframes added at runtime is small. As a result, inconsistencies occur between keyframes that share features with the pre-map and keyframes that do not. Since the position is measured using the feature points of nearby key frames including the key frame where the mismatch occurred, the number of feature points available for position measurement decreases, and stability decreases.

[0060] FIG. 6B is a diagram illustrating correction according to this embodiment. In this embodiment, when the number of feature points used for position measurement is small, the contribution rate of the pre-position map to position calculation is set to be approximately the same as that of the map added at runtime, as shown in the left diagram of FIG. 6B. In this embodiment, doing so maintains consistency between keyframes added at runtime, leading to stable position measurement. Furthermore, in this embodiment, when the number of feature points used for position measurement is large, the contribution rate of the pre-position map to position calculation is increased, as shown in the right diagram of FIG. 6B. In this embodiment, doing so corrects the keyframes added at runtime so that they are consistent with the pre-position map, thereby improving accuracy. In other words, according to this embodiment, it is possible to improve accuracy while maintaining stability of position measurement.

[0061] As described above, in the first embodiment, whether to set the reliability of a previously created portion of a three-dimensional map high is determined based on the stability of the position and orientation, and the three-dimensional map is corrected accordingly. Therefore, when the stability is high, correction is performed to maintain consistency with the prior map, improving accuracy. On the other hand, when the stability is low, accuracy temporarily decreases, but stability is improved and position measurement can be continued. In other words, in position and orientation measurement of an image capture device that uses previously known three-dimensional information, the stability of position measurement can be maintained even when the number of natural feature points is small.

[0062] [Embodiment 2] In the first embodiment, the reliability of a portion of a three-dimensional map of a scene that was created in advance is increased, thereby expanding the measurable range of the position and orientation while maintaining the accuracy of the three-dimensional map. In the second embodiment, auxiliary indices (hereinafter referred to as auxiliary indices) are arranged in the space (scene) in which the subject is located to measure the position and orientation of the imaging device 180 based on a captured image. These auxiliary indices are separate from features that originally exist in the scene. Then, information indicating the three-dimensional position of the feature is given a reliability based on the information of the auxiliary indices. For example, information indicating the three-dimensional position of a feature whose coordinates are known using the auxiliary indices is given a higher reliability than features that originally exist in the scene. In this way, the reliability of the three-dimensional map based on the auxiliary indices is increased. In the second embodiment, this method allows the measurable range of the position and orientation to be expanded while maintaining the accuracy of the three-dimensional map.

[0063] The configuration and processing of an information processing device according to this embodiment will be described with reference to Figures 1 and 2. The configuration and processing of the information processing device according to this embodiment are similar to those of embodiment 1, and only the differences will be described below.

[0064] FIG. 7 is a diagram illustrating auxiliary indices. In this embodiment, a three-dimensional map of the scene created in advance is not used. Instead, as shown in FIG. 7, auxiliary indices (markers) are placed in the scene. In this embodiment, auxiliary indices of a predetermined shape are used, each bearing an identifier that can be read by image analysis. In the example of FIG. 7, black and white square indices with individual identifiers inside are placed as auxiliary indices. The placement information of the auxiliary indices, i.e., the positions and orientations of the auxiliary indices in the world coordinate system, are calibrated in advance. Examples of calibration methods include the methods disclosed in Japanese Patent No. 4,532,982 or U.S. Patent No. 7,529,387. The setting unit 160 holds the calibrated placement information for the auxiliary indices placed in the scene.

[0065] The derivation unit 130 derives the position and orientation of the image capture device 180 in the same manner as in the first embodiment. However, in this embodiment, when the information processing device 1 is started up, no keyframes of the 3D map are included, so the derivation unit 130 derives the position and orientation of the image capture device 180 using auxiliary markers. For example, the derivation unit 130 derives the position and orientation of the image capture device according to the positions of auxiliary markers detected from the input image and the arrangement information of the auxiliary markers. In this embodiment, the method of Kato et al. (Non-Patent Document 4) is used as an example.

[0066] The setting unit 160 sets reliability (information indicating reliability) for information indicating the three-dimensional position of a feature. In this embodiment, a higher reliability is assigned to information indicating the three-dimensional position of a feature (first feature) whose coordinates on an auxiliary index are known than to information indicating the three-dimensional position of a feature (second feature) added by the creation unit 150.

[0067] The correction unit 170 corrects information indicating the three-dimensional positions of features included in the three-dimensional map based on the reliability of the information. For example, the correction unit 170 updates information indicating the three-dimensional positions of features included in the three-dimensional map according to the reliability of the information. Through this process, the correction unit 170 can improve the consistency of the three-dimensional map. This process is known as optimizing the three-dimensional map (or pose graph). In this embodiment, the correction unit 170 updates the information indicating the three-dimensional positions of features by correcting the positions and orientations of key frames so as to ensure consistency across the entire three-dimensional map. For example, the correction unit 170 fixes features whose coordinates on auxiliary markers are known. This makes it possible to expand the range in which the position and orientation of the image capture device 180 can be measured while maintaining accuracy.

[0068] 4, the processes of steps S1010 to S1040 and step S1070 of this embodiment are the same as those of embodiment 1. The processes of steps S1050 to S1060 according to this embodiment will be described below.

[0069] Upon initialization, the holding unit 110 acquires the arrangement information of the pre-calibrated auxiliary indicators from an external storage device (not shown).

[0070] In step S1050, the setting unit 160 calculates the reliability of the information indicating the three-dimensional position of the feature on the auxiliary indicator based on the stability (a value from 0 to 1) of the position and orientation of the imaging device 180 calculated in step S1030. Specifically, the reliability in the case of stability 1 is set in advance. For example, when the reliability of the information indicating the three-dimensional position of the feature generated at runtime is A, and the reliability of the information indicating the three-dimensional position of the feature on the auxiliary indicator is set to a reliability B (A < B) higher than the reliability A. For this reliability B, the value obtained by multiplying the stability is finally calculated as the reliability of the contribution degree of correction.

[0071] The correction of the position and orientation of the key frame and the three-dimensional position of the feature points is obtained by finding the position and orientation that minimizes the sum of the errors represented by the following equation (3). Equation (3) is the sum for the feature points included in the key frame selected in step S1210 among all the feature points included in the three-dimensional map. Σ (reliability of feature points) · (reprojection error of feature points) ··· Equation (3)

[0072] For minimizing equation (3), for example, the Gauss-Newton method that performs iterative calculations is used. Also, by multiplying the reliability of each feature point by the reprojection error, the contribution rate of each constraint condition can be changed. That is, when the reliability is high, the contribution rate becomes high, and when the reliability is low, the contribution rate becomes low.

[0073] Thereby, when the stability is high, the accuracy is improved by matching the features of the auxiliary indicator, and when the stability is low, although the accuracy temporarily decreases, the number of feature points available for position measurement increases, enabling continuous position measurement.

[0074] In step S1060, correction unit 170 corrects the three-dimensional map. For example, correction unit 170 corrects the positions and orientations of key frames included in the three-dimensional map so that the entire three-dimensional map is consistent. While various known methods can be used as the correction method, in this embodiment, correction is performed according to the method of Raul et al. (Non-Patent Document 1).

[0075] On the other hand, in this embodiment, the information indicating the three-dimensional position of the first feature is not updated, but the information indicating the three-dimensional position of the second feature generated at runtime is updated. In this way, the correction unit 170 updates the information indicating the three-dimensional position of the feature while fixing the information indicating the three-dimensional position of the feature on the auxiliary marker whose coordinates are known in advance, thereby improving the consistency of the three-dimensional map. Therefore, according to this embodiment, the position and orientation measurement range can be expanded without changing the information of the highly accurate auxiliary marker.

[0076] As described above, in the second embodiment, whether to set the reliability of feature points whose coordinates on auxiliary markers are known high is determined based on the stability of the position and orientation, and the 3D map is corrected accordingly. Therefore, when the stability is high, correction is performed to maintain consistency with the auxiliary markers, improving accuracy. When the stability is low, accuracy temporarily decreases, but stability is improved and position measurement can be continued. In other words, in position and orientation measurement of an image capture device that uses three-dimensional information known in advance, the stability of position measurement can be maintained even when the number of natural feature points is small.

[0077] (Modifications of Embodiments 1 and 2) Here, a modified example of a variation of the stability evaluation method will be described. For example, the stability may be lowered as the degree of deviation in the distribution of features on the image becomes smaller.

[0078] In the first and second embodiments, the evaluation unit 140 calculates the stability of the position and orientation of the image capture device 180 based on the number of natural feature points, but this is not necessarily limited to this. For example, the stability may be calculated based on the degree of bias in the distribution of feature points in the image. As a specific example, the captured image is divided into a 16 × 16 grid, and whether or not a feature point is present in each grid is determined. Then, the difference between the maximum and minimum grid coordinates in the X-axis (horizontal axis) direction of the image is multiplied by the difference between the maximum and minimum grid coordinates in the Y-axis (vertical axis) direction of the image to calculate a value representing the spread of the feature point distribution. In other words, the larger the value representing the spread of the feature point distribution, the smaller the bias in the feature distribution and the higher the stability. On the other hand, the smaller the value representing the spread of the feature point distribution, the larger the bias in the feature distribution and the lower the stability. Furthermore, the method for calculating the bias in the feature distribution may be any method that can express the degree of variation in the features in the image.

[0079] As another modification, the stability may be lowered as the number of feature points with large depth variance decreases. That is, the stability may be calculated based on the number of feature points with large depth variance detected in the captured image. The depth variance of feature points may be calculated using the method described in Non-Patent Document 5.

[0080] As another modification, the greater the change in luminance over time, the lower the stability may be. That is, the stability may be calculated based on the amount of change in luminance of the captured image over time. As a specific example, the average luminance of the captured image in the most recent five frames may be stored, and the greater the variance of the average luminance, the lower the stability may be calculated.

[0081] As another modification, the stability may be lowered as the amplitude of the position increases over time. That is, the stability may be calculated based on the change in the position of the image capture device over time. As a specific example, the positions of the image capture device in the most recent N frames may be stored, and the stability may be calculated to be lower as the variance of those positions increases.

[0082] As another modification, the stability may be lowered as the number of moving objects in an image increases. That is, the stability may be calculated based on the proportion of the moving objects in the captured image. As a specific example, a recognition model using deep learning may be used to recognize moving objects such as people, and calculate the area of the moving objects in the image. Then, the proportion of the moving object's area in the image may be calculated, and the stability may be calculated to be lower as the proportion increases.

[0083] As another modified example, the stability may be lowered as the velocity and angular velocity increase. That is, the stability may be calculated based on the velocity and angular velocity values of the image capture device 180 calculated from the position and orientation of the image capture device 180. As a specific example, the velocity and angular velocity may be calculated from the difference between the position and orientation of the latest frame and the previous frame, and the stability may be lowered as these values increase.

[0084] As another modification, the greater the difference with the position measurement results of other sensors, the lower the stability may be. In the first and second embodiments, the sensor is only a camera, but this is not necessarily the case, and the present invention may include multiple sensors. The information processing device may then calculate the difference with the position measurement results of other sensors, and calculate such that the greater the difference, the lower the stability. As a specific example, a system including an inertial sensor capable of measuring acceleration and angular velocity in addition to a camera will be described. The difference between the amount of movement calculated from the position and orientation of the latest frame and the previous frame derived by the derivation unit 130 and the amount of movement calculated by integrating the acceleration values of the inertial sensor for that section may be calculated, and the greater the difference, the lower the stability may be calculated.

[0085] Here, a modified example of the setting method will be described. For example, the higher the stability, the higher the reliability of the marker feature. In the first and second embodiments, the setting unit 160 set the reliability of the keyframes of the pre-map and the reliability of the information on the three-dimensional position of the feature whose coordinates on the feature of the auxiliary marker are known to be lower as the stability is lower. In the present invention, these reliability may be set to be higher as the stability is higher.

[0086] Here, we will explain modified examples of variations in indicators. The present invention can also use, for example, BIM or CAD as indicators. BIM is an abbreviation for Building Information Modeling, and CAD is an abbreviation for Computer Aided Design. In the first and second embodiments, auxiliary features such as preliminary maps and auxiliary indicators are listed as highly reliable indicators, but this is not necessarily limited to these as long as the three-dimensional information is highly accurate. For example, the accuracy of the three-dimensional map can be improved by aligning the information of an accurate three-dimensional model such as BIM or CAD with the information of the three-dimensional map to be generated and setting a high reliability for the features obtained from the three-dimensional model.

[0087] [Embodiment 3] In the first and second embodiments, a method for generating a stable and accurate three-dimensional map was described based on highly reliable information such as a prior map and auxiliary features and the stability of the position and orientation of the image capture device 180. In the third embodiment, a case will be described in which constraints from a prior map and auxiliary features are used when deriving the position and orientation of the image capture device 180.

[0088] The configuration and processing of an information processing device according to this embodiment will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing an example of the functional configuration of an information processing device 3 according to embodiment 3 of the present invention. Fig. 9 is an example of a flowchart of an information processing method according to embodiment 3 of the present invention. The configuration and processing of the information processing device according to this embodiment are similar to those of embodiment 2, and differences will be described below.

[0089] The storage unit 110 stores the placement information of the auxiliary features as in the second embodiment, and also stores the three-dimensional map that the creation unit 150 creates at runtime.

[0090] The derivation unit 130 uses the input image, the three-dimensional map, and the arrangement information of the auxiliary features to derive the position and orientation of the image capture device 180 based on the reliability of the information indicating the three-dimensional positions of the features included in the three-dimensional map.

[0091] Next, processing according to this embodiment will be described with reference to the flowchart in Fig. 9. In the initialization processing, the holding unit 110 acquires arrangement information of auxiliary markers calibrated in advance from an external storage device (not shown).

[0092] Step S2010 and steps S2030 to S2060 are similar to steps S1010, steps S1030 to S1050, and step S1070, respectively, and therefore their explanation will be omitted.

[0093] In step S2020, the derivation unit 130 derives the position and orientation of the imaging device 180 when the input image was captured, using the three-dimensional map stored in the storage unit 110 and the input image. As a derivation method, for example, a method of iteratively correcting the position and orientation so as to reduce the difference (reprojection error) between the image position of the feature point on the input image calculated based on the three-dimensional position of the feature point and the derived position and orientation, and the image position of the feature point on the input image. The position and orientation are calculated by reflecting the reliability of the information on the three-dimensional position of the feature set by the setting unit 160 as the contribution to the iterative calculation. Specifically, a position and orientation is found that minimizes the sum of errors expressed by the following equation (4). Σ(reliability of feature points) (reprojection error of feature points) Equation (4)

[0094] This increases the contribution rate of highly reliable feature points, such as auxiliary features, to position and orientation calculation, improving the accuracy of position and orientation measurement. On the other hand, when stability is low, the reliability of the auxiliary features is reduced based on the stability calculated by the evaluation unit 140, thereby maintaining the number of feature points available for position measurement and allowing position measurement to continue.

[0095] As described above, in the third embodiment, the reliability of information indicating the three-dimensional positions of features whose coordinates on auxiliary markers in a three-dimensional map are known is adaptively changed based on the stability of the position and orientation of the image capture device to derive the position and orientation of the image capture device. Therefore, when stability is high, accuracy can be maintained by matching the coordinates to auxiliary features whose coordinates are known. On the other hand, when stability is low, a decrease in the number of feature points due to excessive matching to auxiliary features can be suppressed by relaxing the constraints on the auxiliary features, thereby maintaining stability.

[0096] [Embodiment 4] In the first, second, and third embodiments, the information processing device 1 or 3 may display, on the GUI of the notification unit 190, information about side effects that occur due to lowering the reliability of the prior map or auxiliary features based on the stability of the imaging device 180. GUI is an abbreviation for Graphical User Interface.

[0097] FIG. 10 is a diagram illustrating an example of a GUI according to the fourth embodiment. For example, as shown in display 11a in FIG. 10, a message may be displayed indicating that the position measurement accuracy will decrease due to a temporary mismatch with the auxiliary features. Furthermore, as shown in display 11b, advice to improve stability may be provided to the user. In display 11b, the notification unit 190 may, for example, notify the user to slowly look around the surrounding environment with the camera. This increases the number of feature points in the three-dimensional map, which makes it easier to improve the stability of position and orientation derivation. Furthermore, the notification unit 190 may notify the user to move away from the auxiliary indicator. This increases the number of natural feature points in the field of view, which makes it easier to improve the stability of position and orientation derivation. Furthermore, the notification unit 190 may notify the user to point the camera in a direction with many natural features, such as objects and patterns. This increases the number of natural feature points in the field of view, which makes it easier to improve the stability of position and orientation derivation.

[0098] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0099] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention.

[0100] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) an input means for inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storage means for storing a three-dimensional map including information indicating three-dimensional positions of features of the subject; a deriving means for deriving a position of the imaging device using the captured image and the three-dimensional map; evaluation means for evaluating the stability of the position based on information used by the derivation means to derive the position; a creating means for creating the three-dimensional map based on the captured image and the position; Based on the result of the evaluation by the evaluation means, information indicating three-dimensional positions of features of the subject, obtained based on the captured images in an expanded region of a predetermined region in the three-dimensional map stored in the storage means; and a setting means for setting information indicating the reliability of information indicating the three-dimensional positions of features included in a predetermined area in the three-dimensional map; a correcting means for correcting the three-dimensional map based on the information indicating the reliability set by the setting means; An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the evaluation means lowers the stability of the position as the number of feature points used by the derivation means to derive the position decreases. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the evaluation means lowers the stability of the position as the variance of the positions of the feature points used by the derivation means to derive the position in the captured image increases. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the evaluation means lowers the stability of the position as the depth variance of the feature points used by the derivation means to derive the position increases. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the evaluation means lowers the stability as the amount of change in luminance of the captured image used by the derivation means to derive the position increases. (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the setting means reduces the reliability of information indicating the three-dimensional positions of features included in the predetermined region when the stability evaluated by the evaluation means is low compared to when the stability is high. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the setting means increases the reliability of information indicating the three-dimensional positions of features included in the predetermined region when the stability evaluated by the evaluation means is high compared to when the stability is low. (Configuration 8) an auxiliary indicator for measuring the position of the imaging device based on the captured image is placed in a space in which the subject is located; The setting means sets a reliability based on the information of the auxiliary index for information indicating the three-dimensional position of the feature of the subject. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) a feature included in a previously created map of the three-dimensional map is defined as a first feature; the feature added to the three-dimensional map by the creation means is a second feature; The correction means does not update information indicating the three-dimensional position of the first feature, but updates information indicating the three-dimensional position of the second feature that is less reliable than the first feature. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) the three-dimensional map includes information indicating observation results of the subject from each of a plurality of viewpoints; The correction means updates information indicating the three-dimensional position of the feature so as to reduce an error between an observation result observed at the viewpoint when the feature is at the three-dimensional position and an observation result actually observed at the viewpoint. 10. The information processing device according to configuration 9. (Configuration 11) the three-dimensional map includes information indicating observation results of the subject from each of a plurality of viewpoints; The correction means updates the information indicating the three-dimensional position of the feature so as to reduce an error between the color information of the feature and color information observed for the three-dimensional position at the viewpoint. 11. The information processing device according to configuration 9 or 10. (Configuration 12) an auxiliary indicator for measuring the position of the imaging device based on the captured image is placed in a space in which the subject is located; the stability evaluated by the evaluation means is expressed by the number of feature points used by the derivation means to derive the position; The setting means sets the reliability of the feature based on the auxiliary index lower when the number of feature points is small compared to when the number of feature points is large. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) 13. The information processing device according to any one of configurations 1 to 12, further comprising a notification means for notifying that, when the stability evaluated by the evaluation means is low, the accuracy of the position derived by the derivation means for a predetermined region will decrease. (Configuration 14) The device further includes a notification means for notifying the derivation means of information for increasing the number of feature points used to derive the position when the stability evaluated by the evaluation means is low. 14. The information processing device according to any one of configurations 1 to 13. (Configuration 15) an input means for inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storage means for storing a three-dimensional map including information indicating the three-dimensional positions and reliability of features of the subject; a deriving means for deriving a position of the imaging device using the captured image and the three-dimensional map; evaluation means for evaluating the stability of the position using information used by the derivation means to derive the position; a creating means for creating the three-dimensional map based on the captured image and the position; Based on the result of the evaluation by the evaluation means, information indicating three-dimensional positions of features of the subject, obtained based on the captured images in an expanded region of a predetermined region in the three-dimensional map stored in the storage means; and a setting means for setting information indicating the reliability of information indicating the three-dimensional positions of features included in a predetermined area in the three-dimensional map; An information processing device comprising: (Method 1) an input step of inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storing step of storing a three-dimensional map including information indicating three-dimensional positions of features of the subject; a deriving step of deriving a position of the imaging device using the captured image and the three-dimensional map; an evaluation step of evaluating the stability of the position based on information used to derive the position in the derivation step; a creation step of creating the three-dimensional map based on the captured image and the position; Based on the results of the evaluation in the evaluation step, information indicating three-dimensional positions of features of the subject, obtained based on the captured images in an expanded region of a predetermined region in the three-dimensional map stored in the storing step; and a setting step of setting information indicating the reliability of information indicating the three-dimensional positions of features included in a predetermined area in the three-dimensional map; a correction step of correcting the three-dimensional map based on information indicating the reliability set in the setting step; An information processing method comprising: (Method 2) an input step of inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storing step of storing a three-dimensional map including information indicating three-dimensional positions and reliability of features of the subject; a deriving step of deriving a position of the imaging device using the captured image and the three-dimensional map; an evaluation step of evaluating the stability of the position using information used to derive the position in the derivation step; a creation step of creating the three-dimensional map based on the captured image and the position; Based on the results of the evaluation in the evaluation step, information indicating three-dimensional positions of features of the subject, obtained based on the captured images in an expanded region of a predetermined region in the three-dimensional map stored in the storing step; and a setting step of setting information indicating the reliability of information indicating the three-dimensional positions of features included in a predetermined area in the three-dimensional map; An information processing method comprising: (Program 1) A computer program for causing a computer to function as each of the means recited in claim 1. [Explanation of symbols]

[0101] 1. Information processing equipment 110 Holding part 120 Input section 130 Derivation part 140 Evaluation Department 150 Creation Department 160 Setting section 170 Correction Unit 180 Imaging Device 190 Notification Department

Claims

1. an input means for inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storage means for storing a three-dimensional map including information indicating three-dimensional positions of features of the subject; a deriving means for deriving a position of the imaging device using the captured image and the three-dimensional map; evaluation means for evaluating the stability of the position based on information used by the derivation means to derive the position; a creating means for creating the three-dimensional map based on the captured image and the position; a setting means for setting information indicating the reliability of information indicating the three-dimensional positions of features of the subject, which information is obtained based on the captured image in an extended area of a predetermined area in the three-dimensional map stored in the storage means, and information indicating the three-dimensional positions of features included in the predetermined area in the three-dimensional map, based on the evaluation result of the evaluation means; a correcting means for correcting the three-dimensional map based on the information indicating the reliability set by the setting means; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the evaluation means lowers the stability of the position as the number of feature points used by the derivation means to derive the position decreases.

3. 2. The information processing apparatus according to claim 1, wherein the evaluation means lowers the stability of the position as the variance of the positions of the feature points used by the derivation means to derive the position in the captured image increases.

4. 2. The information processing apparatus according to claim 1, wherein the evaluation means lowers the stability of the position as the variance of the depths of the feature points used by the derivation means to derive the position increases.

5. 2. The information processing apparatus according to claim 1, wherein the evaluation means lowers the stability as the amount of change in brightness of the captured image used by the derivation means to derive the position increases.

6. The information processing device according to claim 1, characterized in that the setting means lowers the reliability of information indicating the three-dimensional position of features included in the predetermined area when the stability evaluated by the evaluation means is low compared to when the stability is high.

7. The information processing device according to claim 1, characterized in that the setting means increases the reliability of information indicating the three-dimensional position of features included in the predetermined area when the stability evaluated by the evaluation means is high compared to when the stability is low.

8. an auxiliary indicator for measuring the position of the imaging device based on the captured image is placed in a space in which the subject is located; The setting means sets a reliability based on the information of the auxiliary index for information indicating the three-dimensional position of the feature of the subject.

2. The information processing apparatus according to claim 1, wherein:

9. a feature included in a previously created map of the three-dimensional map is defined as a first feature; the feature added to the three-dimensional map by the creation means is a second feature; The correction means does not update information indicating the three-dimensional position of the first feature, but updates information indicating the three-dimensional position of the second feature that is less reliable than the first feature.

2. The information processing apparatus according to claim 1, wherein:

10. the three-dimensional map includes information indicating observation results of the subject from each of a plurality of viewpoints; The correction means updates information indicating the three-dimensional position of the feature so as to reduce an error between an observation result observed at the viewpoint when the feature is at the three-dimensional position and an observation result actually observed at the viewpoint.

10. The information processing device according to claim 9,

11. the three-dimensional map includes information indicating observation results of the subject from each of a plurality of viewpoints; The correction means updates the information indicating the three-dimensional position of the feature so as to reduce an error between the color information of the feature and color information observed for the three-dimensional position at the viewpoint.

10. The information processing device according to claim 9,

12. an auxiliary indicator for measuring the position of the imaging device based on the captured image is placed in a space in which the subject is located; the stability evaluated by the evaluation means is expressed by the number of feature points used by the derivation means to derive the position; The setting means sets the reliability of the feature based on the auxiliary index lower when the number of feature points is small compared to when the number of feature points is large.

2. The information processing apparatus according to claim 1, wherein:

13. 2. The information processing apparatus according to claim 1, further comprising a notification means for notifying that the accuracy of the position derived by the derivation means for a predetermined region will decrease if the stability evaluated by the evaluation means is low.

14. The device further includes a notification means for notifying the derivation means of information for increasing the number of feature points used to derive the position when the stability evaluated by the evaluation means is low.

2. The information processing apparatus according to claim 1, wherein:

15. an input means for inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storage means for storing a three-dimensional map including information indicating the three-dimensional positions and reliability of features of the subject; a deriving means for deriving a position of the imaging device using the captured image and the three-dimensional map; evaluation means for evaluating the stability of the position using information used by the derivation means to derive the position; a creating means for creating the three-dimensional map based on the captured image and the position; a setting means for setting information indicating the reliability of information indicating the three-dimensional positions of features of the subject, which information is obtained based on the captured image in an extended area of a predetermined area in the three-dimensional map stored in the storage means, and information indicating the three-dimensional positions of features included in the predetermined area in the three-dimensional map, based on the evaluation result of the evaluation means; An information processing device comprising:

16. an input step of inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storing step of storing a three-dimensional map including information indicating three-dimensional positions of features of the subject; a deriving step of deriving a position of the imaging device using the captured image and the three-dimensional map; an evaluation step of evaluating the stability of the position based on information used to derive the position in the derivation step; a creation step of creating the three-dimensional map based on the captured image and the position; a setting step of setting information indicating the reliability of information indicating the three-dimensional positions of features of the subject, which information is obtained based on the captured images in an extended area of a predetermined area in the three-dimensional map stored in the storage step, and information indicating the three-dimensional positions of features included in the predetermined area in the three-dimensional map, based on the results of the evaluation in the evaluation step; a correction step of correcting the three-dimensional map based on information indicating the reliability set in the setting step; An information processing method comprising:

17. an input step of inputting a captured image of a subject from an image capturing device whose position and orientation are variable; a storing step of storing a three-dimensional map including information indicating three-dimensional positions and reliability of features of the subject; a deriving step of deriving a position of the imaging device using the captured image and the three-dimensional map; an evaluation step of evaluating the stability of the position using information used to derive the position in the derivation step; a creation step of creating the three-dimensional map based on the captured image and the position; a setting step of setting information indicating the reliability of information indicating the three-dimensional positions of features of the subject, which information is obtained based on the captured images in an extended area of a predetermined area in the three-dimensional map stored in the storage step, and information indicating the three-dimensional positions of features included in the predetermined area in the three-dimensional map, based on the results of the evaluation in the evaluation step; An information processing method comprising:

18. A computer program for causing a computer to function as each of the means according to claim 1.

Images