Information processing device, information processing method, and program

Abstract

To enable quick and accurate acquisition of feature points having less possibility of erroneous correspondence among feature points extracted from a captured image.SOLUTION: An information processing device extracts a plurality of feature points from a first image acquired by an imaging device as a feature point group, and uses each feature point included in the feature point group as a feature point to acquire a similarity between that feature point and another feature point in its vicinity as a similarity of the feature point. Then, the information processing device selects feature points for which the similarity is equal to a threshold value or less from the plurality of feature points contained in the feature point group as a selected feature point group.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for estimating the position and orientation of an imaging device based on a captured image. [Background technology]

[0002] In recent years, in the fields of design and manufacturing, there has been a demand for shortening the time and reducing the cost of evaluation using prototypes. For example, systems that provide mixed reality (MR) to evaluate ease of assembly and maintainability using design (shape and design) data created with CAD (Computer-Aided Design) systems have been introduced. Here, when providing mixed reality using an MR system, it is necessary to align the real space and the virtual space accurately and in real time. One of the techniques used to achieve accurate alignment between the real space and the virtual space is to estimate the position and orientation of an image capture device in the real space in real time using an image captured by the image capture device.

[0003] Non-Patent Document 1 discloses that in the process of estimating position and orientation using images acquired by an imaging device, optical flow (tracking of feature points between frames) estimation and depth estimation by stereo matching are performed for local feature points in the images. Both of these are based on a process of determining the correct correspondence of feature points between two images (local feature point correspondence). If mismatched points are included among the matching points, the accuracy of the position and orientation estimation decreases. For this reason, Non-Patent Document 1 randomly extracts some of the matching points and repeats the process of estimating position and orientation, removing outlier matching points as mismatched points. Patent document 1 also discloses a method for searching for corresponding points between two consecutively captured images, in which the similarity between a first similar point and its neighboring points is compared, and the point with the higher similarity is selected as the corresponding point. Furthermore, Non-Patent Document 2 discloses cross-matching in searching for corresponding points, which is a method of removing results of searching for corresponding points from feature points in one image that do not match results of searching for corresponding points from feature points in another image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-41141 [Non-patent literature]

[0005] [Non-Patent Document 1] R.Mur-Artal and JDTardos:"ORB-SLAM: Tracking and Mapping Recognizable Features",Robotics:Science and Systems Workshop on Multi View Geometry in RObotics,2014. [Non-patent document 2] B. Zak and S. Hozyn: "Local Image Features Matching for Real-time Seabed Tracking Applications", Journal of Marine Engineering & Technology, volume 16, 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the method disclosed in Non-Patent Document 1, the ratio of mismatched points among all matching points is high, and if the mismatched points satisfy geometric consistency, it is not possible to remove the mismatched points. For example, if feature points at the four corners of a window in a building are matched with feature points at the four corners of another window that has the same appearance, it is difficult to determine that these are mismatched. Therefore, it is necessary to remove mismatches based not only on geometric consistency but also on the appearance of the image. Furthermore, in the case of the method disclosed in Patent Document 1, erroneous correspondence may occur when the similarity between the first similar point and its neighboring point is equally high. Furthermore, in the case of the method disclosed in Non-Patent Document 2, the number of apparent comparisons between feature points increases, resulting in an increase in the amount of calculation. In other words, these methods are unable to quickly and accurately acquire only feature points that are less likely to be mismatched points, and as a result, it may become impossible to estimate the position and orientation of the imaging device stably and with high accuracy.

[0007] Therefore, an object of the present invention is to enable feature points that are less likely to be mismatched to be obtained with high accuracy and speed from among feature points extracted from a captured image. [Means for solving the problem]

[0008] The information processing device of the present invention includes: a feature point extraction unit that extracts a plurality of feature points as a feature point group from a first image acquired by an imaging device; a feature point extraction unit that sets each feature point included in the feature point group as a target feature point; and a feature point extraction unit that calculates a similarity between the target feature point and other nearby feature points. Based on The similarity of the target feature points is Get a similarity obtaining means for obtaining a similarity between the plurality of feature points included in the feature point group; of the feature points of interest a processing means for selecting feature points whose similarity is equal to or less than a threshold value and setting the selected feature points as a group of selected feature points; a map construction means for estimating three-dimensional coordinates for each feature point included in the selected feature point group and registering the three-dimensional coordinates as a map point group; and a position and orientation estimation means for estimating a position and orientation of the image capture device when the second image was captured, based on correspondence between the map point group and feature points extracted by the feature point extraction means from a second image captured by the image capture device after the first image was captured. The present invention is characterized by having the following. [Effects of the Invention]

[0009] According to the present invention, it is possible to quickly and accurately obtain feature points that are less likely to be mismatched from among feature points extracted from a captured image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] FIG. 10 is a schematic diagram showing an example of a plurality of feature points extracted from an image. [Figure 4] 10 is a flowchart of a process in an information processing device. [Figure 5] 10 is a flowchart of a process for extracting a group of selected feature points. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings. Note that the same configurations or processes will be described with the same reference numerals. In this embodiment, an example in which an information processing device is applied to a mixed reality (MR) system for evaluating ease of assembly and maintenance using design data created by a CAD system will be described. In an MR system, images of real space and virtual space (such as virtual objects and text information drawn by computer graphics) are synthesized and provided to a user. For example, a video see-through image display device used in an MR system displays a synthesized image in which an image of virtual space generated according to the position and orientation of an imaging device is superimposed on an image of real space captured by the imaging device. Furthermore, for example, an optical see-through image display device displays an image of virtual space generated according to the position and orientation of the observer's viewpoint on an optical see-through display worn on the observer's head.

[0012] In this embodiment, the MR system performs accurate registration between the real space and the virtual space based on local feature points in an image captured by an imaging device. However, the process of matching local feature points often results in erroneous matching. One of the reasons for this is the presence of multiple points in the captured image that have similar features when viewed locally. In this embodiment, in order to accurately and quickly determine the correspondence of local feature points between images obtained by an image capture device, a process is performed to remove feature points that are likely to be erroneous correspondences from multiple feature points extracted from the images, that is, to selectively acquire feature points that are unlikely to be erroneous correspondences. In this embodiment, by selectively acquiring feature points that are unlikely to be erroneous correspondences from the local feature points between images, it is possible to more stably estimate the position and orientation of the image capture device.

[0013] Fig. 1 is a block diagram showing an example of a system configuration including an information processing device 1 according to this embodiment. As shown in Fig. 1, the system according to this embodiment is configured such that the information processing device 1 is connected to a camera 100, which is an imaging device. Note that, although an example is given in which the camera 100 has a stereo camera configuration consisting of a camera for the left eye and a camera for the right eye, it may also be a monocular camera or two or more cameras.

[0014] The information processing device 1 includes an image input unit 101, a feature point tracking unit 102, a position and orientation estimation unit 103, a feature point extraction unit 104, a similarity calculation unit 105, a similarity processing unit 106, a map construction unit 107, a map storage unit 108, and an output unit 109. Note that these configurations are merely examples and are not limited to the illustrated example. As will be described in detail later, the information processing device 1 of this embodiment estimates the position and orientation of the image capture device and constructs a map by executing processing using these configurations. Note that, in addition to the captured image, information obtained from sensors such as an acceleration sensor, an angular velocity sensor, an ultrasonic sensor, a magnetic sensor, a depth image sensor, and a laser distance sensor, as well as information obtained from markers placed in real space, may be used during the position and orientation estimation and map construction. In the following description, the position and orientation of the image capture device will be referred to as the camera position and orientation.

[0015] FIG. 2 is a diagram showing an example of a hardware configuration of the present embodiment including the information processing device 1. In FIG. 2, a CPU 201 controls the entire device. A RAM 202 is used as a work area when the CPU 201 performs processing while controlling each unit. A ROM 203 stores a control program, various application programs, data, etc. The CPU 201 loads the control program stored in the ROM 203 into the RAM 202 and executes it. By executing the control program, the CPU 201 realizes each function of the image input unit 101, feature point tracking unit 102, position and orientation estimation unit 103, feature point extraction unit 104, similarity calculation unit 105, similarity processing unit 106, map construction unit 107, map storage unit 108, etc. shown in FIG. 1. An input I / F 204 acquires an image signal from the camera 100 as an input signal in a format processable by the information processing device 1. An output I / F 205 converts a signal generated by the information processing device 1 into a signal in a format processable by an external device (not shown) and outputs it. The input I / F 204 and the output I / F 205 are connected to a communication cable when the information processing device is connected to an external device via a wired connection, and are equipped with an antenna when the information processing device has the function of wirelessly communicating with an external device.

[0016] As described above, the functions of the units of the information processing device 1 shown in FIG. 1 can be realized by the CPU 201 in FIG. 2 executing a program. Note that at least some of the functional units shown in FIG. 1 may be executed by dedicated hardware or a GPU (not shown). In this case, the dedicated hardware or GPU operates under the control of the CPU 201. Alternatively, one or more dedicated hardware components separate from the CPU may be provided, and at least some of the processing performed by the CPU may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). Although not shown in FIG. 2, the information processing device 1 may include or be connected to a display unit, an operation unit, and a large-capacity storage unit. The display unit may be, for example, an LCD display or an LED display, and displays a GUI (graphical user interface) for a user to operate the information processing device. The operation unit includes operating devices operated by the user, such as a mouse, keyboard, joystick, touch panel, etc. The storage unit may include a removable memory card, an internal or external hard disk drive, a solid-state drive, etc.

[0017] The information processing device 1 according to this embodiment and an example of its usage will be described in detail below. The image input unit 101 acquires images captured by the camera 100 of real space and sends them to the feature point tracking unit 102. The camera 100 is a device that captures moving images, and images for each frame of the moving image are input to the image input unit 101. In the following description, the most recent input frame is referred to as the current frame, and the frame immediately before that as the previous frame. Note that if the camera 100 is made up of multiple imaging devices, such as a stereo camera, it is assumed that the multiple image frames input to the image input unit 101 are synchronized (the image capture times are the same). It is also assumed that the internal and external parameters of each camera have been calibrated in advance.

[0018] The feature point tracking unit 102 calculates image coordinates (two-dimensional coordinates) in the current frame corresponding to a plurality of map points (three-dimensional points in the world coordinate system) included in a map point group held in a map holding unit 108 described later. An example of a specific procedure for processing by the feature point tracking unit 102 is shown below. First, the feature point tracking unit 102 projects multiple map points included in the map point cloud stored in the map storage unit 108 onto the image plane of the previous frame based on the camera position and orientation when the image of the previous frame was captured and the internal parameters of the camera 100, to obtain projected coordinates. Then, the feature point tracking unit 102 defines a set of points whose projected coordinates are within the image area of the previous frame as a projected point cloud. Each point in the projected point cloud stores projected coordinates (i.e., two-dimensional coordinates in the previous frame image) and information on the corresponding map point.

[0019] Next, the feature point tracking unit 102 performs a corresponding point search process to find image coordinates of the current frame corresponding to each point of the projected point cloud (hereinafter referred to as a corresponding point cloud) using the image of the previous frame and the image of the current frame. One method for this is the Lucas-Kanade method, but the method of the corresponding point search process in this embodiment is not limited to this. Each point of the corresponding point cloud holds information on the image coordinates (two-dimensional coordinates) in the current frame, and the corresponding projected coordinates and map point. The feature point tracking unit 102 then sends the projected point group and the corresponding point group to the position and orientation estimation unit 103. If the image of the current frame is the first image input to the information processing device 1 and no map point group is stored in the map storage unit 108, the feature point tracking unit 102 does not need to send anything to the position and orientation estimation unit 103.

[0020] The position and orientation estimation unit 103 estimates the position and orientation of the camera when capturing the image of the current frame, based on the map points acquired from the feature point tracking unit 102 and the image coordinates of the current frame corresponding to each of the map points. One method for determining the position and orientation of a camera with six degrees of freedom is to solve a system of equations called the Perspective-n-Point (PnP) problem based on the correspondence between multiple three-dimensional coordinates and two-dimensional coordinates. In this embodiment, the camera position and orientation for the current frame are determined by solving the PnP problem based on the correspondence between the three-dimensional coordinates of map points and the image coordinates (two-dimensional coordinates) of the current frame. Note that, since the correspondence between the three-dimensional coordinates and the two-dimensional coordinates may contain erroneous correspondences, robust estimation such as RANSAC may also be used (see Non-Patent Document 1). However, the camera position and orientation estimation in this embodiment is not limited to this method, and may also be determined using information from other sensors such as a depth image, an acceleration sensor, an angular velocity sensor, or a magnetic sensor. Then, the position and orientation estimation unit 103 sends information indicating the camera position and orientation in the current frame calculated as described above to the output unit 109. At the same time, the information on the camera position and orientation in the current frame is also sent to the feature point extraction unit 104.

[0021] Note that if the image of the current frame is the first image input to the information processing device 1 and no information has been sent from the feature point tracking unit 102, the position and orientation estimation unit 103 may send a default value (such as the origin of the world coordinate system) as the camera position and orientation. Alternatively, the position and orientation estimation unit 103 may send the camera position and orientation calculated by another method. Examples of other methods include a method of detecting markers in an image to calculate the camera position and orientation, and a method of using a sensor that can track the position and orientation of an object, such as motion capture technology. Methods of detecting markers in an image to calculate the camera position and orientation are described in the following reference documents.

[0022] Reference: H. Kato and M. Billinghurst: "Marker Tracking and hmd calibration for a video-based augmented reality conferencing system", Proceedings of 2nd IEEE and ACM International Workshop on Augmented Reality, 1999.

[0023] The feature point extraction unit 104 receives the image of the current frame sent from the image input unit 101 via the feature point tracking unit 102 and the position and orientation estimation unit 103. The feature point extraction unit 104 then extracts multiple feature points from the image of the current frame and outputs them as a feature point group. One method for extracting feature points is to detect points that are locally determined to be corner points as feature points, such as the method of Shi-Tomasi et al., but the feature point extraction method in this embodiment is not limited to this. In this embodiment, the feature point group is described as holding the image coordinates of multiple feature points. However, feature information associated with the feature points may also include image information around the feature points, feature descriptors such as SIFT, edge information, and other information. Furthermore, the image coordinates of the current frame of the map points sent from the feature point tracking unit 102 may also be included as feature points in the feature point group. The feature point extraction unit 104 sends the feature point group extracted using any of these methods to the similarity calculation unit 105.

[0024] 3 is a schematic diagram showing a state in which a group of feature points has been extracted from an image 301. Feature points 302, 303, 305, and 306 are each included in the group of feature points extracted by the feature point extraction unit 104.

[0025] The similarity calculation unit 105 calculates the similarity between each feature point included in the feature point group extracted by the feature point extraction unit 104 and its neighboring feature points by using a feature point similarity acquisition process (feature point similarity calculation process) as described below. The process of calculating the similarity of feature points in the similarity calculation unit 105 will be described with reference to FIG. First, the similarity calculation unit 105 sets a neighborhood search range for each feature point. For example, for feature point 302, a neighborhood search range 304 indicated by a dotted line in the figure is set. In this embodiment, the neighborhood search range is larger than the patch size and smaller than the image size. Note that the patch size is the size of a small region (patch) around a feature point that is used when performing optical flow or stereo matching of the feature point.

[0026] Next, the similarity calculation unit 105 sets each feature point as a target feature point and calculates the similarity between the target feature point and other feature points (i.e., nearby feature points) within the set nearby search range. In this embodiment, a method using a luminance histogram of pixels in a patch (described later) is used to calculate the similarity between two feature points, such as the target feature point and the nearby feature point. However, the method for calculating the similarity between two feature points in this embodiment is not limited to this. For example, the Sum of Squared Difference (SSD) of the luminance of each pixel in the patch may be calculated as the similarity, or the similarity may be defined using color information of each pixel. Furthermore, the patch of one feature point may be used as a template, and the patch of the other feature point may be template-matched, and the maximum similarity obtained may be used as the similarity between the feature points.

[0027] In this embodiment, the similarity calculation unit 105 calculates the similarity using a luminance histogram of pixels in the patch. First, the similarity calculation unit 105 calculates a histogram H1 from the luminance values of each pixel for the patch of the first feature point, which is one of the feature points. Similarly, the similarity calculation unit 105 calculates a histogram H2 from the luminance values of each pixel for the patch of the second feature point, which is the other feature point. Then, the similarity calculation unit 105 calculates the correlation coefficient D of the two histograms using the following equation (1).

[0028]

number

[0029] Here, H(I) in equation (1) is the I-th bin of histogram H, and A(H) is calculated by the following equation (2): N in equation (2) is the number of bins in the histogram.

[0030]

number

[0031] The similarity calculation unit 105 calculates, for each feature point, a correlation coefficient D with feature points within the nearby search range, and determines the similarity of that feature point to be the largest correlation coefficient D. In other words, the similarity calculation unit 105 calculates the highest similarity with a feature point of interest among one or more feature points (neighboring feature points) within the nearby search range as the similarity of that feature point of interest.

[0032] For example, in the situation of FIG. 3 , if feature point 302 is the target feature point, the other feature points (neighboring feature points) within a neighborhood search range 304 of feature point 302 are feature point 303 and feature point 306. For example, if the correlation coefficient between feature point 302 and feature point 303 is D1 and the correlation coefficient between feature point 302 and feature point 306 is D2, and D1 is greater than D2, then the similarity of feature point 302 is D1. Furthermore, if the target feature point is feature point 303, then the only feature point (neighboring feature point) within the neighborhood search range of feature point 303 is feature point 302, and the similarity of feature point 303 is also D1. Furthermore, if the target feature point is feature point 305, then there are no feature points (neighboring feature points) within the neighborhood search range of feature point 305. In this case, the similarity of feature point 305 may be set to a minimum, or a default value indicating that there are no similar feature points. The similarity calculation unit 105 sends the similarity calculated for each feature point included in the feature point group as described above to the similarity processing unit 106 .

[0033] The similarity processing unit 106 determines whether the similarity calculated by the similarity calculation unit 105 is equal to or less than a threshold for each feature point included in the feature point group extracted by the feature point extraction unit 104. The similarity processing unit 106 extracts a set of feature points whose similarity is determined to be equal to or less than the threshold as a selected feature point group, and sends it to the map construction unit 107. For example, in the situation shown in FIG. 3, if the similarity D1 between feature point 302 and feature point 303 exceeds the threshold, and the similarity D2 between feature point 302 and feature point 306 is equal to or less than the threshold, the selected feature point group will consist of only feature point 305 and feature point 306.

[0034] The map construction unit 107 calculates three-dimensional coordinates for each feature point of the selected feature point group sent from the similarity processing unit 106, and sends the three-dimensional coordinates as map points to the map storage unit 108 to store them. In this embodiment, the map construction unit 107 estimates the three-dimensional coordinates of each feature point in the selected feature point group in the camera coordinate system of the current frame by performing stereo matching based on the image input from the camera 100. Furthermore, the map construction unit 107 sets the coordinates converted into three-dimensional coordinates in the world coordinate system using the camera position and orientation calculated by the position and orientation estimation unit 103 as map points. However, the method of calculating the map points in this embodiment is not limited to this, and a method such as performing optimization processing such as bundle adjustment based on correspondence with map points accumulated in the map storage unit 108 may also be used (see Non-Patent Document 1).

[0035] FIG. 4 is a flowchart showing the flow of processing in the information processing device 1 of this embodiment. In step S401, the image input unit 101 imports an image captured by the camera 100 into the information processing device 1. In this embodiment, the camera 100 is a stereo camera, but since stereo images are not required except in step S406, processing may be performed on one of the predetermined images.

[0036] Next, in step S402, the feature point tracking unit 102 calculates a projected point group by projecting the coordinates of each map point included in the map point group held in the map holding unit 108 onto the image of the previous frame. Furthermore, in step S403, the feature point tracking unit 102 performs a corresponding point search process to calculate a corresponding point group in the image of the current frame that corresponds to the projected point group in the image of the previous frame.

[0037] Next, in step S404, the position and orientation estimation unit 103 calculates the camera position and orientation in the current frame based on the information calculated in steps S402 and S403. Then, the position and orientation estimation unit 103 sends the calculated camera position and orientation to the output unit 109.

[0038] Next, in step S405, a process of extracting a group of selected feature points from the image input in step S401 is performed by the feature point extraction unit 104, the similarity calculation unit 105, and the similarity processing unit 106. Details of the process performed in step S405 will be described later using the flowchart in FIG.

[0039] Next, in step S406, the map constructing unit 107 calculates three-dimensional coordinates for each point of the selected feature point group calculated in step S405 based on stereo matching. Furthermore, in step S407, the map construction unit 107 transforms the 3D points of the selected feature points calculated in step S406 into the world coordinate system using the camera position and orientation determined in step S404, and registers them in the map storage unit 108 as map points.

[0040] Thereafter, in step S408, the information processing device 1 determines whether to end the processing, and if not, repeats the processing from step S401 for the next frame. Here, the conditions for ending the processing include, but are not limited to, when there is no more image input from the camera 100 or when there is a request to end the processing from outside the information processing device 1. Through the above processing, the information processing device 1 can output to the output unit 109 the results of camera position and orientation estimation for each frame input to the image input unit 101.

[0041] Here, in the feature point group calculated by the feature point extraction unit 104, there is a possibility that feature points with similar appearances of patches are present in close positions. If this feature point group is used, there is a possibility that a mismatch will occur, in which a feature point is matched with another nearby feature point in the stereo matching in step S406 or the corresponding point search process in step S403 in a subsequent frame. If a mismatch occurs in step S403, an error will occur in the camera position and orientation calculated in step S404. Furthermore, if a mismatch occurs in step S406, an error will occur in the three-dimensional coordinates of the map points registered in step S407, causing errors in the processes from step S402 to step S404 in the subsequent frames.

[0042] Therefore, in step S405, the information processing device 1 of this embodiment extracts a selected feature point group based on the similarity calculated for each feature point in the feature point group, thereby reducing the possibility of erroneous correspondence occurring in steps S406 and S403. As described above, the selected feature point group in this embodiment is a collection of feature points that do not have other feature points that are similar in appearance in the vicinity of each feature point.

[0043] FIG. 5 is a flowchart showing the detailed process flow of step S405 in FIG. In step S501, the feature point extraction unit 104 extracts a plurality of feature points as a feature point group from the captured image. At this point, the selected feature point group is an empty set. Next, in step S502, the similarity calculation unit 105 extracts one feature point (the i-th feature point) from the group of feature points extracted in step S501 as a feature point of interest, and calculates the similarity between that feature point and neighboring feature points.

[0044] Next, in step S503, the similarity processor 106 determines whether the similarity of the i-th feature point calculated in step S502 is equal to or less than a predetermined threshold. If the similarity of the i-th feature point is equal to or less than the threshold, the similarity processor 106 proceeds to step S504, and if it is greater than the threshold, the similarity processor 106 proceeds to step S505.

[0045] If the process proceeds to step S504, that is, if the similarity of the i-th feature point is equal to or less than the threshold, the similarity processing unit 106 adds the i-th feature point to the selected feature point group. On the other hand, if the process proceeds to step S505, that is, if the similarity of the i-th feature point is greater than the threshold value, the similarity processing unit 106 determines whether or not to end the process.

[0046] Here, the condition for ending the process is whether or not there is a next feature point for which similarity calculation has not been processed among the feature points extracted in step S501, and the process ends if there is no next feature point. If there is a feature point for which similarity calculation has not been processed and it is determined in step S505 that the process should not end, the process returns to step S502, and similarity is calculated for the next feature point for which similarity calculation has not been processed (in this case, the i+1th feature point). In other words, the processes from step S502 onwards are repeated until processing has been completed for all feature points included in the feature point group extracted in step S501. Then, when the similarity processing unit 106 has completed the processing for all feature points included in the group of feature points extracted in step S501, it ends the processing of the flowchart in FIG.

[0047] By performing the processing described above, the information processing device 1 of this embodiment can extract a selected feature point group consisting of a plurality of feature points with a low risk of erroneous correspondence. Furthermore, the information processing device 1 of this embodiment can reduce the amount of calculation required for searching for corresponding points in steps S406 and S403, thereby achieving the effect of reducing erroneous correspondence without significantly increasing the amount of calculation required by the system. As a result, the information processing device 1 of this embodiment can reduce erroneous correspondence between local feature points between images, and can more stably and quickly estimate the position and orientation of the camera using images captured by the camera.

[0048] <Variation 1> As a first modification of this embodiment, when extracting a group of feature points from an image, the feature point extraction unit 104 may use both a method of detecting unknown feature points (also called natural feature points) such as corner points in the image and a method of detecting feature points corresponding to predetermined artificially placed indices. In this case, the feature point extraction unit 104 defines a group of unknown natural feature points as a feature point group, and extracts a group of feature points obtained corresponding to predetermined artificially placed indices as an index point group. Note that the three-dimensional coordinates of the index point group are assumed to be known and obtained by prior calibration.

[0049] Here, when using predetermined artificially placed indices, examples of the indices include markers as described in Non-Patent Document 3. This method calculates the coordinates of corner points based on the edges of the markers, thereby ensuring high reliability in detecting corner points using the markers. Furthermore, since the orientation and ID of the markers can be identified, there is a low possibility of mismatches occurring even if the corner points created by the markers appear similar. Therefore, there is no need to perform the processing of the similarity calculation unit 105 and the similarity processing unit 106 described above on the index point group, and the feature point extraction unit 104 sends the extracted feature points to the map construction unit 107.

[0050] In the case of the first modification, the map construction unit 107 performs optimization such as bundle adjustment using the pre-calibrated three-dimensional coordinates of the index point group when estimating the three-dimensional coordinates of the selected feature point group sent from the similarity processing unit 106. This makes it possible to improve the accuracy of the map points.

[0051] <Variation 2> As a second modification of this embodiment, the similarity processing unit 106 may add the similarity calculated by the similarity calculation unit 105 as weight information for each feature point, and then set all points of the feature point group extracted by the feature point extraction unit 104 as a selected feature point group. The weight information of each feature point to be added to the selected feature point group may be referred to when the map construction unit 107 generates map points for each feature point of the selected feature point group, or may be stored in the map storage unit 108 together with the map points.

[0052] <Variation 3> In the above-described embodiment, if there are many apparently similar feature points in an image, the number of feature points included in the selected feature point group will be reduced. As a result, the stability of the camera position and orientation may be reduced. Furthermore, even if the local image similarity is high, comparing a wider range of images may result in differences, reducing the similarity and making it possible to distinguish the feature points as different feature points. Therefore, by setting the patch size of the feature point of interest to an image size that allows it to be distinguished from its neighboring feature points, it is possible to extract more feature points as the selected feature point group while reducing the possibility of mismatches. This is expected to enable more stable camera position and orientation estimation. For the same reason, instead of increasing the patch size, calculating the similarity using a reduced image obtained by reducing the input image may also reduce the similarity and make it possible to distinguish the feature points as different feature points.

[0053] Therefore, as a third modification of this embodiment, when the similarity between two feature points exceeds a threshold, the similarity calculation unit 105 may recalculate the similarity by enlarging the patch size within a preset range or by using a reduced image. Then, when the recalculated similarity falls below the threshold, it may be used as the similarity of the feature points. Furthermore, the similarity processing unit 106 may transmit the calculated similarity of each feature point, together with the patch size (or the scale of the reduced image) used in calculating the similarity.

[0054] <Variation 4> As a fourth modification of this embodiment, the similarity processing unit 106 may determine feature points whose similarity is determined to be greater than a threshold as rejected feature points and extract them as a rejected feature point group. Furthermore, the similarity processing unit 106 may send information about the rejected feature point group to an output image generation unit (not shown) to generate an image that visualizes the coordinates of feature points included in the rejected feature point group in the input image. Furthermore, the similarity processing unit 106 may generate an image that visualizes an area in the input image where there are many feature points included in the rejected feature point group. This allows the user to recognize which feature points in the image are rejected feature points or areas of the rejected feature point group in the image.

[0055] In the above description, an example has been given in which the process of estimating the position and orientation of a camera in real space using an image captured by a camera is applied to an MR system, but the present invention is not limited to this. For example, the information processing device 1 of the present embodiment can also be applied to self-position estimation of a robot or an automatic guided vehicle.

[0056] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and should not be construed as limiting the technical scope of the present invention. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0057] 1: Information processing device, 100: Camera, 101: Image input unit, 102: Feature point tracking unit, 103: Position and orientation estimation unit, 104: Feature point extraction unit, 105: Similarity calculation unit, 106: Similarity processing unit, 107: Map construction unit, 108: Map storage unit, 109: Output unit

Claims

1. a feature point extraction means for extracting a plurality of feature points as a feature point group from a first image acquired by an imaging device; a similarity obtaining means for obtaining a similarity of each feature point included in the group of feature points based on the similarity between the feature point of interest and other feature points in the vicinity of the feature point of interest; a processing means for selecting, from the plurality of feature points included in the feature point group, feature points whose similarity to the target feature point is equal to or less than a threshold, as a selected feature point group; a map construction means for estimating three-dimensional coordinates for each feature point included in the selected feature point group and registering the three-dimensional coordinates as a map point group; a position and orientation estimation means for estimating a position and orientation of the image capturing device when the second image was captured, based on correspondence between the map point cloud and feature points extracted by the feature point extraction means from a second image captured by the image capturing device after the first image was captured; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the similarity acquisition means re-acquires the similarity of the target feature point by using a patch size that is increased within a preset range for the target feature point whose similarity exceeds the threshold, or by using an image that is a reduced version of the first image.

3. the processing means extracts, from among the plurality of feature points included in the feature point group, a feature point of interest whose similarity to the feature point of interest acquired by the similarity acquisition means exceeds the threshold, as a rejected feature point; 3. The information processing apparatus according to claim 1, further comprising display means for visualizing the coordinates of the rejection feature points on the first image.

4. 4. The information processing apparatus according to claim 1, wherein the similarity obtaining means obtains the highest similarity between the feature point of interest and the plurality of neighboring feature points as the similarity of the feature point of interest.

5. the feature point extraction means extracts indices whose three-dimensional coordinates are known from the first image as an index point group; 2. The information processing apparatus according to claim 1, wherein the map construction means estimates three-dimensional coordinates for each feature point included in the selected feature point group based on the three-dimensional coordinates of each index point in the index point group, and registers the three-dimensional coordinates as the map point group.

6. An information processing method executed by an information processing device, a feature point extraction step of extracting a plurality of feature points as a feature point group from a first image acquired by an imaging device; a similarity obtaining step of determining each feature point included in the group of feature points as a feature point of interest and obtaining a similarity of the feature point of interest based on a similarity between the feature point of interest and other feature points in its vicinity; a processing step of selecting, from the plurality of feature points included in the feature point group, feature points whose similarity to the target feature point is equal to or less than a threshold, as a selected feature point group; a map construction step of estimating three-dimensional coordinates for each feature point included in the selected feature point group and registering the three-dimensional coordinates as a map point group; a position and orientation estimation step of estimating a position and orientation of the imaging device when the second image was captured, based on correspondence between the map point cloud and feature points extracted in the feature point extraction step from a second image captured by the imaging device after the first image was captured; An information processing method comprising:

7. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 5.

Images