Information processing device, self-location estimation method, and program

Abstract

The purpose of the present invention is to provide an information processing device that makes it possible to prevent deterioration of the accuracy of self-position estimation when reducing a processing load. An information processing device (10) according to the present disclosure comprises: a detection unit (11) that detects a plurality of new feature points from a first image: a specifying unit (12) that specifies, among the plurality of new feature points, corresponding feature points corresponding with a known feature point associated with a three-dimensional point included in at least one management image used for generating an environment map; and an estimation unit (13) whereby the position and orientation of the imaging device that captured the first image are estimated using the corresponding feature points. The detection unit (11) changes the number of new feature points detected from a target image serving as a target for estimating the position and orientation of the imaging device in accordance with the number of corresponding feature points.

Description

Information processing device, self-location estimation method, and non-transitory computer-readable medium

[0001] The present disclosure relates to an information processing device, a self-location estimation method, and a non-transitory computer-readable medium.

[0002] In recent years, services that assume autonomous robot movement have become widespread. For a robot to move autonomously, it must recognize its surrounding environment and accurately estimate its own position. Therefore, VSLAM (Visual Simultaneous Localization and Mapping) is being considered. This technology simultaneously creates a map of the surrounding environment from video captured by the robot and estimates its own position by referencing the created map. A typical VSLAM recognizes the same point captured in multiple videos as a feature point in the multiple images (still images) that make up the videos. The camera position is then estimated based on the difference between the feature points in the images. Since the camera position on the robot is fixed, the robot's position can be estimated by estimating the camera position. VSLAM estimates the camera position by estimating the 3D positions of feature points in multiple images, projecting the estimated 3D positions onto the images, and then using the difference between the 2D positions and the positions of the feature points in the images. Because VSLAM requires instant processing, reducing the processing load is essential.

[0003] Patent Literature 1 describes the configuration of an autonomous mobile device that estimates its own position by obtaining correspondence between feature points included in image information stored in a storage unit and feature points extracted from a captured image. Patent Literature 1 also describes thinning out images to be stored in the storage unit depending on the number of corresponding feature points obtained during estimation.

[0004] Patent Document 2 describes the configuration of an information processing device that extracts feature points from an input image and detects the position and orientation of an image capturing device that captured the input image based on the extracted feature points. The information processing device of Patent Document 2 changes the number of feature points to be extracted from the input image based on the processing time required to detect the position and orientation of the image capturing device from the input image.

[0005] JP 2020-57187 A JP 2021-9557 A

[0006] However, the autonomous mobile device disclosed in Patent Document 1 increases the number of images to be thinned out and reduces the number of images to be stored as the number of corresponding feature points acquired during estimation increases above a threshold. In this case, the accuracy of self-location estimation deteriorates as the number of images used for self-location estimation decreases. Furthermore, the information processing device disclosed in Patent Document 2 increases the processing time required to detect the position and orientation of the imaging device as the processing load of processes other than the process for detecting the position and orientation of the imaging device increases. In this case, the number of feature points extracted by the information processing device from the input image decreases, resulting in a problem of deterioration in the accuracy of self-location estimation.

[0007] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide an information processing device, a self-location estimation method, and a non-transitory computer-readable medium that can prevent the accuracy of self-location estimation from deteriorating when reducing the processing load.

[0008] An information processing device according to a first aspect of the present disclosure includes a detection unit that detects a plurality of new feature points from a first image, an identification unit that identifies corresponding feature points among the plurality of new feature points that correspond to known feature points that are associated with three-dimensional positions included in at least one control image used to generate an environmental map, and an estimation unit that uses the corresponding feature points to estimate the position and orientation of the image capturing device that captured the first image, wherein the detection unit changes the number of new feature points to be detected from a target image that is the subject of estimation of the position and orientation of the image capturing device, depending on the number of corresponding feature points.

[0009] A self-location estimation method according to a second aspect of the present disclosure detects a plurality of new feature points from a first image, identifies corresponding feature points among the plurality of new feature points that correspond to known feature points associated with three-dimensional positions contained in at least one or more control images used to generate an environmental map, estimates the position and orientation of the image capturing device that captured the first image using the corresponding feature points, and changes the number of new feature points to be detected from a target image from which the position and orientation of the image capturing device are to be estimated depending on the number of corresponding feature points.

[0010] A program according to a third aspect of the present disclosure causes a computer to detect a plurality of new feature points from a first image, identify corresponding feature points among the plurality of new feature points that correspond to known feature points associated with three-dimensional positions contained in at least one or more control images used to generate an environmental map, estimate the position and orientation of the image capturing device that captured the first image using the corresponding feature points, and change the number of new feature points to be detected from a target image from which the position and orientation of the image capturing device are to be estimated, depending on the number of corresponding feature points.

[0011] The present disclosure makes it possible to provide an information processing device, a self-location estimation method, and a non-transitory computer-readable medium that can prevent the accuracy of self-location estimation from deteriorating when reducing processing load.

[0012] Fig. 1 is a configuration diagram of an information processing device according to a first embodiment. Fig. 2 is a diagram showing a flow of a self-location estimation process according to the first embodiment. Fig. 3 is a configuration diagram of an information processing device according to a second embodiment. Fig. 4 is a diagram explaining a feature point matching process according to the second embodiment. Fig. 5 is a diagram explaining a feature point classification process according to the second embodiment. Fig. 6 is a diagram showing a flow of a process of updating the number of target feature points according to the second embodiment. Fig. 7 is a configuration diagram of an information processing device according to each embodiment.

[0013] (First Embodiment) Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. An example configuration of an information processing device 10 according to the first embodiment will be described with reference to FIG. 1. The information processing device 10 may be a computer device that operates when a processor executes a program stored in a memory. The information processing device 10 may be, for example, a server device.

[0014] The information processing device 10 includes a detection unit 11, an identification unit 12, and an estimation unit 13. The detection unit 11, the identification unit 12, and the estimation unit 13 may be software or modules whose processes are performed by a processor executing a program stored in a memory. Alternatively, the detection unit 11, the identification unit 12, and the estimation unit 13 may be hardware such as a circuit or a chip. While FIG. 1 illustrates a configuration in which the detection unit 11, the identification unit 12, and the estimation unit 13 are included in a single information processing device 10, the detection unit 11, the identification unit 12, and the estimation unit 13 may each be located in a different computer device. Alternatively, one of the components of the detection unit 11, the identification unit 12, and the estimation unit 13 may be located in a different computer device. Computers including the detection unit 11, the identification unit 12, and the estimation unit 13 may communicate with each other via a network.

[0015] The detection unit 11 detects multiple new feature points from the first image. The first image may be an image captured by a camera mounted on a moving object such as a vehicle. The camera mounted on the moving object may generate an image by capturing an image in the direction of travel of the moving object or around the moving object while the moving object is moving. The detection unit 11 may receive an image captured by the camera via a network. Alternatively, when the camera is used integrally with the information processing device 10, that is, when the camera is included in or connected to the information processing device 10, the detection unit 11 may acquire an image without going through a network. Alternatively, the first image may be an image received from another information processing device or the like via a network.

[0016] The image capturing device may be, for example, a camera or a device with a camera function. The device with a camera function may be, for example, a mobile terminal such as a smartphone. The image may be, for example, a still image. Alternatively, the image may be a frame image constituting a video. Furthermore, the multiple images may be a data set or data record representing multiple still images, such as multiple frame images constituting a video. Alternatively, the multiple images may be frame images extracted from multiple frame images constituting a video.

[0017] The moving body may be, for example, a robot or a vehicle that moves autonomously. Moving autonomously may mean that the vehicle is operated by a control device mounted on the robot or vehicle, without direct control by a person.

[0018] The new feature points may be detected using, for example, SIFT, SURF, ORB, AKAZE, etc. The new feature points may be indicated using two-dimensional coordinates, which are camera coordinates defined in the image capturing device.

[0019] The identification unit 12 identifies new feature points that correspond to known feature points that are included in at least one management image used to generate the environmental map and that are associated with three-dimensional positions as corresponding feature points.

[0020] An environmental map is three-dimensional information that shows the environment surrounding the camera. Three-dimensional information may also be referred to as 3D information, three-dimensional coordinates, etc. An environmental map includes map information showing the environment surrounding the camera, as well as information about the position and orientation of the camera. The orientation of the camera may be, for example, information about the tilt of the camera. An environmental map is generated by identifying the shooting positions where multiple images were taken and restoring the three-dimensional positions of feature points recorded on the images. In other words, an environmental map includes information about the three-dimensional positions or three-dimensional coordinates of feature points in images captured using the camera. For example, an environmental map may be generated by performing Structure from Motion (SfM) using multiple images. SfM calculates all feature points in a series of previously acquired two-dimensional images (or frames) and estimates matching feature points from multiple images that occur before and after in time. Furthermore, SfM accurately estimates the three-dimensional position or orientation of the camera that captured each frame based on the difference in the two-dimensional position of each feature point in the frame in which it appears. The control image is the image used when executing SfM. The environmental map can also be created by accumulating images previously estimated using VSLAM. In this case, the control image is the image input to VSLAM and the 3D position is estimated.

[0021] A known feature point is a feature point that is included in the control image and is represented using two-dimensional coordinates. Furthermore, a three-dimensional position associated with a known feature point may be represented using, for example, three-dimensional coordinates. A corresponding feature point may be, for example, a feature point that has the same or similar features as a known feature point. A corresponding feature point may be, for example, another way of saying that it matches a known feature point. In other words, the identification unit 12 identifies or extracts new feature points that match a known feature point from among a plurality of new feature points.

[0022] The estimation unit 13 estimates the position and orientation of the image capturing device that captured the first image using the corresponding feature points. The estimation unit 13 may estimate the position and orientation of the image capturing device that captured the first image, for example, by executing VSLAM. Estimating the position and orientation of the image capturing device that captured the first image may also mean estimating the position and orientation of a moving object equipped with the image capturing device.

[0023] Here, the detection unit 11 estimates the position and orientation of the image capturing device that captured each image, even for images captured at a time later than the time the first image was captured. The image captured at a time later than the time the first image was captured is referred to as a target image from which the position and orientation of the image capturing device are to be estimated.

[0024] The detection unit 11 changes the number of new feature points to be detected from the target image depending on the number of corresponding feature points used when estimating the position and orientation of the camera that captured the first image. For example, if the number of corresponding feature points used when estimating the position and orientation of the camera that captured the first image (hereinafter simply referred to as the "number of corresponding feature points") is greater than a predetermined number, the number of new feature points to be detected from the target image may be reduced from the currently set number. If the number of corresponding feature points is less than the predetermined number, the number of new feature points to be detected from the target image may be increased from the currently set number.

[0025] The predetermined number of corresponding feature points can be considered to be a sufficient number of corresponding feature points for estimating the position and orientation of the camera that captured the target image. A sufficient number of corresponding feature points for estimating the position and orientation of the camera that captured the target image means a number of corresponding feature points that allows for highly accurate estimation of the position and orientation of the camera that captured the target image. Therefore, when the number of corresponding feature points is greater than the predetermined number of corresponding feature points, even if the number of new feature points detected from the target image is reduced, the identification unit 12 can identify a sufficient number of corresponding feature points for estimating the position and orientation of the camera that captured the target image. Furthermore, by reducing the number of new feature points detected from the target image, the processing load related to identifying corresponding feature points and the processing load related to estimating the position and orientation of the camera using the corresponding feature points can be reduced.

[0026] On the other hand, if the number of corresponding feature points is less than a predetermined number, it can be said that a sufficient number of corresponding feature points have not been identified to estimate the position and orientation of the camera that captured the target image. Therefore, if the number of corresponding feature points is less than the predetermined number, the identification unit 12 increases the number of corresponding feature points used to estimate the position and orientation of the camera that captured the target image by increasing the number of new feature points detected from the target image. As a result, the estimation unit 13 can improve the accuracy of estimating the position and orientation of the camera with respect to the target image.

[0027] Alternatively, when the number of corresponding feature points used to estimate the position and orientation of the image capturing device for a plurality of target images including the first image is on an increasing trend, the detection unit 11 may reduce the number of new feature points to be detected from the target images from a currently set number. Furthermore, when the number of corresponding feature points used to estimate the position and orientation of the image capturing device for a plurality of target images including the first image is on a decreasing trend, the detection unit 11 may increase the number of new feature points to be detected from the target images from a currently set number.

[0028] Next, a flow of a self-location estimation process executed in the information processing device 10 according to the first embodiment will be described with reference to Fig. 2. The self-location estimation is a process of estimating the position and orientation of an image capturing device that captured an image of a target screen.

[0029] First, the detection unit 11 detects a plurality of new feature points from the first image (S11). Next, the identification unit 12 identifies, from the plurality of new feature points, corresponding feature points that correspond to known feature points associated with three-dimensional positions included in at least one or more control images used to generate the environmental map (S12).

[0030] Next, the estimation unit 13 estimates the position and orientation of the image capturing device that captured the first image using the corresponding feature points (S13). Next, the detection unit 11 changes the number of new feature points to be detected from the target image for which the position and orientation of the image capturing device are to be estimated, depending on the number of corresponding feature points (S14).

[0031] As described above, the information processing device 10 according to the first embodiment changes the number of new feature points to be detected from a target image for estimating the position and orientation of an image capturing device, depending on the number of corresponding feature points. As a result, the information processing device 10 can maintain the accuracy of estimating the position and orientation of the image capturing device that captured the target image, while reducing the processing load.

[0032] (Second embodiment) Next, a configuration example of an information processing device 20 according to a second embodiment will be described with reference to Fig. 3. The information processing device 20 may be a computer device, similar to the information processing device 10. The information processing device 20 has a configuration in which an environment map generation unit 21, a feature point management unit 22, an acquisition unit 23, a feature point determination unit 24, and a detection number management unit 25 are added to the configuration of the information processing device 10. In the following description, detailed description of the same configuration and functions as those of the information processing device 10 in Fig. 1 will be omitted.

[0033] The environment map generation unit 21, feature point management unit 22, acquisition unit 23, feature point determination unit 24, and detection number management unit 25 may be software or modules whose processes are executed by a processor running a program stored in memory. Alternatively, the environment map generation unit 21, feature point management unit 22, acquisition unit 23, feature point determination unit 24, and detection number management unit 25 may be hardware such as a circuit or chip. Alternatively, the feature point management unit 22 and detection number management unit 25 may be memories that store data.

[0034] The information processing device 20 uses multiple images captured by the camera to estimate in real time the position and orientation of the camera that captured each image. For example, the information processing device 20 executes VSLAM to estimate in real time the position and orientation of the camera that captured each image. The information processing device 20 may also be used to correct the position and orientation of an autonomously moving robot. Estimating the position and orientation of an autonomously moving robot involves comparing images captured in real time by the moving robot with environmental images that are similar to the images captured in real time among control images in an environmental map. The environmental images correspond to the control images. The comparison between the images captured in real time and the environmental images is performed using feature points contained in each image. The position and orientation of the robot are estimated and corrected based on the comparison results. Here, the estimation and correction of the robot's position and orientation are performed by VSLAM. In this disclosure, the robot is not limited to a form constituting a device as long as it is capable of moving, and is intended to broadly include, for example, robots that mimic humans or animals, transport vehicles that move using wheels (e.g., automated guided vehicles), etc. The transport vehicle may be, for example, a forklift.

[0035] The environment map generation unit 21 may generate the environment map by performing SfM using multiple images captured by an imaging device. If the information processing device 20 has a camera function, the environment map generation unit 21 may generate the environment map using images captured by the information processing device 20. Alternatively, the environment map generation unit 21 may receive images captured by an imaging device that is a device different from the information processing device 20 via a network or the like, and generate the environment map.

[0036] The environment map generation unit 21 outputs the environment map and multiple images used to generate the environment map to the feature point management unit 22. At this time, the environment map generation unit 21 may output only information about feature points detected in the images to the feature point management unit 22, rather than outputting image information directly to the feature point management unit 22. The feature point management unit 22 manages the environment map and images received from the environment map generation unit 21. Alternatively, the feature point management unit 22 manages information about feature points received from the environment map generation unit 21. The feature point management unit 22 also manages each image received from the environment map generation unit 21 in association with the position and orientation of the camera that captured the image. Furthermore, the feature point management unit 22 manages each image in association with the three-dimensional coordinates on the environment map of feature points in the image. The images managed by the feature point management unit 22 may be referred to as key frames. Here, a key frame can also be considered a frame image that can serve as a starting point for a series of image processing operations described below. Furthermore, the three-dimensional coordinates on the environmental map of the feature points in the key frames may be referred to as landmarks.

[0037] The acquisition unit 23 acquires images captured by a camera or a plurality of frame images constituting a video. The acquisition unit 23 acquires images captured by a camera mounted on an autonomously moving robot in substantially real time. That is, the acquisition unit 23 acquires images captured by the camera in real time to estimate the position and posture of the autonomously moving robot or camera in real time. In the following description, the images acquired by the acquisition unit 23 will be referred to as real-time images.

[0038] The detection unit 11 detects feature points in the real-time image according to a target number of feature points to be detected. The detection unit 11 detects feature points in the real-time image so as to approach the target number. Specifically, the detection unit 11 may detect the same number of feature points as the target number, or may detect a number of feature points within a predetermined range including the target number. In other words, the detection unit 11 may detect more feature points than the target number, or may detect fewer feature points than the target number. The difference between the number of feature points detected by the detection unit 11 and the target number is set to a value sufficiently small compared to the target number. In other words, the difference between the number of feature points detected by the detection unit 11 and the target number is set to a number that is recognized as an error from the target number. The target number may be changed for each real-time image for which feature points are to be detected. Alternatively, the target number may be changed for each of multiple real-time images for which feature points are to be detected. In other words, the same target number may be applied to multiple real-time images.

[0039] The identification unit 12 identifies new feature points from among a plurality of feature points (new feature points) extracted from the real-time image that match feature points (known feature points) managed by the feature point management unit 22. Specifically, the identification unit 12 compares the feature vectors of the known feature points with the feature vectors of the new feature points, and matches feature points whose vectors indicate close distances with each other. The identification unit 12 may extract several images from the plurality of images managed by the feature point management unit 22, and identify new feature points that match known feature points included in each image.

[0040] Here, the feature point matching process executed by the identification unit 12 will be described with reference to Fig. 4. Fig. 4 shows the feature point matching process using a key frame 60 and a real-time image 50. u1, u2, and u3 in the key frame 60 are known feature points, and t1, t2, and t3 in the real-time image 50 are new feature points detected by the detection unit 11. The identification unit 12 identifies t1, t2, and t3 as new feature points that match u1, u2, and u3, respectively. In other words, t1, t2, and t3 are corresponding feature points that correspond to u1, u2, and u3, respectively.

[0041] Furthermore, q1 is a three-dimensional coordinate associated with known feature point u1 and indicates the landmark of known feature point u1, q2 indicates the landmark of known feature point u2, and q3 indicates the landmark of known feature point u3.

[0042] Since the new feature point t1 matches the known feature point u1, the three-dimensional coordinates of the new feature point t1 are the landmark q1. Similarly, the three-dimensional coordinates of the new feature point t2 are the landmark q2, and the three-dimensional coordinates of the new feature point t3 are the landmark q3.

[0043] The estimation unit 13 estimates the position and orientation of the camera device that captured the real-time image 50 using the known feature points u1, u2, and u3, the new feature points t1, t2, and t3, and the landmarks q1, q2, and q3. Specifically, the estimation unit 13 first assumes the position and orientation of the camera device 30 that captured the real-time image 50. The feature point detection unit 23 projects the positions of q1, q2, and q3 onto the real-time image 50 when q1, q2, and q3 are captured at the assumed position and orientation of the camera device 30. The estimation unit 13 repeatedly changes the position and orientation of the camera device 30 that captured the real-time image 50 and projects the positions of q1, q2, and q3 onto the real-time image 50. The estimation unit 13 estimates the position and orientation of the imaging device 30 as the position and orientation of the imaging device 30 that minimizes the difference between the positions where q1, q2, and q3 are projected onto the real-time image 50 and the characteristic points t1, t2, and t3 within the real-time image 50.

[0044] Here, the classification process of feature points executed by the feature point determination unit 24 will be described with reference to Fig. 5. When the position and orientation of the image capture device 30 that captured the real-time image 50 are the position and orientation estimated by the estimation unit 13, the feature point determination unit 24 determines the positions where q1, q2, and q3 are projected onto the real-time image 50 as t'1, t'2, and t'3. In other words, t'1, t'2, and t'3 are the positions of q1, q2, and q3 in the real-time image 50 when the image capture device 30 captured the real-time image 50 at the position and orientation estimated by the estimation unit 13. The dotted circles in the real-time image 50 in Fig. 5 indicate t'1, t'2, and t'3.

[0045] Here, the feature point determination unit 24 calculates the distance between t1 and t'1 associated with the landmark q1, the distance between t2 and t'2 associated with the landmark q2, and the distance between t3 and t'3 associated with the landmark q1. In Fig. 5, t'1 and t'3 are shown to be at substantially the same positions as t1 and t3, respectively, or the distance from t1 and t3 is equal to or less than a predetermined distance. Fig. 5 also shows that the position of t'2 is different from t2 and is separated from t2 by a predetermined distance or more.

[0046] The fact that the position of t'2 is different from t2 indicates that t2 is shifted from the position of the landmark q2 that should be displayed in the real-time image 50. In other words, this indicates that the matching accuracy of the new feature point t2 with respect to the known feature point u2 is low. On the other hand, the fact that the positions of t'1 and t'3 are substantially the same as t1 and t3 indicates that t1 and t3 match the landmarks q1 and q3 that should be displayed in the real-time image 50. In other words, this indicates that the matching accuracy of the new feature points t1 and t3 with respect to the known feature points u1 and u3 is high. The feature point determination unit 24 refers to t2 in FIG. 5 as a low-accuracy feature point, and t1 and t3 as high-accuracy feature points. In other words, the feature point determination unit 24 classifies t2 in FIG. 5 as a low-accuracy feature point, and t1 and t3 as high-accuracy feature points. High-accuracy feature points may also be referred to as inlier feature points, and low-accuracy feature points may also be referred to as outlier feature points.

[0047] The feature point determination unit 24 outputs the number of high-precision feature points and the number of low-precision feature points among the new feature points used to estimate the position and orientation of the image capturing device 30 in the real-time image 50 to the detection number management unit 25. Alternatively, the feature point determination unit 24 may output only the number of high-precision feature points to the detection number management unit 25.

[0048] The detection number management unit 25 calculates the target number of feature points (target number of feature points) to be detected from the real-time image acquired by the acquisition unit 23 using the number of high-precision feature points received from the feature point determination unit 24. The target number of feature points f to be detected from the n-th real-time image acquired by the acquisition unit 23 is n may be calculated using, for example, the following equation 1:

[0049] (Formula 1) f n = f n-1 +α×(Ii n-i )

[0050] "I" indicates the target number of high-precision feature points, and i n-i indicates the number of high-precision feature points in the previous frame (previous image). Also, α indicates a coefficient and is a number greater than 0. α is the coefficient of Ii n-i ≧0 and Ii n-i< 0, and Ii n-i ≧0 and Ii n-i < 0 may be different values.

[0051] For example, Ii n-i If ≧0, set α=2 and Ii n-i If <0, α may be set to 1. n-i ≧0 means that the number of identified high-precision feature points does not reach the target number of high-precision feature points, and Ii n-i <0 means that the number of identified high-precision feature points exceeds the target number of high-precision feature points. n-i The value of α when ≧0 is Ii n-i Increasing the value of α from the value when Ii < 0 means that the increase in the number of target feature points is increased when the number of identified high-precision feature points does not reach the target number of high-precision feature points. n-i The value of α when ≧0 is Ii n-i Increasing the value of α beyond that when α is < 0 means that the decrease in the target number of feature points when the number of identified high-precision feature points exceeds the target number of high-precision feature points is reduced. In other words, this means that the emphasis is on improving the estimation accuracy of the position and orientation of the image capture device rather than reducing the processing load of the estimation process of the position and orientation of the image capture device.

[0052] On the other hand, Ii n-i The value of α when ≧0 is Ii n-i The value of α may be smaller than the value when α is < 0. This means that the emphasis is on reducing the load of the estimation process for the position and orientation of the image capture device rather than improving the estimation accuracy of the estimation process for the position and orientation of the image capture device.

[0053] The coefficient α is the target feature point number f n and the number of high-precision feature points i n The correlation function between the high-precision feature points i n The function g(i n) Alternatively, the coefficient α may be determined by holding the estimation results of the position and orientation of the image for a certain period of time and using a function that uses the amount of change in position and orientation between the most recent real-time images as a variable. The amount of change may be, for example, velocity, and the function that uses the amount of change as a variable may be a function that uses velocity as a variable.

[0054] Furthermore, if the number of target feature points is too large, the load on the estimation process of the position and orientation of the image capture device increases, and if the number of target feature points is too small, the estimation accuracy of the position and orientation of the image capture device decreases. Therefore, a maximum and minimum value may be defined for the number of target feature points, and a value between the minimum and maximum values ​​may be used as the number of target feature points.

[0055] The number of target high-precision feature points may be set to any value by an administrator or the like of the information processing device 20. For example, a value that the administrator or the like of the information processing device 20 considers appropriate may be set as the number of target feature points. Alternatively, the number of target high-precision feature points may be determined using machine learning. For example, the number of target high-precision feature points may be determined using a learning model that has learned the relationship between the number of high-precision feature points and the processing load or the estimation accuracy of the position and orientation of the information processing device 20.

[0056] Next, the flow of the process of updating the number of target feature points according to the second embodiment will be described with reference to Fig. 6. First, the detection unit 11 detects feature points according to the target number of feature points from the real-time image acquired by the acquisition unit 23 (S21). Next, the identification unit 12 identifies new feature points that match known feature points managed by the feature point management unit 22 from among the new feature points extracted from the real-time image (S22).

[0057] Next, the feature point determination unit 24 classifies new feature points that match known feature points managed by the feature point management unit 22 into high-precision feature points and low-precision feature points, and identifies the number of high-precision feature points (S23).

[0058] Next, the detection number management unit 25 determines whether the number of high-precision feature points is equal to or greater than the target number of high-precision feature points (S24). If the detection number management unit 25 determines that the number of high-precision feature points is equal to or greater than the target number of high-precision feature points (YES determination in S24), it updates the target number of feature points so as to decrease the target number of feature points (S25). If the detection number management unit 25 determines that the number of high-precision feature points is less than the target number of high-precision feature points (NO determination in S24), it updates the target number of feature points so as to increase the target number of feature points (S26).

[0059] As described above, the information processing device 20 according to the second embodiment identifies new feature points included in a real-time image that match known feature points. Furthermore, the information processing device 20 identifies, among the new feature points that match known feature points, high-precision feature points whose distances to projection points obtained by projecting the three-dimensional positions of the known feature points onto the real-time image are shorter than a predetermined distance. Furthermore, the detection number management unit 25 determines a target number of feature points to be extracted from the real-time image according to the number of high-precision feature points.

[0060] Generally, the accuracy of position and orientation estimation improves as the number of high-precision feature points used to estimate the position and orientation of the image capture device increases. Assume that a certain number of high-precision feature points are included in new feature points extracted from real-time images. In this case, as the number of high-precision feature points increases, the number of new feature points extracted from real-time images also increases. Therefore, the number of feature points used to estimate the position and orientation also increases, and the processing load on the information processing device 20 also increases. Therefore, when the number of high-precision feature points exceeds a target number of high-precision feature points, it is considered that the position and orientation estimation accuracy can be maintained at a sufficiently high level, and the target number of feature points extracted from real-time images can be lowered. As a result, it is possible to prevent an increase in the processing load on the position and orientation while maintaining the position and orientation estimation accuracy.

[0061] FIG. 7 is a block diagram showing an example configuration of the information processing device 10 and information processing device 20 (hereinafter referred to as information processing device 10, etc.) described in the above-described embodiment. Referring to FIG. 7, the information processing device 10, etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 may be used to communicate with a network node. The network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers.

[0062] The processor 1202 reads and executes software (computer programs) from the memory 1203 to perform the processing of the information processing device 10 and the like described using flowcharts in the above-described embodiments. The processor 1202 may be, for example, a microprocessor, an MPU, or a CPU. The processor 1202 may include multiple processors.

[0063] The memory 1203 is configured by a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located remotely from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O (Input / Output) interface (not shown).

[0064] 7, the memory 1203 is used to store a group of software modules. The processor 1202 reads and executes these software modules from the memory 1203, thereby performing the processing of the information processing device 10 and the like described in the above-described embodiment.

[0065] As explained using Figure 7, each of the processors possessed by the information processing device 10, etc. in the above-mentioned embodiments executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawings.

[0066] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0067] The technical ideas of the present disclosure are not limited to the above-described embodiments, and can be modified as appropriate within the scope of the gist of the present disclosure.

[0068] REFERENCE SIGNS LIST 10 Information processing device 11 Detection unit 12 Identification unit 13 Estimation unit 20 Information processing device 21 Environmental map generation unit 22 Feature point management unit 23 Acquisition unit 24 Feature point determination unit 25 Detection number management unit 30 Photography device 50 Real-time image 60 Key frame

Claims

1. a detection unit for detecting a plurality of new feature points from the first image; an identification unit that identifies, among the plurality of new feature points, corresponding feature points that correspond to known feature points associated with three-dimensional positions included in at least one or more control images used to generate the environmental map; an estimation unit that estimates a position and an orientation of an image capturing device that captured the first image by using the corresponding feature points; The detection unit is The information processing device changes, depending on the number of the corresponding feature points, the number of new feature points to be detected from a target image from which the position and orientation of the image capturing device are to be estimated.

2. The detection unit is 2. The information processing device according to claim 1, wherein if the number of the corresponding feature points is greater than a target number, the number of new feature points to be detected from the target image is reduced from a currently set number, and if the number of the corresponding feature points is less than the target number, the number of new feature points to be detected from the target image is increased from a currently set number.

3. The identification unit is identifying high-precision feature points, the distance between which is shorter than a predetermined distance between the corresponding feature points and projection points, which are obtained by projecting three-dimensional positions associated with the known feature points onto the first image, and low-precision feature points, the distance between which is longer than the predetermined distance between the projection points and the corresponding feature points; The detection unit is The information processing apparatus according to claim 1 , further comprising: a step of changing a number of new feature points to be detected from a target image for estimating a position and orientation of the image capturing apparatus, in accordance with a number of the high-precision feature points.

4. The detection unit is 4. The information processing device according to claim 3, wherein when the number of high-precision feature points is greater than the target number of high-precision feature points, the number of new feature points to be detected from the target image is reduced from a currently set number, and when the number of high-precision feature points is less than the target number of high-precision feature points, the number of new feature points to be detected from the target image is increased from the currently set number.

5. The detection unit is 5. The information processing device according to claim 4, wherein, when the number of high-precision feature points is greater than the target number of high-precision feature points, a value obtained by subtracting the target number of high-precision feature points from the number of high-precision feature points is subtracted from the currently set number of detected new feature points, and when the number of high-precision feature points is less than the target number of high-precision feature points, a value obtained by subtracting the number of high-precision feature points from the target number of high-precision feature points is added to the currently set number of detected new feature points.

6. The detection unit is 6. The information processing device according to claim 5, wherein, when the number of high-precision feature points is greater than the target number of high-precision feature points, a value obtained by subtracting the target number of high-precision feature points from the number of high-precision feature points, multiplying the result by a first coefficient, and subtracting the result from the currently set number of detected new feature points, and when the number of high-precision feature points is less than the target number of high-precision feature points, a value obtained by subtracting the number of high-precision feature points from the target number of high-precision feature points, multiplying the result by a second coefficient, is added to the currently set number of detected new feature points, wherein the first coefficient and the second coefficient are positive numbers, and the first coefficient is a value smaller than the second coefficient.

7. The detection unit is The information processing apparatus according to claim 1 , wherein the number of new feature points to be detected from the target image is changed within a range between a maximum value and a minimum value of the number of new feature points to be detected from the target image.

8. Detecting a plurality of novel feature points from the first image; Among the plurality of new feature points, corresponding feature points corresponding to known feature points associated with three-dimensional positions included in at least one or more control images used to generate the environmental map are identified. Using the corresponding feature points, a position and orientation of an image capturing device that captured the first image is estimated. A self-location estimation method, comprising: changing, depending on the number of the corresponding feature points, the number of new feature points to be detected from a target image from which the position and orientation of the image capturing device are to be estimated.

9. Detecting a plurality of novel feature points from the first image; Among the plurality of new feature points, corresponding feature points corresponding to known feature points associated with three-dimensional positions included in at least one or more control images used to generate the environmental map are identified. Using the corresponding feature points, a position and orientation of an image capturing device that captured the first image is estimated. A program that causes a computer to execute the following: changing the number of new feature points to be detected from a target image from which the position and orientation of the image capturing device are to be estimated, depending on the number of corresponding feature points.