Program, information processing device, and information processing method

Abstract

To provide a program, an information processing device, and an information processing method capable of reducing computational costs for at least one computation of the positional attitude of a camera and of three-dimensional information of an image feature while maintaining accuracy.SOLUTION: The program of the present embodiment makes a computer function as a correspondence acquisition part, a selection part, and a calculation part. The correspondence acquisition part calculates multiple image features from multiple images captured by a camera and acquires the correspondence of the multiple image features. The selection part selects the multiple correspondences based on correspondence effectiveness and on the influence on at least one image of the multiple images when the correspondence where the effectiveness is lower than effectiveness threshold is deleted. The calculation part calculates at least one of the positional attitudes of the camera and of three-dimensional information of the image feature from the selected correspondence of the multiple correspondences.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method. [Background Art]

[0002] As techniques for simultaneously performing self-localization and three-dimensional measurement of the surrounding environment using a camera, Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) have attracted attention. Since cameras can be used indoors and are relatively inexpensive and easy to use, applications of techniques using cameras are expected. Examples include applications to measurement of shooting positions and target positions in maintenance inspection and patrol of infrastructure, estimation of the movement of the own vehicle in accident footage, and digital archiving of cultural properties for the purpose of protecting cultural heritage. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent No. 5953142 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, with the conventional techniques, it has been difficult to reduce the calculation cost required for calculating at least one of the position and orientation of the camera and the three-dimensional information of image features while maintaining accuracy. [Means for Solving the Problem]

[0005] The program of the embodiment causes the computer to function as a correspondence acquisition unit, a selection unit, and a calculation unit. The correspondence acquisition unit calculates multiple image features from multiple images captured by the camera and acquires the correspondence between the multiple image features. The selection unit selects the multiple correspondences based on the effectiveness of the correspondence and the impact on at least one of the multiple images if correspondences with an effectiveness lower than the effectiveness threshold are deleted. The calculation unit calculates at least one of the camera's position and orientation and the three-dimensional information of the image features from the correspondences selected from the multiple correspondences. [Brief explanation of the drawing]

[0006] [Figure 1] A diagram showing an example of the functional configuration of an information processing device according to an embodiment. [Figure 2A] A figure showing an example 1 of the amount of displacement of image features in the embodiment. [Figure 2B] A figure showing example 2 of the amount of displacement of image features in the embodiment. [Figure 3] A diagram illustrating the number of images containing corresponding image features in the embodiment. [Figure 4A] A figure showing example 1 of the distribution of image features in the embodiment. [Figure 4B] A figure showing example 2 of the distribution of image features in the embodiment. [Figure 5A] A diagram illustrating the mapping of image features in SLAM. [Figure 5B] A diagram illustrating the mapping of image features in SfM. [Figure 6] A flowchart showing an example of the sorting process in the embodiment. [Figure 7] A flowchart showing an example of the sorting process in a modified example of the embodiment 1. [Figure 8] This figure shows an example of the functional configuration of the information processing device in a modified example of the embodiment 2. [Figure 9] A diagram showing an example of the hardware configuration of an information processing device according to an embodiment. [Modes for carrying out the invention]

[0007] The following describes in detail embodiments of the program, information processing device, and information processing method with reference to the attached drawings.

[0008] First, let's explain the difference between SLAM and SfM. Generally, SLAM is used in applications that require real-time processing, such as autonomous driving, and performs sequential position estimation using a continuous sequence of images as input. Because processing speed is prioritized over processing accuracy, techniques such as narrowing the range when optimizing the camera's position and orientation, as well as the camera's surrounding environment, are employed.

[0009] On the other hand, SfM assumes offline processing, and the input is not limited to a continuous sequence of images but uses the entire image set, with optimization prioritizing accuracy.

[0010] For applications such as measuring shooting and target locations during maintenance, inspection, and patrols, and estimating the movement of one's own vehicle in accident footage, SfM (Structure from Motion) is suitable because it does not require real-time processing and therefore prioritizes accuracy. On the other hand, even with SfM that uses the entire image set, speeding up processing time can reduce server usage fees and waiting times for application users, so there is a demand for SfM speed-up technologies.

[0011] A typical processing flow in SfM involves detecting feature points from a set of images, creating corresponding points by mapping the feature points, estimating the initial values ​​of the camera position and orientation, estimating the initial values ​​of the 3D points of the corresponding points, and optimizing the estimated camera position, orientation, and 3D points. The optimization process is often performed using Bundle Adjustment, which minimizes the reprojection error when the 3D points are reprojected onto the image using the camera position and orientation.

[0012] In general, among processing flows, optimization processing takes the longest processing time, and the greater the number of corresponding points, the more estimated variables there are. Therefore, speedup can be achieved by reducing the number of corresponding points, but there is a problem that accuracy decreases even if the number of corresponding points is randomly reduced. For example, the number of corresponding points can be reduced by decreasing the number of detected feature points or increasing the lower limit of the number of matched images for corresponding points. However, optimal parameters for maintaining accuracy differ depending on the scene, making it difficult to uniquely determine such parameters.

[0013] Hereinafter, an embodiment will be described in which correspondence relationships of image features (e.g., feature points, etc.) included in two or more camera images are sorted, and at least one of a camera position and orientation and three-dimensional information of the image features is calculated based on the sorted correspondence relationships.

[0014] [Example of Functional Configuration] Figure 1 is a diagram showing an example of the functional configuration of the information processing apparatus 1 according to the embodiment. The information processing apparatus 1 of the embodiment includes a correspondence relationship acquisition unit 11, a sorting unit 12, and a calculation unit 13.

[0015] The correspondence relationship acquisition unit 11 acquires correspondence relationships of image features on two or more camera images. Specifically, the correspondence relationship acquisition unit 11 calculates image features indicating characteristic regions on camera images, and matches image features indicating the same location from among image features on two or more camera images. Here, the image features may be point-shaped (feature points) or line-shaped. Image features are calculated from, for example, luminance gradients of an image.

[0016] For calculation of image features, feature point detection algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and AKAZE (Accelerated KAZE) may be used, or a deep neural network may be used.

[0017] The association of image features may be performed based on the similarity (matching degree) of feature quantities obtained by digitizing image features with feature descriptors, or may be calculated based on the similarity of pixels in the peripheral region of the image feature. For calculating feature descriptors, feature point detection algorithms such as SIFT, SURF and AKAZE may be used, or a deep neural network may be used.

[0018] The similarity of feature quantities is calculated from, for example, L1 norm, L2 norm, Hamming distance, and cosine similarity between N-dimensional feature quantities. In the case of the similarity of pixels in the peripheral region, the similarity is calculated from, for example, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), and normalized cross-correlation.

[0019] When the above-mentioned similarity is equal to or higher than a predetermined threshold, the correspondence acquisition unit 11 associates image features on a plurality of images indicating the same location, and stores the correspondence as the correspondence relationship of the plurality of image features in a storage device of the information processing apparatus 1. The association of image features is not limited to between two images, and may be performed for two or more images. That is, the correspondence acquisition unit 11 obtains a correspondence relationship in which image features on two or more images are associated.

[0020] The sorting unit 12 sorts, from the correspondence relationship obtained by the correspondence acquisition unit 11, correspondence relationships effective for calculation of at least one of the position and orientation of a camera and three-dimensional information of image features. The sorting unit 12 reduces only correspondence relationships unnecessary for maintaining the accuracy of position and orientation and three-dimensional information, whereby the number of estimation parameters in optimization processing such as Bundle Adjustment in SfM can be reduced, and processing time can be reduced.

[0021] Here, when the image feature is a feature point, the estimation parameters are the position and orientation of the camera and the three-dimensional point for each corresponding point. Reducing the number of correspondence relationships leads to reducing the number of three-dimensional points, and directly reduces the number of estimation parameters. On the other hand, since correspondence relationships necessary for maintaining accuracy are retained, the accuracy of position and orientation and three-dimensional information can be maintained.

[0022] The following sections will describe methods for selecting valid correspondences.

[0023] Generally, the following five relationships are considered effective in contributing to the accuracy of position and orientation estimation in SfM: • The amount of movement of image features in the correspondence is greater. • A larger number of images contain image features that correspond to the image features of other images. • The degree of similarity (match) in the correspondence relationship is higher. • Image features that correspond to image features in other images are more widely distributed within the image. • The reliability of the correspondence is higher.

[0024] The following are examples of the five indicators used to evaluate the effectiveness of the correspondence relationships.

[0025] First, the amount of movement of image features in a correspondence refers to the amount by which an image feature indicating the same location moves across different images (the amount of movement between the positions indicated by two corresponding image features). When the image feature is a feature point, it is generally called a flow.

[0026] Figure 2A shows an example 1 of the displacement amount of image features in the embodiment. Figure 2B shows an example 2 of the displacement amount of image features in the embodiment. The examples in Figures 2A and 2B show the case where image features are represented by feature points.

[0027] For example, correspondence 101a in Figure 2A is obtained from corresponding feature points (feature points indicating the same location) in two different images. Similarly, correspondence 101b in Figure 2B is obtained from corresponding feature points (feature points indicating the same location) in two different images.

[0028] The greater the distance between the corresponding feature points in two different images, the larger the flow indicating the amount of movement. In the example in Figures 2A and 2B, for instance, the amount of movement of the image feature in image correspondence 101a is greater than the amount of movement of the image feature in image correspondence 101b.

[0029] Figures 2A and 2B show examples of flow between two images. However, if feature point mapping is performed using two or more images, a method may be used to calculate the flow using the difference between the position of the feature in the image with the earliest observed time and the position of the feature in the image with the latest observed time.

[0030] For example, if the feature point mapping is performed using two or more images, a method may be used in which the difference between the maximum and minimum x-coordinates of the feature points in the images, and the difference between the maximum and minimum y-coordinates of the feature points in the images are calculated, and the flow between feature points with larger differences is calculated.

[0031] In SfM, the camera's position and orientation, as well as the 3D information of image features (3D information of the camera's surrounding environment), are estimated based on changes in how the object appears. Therefore, the magnitude of the movement of image features on the image (e.g., the flow of feature points) is an important indicator for estimating the camera's position and orientation, and the 3D information of image features.

[0032] For example, the selection unit 12 selects multiple correspondences such that it does not delete correspondences where the amount of movement between the positions indicated by the two corresponding image features is greater.

[0033] Next, the number of images containing image features that correspond to image features of other images refers to the number of images associated with image features that indicate the same location, obtained by observing the same location as an image feature in different images.

[0034] Figure 3 is a diagram illustrating the number of images containing corresponding image features 100a and 100b in the embodiment. The example in Figure 3 shows the case where image features 100a and 100b are feature points. The example in Figure 3 also shows the case where images are captured while moving by a camera 10 mounted on a moving object such as an automobile. Image feature 100a is included in two images captured by the camera 10. On the other hand, image feature 100b is included in six images captured by the camera 10.

[0035] In the example in Figure 3, the number of images containing image feature 100b is greater than the number of images containing image feature 100a, so the correspondence for image feature 100b is more valid than the correspondence for image feature 100a.

[0036] In SfM, the more areas observed in multiple images, the more accumulated errors such as scale drift can be reduced in the position and orientation of camera 10. Therefore, the number of images associated with image features that indicate the same area is an important indicator.

[0037] For example, the selection unit 12 selects multiple correspondences so as not to delete correspondences that have a larger number of images containing the corresponding image features.

[0038] Next, the correspondence similarity (matching degree) refers to the correspondence similarity calculated by the correspondence relationship acquisition unit 11. The higher the similarity of the associated image features, the fewer incorrect associations there will be, and the higher the accuracy of estimating the position and orientation of the camera 10. Therefore, the correspondence similarity is an important indicator.

[0039] For example, the selection unit 12 selects multiple correspondences so as not to delete correspondences where the similarity of the corresponding image features is higher.

[0040] Next, the distribution of image features refers to the distribution of image features on a single image.

[0041] Figure 4A shows example 1 of the distribution of image features in the embodiment. Figure 4B shows example 2 of the distribution of image features in the embodiment. The examples in Figures 4A and 4B show the case where the image features are feature points. The positions of the image features included in the image in Figure 4B are biased towards the left side of the image. On the other hand, the image features included in the image in Figure 4A have greater variability and are more widely distributed within the image than the image features included in the image in Figure 4B.

[0042] The distribution of image features is an important indicator because the wider the distribution of image features that correspond to image features in other images, the more diverse the changes in how the object appears become, and the higher the accuracy of estimating the position and orientation of camera 10.

[0043] For example, the selection unit 12 selects multiple correspondences such that, when correspondences with an effectiveness lower than the effectiveness threshold are removed, the evaluation value that evaluates the variability of the position of image features within the image does not fall below the influence threshold.

[0044] Next, the confidence level of a correspondence is calculated by a neural network that takes a correspondence as input and outputs the confidence level of that correspondence. The confidence level of a correspondence is a direct indicator of the effectiveness of the correspondence. The confidence level of a correspondence is output in such a way that it increases as the accuracy of the estimation of the position and orientation of camera 10 increases, making the confidence level of a correspondence an important indicator.

[0045] For example, the correspondence acquisition unit 11 further acquires the confidence level of the correspondence using a neural network that calculates the confidence level of the correspondence. Then, the selection unit selects multiple correspondences so as not to delete correspondences with a higher confidence level.

[0046] While correspondences with high values ​​for the five indicators listed above are effective for calculating position, orientation, and surrounding 3D information, it is not possible to simply select correspondences based on whether each indicator is high or low. This is because the characteristics of the correspondences obtained differ depending on the scene. In scenes with a large number of image features, for example, it is acceptable to retain effective correspondences that meet one of the five indicators above and delete the remaining ones. However, in scenes with fewer image features, securing a sufficient number of correspondences becomes more important, so it is necessary to retain correspondences that do not meet any of the five indicators above.

[0047] Therefore, the selection unit 12 selects correspondences while also considering the impact on the entire scene (at least one image out of multiple images). In other words, when selecting whether a particular correspondence is valid or not, it also considers the impact on camera images other than the camera image associated with the correspondence being selected. Specifically, the selection unit 12 performs the selection by considering not only the values ​​of the five indicators that evaluate the validity of the correspondence in the correspondence being selected, but also the values ​​of another indicator that evaluates the impact on the entire scene.

[0048] Furthermore, selecting correspondences while considering the overall scene relationships is only possible with SfM, which is based on offline processing, and not with SLAM, which is based on sequential processing.

[0049] Figure 5A is a diagram illustrating the mapping of image features in SLAM. Figure 5B is a diagram illustrating the mapping of image features in SfM.

[0050] As shown in Figure 5A, SLAM does not have image information for future time points, so the correspondences that have just been detected and matched to image features may have a small number of matched images or small movement of features on the image. However, these image features may become important correspondences for estimating key points (important image features) in images for future time points. Therefore, it is difficult for SLAM to select image features while also considering the values ​​of other metrics that evaluate the impact on the entire scene.

[0051] On the other hand, in the case of SfM, offline processing is assumed, and as shown in Figure 5B, the correspondence of the entire scene is known in advance, which optimizes the computational cost for calculating at least one of the camera's position and orientation, and the 3D information of the image features. Therefore, it becomes possible to select image features while also considering the value of another indicator that evaluates the impact on the entire scene.

[0052] As shown in Figures 5A and 5B, the speedup achieved by this embodiment is not possible with SLAM, but it can be applied to SfM, which is a key difference between this embodiment and existing algorithms for speeding up SLAM.

[0053] When selecting based on each indicator while considering the entire scene, one possible method is to calculate the threshold for each indicator for each scene. For example, one could determine the threshold (effectiveness threshold) based on the average or percentile of each indicator in all correspondences. The selection unit 12 selects correspondences that are above the effectiveness threshold as correspondences that are effective for calculating the camera's position and orientation and 3D information of the surrounding environment (they are kept without being deleted).

[0054] Furthermore, in order to maintain the accuracy of calculating the position and orientation of camera 10 and the 3D information of image features (3D information of the surrounding environment), a judgment process is performed to consider the impact on the entire scene.

[0055] For example, one of the factors that influences at least one of multiple images is the variability in the position of image features distributed across each image. The selection unit 12 selects multiple correspondences such that, when correspondences with an effectiveness lower than the effectiveness threshold are removed, the evaluation value for assessing the variability in the position of image features does not fall below the influence threshold.

[0056] For example, as a judgment process to consider the entire scene, a process is performed to determine at least one of the total number of correspondences and the total number of image features within a single image, based on a predetermined threshold.

[0057] Here, the total number of correspondences is equal to the number of sets of image features (sets of image features that indicate the same location) that are associated with image features of other images by any of the correspondences. A set of image features corresponding to one correspondence contains image features that indicate the same location in two or more different images.

[0058] When one correspondence is deleted, the set of image features corresponding to that correspondence is also deleted. The number of image features in the set of image features is 2 if it is a correspondence between two images, and n if it is a correspondence between n images.

[0059] The fewer the total number of correspondences, or the fewer the number of image features on a single image, the more difficult it becomes to maintain the accuracy of calculating the position and orientation of the camera 10 and the 3D information of the image features (3D information of the surrounding environment). Therefore, for example, the sorting unit 12 of the embodiment is processed according to the flowchart in Figure 6 below.

[0060] Figure 6 is a flowchart showing an example of the processing of the sorting unit 12 in the embodiment. First, the sorting unit 12 selects one correspondence to be processed and determines whether the index for evaluating the effectiveness of the correspondence is equal to or greater than a threshold (effectiveness threshold) (step S1). For example, one or more of the above five examples of effectiveness evaluation indicators are used as the index for evaluating effectiveness.

[0061] If the indicator is above the threshold (Step S1, Yes), the sorting unit 12 selects a new correspondence to be processed from the unprocessed correspondences and executes the process in Step S1.

[0062] If the index is below the threshold (Step S1, No), the sorting unit 12 determines whether, once the correspondence to be processed is deleted, there exists an image in the entire scene (multiple images) where the number of image features within the image is below the threshold (Step S2).

[0063] If there are images in which the number of image features is below a threshold (Step S2, Yes), the selection unit 12 selects a new correspondence to be processed from the unprocessed correspondences and returns to the process in Step S1. In other words, the selection unit 12 selects multiple correspondences so that when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of image features in the image does not fall below the influence threshold.

[0064] If there are no images in which the number of image features in the image is less than or equal to the threshold (Step S2, No), the sorting unit 12 determines whether the total number of correspondences will be less than or equal to the threshold when the correspondence to be processed is deleted (Step S3).

[0065] If the total number of correspondences falls below a threshold (Step S3, Yes), the selection unit 12 selects new correspondences to be processed from the unprocessed correspondences and returns to the process in Step S1. In other words, the selection unit 12 selects multiple correspondences so that when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of correspondences in multiple images does not fall below the influence threshold.

[0066] If the total number of correspondences does not fall below the threshold (Step S3, No), the sorting unit 12 deletes the correspondences to be processed, selects new correspondences to be processed from the unprocessed correspondences, and returns to the process in Step S1.

[0067] Once processing for all correspondences is complete, the selection process shown in the flowchart in Figure 6 is finished.

[0068] The above is a description of the sorting unit 12.

[0069] Returning to Figure 1, the calculation unit 13 calculates at least one of the three-dimensional information of the camera's position and orientation and image features from the correspondence relationships selected by the selection unit 12. Specifically, first, the calculation unit 13 calculates a basic matrix from the selected correspondence relationship information and calculates the initial values ​​of the camera's position and orientation.

[0070] Next, the calculation unit 13 calculates initial values ​​of the 3D information of the image features (3D information of the surrounding environment) from the initial values ​​of the position and orientation of the camera 10 and the corresponding relationships using triangulation. Finally, the calculation unit 13 reprojects the initial values ​​of the 3D information onto the image based on the position and orientation of the camera 10, and optimizes the position and orientation of the camera 10 and the 3D information of the surrounding environment using Bundle Adjustment, which minimizes the reprojection error.

[0071] At this time, the calculation unit 13 may optimize the entire scene in a single Bundle Adjustment, or it may sequentially increase the number of images to be optimized. For example, one can consider a method in which the series of processes are repeated while sequentially increasing the number of images to be optimized. Specifically, the calculation unit 13 may perform partial optimization and then calculate the initial position and orientation of the camera 10 by solving the PnP (Perspective-n-Point) problem from the optimized 3D information and its correspondence. Then, the calculation unit 13 may increase the 3D information of the surrounding environment by triangulation using the initial value and its correspondence, and then perform optimization again with Bundle Adjustment.

[0072] As described above, in this embodiment, the correspondence acquisition unit 11 calculates multiple image features from multiple images captured by the camera 10 and acquires the correspondence between the multiple image features. The selection unit 12 selects the multiple correspondences based on the effectiveness of the correspondence and the impact on at least one of the multiple images if correspondences with an effectiveness lower than the effectiveness threshold are deleted. Then, the calculation unit 13 calculates at least one of the camera 10's position and orientation and the three-dimensional information of the image features from the selected correspondences.

[0073] As a result, according to this embodiment, the computational cost required to calculate at least one of the camera 10's position and orientation, and the 3D information of its image features, can be reduced while maintaining accuracy. For example, by using the correspondence relationships selected by the selection unit 102, the optimization process in the calculation unit 13 can be sped up. Furthermore, since the selection unit 12 selects correspondence relationships while also considering the impact on the entire scene (at least one image among multiple images), the accuracy of the calculation of at least one of the camera 10's position and orientation, and the 3D information of its image features can be maintained.

[0074] (Modification of Embodiment 1) Next, a first modification of the embodiment will be described. In the description of the first modification, explanations similar to those of the embodiment will be omitted, and only the differences from the embodiment will be described. The first modification describes a method for adaptively calculating the threshold value of an index for evaluating the effectiveness of the correspondence relationship through iterative processing.

[0075] In Modification 1, a lower threshold for the total number of correspondences or a lower threshold for the number of image features on a single image is defined beforehand.

[0076] Next, initial values ​​for the lower thresholds are defined for one or more of the five effectiveness evaluation indicators described in the embodiment.

[0077] The method for calculating thresholds using iterative processing will be explained below, following the flowchart in Figure 7.

[0078] Figure 7 is a flowchart showing an example of processing by the sorting unit 12 in a modified example 1 of the embodiment. In the example in Figure 7, the amount of feature movement on the image and the number of images associated with the correspondence are used as indicators for selecting valid correspondences, but a combination of one or more arbitrary indicators for evaluating the effectiveness of the correspondence may be used.

[0079] First, the sorting unit 12 selects one correspondence to be processed and determines whether the number of images associated with that correspondence (the number of images containing image features that have the correspondence) is equal to or greater than a threshold (step S11).

[0080] If the number of images associated with a correspondence relationship is greater than or equal to a threshold (Step S11, Yes), the sorting unit 12 selects a new correspondence relationship to be processed from the unprocessed correspondence relationships and executes the process in Step S11.

[0081] If the number of images associated with a correspondence is less than the threshold (Step S11, No), the sorting unit 12 determines whether the amount of movement of the image features in the correspondence to be processed is greater than or equal to the threshold (Step S12).

[0082] If the amount of movement of an image feature on the image is greater than or equal to a threshold (step S12, Yes), the sorting unit 12 selects a new correspondence to be processed from the unprocessed correspondences and executes the process in step S11.

[0083] In other words, the correspondence that is answered "Yes" in step S11 or S12 is determined by the sorting unit 12 to be a correspondence that is effective for calculating at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features, and is stored in the storage device of the information processing device 1.

[0084] If the amount of movement of image features on the image is less than the threshold (step S12, No), the sorting unit 12 determines whether, once the correspondence to be processed is deleted, there exists an image in the entire scene (multiple images) where the number of image features in the image is less than or equal to the threshold (step S13).

[0085] If there are images where the number of image features in the image is below a threshold (Step S13, Yes), deleting the correspondence to be processed would be disadvantageous to the calculation of at least one of the camera 10's position and orientation, and the 3D information of the image features, so this correspondence is not deleted. The selection unit 12 newly selects a correspondence to be processed from the unprocessed correspondences and returns to the process in Step S11.

[0086] If there are no images in which the number of image features in the image is less than or equal to the threshold (step S13, No), the sorting unit 12 determines whether the total number of correspondences will be less than or equal to the threshold when the correspondence to be processed is deleted (step S14).

[0087] If the total number of correspondences falls below the threshold (Step S14, Yes), the selection process ends because no further correspondences can be deleted.

[0088] If the total number of correspondences does not fall below the threshold (step S14, No), the sorting unit 12 deletes the correspondences to be processed because the impact on the calculation process of at least one of the camera's position and orientation and the 3D information of the image features is negligible. Then, the sorting unit 12 selects new correspondences to be processed from the unprocessed correspondences and returns to the process in step S11.

[0089] Once processing for all correspondences is complete, the sorting unit 12 increases the threshold for the number of images used in step S11 (step S16) and increases the threshold for the amount of movement used in step S12 (step S17).

[0090] By repeating the above process a predetermined number of times (N times), the threshold for the index used to evaluate the effectiveness of the correspondence can be adaptively calculated.

[0091] As explained above, in Modification 1, for example, the selection unit 12 changes the effectiveness threshold (in the example in Figure 7, the threshold for the number of images and the threshold for the amount of movement) until the total number of image features included in each image approaches a predetermined total number. The selection unit 12 then selects multiple correspondences such that when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of image features included in each image does not fall below the influence threshold.

[0092] For example, the selection unit 12 changes the effectiveness threshold (in the example in Figure 7, the threshold for the number of images and the threshold for the amount of movement) until the total number of correspondences approaches a predetermined total. Then, the selection unit 12 selects multiple correspondences so that when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of correspondences in multiple images does not fall below the influence threshold.

[0093] As a result, according to Modification 1, unnecessary correspondences can be sequentially removed while maintaining the number of features on a single image and the total number of correspondences necessary for calculating at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features.

[0094] (Modified embodiment 2) Next, a modified example 2 of the embodiment will be described. In the description of modified example 2, explanations similar to those of the embodiment will be omitted, and the differences from the embodiment will be described. Modified example 2 will describe a case in which an initial value acquisition unit is further provided to acquire the initial position and orientation values ​​of the camera 10.

[0095] [Example of functional configuration] Figure 8 shows an example of the functional configuration of the information processing device 1-2 in modified example 2 of the embodiment. The information processing device 1-2 in modified example 2 includes a correspondence relationship acquisition unit 11, a sorting unit 12, a calculation unit 13, and an initial value acquisition unit 14. The explanation of the correspondence relationship acquisition unit 11 and the sorting unit 12 is the same as in the embodiment and will therefore be omitted.

[0096] The initial value acquisition unit 14 acquires position and orientation information that includes at least one of the initial value of the camera 10's shooting position and the initial value of the camera 10's orientation. Specifically, the initial value acquisition unit 14 acquires position and orientation information of the camera 10 obtained from information other than the correspondence between image features.

[0097] For example, the initial value acquisition unit 14 acquires position and orientation information of the camera 10 estimated by a neural network that takes the camera image as input. Alternatively, the initial value acquisition unit 14 may acquire position and orientation information of the camera 10 estimated by GPS (Global Positioning System). Alternatively, position and orientation information may be obtained from sensors other than the camera, such as a wheel encoder. Alternatively, the initial value acquisition unit 14 may acquire integrated position and orientation information by using the average or median of these multiple pieces of position and orientation information.

[0098] The calculation unit 13 calculates at least one of the camera's position and orientation, and the 3D information of the image features, from the correspondence relationships selected by the selection unit 12 and the initial values ​​included in the camera's position and orientation information acquired by the initial value acquisition unit 14. Specifically, first, the calculation unit 13 calculates the initial values ​​of the 3D information of the surrounding environment from the initial values ​​of the camera's position and orientation and the correspondence relationships using triangulation. Next, the calculation unit 13 reprojects the initial values ​​of the 3D information onto the image using the camera's position and orientation, and optimizes the camera's position and orientation and the 3D information of the surrounding environment using Bundle Adjustment, which minimizes the reprojection error.

[0099] At this time, the calculation unit 13 may optimize the entire scene in a single Bundle Adjustment, or it may sequentially increase the number of images to be optimized. For example, one can consider a method in which the series of processes are repeated while sequentially increasing the number of images to be optimized. Specifically, after performing partial optimization, the calculation unit 13 adds the position and orientation of the camera 10 to be optimized using the initial position and orientation values ​​of the camera 10 obtained by the initial value acquisition unit 14, or the initial position and orientation values ​​of the camera 10 calculated by solving a PnP problem from the correspondence between the optimized 3D information and the camera 10. Furthermore, the calculation unit 13 increases the 3D information of the surrounding environment by triangulation using these initial values ​​and the correspondence. After that, the calculation unit 13 repeats the series of processes, such as performing optimization again with Bundle Adjustment.

[0100] Finally, an example of the hardware configuration of the information processing device 1 of this embodiment will be described.

[0101] [Example hardware configuration] Figure 9 shows an example of the hardware configuration of the information processing device 1 in an embodiment. The information processing device 1 includes a processor 201, a main memory 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main memory 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0102] Furthermore, the information processing device 1 may not be equipped with some of the above-described components. For example, if the information processing device 1 can utilize the input and display functions of an external device, the information processing device 1 may not be equipped with the display device 204 and the input device 205.

[0103] The processor 201 executes the program read from the auxiliary storage device 203 into the main memory 202. The main memory 202 is memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The auxiliary storage device 203 is such as an HDD (Hard Disk Drive) and a memory card.

[0104] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the information processing device 1. The display device 204 and the input device 205 may be implemented by a touch panel or the like that has both display and input functions. The communication device 206 is an interface for communicating with other devices.

[0105] For example, a program executed by the information processing device 1 is provided as a computer program product, recorded in an installable or executable file format on a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0106] Alternatively, for example, the program executed by the information processing device 1 may be stored on a computer connected to a network such as the Internet, and provided by allowing users to download it via the network.

[0107] Alternatively, the information processing device 1 may be configured to provide programs via a network such as the Internet without requiring users to download them. Specifically, it may be configured, for example, as an ASP (Application Service Provider) type cloud service.

[0108] Alternatively, for example, the program for the information processing device 1 may be pre-installed and provided in ROM or the like.

[0109] The program executed by the information processing device 1 has a modular configuration that includes functions that can also be implemented by the program, as described above. In actual hardware terms, each of these functions is loaded onto the main memory 202 by the processor 201 reading and executing the program from the storage medium. In other words, each of the above function blocks is generated on the main memory 202.

[0110] Furthermore, some or all of the above-mentioned functions may be implemented using hardware such as an IC (Integrated Circuit) instead of software.

[0111] Alternatively, multiple processors 201 may be used to implement each function, in which case each processor 201 may implement one of the functions, or two or more of the functions.

[0112] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0113] 1. Information Processing Device 11. Correspondence Acquisition Unit 12. Sorting Section 13 Calculation Section 14. Initial Value Acquisition Section 201 Processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input device 206 Communication equipment 210 Bus

Claims

1. Computers, A correspondence relationship acquisition unit calculates multiple image features from multiple images captured by a camera and acquires the correspondence relationship between the multiple image features, A selection unit that selects multiple correspondences based on the effectiveness of the correspondence and the effect on at least one of the multiple images when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted, A calculation unit calculates at least one of the camera's position and orientation, and the three-dimensional information of the image features, from the correspondence selected from the plurality of correspondences. To make it function as, The effectiveness of the aforementioned correspondence is that the amount of movement between the positions indicated by the two image features in the aforementioned correspondence is greater than the amount of movement between them. The aforementioned effectiveness threshold indicates a predetermined amount of movement. The sorting unit sorts the multiple correspondences such that it does not delete correspondences where the amount of movement between the positions indicated by the two corresponding image features is larger. program.

2. A computer, A correspondence relationship acquisition unit calculates multiple image features from multiple images captured by a camera and acquires the correspondence relationship between the multiple image features, A selection unit that selects multiple correspondences based on the effectiveness of the correspondence and the effect on at least one of the multiple images when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted, A calculation unit calculates at least one of the camera's position and orientation, and the three-dimensional information of the image features, from the correspondence selected from the plurality of correspondences. To make it function as, The effectiveness of the aforementioned correspondence is that the number of images containing the image features for which the aforementioned correspondence exists is greater. The aforementioned effectiveness threshold indicates a predetermined number of images. The selection unit selects the multiple correspondences such that it does not delete correspondences where the number of images containing the corresponding image features is greater than the number of correspondences. program.

3. The aforementioned computer, The unit further functions as an initial value acquisition unit that acquires at least one of the initial value of the camera's shooting position and the initial value of the camera's orientation. The calculation unit further calculates the camera's position and orientation, and at least one of the three-dimensional information of the image features, based on at least one of the initial value of the camera's shooting position and the initial value of the camera's orientation. The program according to claim 1 or 2.

4. The effect on at least one of the aforementioned multiple images is the variation in the position of image features distributed in each of the aforementioned images. The selection unit selects the multiple correspondences such that, when correspondences with an effectiveness lower than the effectiveness threshold are removed, the evaluation value for evaluating the variability of the position of the image features does not fall below the influence threshold. The program according to claim 1 or 2.

5. The effect on at least one of the aforementioned multiple images is the total number of the aforementioned correspondences in the aforementioned multiple images. The selection unit selects the multiple correspondences such that, when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of correspondences in the multiple images does not fall below the influence threshold. The program according to claim 1 or 2.

6. The selection unit changes the effectiveness threshold until the total number of correspondences approaches a predetermined total, and selects the multiple correspondences such that when correspondences with an effectiveness lower than the effectiveness threshold are deleted, the total number of correspondences in the multiple images does not fall below the influence threshold. The program according to claim 5.

7. The effect on at least one of the aforementioned multiple images is the total number of image features contained in each of the aforementioned images. The selection unit selects the multiple correspondences such that, when correspondences with an effectiveness lower than the effectiveness threshold are removed, the total number of image features does not fall below the influence threshold. The program according to claim 1 or 2.

8. The selection unit changes the effectiveness threshold until the total number of image features included in each image approaches a predetermined total number, and selects the multiple correspondences such that when correspondences with an effectiveness lower than the effectiveness threshold are removed, the total number of image features included in each image does not fall below the influence threshold. The program according to claim 7.

9. A correspondence relationship acquisition unit calculates multiple image features from multiple images captured by a camera and acquires the correspondence relationship between the multiple image features, A selection unit that selects multiple correspondences based on the effectiveness of the correspondence and the effect on at least one of the multiple images when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted, The system includes a calculation unit that calculates at least one of the camera's position and orientation, and the three-dimensional information of the image features, from the correspondence selected from the plurality of correspondences. The effectiveness of the aforementioned correspondence is that the amount of movement between the positions indicated by the two image features in the aforementioned correspondence is greater than the amount of movement between them. The aforementioned effectiveness threshold indicates a predetermined amount of movement. The sorting unit sorts the multiple correspondences such that it does not delete correspondences where the amount of movement between the positions indicated by the two corresponding image features is larger. Information processing device.

10. A correspondence relationship acquisition unit that calculates multiple image features from multiple images captured by a camera and acquires the correspondence relationship of the multiple image features, A selection unit that selects multiple correspondences based on the effectiveness of the correspondence and the effect on at least one of the multiple images when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted, The system includes a calculation unit that calculates at least one of the camera's position and orientation, and the three-dimensional information of the image features, from the correspondence selected from the plurality of correspondences. The effectiveness of the aforementioned correspondence is that the number of images containing the image features for which the aforementioned correspondence exists is greater. The aforementioned effectiveness threshold indicates a predetermined number of images. The selection unit selects the multiple correspondences such that it does not delete correspondences where the number of images containing the corresponding image features is greater than the number of correspondences. Information processing device.

11. The information processing device calculates multiple image features from multiple images captured by a camera and obtains the correspondence between the multiple image features. The information processing device selects a plurality of correspondences based on the effectiveness of the correspondence and the effect on at least one of the plurality of images if a correspondence whose effectiveness is lower than the effectiveness threshold is deleted. The information processing device includes the step of calculating at least one of the camera's position and orientation and the three-dimensional information of the image features from the correspondence selected from the plurality of correspondences, The effectiveness of the aforementioned correspondence is that the amount of movement between the positions indicated by the two image features in the aforementioned correspondence is greater than the amount of movement between them. The aforementioned effectiveness threshold indicates a predetermined amount of movement. The selection step involves selecting multiple correspondences such that correspondences with a larger displacement between the positions indicated by the two corresponding image features are not deleted. Information processing methods.

12. An information processing device calculates multiple image features from multiple images captured by a camera and obtains the correspondence between the multiple image features, The information processing device selects a plurality of correspondences based on the effectiveness of the correspondence and the effect on at least one of the plurality of images if a correspondence whose effectiveness is lower than the effectiveness threshold is deleted. The information processing device includes the step of calculating at least one of the camera's position and orientation and the three-dimensional information of the image features from the correspondence selected from the plurality of correspondences, The effectiveness of the aforementioned correspondence is that the number of images containing the image features for which the aforementioned correspondence exists is greater. The aforementioned effectiveness threshold indicates a predetermined number of images. The selection step involves selecting multiple correspondences such that the number of images containing the corresponding image features is greater than the number of correspondences, and not deleting the correspondences. Information processing methods.

Images