Program, information processing device, and information processing method

Abstract

To provide a program, an information processing device, and an information processing method capable of reducing computational costs for at least one computation of the positional attitude of a camera and of three-dimensional information of an image feature while maintaining accuracy.SOLUTION: The program of the present embodiment makes a computer function as a correspondence acquisition part, a selection part, and a calculation part. The correspondence acquisition part calculates multiple image features from multiple images captured by a camera and acquires the correspondence of the multiple image features. The selection part selects the multiple correspondences based on correspondence effectiveness and on the influence on at least one image of the multiple images when the correspondence where the effectiveness is lower than effectiveness threshold is deleted. The calculation part calculates at least one of the positional attitudes of the camera and of three-dimensional information of the image feature from the selected correspondence of the multiple correspondences.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) are attracting attention as technologies that use cameras to simultaneously estimate the vehicle's position and measure the surrounding environment in three dimensions. Cameras can be used indoors and are relatively inexpensive and easy to use, so there are high expectations for applications of camera-based technologies. Examples of applications include measuring the position of a vehicle or target during infrastructure maintenance and inspection, estimating the vehicle's movement in accident footage, and digital archiving of cultural assets for the purpose of protecting cultural heritage. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5953142 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, it has been difficult to reduce the computational cost required to calculate at least one of the camera position and orientation and the three-dimensional information of image features while maintaining accuracy. [Means for solving the problem]

[0005] A program according to an embodiment causes a computer to function as a correspondence acquisition unit, a selection unit, and a calculation unit. The correspondence acquisition unit calculates a plurality of image features from a plurality of images captured by a camera, and acquires correspondences between the plurality of image features. The selection unit selects the plurality of correspondences based on the validity of the correspondences and the effect on at least one of the plurality of images when a correspondence whose validity is lower than a validity threshold is deleted. The calculation unit calculates at least one of the position and orientation of the camera and three-dimensional information of the image features from the correspondence selected from the plurality of correspondences. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a diagram showing an example of a functional configuration of the information processing apparatus according to the embodiment. [Figure 2A] FIG. 4 is a diagram showing Example 1 of the movement amount of an image feature according to the embodiment. [Figure 2B] 11A and 11B are diagrams showing a second example of the movement amount of an image feature according to the embodiment. [Diagram 3] 6A and 6B are diagrams for explaining the number of images including image features in a corresponding relationship according to the embodiment. [Figure 4A] FIG. 4 is a diagram showing Example 1 of the distribution of image features according to the embodiment. [Figure 4B] FIG. 11 is a diagram showing a second example of distribution of image features according to the embodiment. [Figure 5A] FIG. 1 is a diagram for explaining image feature matching in SLAM. [Figure 5B] FIG. 1 is a diagram for explaining image feature correspondence in SfM. [Figure 6] 10 is a flowchart illustrating an example of processing by a selection unit according to the embodiment. [Figure 7] 11 is a flowchart showing an example of processing by a selection unit according to the first modified example of the embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a functional configuration of an information processing device according to a second modified example of the embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a hardware configuration of an information processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of a program, an information processing device, and an information processing method will be described in detail with reference to the accompanying drawings.

[0008] First, we will explain the difference between SLAM and SfM. SLAM is generally used in applications that require real-time processing, such as autonomous driving, and performs position estimation sequentially using a continuous image sequence as input. Since it prioritizes processing speed over processing accuracy, it is designed to narrow the range when optimizing the camera position and orientation and the surrounding environment of the camera.

[0009] On the other hand, SfM is based on offline processing, and the input is not limited to a continuous image sequence; it uses the entire image set and performs optimization with an emphasis on accuracy.

[0010] SfM, which emphasizes accuracy, is suitable for applications such as measuring shooting positions and target positions during maintenance inspections and patrols, and estimating the movement of the vehicle in accident footage, as it does not require real-time processing. On the other hand, even in SfM, which uses the entire image group, speeding up the processing time has the effect of reducing server usage fees and reducing waiting times for users using applications, and there is a demand for technology to increase the speed of SfM.

[0011] A typical processing flow in SfM includes detecting feature points from a group of images, creating corresponding points by matching feature points, estimating the initial value of the camera position and orientation, estimating the initial value of the corresponding 3D points, and optimizing the estimated camera position and orientation and 3D points. The optimization process is often performed using Bundle Adjustment, which minimizes the reprojection error when 3D points are reprojected onto the images using the camera position and orientation.

[0012] In general, the optimization process takes the longest time in the processing flow, and the more corresponding points there are, the more estimation variables there are. Therefore, the processing speed can be increased by reducing the number of corresponding points, but there is a problem that accuracy decreases even if the number of corresponding points is reduced randomly. For example, the number of corresponding points can be reduced by reducing the number of detected feature points or by increasing the lower limit of the number of images that are matched at the corresponding points, but it is difficult to uniquely determine the optimal parameters for maintaining accuracy because they differ depending on the scene.

[0013] Below, we will explain an embodiment in which correspondences between image features (e.g., feature points, etc.) contained in two or more camera images are selected, and at least one of the camera position and orientation and three-dimensional information of the image features is calculated based on the selected correspondences.

[0014] [Example of functional configuration] 1 is a diagram illustrating an example of a functional configuration of an information processing device 1 according to an embodiment. The information processing device 1 according to the embodiment includes a correspondence relationship acquisition unit 11, a selection unit 12, and a calculation unit 13.

[0015] The correspondence acquisition unit 11 acquires the correspondence between image features on two or more camera images. Specifically, the correspondence acquisition unit 11 calculates image features that indicate characteristic areas on the camera images, and performs correspondence between image features that indicate the same location from the image features on two or more camera images. Here, the image features may be in the form of points (feature points) or lines. The image features are calculated from the luminance gradient of the image, etc.

[0016] To calculate image features, feature point detection algorithms such as Scaled Invariance Feature Transform (SIFT), Speed-Upped Robust Feature (SURF), and Accelerated KAZE (AKAZE) may be used, or a deep neural network may be used.

[0017] The correspondence of image features may be performed based on the similarity (degree of agreement) of feature amounts quantified by a feature descriptor, or may be calculated based on the similarity of pixels in the peripheral area of ​​the image feature. For the calculation of the feature descriptor, a feature point detection algorithm such as SIFT, SURF, or AKAZE may be used, or a deep neural network may be used.

[0018] The similarity of the feature is calculated, for example, from the L1 norm, L2 norm, Hamming distance, cosine similarity, etc. between N-dimensional feature. The similarity of pixels in the surrounding area is calculated, for example, from the Sum of Squared Difference (SSD), Sum of Absolute Difference (SAD), normalized cross-correlation, etc.

[0019] When the similarity is equal to or greater than a predetermined threshold, the correspondence acquisition unit 11 associates image features on multiple images showing the same location, and stores the association of the image features in the storage device of the information processing device 1. The association of image features is not limited to between two images, but is performed for two or more images. That is, the correspondence acquisition unit 11 obtains a correspondence in which image features on two or more images are associated with each other.

[0020] The selection unit 12 selects correspondence relationships that are effective for calculating at least one of the position and orientation of the camera and the three-dimensional information of the image features from the correspondence relationships obtained by the correspondence relationship acquisition unit 11. The selection unit 12 removes only correspondence relationships that are unnecessary for maintaining the accuracy of the position and orientation and the three-dimensional information, thereby making it possible to reduce the number of estimated parameters for optimization processing such as bundle adjustment in SfM and to reduce processing time.

[0021] Here, when the image features are feature points, the estimation parameters are the camera position and orientation and the 3D points for each corresponding point, and reducing the correspondences reduces the number of 3D points, which directly reduces the estimation parameters. On the other hand, the correspondences necessary to maintain accuracy are left, so the accuracy of the position and orientation and 3D information can be maintained.

[0022] In the following, a method for selecting valid correspondences will be described.

[0023] In general, the following five types of correspondences can be considered as effective correspondences that contribute to the estimation accuracy of position and orientation in SfM. · The amount of movement of image features across images in the correspondence is larger. A larger number of images contain image features that have corresponding relationships with image features in other images. · The similarity (matching) of the correspondence relationships is higher. Image features that have corresponding relationships with image features in other images are more widely distributed within the image. -Higher confidence in correspondence.

[0024] Examples of the above five indicators for evaluating the validity of the correspondence will be explained in turn.

[0025] First, the amount of movement of image features in a corresponding relationship refers to the amount of movement of image features showing the same location between different images (the amount of movement between the positions shown by two corresponding image features). When the image features are feature points, they are generally called flow.

[0026] Fig. 2A is a diagram showing Example 1 of the amount of movement of an image feature according to an embodiment. Fig. 2B is a diagram showing Example 2 of the amount of movement of an image feature according to an embodiment. The examples of Figs. 2A and 2B show a case where an image feature is represented by a feature point.

[0027] For example, the correspondence relationship 101a in Fig. 2A is obtained from corresponding feature points (feature points showing the same location) in two different images. Similarly, the correspondence relationship 101b in Fig. 2B is obtained from corresponding feature points (feature points showing the same location) in two different images.

[0028] The greater the distance between corresponding feature points in two different images, the greater the flow indicating the amount of movement. In the examples of Figures 2A and 2B, for example, the amount of movement of the image feature on the image in correspondence relationship 101a is greater than the amount of movement of the image feature on the image in correspondence relationship 101b.

[0029] Note that Figures 2A and 2B are examples of flows between two images. When feature points are matched between two or more images, a method may be used in which the flow is calculated using, for example, the difference between the position of the feature on the image with the earliest observed time in the matching and the position of the feature on the image with the latest observed time.

[0030] Furthermore, for example, when matching of feature points is performed on two or more images, a method may be used in which the difference between the maximum and minimum values ​​of the x coordinates of feature points on the images and the difference between the maximum and minimum values ​​of the y coordinates of feature points on the images are calculated, and the flow between feature points with the larger difference is calculated.

[0031] In SfM, the position and orientation of the camera and the 3D information of the image features (3D information of the camera's surrounding environment) are estimated from the change in the appearance of the object. Therefore, the amount of movement of the image features on the image (e.g., the flow of feature points) is an important index for estimating the position and orientation of the camera and the 3D information of the image features.

[0032] For example, the selection unit 12 selects a plurality of correspondences so as not to delete a correspondence in which the amount of movement between the positions indicated by two corresponding image features is greater.

[0033] Next, the number of images that contain image features that correspond to image features of other images refers to the number of images that correspond to image features that indicate the same location, obtained by observing the same location as an image feature in a different image.

[0034] Fig. 3 is a diagram for explaining the number of images including image features 100a and 100b in a corresponding relationship in the embodiment. The example of Fig. 3 shows a case where image features 100a and 100b are feature points. The example of Fig. 3 also shows a case where images are captured by a camera 10 mounted on a moving object such as an automobile while moving. Image feature 100a is included in two images captured by the camera 10. On the other hand, image feature 100b is included in six images captured by the camera 10.

[0035] In the example of FIG. 3, the number of images containing image feature 100b is greater than the number of images containing image feature 100a, so the correspondence relationship of image feature 100b is more valid than the correspondence relationship of image feature 100a.

[0036] In SfM, the more a location is observed in multiple images, the more accumulated errors such as scale drift in the position and orientation of the camera 10 can be reduced, so the number of images associated with image features indicating the same location is an important indicator.

[0037] For example, the selection unit 12 selects a plurality of correspondences so as not to delete a correspondence that has a larger number of images that include corresponding image features.

[0038] Next, the similarity (degree of agreement) of the correspondence refers to the similarity of the correspondence calculated by the correspondence acquisition unit 11. The higher the similarity of the associated image features, the fewer the erroneous correspondences will be, and the higher the estimation accuracy of the position and orientation of the camera 10 will be, so the similarity of the correspondence is an important index.

[0039] For example, the selection unit 12 selects a plurality of correspondences so as not to delete a correspondence having a higher degree of similarity between image features in the correspondences.

[0040] Next, the distribution of image features refers to the distribution of image features on a single image.

[0041] Fig. 4A is a diagram showing Example 1 of the distribution of image features according to the embodiment. Fig. 4B is a diagram showing Example 2 of the distribution of image features according to the embodiment. The examples of Figs. 4A and 4B show a case where the image features are feature points. The positions of the image features included in the image of Fig. 4B are biased toward the left side in the image. On the other hand, the image features included in the image of Fig. 4A have greater variation and are distributed more widely in the image than the image features included in the image of Fig. 4B.

[0042] The distribution of image features is an important indicator because the more widely distributed image features that correspond to image features in other images are within an image, the more diverse the changes in the appearance of the object will be and the higher the accuracy of estimating the position and orientation of the camera 10 will be.

[0043] For example, the selection unit 12 selects multiple correspondences so that when a correspondence whose validity is lower than a validity threshold is deleted, the evaluation value that evaluates the variation in the position of image features within an image does not become smaller than an influence threshold.

[0044] Next, the reliability of the correspondence is calculated by a neural network that inputs the correspondence and outputs the reliability of the correspondence. The reliability of the correspondence is a direct index that indicates the validity of the correspondence. The reliability of the correspondence is output so as to be higher as the estimation accuracy of the position and orientation of the camera 10 becomes higher, so the reliability of the correspondence is an important index.

[0045] For example, the correspondence acquisition unit 11 further acquires the reliability by using a neural network that calculates the reliability of the correspondence. Then, the selection unit selects the multiple correspondences so as not to delete a correspondence with a higher reliability.

[0046] Correspondences with large values ​​for the five indices listed above are effective for calculating position and orientation and surrounding 3D information, but it is not possible to simply select only correspondences with large or small indices. This is because the characteristics of the correspondences obtained differ depending on the scene. In scenes with a larger number of image features, for example, it is no problem to leave valid correspondences that meet any of the five indices listed above and delete the remaining correspondences. However, the fewer the number of image features in a scene, the more important it is to ensure a large number of correspondences, so it is necessary to leave correspondences that do not meet any of the five indices listed above.

[0047] Therefore, the selection unit 12 selects the correspondence relationship while also considering the influence on the entire scene (at least one of the multiple images). In other words, when selecting whether a certain correspondence relationship is valid or not, the selection unit 12 also considers the influence on camera images other than the camera image corresponding to the correspondence relationship related to the selection. Specifically, the selection unit 12 performs the selection while considering not only the values ​​of the five indices for evaluating the valid correspondence relationship in the correspondence relationship related to the selection, but also the value of another indices for evaluating the influence on the entire scene.

[0048] In addition, selecting correspondences taking into account the correspondences of the entire scene is possible only with SfM, which is based on offline processing, and not with SLAM, which is based on sequential processing.

[0049] Fig. 5A is a diagram for explaining image feature matching in SLAM, and Fig. 5B is a diagram for explaining image feature matching in SfM.

[0050] As shown in Figure 5A, SLAM does not know image information at future times, so when image features are detected and matched recently, the number of matched images may be small and the amount of movement of features on the images may be small. However, in images at future times, these image features may be important for estimating keypoints (important image features), etc. Therefore, in SLAM, it is difficult to select image features while also taking into account the value of another index that evaluates the influence on the entire scene.

[0051] On the other hand, in the case of SfM, offline processing is assumed, and the correspondence relationship of the entire scene for optimizing the calculation cost required for calculating at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features is known in advance, as shown in Fig. 5B. Therefore, it becomes possible to select image features by taking into account the value of another index that evaluates the influence on the entire scene.

[0052] As shown in FIGS. 5A and 5B, the speed-up according to this embodiment cannot be achieved in SLAM, but can be applied to SfM, which is a difference between this embodiment and existing algorithms for speed-up and the like in SLAM.

[0053] When selecting based on each index while taking the entire scene into consideration, a method of calculating a threshold for each index for each scene is conceivable. For example, a method of determining a threshold (validity threshold) based on the average or percentile of each index for all correspondences is conceivable. The selection unit 12 selects (and does not delete) correspondences that are equal to or greater than the validity threshold as correspondences that are valid for calculating the position and orientation of the camera and 3D information about the surrounding environment.

[0054] In addition, when performing the selection, a determination process is performed to take into account the influence on the entire scene in order to maintain the calculation accuracy of the position and orientation of the camera 10 and the three-dimensional information of the image features (three-dimensional information of the surrounding environment).

[0055] For example, the influence on at least one of the multiple images is the variation in the positions of image features distributed among the images. When a correspondence relationship whose effectiveness is lower than the effectiveness threshold is deleted, the selection unit 12 selects the multiple correspondence relationships so that an evaluation value for evaluating the variation in the positions of the image features is not smaller than the influence threshold.

[0056] Furthermore, for example, as a determination process for taking the entire scene into consideration, a process is performed in which at least one of the total number of correspondence relationships and the total number of image features in one image is determined using a preset threshold value.

[0057] Here, the total number of correspondences is equal to the number of sets of image features (sets of image features showing the same location) that are associated with image features of other images through any of the correspondences. A set of image features corresponding to one correspondence includes image features showing the same location in two or more different images.

[0058] When a correspondence is deleted, the set of image features corresponding to this correspondence is also deleted. The number of image features in the set of image features is 2 if the correspondence is between two images, and n if the correspondence is between n images.

[0059] The smaller the total number of correspondence relationships is, or the smaller the number of image features on one image is, the more difficult it becomes to maintain the accuracy of calculation of the position and orientation of the camera 10 and the three-dimensional information of the image features (three-dimensional information of the surrounding environment). Therefore, the processing of the selection unit 12 of the embodiment is performed according to, for example, the flowchart in FIG. 6 below.

[0060] 6 is a flowchart showing an example of processing by the selection unit 12 of the embodiment. First, the selection unit 12 selects one correspondence to be processed, and determines whether or not an index for evaluating the validity of the correspondence is equal to or greater than a threshold (validity threshold) (step S1). For example, the index for evaluating the validity may be any one or more of the above-mentioned five examples of the index for evaluating the validity.

[0061] If the index is equal to or greater than the threshold (Yes at step S1), the selection unit 12 selects a new correspondence relationship to be processed from among the unprocessed correspondence relationships, and executes the process at step S1.

[0062] If the index is less than the threshold (step S1, No), the selection unit 12 determines whether or not there is an image in the entire scene (multiple images) in which the number of image features in the image will be less than or equal to the threshold when the correspondence to be processed is deleted (step S2).

[0063] If there is an image in which the number of image features in the image is equal to or smaller than the threshold (Yes in step S2), the selection unit 12 selects a new correspondence to be processed from among the unprocessed correspondences, and returns to the process of step S1. That is, the selection unit 12 selects a plurality of correspondences such that the total number of image features in the image is not smaller than the influence threshold when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted.

[0064] If there is no image in which the number of image features is below the threshold (step S2, No), the selection unit 12 determines whether the total number of correspondences will be below the threshold when the correspondence to be processed is deleted (step S3).

[0065] If the total number of correspondences is equal to or smaller than the threshold (Yes at step S3), the selection unit 12 selects a new correspondence to be processed from among the unprocessed correspondences, and returns to the process at step S1. That is, the selection unit 12 selects multiple correspondences such that the total number of correspondences in multiple images does not become smaller than the influence threshold when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted.

[0066] If the total number of correspondence relationships is not equal to or less than the threshold (No at step S3), the selection unit 12 deletes the correspondence relationship to be processed, selects a new correspondence relationship to be processed from the unprocessed correspondence relationships, and returns to the process at step S1.

[0067] When the processing for all the correspondence relationships is completed, the selection processing according to the flowchart in FIG. 6 ends.

[0068] The above is the explanation of the selection unit 12.

[0069] 1, the calculation unit 13 calculates at least one of the position and orientation of the camera 10 and three-dimensional information of the image features from the correspondence relationships selected by the selection unit 12. Specifically, the calculation unit 13 first calculates a fundamental matrix from the information on the selected correspondence relationships, and calculates an initial value of the position and orientation of the camera 10.

[0070] Next, the calculation unit 13 calculates an initial value of three-dimensional information of the image feature (three-dimensional information of the surrounding environment) by triangulation from the initial value of the position and orientation of the camera 10 and the correspondence relationship. Finally, the calculation unit 13 reprojects the initial value of the three-dimensional information onto the image based on the position and orientation of the camera 10, and optimizes the position and orientation of the camera 10 and the three-dimensional information of the surrounding environment by bundle adjustment that minimizes the reprojection error.

[0071] At this time, the calculation unit 13 may optimize the entire scene by one Bundle Adjustment, or may sequentially increase the number of images to be optimized. For example, a method of repeating a series of processes while sequentially increasing the number of images to be optimized is conceivable. Specifically, the calculation unit 13 may perform partial optimization, and then calculate an initial value of the position and orientation of the camera 10 by solving a Perspective-n-Point (PnP) problem from the three-dimensional information and the correspondence relationship after the optimization. Then, the calculation unit 13 may increase the three-dimensional information of the surrounding environment by triangulation using the initial value and the correspondence relationship, and then perform optimization again by Bundle Adjustment.

[0072] As described above, in the embodiment, the correspondence acquisition unit 11 calculates a plurality of image features from a plurality of images captured by the camera 10, and acquires correspondences between the plurality of image features. The selection unit 12 selects the plurality of correspondences based on the validity of the correspondences and the effect on at least one of the plurality of images when a correspondence whose validity is lower than a validity threshold is deleted. Then, the calculation unit 13 calculates at least one of the position and orientation of the camera 10 and three-dimensional information of the image features from the correspondence selected from the plurality of correspondences.

[0073] As a result, according to the embodiment, it is possible to reduce the calculation cost required for calculating at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features while maintaining accuracy. For example, by using the correspondence relationships selected by the selection unit 102, it is possible to speed up the optimization process in the calculation unit 13. In addition, since the selection unit 12 selects the correspondence relationships while taking into consideration the influence on the entire scene (at least one of the multiple images), it is possible to maintain accuracy related to the calculation of at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features.

[0074] (First Modification of the Embodiment) Next, a first modification of the embodiment will be described. In the description of the first modification, the same description as in the embodiment will be omitted, and only the differences from the embodiment will be described. In the first modification, a method for adaptively calculating a threshold value of an index for evaluating the validity of a correspondence relationship by repeated processing will be described.

[0075] In the first modification, first, a lower threshold for the total number of correspondence relationships or a lower threshold for the number of image features on one image is defined in advance.

[0076] Next, for one or more of the five indices for evaluating the effectiveness described in the embodiment, an initial value of the lower limit threshold is defined.

[0077] Hereinafter, the method of calculating the threshold value by the iterative process will be described with reference to the flowchart in FIG.

[0078] Fig. 7 is a flowchart showing an example of processing by the selection unit 12 in the first modified example of the embodiment. In the example of Fig. 7, the amount of movement of features on the images and the number of images corresponding to the correspondence are used as indices for selecting valid correspondences, but a combination of one or more arbitrary indices for evaluating the validity of the correspondences may be used.

[0079] First, the selection unit 12 selects one correspondence to be processed, and determines whether the number of images corresponding to the correspondence (the number of images including image features having the correspondence) is equal to or greater than a threshold (step S11).

[0080] If the number of images associated with the correspondence relationship is equal to or greater than the threshold (Yes at step S11), the selection unit 12 selects a new correspondence relationship to be processed from the unprocessed correspondence relationships, and executes the process at step S11.

[0081] If the number of images corresponding to the correspondence relationship is less than the threshold (No at step S11), the selection unit 12 determines whether the amount of movement on the image of the image feature in the correspondence relationship to be processed is equal to or greater than the threshold (step S12).

[0082] If the amount of movement of the image feature on the image is equal to or greater than the threshold (Yes at step S12), the selection unit 12 selects a new correspondence relationship to be processed from among the unprocessed correspondence relationships, and executes the process at step S11.

[0083] In other words, the correspondence relationship for which the answer is Yes in step S11 or S12 is determined by the selection unit 12 to be a correspondence relationship that is valid for calculating at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features, and is stored in the storage device of the information processing device 1.

[0084] If the amount of movement of the image feature in the image is less than the threshold value (step S12, No), the selection unit 12 determines whether or not there is an image in the entire scene (multiple images) in which the number of image features in the image will be less than or equal to the threshold value when the correspondence to be processed is deleted (step S13).

[0085] If there is an image in which the number of image features is equal to or less than the threshold (Yes in step S13), the correspondence relationship to be processed is not deleted because deleting the correspondence relationship would be detrimental to the calculation process of at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features. The selection unit 12 selects a new correspondence relationship to be processed from the unprocessed correspondence relationships, and returns to the process of step S11.

[0086] If there is no image in which the number of image features is below the threshold (step S13, No), the selection unit 12 determines whether the total number of correspondences will be below the threshold when the correspondence to be processed is deleted (step S14).

[0087] If the total number of correspondence relationships is equal to or less than the threshold value (Yes at step S14), no more correspondence relationships can be deleted, and the selection process ends.

[0088] If the total number of correspondence relationships is not equal to or less than the threshold (step S14, No), the selection unit 12 deletes the correspondence relationship to be processed because the effect on the calculation process of at least one of the position and orientation of the camera 10 and the three-dimensional information of the image features is minor. Then, the selection unit 12 selects a new correspondence relationship to be processed from the unprocessed correspondence relationships, and returns to the process of step S11.

[0089] When the processing for all the correspondence relationships is completed, the selection unit 12 increases the threshold value for the number of images used in step S11 (step S16), and increases the threshold value for the amount of movement used in step S12 (step S17).

[0090] By repeating the above process a predetermined number of times (N times), it is possible to adaptively calculate a threshold value for an index for evaluating the validity of the correspondence relationship.

[0091] As described above, in Modification 1, for example, the selection unit 12 changes the effectiveness threshold (the threshold for the number of images and the threshold for the amount of movement in the example of FIG. 7) until the total number of image features included in each image approaches a predetermined total number. Then, when a correspondence relationship whose effectiveness is lower than the effectiveness threshold is deleted, the selection unit 12 selects a plurality of correspondence relationships so that the total number of image features included in each image does not become smaller than the influence threshold.

[0092] Furthermore, for example, the selection unit 12 changes the effectiveness threshold (the threshold for the number of images and the threshold for the amount of movement in the example of FIG. 7) until the total number of correspondences approaches a predetermined total number. Then, when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted, the selection unit 12 selects the multiple correspondences such that the total number of correspondences in the multiple images does not become smaller than the influence threshold.

[0093] As a result, according to variant example 1, unnecessary correspondences can be removed in order while maintaining the position and orientation of camera 10, the number of features on one image required to calculate at least one of the three-dimensional information of the image features, and the total number of correspondences.

[0094] (Modification 2 of the embodiment) Next, a second modification of the embodiment will be described. In the description of the second modification, the same description as in the embodiment will be omitted, and only the differences from the embodiment will be described. In the second modification, a case will be described in which an initial value acquisition unit that acquires an initial value of the position and orientation of the camera 10 is further provided.

[0095] [Example of functional configuration] 8 is a diagram showing an example of a functional configuration of an information processing device 1-2 of Modification 2 of the embodiment. The information processing device 1-2 of Modification 2 includes a correspondence relationship acquisition unit 11, a selection unit 12, a calculation unit 13, and an initial value acquisition unit 14. The explanation of the correspondence relationship acquisition unit 11 and the selection unit 12 will be omitted since they are the same as those in the embodiment.

[0096] The initial value acquisition unit 14 acquires position and orientation information including at least one of an initial value of the shooting position of the camera 10 and an initial value of the orientation of the camera 10. Specifically, the initial value acquisition unit 14 acquires the position and orientation information of the camera 10 obtained from information other than the correspondence relationship of image features.

[0097] For example, the initial value acquisition unit 14 acquires position and orientation information of the camera 10 estimated by a neural network that receives a camera image as an input. Alternatively, for example, the initial value acquisition unit 14 acquires position and orientation information of the camera 10 estimated by a global positioning system (GPS). Alternatively, for example, the initial value acquisition unit 14 may acquire position and orientation information obtained from sensor information other than the camera, such as a wheel encoder. Alternatively, for example, the initial value acquisition unit 14 may acquire position and orientation information in which the multiple pieces of position and orientation information are integrated by using an average value or a median value of the multiple pieces of position and orientation information.

[0098] The calculation unit 13 calculates at least one of the position and orientation of the camera 10 and three-dimensional information of the image features from the correspondence selected by the selection unit 12 and the initial value included in the position and orientation information of the camera 10 acquired by the initial value acquisition unit 14. Specifically, the calculation unit 13 first calculates an initial value of the three-dimensional information of the surrounding environment from the initial value of the position and orientation of the camera 10 and the correspondence by triangulation. Next, the calculation unit 13 reprojects the initial value of the three-dimensional information onto the image using the position and orientation of the camera, and optimizes the position and orientation of the camera and the three-dimensional information of the surrounding environment by bundle adjustment that minimizes the reprojection error.

[0099] At this time, the calculation unit 13 may optimize the entire scene by one Bundle Adjustment, or may sequentially increase the number of images to be optimized. For example, a method of repeating a series of processes while sequentially increasing the number of images to be optimized is conceivable. Specifically, after performing partial optimization, the calculation unit 13 adds the position and orientation of the camera 10 to be optimized using the initial value of the position and orientation of the camera 10 obtained by the initial value acquisition unit 14, or the initial value of the position and orientation of the camera 10 calculated by solving the PnP problem from the three-dimensional information after optimization and the correspondence relationship. Furthermore, the calculation unit 13 increases the three-dimensional information of the surrounding environment by triangulation using the initial value and the correspondence relationship. After that, the calculation unit 13 repeats a series of processes such as performing optimization by Bundle Adjustment again.

[0100] Finally, an example of the hardware configuration of the information processing device 1 according to the embodiment will be described.

[0101] [Example of hardware configuration] 9 is a diagram showing an example of a hardware configuration of the information processing device 1 according to the embodiment. The information processing device 1 includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0102] Note that the information processing device 1 may not include some of the above configurations. For example, if the information processing device 1 can use the input function and display function of an external device, the information processing device 1 may not include the display device 204 and the input device 205.

[0103] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a read only memory (ROM) and a random access memory (RAM). The auxiliary storage device 203 is a hard disk drive (HDD), a memory card, or the like.

[0104] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the information processing device 1. The display device 204 and the input device 205 may be realized by a touch panel or the like having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0105] For example, the program executed by the information processing device 1 is provided as a computer program product in the form of a file in an installable or executable format, recorded on a computer-readable storage medium such as a memory card, a hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0106] Also, for example, the program executed by the information processing device 1 may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0107] Also, for example, the program executed by the information processing device 1 may be provided via a network such as the Internet without being downloaded. Specifically, for example, the program may be configured as an ASP (Application Service Provider) type cloud service.

[0108] Also, for example, the program for the information processing device 1 may be provided by being pre-installed in a ROM or the like.

[0109] The program executed by the information processing device 1 has a modular configuration including functions that can be realized by the program among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 201 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 202. In other words, the above-mentioned functional blocks are generated on the main storage device 202.

[0110] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) instead of by software.

[0111] Furthermore, each function may be realized using a plurality of processors 201. In this case, each processor 201 may realize one of the functions, or may realize two or more of the functions.

[0112] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0113] 1. Information processing device 11 Correspondence Acquisition Unit 12 Sorting Department 13 Calculation section 14 Initial value acquisition section 201 Processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input Device 206 Communication Equipment 210 Bus

Claims

1. Computer, a correspondence relationship acquisition unit that calculates a plurality of image features from a plurality of images captured by a camera and acquires a correspondence relationship between the plurality of image features; a selection unit that selects the plurality of correspondence relationships based on the validity of the correspondence relationship and an influence on at least one image among the plurality of images when a correspondence relationship whose validity is lower than a validity threshold is deleted; a calculation unit that calculates at least one of a position and orientation of the camera and three-dimensional information of the image features from a correspondence relationship selected from the plurality of correspondence relationships; A program to function as a

2. The computer, and further functioning as an initial value acquisition unit that acquires at least one of an initial value of the shooting position of the camera and an initial value of the attitude of the camera; the calculation unit calculates at least one of a position and orientation of the camera and three-dimensional information of the image feature based on at least one of an initial value of a shooting position of the camera and an initial value of an orientation of the camera. The program according to claim 1.

3. The validity of the correspondence is determined by a larger amount of movement between the positions of the two image features in the correspondence, The validity threshold indicates a predetermined amount of movement, the selection unit selects the plurality of correspondence relationships so as not to delete a correspondence relationship having a larger amount of movement between positions indicated by the two image features in the correspondence relationship.

3. The program according to claim 1 or 2.

4. The effectiveness of the correspondence is that the number of images that contain the image features with which the correspondence is established is greater; the validity threshold indicates a predetermined number of images; the selection unit selects the plurality of correspondence relationships so as not to delete a correspondence relationship having a larger number of images including the image feature having the correspondence relationship.

3. The program according to claim 1 or 2.

5. The effectiveness of the correspondence is such that the similarity between the image features in the correspondence is higher; the validity threshold indicates a predetermined degree of similarity; the selection unit selects the plurality of correspondence relationships so as not to delete a correspondence relationship having a higher similarity between image features in the correspondence relationship.

3. The program according to claim 1 or 2.

6. the correspondence acquisition unit further acquires the reliability by using a neural network that calculates the reliability of the correspondence; The validity of the correspondence is such that the confidence level is higher; the validity threshold indicates a predetermined confidence level; the selection unit selects the plurality of correspondence relationships so as not to delete a correspondence relationship having a higher reliability.

3. The program according to claim 1 or 2.

7. the influence on at least one image among the plurality of images is a variation in the position of an image feature distributed among the images; the selection unit selects the plurality of correspondence relationships such that an evaluation value for evaluating a variation in positions of the image features is not smaller than an influence threshold value when a correspondence relationship whose effectiveness is lower than the effectiveness threshold value is deleted.

3. The program according to claim 1 or 2.

8. the influence on at least one image among the plurality of images is the total number of the correspondence relationships among the plurality of images; the selection unit selects the plurality of correspondence relationships such that a total number of the correspondence relationships in the plurality of images does not become smaller than an influence threshold when a correspondence relationship whose effectiveness is lower than the effectiveness threshold is deleted.

3. The program according to claim 1 or 2.

9. the selection unit changes the validity threshold until the total number of the correspondence relationships approaches a predetermined total number, and selects the plurality of correspondence relationships such that the total number of the correspondence relationships in the plurality of images does not become smaller than the influence threshold when a correspondence relationship whose validity is lower than the validity threshold is deleted. The program according to claim 8.

10. the influence on at least one image among the plurality of images is a total number of the image features included in each of the images; the selection unit selects the plurality of correspondence relationships such that a total number of the image features does not become smaller than an influence threshold when a correspondence relationship whose effectiveness is lower than the effectiveness threshold is deleted.

3. The program according to claim 1 or 2.

11. the selection unit changes the effectiveness threshold until a total number of the image features included in each of the images approaches a predetermined total number, and selects the plurality of correspondences such that the total number of the image features included in each of the images does not become smaller than the influence threshold when a correspondence whose effectiveness is lower than the effectiveness threshold is deleted. The program according to claim 10.

12. a correspondence relationship acquisition unit that calculates a plurality of image features from a plurality of images captured by a camera and acquires a correspondence relationship between the plurality of image features; a selection unit that selects the plurality of correspondence relationships based on the validity of the correspondence relationship and an influence on at least one image among the plurality of images when a correspondence relationship whose validity is lower than a validity threshold is deleted; a calculation unit that calculates at least one of a position and orientation of the camera and three-dimensional information of the image features from a correspondence relationship selected from the plurality of correspondence relationships; An information processing device comprising:

13. An information processing device calculates a plurality of image features from a plurality of images captured by a camera, and obtains a correspondence relationship between the plurality of image features; a step of selecting, by the information processing device, the plurality of correspondence relationships based on validity of the correspondence relationships and an influence on at least one image among the plurality of images when a correspondence relationship whose validity is lower than a validity threshold is deleted; a step of calculating at least one of a position and orientation of the camera and three-dimensional information of the image features from the correspondence relationship selected from the plurality of correspondence relationships by the information processing device; An information processing method comprising:

Images