Camera posture estimation device, camera posture estimation method, and program

Abstract

A camera orientation estimating device 10 comprises a provisional camera orientation estimating unit 11 which selects a plurality of sets from among sets of corresponding two-dimensional points and three-dimensional points, and uses the selected sets to estimate a plurality of provisional camera orientations; a corresponding point inlier calculating unit 12 that uses the provisional camera orientations to identify inliers among the sets of two-dimensional points and three-dimensional points; a corresponding line segment inlier calculating unit 13 that uses one of the provisional camera orientations to identify inliers among sets of two-dimensional line segments and three-dimensional line segments; a score calculating unit 14 that calculates a score from the sets of two-dimensional points and three-dimensional points that form inliers and the sets of two-dimensional line segments and three-dimensional line segments that form inliers, for each provisional camera orientation; and a camera orientation estimating unit 14 that estimates a camera orientation using at least one of the sets of two-dimensional points and three-dimensional points that form inliers and the sets of two-dimensional line segments and three-dimensional line segments that form inliers, when the score is maximum.

Description

[Technical Field]

[0001] The present disclosure relates to a camera posture estimation device and a camera posture estimation method for estimating the posture of a camera that captured an image, and further relates to a program for realizing these. Mu Regarding. [Background technology]

[0002] Patent Document 1 discloses an apparatus that detects defects such as minute cracks that have occurred on the surface of a structure from an image of the structure taken with a camera and identifies the location of the detected defects. The apparatus disclosed in Patent Document 1 first performs image analysis on the image taken with the camera to detect defects that have occurred on the surface of the structure, and then measures the distance to the defects using a rangefinder. Next, the apparatus disclosed in Patent Document 1 calculates the coordinates of the detected defects in a global coordinate system using coordinate data associated with the image and the measured distance.

[0003] The device disclosed in the patent document can identify the actual position of a defect in a structure detected on an image. However, the device disclosed in the patent document requires a rangefinder in addition to a camera to calculate the position, which makes it difficult to reduce the cost of the device.

[0004] In response to this, a technology has been proposed that pairs the 2D coordinates of feature points of an object in a 2D image with the 3D coordinates of feature points in a 3D model of the object to estimate the camera pose (camera parameters) of the camera that captured the 2D image.This technology makes it possible to calculate the 3D coordinates corresponding to a specified point on the 2D image from the estimated camera pose without using a rangefinder, thereby reducing equipment costs. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2017 / 103982 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology disclosed in the above-mentioned patent document has a problem in that it cannot accurately estimate the camera pose when an object with many feature points is located far from the camera in the 2D image, because in such a case, it is difficult to accurately pair the 2D coordinates of the feature points with the 3D coordinates of the feature points.

[0007] An example of an objective of the present disclosure is to improve the accuracy of estimating the camera posture of a camera that captures a two-dimensional image. [Means for solving the problem]

[0008] In order to achieve the above object, a camera pose estimation device according to one aspect of the present disclosure includes: a temporary camera pose estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation unit that identifies pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation unit that identifies pairs of corresponding two-dimensional and three-dimensional line segments that are inliers, using one of the plurality of temporary camera postures; a score calculation unit that calculates a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation unit that estimates a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; The present invention is characterized in that it is provided with:

[0009] In order to achieve the above object, a camera posture estimation method according to one aspect of the present disclosure includes: a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying pairs of corresponding two-dimensional line segments and three-dimensional line segments that are inliers, using one of the plurality of temporary camera poses; a score calculation step of calculating a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation step of estimating a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; The present invention is characterized by having the following:

[0010] Furthermore, in order to achieve the above object, in one aspect of the present disclosure, program teeth, On the computer, a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying pairs of corresponding two-dimensional line segments and three-dimensional line segments that are inliers, using one of the plurality of temporary camera poses; a score calculation step of calculating a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation step of estimating a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; Run 、 It is characterized by: [Effects of the Invention]

[0011] As described above, according to the present disclosure, it is possible to improve the accuracy of estimating the camera posture of a camera that captured a two-dimensional image. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a camera posture estimation device according to the first embodiment. [Figure 2] FIG. 2 is a diagram specifically illustrating the configuration of the camera posture estimation device according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of two-dimensional image data and three-dimensional data used in the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the operation of the camera posture estimation device according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing the configuration of a camera posture estimation device according to the second embodiment. [Figure 6] FIG. 6 is a flowchart showing the operation of the camera posture estimation device according to the second embodiment. [Figure 7] FIG. 7 is a block diagram showing an example of a computer that realizes the camera posture estimation device according to the first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Embodiment 1) The camera posture estimation device, camera posture estimation method, and program according to the first embodiment will be described below with reference to FIGS.

[0014] [Device configuration] First, a schematic configuration of the camera posture estimation device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of the camera posture estimation device according to the first embodiment.

[0015] 1, camera pose estimation device 10 according to the first embodiment is a device for estimating the pose of a camera that captured an image. As shown in FIG. 1, camera pose estimation device 10 includes a virtual camera pose estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, and a camera pose estimation unit 15.

[0016] The temporary camera posture estimation unit 11 selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera postures using the selected plurality of pairs.

[0017] The corresponding point inlier calculation unit 12 uses one of a plurality of tentative camera postures to identify pairs of corresponding two-dimensional points and three-dimensional points that are inliers.

[0018] The corresponding line segment inlier calculation unit 13 uses one of a plurality of tentative camera postures to identify pairs of corresponding two-dimensional and three-dimensional line segments that are inliers.

[0019] For each estimated tentative camera posture, the score calculation unit 14 calculates a score from pairs of inliers between two-dimensional points and three-dimensional points and pairs of inliers between two-dimensional line segments and three-dimensional line segments, which are identified using the tentative camera posture.

[0020] The camera posture estimation unit 15 estimates the camera posture using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

[0021] In this way, the camera pose estimation device 10 estimates multiple tentative camera poses and calculates a score representing the error between the point and the line segment for each tentative camera pose. Then, the camera pose estimation device 10 uses the score to identify an appropriate pair and calculate the final camera pose. In other words, the camera pose estimation device 10 can estimate the camera pose after performing accurate pairing, and as a result, can improve the estimation accuracy of the camera pose.

[0022] Next, the configuration and functions of the camera posture estimation device according to Embodiment 1 will be specifically described with reference to Figures 2 and 3. Figure 2 is a configuration diagram specifically showing the configuration of the camera posture estimation device according to Embodiment 1. Figure 3 is a diagram showing an example of two-dimensional image data and three-dimensional data used in Embodiment 1.

[0023] As shown in FIG. 2, in the first embodiment, the camera posture estimation device 10 includes a corresponding point setting unit 16, a two-dimensional line segment extraction unit 17, and a three-dimensional line segment extraction unit 18 in addition to the above-mentioned virtual camera posture estimation unit 11, corresponding point inlier calculation unit 12, corresponding line segment inlier calculation unit 13, score calculation unit 14, and camera posture estimation unit 15.

[0024] 2, image data of a two-dimensional image captured by a camera 21 (hereinafter referred to as "two-dimensional image data") is input to the camera posture estimation device 10. This two-dimensional image is an image for which the camera posture is to be estimated, and will also be referred to as a "query image" hereinafter.

[0025] Furthermore, in the first embodiment, camera 21 may be a camera held by a user or may be a camera mounted on a moving object such as a robot or an unmanned aerial vehicle (drone). The two-dimensional image data output from camera 21 may be temporarily stored in a storage device and then transmitted to camera posture estimation device 10, or may be transmitted directly from the camera to camera posture estimation device 10.

[0026] 2, the camera posture estimation device 10 is connected to an image database 30 that stores three-dimensional data so as to be capable of data communication. In the first embodiment, the three-dimensional data includes image data with depth captured by an RGB-D (depth) camera, three-dimensional point cloud data of an object, and the like.

[0027] The corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Specifically, the corresponding point setting unit 16 identifies feature points by calculating, for example, SIFT feature amounts or SURF feature amounts on each image data. Note that if the three-dimensional data is three-dimensional point cloud data, the feature points have already been extracted. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of corresponding two-dimensional points and three-dimensional points (hereinafter referred to as "feature point pairs").

[0028] The two-dimensional line segment extraction unit 17 extracts two-dimensional line segments from the two-dimensional image data 20 sent from the camera 21. Existing technology is used as a method for extracting two-dimensional line segments. The three-dimensional line segment extraction unit 18 extracts three-dimensional line segments from the three-dimensional data stored in the image database 30. Existing technology is also used as a method for extracting two-dimensional line segments. Note that if the three-dimensional data is the above-mentioned depth-added image data, extracting two-dimensional line segments will result in the extraction of three-dimensional line segments.

[0029] When a large number of feature point pairs are set by the corresponding point setting unit 16, the temporary camera posture estimation unit 11 selects an arbitrary number of pairs, for example, three pairs, from among them. Then, the temporary camera posture estimation unit 11 uses the selected feature point pairs to estimate a temporary camera posture, i.e., the extrinsic parameters of the camera 21. The extrinsic parameters are matrices for transforming the positions of the feature points from the world coordinate system to the camera coordinate system, and are composed of a rotation matrix R and a translation vector t. An existing estimation method is used as the method for estimating the extrinsic parameters.

[0030] In the first embodiment, each time a tentative camera pose is estimated by the tentative camera pose estimation unit 11, the corresponding point inlier calculation unit 12 uses the estimated tentative camera pose to identify pairs of corresponding two-dimensional points and three-dimensional points that are inliers. Pairs that are not identified are outliers (outliers, errors).

[0031] Specifically, for each of all the set pairs of feature points, the corresponding point inlier calculation unit 12 projects the three-dimensional point onto a two-dimensional coordinate system (camera coordinate system) using the estimated virtual camera parameters, and calculates the distance between the projected three-dimensional point and the two-dimensional point. If the calculated distance is equal to or less than a preset threshold, the corresponding point inlier calculation unit 12 determines that the corresponding pair of feature points is an inlier.

[0032] First, each time a provisional camera posture is estimated by the provisional camera posture estimation unit 11, the corresponding line segment inlier calculation unit 13 uses the extracted two-dimensional line segment and the extracted three-dimensional line segment to set a pair of corresponding two-dimensional line segments and three-dimensional line segments (hereinafter referred to as a "pair of characteristic line segments").

[0033] Then, each time a tentative camera posture is estimated, the corresponding line segment inlier calculation unit 13 also uses the estimated tentative camera posture to identify pairs of corresponding two-dimensional and three-dimensional line segments that are inliers. Pairs that are not identified are outliers (outliers, errors).

[0034] Specifically, for each of all the set pairs of characteristic line segments, the corresponding line segment inlier calculation unit 13 projects the three-dimensional line segments onto a two-dimensional coordinate system (camera coordinate system) using the estimated virtual camera parameters, and calculates the angle θ between the projected three-dimensional line segment and the two-dimensional line segment. If the calculated angle θ is equal to or smaller than a preset threshold, the corresponding line segment inlier calculation unit 13 determines that the corresponding pair of characteristic line segments is an inlier.

[0035] For each provisional camera posture, the score calculation unit 14 calculates the sum or product of the number of inlier pairs of two-dimensional points and three-dimensional points and the number of inlier pairs of two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score of the provisional camera posture. Note that in the first embodiment, a method other than the method of calculating the sum or product described above may be used as the score calculation method.

[0036] However, if the set of feature points selected by the temporary camera pose estimation unit 11 is incorrect, the temporary camera pose estimated at this time will also be incorrect. Therefore, the temporary camera pose estimation unit 11 selects a set of feature points multiple times while changing the set of feature points to be selected, and estimates a temporary camera pose each time. Then, once the temporary camera pose is estimated, processing is performed by the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, and the score calculation unit 14 each time estimation is performed.

[0037] In this way, the virtual camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, and the score calculation unit 14 form a RANSAC (Random Sample Consensus) loop (the part surrounded by the dashed line in Figure 2), and each process is executed repeatedly.

[0038] The camera posture estimation unit 15 identifies a tentative camera posture that maximizes the score calculated by the score calculation unit 14. The camera posture estimation unit 15 also identifies, for the identified tentative camera posture, a set of feature points that have been determined to be inliers by the corresponding point inlier calculation unit 12 and a set of feature line segments that have been determined to be inliers by the corresponding line segment inlier calculation unit 13.

[0039] Then, the camera posture estimation unit 15 estimates the camera posture using the identified set of inlier feature points, the identified set of inlier feature line segments, or both of them. In this case, an existing estimation method is also used as the camera posture estimation method.

[0040] [Device operation] Next, the operation of camera posture estimation device 10 in embodiment 1 will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the operation of the camera posture estimation device in embodiment 1. In the following description, Figs. 1 to 3 will be referenced as appropriate. Also, in embodiment 1, a camera posture estimation method is implemented by operating camera posture estimation device 10. Therefore, the description of the camera posture estimation method in embodiment 1 will be replaced by the following description of the operation of camera posture estimation device 10.

[0041] 4, first, the corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20 sent from the camera 21, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of feature points (step A1).

[0042] Next, the two-dimensional line segment extracting unit 17 extracts two-dimensional line segments from the two-dimensional image data 20 sent from the camera 21 (step A2).

[0043] Next, the three-dimensional line segment extraction unit 18 extracts three-dimensional line segments from the three-dimensional data stored in the image database 30 (step A3). The order of steps A1 to A3 is not particularly limited, and these steps may be executed simultaneously.

[0044] Next, the temporary camera posture estimation unit 11 selects any number of pairs of feature points from the many pairs of feature points set in step A1 (step A4), and then estimates a temporary camera posture using the pairs of feature points selected in step A4 (step A5).

[0045] Next, the corresponding point inlier calculation unit 12 uses the tentative camera posture estimated in step A5 to identify pairs of inlier feature points from among the pairs of feature points set in step A1 (step A6).

[0046] Next, the corresponding line segment inlier calculation unit 13 sets pairs of corresponding characteristic line segments using the two-dimensional line segments extracted in step A2 and the three-dimensional line segments extracted in step A3 (step A7).

[0047] Next, the corresponding line segment inlier calculation unit 13 also uses the tentative camera posture estimated in step A5 to identify pairs of characteristic line segments that are inliers from among the pairs of characteristic line segments set in step A7 (step A8).

[0048] Next, the score calculation unit 14 calculates the sum or product of the number of inlier pairs identified in step A6 and the number of inlier pairs identified in step A8, and sets the calculated value as the score of the provisional camera pose estimated in step a5 (step A9).

[0049] Next, the score calculation unit 14 determines whether the number of times the score has been calculated has reached a predetermined number (step A10). If the result of the determination in step A10 is that the number of times the score has been calculated has not reached the predetermined number (step A10: No), the score calculation unit 14 instructs the temporary camera pose estimation unit 11 to execute step A4 again. In this case, the temporary camera pose estimation unit 11 selects a set of feature points that has not yet been selected, and executes step A4 again.

[0050] On the other hand, if the result of the determination in step A10 shows that the number of score calculations has reached the predetermined number (step A10: YES), the score calculation unit 14 notifies the camera pose estimation unit 15 to execute processing. As a result, the camera pose estimation unit 15 identifies a tentative camera pose when the score calculated in step A9 is maximized, and further identifies, for this tentative camera pose, the set of feature points determined to be inliers in step A6 and the set of feature line segments determined to be inliers in step A8 (step A11).

[0051] Next, the camera posture estimation unit 15 estimates the camera posture using the sets of inlier feature points, the sets of inlier feature line segments, or both identified in step A11 (step A12). Note that if only the sets of feature points are used in step A12, then only the sets of feature points may be identified in step A11. Furthermore, if only the sets of feature line segments are used in step A12, then only the sets of feature line segments may be identified in step A11.

[0052] As described above, in the first embodiment, the camera pose estimation device 10 estimates a plurality of tentative camera poses using pairs of corresponding two-dimensional points and three-dimensional points. Furthermore, for each tentative camera pose, the camera pose estimation device 10 calculates a score using a pair of feature points and a pair of feature line segments so that the score is maximized when the inlier between the two is highest, and calculates the final camera pose using the pair that maximizes the score. Therefore, the camera pose estimation device 10 can estimate the camera pose after removing pairs that are outliers, thereby improving the accuracy of the camera pose estimation.

[0053] Furthermore, the camera posture estimation device 10 can accurately estimate the camera posture from a two-dimensional image, and is therefore useful for robots, unmanned aerial vehicles (drones), self-driving vehicles, and the like, which require estimation of their own positions.

[0054] [program] The program in the first embodiment may be any program that causes a computer to execute steps A1 to A12 shown in Fig. 4. By installing and executing this program on a computer, the camera posture estimation device 10 and the camera posture estimation method in the embodiment can be realized. In this case, the processor of the computer functions as and performs processing for a provisional camera posture estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, a camera posture estimation unit 15, a corresponding point setting unit 16, a two-dimensional line segment extraction unit 17, and a three-dimensional line segment extraction unit 18. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0055] The program according to the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of a provisional camera posture estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, a camera posture estimation unit 15, a corresponding point setting unit 16, a two-dimensional line segment extraction unit 17, and a three-dimensional line segment extraction unit 18.

[0056] (Embodiment 2) Next, a camera posture estimation device, a camera posture estimation method, and a program according to the second embodiment will be described with reference to FIGS.

[0057] [Device configuration] First, the configuration of the camera posture estimation device according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing the configuration of the camera posture estimation device according to the second embodiment.

[0058] As shown in Fig. 5, camera posture estimation device 40 in embodiment 2 differs from camera posture estimation device 10 in embodiment 1 shown in Fig. 2 in that it does not include two-dimensional line segment extraction unit 17 and three-dimensional line segment extraction unit 18, but instead includes corresponding line segment setting unit 41. Furthermore, in embodiment 2, the processing in temporary camera posture estimation unit 11 is also different. The following description will focus on the differences from embodiment 1.

[0059] The corresponding line segment setting unit 41 extracts two-dimensional line segments from the two-dimensional image data 20, and extracts three-dimensional line segments from the three-dimensional data stored in the image database 30. Then, the corresponding line segment setting unit 41 uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of corresponding two-dimensional and three-dimensional line segments (pairs of characteristic line segments). The corresponding line segment setting unit 41 also inputs the set pairs of characteristic line segments to the temporary camera posture estimation unit 11.

[0060] In a modified example, the temporary camera posture estimation unit 11 selects multiple pairs from pairs of corresponding feature points and pairs of corresponding feature line segments, and estimates multiple temporary camera postures using the selected multiple pairs.

[0061] In the modified example, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, and the camera posture estimation unit 15 function in the same manner as in the first embodiment.

[0062] [Device operation] Next, the operation of camera posture estimation device 40 in embodiment 2 will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the operation of the camera posture estimation device in embodiment 2. In the following description, Fig. 5 will be referred to as appropriate. Also, in embodiment 2, a camera posture estimation method is implemented by operating camera posture estimation device 40. Therefore, the description of the camera posture estimation method in embodiment 2 will be replaced by the following description of the operation of camera posture estimation device 40.

[0063] 5, first, the corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20 sent from the camera 21, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of feature points (step B1). Step B1 is the same step as step A1 shown in FIG.

[0064] Next, the corresponding line segment setting unit 41 extracts two-dimensional line segments from the two-dimensional image data 20, and extracts three-dimensional line segments from the three-dimensional data stored in the image database 30 (step B2).

[0065] Next, the corresponding line segment setting unit 41 uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of corresponding two-dimensional line segments and three-dimensional line segments (pairs of characteristic line segments) (step B3).

[0066] Next, the temporary camera posture estimation unit 11 selects any number of pairs from the pairs of feature points set in step B1 and the pairs of feature line segments set in step B3 (step B4).The temporary camera posture estimation unit 11 then estimates multiple temporary camera postures using the multiple pairs selected in step B4 (step B5).

[0067] Next, the corresponding point inlier calculation unit 12 identifies inlier pairs from among the pairs of feature points set in step A1 using the tentative camera posture estimated in step B4 (step B6). Step B6 is the same as step A6 shown in FIG.

[0068] Next, the corresponding line segment inlier calculation unit 13 also uses the tentative camera posture estimated in step B4 to identify pairs of characteristic line segments that are inliers among the pairs of characteristic line segments set in step B3 (step B7). Step B7 is a step similar to step A8 shown in FIG.

[0069] Next, the score calculation unit 14 calculates the sum or product of the number of inlier pairs identified in step B6 and the number of inlier pairs identified in step B7, and sets the calculated value as the score of the provisional camera pose estimated in step a5 (step B8). Step B6 is a step similar to step A9 shown in FIG.

[0070] Next, the score calculation unit 14 determines whether the number of times the score has been calculated has reached a predetermined number (step B9). If the result of the determination in step B9 is that the number of times the score has been calculated has not reached the predetermined number (step B9: No), the score calculation unit 14 instructs the temporary camera pose estimation unit 11 to execute step B4 again. In this case, the temporary camera pose estimation unit 11 selects a set of feature points that has not yet been selected, and executes step B4 again.

[0071] On the other hand, if the result of the determination in step B9 shows that the number of score calculations has reached the predetermined number (step B9: YES), the score calculation unit 14 notifies the camera pose estimation unit 15 to execute processing. As a result, the camera pose estimation unit 15 identifies a tentative camera pose when the score calculated in step B8 is maximized, and further identifies, for this tentative camera pose, the set of feature points determined to be inliers in step B5 and the set of feature line segments determined to be inliers in step B6 (step B10).

[0072] Next, the camera posture estimation unit 15 estimates the camera posture using the set of inlier feature points, the set of inlier feature line segments, or both identified in step B10 (step B11). Note that if only the set of feature points is used in step B11, then only the set of feature points may be identified in step B10. Furthermore, if only the set of feature line segments is used in step B11, then only the set of feature line segments may be identified in step B10.

[0073] In this way, in the second embodiment as well, as in the first embodiment, it is possible to estimate the camera pose after removing outlier pairs, thereby improving the accuracy of estimating the camera pose.

[0074] [program] The program in the second embodiment may be any program that causes a computer to execute steps B1 to B11 shown in Fig. 6. By installing and executing this program on a computer, it is possible to realize the camera posture estimation device 40 and the camera posture estimation method according to the embodiment. In this case, the processor of the computer functions as and performs processing for a temporary camera posture estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, a camera posture estimation unit 15, a corresponding point setting unit 16, and a corresponding line segment setting unit 41. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0075] The program according to the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, the camera posture estimation unit 15, the corresponding point setting unit 16, and the corresponding line segment setting unit 41.

[0076] [Physical configuration] Here, a computer that realizes the camera posture estimation device by executing the program according to the first and second embodiments will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of a computer that realizes the camera posture estimation device according to the first and second embodiments.

[0077] 7, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0078] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0079] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0080] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0081] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0082] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0083] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0084] Note that the camera posture estimation device in the embodiments can be realized not by a computer on which a program is installed, but by hardware corresponding to each unit, for example, an electronic circuit. Furthermore, the camera posture estimation device may be partially realized by a program and the remaining unit by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 7.

[0085] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0086] (Appendix 1) a temporary camera pose estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation unit that identifies pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation unit that identifies pairs of corresponding two-dimensional and three-dimensional line segments that are inliers, using one of the plurality of temporary camera postures; a score calculation unit that calculates a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation unit that estimates a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; A camera posture estimation device comprising:

[0087] (Appendix 2) a corresponding point setting unit that extracts two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracts three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combines the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other, 2. The camera pose estimation device according to claim 1.

[0088] (Appendix 3) a two-dimensional line segment extraction unit that extracts the two-dimensional line segments from the two-dimensional image data; a three-dimensional line segment extraction unit that extracts the three-dimensional line segments from the three-dimensional data; Further provided with the corresponding line segment inlier calculation unit sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; 2. The camera pose estimation device according to claim 1.

[0089] (Appendix 4) a corresponding line segment setting unit that extracts the two-dimensional line segments from the two-dimensional image data, extracts the three-dimensional line segments from the three-dimensional data, and sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; the temporary camera posture estimation unit selects a plurality of pairs from among the pairs of corresponding two-dimensional points and three-dimensional points and the pairs of corresponding two-dimensional line segments and three-dimensional line segments, and estimates a plurality of temporary camera postures using the selected plurality of pairs. 2. The camera pose estimation device according to claim 1.

[0090] (Appendix 5) the score calculation unit calculates, for each of the estimated virtual camera postures, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score. 2. The camera pose estimation device according to claim 1.

[0091] (Appendix 6) a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying pairs of corresponding two-dimensional line segments and three-dimensional line segments that are inliers, using one of the plurality of temporary camera poses; a score calculation step of calculating a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation step of estimating a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; A camera posture estimation method comprising:

[0092] (Appendix 7) The method further comprises a corresponding point setting step of extracting two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracting three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other. 7. The camera pose estimation method according to claim 6.

[0093] (Appendix 8) a two-dimensional line segment extraction step of extracting the two-dimensional line segments from the two-dimensional image data; a three-dimensional line segment extraction step of extracting the three-dimensional line segments from the three-dimensional data; and in the corresponding line segment inlier calculation step, the extracted two-dimensional line segment and the extracted three-dimensional line segment are used to set the corresponding two-dimensional line segment and three-dimensional line segment pair; 7. The camera pose estimation method according to claim 6.

[0094] (Appendix 9) a corresponding line segment setting step of extracting the two-dimensional line segments from the two-dimensional image data, extracting the three-dimensional line segments from the three-dimensional data, and then setting pairs of the two-dimensional line segments and the three-dimensional line segments that correspond to each other using the extracted two-dimensional line segments and the extracted three-dimensional line segments; In the temporary camera posture estimation step, a plurality of pairs are selected from the pairs of the corresponding two-dimensional points and three-dimensional points and the pairs of the corresponding two-dimensional line segments and three-dimensional line segments, and a plurality of temporary camera postures are estimated using the selected plurality of pairs. 7. The camera pose estimation method according to claim 6.

[0095] (Appendix 10) In the score calculation step, for each of the estimated virtual camera postures, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of pairs of inliers between two-dimensional line segments and three-dimensional line segments is calculated, and the calculated value is set as the score. 7. The camera pose estimation method according to claim 6.

[0096] (Appendix 11) On the computer, a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying pairs of corresponding two-dimensional points and three-dimensional points that are inliers using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying pairs of corresponding two-dimensional line segments and three-dimensional line segments that are inliers, using one of the plurality of temporary camera poses; a score calculation step of calculating a score for each of the estimated tentative camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the tentative camera posture; a camera pose estimation step of estimating a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; Run Ru, Program Hmm.

[0097] (Appendix 12) before The computer further executing a corresponding point setting step of extracting two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracting three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other; As described in Appendix 11 program .

[0098] (Appendix 13) before The computer a two-dimensional line segment extraction step of extracting the two-dimensional line segments from the two-dimensional image data; a three-dimensional line segment extraction step of extracting the three-dimensional line segments from the three-dimensional data; Then run in the corresponding line segment inlier calculation step, the extracted two-dimensional line segment and the extracted three-dimensional line segment are used to set the corresponding two-dimensional line segment and three-dimensional line segment pair; As described in Appendix 11 program .

[0099] (Appendix 14) before The computer further executing a corresponding line segment setting step of extracting the two-dimensional line segments from the two-dimensional image data, extracting the three-dimensional line segments from the three-dimensional data, and setting pairs of the two-dimensional line segments and the three-dimensional line segments that correspond to each other using the extracted two-dimensional line segments and the extracted three-dimensional line segments; In the temporary camera posture estimation step, a plurality of pairs are selected from the pairs of the corresponding two-dimensional points and three-dimensional points and the pairs of the corresponding two-dimensional line segments and three-dimensional line segments, and a plurality of temporary camera postures are estimated using the selected plurality of pairs. As described in Appendix 11 program .

[0100] (Appendix 15) In the score calculation step, for each of the estimated virtual camera postures, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of pairs of inliers between two-dimensional line segments and three-dimensional line segments is calculated, and the calculated value is set as the score. As described in Appendix 11 program .

[0101] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Industrial Applicability]

[0102] As described above, the present disclosure can improve the accuracy of estimating the camera posture of a camera that captures a two-dimensional image. The present disclosure is useful for systems that require estimating the camera posture from an image, such as autonomous robots, autonomous unmanned aerial vehicles, and autonomous driving systems. [Explanation of symbols]

[0103] 10. Camera Pose Estimation Device (First Embodiment) 11. Virtual camera pose estimation unit 12 Corresponding point inlier calculation unit 13 Corresponding line segment inlier calculation unit 14 Score calculation section 15 Camera pose estimation unit 16 Corresponding point setting section 17 2D line segment extraction section 18 3D line segment extraction section 20 2D image data 21 Camera 30 Image Database 40 Camera Pose Estimation Device (Embodiment 2) 41 Corresponding line setting section 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a temporary camera pose estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation unit that identifies pairs of corresponding two-dimensional points and three-dimensional points that are inliers, using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation unit that identifies pairs of corresponding two-dimensional and three-dimensional line segments that are inliers, using one of the plurality of temporary camera postures; a score calculation unit that calculates a score for each of the estimated temporary camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the temporary camera posture; a camera pose estimation unit that estimates a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; A camera posture estimation device comprising:

2. a corresponding point setting unit that extracts two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracts three-dimensional feature points as the three-dimensional points from the three-dimensional data, and sets pairs of the corresponding two-dimensional points and three-dimensional points by combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other; The camera posture estimation device according to claim 1 .

3. a two-dimensional line segment extraction unit that extracts the two-dimensional line segments from the two-dimensional image data; a three-dimensional line segment extraction unit that extracts the three-dimensional line segments from the three-dimensional data; Further provided with the corresponding line segment inlier calculation unit sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; The camera posture estimation device according to claim 1 .

4. a corresponding line segment setting unit that extracts the two-dimensional line segments from the two-dimensional image data, extracts the three-dimensional line segments from the three-dimensional data, and sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; the temporary camera posture estimation unit selects a plurality of pairs from among the pairs of corresponding two-dimensional points and three-dimensional points and the pairs of corresponding two-dimensional line segments and three-dimensional line segments, and estimates a plurality of temporary camera postures using the selected plurality of pairs. The camera posture estimation device according to claim 1 .

5. the score calculation unit calculates, for each of the estimated virtual camera postures, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score. The camera posture estimation device according to claim 1 .

6. selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of tentative camera poses using the selected plurality of pairs; identifying inlier pairs of corresponding two-dimensional and three-dimensional points using one of the plurality of temporary camera poses; using one of the plurality of temporary camera poses, identifying pairs of corresponding two-dimensional and three-dimensional line segments that are inliers; For each estimated temporary camera pose, a score is calculated from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the temporary camera pose; a camera pose estimation method, characterized in that the camera pose is estimated using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

7. On the computer, selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of tentative camera poses using the selected plurality of pairs; using one of the plurality of virtual camera poses to identify inlier pairs of corresponding two-dimensional and three-dimensional points; using one of the plurality of virtual camera poses, identifying pairs of corresponding two-dimensional and three-dimensional line segments that are inliers; For each estimated temporary camera posture, a score is calculated from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, which are identified using the temporary camera posture; a camera pose is estimated using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; program.

Images