Camera posture estimation device, camera posture estimation method, and program

Abstract

A camera orientation estimating device 10 comprises a provisional camera orientation estimating unit 11 which selects a plurality of sets from among sets of corresponding two-dimensional points and three-dimensional points, and uses the selected sets to estimate a plurality of provisional camera orientations; a corresponding point inlier calculating unit 12 that uses the provisional camera orientations to identify inliers among the sets of two-dimensional points and three-dimensional points; a corresponding line segment inlier calculating unit 13 that uses one of the provisional camera orientations to identify inliers among sets of two-dimensional line segments and three-dimensional line segments; a score calculating unit 14 that calculates a score from the sets of two-dimensional points and three-dimensional points that form inliers and the sets of two-dimensional line segments and three-dimensional line segments that form inliers, for each provisional camera orientation; and a camera orientation estimating unit 14 that estimates a camera orientation using at least one of the sets of two-dimensional points and three-dimensional points that form inliers and the sets of two-dimensional line segments and three-dimensional line segments that form inliers, when the score is maximum.

Description

Camera posture estimation device, camera posture estimation method, and computer-readable recording medium

[0001] The present disclosure relates to a camera posture estimation device and a camera posture estimation method for estimating the posture of a camera that captured an image, and further relates to a computer-readable recording medium on which a program for realizing these is recorded.

[0002] Patent Document 1 discloses an apparatus for detecting defects such as minute cracks on the surface of a structure from an image of the structure taken with a camera and identifying the location of the detected defects. The apparatus disclosed in Patent Document 1 first performs image analysis on the image taken with the camera to detect defects on the surface of the structure, and then measures the distance to the defects using a rangefinder. Next, the apparatus disclosed in Patent Document 1 calculates the coordinates of the detected defects in a global coordinate system using coordinate data associated with the image and the measured distance.

[0003] The device disclosed in the patent document can identify the actual position of a defect in a structure detected on an image. However, the device disclosed in the patent document requires a rangefinder in addition to a camera to calculate the position, which makes it difficult to reduce the cost of the device.

[0004] In response to this, a technique has been proposed in which the 2D coordinates of feature points of an object in a 2D image are paired with the 3D coordinates of feature points in a 3D model of the object to estimate the camera pose (camera parameters) of the camera that captured the 2D image. This technique makes it possible to calculate the 3D coordinates corresponding to a specified point on the 2D image from the estimated camera pose without using a rangefinder, thereby reducing equipment costs.

[0005] International Publication No. 2017 / 103982

[0006] However, the technique disclosed in the above-mentioned patent document has a problem in that it is not possible to accurately estimate the camera pose when an object with many feature points is located far from the camera in the two-dimensional image, because in such a case, it is difficult to accurately pair the two-dimensional coordinates of the feature points with the three-dimensional coordinates of the feature points.

[0007] An example of an objective of the present disclosure is to improve the accuracy of estimating the camera posture of a camera that captures a two-dimensional image.

[0008] In order to achieve the above object, a camera pose estimation device according to one aspect of the present disclosure includes: a temporary camera pose estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation unit that uses one of the plurality of temporary camera poses to identify an inlier pair of corresponding two-dimensional points and three-dimensional points; a corresponding line segment inlier calculation unit that uses one of the plurality of temporary camera poses to identify an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments; and a score calculation unit that calculates a score for each estimated temporary camera pose from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera pose. a camera pose estimation unit that estimates the camera pose using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment when the score is maximized.

[0009] Furthermore, in order to achieve the above object, a camera pose estimation method according to one aspect of the present disclosure includes: a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying an inlier pair of corresponding two-dimensional points and three-dimensional points using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments using one of the plurality of temporary camera poses; and a score calculation step of calculating a score for each estimated temporary camera pose from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera pose. a camera pose estimation step of estimating the camera pose using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

[0010] Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure includes a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying an inlier pair of corresponding two-dimensional points and three-dimensional points using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments using one of the plurality of temporary camera poses; and a score calculation step of calculating a score for each estimated temporary camera pose from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera pose. a camera pose estimation step of estimating the camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point, and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, when the score is maximized, and

[0011] As described above, according to the present disclosure, it is possible to improve the accuracy of estimating the camera posture of a camera that captured a two-dimensional image.

[0012] FIG. 1 is a configuration diagram showing a schematic configuration of a camera posture estimation device according to Embodiment 1. FIG. 2 is a configuration diagram specifically showing the configuration of the camera posture estimation device according to Embodiment 1. FIG. 3 is a diagram showing an example of two-dimensional image data and three-dimensional data used in Embodiment 1. FIG. 4 is a flow diagram showing the operation of the camera posture estimation device according to Embodiment 1. FIG. 5 is a configuration diagram showing the configuration of a camera posture estimation device according to Embodiment 2. FIG. 6 is a flow diagram showing the operation of the camera posture estimation device according to Embodiment 2. FIG. 7 is a block diagram showing an example of a computer that realizes the camera posture estimation devices according to Embodiments 1 and 2.

[0013] First Embodiment A camera posture estimation device, a camera posture estimation method, and a program according to a first embodiment will be described below with reference to FIGS.

[0014] [Device Configuration] First, a schematic configuration of the camera posture estimation device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of the camera posture estimation device according to the first embodiment.

[0015] 1, a camera posture estimation device 10 according to the first embodiment is a device for estimating the posture of a camera that captured an image. As shown in FIG. 1, the camera posture estimation device 10 includes a provisional camera posture estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, and a camera posture estimation unit 15.

[0016] The provisional camera posture estimation unit 11 selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of provisional camera postures using the selected plurality of pairs.

[0017] The corresponding point inlier calculation unit 12 uses one of a plurality of tentative camera poses to identify pairs of corresponding two-dimensional and three-dimensional points that are inliers.

[0018] The corresponding line segment inlier calculation unit 13 uses one of a plurality of provisional camera postures to identify pairs of corresponding two-dimensional and three-dimensional line segments that are inliers.

[0019] For each estimated virtual camera posture, the score calculation unit 14 calculates a score from pairs of inliers between two-dimensional points and three-dimensional points and pairs of inliers between two-dimensional line segments and three-dimensional line segments, which are identified using the virtual camera posture.

[0020] The camera posture estimation unit 15 estimates the camera posture using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

[0021] In this way, the camera pose estimation device 10 estimates multiple tentative camera poses and calculates a score representing the error between the point and the line segment for each tentative camera pose. The camera pose estimation device 10 then uses the score to identify an appropriate pair and calculate the final camera pose. In other words, the camera pose estimation device 10 can estimate the camera pose after performing accurate pairing, thereby improving the accuracy of camera pose estimation.

[0022] Next, the configuration and functions of the camera posture estimation device in Embodiment 1 will be specifically described with reference to Figures 2 and 3. Figure 2 is a configuration diagram specifically showing the configuration of the camera posture estimation device in Embodiment 1. Figure 3 is a diagram showing an example of two-dimensional image data and three-dimensional data used in Embodiment 1.

[0023] As shown in FIG. 2 , in the first embodiment, the camera posture estimation device 10 includes a corresponding point setting unit 16, a two-dimensional line segment extraction unit 17, and a three-dimensional line segment extraction unit 18 in addition to the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, and the camera posture estimation unit 15 described above.

[0024] 2, image data of a two-dimensional image captured by a camera 21 (hereinafter referred to as "two-dimensional image data") is input to the camera posture estimation device 10. This two-dimensional image is an image for which the camera posture is to be estimated, and will also be referred to as a "query image" hereinafter.

[0025] Furthermore, in the first embodiment, the camera 21 may be a camera held by a user or may be a camera mounted on a moving object such as a robot or an unmanned aerial vehicle (drone). The two-dimensional image data output from the camera 21 may be temporarily stored in a storage device and then transmitted to the camera posture estimation device 10, or may be transmitted directly from the camera to the camera posture estimation device 10.

[0026] 2, the camera posture estimation device 10 is connected to an image database 30 that stores three-dimensional data so as to enable data communication. In the first embodiment, the three-dimensional data includes image data with depth captured by an RGB-D (depth) camera, three-dimensional point cloud data of an object, and the like.

[0027] The corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Specifically, the corresponding point setting unit 16 identifies feature points by calculating, for example, SIFT feature values ​​or SURF feature values ​​on each image data. Note that if the three-dimensional data is three-dimensional point cloud data, the feature points have already been extracted. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of corresponding two-dimensional points and three-dimensional points (hereinafter referred to as "feature point pairs").

[0028] The two-dimensional line segment extraction unit 17 extracts two-dimensional line segments from the two-dimensional image data 20 sent from the camera 21. Existing technology is used as a method for extracting two-dimensional line segments. The three-dimensional line segment extraction unit 18 extracts three-dimensional line segments from the three-dimensional data stored in the image database 30. Existing technology is also used as a method for extracting two-dimensional line segments. Note that if the three-dimensional data is the above-mentioned depth-added image data, extracting two-dimensional line segments will result in extracting three-dimensional line segments.

[0029] When a large number of feature point pairs are set by the corresponding point setting unit 16, the temporary camera posture estimation unit 11 selects an arbitrary number of pairs, for example, three pairs, from among them. Then, the temporary camera posture estimation unit 11 uses the selected feature point pairs to estimate a temporary camera posture, i.e., the extrinsic parameters of the camera 21. The extrinsic parameters are matrices for transforming the positions of the feature points from the world coordinate system to the camera coordinate system, and are composed of a rotation matrix R and a translation vector t. An existing estimation method is used as the method for estimating the extrinsic parameters.

[0030] In the first embodiment, each time a tentative camera pose is estimated by the tentative camera pose estimation unit 11, the corresponding point inlier calculation unit 12 uses the estimated tentative camera pose to identify pairs of corresponding two-dimensional points and three-dimensional points that are inliers. Pairs that are not identified are outliers (outliers, errors).

[0031] Specifically, for each of all the set pairs of feature points, the corresponding point inlier calculation unit 12 projects the three-dimensional points onto a two-dimensional coordinate system (camera coordinate system) using the estimated virtual camera parameters, and calculates the distance between the projected three-dimensional points and the two-dimensional points. If the calculated distance is equal to or smaller than a preset threshold, the corresponding point inlier calculation unit 12 determines that the corresponding pair of feature points is an inlier.

[0032] First, each time a provisional camera posture is estimated by the provisional camera posture estimation unit 11, the corresponding line segment inlier calculation unit 13 uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of corresponding two-dimensional and three-dimensional line segments (hereinafter referred to as "pairs of characteristic line segments").

[0033] Then, each time a tentative camera posture is estimated, the corresponding line segment inlier calculation unit 13 also uses the estimated tentative camera posture to identify pairs of corresponding two-dimensional and three-dimensional line segments that are inliers. Pairs that are not identified are outliers (outliers, errors).

[0034] Specifically, for each of all the pairs of characteristic line segments that have been set, the corresponding line segment inlier calculation unit 13 projects the three-dimensional line segments onto a two-dimensional coordinate system (camera coordinate system) using the estimated virtual camera parameters, and calculates the angle θ between the projected three-dimensional line segment and the two-dimensional line segment. If the calculated angle θ is equal to or smaller than a predetermined threshold, the corresponding line segment inlier calculation unit 13 determines that the corresponding pair of characteristic line segments is an inlier.

[0035] The score calculation unit 14 calculates, for each provisional camera posture, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of pairs of inliers between two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score of the provisional camera posture. Note that in the first embodiment, a method other than the method of calculating the sum or product described above may be used as the score calculation method.

[0036] However, if the set of feature points selected by the temporary camera pose estimation unit 11 is incorrect, the temporary camera pose estimated at this time will also be incorrect. Therefore, the temporary camera pose estimation unit 11 selects a set of feature points multiple times while changing the set of feature points to be selected, and estimates a temporary camera pose each time. Then, once the temporary camera pose is estimated, processing is performed by the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, and the score calculation unit 14 each time estimation is performed.

[0037] In this way, the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, and the score calculation unit 14 form a RANSAC (Random Sample Consensus) loop (the part surrounded by a dashed line in Figure 2), and each process is executed repeatedly.

[0038] The camera posture estimation unit 15 identifies a tentative camera posture that maximizes the score calculated by the score calculation unit 14. The camera posture estimation unit 15 also identifies, for the identified tentative camera posture, a set of feature points that have been determined to be inliers by the corresponding point inlier calculation unit 12 and a set of feature line segments that have been determined to be inliers by the corresponding line segment inlier calculation unit 13.

[0039] The camera posture estimation unit 15 then estimates the camera posture using the identified set of inlier feature points, the identified set of inlier feature line segments, or both of them. In this case, an existing estimation method is also used as the camera posture estimation method.

[0040] [Device Operation] Next, the operation of camera pose estimation device 10 in embodiment 1 will be described with reference to FIG. 4. FIG. 4 is a flow diagram showing the operation of the camera pose estimation device in embodiment 1. In the following description, reference will be made to FIGS. 1 to 3 as appropriate. In embodiment 1, a camera pose estimation method is implemented by operating camera pose estimation device 10. Therefore, the description of the camera pose estimation method in embodiment 1 will be replaced by the following description of the operation of camera pose estimation device 10.

[0041] 4, first, the corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20 sent from the camera 21, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of feature points (step A1).

[0042] Next, the two-dimensional line segment extracting unit 17 extracts two-dimensional line segments from the two-dimensional image data 20 sent from the camera 21 (step A2).

[0043] Next, the three-dimensional line segment extraction unit 18 extracts three-dimensional line segments from the three-dimensional data stored in the image database 30 (step A3). Note that the order of steps A1 to A3 is not particularly limited, and these steps may be executed simultaneously.

[0044] Next, the temporary camera posture estimation unit 11 selects any number of sets of feature points from the many sets of feature points set in step A1 (step A4), and estimates a temporary camera posture using the sets of feature points selected in step A4 (step A5).

[0045] Next, the corresponding point inlier calculation unit 12 uses the tentative camera posture estimated in step A5 to identify inlier pairs of feature points from among the pairs set in step A1 (step A6).

[0046] Next, the corresponding line segment inlier calculation unit 13 sets pairs of corresponding characteristic line segments using the two-dimensional line segments extracted in step A2 and the three-dimensional line segments extracted in step A3 (step A7).

[0047] Next, the corresponding line segment inlier calculation unit 13 also uses the tentative camera posture estimated in step A5 to identify pairs of characteristic line segments that are inliers from among the pairs of characteristic line segments set in step A7 (step A8).

[0048] Next, the score calculation unit 14 calculates the sum or product of the number of inlier pairs identified in step A6 and the number of inlier pairs identified in step A8, and sets the calculated value as the score of the provisional camera posture estimated in step a5 (step A9).

[0049] Next, the score calculation unit 14 determines whether the number of score calculations has reached a predetermined number (step A10). If the result of the determination in step A10 is that the number of score calculations has not reached the predetermined number (step A10: No), the score calculation unit 14 instructs the temporary camera pose estimation unit 11 to execute step A4 again. In this case, the temporary camera pose estimation unit 11 selects a set of feature points that has not yet been selected, and executes step A4 again.

[0050] On the other hand, if the result of the determination in step A10 is that the number of score calculations has reached the predetermined number (step A10: YES), the score calculation unit 14 notifies the camera pose estimation unit 15 to execute processing. As a result, the camera pose estimation unit 15 identifies a tentative camera pose when the score calculated in step A9 is maximized, and further identifies, for this tentative camera pose, the set of feature points determined to be inliers in step A6 and the set of feature line segments determined to be inliers in step A8 (step A11).

[0051] Next, the camera posture estimation unit 15 estimates the camera posture using the set of inlier feature points, the set of inlier feature line segments, or both identified in step A11 (step A12). Note that if only the set of feature points is used in step A12, only the set of feature points may be identified in step A11. Furthermore, if only the set of feature line segments is used in step A12, only the set of feature line segments may be identified in step A11.

[0052] As described above, in the first embodiment, the camera pose estimation device 10 estimates multiple tentative camera poses using pairs of corresponding two-dimensional points and three-dimensional points. Furthermore, for each tentative camera pose, the camera pose estimation device 10 calculates a score using a pair of feature points and a pair of feature line segments so that the score is maximized when the inliers between the two are highest, and calculates the final camera pose using the pair that maximizes the score. Therefore, the camera pose estimation device 10 can estimate the camera pose after removing outlier pairs, thereby improving the accuracy of camera pose estimation.

[0053] Furthermore, the camera posture estimation device 10 can accurately estimate the camera posture from a two-dimensional image, and is therefore useful for robots, unmanned aerial vehicles (drones), self-driving vehicles, and the like, which require estimation of their own position.

[0054] [Program] The program in the first embodiment may be a program that causes a computer to execute steps A1 to A12 shown in FIG. By installing and executing this program on a computer, the camera posture estimation device 10 and the camera posture estimation method according to the embodiment can be realized. In this case, the processor of the computer functions as and performs processing for a provisional camera posture estimation unit 11, a corresponding point inlier calculation unit 12, a corresponding line segment inlier calculation unit 13, a score calculation unit 14, a camera posture estimation unit 15, a corresponding point setting unit 16, a two-dimensional line segment extraction unit 17, and a three-dimensional line segment extraction unit 18. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0055] The program according to the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, the camera posture estimation unit 15, the corresponding point setting unit 16, the two-dimensional line segment extraction unit 17, and the three-dimensional line segment extraction unit 18.

[0056] Second Embodiment Next, a camera posture estimation device, a camera posture estimation method, and a program according to a second embodiment will be described with reference to FIGS. 5 and 6. FIG.

[0057] [Device Configuration] First, the configuration of the camera posture estimation device in the second embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing the configuration of the camera posture estimation device in the second embodiment.

[0058] As shown in Fig. 5, camera posture estimation device 40 in embodiment 2 differs from camera posture estimation device 10 in embodiment 1 shown in Fig. 2 in that it does not include two-dimensional line segment extraction unit 17 and three-dimensional line segment extraction unit 18, but instead includes a corresponding line segment setting unit 41. Furthermore, in embodiment 2, the processing in temporary camera posture estimation unit 11 is also different. The following description will focus on the differences from embodiment 1.

[0059] The corresponding line segment setting unit 41 extracts two-dimensional line segments from the two-dimensional image data 20 and extracts three-dimensional line segments from the three-dimensional data stored in the image database 30. Then, the corresponding line segment setting unit 41 uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of corresponding two-dimensional and three-dimensional line segments (pairs of characteristic line segments). The corresponding line segment setting unit 41 also inputs the set pairs of characteristic line segments to the virtual camera posture estimation unit 11.

[0060] In a modified example, the provisional camera posture estimation unit 11 selects multiple pairs from pairs of corresponding feature points and pairs of corresponding feature line segments, and estimates multiple provisional camera postures using the selected multiple pairs.

[0061] In the modified example, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, and the camera posture estimation unit 15 function in the same manner as in the first embodiment.

[0062] [Device Operation] Next, the operation of camera posture estimation device 40 in embodiment 2 will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the operation of the camera posture estimation device in embodiment 2. In the following description, Fig. 5 will be referenced as appropriate. Also, in embodiment 2, a camera posture estimation method is implemented by operating camera posture estimation device 40. Therefore, the description of the camera posture estimation method in embodiment 2 will be replaced by the following description of the operation of camera posture estimation device 40.

[0063] 5, first, the corresponding point setting unit 16 extracts two-dimensional feature points as two-dimensional points from the two-dimensional image data 20 sent from the camera 21, and extracts three-dimensional feature points as three-dimensional points from the three-dimensional data stored in the image database 30. Then, the corresponding point setting unit 16 combines corresponding two-dimensional feature points and three-dimensional feature points to set a large number of pairs of feature points (step B1). Step B1 is the same as step A1 shown in FIG.

[0064] Next, the corresponding line segment setting unit 41 extracts two-dimensional line segments from the two-dimensional image data 20, and extracts three-dimensional line segments from the three-dimensional data stored in the image database 30 (step B2).

[0065] Next, the corresponding line segment setting unit 41 uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of corresponding two-dimensional line segments and three-dimensional line segments (pairs of characteristic line segments) (step B3).

[0066] Next, the temporary camera posture estimation unit 11 selects any number of pairs from the pairs of feature points set in step B1 and the pairs of feature line segments set in step B3 (step B4), and estimates multiple temporary camera postures using the multiple pairs selected in step B4 (step B5).

[0067] Next, the corresponding point inlier calculation unit 12 identifies inlier pairs of feature points from among the pairs of feature points set in step A1 using the tentative camera pose estimated in step B4 (step B6). Step B6 is the same as step A6 shown in FIG. 4.

[0068] Next, the corresponding line segment inlier calculation unit 13 also uses the virtual camera posture estimated in step B4 to identify pairs of characteristic line segments that are inliers among the pairs of characteristic line segments set in step B3 (step B7). Step B7 is a step similar to step A8 shown in FIG. 4.

[0069] Next, the score calculation unit 14 calculates the sum or product of the number of inlier pairs identified in step B6 and the number of inlier pairs identified in step B7, and sets the calculated value as the score of the provisional camera pose estimated in step a5 (step B8). Step B6 is a step similar to step A9 shown in FIG. 4.

[0070] Next, the score calculation unit 14 determines whether the number of score calculations has reached a predetermined number (step B9). If the result of the determination in step B9 is that the number of score calculations has not reached the predetermined number (step B9: No), the score calculation unit 14 instructs the temporary camera posture estimation unit 11 to execute step B4 again. In this case, the temporary camera posture estimation unit 11 selects a set of feature points that has not yet been selected, and executes step B4 again.

[0071] On the other hand, if the result of the determination in step B9 shows that the number of score calculations has reached the predetermined number (step B9: YES), the score calculation unit 14 notifies the camera pose estimation unit 15 to execute processing. As a result, the camera pose estimation unit 15 identifies a tentative camera pose when the score calculated in step B8 is maximized, and further identifies, for this tentative camera pose, the set of feature points determined to be inliers in step B5 and the set of feature line segments determined to be inliers in step B6 (step B10).

[0072] Next, the camera posture estimation unit 15 estimates the camera posture using the set of inlier feature points, the set of inlier feature line segments, or both identified in step B10 (step B11). Note that if only the set of feature points is used in step B11, then only the set of feature points may be identified in step B10. Furthermore, if only the set of feature line segments is used in step B11, then only the set of feature line segments may be identified in step B10.

[0073] In this way, in the second embodiment as well, as in the first embodiment, it is possible to estimate the camera pose after removing outlier pairs, thereby improving the accuracy of the camera pose estimation.

[0074] [Program] The program in the second embodiment may be any program that causes a computer to execute steps B1 to B11 shown in FIG. 6. By installing and executing this program on a computer, the camera posture estimation device 40 and the camera posture estimation method according to the embodiment can be realized. In this case, the processor of the computer functions as and performs processing for the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, the camera posture estimation unit 15, the corresponding point setting unit 16, and the corresponding line segment setting unit 41. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0075] The program in embodiment 2 may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the provisional camera posture estimation unit 11, the corresponding point inlier calculation unit 12, the corresponding line segment inlier calculation unit 13, the score calculation unit 14, the camera posture estimation unit 15, the corresponding point setting unit 16, and the corresponding line segment setting unit 41.

[0076] [Physical Configuration] Here, a computer that realizes the camera posture estimation device by executing the program according to the first and second embodiments will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of a computer that realizes the camera posture estimation device according to the first and second embodiments.

[0077] 7, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0078] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0079] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0080] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0081] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0082] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0083] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0084] Note that the camera posture estimation device in the embodiments can be realized not by a computer on which a program is installed, but by hardware corresponding to each unit, such as an electronic circuit. Furthermore, the camera posture estimation device may be partially realized by a program and the remaining unit by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 7.

[0085] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0086] (Supplementary Note 1) A temporary camera posture estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera postures using the selected plurality of pairs; a corresponding point inlier calculation unit that uses one of the plurality of temporary camera postures to identify an inlier pair of corresponding two-dimensional points and three-dimensional points; a corresponding line segment inlier calculation unit that uses one of the plurality of temporary camera postures to identify an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments; and a score calculation unit that calculates a score for each estimated temporary camera posture from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera posture. a camera pose estimation unit that estimates the camera pose using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

[0087] (Supplementary Note 2) The camera posture estimation device according to Supplementary Note 1, further comprising a corresponding point setting unit that extracts two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracts three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combines the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other.

[0088] (Supplementary Note 3) The camera posture estimation device according to Supplementary Note 1, further comprising: a two-dimensional line segment extraction unit that extracts the two-dimensional line segments from the two-dimensional image data; and a three-dimensional line segment extraction unit that extracts the three-dimensional line segments from the three-dimensional data, wherein the corresponding line segment inlier calculation unit uses the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of the corresponding two-dimensional line segments and three-dimensional line segments.

[0089] (Supplementary Note 4) The camera posture estimation device according to Supplementary Note 1, further comprising a corresponding line segment setting unit that extracts the two-dimensional line segments from the two-dimensional image data, extracts the three-dimensional line segments from the three-dimensional data, and sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; wherein the provisional camera posture estimation unit selects a plurality of pairs from the pairs of the corresponding two-dimensional points and three-dimensional points and the pairs of the corresponding two-dimensional line segments and three-dimensional line segments, and estimates a plurality of provisional camera postures using the selected plurality of pairs.

[0090] (Supplementary Note 5) The camera pose estimation device according to Supplementary Note 1, wherein the score calculation unit calculates, for each of the estimated tentative camera poses, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score.

[0091] (Supplementary Note 6) A temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying an inlier pair of corresponding two-dimensional points and three-dimensional points using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments using one of the plurality of temporary camera poses; and a score calculation step of calculating a score for each estimated temporary camera pose from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera pose. a camera pose estimation step of estimating the camera pose using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

[0092] (Supplementary Note 7) The camera posture estimation method according to Supplementary Note 6, further comprising a corresponding point setting step of extracting two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracting three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other.

[0093] (Supplementary Note 8) The camera posture estimation method according to Supplementary Note 6, further comprising: a two-dimensional line segment extraction step of extracting the two-dimensional line segments from two-dimensional image data; and a three-dimensional line segment extraction step of extracting the three-dimensional line segments from three-dimensional data, wherein in the corresponding line segment inlier calculation step, pairs of the corresponding two-dimensional line segments and three-dimensional line segments are set using the extracted two-dimensional line segments and the extracted three-dimensional line segments.

[0094] (Supplementary Note 9) The camera pose estimation method according to Supplementary Note 6, further comprising a corresponding line segment setting step of extracting the two-dimensional line segments from the two-dimensional image data, extracting the three-dimensional line segments from the three-dimensional data, and then using the extracted two-dimensional line segments and the extracted three-dimensional line segments to set pairs of the corresponding two-dimensional line segments and three-dimensional line segments, wherein in the provisional camera pose estimation step, a plurality of pairs are selected from the pairs of the corresponding two-dimensional points and three-dimensional points and the pairs of the corresponding two-dimensional line segments and three-dimensional line segments, and a plurality of provisional camera poses are estimated using the selected plurality of pairs.

[0095] (Supplementary Note 10) The camera pose estimation method according to Supplementary Note 6, wherein in the score calculation step, for each estimated tentative camera pose, a sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments is calculated, and the calculated value is used as the score.

[0096] (Supplementary Note 11) A computer is provided with: a temporary camera pose estimation step of selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation step of identifying an inlier pair of corresponding two-dimensional points and three-dimensional points using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation step of identifying an inlier pair of corresponding two-dimensional line segments and three-dimensional line segments using one of the plurality of temporary camera poses; and a score calculation step of calculating a score for each estimated temporary camera pose from the inlier pair of two-dimensional points and three-dimensional points and the inlier pair of two-dimensional line segments and three-dimensional line segments identified using the temporary camera pose. a camera pose estimation step of estimating the camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point, and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, when the score is maximized, and a computer-readable recording medium having a program recorded thereon, the computer-readable recording medium including instructions for causing the computer to execute the steps.

[0097] (Supplementary Note 12) The computer-readable recording medium according to Supplementary Note 11, wherein the program further causes the computer to execute a corresponding point setting step of extracting two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracting three-dimensional feature points as the three-dimensional points from the three-dimensional data, and combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other to set pairs of the two-dimensional points and the three-dimensional points that correspond to each other.

[0098] (Supplementary Note 13) The computer-readable recording medium according to Supplementary Note 11, wherein the program further causes the computer to execute a two-dimensional line segment extraction step of extracting the two-dimensional line segments from two-dimensional image data, and a three-dimensional line segment extraction step of extracting the three-dimensional line segments from three-dimensional data, and wherein in the corresponding line segment inlier calculation step, pairs of the corresponding two-dimensional line segments and three-dimensional line segments are set using the extracted two-dimensional line segments and the extracted three-dimensional line segments.

[0099] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 11, wherein the program further causes the computer to execute a corresponding line segment setting step of extracting the two-dimensional line segments from the two-dimensional image data, extracting the three-dimensional line segments from the three-dimensional data, and setting pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; and in the provisional camera posture estimation step, selecting a plurality of pairs from the pairs of the corresponding two-dimensional points and three-dimensional points and the pairs of the corresponding two-dimensional line segments and three-dimensional line segments, and estimating a plurality of provisional camera postures using the selected plurality of pairs.

[0100] (Supplementary Note 15) The computer-readable recording medium according to Supplementary Note 11, wherein in the score calculation step, for each of the estimated virtual camera poses, a sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments is calculated, and the calculated value is used as the score.

[0101] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0102] As described above, the present disclosure can improve the accuracy of estimating the camera posture of a camera that captures a two-dimensional image. The present disclosure is useful for systems that require estimating the camera posture from an image, such as autonomous robots, autonomous unmanned aerial vehicles, and autonomous driving systems.

[0103] REFERENCE SIGNS LIST 10 Camera pose estimation device (first embodiment) 11 Virtual camera pose estimation unit 12 Corresponding point inlier calculation unit 13 Corresponding line segment inlier calculation unit 14 Score calculation unit 15 Camera pose estimation unit 16 Corresponding point setting unit 17 Two-dimensional line segment extraction unit 18 Three-dimensional line segment extraction unit 20 Two-dimensional image data 21 Camera 30 Image database 40 Camera pose estimation device (second embodiment) 41 Corresponding line segment setting unit 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. a temporary camera pose estimation unit that selects a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimates a plurality of temporary camera poses using the selected plurality of pairs; a corresponding point inlier calculation unit that identifies pairs of corresponding two-dimensional points and three-dimensional points that are inliers, using one of the plurality of temporary camera poses; a corresponding line segment inlier calculation unit that identifies pairs of corresponding two-dimensional and three-dimensional line segments that are inliers, using one of the plurality of temporary camera postures; a score calculation unit that calculates a score for each of the estimated temporary camera postures from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the temporary camera posture; a camera pose estimation unit that estimates a camera pose using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; A camera posture estimation device comprising:

2. a corresponding point setting unit that extracts two-dimensional feature points as the two-dimensional points from the two-dimensional image data, extracts three-dimensional feature points as the three-dimensional points from the three-dimensional data, and sets pairs of the corresponding two-dimensional points and three-dimensional points by combining the two-dimensional feature points and the three-dimensional feature points that correspond to each other; The camera posture estimation device according to claim 1 .

3. a two-dimensional line segment extraction unit that extracts the two-dimensional line segments from the two-dimensional image data; a three-dimensional line segment extraction unit that extracts the three-dimensional line segments from the three-dimensional data; Further provided with the corresponding line segment inlier calculation unit sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; The camera posture estimation device according to claim 1 .

4. a corresponding line segment setting unit that extracts the two-dimensional line segments from the two-dimensional image data, extracts the three-dimensional line segments from the three-dimensional data, and sets pairs of the corresponding two-dimensional line segments and three-dimensional line segments using the extracted two-dimensional line segments and the extracted three-dimensional line segments; the temporary camera posture estimation unit selects a plurality of pairs from among the pairs of corresponding two-dimensional points and three-dimensional points and the pairs of corresponding two-dimensional line segments and three-dimensional line segments, and estimates a plurality of temporary camera postures using the selected plurality of pairs. The camera posture estimation device according to claim 1 .

5. the score calculation unit calculates, for each of the estimated virtual camera postures, the sum or product of the number of pairs of inliers between two-dimensional points and three-dimensional points and the number of inliers between two-dimensional line segments and three-dimensional line segments, and sets the calculated value as the score. The camera posture estimation device according to claim 1 .

6. selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of tentative camera poses using the selected plurality of pairs; identifying inlier pairs of corresponding two-dimensional and three-dimensional points using one of the plurality of temporary camera poses; using one of the plurality of temporary camera poses, identifying pairs of corresponding two-dimensional and three-dimensional line segments that are inliers; For each estimated temporary camera pose, a score is calculated from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, both of which are identified using the temporary camera pose; a camera pose estimation method, characterized in that the camera pose is estimated using at least one of a pair of inliers between a two-dimensional point and a three-dimensional point, and a pair of inliers between a two-dimensional line segment and a three-dimensional line segment, when the score is maximized.

7. On the computer, selecting a plurality of pairs of corresponding two-dimensional points and three-dimensional points, and estimating a plurality of tentative camera poses using the selected plurality of pairs; using one of the plurality of virtual camera poses to identify inlier pairs of corresponding two-dimensional and three-dimensional points; using one of the plurality of virtual camera poses, identifying pairs of corresponding two-dimensional and three-dimensional line segments that are inliers; For each estimated temporary camera posture, a score is calculated from an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment, which are identified using the temporary camera posture; a camera pose is estimated using at least one of an inlier pair of a two-dimensional point and a three-dimensional point and an inlier pair of a two-dimensional line segment and a three-dimensional line segment when the score is maximized; program.