Comparison image search device, method and program

Abstract

To provide a comparison image retrieval apparatus which retrieves a comparison image suitable for use in estimating an attitude of a camera based on an image.SOLUTION: A comparison image retrieval apparatus is configured to: repeat estimating camera positions / attitudes, for each estimation, in a three-dimensional map of a query image, by performing image matching between the query image and a group of comparison images for each estimation selected from a group of reference images including images which have been rendered in advance in one or more camera positions / attitudes from the three-dimensional map, or images which have been captured in advance in one or more camera positions / attitudes, in a real world that the three-dimensional map is created for; repeat, until convergence is determined, estimating a camera position / attitude for the current estimation in the three-dimensional map of the query image by performing image matching between the query image and a group of comparison images for the current estimation selected from the group of reference images on the basis of a camera position / attitude for the previous estimation, in each of the repeated estimations; and output the group of comparison images used at the estimation in which the convergence is determined.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a comparison image search device, method, and program for searching for a comparison image suitable for use in image-based camera pose estimation. [Background technology]

[0002] Camera pose estimation based on an image is called a Visual Positioning System (VPS) and is used as a core technology in various fields such as robotics and augmented reality. Patent Document 1 discloses a technology related to a VPS that matches a three-dimensional point cloud map with a query image captured later and estimates the pose of the query image in the map. Patent Document 2 discloses a technology that estimates an approximate camera pose of a query image in a three-dimensional mesh map, renders a two-dimensional comparison image from the three-dimensional mesh based on the estimated approximate camera pose, and matches the comparison image with the query image in order to estimate the pose of a query image using a general-purpose three-dimensional mesh map instead of a three-dimensional point cloud map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-174285 [Patent Document 2] Patent application No. 2022-063059 Summary of the Invention [Problem to be solved by the invention]

[0004] When comparing a query image with images stored in a database, it is necessary to extract images suitable for comparison from the database. However, the accuracy of the method of extracting images based on the similarity of the images is insufficient. Patent Document 2 also assumes the use of GPS (Global Positioning System) information and geomagnetic sensors, but the locations where they can be used are limited.

[0005] In view of the problems with the conventional technology, the present invention aims to provide a comparison image search device, method, and program capable of searching for comparison images suitable for use in image-based camera pose estimation. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention is a comparison image search device, which repeats estimating a camera position and orientation of the query image in the 3D map for each round by performing image matching between a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a 3D map at one or more camera positions and orientations and / or images obtained by photographing in advance at one or more camera positions and orientations in the real world that is the subject of the 3D map, and repeats estimating a camera position and orientation of the query image in the 3D map for each round by performing image matching between a current comparison image set selected from the reference image set and the query image based on the camera position and orientation estimated in the previous round until a convergence determination is obtained, and outputs the comparison image set used in the round for which the convergence determination is obtained. Also, the present invention is characterized by a method and a program corresponding to the device. Effect of the Invention

[0007] According to the present invention, the camera position and orientation are updated and estimated through repeated processing until a convergence judgment is obtained, and at this time, a comparison image set selected from the reference image set based on this camera position and orientation is updated and used, thereby making it possible to ultimately output an appropriate image set to be compared with the query image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a functional block diagram of a comparison image search device according to an embodiment. [Diagram 2] 1 is a flowchart of an operation of a comparison image search device according to an embodiment. [Diagram 3] FIG. 13 is a schematic representation of the rendering of a comparison image from a three-dimensional map. [Figure 4] FIG. 13 is a schematic diagram illustrating the rendering of multiple comparison images to obtain a comparison image set. [Diagram 5] FIG. 13 is a schematic diagram of an estimated camera pose. [Figure 6] 10 is a diagram illustrating each process up to calculation of an estimated camera posture based on corresponding point pairs obtained by an image matching unit. FIG. [Figure 7] FIG. 13 is a schematic diagram showing a state when the result of convergence determination is NG. [Figure 8] 10 is a diagram illustrating a method for determining a pair of corresponding points between three-dimensional coordinates of a three-dimensional map and two-dimensional coordinates of a comparison image. [Figure 9] FIG. 5 is a diagram illustrating a method (5-1) for selecting an optimal comparison image. [Figure 10] FIG. 13 is a diagram illustrating a method (5-2) for selecting an optimal comparison image. [Figure 11] FIG. 13 is a diagram illustrating a method (5-4) for selecting an optimal comparison image. [Figure 12] FIG. 13 is a diagram showing an example of the relationship between two-dimensional coordinates (uj, vj) and (width, height). [Figure 13] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1 is a functional block diagram of an optimal comparison image retrieval device 10 for image matching with a 3D map according to an embodiment. The comparison image retrieval device 10 includes a comparison image rendering unit 11, an initial comparison image selection unit 12, an image matching unit 13, an estimated camera posture calculation unit 14, an optimal comparison image selection unit 15, and a convergence determination unit 16.

[0010] Fig. 2 is a flowchart of the operation of the comparison image search device 10 according to the embodiment. Below, the operation of each functional unit of the comparison image search device 10 in Fig. 1 will be described in detail while explaining each step in Fig. 2.

[0011] The flow in Fig. 2 has a structure in which the processes of steps S3 to S7 are repeated until the convergence determination in step S7 is established. The number of repetitions is referred to as N (N=1 is the first time, and N increases with each repetition as N=1, 2, 3, ...), and the current number of repetitions in the explanation of the flow in Fig. 2 is an arbitrary N, and will be explained separately from the previous N-1th time and the future N+1th time. Here, steps S1 and S2 belong only to the first time N=1.

[0012] (1) Step S1...Comparative image rendering unit 11 In step S1, the comparison image rendering unit 11 renders a set of m comparison images PS={P(i)|i=1, 2, ..., m} (reference image set PS) as a process of constructing an entire database of reference images, and then proceeds to step S2.

[0013] Specifically, the comparison image rendering unit 11 renders a plurality of comparison images PS from a three-dimensional map M prepared in advance as input data. The rendered comparison images PS are output to an initial comparison image selection unit 12, which will be described later, and are used for image matching with the query image. The rendered comparison images PS are also output to an optimal comparison image selection unit 15, which will be described later, and become candidates for the comparison images to be selected.

[0014] Fig. 3 is a diagram showing a schematic representation of the rendering of a comparison image P(i) from a three-dimensional map M. By providing external parameters EX(i) (corresponding to the camera position and orientation) and internal parameters IN(i) in the world coordinates of the three-dimensional map M for each comparison image P(i) (i=1, 2, ..., m) belonging to the comparison image set PS, each comparison image P(i) can be rendered using an existing method in three-dimensional CG (computer graphics). Note that this rendering is possible by preparing, for example, polygon vertex coordinates and polygon textures for the structures that make up the three-dimensional map M.

[0015] In this way, by rendering each of the comparison images P(i) belonging to the comparison image set PS, a comparison image set PS consisting of multiple m images can be obtained, as shown in Fig. 4. (Note that Fig. 4 shows, as a schematic example, comparison images P(1) to P(13), which are part of the comparison image set PS.)

[0016] The rendering position given by the external parameters EX(i) may be randomly determined at the ground level where the query image Q is assumed to be captured, or in a space without structures (such as a road in the real world). (Each of the external parameters EX(i) may be set comprehensively within the spatial range in which the 3D map M is defined so that the position and orientation of the query image Q can be estimated by matching it with the comparison image set PS.) The internal parameters IN(i) may be randomly selected from a predetermined setting, or the same internal parameters as those of the query image Q may be used for all comparison images P(i) (i=1, 2, ..., m).

[0017] Furthermore, for all or a part of the rendering images constituting the comparison image set PS, instead of using rendering images generated by rendering in the comparison image rendering unit 11, real photographs used when generating the 3D map M (real photographs taken in the real world, such as a street corner, which is the subject of the generation of the 3D map M) may be used. Furthermore, for the comparison image set PS as a reference image database, instead of executing the processing of the comparison image rendering unit 11 as internal processing in the comparison image retrieval device 10, the comparison image set PS corresponding to the 3D map M may be prepared in advance in an external server or the like, and the comparison image set PS may be read from this external server by the comparison image retrieval device 10 (read by the initial comparison image selection unit 12 and the optimum comparison image selection unit 15), thereby omitting the processing of the comparison image rendering unit 11.

[0018] (2) Step S2...First comparison image selection unit 12 In step S2, the initial comparison image selection unit 12 selects the initial comparison image set PS[1] for N=1 from the comparison image set PS obtained by the comparison image rendering unit 11 (i.e., selects it as PS[1] ⊂ PS), and proceeds to step S3.

[0019] Specifically, the initial comparison image selection unit 12 selects comparison images suitable for image matching with the query image Q in the image matching unit 13 (first time when N=1), described later, from among the multiple comparison images (the entire comparison image set PS) rendered by the initial comparison image rendering unit 11, to obtain a set PS[1]. The selected comparison images are output to the image matching unit 13, described later, for the first processing when N=1. Here, only one comparison image may be selected, or multiple comparison images may be selected.

[0020] (3) Step S3...Image matching unit 13 In step S3, the image matching unit 13 performs image matching between the set of comparison images PS[N] corresponding to the number of repetitions N and the query image Q to obtain corresponding point pairs, and then the process proceeds to step S4.

[0021] Specifically, the image matching unit 13 performs image matching between the comparison image set PS[1] selected by the initial comparison image selection unit 12 and the query image Q for the first time when N=1, and from the second time onwards when N≧2, performs image matching between the comparison image set PS[N] selected by the optimal comparison image selection unit 15 described later and the query image Q.

[0022] As described above, the matching target set PS[N] differs depending on the number of times N, but the image matching process itself is common regardless of the number of times N, and a corresponding point pair between the two-dimensional coordinates of the query image Q and the two-dimensional coordinates of the comparison image P(i)∈PS[N] is obtained. The obtained corresponding point pair is output to an estimated camera posture calculation unit 14, which will be described later. For image matching, the technology described in Non-Patent Document 1 or the technology described in Patent Document 2, mentioned above, may be used. [Non-patent Document 1] "Wang, Qianqian, et al. "Learning feature descriptors using camera pose supervision." European Conference on Computer Vision. Springer, Cham, 2020."

[0023] When multiple comparison images are input, image matching is performed with each comparison image against one query image, and the result is output to the estimated camera posture calculation unit 14 described later. For example, when there are five comparison images, image matching is performed five times, and corresponding point pairs for five times are output to the estimated camera posture estimation unit 14 described later. However, corresponding point pairs obtained as a result of performing image matching multiple times may be aggregated as a result for one time and output to the estimated camera posture estimation unit 14 described later.

[0024] (4) Step S4...Estimated camera attitude calculation unit 14 In step S4, the estimated camera posture calculation unit 14 calculates an estimated camera posture of the query image Q from the result (corresponding point pair) in the image matching unit 13, and the process proceeds to step S5.

[0025] Specifically, the estimated camera posture calculation unit 14 obtains a pair of corresponding points between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the query image based on the corresponding point pairs between the query image PS and the comparison image P(i)∈PS[N] obtained by the image matching unit 13, and calculates an estimated camera posture of the query image Q using the obtained pair. When the results of multiple image matching operations are input from the image matching unit 13, the estimated camera postures are calculated for the multiple operations, and the better camera posture (the camera posture determined to be the best) is adopted as the estimated camera posture. The method of determining the better camera posture may be, for example, to apply Ransac (random sample voting) described in Non-Patent Document 4 (described later) when estimating the camera posture, and adopt the one with the largest number of calculated inliers.

[0026] In this way, the image matching unit 13 can obtain image matching results integrated into a single result or multiple image matching results, and in either case, the estimated camera posture calculation unit 14 can obtain one estimated camera posture corresponding to one external parameter EX[N] as the Nth result.

[0027] The calculated estimated camera pose of the query image Q is output to an optimum comparison image selection unit 15, which will be described later, and is used to select an optimum comparison image (a set of comparison images PS[N+1] for the next N+1th iteration) from among the multiple comparison images PS rendered by the initial comparison image rendering unit 11. The estimated camera pose of the query image Q is found by first calculating a corresponding point pair between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the query image, and then solving a PnP (Perspective-n-Point) problem, which is a technique described in Non-Patent Document 2. [Non-Patent Document 2]: "Lepetit, Vincent, Francesc Moreno-Noguer, and Pascal Fua. "Epnp: An accurate o (n) solution to the pnp problem." International journal of computer vision 81.2 (2009): 155."

[0028] As shown in Figure 5, which is a schematic diagram of the estimated camera pose of the calculated query image Q, similar to Figure 3 which explains rendering, the Nth estimated camera pose can be estimated as an external parameter EX[N], which is the position and pose of the camera in the world coordinate system in which the 3D map M is defined.

[0029] In addition, corresponding point pairs between the three-dimensional coordinates of the three-dimensional map and the two-dimensional coordinates of the query image can be obtained via the two-dimensional coordinates of the comparison image based on corresponding point pairs between the two-dimensional coordinates of the comparison image and the two-dimensional coordinates of the query image, and corresponding point pairs between the three-dimensional coordinates of the three-dimensional map and the two-dimensional coordinates of the comparison image.

[0030] FIG. 6 is a diagram showing each process p1 to p4 up to the calculation of the estimated camera posture based on the corresponding point pair obtained by the image matching unit 13. Process p1 is the result of the image matching unit 13, and for the result of this process p1 (corresponding point pair of the two-dimensional coordinates of the comparison image P(i) and the two-dimensional coordinates of the query image Q), as shown in process p2, the two-dimensional coordinates of the comparison image P(i) among the corresponding point pair are determined as corresponding three-dimensional coordinates by referring to the original three-dimensional map M. Therefore, by checking the results of processes p1 and p2, the three-dimensional coordinates of the corresponding point pair corresponding to the two-dimensional coordinates of the query image Q are determined as shown in process p3. Thereby, the camera posture of the query image Q can be estimated by solving the PnP problem shown in process p4.

[0031] (5) Step S5...Optimal comparison image selection unit 15 In step S5, optimal comparison image selection unit 15 selects an optimal set of comparison images PS[N+1] for the next time (N+1th time), and then the process proceeds to step S6.

[0032] Specifically, the optimum comparison image selection unit 15 selects an optimum set PS[N+1] of comparison images for the next N+1th iteration from among a plurality of comparison images (comparison image set PS) rendered by the comparison image rendering unit 11, based on the estimated camera pose of the query image Q estimated by the estimated camera pose calculation unit 14 (the current estimated camera pose estimated as the external parameter EX[N]). The selected optimum comparison image set PS[N+1] is output to the convergence determination unit 16, which will be described later, and used for convergence determination. (Note that if the result of the convergence determination is that the processing is not to be terminated in the current Nth iteration but is to be repeated for the next N+1th iteration, the set PS[N+1] obtained in the current Nth iteration will be used as the target for matching processing with the query image Q in the image matching unit 13 in step S3 for the next N+1th iteration.)

[0033] The comparison image to be selected as the set of comparison images PS[N+1] may be one or more. Details of the process of the optimum comparison image selection unit 15 will be described later as an embodiment.

[0034] (6) Steps S6, S7, S8...convergence determination unit 16 In step S6, the convergence determination unit 16 performs a convergence determination, and then the process proceeds to step S7. In step S7, if the determination result in step S6 is convergence, the process proceeds to step S8, and if convergence is not determined, the process returns to step S3, and the above-mentioned processes from step S3 onwards are repeated as the next (N+1)th process. In step S8, the comparison image PS[N+1] (or PS[N]) is output as the search result in the comparison image search device 10, and the flow in FIG. 2 ends.

[0035] Specifically, the convergence determination unit 16 performs a convergence determination on the comparison image set PS[N+1] selected by the optimal comparison image selection unit 15. If the convergence determination result is OK (converged), the latest comparison image set PS[N+1] selected is output. If the convergence determination result is NG (not converged), the process proceeds to "steps S6 → S7 → S3", and in the functional block configuration, the process returns to the image matching unit 13, and image matching is performed again using the latest comparison image PS[N+1] selected (as the current N+1th image), and the estimated camera posture calculation unit 14 calculates the estimated camera posture of the query image Q. After that, the optimal comparison image selection unit 15 reselects a comparison image based on the estimated camera posture (external parameter EX[N+1]) of the query image calculated by the estimated camera posture calculation unit 14, and the comparison image set PS[(N+1)+1]=PS[N+2] is output to the convergence determination unit 16 again.

[0036] As described above with respect to the flow structure of Fig. 2, this cycle is repeated until the result of the judgment by the convergence judgment unit 16 becomes OK. Convergence may be judged to have occurred when it is judged that the latest (Nth) comparison image PS[N+1] is the same as the comparison image PS[N] selected immediately before (N-1th) that is the same. The sameness may be judged by whether the overlap rate (rate at which identical images exist) of the image sets PS[N+1] and PS[N] is equal to or greater than a threshold value.

[0037] Alternatively, it may be determined that convergence has occurred if the change between the estimated camera position and orientation (external parameters EX[N]) of the latest (Nth) query image Q and the position and orientation (external parameters EX[N-1]) of the estimated camera orientation of the previous (N-1th) query image Q is within a threshold. The threshold may be set freely, for example, to 1 m, for the position. However, in the case of N=1, since there is no previous (N-1th) result, it is always determined as NG, and the process branches to the image matching unit 13. (Note that in the case of convergence determination based on the overlap ratio of sets PS[N+1], PS[N], etc., rather than the position and orientation, it is possible that the convergence determination is obtained by chance at the first time when N=1.) Note that, regarding the threshold determination of the change, the estimated camera position and orientation are composed of a position element (translation component) and an orientation element (rotation element), but the changes in both the position element and the orientation element may be used as the target of the threshold determination, or only one of the changes (for example, only the position) may be used as the target of the threshold determination.

[0038] Figure 7 is a schematic diagram showing the state when the convergence judgment result is NG. Since it is judged that the current Nth comparison image set PS[N+1] does not match the previous N-1th comparison image set PS[N], it is judged that convergence has not occurred.

[0039] Above, each functional block in Fig. 1 was described while explaining each step in Fig. 2. In the following, an embodiment of the process of each functional block in Fig. 1 will be further described.

[0040] (2) The initial comparison image selection unit 12 selects the comparison image set PS as follows: (2-1) You may use the Image Retrieval technology described in Non-Patent Document 3, (2-2) When performing image matching using the technique described in Non-Patent Document 1 or the technique described in Patent Document 2, the Ransac (random sample voting) described in Non-Patent Document 4 may be applied to select a comparison image having a large number of values ​​(inliers) other than the calculated outliers, or (2-3) After performing image matching using the technology described in Non-Patent Document 1 or the technology described in Patent Document 2, the camera posture of the query image may be estimated using the technology described in Non-Patent Document 2. Then, Ransac described in Non-Patent Document 4 may be applied to select a comparison image with a large number of calculated inliers. [Non-Patent Document 3]: "Revaud, J., Almazan, J., Rezende, RS, & Souza, CRD (2019). Learning with average precision: Training image retrieval with a listwise loss. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 5107-5116)." [Non-Patent Document 4]: "MA Fischler and RC Bolles, ``Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,'' Communications of the ACM 24(6):381-395 (1981).''

[0041] (4) When the estimated camera attitude calculation unit 14 obtains a pair of corresponding points between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the comparison image P(i), (4-1) As shown in FIG. 8, the camera attitude position Pos(i) of the comparison image P(i) (given as a translation component in the external parameters EX(i) consisting of translation components and rotation components during rendering in the comparison image rendering unit 11, and in the case of an actual image rather than a rendered image, similar external parameters EX(i) may be associated and acquired in advance at the time of shooting) and the two-dimensional coordinates of the comparison image P(i), and the extension of the straight line connecting the position Pos(i) of the camera attitude of the comparison image P(i) and the two-dimensional coordinates of the comparison image P(i) may be obtained from the point of intersection with the three-dimensional map M (a structure therein); (4-2) When the method described in Patent Document 2 is used for image matching, the three-dimensional coordinates of the three-dimensional map may be converted into two-dimensional coordinates on the comparison image P(i) based on the camera posture of the comparison image P(i).

[0042] (5) Various embodiments of the method for selecting the optimal comparison image (set PS[N+1]) in optimal comparison image selection unit 15 will be described below. (5-1) For example, as shown in FIG. 9, a comparison image located within a range R close to the position of the estimated camera pose of the query image Q (the translation component of the external parameter EX[N]) may be selected. Q ,Y Q ,Z Q ) is the estimated camera pose (rotation matrix R Q and the translation vector t Q ) and

[0043]

number

[0044] This can be expressed as: The coordinates (X i ,Y i ,Z i ) is the value used when rendering the comparison image by the comparison image rendering unit 11 (if it is a real-photographed image, the value is associated in advance at the time of shooting). Q ,Y Q ,Z Q ) and the coordinates of the camera posture of the comparison image P(i) (X i ,Y i ,Z i ) the distance d between the two points l Find the distance d l If d is smaller than d, it may be determined that the distance is close. In addition, the number of comparison images (set PS[N+1]) to be selected is d l You can choose the one with the smallest distance d l Alternatively, multiple smaller ones may be selected.

[0045] (5-2) For example, as shown in FIG. 10, a comparison image may be selected in which the rotation of the estimated camera pose of the query image Q (the rotation component of the external parameter EX[N]) is similar to the rotation of the camera pose of the comparison image. The determination of whether the rotation of the camera pose is similar is performed by using the rotation angle θ Q and the rotation angle θ in the quaternion representation of the camera pose of the comparison image P(i) i Difference d θ Take the difference d θ In addition, the number of images to be selected for comparison (set PS[N+1]) may be d θ Alternatively, multiple values ​​with smallest d_θ may be selected.

[0046] (5-3) For example, the distance d between the estimated pose position of the query image obtained by the method of (5-1) and the pose position of the comparison image l The difference d between the rotation angle in the quaternion representation of the orientation of the query image and the rotation angle in the quaternion representation of the orientation of the comparison image obtained by the method of (5-2) is θ Using the weighted average

[0047]

number

[0048] A comparison image with a small w l and w θ is a tunable parameter, and w l If we reduce d l Since the influence of w becomes larger, comparison images closer to the position of the query image are more likely to be selected. θ If we reduce d θ Since the influence of becomes large, it becomes easier to select a comparison image with a smaller difference in rotation angle from the query image. In addition, the number of comparison images (set PS[N+1]) to be selected may be the one with the smallest W, or dθ Alternatively, multiple smaller ones may be selected.

[0049] (5-4) For example, as shown in FIG. 11, if the camera pose of a comparison image exists within the field of view V of the estimated camera pose of a query image Q, that comparison image may be selected. If the following expressions (3) and (4) are satisfied simultaneously, it may be determined that the camera pose of a comparison image P(j)∈PS[N+1] exists within a rendering image (i.e., within the field of view V) using the estimated pose of the query image Q (pose estimated as the external parameter EX[N]). (In other words, if either expression (3) or (4) does not hold, it is determined that the comparison image P(j) is not "P(j)∈PS[N+1]".) 0≦u j ≦width …(3) 0≦v j ≦height …(4)

[0050] Here, (width, height) are the width and height of the image size of the query image Q, and (u j ,v j ) is the position of the camera posture of the comparison image P(j) (each candidate image P(j) to be judged as belonging to PS[N+1] or not) in the camera coordinate system of the query image Q. j ,v j An example of the relationship between (u) and (width, height) is shown in Figure 12. j ,v j ) is calculated by using the perspective projection matrix P (the matrix P corresponding to the internal parameter information of the query image Q) to calculate the coordinates (X j ,Y j ,Z j ) can be converted from

[0051]

number

[0052] However, the projection matrix P is calculated by using the camera rotation matrix R, translation vector t, and internal parameters K when rendering the comparison image. P = K[R t] …(6) If no camera pose of a comparison image is within the field of view of the estimated camera pose of the query image, a comparison image may be selected using any of methods (5-1), (5-2), or (5-3).

[0053] As described above, according to each embodiment and example of the present invention, by repeatedly calculating the estimated camera attitude of the query image and selecting the optimal comparison image, it is possible to select a comparison image suitable for image matching, and therefore it is possible to select the optimal comparison image from multiple candidates prepared in advance even in a location where a GPS or a geomagnetic sensor cannot be used. The repeated processing is as described above, but in one embodiment, for example, the following result may be obtained. ●At N=1 (first time), a comparison image set PS[1] is selected from the reference image set PS, an estimated camera pose EX[1] is estimated from the set PS[1], and a comparison image set PS[2] for the next time N=2 is selected from the reference image set PS. Since there is little overlap between the sets PS[1] and PS[2], it is determined that convergence has not occurred. ●At N=2, estimate the estimated camera pose EX[2] from the set PS[2], select a comparison image set PS[3] for the next N=3 from the reference image set PS, and determine that convergence has occurred since there is a large overlap between the sets PS[2] and PS[3], and output PS[3] as the result. (Note that it is also possible to output PS[N] (PS[2] in this example) instead of PS[N+1] (PS[3] in this example) as the result.)

[0054] In particular, when the reference image set PS prepared initially is dense and spans a large number of images (for example, when the intervals of the external parameters EX(i) corresponding to the position and orientation of each image P(i) are dense, such as 1 cm intervals and 1° intervals for all three axes of rotation, and as a result the number of images that make up the image set PS is also large), it is expected that a more appropriate comparison image set PS[N] will be selected, and ultimately convergence to an appropriate set PS[N], as the comparison image set PS[N] selected changes each time, from PS[1] → PS[2] → PS[3] → ..., through the repetition of each embodiment or example of the present invention.

[0055] Various supplementary examples, alternative examples, additional examples, etc. will be described below.

[0056] <1> The comparative image search device 10 of this embodiment can contribute to improving the accuracy of self-location estimation of a robot as an element technology of VPS, etc., even in a situation where GPS, etc. are not used. This improves the convenience of remote robots, does not require leaders to travel to remote sites, and reduces carbon dioxide emissions by saving energy resources required for user movement, making it possible to contribute to Goal 13 of the Sustainable Development Goals (SDGs) led by the United Nations, "Take urgent action to combat climate change and its impacts."

[0057] (14) FIG. 13 is a diagram showing an example of a hardware configuration of a general computer device 70. The comparison image search device 10 can be realized as one or more computer devices 70 having such a configuration. When the comparison image search device 10 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, a camera 78, and a bus BS for transmitting and receiving data between them.

[0058] Each functional unit of the comparison image search device 10 can be realized by a CPU 71 and / or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the above, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the above. Processing results by the comparison image search device 10 may be displayed and output on the display 76. [Explanation of symbols]

[0059] 10: Comparison image search device, 11: Comparison image rendering unit, 12: Initial comparison image selection unit, 13: Image matching unit, 14: Estimated camera posture calculation unit, 15: Optimal comparison image selection unit, 16: Convergence determination unit

Claims

1. by performing image matching between a query image and a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a 3D map at one or more camera positions and / or images obtained by photographing in advance at one or more camera positions and / or in the real world that is the subject of the 3D map, estimating the camera position and orientation of the query image in the 3D map for each round repeatedly; In each iteration, image matching is performed between the current set of comparison images selected from the set of reference images and the query image based on the camera position and orientation estimated as the previous one, thereby estimating the current camera position and orientation of the query image in the 3D map until a convergence determination is obtained; a comparison image search device that outputs a set of comparison images used in the round in which the convergence judgment was obtained.

2. 2. The comparison image search device according to claim 1, wherein the convergence determination is obtained when it is determined that there is little change between the current comparison image set and the previous comparison image set.

3. The comparison image search device according to claim 1, characterized in that the convergence judgment is determined to have been obtained when it is determined that there is little change between the position and / or orientation in the camera position and orientation estimated for the current time and the position and / or orientation in the camera position and orientation estimated for the previous time.

4. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the set of reference images those images that are determined to have a position close to the camera position and orientation estimated for the previous image.

5. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the set of reference images those images whose pose is determined to be close to the pose of the camera position and pose estimated as the previous image.

6. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the set of reference images those whose position and orientation are determined to be close to the camera position and orientation estimated as the previous image.

7. The comparison image search device described in claim 1, characterized in that the current set of comparison images is determined by selecting from the set of reference images those images that are determined to be located within the camera field of view at the camera position and orientation estimated as the previous one.

8. by performing image matching between a query image and a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a 3D map at one or more camera positions and / or images obtained by photographing in advance at one or more camera positions and / or in the real world that is the subject of the 3D map, estimating the camera position and orientation of the query image in the 3D map for each round repeatedly; In each iteration, image matching is performed between the current set of comparison images selected from the set of reference images and the query image based on the camera position and orientation estimated as the previous one, thereby estimating the current camera position and orientation of the query image in the 3D map until a convergence determination is obtained; A comparison image retrieval method comprising: outputting a set of comparison images used in the round in which the convergence judgment was obtained.

9. 8. A program for causing a computer to function as the comparison image search device according to claim 1.

Images