Matching device, method and program

Abstract

To perform proper matching between a photographed image and three-dimensional mesh data in which a texture may not necessarily match an actual photographed image or a texture may be unclear.SOLUTION: A matching device 10 for performing matching of a feature amount between a photographed image and a three-dimensional mesh constituted by information on a three-dimensional apex and a texture of a surface where the three-dimensional apex is formed, executes: processing 11, 12, 13, 21 and 22 of obtaining a rendering image by rendering the three-dimensional mesh in a standard camera posture obtained by estimating a posture of a camera that has captured the photographed image, and extracting a first feature amount from a first feature point with all or a portion of a two-dimensional position corresponding to the three-dimensional apex of the three-dimensional mesh with respect to the rendering image as the first feature point; processing 23 and 24 of detecting a second feature point from the photographed image and extracting a second feature amount from the second feature point; and processing 30 of performing matching between the first feature amount and the second feature amount.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a matching device, method, and program for performing matching between a three-dimensional mesh to which a texture is assigned and a captured image. [Background technology]

[0002] Accurately obtaining the correspondence between two images plays an important role in the restoration of three-dimensional structures, augmented reality, autonomous driving, autonomous robots, and VPS (Visual Positioning System), among others.

[0003] Patent Document 1 describes a technology called VPS that uses images to estimate the vehicle's position and orientation based on a 3D point cloud map. To create a 3D point cloud map, a 3D point cloud is reconstructed from video captured with an RGB camera or similar device through processing such as feature point matching. To estimate the vehicle's position and orientation, the camera pose is estimated by solving the Perspective-n-Point (PnP) problem (the technology described in Non-Patent Document 1) based on the feature points of the 3D point cloud and the captured photograph. The correspondence between the feature points of the 3D point cloud and the captured photograph is obtained by obtaining the correspondence between the image used to generate the 3D point cloud and the captured photograph (feature point matching). Note that feature points are characteristic points in an image, and various algorithms exist for detecting them. For example, Non-Patent Document 2 proposes a feature point detection method called SIFT, which uses DoG (Difference of Gaussian) differential images.

[0004] Accurate feature point matching between two images is essential for accurate estimation of self-position and direction, and it is desirable that the positions of feature points on an image are evenly distributed across the entire image. Various techniques have been proposed for accurate feature point matching. Non-Patent Document 2 proposes SIFT, which detects feature points from an image and performs scale-invariant feature transformation. Classic image matching techniques, such as this one, are achieved through three steps: detecting feature points from two images, describing the feature values ​​of each feature point, and comparing the feature values.

[0005] Meanwhile, in recent years, deep learning-based methods have also been proposed that obtain correspondence between two images without detecting feature points as employed in classical methods. In Non-Patent Document 3, a method based on convolutional neural networks is used to extract features from the entire image, and an attention mechanism is used to capture areas with high similarity between the two images, narrowing down the corresponding areas, and then performing more detailed matching within those areas to derive correspondence. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-174285 [Non-patent literature]

[0007] [Non-Patent Document 1] Lepetit, Vincent, Francesc Moreno-Noguer, and Pascal Fua. "Epnp: An accurate o (n) solution to the pnp problem." International journal of computer vision 81.2 (2009): 155. [Non-patent document 2] Lowe, David G. "Distinctive image features from scale-invariant keypoints." International journal of computer vision 60.2 (2004): 91-110. [Non-patent document 3] Sun, Jiaming et al. "LoFTR: Detector-free local feature matching with transformers." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021. Summary of the Invention [Problem to be solved by the invention]

[0008] In Patent Document 1, a dedicated 3D point cloud map must be created each time a VPS service is provided in an area. However, in recent years, various 3D scanners, such as Matterport (registered trademark) and LiDAR (light detection and ranging), have made it possible to acquire 3D mesh data of solids and space, and 3D mesh spatial data is being created for various purposes, such as city models, CAD (computer-aided design) data, VR (virtual reality simulation), and indoor 3D maps for previews. If 3D mesh data created for various purposes could be reused for VPS as a standard, there would be no need to create a 3D point cloud map solely for the purpose of providing VPS services, and map creation costs could be reduced.

[0009] However, conventional technology has not considered using 3D mesh data prepared for various purposes in VPS services, which compare it with images taken on-site and use it for positioning.

[0010] Here, 3D mesh data consists of vertex information of the scanned object, polygons (surfaces) created based on those vertices, and texture information to be applied to the surfaces, but it does not perfectly reproduce the real space, and in particular, detailed textures may differ from the real thing. Also, if it is outdoors, changes may occur to the paint on the wall or signs.

[0011] Figure 1 shows an example of 3D mesh data and the corresponding image taken at the actual site. As an example, Figure 1 compares image P101, which is a rendered image of a section of the mesh of Plateau's 3D mesh data published by the Ministry of Land, Infrastructure, Transport and Tourism, with photograph P102 taken at the site. In order to use 3D mesh data in VPS, it is necessary to perform feature point matching between images that are similar but not the same: image P101 (rendered image P101) generated based on the 3D mesh as shown in Figure 1, and photograph P102 taken at the site.

[0012] However, the classical image comparison method typified by Non-Patent Document 2 is a technology that assumes that the same object exists in each image, and detects feature points in each image and then compares the feature points. Therefore, in cases where the texture of the 3D scanned object differs from the actual object, as in the example of Figure 1, the feature points detected in the photograph may not be detected in the rendered image, and as a result, feature point detection may not be performed correctly, which may result in a decrease in the accuracy of feature point matching.

[0013] In Non-Patent Document 3, feature points are not detected, but rather features are acquired from the entire image based on a convolutional neural network. Furthermore, location information is added to the acquired features, and self-attention and cross-attention are used to acquire features for each region divided into a certain grid size. The authors propose a method for acquiring feature point pairs by comparing the similarity between two images using the features for each region and performing more detailed matching within similar regions between the two images.

[0014] However, in a rendered image of 3D mesh data, such as image P101 in Figure 1, there are structures with clear textures and structures with unclear textures. In Non-Patent Document 3, as described above, feature point detection is not performed, and feature values ​​are acquired based on a convolutional neural network. Therefore, there is a high possibility that feature points can be acquired densely for structures with clear textures, but conversely, there is a high possibility that feature points cannot be acquired densely for structures with unclear textures. As a result, the positions of feature point pairs that can be acquired after feature point matching may be concentrated in areas with clear textures. If feature point pairs are not evenly distributed throughout the image, the accuracy of camera pose estimation may decrease, making it impossible to perform accurate self-localization.

[0015] In view of the above-mentioned problems with the conventional technology, the present invention aims to provide a matching device, method, and program that can perform appropriate matching between a photographed image and 3D mesh data, the texture of which may not necessarily match that of the actual photographed image or may be unclear. [Means for solving the problem]

[0016] To achieve the above object, the present invention provides a matching device that matches features between a captured image and a 3D mesh formed by information on 3D vertices and the texture of the surfaces formed by the 3D vertices, and the device performs the following steps: rendering the 3D mesh at a reference camera posture that estimates the posture of the camera that captured the captured image to obtain a rendered image; extracting first feature values ​​from the first feature values, using all or part of two-dimensional positions corresponding to the 3D vertices of the 3D mesh from the rendered image as first feature values; detecting second feature values ​​from the captured image and extracting second feature values ​​from the second feature values; and matching the first feature values ​​with the second feature values. The present invention also provides a method and a program corresponding to the device. [Effects of the Invention]

[0017] According to the present invention, features of a 2D rendering image of a 3D mesh are extracted from the 3D vertices that represent the structural features of the object represented by the 3D mesh, and are matched with features extracted from the captured image, thereby enabling appropriate feature matching between a 3D mesh that does not necessarily have sufficient texture and a captured image. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing an example of three-dimensional mesh data and corresponding images taken at an actual site. [Figure 2] FIG. 1 is a functional block diagram of a matching device according to an embodiment. [Figure 3] FIG. 1 is a schematic diagram showing a reference camera pose in a three-dimensional mesh. [Figure 4] FIG. 2 is a diagram showing a schematic representation of the elevation angle, tilt angle, and azimuth angle of a camera for a captured image. [Figure 5] FIG. 10 is a diagram showing that the rotation of the camera object and the rotation of the coordinates have an inverse relationship. [Figure 6] 10A and 10B are diagrams illustrating an example in which matching candidates are limited, in comparison with an example in which no limit is imposed. [Figure 7] 3 is a functional block diagram of a matching device according to an additional embodiment to the configuration of FIG. 2. FIG. [Figure 8] FIG. 10 is a diagram showing a schematic example of a two-dimensional texture associated with a three-dimensional mesh in the case of a triangular polygon. [Figure 9] FIG. 10 is a diagram illustrating a unit normal vector of a polygon. [Figure 10] 10A and 10B are diagrams showing examples of the relationship between unit normals as a schematic example for explaining the processing content in the mesh vertex reduction unit. [Figure 11] FIG. 10 is a schematic diagram for explaining the processing content of a three-dimensional feature point restriction unit. [Figure 12] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE INVENTION

[0019] 2 is a functional block diagram of a matching device 10 according to an embodiment of the present invention, which performs a matching process after performing a composite feature point detection process based on the structure and texture of a 3D mesh. As shown in the figure, the matching device 10 includes a mesh vertex extraction unit 11, a reference camera posture estimation unit 12, an image rendering unit 13, a rendered image feature point conversion unit 21, a rendered image feature amount description unit 22, a captured image feature point detection unit 23, a captured image feature amount description unit 24, and a feature point matching unit 25.

[0020] As shown schematically in FIG. 2, the overall processing of the matching device 10 is to read two pieces of input data, namely, 3D mesh data and a captured image, extract feature points from both the 3D mesh data and the captured image, and then output the matching results between the feature points extracted from the 3D mesh data and the feature points extracted from the captured image in the form of sets (pairs) of corresponding feature points.

[0021] The following describes in detail the processing content of each functional unit of the matching device 10 that realizes this overall processing.

[0022] The input data is as described above, and the 3D mesh data, which is one of the two input data, is prepared in a format that links vertex information of the object obtained by scanning or other means, polygons (surfaces) created based on those vertices, and texture information to be applied to the surfaces. The other of the two input data, the photographed images, are prepared by taking images using a regular RGB camera or the like. Examples of 3D mesh data (rendered images) and photographed images P101 and P102 of the same street scene are as shown in Figure 1 above.

[0023] (1) The mesh vertex extraction unit 11 extracts vertex information from a three-dimensional mesh (information linking vertices, faces surrounded by the vertices, and texture information for the faces), which is one of the input data. The extracted vertex information is output to the downstream rendering image feature point conversion unit 21, where it is converted into two-dimensional feature points. A three-dimensional mesh is made up of polygons such as triangles, and has vertices that make up the polygons. For example, a triangle has three vertices. These vertices have three-dimensional coordinates defined, and the mesh vertex extraction unit 11 extracts the vertices of the three-dimensional mesh.

[0024] (2) The reference camera posture estimation unit 12 estimates an approximate camera posture (reference camera posture) from a captured image, which is one of the input data, with the correct answer being the exact camera posture (actual camera posture) at the time the captured image was captured. Here, the camera posture refers to the shooting position and direction, which can be expressed by a rotation matrix R and a translation vector t, respectively. The reference camera posture estimated by the reference camera posture estimation unit 12 is output to the downstream image rendering unit 13 and rendering image feature point conversion unit 21, and is used as the camera posture in the 3D mesh.

[0025] A schematic diagram showing the reference camera posture in a 3D mesh is shown in Figure 3. In the subsequent rendering image feature point conversion unit 21, the 3D feature points extracted by the vertex extraction unit 11 are converted into 2D feature points based on the reference camera posture estimated by the reference camera posture estimation unit 12. Finally, the resulting 2D feature points are matched with the 2D feature points of the captured image in the subsequent feature point matching unit 30. To improve the accuracy of this matching, it is desirable that the reference camera posture be as close as possible to the actual camera posture of the captured image. Since the actual camera posture is unknown at the time the captured image (photograph) is taken, the reference camera posture, which is a rough estimate, is estimated using information on the GPS (Global Positioning System) and the camera's rotation angle (information previously associated with the captured image as values ​​acquired by a GPS sensor or orientation sensor equipped on the camera when the image was taken), as described below.

[0026] The rotation matrix R of the reference camera posture is, for example, the elevation angle θ when the photograph was taken. x , inclination angle θ y , azimuth θ z (These angles may be obtained by using information previously linked to the captured image, such as values ​​acquired by an orientation sensor or the like provided on the camera when the image was captured.) In the world coordinates of the 3D mesh, the elevation angle represents the angle of rotation around the x-axis, the tilt angle represents the angle of rotation around the y-axis, and the azimuth angle represents the angle of rotation around the z-axis. Figure 4 shows a diagram that schematically illustrates the elevation angle, tilt angle, and azimuth angle of the camera of the captured image. In the world coordinates of the 3D mesh, the camera of the captured image is rotated around the x-axis at an angle θ x Rotation matrix R when rotating x , angle θ around the y-axis y Rotation matrix R when rotating y , angle θ around the z axis z Rotation matrix R when rotating z As is well known in the field of 3D CG (computer graphics), etc., can be expressed as the following equations (1), (2), and (3).

[0027]

number

[0028] Rotation matrix R x , rotation matrix R y , rotation matrix R z Using this, the camera of the captured image is rotated around the x-axis of the world coordinate system by θ x and then rotate it by θ around the y-axis of the world coordinate system. y and then rotate it by θ around the z-axis of the world coordinate system. z The camera rotation matrix R when the camera is rotated in this order is C can be expressed as the following equation (4). R C =R z R y R x …(4)

[0029] The rotation angle expressed as above is generally called a fixed angle. Here, the rotation matrix R of the reference camera posture calculated by the reference camera posture estimation unit 12 expresses a rotation matrix that rotates the "coordinates." C is the rotation matrix that rotates the camera. As shown in Figure 5 (a diagram showing an example of rotation around the z axis by an angle φ, which shows that the rotation of the camera object and the rotation of the coordinates are inversely related), the "camera orientation when the camera object is rotated by φ" is the same as the "camera orientation in the coordinate system after rotating the coordinates by -φ", so the rotation matrix R is expressed as R, as shown in the following equation (5). C (Furthermore, from the properties of the rotation matrix, R C (the transpose of the matrix). R=R C -1 =R C T …(5)

[0030] In addition, the translation vector t of the reference camera orientation is calculated by the rotation matrix R and the approximate shooting position (X C ,Y C ,Z C) can be used to calculate the following equation (6): (Note that this GPS information may be information previously associated with the captured image, such as a value acquired by a GPS sensor or the like attached to the camera when the image was captured.)

[0031]

number

[0032] (3) The image rendering unit 13 renders the 3D mesh, which is one of the input data, into a 2D image based on the reference camera posture (rotation matrix R and translation vector t) estimated by the reference camera posture estimation unit 12. (That is, the mesh defined by the 3D vertex information among the 3D mesh information and the texture applied to this mesh are projected in the same manner as in equations (7) and (8) described below, thereby rendering into a 2D image as seen from the reference camera posture.) The rendered image is output to the subsequent rendering image feature description unit 22. This rendering can be achieved by an existing method used in the field of 3D CG, and for actual rendering, a 3D computer graphics software tool set such as Blender (product name) may be used.

[0033] (4) The rendering image feature point conversion unit 21 converts the 3D feature points extracted by the mesh vertex extraction unit 11 into image coordinates using the reference camera orientation (rotation matrix R and translation vector t) obtained by the reference camera orientation estimation unit 12, converting them into 2D feature points. The obtained 2D feature points are output to the subsequent rendering image feature description unit 22. Here, by using a camera geometric model known in the field of 3D CG, the 3D world coordinates (X, Y, Z) can be converted into 2D image coordinates (u, v) using a projection matrix P as shown in the following equation (7). The projection matrix P in equation (7) can be expressed as the following equation (8) using the camera rotation matrix R, translation vector t, and internal camera parameter K. Here, the camera rotation matrix R and translation vector t use the values ​​obtained by the reference camera orientation estimation unit 12, and the camera internal parameter K uses the same value as used in the image rendering unit 13.

[0034]

number

[0035] (5) The rendering image feature description unit 22 calculates feature amounts for the two-dimensional feature points obtained by the rendering image feature point conversion unit 21 based on the image generated by the image rendering unit 13. (That is, the feature amounts are calculated at the positions of the two-dimensional feature points obtained by the rendering image feature point conversion unit 21 on the image generated by the image rendering unit 13.) The obtained feature amounts are output to the feature point matching unit 30 at the downstream side.

[0036] Any existing method may be used for calculating this feature amount. For example, the feature amount may be described using the technology described in the above-mentioned Non-Patent Document 2 or the following Non-Patent Document 4. Alternatively, the technology described in the following Non-Patent Document 5 may be used. In this case, since the method uses the 2D feature points obtained by the rendering image feature point conversion unit 21 as a query to search for corresponding feature points on the captured image side, the captured image feature point detection unit 23 at the subsequent stage, the captured image feature amount description unit 24 does not perform feature point description on the captured image side, and the feature point matching unit 30 does not perform feature point matching on the captured image side. Instead, the rendering image feature amount description unit 22 performs the process up to obtaining pairs of feature points between the rendering image and the captured photographic image. [Non-patent Document 4] Wang, Qianqian, et al. "Learning feature descriptors using camera pose supervision." European Conference on Computer Vision. Springer, Cham, 2020. [Non-patent Document 5] Jiang, Wei, et al. "COTR: Correspondence Transformer for Matching Across Images." arXiv preprint arXiv:2103.14167 (2021)

[0037] That is, the technique of Non-Patent Document 5 is a technique in which the captured image feature point detection unit 23 is omitted, and the processes of the rendering image feature amount description unit 22, the captured image feature amount description unit 24, and the feature point matching unit 30 are processed collectively by a deep learning network NW consisting of an encoder, converter, decoder, etc. This network NW receives three inputs: the captured image, the rendering image, and the query two-dimensional feature points obtained from the rendering image feature point conversion unit 21, and obtains a matching result of the reference feature points in the captured image without explicitly detecting two-dimensional feature points (reference feature points) from the captured image.

[0038] In other words, when the technique of Non-Patent Document 5 is applied to the matching device 10 of this embodiment, the processes corresponding to the processing of the captured image feature point detection unit 23, the processing of the rendered image feature amount description unit 22, the processing of the captured image feature amount description unit 24, and the processing of the feature point matching unit 30 are all realized implicitly (as internal processing within the network) in a deep learning network NW consisting of an encoder, converter, decoder, etc. (The two-dimensional feature points of the captured image and the feature points and feature amounts of the rendered image and captured image as intermediate data are not explicitly obtained, and matching in feature amount space is not explicitly performed; these processes are performed implicitly within the deep learning network NW, and only the feature point matching result, which is the final result, is output.)

[0039] Specifically, in the technique of Non-Patent Document 5, a deep learning network NW is configured in the order of "encoder → converter → decoder." When applied to the matching device 10 of this embodiment, the encoder encodes two images, a captured image and a rendered image, to obtain two feature maps, which are then combined to obtain a feature map. Furthermore, position information is added to the feature maps. The converter reads the combined feature map and the two-dimensional feature points obtained by the rendered image feature point conversion unit 21 as inputs to obtain a converted output. The decoder decodes this converted output to output a matching result of the feature points in the captured image relative to the two-dimensional feature points obtained by the rendered image feature point conversion unit 21. This "encoder → converter → decoder" configuration allows the deep learning network NW to implicitly realize processes equivalent to the processes of the captured image feature point detection unit 23, the rendered image feature amount description unit 22, the captured image feature amount description unit 24, and the feature point matching unit 30 in Figures 2 and 7, all at once.

[0040] (6) The photographed image feature point detection unit 23 detects feature points from the photographed image, which is one of the input data. The obtained feature points are output to the photographed image feature amount description unit 24 at the subsequent stage. For this feature point detection, for example, existing techniques described in Non-Patent Document 6 and Non-Patent Document 7 below may be used. [Non-Patent Document 6] "Revaud, Jerome, et al. "R2D2: repeatable and reliable detector and descriptor." arXiv preprint arXiv:1906.06195 (2019)" [Non-Patent Document 7] "Rublee, Ethan, et al. "ORB: An efficient alternative to SIFT or SURF." 2011 International Conference on Computer Vision. IEEE, 2011."

[0041] (7) The captured image feature amount description unit 24 calculates feature amounts for the feature points detected by the captured image feature point detection unit 23. The obtained feature amounts are output to the feature point matching unit 30 at the subsequent stage. This feature point matching unit 30 matches the two-dimensional feature points of the rendered image acquired by the rendered image feature amount description unit 22 and the feature amounts of the captured image acquired by the captured image feature amount description unit 24, so that the captured image feature amount description unit 24 describes the feature amounts using the same method as the feature amount description technique used by the rendered image feature amount description unit 22. For example, the feature amounts may be described using the techniques described in the above-mentioned Non-Patent Document 2 or Non-Patent Document 4.

[0042] In other words, the rendering image feature description unit 22 and the captured image feature description unit 24 describe the same types of features, and if the rendering image feature description unit 22 uses the method of non-patent document 2, the captured image feature description unit 24 should also similarly use the method of non-patent document 2.

[0043] (8) The feature point matching unit 30 matches the feature amounts of the rendering image calculated by the rendering image feature amount description unit 22 with the feature amounts of the captured image calculated by the captured image feature amount description unit 24, and outputs the result as a matching result in the matching device 10. For this matching, for example, the technology (Nearest Neighbor Matching) described in the following Non-Patent Document 8 may be used as an existing method. [Non-Patent Document 8] Rubin, Donald B. "Matching to remove bias in observational studies." Biometrics (1973): 159-183.

[0044] (8-1) Here, the feature point matching unit 30 may further limit the objects of comparison when comparing and matching the features of the rendering image of the 3D mesh calculated by the rendering image feature description unit 22 with the features of the captured image calculated by the captured image feature description unit 24.

[0045] Specifically, in the nearest neighbor matching of Non-Patent Document 8, feature amounts are calculated in the rendering image feature amount description unit 22 and the photographed image feature amount description unit 24 for the two-dimensional feature points of the rendering image acquired by the rendering image feature point conversion unit 21 and the two-dimensional feature points of the photographed photograph acquired by the photographed image feature point detection unit 23, respectively, and the similarities between the acquired feature amounts are compared in a brute-force manner, and feature point pairs whose feature amounts are determined to be similar by threshold judgment or the like are regarded as matching, but when comparing the similarities, it is also possible to limit in advance the targets for which the similarity of the feature amounts of the rendering image and the photographed image are compared.

[0046] Figure 6 is a diagram that schematically illustrates this limitation technique, showing a comprehensive similarity comparison example EX1 before limiting the two-dimensional feature points of the captured image P102 to the two-dimensional feature points of the rendered image P101, and a limited similarity comparison example EX2 after the limitation. (Note that the examples of images P101 and P102 whose features are being compared in Figure 6 are the same as the example in Figure 1, as they are given common reference numerals. Figure 6 schematically illustrates multiple feature points (feature amounts) in image P102 that are the subject of similarity comparison with one feature point (feature amount) in image P101, connected by straight lines.) In example EX2 after limiting the two-dimensional feature points of the captured image, the number of matching candidates has been limited and reduced compared to example EX1 when no limitation is performed.

[0047] The similarity comparison in nearest neighbor matching compares the feature values ​​of the rendered image and the feature values ​​of the captured photo in a brute force manner, ignoring the coordinate (u,v) information of the 2D feature points corresponding to the feature values.In contrast, by using this coordinate (u,v) information and limiting the objects to be compared in advance to only those within the range where the 2D coordinates (u,v) are determined to be close using a threshold judgment or the like (shown schematically as a rectangular range in Example EX2 in Figure 6), the number of matching candidates can be reduced, which is expected to be effective in avoiding incorrect matching.

[0048] Here, whether the coordinates (u, v) are close may be determined by directly using the values ​​of the coordinates (u, v) as described above. However, although the 3D camera coordinates and shooting direction from which the coordinates (u, v) are calculated are used as the reference camera posture by the reference camera posture estimation unit 12, this may not be accurate and may involve errors. For this reason, a method for determining the range in which the coordinates (u, v) are determined to be close and limiting the objects to be compared may use, for example, the coarse-level matches technique, which is an intermediate module described in Non-Patent Document 3. In this technique, features based on a convolutional neural network are acquired from the entire image, position information is assigned to the features, and features for each region divided into a certain grid size are acquired using self-attention and cross-attention, and the similarity between the two images is compared.

[0049] That is, feature matching may be performed within grids determined to be similar between the division grid of the rendered image and the division grid of the captured image by the method of Non-Patent Document 3. (As a result, comprehensive matching is performed within the grids determined to be similar, but the search range is more limited than when comprehensive matching is performed within the entire image.)

[0050] As described above, according to the embodiment of the matching device 10 in Fig. 2, vertices (vertices thought to correspond to corners of a real object, etc.) of a 3D mesh (consisting of structural information about the 3D mesh and texture information about each polygon (face)) are set as feature points to be used for calculating feature amounts, and the feature amounts are extracted after rendering at an approximate camera attitude (reference camera attitude) estimated based on the camera attitude at which the captured image was taken, and then matched with the feature amounts of the captured image, thereby achieving matching of feature amounts between the 3D mesh and the captured image, and making it possible to utilize this as a basic technology for VPS, etc. Furthermore, during matching, it is also possible to limit the feature points to be matched between two images, which is expected to improve matching accuracy.

[0051] Furthermore, according to the matching device 10 of an additional embodiment described below, it is possible to, for example, detect feature points evenly across the entire image, and to add or select appropriate feature points to be matched. By improving the matching accuracy, this can contribute to improving the accuracy of estimating the camera's posture for captured images, even in VPS applications, etc.

[0052] Fig. 7 is a functional block diagram of a matching device 10 according to an embodiment additional to the configuration of Fig. 2. As shown in the figure, the matching device 10 of Fig. 7 additionally includes four functional blocks, namely, a mesh structure simplification unit 41, a texture feature extraction unit 42, a mesh vertex reduction unit 43, and a three-dimensional feature point restriction unit 44, in addition to the configuration of Fig. 2, and performs additional processing in these locations.

[0053] 7 shows a state in which all four additional function blocks 41 to 44 are added to the matching device 10 having the configuration of FIG. 2, but it is possible to add all or any part of these four to the matching device 10 having the configuration of FIG. 2. Below, we will explain the additional processing content to the configuration of FIG. 2 when each of the four additional function blocks 41 to 44 is added individually, but it is possible to add all or some of these four additional components.

[0054] (9) The matching device 10 may further include a mesh structure simplification unit 41 at the location shown in Fig. 7, and may output a simplified 3D mesh to the mesh vertex extraction unit 11 and the image rendering unit 13. In other words, the mesh structure simplification unit 41 can be used as an additional component that performs simplification processing as preprocessing on the 3D mesh that is input data in the configuration of Fig. 2.

[0055] Simplifying the 3D mesh structure by the mesh structure simplification unit 41 means reducing the number of polygons in the 3D mesh and the number of vertices that make up the 3D mesh. 3D mesh vertices may be concentrated in areas where the 3D mesh structure is complex. If there is an area where 3D mesh vertices are concentrated, the 2D feature points converted from the 3D feature points by the rendering image feature point conversion unit 21 may not be distributed evenly throughout the rendering image, resulting in a bias. To eliminate this bias, an existing tool such as MeshLab (product name) in Non-Patent Document 9 below may be used to simplify the 3D mesh structure, reduce the number of vertices, and update the texture to a correspondingly simplified form. (Other methods may include simple thinning of the 3D vertices of the 3D mesh and mesh integration after thinning.)

[0056] (10) The matching device 10 may further include a texture feature extraction unit 42 at the location shown in FIG. 7 to extract 3D feature points with distinctive textures from the 3D mesh. In other words, the texture feature extraction unit 42 is configured to prepare 3D feature points to be input to the rendering image feature point conversion unit 21 in the configuration of FIG. 2, separately from the 3D feature points obtained from the mesh vertex extraction unit 11. In FIG. 7, the 3D feature points added by the texture feature extraction unit 42 are depicted as being skipped from the targets of processing by the mesh vertex reduction unit 43 (described later) and included in the targets of processing by the 3D feature point restriction unit 44. However, as depicted by the dotted lines, they may also be included in the targets of processing by the mesh vertex reduction unit 43 and not skipped. Furthermore, when the mesh structure simplification unit 41 is additionally used at the same time, a simplified 3D mesh may be input to the texture feature extraction unit 42.

[0057] The characteristic texture used to add three-dimensional feature points in the texture feature extraction unit 42 refers to a texture that has a unique pattern or design, and thus the feature points extracted from this texture increase the accuracy of matching between two images (contributing to ensuring the accuracy of matching between two images). In a three-dimensional mesh, as shown in FIG. 8, a two-dimensional texture is attached to a polygon. Texture information is often saved as a two-dimensional image file, such as PNG. Furthermore, the correspondence between the vertex coordinates of the polygons in the three-dimensional mesh and the two-dimensional coordinates ((u,v) coordinates) in the two-dimensional image file is saved, and the texture is often attached to the polygon.

[0058] To actually extract texture feature points, for example, feature points are detected from a two-dimensional image file of the texture, and the (u, v) coordinates of the detected feature points are converted into three-dimensional coordinates based on the correspondence between the (u, v) coordinates and the coordinates of the three-dimensional mesh. These three-dimensional coordinates are then used as additional outputs from the texture feature extraction unit 42. In FIG. 8, two-dimensional vertices a, b, and c of a two-dimensional polygon correspond to three-dimensional vertices A, B, and C of the three-dimensional mesh. Similarly, feature points extracted from the texture of a triangular polygon abc (polygon ABC viewed from the front) surrounded by the two-dimensional vertices correspond to feature points within the triangular polygon (triangle ABC) in three-dimensional space. Feature points may be detected from a two-dimensional image file (the texture of triangular polygon abc in FIG. 8) using, for example, the techniques described in Non-Patent Document 2 or Non-Patent Document 6.

[0059] (11) The matching device 10 may further include a mesh vertex reduction unit 43 at the location shown in FIG. 7, which may reduce the number of three-dimensional mesh vertices extracted by the mesh vertex extraction unit 11 and output the reduced number to a subsequent stage. As mentioned above, various combinations are possible, and three-dimensional feature points may be input to the mesh vertex reduction unit 43 only from the mesh vertex extraction unit 11, or, as shown by the dotted line as an example of no skipping, three-dimensional feature points may also be input from the texture feature extraction unit 42. The three-dimensional feature points reduced by the mesh vertex reduction unit 43 are output to the rendering image feature point conversion unit 21. (Note that, if a three-dimensional feature point restriction unit 44 is present, they are output to this unit.)

[0060] 1, which is a diagram showing an example of a rendered image P101 generated by the image rendering unit 13 and a captured image P102, and as can be seen from this example, in the rendered image P101, which has an unclear texture, corners and edges of structures are often feature points that are easy to identify and match. The vertices of the three-dimensional mesh extracted by the mesh vertex extraction unit 11 or the like in the preceding stage are converted into two-dimensional feature points by the rendered image feature point conversion unit 21, and are matched with the two-dimensional feature points of the captured image by the feature point matching unit 30. However, in order to leave only feature points that are easy to match in the feature point matching unit 30, the three-dimensional mesh vertices extracted by the mesh vertex extraction unit 11 or the like may be reduced to only vertices such as the corners and edges of structures.

[0061] For the purpose based on the above considerations, the mesh vertex reduction unit 43 plays a role in reducing the number of 3D vertices by limiting them in advance to only those that are expected to be easy to match. A specific reduction method, for example, is as follows: Of the vertices of the 3D mesh (including the vertices added by the texture feature extraction unit 42 if any), only those that are determined to be in pointed positions (the apexes of sharp convex shapes) in relation to the polygon are limited.

[0062] For example, when reducing only vertices such as corners and edges, the unit normal vector U of the vertex X of the 3D mesh is calculated as shown in the following equation (9). X and N polygons P1, P2, ..., P containing vertex X. N unit normal vector V Pn (where n=1, 2, …, N) and calculate the inner product of each, V P1 , V P2 ,…, V Pn Among them, at least one threshold θ i The following vertex X may be extracted:

[0063]

number

[0064] However, U X ·V Pn is U X and V Pn In this case, the dot product is U X and V Pn If the vector directions are the same, the value is 1, and if the vector directions are opposite, the value is -1. Therefore, the threshold value θ is set in the range from 1 to 0. i The smaller the setting, the X and V Pn This allows us to extract vertices X with a different vector direction from the normal vector of the polygon containing vertex X, thereby obtaining non-flat vertices such as corners and edges of structures. n unit normal vector V Pn is Polygon P n As shown in the following equation (10), the polygon P shown in FIG. n The two vectors a n ,b n Regarding the cross product a n ×b n The unit normal vector of vertex X is a vector obtained by averaging the normal vectors of the faces that include vertex X. As shown in the following formula (11), the unit normal vector of N polygons P1, P2, ..., P that include vertex X isN unit normal vector V P1 , V P2 ,…, V Pn is obtained by normalizing a linear combination of

[0065]

number

[0066] As a schematic example for explaining the processing contents of the above equations (9) to (11) in the mesh vertex reduction unit 43, the unit normal vector V P1 , V P2 ,…, V Pn and the unit normal vector U X An example of the relationship is shown below.

[0067] (12) The matching device 10 may further include a three-dimensional feature point restriction unit 44 at the location shown in FIG. 7 , and may restrict the three-dimensional feature points extracted by the mesh vertex extraction unit 11 based on the reference camera posture estimated by the reference camera posture estimation unit 12. As described above, various combinations are possible, and a three-dimensional mesh simplified by the mesh structure simplification unit 41 may be input, or three-dimensional feature points reduced by the mesh vertex reduction unit 43 or three-dimensional feature points extracted by the texture feature extraction unit 42 may be input. The three-dimensional feature points restricted by the three-dimensional feature point restriction unit 44 are output to the rendering image feature point conversion unit 21. Various embodiments of this three-dimensional feature point restriction unit 44 will be described below.

[0068] (12-1) The 3D feature point limiting unit 44 may limit the 3D feature points extracted by the mesh vertex extracting unit 11, the 3D feature points reduced by the mesh vertex reducing unit 43, or the 3D feature points extracted by the texture feature extracting unit, to other than occluded 3D feature points, starting from the reference camera posture acquired by the reference camera posture estimating unit 12. (In other words, it may limit to only 3D feature points that are not occluded.) That is, when the 3D feature points are converted into 2D feature points by the downstream rendering image feature point converting unit 21, there is a possibility that points occluded (hidden) by structures may also be converted into 2D feature points. However, these points are unlikely to be feature points that clearly represent the corners, edges, or textures of structures, which is the original purpose of using 3D vertices as feature points for the downstream feature point matching unit 30. Therefore, these feature points may not necessarily be feature points that are easy to match in this feature point matching unit 30. Therefore, it is possible to limit the 3D feature points to other than occluded 3D feature points.

[0069] Specifically, as shown in FIG. 11 , a 3D mesh object may be determined to be occluded if another 3D mesh object intersects with a line connecting the 3D feature points extracted by the mesh vertex extraction unit 11, mesh vertex reduction unit 43, and texture feature extraction unit 42 with the position of the reference camera posture determined by the reference camera posture estimation unit 12. To actually determine this, an existing method in the field of 3D CG may be used, such as ray_cast, an API (application programming interface) provided by a 3D CG software tool set such as Blender (product name). Furthermore, to exclude in advance 3D feature points located on the back side of an object as viewed from the reference camera position, the normal vector of a polygon, as illustrated in FIG. 9 , may be defined as the front-facing direction of the polygon (the direction of the object surface from the inside to the outside of the object), and the 3D vertices may be limited to those associated with at least one polygon that faces the front as viewed from the reference camera position.

[0070] (12-2) The 3D feature point limiting unit 44 may limit the 3D feature points extracted by the mesh vertex extraction unit 11, the 3D feature points reduced by the mesh vertex reduction unit 43, or the 3D feature points extracted by the texture feature extraction unit to only those 3D feature points whose distance from the approximate shooting position acquired by the reference camera posture estimation unit 12 is equal to or less than a threshold. In other words, even if a 3D feature point whose distance from the approximate shooting position estimated by the reference camera posture estimation unit 12 is equal to or greater than a threshold is a 3D feature point with a clear corner, edge, or texture of a structure, it is located far from the camera position used for rendering. Therefore, when the 3D mesh is rendered into a 2D image by the image rendering unit 13, the scale of the 3D structure becomes small, and the feature point may not necessarily be a feature point that is easy to match in the feature point matching unit 30 at the subsequent stage. Therefore, the 3D feature point limiting unit 44 may limit the 3D feature points whose distance from the approximate shooting position is equal to or less than a threshold. For example, as shown in the following equation (12), the world coordinates (X C ,Y C ,Z C ) and the world coordinates (X, Y, Z) of the three-dimensional feature points, the distance between which is less than or equal to a threshold value d. C ,Y C ,Z C ) can be calculated using the reference camera orientation (rotation matrix R and translation vector t) obtained by the reference camera orientation estimation unit 12, as shown in the following equation (13).

[0071]

number

[0072] (13) The matching device 10 of this embodiment, as a component technology of a VPS or the like, can contribute to improving the accuracy of image positioning when utilizing mesh maps while reducing the effort required for map creation, compared to when conventional methods are applied as is (such as the image matching methods described in Non-Patent Documents 2 and 3). This allows for smooth implementation of applications such as a worker at a remote site taking an image of an object to be observed and sending the image to an instructor over the network, who can then use the image to reconstruct the position of a 3D model of the remote site in computer graphics and provide advice to the worker. This eliminates the need for the instructor to travel to the remote site, and reduces carbon dioxide emissions by saving the energy resources required for user travel, thereby contributing to Goal 13 of the United Nations' Sustainable Development Goals (SDGs), which states, "Take urgent action to combat climate change and its impacts."

[0073] (14) FIG. 12 is a diagram showing an example of the hardware configuration of a general computer device 70. The matching device 10 can be realized as one or more computer devices 70 having such a configuration. When the matching device 10 is realized using two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71, a RAM 73 as a main memory device that provides a work area for the CPU 71 (and GPU 72), a ROM 74 as an auxiliary memory device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, and a bus BS for transmitting and receiving data among them.

[0074] Each functional unit of the matching device 10 can be realized by a CPU 71 and / or a GPU 72 that reads from a ROM 74 and executes a predetermined program corresponding to the function of each unit. Both the CPU 71 and the GPU 72 are types of arithmetic devices (processors). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and the GPU 72. Processing results by the matching device 10 may be displayed and output on the display 76. An image captured by a camera 78 may be used as a captured image used as one of the inputs to the matching device 10. [Explanation of symbols]

[0075] 10...Matching device, 11...Mesh vertex extraction unit, 12...Reference camera posture estimation unit, 13...Image rendering unit, 21...Rendered image feature point conversion unit, 22...Rendered image feature amount description unit, 23...Captured image feature point detection unit, 24...Captured image feature amount description unit, 30...Feature point matching unit, 41...Mesh structure simplification unit, 42...Texture feature extraction unit, 43...Mesh vertex reduction unit, 44...3D feature point restriction unit

Claims

1. A matching device that matches features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount; As a process of adding a first feature point used to extract the first feature amount, A matching device characterized in that three-dimensional feature points are extracted from a texture of a surface formed by three-dimensional vertices in the three-dimensional mesh when viewed from the front, and the points at which the three-dimensional feature points are rendered on the rendering image are used as the points to be added to the first feature points.

2. 2. The matching device according to claim 1, wherein in the adding process, the texture from which the three-dimensional feature points are extracted relates to a unique pattern or design, which contributes to ensuring matching accuracy using the extracted feature points.

3. A matching device that matches features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount; As a process of reducing the first feature points used to extract the first feature amount, A matching device characterized in that the first feature points are obtained from only a portion of the three-dimensional vertices of the three-dimensional mesh that are determined to be at the top of a sharp convex shape in relation to the surface, out of all the three-dimensional vertices of the three-dimensional mesh.

4. A matching device that matches features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount; As a process of reducing the first feature points used to extract the first feature amount, A matching device characterized in that the first feature points are obtained from only a portion of the three-dimensional vertices of the three-dimensional mesh that are determined to be close in distance from the camera position in the camera orientation.

5. 5. A matching device according to claim 1, wherein in the process of matching the first feature and the second feature, matching candidates are limited to feature values ​​for which the position of the first feature point in the rendering image and the position of the second feature point in the captured image are determined to be in the vicinity of each other.

6. The matching device according to claim 5, characterized in that a similarity determination is performed between a divided area in the rendering image and a divided area in the captured image, and a first feature point and a second feature point belonging to the divided area determined to be similar are treated as being determined to have positions that are nearby.

7. A matching device as described in any one of claims 1 to 4, characterized in that when extracting the first feature, the structure of the three-dimensional mesh is simplified in advance so as to uniformize the spatial distribution of the three-dimensional vertices of the three-dimensional mesh, thereby reducing the number of three-dimensional vertices compared to before the simplification.

8. As a process of reducing the first feature points used to extract the first feature amount, A matching device as described in any one of claims 1 to 4, characterized in that the first feature points are obtained from only a portion of the three-dimensional vertices of the three-dimensional mesh that are determined to be not occluded from the positional relationship of the structure corresponding to the three-dimensional mesh at the camera attitude, out of all the three-dimensional vertices of the three-dimensional mesh.

9. A method executed by a computer for matching features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, the method comprising: a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount, As a process of adding a first feature point used to extract the first feature amount, A method characterized by extracting three-dimensional feature points from a texture of a surface formed by three-dimensional vertices in the three-dimensional mesh as viewed from the front, and using the points at which the three-dimensional feature points are rendered on the rendering image as the points to be added to the first feature points.

10. The method according to claim 9, wherein in the adding process, the texture from which the three-dimensional feature points are extracted is related to a unique pattern or design, which contributes to ensuring matching accuracy using the extracted feature points.

11. A method executed by a computer for matching features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, the method comprising: a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount, As a process of reducing the first feature points used to extract the first feature amount, A method characterized by obtaining the first feature points from only a portion of the three-dimensional vertices of the three-dimensional mesh that are determined to be at the apex of a sharp convex shape in relation to the surface, out of all the three-dimensional vertices of the three-dimensional mesh.

12. A method executed by a computer for matching features between a three-dimensional mesh formed by information on three-dimensional vertices and textures of surfaces formed by the three-dimensional vertices and a captured image, the method comprising: a process of rendering the three-dimensional mesh at a reference camera posture that is an estimated posture of a camera that captured the captured image to obtain a rendered image, and extracting first feature amounts from the first feature points that are all or part of two-dimensional positions corresponding to three-dimensional vertices of the three-dimensional mesh from the rendered image; a process of detecting second feature points from the captured image and extracting second feature amounts from the second feature points; a process of matching the first feature amount with the second feature amount, As a process of reducing the first feature points used to extract the first feature amount, A method characterized by obtaining the first feature point from only a portion of the three-dimensional vertices of the three-dimensional mesh that are determined to be close to the camera position in the camera pose.

13. 5. A program that causes a computer to function as the matching device according to claim 1.

Images