Image feature extraction model and method using image features under low resolution, and image matching device and program

Abstract

To provide an image feature amount extraction model which extracts an image feature amount by which image matching processing corresponding to even the deviation of the resolution between images can be executed.SOLUTION: A method causes a computer to function as: means which generates a low-resolution key point and a low-resolution descriptor in each low-resolution stage for each of two original images; means which executes matching processing for each low-resolution stage in each image and generates a pair of the matching low-resolution key points; means which executes up-sampling processing to the matching low-resolution key point for each low-resolution stage in each image, generates and outputs an original resolution corresponding key point; and means which generates an original image corresponding image feature amount using a low-resolution image entire feature amount in each low-resolution stage or a low-resolution corresponding image feature amount generated from an image entire feature amount, and generates and outputs an original resolution corresponding descriptor DS on the basis of the original resolution corresponding key point from the original image corresponding image feature amount.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image matching technology and an image feature extraction technology that is the core of the technology. [Background technology]

[0002] Image feature extraction and image matching are important techniques for selecting and generating visual artifacts from images, such as image retrieval, visual search, and three-dimensional (3D) image construction. In many cases, methods have been proposed that first extract keypoints and descriptors as image features from the images to be matched, and then perform keypoint matching using these extracted features.

[0003] Regarding image feature extraction, for example, Patent Document 1 discloses a technique for extracting keypoints using a plurality of images (octaves) with different resolutions forming a Gaussian pyramid. Patent Document 2 discloses a technique for extracting keypoint candidates from corner regions and then determining the keypoints and descriptors.

[0004] Furthermore, Non-Patent Document 1 discloses a technique for using a neural network backbone to extract key points along lines of an image, which are low-level features, to perform line detection, and then to generate a high-level feature, a descriptor corresponding to the detected line. After this, line matching is performed using the detected line and the generated descriptor.

[0005] In addition, Non-Patent Document 2 discloses a technology that uses a multi-scale pyramid filter that combines a Convolutional Neural Network (CNN) for handcrafted features designed based on human knowledge with a CNN for machine learning-based features to extract keypoints at different resolutions, and then determines the final keypoints.

[0006] As explained above, in many conventional studies, image feature extraction and image matching are independent processes. On the other hand, some techniques have been proposed that treat image feature extraction and image matching as a single task.

[0007] For example, in Non-Patent Document 3, a confidence heatmap is generated as image features under low resolution from input query keypoints, and correspondence keypoints under high resolution are determined using this heatmap. Here, the correspondence keypoints are keypoints that match the query keypoints in the other image (matching partner).

[0008] In addition, Non-Patent Document 4 discloses a technology for image matching by determining matched pixel pairs without keypoint detection through semi-dense matching, which performs pixel-by-pixel matching between images. Specifically, first, local features are extracted from images using a neural network backbone, then dense matching with multi-heads attention is performed using a low-resolution compatible transformer, thereby determining matched pixel pairs at low resolution, and then highly accurate matched keypoints are determined from the matched pixel pairs using a high-resolution compatible transformer.

[0009] Furthermore, Non-Patent Document 5 discloses a machine learning-based image registration algorithm at each stage from low to high resolution that can handle multi-modality images (images with various image ranges, resolutions, etc.). In this algorithm, a trained BEL detector is first used to extract keypoints at multiple low resolution stages, and then these keypoints are matched using a trained classifier in which boosting nodes corresponding to each low resolution stage are successively provided. Next, after learning the degree of agreement between the keypoints at the high resolution stage and given landmark points, keypoint matching at high resolution is performed using an affine transformer whose learning parameters have been trained using the RANSAC (RANdam SAmple Concensus) algorithm. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] US Patent Application Publication No. 2012 / 0027290 [Patent Document 2] US Patent Application Publication No. 2016 / 0292887 [Non-patent literature]

[0011] [Non-Patent Document 1] Remi Pautrat, Juan-Ting Lin, Viktor Larsson, Martin R. Oswald, and Marc Pollefeys, “SOLD2: Self-supervised Occlusion-aware Line Description and Detection”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 2104.03362.pdf> , 2021 [Non-Patent Document 2] Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, Krystian Mikolajczyk, “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”, International Conference on Computer Vision,<https: / / arxiv.org / pdf / 1904.00889.pdf> , 2019 [Non-Patent Document 3] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, Noah Snavely, “Learning Feature Descriptors using Camera Pose Supervision”, European Conference on Computer Vision,<https: / / arxiv.org / pdf / 2004.13324.pdf> , 2020 [Non-Patent Document 4] Jiaming Sun, Zehong Shen, Yu'ang Wang, Hujun Bao, and Xiaowei Zhou, “LoFTR: Detector-Free Local Feature Matching with Transformers”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, <https: / / arxiv.org / pdf / 2104.00680.pdf>, 2021 [Non-Patent Document 5] Jiayan Jiang, Songfeng Zheng, Arthur W. Toga, and Zhuowen Tu, “Learning based coarse-to-fine image registration”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, <https: / / doi.org / 10.1109 / CVPR.2008.<4587396>, 2008 [Summary of the Invention] [Problems to be Solved by the Invention]

[0012] However, in conventional image feature extraction technologies and image matching technologies, for example, when one of the images to be matched is a very high-resolution image taken by a smartphone-mounted camera and the other is a relatively low-resolution image rendered (drawn) from a 3D model related to, for example, the 3D (three-dimensional) urban model open data project led by the Ministry of Land, Infrastructure, Transport and Tourism (Plateau, <https: / / www.mlit.go.jp / plateau / >), etc., it is often difficult to perform image matching with sufficient accuracy.

[0013] Specifically, the two images described above usually have significantly different resolutions in the texture region, and the resolution of a rendering image of a 3D model is not constant even within the image. As a result, in the conventional image feature extraction technology and image matching technology, it is not rare that many of the determined matched pairs are actually unmatched pairs, resulting in low accuracy results. Incidentally, as a conventional example of such a low accuracy result, FIG. 5(A) shows an image matching result (conventional example 1) using SIFT (Scale-Invariant Feature Transform), which is a typical image feature.

[0014] It is generally known that a more accurate image matching result can be obtained by searching for keypoints based on low-resolution image features corresponding to the lines and shapes of objects contained in the images. This is because such low-resolution image features tend to reflect the degree of geometric matching. However, in searching for keypoints based on low-resolution image features, it is difficult to obtain a sufficient number of keypoints for image matching, as in the conventional example 2 shown in FIG. 5(B).

[0015] Furthermore, for example, in a rendering image of a 3D model of a city with buildings, the general shapes of the buildings usually roughly match or at least resemble their actual shapes. However, the detailed structure of the buildings, for example the shape of the window frames, often differs significantly from the actual ones. Incidentally, in the above-mentioned 3D city model, there may be cases where buildings incidental to a certain building (for example, next to it) are not digitized, which also has a significant impact on the image matching results.

[0016] Fig. 5(C) shows the conventional example 3 in which the detailed structures and incidental objects differ between the images to be matched. Fig. 5(C) shows that the detailed structures of the corresponding buildings in both images and the incidental structures at the base of the buildings are significantly different. This is because the detailed structures and incidental structures are not captured in the 3D model that is the basis of the 3D rendering image.

[0017] In view of the problems with the conventional technology as described above, it is understood that it is extremely important to extract keypoints that can handle the difference in resolution between images, to find more keypoints in addition to line- and shape-based keypoints, and to extract image features that can also handle the detailed structure of a building and ancillary buildings.

[0018] However, in the conventional image feature extraction techniques and image matching techniques described above, for example, the techniques disclosed in Patent Documents 1 and 2 and Non-Patent Documents 2, 3 and 4, initially, processing is performed using image features at low resolution only, and it is far from being able to solve the problems described above.

[0019] Here, the technology disclosed in Non-Patent Document 4 does indeed generate keypoints at high resolution using matched pixel pairs at low resolution. However, the dense matching used here requires huge calculation costs, so it only handles one low resolution image, and furthermore, the information generated at low resolution in this way is not used for image matching at all.

[0020] On the other hand, the techniques disclosed in Non-Patent Documents 1 and 5 certainly perform image matching using both line-based image features at low resolution and information generated at low resolution. Here, in the technique of Non-Patent Document 1, the keypoints are only line- and edge-based at low resolution, and the number of keypoints is limited. In contrast, the technique of Non-Patent Document 5 detects and matches keypoints at multiple low resolution stages, and then performs upsampling processing (processing to increase the number by interpolation) on these keypoints before performing keypoint matching at high resolution.

[0021] However, even in the technology disclosed in Non-Patent Document 5, there is no attempt to extract a high-resolution descriptor or re-detect a keypoint. Furthermore, keypoint matching is performed only using a geometric transformer, and no attempt is made to improve the matching accuracy at high resolution using, for example, visual similarity. As a result, it is impossible to find image features that can correspond to, for example, the detailed structure of a building or incidental (for example, side) buildings.

[0022] Therefore, the present invention aims to provide an image matching device and an image matching program capable of performing image matching processing that can handle differences in resolution between images, generate more key points, and take into consideration more details and additional information of objects in images, as well as an image feature extraction model and an image feature extraction method capable of extracting image features that enable such image matching processing. [Means for solving the problem]

[0023] According to the present invention, there is provided an image feature extraction model for causing a computer to extract key points and descriptors from each of two original images, the model comprising: a low-resolution feature generating means for generating, for each of the images, low-resolution keypoints and low-resolution descriptors in at least one low-resolution step lower than an original resolution that is a resolution of the original image, from the extracted features of the entire image; For each low resolution step, Former low-resolution keypoint matching means for performing a matching process of the low-resolution keypoints between images to generate pairs of matched low-resolution keypoints; an original resolution corresponding keypoint extraction means for performing upsampling processing on the matched low-resolution keypoints for each of the images at each low-resolution stage, generating original resolution corresponding keypoints corresponding to the original resolution, the number of which is greater than the number of the low-resolution keypoints, and outputting the generated keypoints as an extraction result; For each image, low-resolution image overall feature values ​​at each low-resolution stage are generated from the extracted image overall feature values. or by performing keypoint detection processing on the low-resolution image overall feature amount, At each low resolution stage Okeru an original resolution corresponding descriptor extraction means for generating an original image corresponding image feature corresponding to the original image using the low resolution corresponding image feature, generating an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image feature based on the original resolution corresponding key points, and outputting the generated descriptor as an extraction result; An image feature extraction model that functions as a computer is provided.

[0024] As an embodiment of the image feature extraction model according to the present invention, the low-resolution keypoint matching means includes: For each image, receive updated original resolution corresponding keypoints and updated original resolution corresponding descriptors, the updated original resolution corresponding keypoints and original resolution corresponding descriptors being output and externally updated; For each of the images, for each of the low-resolution stages, a pooling process is performed on the updated original resolution corresponding keypoints that have been re-scaled to the low-resolution stage and the updated original resolution corresponding descriptors to generate low-resolution keypoints and low-resolution descriptors that correspond to the low-resolution stage; For each image, for each resolution step, the resolution reduction descriptor for that resolution step and the previously generated resolution descriptor for the entire image are of merging the feature vector with a descriptor generated based on the reduced resolution keypoints to generate an updated reduced resolution descriptor; For each image, for each low-resolution stage, the low-resolution keypoint and the previously generated low-resolution keypoint are combined to form a new low-resolution keypoint, and the updated low-resolution descriptor and the previously generated low-resolution descriptor are combined to form a new low-resolution descriptor. Former A matching process is performed for the new low-resolution keypoints between the images to generate a pair of matched low-resolution keypoints. It is also preferable.

[0025] In addition, the image feature extraction model according to the present invention is (a) a process in which a low-resolution keypoint matching means receives the updated original resolution corresponding keypoints and the updated original resolution corresponding descriptors and generates a pair of matched low-resolution keypoints; (b) a process in which the original resolution corresponding keypoint extraction means generates the original resolution corresponding keypoints and outputs them as an extraction result; and (c) A process in which the original resolution corresponding descriptor extraction means generates the original resolution corresponding descriptor and outputs it as an extraction result. It is also preferable to execute the above set of steps a plurality of times and to cause the computer to function so as to output the finally generated original resolution corresponding keypoints and original resolution corresponding descriptors as extraction results.

[0026] Furthermore, as another embodiment of the image feature extraction model according to the present invention, it is also preferable that the low-resolution feature generation means generates, for each of the images, for each low-resolution stage, low-resolution keypoints and low-resolution image whole features at that low-resolution stage from the features of the whole image, and generates the low-resolution descriptor from the low-resolution keypoints based on the whole low-resolution image features.

[0027] In still another embodiment of the image feature extraction model according to the present invention, the original resolution corresponding keypoint extraction means preferably performs an unpooling process after rescaling the matched low resolution keypoints to the original resolution for each of the images for each low resolution stage, generates original resolution corresponding keypoints corresponding to the original resolution which are greater in number than the low resolution keypoints at that low resolution stage, consolidates the original resolution corresponding keypoints at each low resolution stage, and outputs them as the original resolution corresponding keypoints.

[0028] In addition, as a first embodiment, the original resolution corresponding descriptor extraction means of the present invention performs a pypooling process on the overall image features for each image to generate overall low-resolution image features at each low resolution stage, performs an upsampling process on the overall low-resolution image features that have been subjected to the convolution process at each low resolution stage, and further generates image features corresponding to the original image by combining them with the overall image features, and generates an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image features based on the original resolution corresponding keypoints.

[0029] Furthermore, as a second embodiment, the original resolution corresponding descriptor extraction means of the present invention is also preferably a process of merging the low resolution corresponding image features for each image, in which the result of the merging process at one low resolution stage is subjected to a deconvolution process and then merged with the low resolution corresponding image features, proceeding sequentially from lowest resolution to the original resolution stage to generate the original image corresponding image features, and generating an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image features based on the original resolution corresponding keypoints.

[0030] According to the present invention, there is also provided an image feature extraction means for extracting, from each of the two original images, key points corresponding to the original resolution and descriptors corresponding to the original resolution, which are image features of the original images, by using the image feature extraction model described above; an image matching means for matching the two original images using the extracted original resolution keypoints and the original resolution descriptors; An image matching apparatus is provided having:

[0031] According to the present invention, there is further provided an image feature extraction means for extracting, from each of the two original images, image feature values ​​of the original image, key points corresponding to the original resolution and descriptors corresponding to the original resolution, using the image feature extraction model described above; an image matching means for matching the two original images using the extracted original resolution keypoints and the original resolution descriptors; An image matching program is provided that causes a computer to function as:

[0032] In one embodiment of the image matching program according to the present invention, the image matching means includes: a graph generating means for generating, for each of the images, a graph representing the original image using the original resolution corresponding keypoints; graph attention means for performing a graph attention process using the graph on the original resolution corresponding descriptor for each of the images to generate an updated original resolution corresponding descriptor; a keypoint matching means for performing matching processing of the original resolution-corresponding keypoints using the updated original resolution-corresponding descriptor; It is also preferred that the compound has the following structure:

[0033] The present invention also provides a computer-implemented image feature extraction method for extracting keypoints and descriptors from each image of two source images, the method comprising the steps of: generating, for each of the images, low-resolution keypoints and low-resolution descriptors at at least one low-resolution level lower than the original resolution of the original image from the extracted features of the entire image; For each low resolution step, Former performing a matching process of the low-resolution keypoints between the images to generate a pair of matched low-resolution keypoints; performing upsampling on the matched low-resolution keypoints for each low-resolution stage of the image to generate original resolution corresponding keypoints corresponding to the original resolution, the number of which is greater than the number of the low-resolution keypoints, and outputting the generated keypoints as extraction results; For each image, a feature vector is generated from the extracted image features. 、 Using the overall features of the low-resolution image at each low-resolution stage, or By performing keypoint detection processing on the overall feature amount of the low-resolution image, Generated , at each low resolution stage generating an original image corresponding image feature corresponding to the original image using the low resolution corresponding image feature, generating an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image feature based on the original resolution corresponding keypoints, and outputting the generated descriptor as an extraction result; An image feature extraction method is provided, comprising: Effect of the Invention

[0034] According to the image matching device and the image matching program of the present invention, it is possible to carry out an image matching process that can handle the difference in resolution between images, generate more key points, and take into consideration more details and additional information of objects in the images. Also, according to the image feature extraction model and the image feature extraction method of the present invention, it is possible to extract image features that enable such an image matching process to be carried out. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a schematic diagram showing an embodiment of an image feature extraction model according to the present invention. [Diagram 2] 1A to 1C are schematic diagrams showing examples in which various processes according to the present invention are applied to a smartphone camera image and a 3D rendering image. [Diagram 3] 1 is a flowchart showing an outline of an embodiment of an image feature extraction method according to the present invention. [Figure 4] 1 is a functional block diagram showing an embodiment of an image matching device 9 according to the present invention. [Diagram 5] FIG. 1 is a schematic diagram showing a conventional example of an image feature extraction technique and an image matching technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0037] [Image feature extraction model] FIG. 1 is a schematic diagram showing an embodiment of an image feature extraction model according to the present invention.

[0038] The image feature extraction model 1 of this embodiment shown in FIG. (a) Image data of original image A and original image B to be subjected to image matching are input, (b) A set of key points and a set of descriptors of an original image A, the set of key points and a set of descriptors corresponding to the resolution (original resolution) of the original image A (K Fin , F Fin ) A and a set of key points and a set of descriptors of the original image B, which correspond to the resolution (original resolution) of the original image B (K Fin , F Fin ) B and output It is a machine learning model.

[0039] Here, a keypoint is a part of an image that is judged to be characteristic (a part that is important for image classification) when the target image is viewed from a set, predetermined viewpoint. A descriptor is a feature extracted from the image area around a keypoint, and specifically, is a high-dimensional vector that describes the semantic information of the keypoint. These keypoints and descriptors are important image features for image matching.

[0040] In order to realize the image feature processing described above, the image feature extraction model 1 specifically has the following features: (A) For each of the original images A and B, from the extracted “feature of the entire image” D0, “low-resolution keypoints” K at least one (L) low-resolution step lower than the original resolution, which is the resolution of the original image, are extracted. 1 C ~K L C and "low-resolution descriptor" F 1 C ~F L C A low-resolution feature generator 11 that generates a low-resolution feature (B) For each of the above low-resolution steps, a matching process of "low-resolution keypoints" is performed between the original images A and B, and a pair of matched "low-resolution keypoints" M 1 C ~M L C A low-resolution keypoint (KP) matching unit 12 for generating a low-resolution keypoint (KP) (C) For each of the original images A and B, an upsampling process is performed on the matched "low-resolution keypoints" at each of the above low-resolution steps, and "original resolution-corresponding keypoints" K corresponding to the original resolution, which are greater than the "low-resolution keypoints", are generated. Fin and an original resolution corresponding key point (KP) extraction unit 13 that generates and outputs the extracted result. (D) For each of the original images A and B, (d1) the “low-resolution image overall feature quantity” D at each low-resolution stage is generated from the extracted “image overall feature quantity” D0″. i'' or (d2) generate ``low-resolution corresponding image features'' D from the extracted ``whole image features'' D0'' i C '' to obtain the "image feature value corresponding to the original image" D Fin and generate "image features corresponding to the original image" D Fin From "Original Resolution Key Points" K Fin Based on this, the "original resolution correspondence descriptor" F Fin and an original resolution corresponding descriptor (DS) extraction unit 14 that generates and outputs the extracted result. to make the computer function.

[0041] In this way, in the image feature extraction model 1, the low-resolution KP matching unit 12 (B) performs keypoint matching for each low-resolution stage (which generally produces more accurate image matching results), and the original resolution corresponding KP extraction unit 13 (C) uses the matching results to extract the “original resolution corresponding keypoints” K Fin As a result, even if the resolutions of original images A and B are significantly different, it is possible to generate a group of key points that will enable a highly accurate image matching process to be performed later.

[0042] An example of a case where the resolutions are different is when original image A is an extremely high-resolution image captured by a smartphone camera, and original image B is a relatively low-resolution image rendered (drawn) from a 3D model. In fact, such a case is commonly seen in various fields where image matching processing is performed.

[0043] In addition, in the image feature extraction model 1, the original resolution corresponding KP extraction unit 13 (C) performs upsampling processing on the matched “low-resolution keypoints” acquired from the low-resolution KP matching unit 12 (B) above, and extracts “original resolution corresponding keypoints” K that are greater than the “low-resolution keypoints”. FinAs a result, a larger number of keypoints can be used for subsequent image matching with higher accuracy.

[0044] Furthermore, in the above (D) original resolution corresponding DS extraction unit 14, the “original image corresponding image feature quantity” D Fin From the (more appropriate and larger number of) "original resolution-compatible keypoints" K Fin Based on this, the "original resolution correspondence descriptor" F Fin This "original resolution descriptor" F Fin Therefore, it is an image feature that has higher descriptiveness (the degree to which it accurately describes the characteristics of an image without missing anything) and can more reliably describe, for example, the details of objects in an image and additional information.

[0045] Therefore, for example, if original image A is a very high-resolution image generated by taking a picture of a street lined with buildings with a smartphone camera, and original image B is a relatively low-resolution image rendered (drawn) from a 3D model of the same street, even if the detailed structure of the buildings (e.g., the shape of the window frames) and the presence or absence of buildings incidental to the buildings (e.g., next to them) are significantly different in both images, this highly descriptive "original resolution correspondence descriptor" F Fin By using the above, it is possible to perform image matching processing with higher accuracy.

[0046] As described above, the image feature extraction model 1 can handle the difference in resolution between images, generate more key points, and perform image matching processing that can take into account more details and additional information of objects in the images. Fin , F Fin ) can be extracted.

[0047] In this specification, the subscripted K for keypoints and the subscripted F for descriptors represent, more precisely, a set of keypoints k (K={k}) and a set of descriptors d (F={d}), respectively. However, for the sake of simplicity, it should be noted that in the following description, they are often simply referred to as keypoints and descriptors.

[0048] [Model configuration] The configuration of the image feature extraction model 1 of this embodiment will be described in more detail below. Also referring to FIG. 1, in this embodiment, the image feature extraction model 1 is as follows: (A) A low-resolution feature generation unit 11 having a CNN unit 11a, a low KP generation unit (11-1a, . . . , 11-La) and a low DS generation unit (11-1b, . . . , 11-Lb); (a) a low-resolution KP matching unit 12 having a downsampling unit 12a, a merging unit 12b, and a KP matching unit 12c; (c) an original resolution corresponding KP extraction unit 13 having an upsampling unit 13a and an aggregation unit 13b; (D) An original resolution corresponding DS extraction unit 14 having a ResNet unit 14a, low image feature generation units (14-1, . . . , 14-L), and an original resolution corresponding DS generation unit 14b; These are functional components embodied by a program installed in the computer. In other words, these functional components are a model that makes the computer function. Each of the above-mentioned functional components will be specifically described below.

[0049] <Means for generating low-resolution features> As also shown in FIG. 1, in the low-resolution feature generator 11 of this embodiment, (A1) The CNN unit 11a extracts the whole image feature quantity D0 from each image data I of the original images A and B, (A2) The low KP generation unit (11-1a, . . . , 11-La) generates low-resolution keypoints (KPs) at each of L low-resolution stages (L≧2 in this embodiment) lower than the original resolution, which is the resolution of the original images (A, B), from the whole image feature quantity D0 for each of the original images A and B. 1 C , ··, K L C ) and low-resolution image total feature (D 1 C , , D L C ), (A3) The low-resolution image feature quantity (D 1 C , , D L C ) to obtain the low-resolution keypoints (K 1 C , ··, K L C ) based on the low-resolution descriptor (F 1 C , , F L C )

[0050] Hereinafter, the above processes (A1) to (A3) will be described in more detail. First, in this embodiment, the CNN unit 11a of (A1) above calculates image data I (I (A) or I (B) ) to obtain the following equation: (1) D0=CNN BB (I) The whole image feature quantity D0 is calculated by the above.

[0051] Here, CNN BB (·) is an operator that represents processing in a known CNN (Convolutional Neural Network) backbone, and is the image space (R H×W×3 ) to H 0 C ( <H)×W0 C ( <W)×N F The feature space (R H0C×W0C×NF ) where the (pixel of the keypoint) resolution H 0 C ×W 0 C is a value (spatial dimensionality) smaller than the resolution H × W.

[0052] Next, the low KP generation unit 11-(i)a of the i-th (1≦i≦L)-th low-resolution stage in (a2) calculates the following equation from the total image feature quantity D0 of each of the original images A and B: (2) K i C , D i C =Det i (D0) Using the i-th low-resolution stage, we estimate the low-resolution keypoints K i C (1≦i≦L) and the low-resolution image feature vector D i C (1≦i≦L) is calculated. Here, Det i (·) is a keypoint detection operator that performs keypoint detection on an image (with a resolution equivalent to the i-th low-resolution stage). i For example, a line-based detector as disclosed in Non-Patent Document 1 may be used as (·), and various other known keypoint detection operators such as a shape-based detector can also be used.

[0053] In addition, the low-resolution image total feature value D calculated by the above formula (2) i C is H i C ×W i C ×N i C (Number of descriptor dimensions) dimensional low-resolution image space (R HiC×WiC×NiC ) and are so-called dense descriptors. Furthermore, the low-resolution keypoints Ki C is the image coordinate system R 2 is the set of points in K i C ={(x, y)∈R 2}

[0054] Next, the low-resolution DS generating unit 11-(i)b of the i-th (1≦i≦L)-th low-resolution stage in (A3) generates the low-resolution image feature vector D i C From the low-resolution keypoints K i C Based on this, the following equation (3) F i C =D i C (x,y,:) (x,y)∈KiC Using the i-th low-resolution stage, the low-resolution descriptor F i C (1≦i≦L). The above formula (3) calculates the low-resolution descriptor F i C However, the low-resolution image feature vector D i C Among the image features (descriptors) that make up i C In other words, the low-resolution descriptor F i C is the low-resolution keypoint K i C Based on the spatial position defined by, the low-resolution image full feature D i C A descriptor(s) selected from (the set of descriptors constituting the low-resolution keypoint K i C Number of images in (density, cardinality)|K i C | i Then, k i ×N i C dimensional low-resolution descriptor space (R ki×NiC) is a descriptor (group) that belongs to the

[0055] As described above, the low-resolution feature generator 11 generates pairs of low-resolution keypoints and low-resolution descriptors at each low-resolution stage (1, 2, . . . , L) in each of the original images A and B through the processes in (A1) to (A3) above, i.e. (4) {(K 1 C , F 1 C ), (K 2 C , F 2 C ), ··, (K L C , F L C )} (K i C , F i C )={(k, d)|k∈K i C , d∈F i C} In this way, the low-resolution feature generator 11 generates the pair (K i C , F i C ) is output as is.

[0056] In this regard, the prior art, for example, Non-Patent Documents 1 and 2, and Non-Patent Document: Zixin Luo, et al., “ASLFeat: Learning Local Features of Accurate Shape and Localization”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 2003.10071.pdf> The method disclosed in

[2020] generates pairs of low-resolution keypoints and low-resolution descriptors, but does not provide any support for subsequently performing low-resolution keypoint matching processing for each low-resolution stage.

[0057] Here, the pair (K i C , F i C ) resolution is H i C ×W i C However, this resolution is the CNN of the above formula (1). BB Depending on the backbone used as (·), it is determined whether the result is highest when i=1 (lowest when i=L) or vice versa. For example, CNN BB If we use an autoencoder as (·), the result will be lowest when i=1 and highest when i=L.

[0058] The low-resolution feature generation process of this embodiment has been described above. FIG. 2A shows an example in which keypoint detection in this process is performed on a smartphone camera image and a 3D rendering image.

[0059] In this embodiment, Det iAs (·), we use the line-based detector disclosed in Non-Patent Document 1. It has been confirmed that by performing line-based detection with high expected accuracy at low resolution, it is possible to obtain keypoints that match with higher accuracy. In other words, keypoint detection that compensates for the large difference in resolution between smartphone camera images and 3D rendering images has been realized.

[0060] <Low resolution KP matching method> As also shown in FIG. 1, the low-resolution KP matching unit 12 of this embodiment performs the following for each of the original images A and B: (*) Updated original resolution corresponding key points K, which are original resolution corresponding key points and original resolution corresponding descriptors updated outside the model (output from the original resolution corresponding DS extraction unit 14 described later). Fin and the updated original resolution correspondence descriptor F Fin Then, by performing the steps (i) to (i) described below, we obtain the matched low-resolution keypoint pairs M 1 C ~M L C Generate.

[0061] Here, the updated original resolution keypoint K Fin and the updated original resolution correspondence descriptor F Fin In this embodiment, these are image features obtained as a result of key point matching processing in the image matching unit 93 (FIG. 4), which will be described later in detail. Therefore, at the stage when the original resolution corresponding DS extraction unit 14 has not yet output the extraction result to the image matching unit 93, that is, at the initial stage of the low-resolution KP matching processing, the following processes (A1) and (A2) are omitted, and the low-resolution KP matching processing is performed starting from process (A3).

[0062] In another embodiment, the updated original resolution keypoints K Fin and the updated original resolution correspondence descriptor F FinIt is also possible to perform a low-resolution KP matching process that does not use the KP matching algorithm. In this case, the calculation cost can be reduced.

[0063] As also shown in FIG. 1, in the low-resolution KP matching process of this embodiment, specifically, (a) The downsampling unit 12a down-samples each of the original images A and B to generate updated original resolution keypoints K that have been rescaled to the corresponding low resolution stage (1, 2, . . . , L). Fin (In the following, i F→C_LOC ) and the updated original resolution correspondence descriptor F Fin Then, the pooling process is performed on the low-resolution keypoints K i F→C and low-resolution descriptor F i F→C Generate (a2) The merging unit 12b performs the following for each of the original images A and B: (a) For each low-resolution stage (1, 2, . . . , L), a low-resolution descriptor F i F→C and the total image feature quantity D0 generated earlier (the total low-resolution image feature quantity D i C ) to the low-resolution keypoint K i F→C The generated descriptor F i C_F→C and the updated low-resolution descriptor F i F→C _M, and then use this updated low-resolution descriptor F i F→C _M and the previously generated low-resolution descriptor F i C Combined with this, we create a new low-resolution descriptor F i C ' and also, (b) For each low-resolution step (1, 2, . . . , L), the low-resolution keypoints K i F→Cand the previously generated low-resolution keypoints K i C Combined with this, we create a new low-resolution keypoint K i C 'year, (a3) The KP matching unit 12c generates a new low-resolution descriptor F between the original images A and B for each low-resolution step (1, 2, . . . , L). i C ' to find new low-resolution keypoints K i C ' and find the matched low-resolution keypoint pairs {M i C} is generated.

[0064] Hereinafter, the above processes (a1) to (a3) ​​will be described in more detail. First, in this embodiment, the downsampling unit 12a of (a1) above calculates the received updated original resolution corresponding keypoints K Fin and the updated original resolution correspondence descriptor F Fin From this, the following equation (5) K i F→C , F i F→C =Pool i (K i F→C_LOC , F Fin ) where K i F→C_LOC =Re i Down (K Fin ) Using this, we reduce the resolution of the keypoints K i F→C , and the low-resolution descriptor F i F→C Generate.

[0065] In the above equation (5), Re i Down (·) is an operator that rescales the position of the keypoint corresponding to the original resolution to the position corresponding to the low resolution stage (i). i(·,·) is a pooling operator that pools the target keypoints into a super node (when multiple keypoints are considered to be roughly in the same position).

[0066] As explained above, the low-resolution keypoints K i F→C In this case, the number (cardinality)|K i F→C | is the K generated by the rescaling process. i F→C_LOC Number of (cardinality, cardinality)|K i F→C_LOC |The following is true (|K i F→C |≦|K i F→C_LOC On the other hand, the low-resolution descriptor F i F→C (∈R Ki×NiF→C ) with the number of dimensions N i F→C is the low-resolution descriptor F i C The number of dimensions of N i C or the number of dimensions N depending on the operator used in the next merge step. i C It is possible to set the value to a different value.

[0067] In any case, the reduced resolution keypoints K i F→C and low-resolution descriptor F i F→C can then be used to find better keypoints and increase the number of keypoints in the original resolution (upsampling).

[0068] Next, in this embodiment, the merging unit 12b in (a2) above merges the low-resolution image total feature value D generated from the previously generated image total feature value D0 for each low-resolution stage (1, 2, . . . , L) for each of the original images A and B. i C From the low-resolution keypoint K i F→C Based on this, the following equation (6) F i C_F→C =D i C (x,y,:) (x,y)∈KiF→C Using the i-th low-resolution stage, the descriptor F i C_F→C (1≦i≦L). The above formula (6) calculates the descriptor F i C_F→C However, the low-resolution image feature vector D i C Among the image features (descriptors) that make up i F→C In other words, this descriptor F i C_F→C is the low-resolution keypoint K i F→C Based on the spatial position defined by, the low-resolution image full feature D i C It is a descriptor (group) selected from (the group of descriptors that make up).

[0069] Next, the merger unit 12b merges the descriptor F i C_F→C and the low-resolution descriptor F i F→C and the following equation (7) F i F→C _M=Merge(F i F→C , F i C_F→C ) The updated low-resolution descriptor F is merged by i F→CHere, Merge(·,·) may be any operator that combines (combines) two descriptors, such as an operator that performs (vector) addition, shortcut, or concatenation. Finally, the merge unit 12b generates the following equation for each low-resolution stage (1, 2, . . . , L): (8) (K i C ', F i C ')={(k,d)|k:=(x,y)∈K i C ∪K i F→C , d∈F i C ∪F i F→C _M} The new low-resolution keypoint K is denoted by i C ' and a new low-resolution descriptor F i C ' and pair (K i C ', F i C ').

[0070] As described above, the merger unit 12b merges the original resolution-compatible key points K Fin and the updated original resolution correspondence descriptor F Fin The reduced-resolution keypoints K generated from i F→C and low-resolution descriptor F i F→C At each low-resolution stage, the newly generated keypoints (K i C ') to create a more descriptive descriptor (F i C ').

[0071] Here, the low-resolution KP matching unit 12 matches the updated original resolution keypoints K Fin and the updated original resolution correspondence descriptor F FinIf the low-resolution KP matching process is not received (for example, when the low-resolution KP matching process is performed for the first time), i C ', F i C ') received from the low-resolution feature generator 11 as (K i C , F i C ) will be used in the KP matching process described below.

[0072] Next, in this embodiment, the KP matching unit 12c in (a3) ​​above creates a new low-resolution descriptor F i C ' to find new low-resolution keypoints K i C Specifically, the matching process of the following formula is performed: (9) M i C =Mashing i C {(K i C ', F i C ') A , (K i C ', F i C ') B} With this, we have a set of matched low-resolution keypoint pairs M i C is generated, and the KP matching process result is {M 1 C ,M 2 C ,··,M L C} is output.

[0073] Here, Maching i CIt is also preferable to apply different known matching methods to each low-resolution stage (i) as (·,·) to generate information on matching pairs related to different types of low resolution. By appropriately adjusting the matching method at each low-resolution stage (i), it is possible to finally obtain image feature quantities (K Fin , F Fin Alternatively, at any resolution level, a lightweight and fast matching method, such as nearest neighbor matching, can be used to further reduce the computational cost.

[0074] In any case, the pair M generated by the above formula (9) i C is expressed as follows: (10) M i C :={(x,y) A -(x,y) B} where (x,y) A ∈K i C ' (A) ,(x,y) B ∈K i C ' (B) Here, "-" is a symbol representing a matched pair, and the superscripts A and (A) (superscripts B and (B)) indicate that the subscripted quantity is a quantity related to original image A (original image B).

[0075] In this embodiment, the KP matching unit 12c matches the updated original resolution corresponding key points K Fin and the updated original resolution correspondence descriptor F Fin Similarly, from outside the model (for example, the image matching unit 93 (FIG. 4) described later), (11) M Fin ={(x,y) A -(x,y) B} where (x,y) A ∈K Fin(A) ,(x,y) B ∈K Fin (B) The matching pair M as the result of image matching is defined as Fin If you receive the following, the result will be: {M 1 C ,M 2 C ,··,M L C}∪M Fin to the original resolution corresponding KP extraction unit 13.

[0076] <How to extract KP in original resolution> As also shown in FIG. 1, the original resolution corresponding KP extraction unit 13 of this embodiment calculates the following for each of the original images A and B: (c) First, the upsampling unit 13a performs the upsampling on the low-resolution keypoints KP received from the KP matching unit 12c for each low-resolution stage (1, 2, . . . , L). i C_M (K i C_M (A) , K i C_M (B) ) are rescaled to the original resolution and then unpooled (i.e., upsampled) to obtain a number of original resolution-corresponding keypoints K i F Generate (c2) Next, as the aggregation part 13b, the original resolution corresponding key points K i F The original resolution key point K Fin The output is as follows:

[0077] More specifically, in this embodiment, the upsampling unit 13a of (iii) above calculates the matched low-resolution keypoints KP received from the KP matching unit 12c for each low-resolution stage (1, 2, . . . , L) for each of the original images A and B. i C_M From the above, (12) K i F =Unpool i (K i C_M→F ) where K i C_M→F =Re i Up (K i C_M ) Using the above, the original resolution corresponding key point K i F Generate.

[0078] Here, Re in equation (12) above i Up (·) is an operator that rescales the position of the keypoint corresponding to the low-resolution stage (i) to the position corresponding to the original resolution. i Down It is the opposite operator to (·). Also, Unpool i (·) is an operator that performs the inverse process of the pooling process (unpooling process) on the keypoints corresponding to the low-resolution stage (i) to convert them into keypoints corresponding to the original resolution.

[0079] This Unpool i (·) and the Pool of the above equation (5) i (·) is the Re related to the above position i Up (·) and Re i Down Although it is not required to be an inverse operator like (·), it is preferable that they have consistency and meshing in the calculation process. i (·) and Pool iAs (·), the operator of the well-known grid-sampling process, or the operator of the non-patent literature: Jiahui Zhang et al., “Learning Two-View Correspondences and Geometry Using Order-Aware Network”, Published in 2019 IEEE / CVF International Conference on Computer Vision (ICCV),<https: / / doi.org / 10.1109 / ICCV.2019.00594> , 2019, and non-patent literature: Jiatao Ying et al., “Hierarchical Graph Representation Learning with Differentiable Pooling”, Neural Information Processing Systems 31,<https: / / arxiv.org / pdf / 1806.08804.pdf> The DIFFPOOL operator and the DIFFUNPOOL operator disclosed in 2018 can be adopted.

[0080] Next, in the above (c2) aggregation unit 13b, in this embodiment, the original resolution associated key points K i F , as follows: (13) K Fin =∪ i∈[1,2,··,L] {(x,y)∈K i F} The original resolution key point K is expressed as Fin is generated and output as the extraction result. Here, ∪ i∈[1,2,··,L] is a union operator that generates the union of the following sets when i is 1, 2, . . . , L. According to the above formula (13) (as well as the following formula (14)), the original resolution-matched keypoint (K i F ) can be increased by setting L (the number of low resolution steps) to a larger value.

[0081] The aggregation unit 13b also receives the KP matching results {M 1 C ,M 2 C ,··,M L C}∪M Fin If you have received K L+1 F =K Fin As the following formula (14) K Fin =∪ i∈[1,2,··,L+1] {(x,y)∈K i F} By the original resolution corresponding key point K Fin In any case, the original resolution corresponding KP extraction unit 13 uses the low-resolution KP matching processing result received from the low-resolution KP matching unit 12 to project the positions of the matched keypoints to the original resolution and generates an increased number of original resolution corresponding keypoints K that can ultimately result in high-precision image matching. Fin (K Fin (A) , K Fin (B) ) can be obtained.

[0082] Here, FIGS. 2B and 2C respectively show an embodiment of the KP matching process and the original resolution compatible KP extraction process according to the present invention.

[0083] First, according to Fig. 2(B), it can be seen that the low-resolution keypoint pairs (at a certain low-resolution stage) generated by the KP matching process in the KP matching unit 12c are well-matched keypoint pairs between the smartphone camera image and the 3D rendering image. However, the number of such pairs is not sufficiently large.

[0084] Next, according to FIG. 2(C), as a result of performing an original resolution corresponding KP extraction process (upsampling process) in the original resolution corresponding KP extraction unit 13 for each key point of the matched pair shown in FIG. 2(B), a larger number of original resolution corresponding key points (K i F ) are generated. In this way, a large number of original resolution corresponding keypoints (K i F ) are compiled and the original resolution key points K Fin (K Fin (A) , K Fin (B) ) is generated and output as the extraction result.

[0085] The original resolution corresponding DS extraction unit 14 will be described below. The original resolution corresponding DS extraction process carried out here can be roughly divided into two embodiments (first embodiment and second embodiment).

[0086] <Original Resolution Corresponding DS Extraction Means: First Embodiment> Returning to FIG. 1, in the original resolution corresponding DS extraction unit 14 of the first embodiment, for each of the original images A and B, (D1) The ResNet unit 14a extracts an overall image feature amount D0″ from each image data I of the original images A and B, (D2) The low-resolution image feature generation unit (14-1, . . . , 14-L) performs pypooling processing on the image overall feature D0'' for each of the original images A and B to generate low-resolution image overall feature D0'' at each low-resolution stage (1, 2, . . . , L). i '' (D3) The original resolution corresponding DS generating unit 14b generates the low-resolution image overall feature quantity D i '' is upsampled and combined with the overall image feature D0'', resulting in the original image feature D FinThe generated image feature vector D Fin From the original resolution corresponding key point K (received from the original resolution corresponding KP extraction unit 13) Fin Based on this, the original resolution descriptor F Fin is generated and output as the extraction result.

[0087] Hereinafter, the above processes (D1) to (D3) will be described in more detail. First, in the first embodiment, the ResNet unit 14a of (D1) above processes image data I (I (A) or I (B) ) to obtain the following equation: (15) D0''=ResNet BB (I) The overall image feature D0'' is calculated by the following. Here, ResNet BB The (·) operator represents processing in the feature map layer of PSPNet (Pyramid Scene Parsing Network), which is the backbone of semantic segmentation, which assigns meaning (by associating a label or category) to every pixel in an image.

[0088] As a result, the overall image feature D0'' calculated by equation (15) is a high-level feature that reflects semantic information in the image (e.g., it may include details of objects in the image and information on accessories). Incidentally, this feature map layer of PSPNet is a variation of ResNet.

[0089] For more information on PSPNet, see the following non-patent literature: Hengshuang Zhao et al., “Pyramid Scene Parsing Network”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 1612.01105.pdf> , 2017. For more information on ResNet, see Kaiming He et al., “Deep Residual Learning for Image Recognition”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, Volume 1, pp.770-778,<https: / / doi.org / 10.1109 / CVPR.2016.90> , details of which were disclosed in 2016.

[0090] In addition, the ResNet unit 14a may generate the overall image feature D0″ using a backbone other than the feature map layer of PSPNet, as long as the backbone is a semantic segmentation-based backbone. Furthermore, the ResNet unit 14a and the CNN unit 11a (of the low-resolution feature generator 11) may be integrated (integrated into one functional configuration unit) to, for example, use the above formula (1) (D0=CNN BB (I))CNN BB (·) as ResNet BB It is also possible to adopt (·). In this case, the whole image feature D0 and the whole image feature D0'' are the same feature. In this case, the low-resolution feature generator 11 and the original resolution corresponding DS extractor 14 become two CNN branches sharing the same backbone.

[0091] Next, in the first embodiment, the low image feature generation units (14-1, . . . , 14-L) in (D2) above calculate the following equation for each of the original images A and B using the overall image feature amount D0″: (16) D i ''=PyPooling i (D0'') The low-resolution image overall feature value D i '' where PyPooling i (·) is the pypooling operator used in PSPNet mentioned above.

[0092] Next, in the first embodiment, the original resolution corresponding DS generating unit 14b of (D3) generates the low resolution image overall feature quantity D i Using the formula (17) D Fin =CONCAT(D0'', {UPSAMPLE(CONV i (D i ''))} i∈[1,2,··, L] ) Then, the image feature vector D corresponding to the original image is obtained. Fin Generate.

[0093] Here, CONV i (·) is a convolution operator that performs convolution processing on the target quantity of the low resolution stage (i). Also, UPSAMPLE(·) is an upsampling operator that performs upsampling processing on the target quantity. Furthermore, CONCAT(D0'', {UPSAMPLE(CONV i (D i ''))} i∈[1,2,··, L] ) is D0'' and UPSAMPLE(CONV 1 (D1'') and UPSAMPLE(CONV 2 (D2'') and, UPSAMPLE(CONV L (D L '').

[0094] The process in the form of the above equation (17) is usually performed in the above-mentioned PSPNet, and is the process of extracting the CONV i (D i'') directly, we extract dense features (image features corresponding to the original image) D Fin Since the above process is performed by extracting the above-mentioned vectors, the calculation cost is reduced.

[0095] Next, the original resolution corresponding DS generating unit 14b generates the original image corresponding image feature quantity D Fin From the original resolution corresponding key point K (received from the original resolution corresponding KP extraction unit 13) Fin Based on this, the following equation (18) F Fin =D Fin (x,y,:) (x,y)∈KFin Using this, we obtain the original resolution correspondence descriptor F Fin Here, the above equation (18) generates the original resolution correspondence descriptor F Fin is the image feature vector D Fin Among the image features (descriptors) that make up Fin In other words, the original resolution descriptor F Fin is the key point K corresponding to the original resolution Fin Based on the spatial position defined by Fin It is a descriptor (group) selected from (the group of descriptors that make up).

[0096] <Original Resolution DS Extraction Means: Second Embodiment> As also shown in FIG. 1, in the original resolution corresponding DS extraction unit 14 of the second embodiment, for each of the original images A and B, (D1) The ResNet unit 14a extracts an overall image feature amount D0″ from each image data I of the original images A and B, (E2') The low image feature generation unit (14-1, . . . , 14-L) calculates, for each of the original images A and B, a low-resolution corresponding image feature D i C '' (E3') The original resolution corresponding DS generating unit 14b generates the low resolution corresponding image feature quantity D i C '' is merged, and the result of the previous merge process is deconvolved and the low-resolution corresponding image feature D i C The process of merging the original image feature vector D Fin and generate the image feature vector D Fin From the original resolution corresponding key point K (received from the original resolution corresponding KP extraction unit 13) Fin Based on this, the original resolution descriptor F Fin is generated and output as the extraction result.

[0097] Here, the ResNet unit 14a in (D1) above performs the same processing as in the first embodiment. Below, (D2') and (D3') above will be described in more detail.

[0098] First, the low-resolution stage low-image feature generator 14-(i) in the above (D2') (1≦i≦L) calculates the overall image feature quantity D0″ using the following formula in the second embodiment: (19) K i C ", D i C ''=Det i (PyPooling i (D0'') Using the low-resolution keypoint K i C ''(1≦i≦L) and low-resolution corresponding image feature D i C Calculate (1≦i≦L) where Det i (·) is the Det i Similar to (·), this is a keypoint detection operator that detects keypoints in an image (at a resolution equivalent to the i-th low-resolution step). i(·) is the pypooling operator used in PSPNet mentioned above.

[0099] Next, in the second embodiment, the original resolution corresponding DS generating unit 14b in (E3') generates low resolution corresponding image feature quantities D i C '' from i=1 to i=L in sequence, (20) D i+1 F =Merge(DECONV i H (D i F ), D i C '') Then, the image feature vector D corresponding to the original image is obtained. Fin (=D L+1 F ) is calculated, where D 1 F is D0'' (D 1 F =D0''). Furthermore, DECONV i H (·) is the deconvolution operator to the original resolution level, and Merge(·,·) is the operator that merges the target quantities. In other words, the above formula (20) simply expresses the whole image feature quantity D0″ as the low-resolution corresponding image feature quantity D i C This represents a decoder that uses '' to upscale to the original resolution.

[0100] Next, the original resolution corresponding DS generating unit 14b generates the original image corresponding image feature quantity D Fin From the original resolution corresponding key point K (received from the original resolution corresponding KP extraction unit 13) Fin Based on this, the following equation (21) F Fin =D Fin (x,y,:) (x,y)∈KFin (The same form as equation (18) above) Using this, we obtain the original resolution correspondence descriptor F Finis generated and output as the extraction result.

[0101] As described above, two embodiments (first embodiment and second embodiment) of the original resolution corresponding DS extraction unit 14 have been described. In either case, the original resolution corresponding DS extraction unit 14 extracts the original resolution corresponding key points K (received from the original resolution corresponding KP extraction unit 13) for each of the original images A and B. Fin and the corresponding (self-generated) original resolution correspondence descriptor F Fin Pairs with, i.e., (K Fin , F Fin ) A , (K Fin , F Fin ) B This is generated and output as the extraction result (outcome) of this image feature extraction model.

[0102] Among these, the original resolution corresponding key point K Fin As mentioned above, the original resolution descriptor F Fin is generated from the global image features D0'' based on semantic segmentation, and is a high-level descriptor that also reflects semantic information in the image (which may include, for example, details of objects in the image and information about their surroundings).

[0103] [Image feature extraction method] 3 is a flowchart outlining an image feature extraction method according to an embodiment of the present invention. The process flow in steps S101 to S109 of this method will be described below.

[0104] (S101) Image data of original images A and B are acquired from outside. (S102) Using the acquired image data, "low-resolution keypoints" and "low-resolution descriptors" are generated for each image.

[0105] Next, steps S103 to S108 are repeated a predetermined number of times (N times) to generate KP·DS for the original resolution. It has been confirmed that good results can be obtained by setting the predetermined number of times to, for example, 4 to 8 times (4≦N≦8). (S103) It is determined whether or not there are any "updated original resolution corresponding keypoints" and "updated original resolution corresponding descriptors" that have been generated externally. As described above, in this embodiment, the determination in step S103 is the same as the determination as to whether or not the current original resolution compatible KP·DS generation process is the first time. If the determination is false (non-existent or the first time), the process proceeds to step S106.

[0106] (S104) On the other hand, if the determination in step S103 is true (exists), downsampling processing is performed on the "updated original resolution-corresponding keypoints" and the "updated original resolution-corresponding descriptors." (S105) Using the result of the downsampling process in step S104 and the "low-resolution keypoints" and "low-resolution descriptors" generated in step S102, new "low-resolution keypoints" and new "low-resolution descriptors" are generated.

[0107] (S106) A low-resolution KP matching process is performed using the (new) "low-resolution keypoints" and the (new) "low-resolution descriptors." (S107) An original resolution corresponding KP extraction process is performed on the processing result of step S106 to generate "original resolution corresponding key points." (S108) For the overall image features generated from the acquired image data, an original resolution-compatible DS extraction process is performed using the "original resolution-compatible keypoints" to generate an "original resolution-compatible descriptor."

[0108] (S109) After the predetermined number of times (N times) has been completed, the "keypoints corresponding to the original resolution" and the "descriptor corresponding to the original resolution" for each of the original images A and B generated in steps S107 and S108 are output as extraction results.

[0109] [Image matching device / program] FIG. 4 is a functional block diagram showing an embodiment of an image matching device 9 according to the present invention.

[0110] According to FIG. 4, the image matching device 9 of this embodiment: (a) an input unit 91 for inputting image data of original images A and B to be subjected to image matching; (b) Using the image feature extraction model 1 described above, the image feature (K Fin , F Fin ) A and the original image B (K Fin , F Fin ) B An image feature extraction unit 92 that generates (c) Generated image features (K Fin , F Fin ) A and (K Fin , F Fin ) B an image matching unit 93 that performs image matching processing of the original images A and B using (e) The result of the image matching process in (c) is a matching pair M (a pair of matching keypoints). Fin An output unit 94 that provides the It has.

[0111] Here, the image feature extraction unit 92 and the image matching unit 93 are main functional components for implementing an embodiment of an image matching method (including an image feature extraction method) according to the present invention, and further, are functions of a processor / memory that stores an embodiment of an image matching program (including an image feature extraction program) according to the present invention. As a result, the image matching device 9 may be a dedicated device for image matching processing, but it can also be, for example, a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or a smartphone equipped with the image matching program according to the present invention.

[0112] 4, the input unit 91 may be, for example, equipped with a communication function and receive the original image A and the original image B from an external information processing device, such as an image management server, or may be an interface such as a Universal Serial Bus (USB) that allows the user to directly input the original image A and the original image B. Furthermore, the output unit 94 may also be, for example, equipped with a communication function and transmit the image matching result to an external information processing device, such as an image management server that provides the images, or may be a display that can display the image matching result.

[0113] In addition, the image matching unit 93 may perform image matching processing using various known image matching methods, such as nearest neighbor matching, but in this embodiment, image matching is performed using graph attention (GAT) processing, which can further improve matching accuracy.

[0114] Specifically, in the image matching unit 93 (a) The graph generating unit 931 generates original resolution corresponding key points K for each of the original images A and B. Fin A graph is generated to represent the original image using (b) The graph attention unit 932 generates an original resolution correspondence descriptor F for each of the original images A and B.Fin The GAT process is performed using the generated graph to obtain the updated original resolution descriptor F Fin Generate (c) The KP matching unit 933 matches the updated original resolution correspondence descriptor F Fin Using the original resolution corresponding keypoint K Fin The matching process is performed to find the matching pair M Fin This generates:

[0115] In this embodiment, the image matching unit 93 calculates key points K Fin and the original resolution correspondence descriptor F Fin The updated pair with (K Fin , F Fin ), and the matching pair M Fin is input to the image feature extraction model 1 (of the image feature extraction unit 92) and used for image feature extraction, and the process of receiving the image feature extraction result therefrom is repeated a predetermined number of times (N times). Fin is output to the output unit 94.

[0116] This enables the image matching unit 93 to better accommodate differences in resolution between the original images A and B, generate more key points, and perform image matching processing that can take into account more details and additional information about objects in the images.

[0117] Incidentally, the image matching method in the KP matching unit 933 in (c) above is described in the non-patent document: Paul-Edouard Sarlin et al., “SuperGlue: Learning Feature Matching with Graph Neural Networks”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,<https: / / arxiv.org / pdf / 1911.11763.pdf> , 2020 can be applied.

[0118] The image matching method of this non-patent document (Sarlin et al.) employs a deep learning model called the Transformer, which uses an attention mechanism that weights and considers the influence of different input image parts. This Transformer is trained to perform image matching based on both the descriptors and the positions of keypoints, and includes (a) a keypoint encoder that grasps the positions of keypoints, (b) a self / cross attention layer (9 layers) related to the features of the keypoints and the descriptors within each image or between images, and (c) a matching layer including the Sinkhorn algorithm for matching two embedded representation features. By using such a Transformer, it is possible to perform image matching with very high accuracy.

[0119] As described above in detail, according to the present invention, it is possible to carry out image matching processing that can handle differences in resolution between images, generate more key points, and take into account more details and additional information of objects in images. It is also possible to extract image features that can be used to carry out such image matching processing.

[0120] Furthermore, the highly accurate image matching process of the present invention can be utilized in the analysis of a huge amount of camera images from numerous security cameras installed in a city, which can facilitate prediction and detection of trouble and crime occurrence, and even arrest of perpetrators, etc. In other words, the present invention can contribute to Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

[0121] Furthermore, the highly accurate image matching process of the present invention can be used to analyze satellite and aerial photographs of target areas, particularly agricultural areas, forest areas, and arid areas, as well as satellite and aerial photographs of target sea areas, to investigate various conditions in such areas and sea areas, such as the growth status of crops, the current state of ecosystems, and the impact of climate change. In other words, the present invention can contribute to Goal 13 "Take urgent action to combat climate change and its impacts," Goal 14 "Conserve and sustainably use the oceans and marine resources," and Goal 15 "Sustainably manage forests, combat desertification, halt and reverse land degradation, and halt biodiversity loss" in the Sustainable Development Goals (SDGs) led by the United Nations.

[0122] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0123] 1. Image feature extraction model 11 Low-resolution feature generation unit 11a CNN Department 11-1a, 11-2a, 11-La Low Key Point (KP) Generation Section 11-1b,11-2b,11-Lb Low descriptor (DS) generator 12 Low-resolution KP matching section 12a Downsampling section 12b Merge section 12c KP Matching Section 13 Original resolution compatible KP extraction section 13a Upsampling section 13b Assembly area 14 Original resolution compatible DS extraction section 14a ResNet Department 14-1, 14-2, 14-L Low image feature generation section 14b Original resolution compatible DS generation section 9. Image Matching Device 91 Input section 92 Image feature extraction unit 93 Image Matching Section 931 Graph Generation Unit 932 Graph Attention Section 933 KP Matching Department 94 Output section

Claims

1. An image feature extraction model for causing a computer to extract key points and descriptors from each of two original images, comprising: a low-resolution feature generating means for generating, for each of the images, low-resolution keypoints and low-resolution descriptors in at least one low-resolution step lower than an original resolution that is a resolution of the original image, from the extracted features of the entire image; a low-resolution keypoint matching means for performing a matching process of the low-resolution keypoints between the two original images for each low-resolution stage, and generating a pair of matched low-resolution keypoints; an original resolution corresponding keypoint extraction means for performing upsampling on the matched low-resolution keypoints for each of the images at each low-resolution stage, generating original resolution corresponding keypoints corresponding to the original resolution, the number of which is greater than the number of the low-resolution keypoints, and outputting the generated keypoints as an extraction result; an original resolution corresponding descriptor extraction means for generating an original image corresponding image feature quantity corresponding to the original image for each of the images by using a low-resolution image overall feature quantity at each low resolution stage generated from the extracted overall image feature quantity, or by using a low-resolution corresponding image feature quantity at each low resolution stage generated by performing a keypoint detection process on the low-resolution image overall feature quantity, generating an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image feature quantity based on the original resolution corresponding keypoint, and outputting the generated descriptor as an extraction result; The image feature extraction model is characterized by causing a computer to function as follows.

2. The low resolution keypoint matching means includes: For each image, receive updated original resolution corresponding keypoints and updated original resolution corresponding descriptors, which are output and externally updated original resolution corresponding keypoints and original resolution corresponding descriptors; For each of the images, for each of the low-resolution stages, a pooling process is performed on the updated original resolution corresponding keypoints that have been re-scaled to the low-resolution stage and the updated original resolution corresponding descriptors to generate low-resolution keypoints and low-resolution descriptors that correspond to the low-resolution stage; for each of the images, for each of the low resolution stages, merging the low resolution descriptor for that low resolution stage with a previously generated descriptor based on the low resolution keypoints from features of all of the images to generate an updated low resolution descriptor; For each image, for each low-resolution stage, combine the low-resolution keypoint with the previously generated low-resolution keypoint to obtain a new low-resolution keypoint, combine the updated low-resolution descriptor with the previously generated low-resolution descriptor to obtain a new low-resolution descriptor, and perform a matching process of the new low-resolution keypoint between the two original images to generate a pair of matched low-resolution keypoints.

2. The image feature extraction model according to claim 1 .

3. The image feature extraction model is a process in the low-resolution keypoint matching means for receiving the updated original resolution corresponding keypoints and the updated original resolution corresponding descriptors and generating a pair of matched low-resolution keypoints; A process in which the original resolution corresponding keypoint extraction means generates the original resolution corresponding keypoints and outputs them as an extraction result; and A process in which the original resolution corresponding descriptor extraction means generates the original resolution corresponding descriptor and outputs the generated descriptor as an extraction result. The image feature extraction model according to claim 2, characterized in that the computer is caused to function so as to perform the above set of steps a plurality of times and to output the finally generated keypoints corresponding to the original resolution and descriptors corresponding to the original resolution as extraction results.

4. The image feature extraction model according to any one of claims 1 to 3, characterized in that the low-resolution feature generation means generates, for each of the images, for each of the low-resolution stages, low-resolution keypoints and low-resolution image whole features at that low-resolution stage from the features of the entire image, and generates the low-resolution descriptor from the low-resolution image whole features based on the low-resolution keypoints.

5. 5. The image feature extraction model according to claim 1, wherein the original resolution corresponding keypoint extraction means performs an unpooling process on the matched low-resolution keypoints for each of the images, for each of the low-resolution stages, after performing a rescale process to the original resolution on the matched low-resolution keypoints, to generate original resolution corresponding keypoints corresponding to the original resolution in a number greater than the number of the low-resolution keypoints in the low-resolution stage, and outputs the original resolution corresponding keypoints in each low-resolution stage as a compilation of the original resolution corresponding keypoints.

6. The image feature extraction model according to any one of claims 1 to 5, characterized in that the original resolution corresponding descriptor extraction means performs a pypooling process on the features of the entire image for each of the images to generate low-resolution image overall features at each low resolution stage, performs an upsampling process on the low-resolution image overall features subjected to the convolution process at each low resolution stage, and further generates image features corresponding to the original image by combining them with the features of the entire image, and generates an original resolution corresponding descriptor corresponding to the original resolution from the image features corresponding to the original image based on the original resolution corresponding keypoints.

7. The original resolution corresponding descriptor extraction means is a process of merging, for each of the images, the low resolution corresponding image features, which is a result of the merging process at one of the low resolution stages, which is subjected to a deconvolution process, with the low resolution corresponding image features, and this process is carried out sequentially from lowest resolution to the original resolution stage to generate the original image corresponding image features, and generates an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image features based on the original resolution corresponding keypoints.

8. an image feature extraction means for extracting, from each of the two original images, key points corresponding to the original resolution and descriptors corresponding to the original resolution, which are image features of the original images, by using an image feature extraction model according to any one of claims 1 to 7; an image matching means for performing matching of the two original images using the extracted original resolution-corresponding keypoints and the original resolution-corresponding descriptors; 1. An image matching device comprising:

9. an image feature extraction means for extracting, from each of the two original images, key points corresponding to the original resolution and descriptors corresponding to the original resolution, which are image features of the original images, by using an image feature extraction model according to any one of claims 1 to 7; an image matching means for performing matching of the two original images using the extracted original resolution-corresponding keypoints and the original resolution-corresponding descriptors; The image matching program causes a computer to function as follows.

10. The image matching means is a graph generating means for generating, for each of the images, a graph representing the original image using the original resolution corresponding keypoints; graph attention means for performing a graph attention process using the graph on the original resolution corresponding descriptor for each of the images to generate an updated original resolution corresponding descriptor; a keypoint matching means for performing matching processing of the original resolution-corresponding keypoints using the updated original resolution-corresponding descriptor; 10. The image matching program according to claim 9, further comprising:

11. 1. A computer-implemented image feature extraction method for extracting keypoints and descriptors from each image of two source images, comprising: generating, for each of the images, low-resolution keypoints and low-resolution descriptors at at least one low-resolution level lower than the original resolution of the original image from the extracted features of the entire image; performing a matching process of the low-resolution keypoints between the two source images for each low-resolution stage to generate a pair of matched low-resolution keypoints; performing upsampling on the matched low-resolution keypoints for each of the images at each low-resolution stage to generate original resolution corresponding keypoints corresponding to the original resolution, the number of which is greater than the number of the low-resolution keypoints, and outputting the generated keypoints as extraction results; generating an original image corresponding image feature corresponding to the original image for each of the images using a low-resolution image overall feature at each low-resolution stage generated from the extracted overall image feature, or using a low-resolution corresponding image feature at each low-resolution stage generated by performing a keypoint detection process on the low-resolution image overall feature, generating an original resolution corresponding descriptor corresponding to the original resolution from the original image corresponding image feature based on the original resolution corresponding keypoint, and outputting the generated descriptor as an extraction result; 1. An image feature extraction method comprising the steps of:

Images