Feature point upsampling device, method, and program

Abstract

To increase the number of geometrically matched feature points that can be detected from an image pair of the same subject taken from different viewpoints by up-sampling.SOLUTION: A feature point extraction unit A extracts feature points from an image pair of the same subject taken from different viewpoints. A feature point matcher B applies the feature points extracted from the image pair to a first inference model to detect an initial matching pair. A feature point up-sampler C applies the feature points extracted from the image pair and the initial matching pair to a second inference model to up-sample the feature points. A correct data generation unit D generates correct data using the feature point extraction unit A for a teacher model that has learned in advance the relationship between the feature points extracted from the image pair of the same subject taken from different viewpoints and matching results. The correct data generation unit D distills the knowledge of the teacher model to train the inference models of the feature point matcher B and feature point up-sampler C.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a feature point upsampling device, method, and program, and more particularly to a feature point upsampling device, method, and program that increases the number of feature points extracted from each of a pair of images of the same subject taken from different viewpoints by patch-based upsampling based on a feature matching set. [Background technology]

[0002] Many deep learning methods designed for image matching use ground truth matches (GTM). Therefore, accurately detecting a large number of keypoints from an image pair can improve the learning effect by increasing the number of matches in the GTM. This has been proven in the high-density matching task disclosed in Non-Patent Document 1, but this method often lacks efficiency because all image pixels are considered equally important and the number of detected keypoints is equal to the number of pixels in the image.

[0003] Patent Document 1 discloses an image comparison tool trained by a machine learning system that estimates the similarity between images based on image features. The authenticity of image matching is determined by comparing the pixel values ​​of two images and determining the similarity between the pixels. For example, the pixel value comparison can be performed using the average difference, and the reciprocal of the average difference represents the similarity between image pixels.

[0004] Patent Document 2 discloses an image matching system based on a spatial channel attention neural network. In Patent Document 2, a matching set of features is obtained to predict the probability of a matching pair, and a weighted 8-point algorithm is calculated to obtain an essential matrix, and an accurate camera pose is obtained according to the essential matrix. This method employs geometry information to supplement the GTM with relative camera poses to train a spatial channel attention neural network used to predict the matching probability.

[0005] Patent Document 3 discloses a data generation system that uses the output of a neural network as training data for training a target neural network. The neural network from which the output is derived is the target network or the teacher network. In Patent Document 3, the activation map from the neural network is the output of the teacher network.

[0006] Non-Patent Document 1 discloses a technique for finding matching pixel pairs without feature detection by performing dense matching that matches all pixels between two images. Non-Patent Document 1 employs a neural network backbone to extract local features from images and two Transformers to match pixels at coarse and fine scales.

[0007] The coarse-level Transformer performs fine matching using self-attention / cross-attention, which is multi-head attention, resulting in matched pixel pairs at the coarse scale. The matched pixel pairs at the coarse scale are then used to initialize match pairs for the fine-level Transformer. The fine-level Transformer performs matching using self-attention / cross-attention to extract fine matching keypoints.

[0008] Non-Patent Document 2 proposes sparse matching, which matches two sets of features from a pair of images. Non-Patent Document 2 employs the Transformer, a deep learning model with an attention structure that weights the influence of different parts of the input. The Transformer is trained to match images based on both the feature positions and descriptors.

[0009] The Transformer has the following components: (1) a feature point encoder to capture the positions of feature points, (2) a nine-layer self-attention / cross-attention multi-head attention to aggregate information on feature points and their feature descriptions within / between two images, and (3) a matching layer using the Sinkhorn algorithm to capture correspondences between features embedded with their positions and feature descriptions between two images.

[0010] [3] proposes a detail-driven deep neural network for upsampling point sets. This method utilizes patch-based progressive upsampling for point sets. The neural network learns different levels of detail in multiple steps, with each step focusing on a local patch from the output of the previous step. [3] trains the network end-to-end incrementally, gradually upsampling a sparse input point set to generate a dense point set rich in geometric details. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] US10489678B2 [Patent Document 2] CN112489098A [Patent Document 3] US20200302295A1 [Patent Document 4] US20190205748A1 [Non-patent literature]

[0012] [Non-Patent Document 1] "LoFTR: Detector-Free Local Feature Matching with Transformers", Sun et al., 2021 [Non-patent document 2] "SuperGlue: Learning Feature Matching with Graph Neural Networks",Sarlin et al., 2020 [Non-patent document 3] "Patch-based Progressive 3D Point Set Upsampling",Yifan et al., 2019 Summary of the Invention [Problem to be solved by the invention]

[0013] As shown in Figure 7, image pair I is a pair of images of the same subject taken from different viewpoints. A ,I B In image matching, feature points are detected first and then each camera image I A ,I B Multiple feature points are detected from each image [Fig. 1(a)].

[0014] Next, the feature points detected from one camera image are transferred to the other camera image (here, camera image I A Camera image I B ) projective transformation [Fig. 1(b)]. The camera orientation and depth information to the subject required for this calculation are stored in the image pair I A ,I B is given along with.

[0015] Finally, feature points that overlap and have matching feature descriptions are registered as a correct matching pair [Fig. 1(c)]. The feature points that make up this pair are called correct matching feature points. This type of image matching method has the following technical challenges:

[0016] First, even if the positions of detected feature points are geometrically consistent between two images, they may not overlap after projection using the above method. In such cases, the feature points cannot be verified as correct data. The number of correct matching feature points is often 35% or less of the number of detected feature points.

[0017] Second, existing correct matching pairs are represented by a binary matrix in which matched feature point pairs are "1" and unmatched feature point pairs are "0", but the number of matching points is insufficient because the number of matching points is less than 35% of the total feature points.

[0018] The object of the present invention is to solve the above technical problems and provide a feature point upsampling device, method, and program that increases the number of geometrically matching feature points that can be extracted from an image pair through upsampling, and further relaxes the conditions for correct matching pairs through a soft labeling method that adds new information during feature point matcher training. [Means for solving the problem]

[0019] To achieve the above objective, the present invention solves the problem of generating correct data by (1) upsampling feature points and (2) generating additional labeling for matcher learning of the upsampled feature points.

[0020] To increase the number of feature points, we design a new neural network architecture that learns geometrically matching keypoints by upsampling the matched feature points and further interpolating their depth, correcting the upsampling of the point set, and also learns and corrects the positions of the upsampled feature points.

[0021] We modify the knowledge distillation technique for matcher learning and provide ground truth data for learning feature point upsampling and matching.

[0022] The present invention can be realized not only as a feature point upsampling device equipped with such characteristic processing units, but also as a feature point upsampling method including such characteristic processing steps, and further as a feature point upsampling program that causes a computer to execute such steps. [Effects of the Invention]

[0023] According to the present invention, the following effects can be achieved.

[0024] (1) Feature points that could not be extracted from the image can be added by upsampling, so the total number of feature points that can be used for matching can be increased.

[0025] (2) By complementing the depth of the increased feature points through upsampling, high-precision projective transformation based on a large number of feature points can be realized.

[0026] (3) Upsampling of feature points is performed on a patch basis, using the initial feature matching pairs as seeds, thereby reducing the matcher calculation time.

[0027] (4) By using the positions of matched feature points to prevent unnecessary matching, the efficiency of the entire pipeline, including matching, 3D reconstruction, and visual localization, can be improved. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a functional block diagram showing the configuration of the main parts of a feature point upsampling device according to an embodiment of the present invention. [Figure 2] FIG. 10 is a flowchart illustrating the processing of the feature point upsampler. [Figure 3] FIG. 10 is a diagram illustrating a patch division process. [Figure 4] FIG. 10 is a diagram illustrating an upsampling process. [Figure 5] FIG. 1 is a diagram (part 1) for explaining the effect of the present invention. [Figure 6] FIG. 2 is a diagram (part 2) for explaining the effect of the present invention. [Figure 7] FIG. 1 is a diagram for explaining a problem to be solved by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029]

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main parts of a feature point upsampling device according to an embodiment of the present invention, and the main components include a feature point extraction unit A, a feature point matcher B, a feature point upsampler C, and a supervised data addition and generation unit D.

[0030] Such a feature point upsampling device can be configured by installing an application (program) that realizes each of the functions detailed below on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0031] The feature point extraction unit A extracts feature points from two images (image pair) of the same subject taken from different viewpoints. The extracted feature points consist of two elements: the position of the feature point and its feature descriptor. The feature descriptor is a high-dimensional vector that describes the semantic information of the feature point. In this embodiment, the feature point extraction unit A may employ an existing method such as SIFT or SuperPoint, or may employ a neural network capable of learning in an end-to-end manner.

[0032] The feature point extraction unit A extracts the image I A Extract feature points from the image and use the position vector Ko to represent their positions (x, y) on the two-dimensional image. A ∈R No×2 and a high-dimensional vector Fo representing the feature descriptor. A ∈R No×F Therefore, the processing of the feature point extraction unit A can be expressed by the following equations (1-1) and (1-2). The extracted position vector Ko A and the high-dimensional vector of feature descriptors, Fo A are provided to the feature point matcher B, the feature point upsampler C, and the ground truth data additional generation unit D.

[0033]

number

[0034] Feature point matcher B compares two images I A ,I B Feature matching is performed based on the feature points extracted from the image, and corresponding feature points are determined as initial matching pairs. Feature Matcher B may use an existing feature matching method that uses a neural network (a small learning model for inference: inference model) such as MNN (Matching Neural Network) Matching, SuperGlue, or SGMNet, or it may use a neural network that can be trained in an end-to-end manner.

[0035] For example, to train feature point matcher B in an end-to-end manner, we need to train both the SuperGlue graph neural network (GNN) and the Sinkhorn layer. The GNN part can be defined as follows: (2) The Sinkhorn layer can be defined as follows: (3)

[0036]

number

[0037]

number

[0038] The set of initial matching pairs is M score In this way, the feature point matcher B can find the nearest neighbors of two images I A ,I B The initial matching pair set M of feature points o and outputs it to the feature point upsampler C. Therefore, the operation of the feature point matcher B can be expressed by the following equation (4). Here, Mo is the image I A Image from I B represents the set of initial matching pairs to

[0039]

number

[0040] As shown in Figure 2, the feature point upsampler C performs the processes of patch division C.1, upsampling C.2, patch-based matching C.3, depth interpolation C.4, and geometry-aware learning C.5, and increases the total number of feature points and matching pairs by local upsampling on a patch basis based on the initial matching pairs.

[0041] In this embodiment, each feature point of the initial matching pair is divided into multiple small regions (patches) based on its position. Next, each feature point for each patch is input to a neural network (a small learning model for inference: an inference model) to extract (upsample) new feature points related to the input feature points, and patch-based local feature point matching is performed. Finally, depth information is interpolated for each matching pair of the upsampled feature points.

[0042] In patch division C.1, for example, using the K-NN (k-nearest neighbor) algorithm, we divide each image I A ,I B A large number of feature points are extracted from the image and divided into K small regions (patches: l1, l2…l K This process can be expressed by the following equations (5-1) and (5-2).

[0043]

number

[0044] Here, I i A is image I A The i-th patch of l i A A set of feature point indices in I j B is image I B The jth patch of l j BIn this embodiment, the set of feature point indices in the image I A ,I B The feature point index specifies each patch of each feature point at the position k i A ,k j B and its feature descriptor d i A ,d j B This process can be expressed by the following equations (6-1) and (6-2).

[0045]

number

[0046] The feature point position k of each patch i A and k j B has a size of K × 2, and the feature descriptor d i A ,d j B has a size of K × F, where K is the number of feature points and their feature descriptors, and F is the size of one feature descriptor. Figure 3 shows a schematic representation of the patch division process C.1.

[0047] In upsampling C.2, the position and feature descriptor of each feature point for each patch are input to a neural network to generate feature descriptors for a large number of points. This process can be expressed by the following equations (7-1) and (7-2). (k i A ,d i A ) contains the positions of the K nearest neighbors associated with the i-th patch and their feature descriptors.

[0048]

number

[0049] As shown in Figure 4, when the scaling factor is set to P, the resulting feature descriptor d - i A is associated with P upsampled points from the given feature point. Deep neural networks such as OA-Net [Zhang et al., 2019] and PU-GCN [Qian et al., 2019] can be used to upsample the associated feature descriptors.

[0050] For example, UPSAMP(·) can upsample the number of feature points by using a graph neural network (GCN) layer to increase the feature dimension in a trainable feature point upsampling unit such as PU-GCN (Point Cloud Upsampling using Graph Convolutional Networks) and periodically shuffling the increased feature dimension. In this embodiment, the feature dimension is first expanded as shown in the following equation (8).

[0051]

number

[0052] Here, GCN means that it receives an input of size F×K and gives an output of size PF×K, as shown in the following equation (9).

[0053]

number

[0054] Then, as shown in the following equation (10), a reduction operation is performed so that the final dimension has the maximum value.

[0055]

number

[0056] Finally, a feature descriptor d containing P upsampled points from the i^th seed position is obtained. - i A Based on the following equation (11), ~ i A Permutation of the dimensions of

[0057]

number

[0058] The position of the feature point relative to the upsampled point is calculated by (13) using a DNN layer that maps the feature descriptor to two dimensions, i.e. (12)

[0059]

number

[0060]

number

[0061] where k - i A and k - j B are the sets of upsampled feature points in the i-th and j-th patches, respectively. - i A and k - j B can take any value outside the range of image coordinates. Furthermore, the new coordinates do not guarantee that the positional relationship between the two images will be maintained. For example, there is a possibility that the positions of points aligned from the left in image A and the corresponding points in image B will be swapped. To prevent this, in this embodiment, the output of the above equation (13) is used as a deviation from the seed position, and the output k - i A and k -j B For example, if P=4, then k - i A You will get 4 points from

[0062] Therefore, in this embodiment, a directional filter such as [(0, 1), (0, -1), (-1, 0), (0, 1)] is used to calculate k - i A Therefore, if the directional filter is represented as Filt(·), the position of the new feature point can be calculated using the following equations (14-1) and (14-2).

[0063]

number

[0064] The upsampled feature point locations k-iA and feature descriptor d ~ The concept of iA can be explained by the following equations (15) and (16), respectively.

[0065]

number

[0066]

number

[0067] where P i A and P j B are the images I A From the ith source patch of image I B represents the set of indexes of feature points to the j-th target patch of P i A ,P j BIf we collect all patches with index (p, q) from up can be defined by the following equation (17).

[0068]

number

[0069] In patch-based matching (C.3), local matching of feature points is performed between patches. Feature points are input via their feature descriptors and projected to higher dimensions.

[0070] Finally, local matching is performed between each patch, and the matched feature points are collected and stored. Local matching of feature points can be performed using any matcher and filtering. The processing of the patch-by-patch matching unit C.3 can be expressed by the following equation (18). Here, M up is image I A Image from I B represents the set of matching pairs of feature points upsampled to

[0071]

number

[0072] In depth interpolation C.4, the depth of the upsampled feature points is estimated by interpolation. The input 2D feature points do not have depth information. The index set P i A ,P j B The upsampled positions and feature descriptors are (k - i A ,d - i A ),(k - j B ,d - j B), the depth can be estimated using the following equation (19).

[0073]

number

[0074] In geometry-aware learning C.5, various losses such as (1) geometry loss, (2) description loss, (3) upsampling loss, and (4) depth estimation loss are learned so that upsampled feature points can be detected while maintaining geometric consistency between images.

[0075] (1) Geometric loss To optimize the network parameters to accurately detect the positions of feature points, the upsampling of feature points is learned using the camera pose and depth information (CAM). The geometric loss learning method can be expressed as the following equation (20): where k - Mup A and k - Mup B is M up indicates the positions of the feature points of the matching pair.

[0076]

number

[0077] (2) Feature Descriptor Loss To further improve discrimination, a feature descriptor for the upsampled feature points is trained. The training method for the feature descriptor loss can be expressed as follows:

[0078]

number

[0079] (3) Upsampling loss The augmented ground truth data is used to train the upsampled keypoints. The training method for the upsampling loss can be expressed as follows:

[0080]

number

[0081] (4) Depth estimation loss The extended ground truth data is used to train the upsampled feature points. The training method for the depth estimation loss can be expressed as follows: where Chamfer(·) corresponds to the Chamfer loss disclosed in Non-Patent Document 3.

[0082]

number

[0083] On the other hand, to achieve good convergence, M up should be as close as possible to the correct matching pair of upsampled points. However, extracting the correct matching pair of upsampled points may incur high computational costs due to the mismatch of different sampling points. In this embodiment, we use a knowledge distillation method to solve this problem.

[0084] In this embodiment, a large-scale and complex neural network (a large model for learning: a teacher model) that has learned the relationship between feature points extracted from image pairs and matching results is prepared in advance in the supervised data addition and generation unit D, and the output of the teacher model is provided to the feature point matcher B and feature point upsampler C as supervised data.

[0085] The supervised data addition and generation unit D includes a soft label extraction unit D.1, an upsampling supervised data matching unit D.2, and a supervised depth interpolation unit D.3, and extracts missing information from the 2D and 3D information required for neural network training in the feature point matcher B and the feature point upsampler C.

[0086] In this embodiment, a trained neural network such as SuperGlue or SGMNet is used as a teacher model to provide correct labels for both soft and hard labels representing missing two-dimensional and three-dimensional information. The correct data additionally generated by the correct data additional generation unit D is generated for training the output layer and loss function described above, as will be described in detail below.

[0087] The soft label extraction unit D.1 uses intermediate outputs and prediction scores of a teacher model such as SuperGlue that has been trained as a matcher as soft labels (soft targets) for the feature point matcher B to match feature points. In this embodiment, the output of the Sinkhorn layer is used as the soft labels. The operation of the soft label extraction unit D.1 can be expressed by the following equation (24).

[0088]

number

[0089] where M^ score represents the matching score from the Sinkhorn layer of the trained SuperGlue, and F^ A ,F^ B represents the intermediate output from the trained SuperGlue GNN.

[0090] The matching score M^ obtained by the above equation (24) score can be used as correct data when the feature point matcher B learns the matcher in the above equation (4) together with the loss function of the following equation (25). scoreis the matching score estimated by equation (4) above, and KL(·||·) is the Kullback-Leibler divergence loss.

[0091]

number

[0092] The upsampling ground truth data match generation unit D.2 generates ground truth matching pairs M^ for the two-dimensional upsampled feature points using the trained teacher model. up In this embodiment, the feature point extraction unit A is used to generate the positions of more feature points and their feature descriptors (K up A ,F up A ),(K up B ,F up B ) are extracted and applied to the following equation (26) to obtain the correct matching pair M^ up For example, SuperPoint can change the number of feature points to be output to 1000 or 5000.

[0093]

number

[0094] In this embodiment, M in the above formulas (20) and (21) up M^ up Then, in order to select correct data for learning upsampling feature points in the above equation (22), the information in the following equation (27) is used.

[0095]

number

[0096] The correct depth interpolation unit (D.3) generates correct depth information for the upsampled feature points for which the depth was not found by interpolation calculation. i A and P j B If the interpolation depth is expressed as the upsampling index of the ground truth data corresponding to the interpolation depth, the interpolation depth can be defined by the following equation (28).

[0097]

number

[0098] In addition, the above embodiment in In the above description, the supervised data addition and generation unit D simultaneously trains the inference models of the feature point matcher B and the feature point upsampler C, but the present invention is not limited to this, and it is also possible to train at least one of the inference models of the feature point matcher B and the feature point upsampler C.

[0099] According to the above embodiment, a large number of matching pairs can be obtained in a short time using a small-scale inference model from image pairs, thereby realizing highly accurate projective transformation based on a large number of feature points. This makes it possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the United Nations Sustainable Development Goals (SDGs). [Explanation of symbols]

[0100] A...Feature point extraction unit, B...Feature point matcher, C...Feature point upsampler, C.1...Patch division processing, C.2...Upsampling processing, C.3...Patch-based matching processing, C.4...Depth interpolation processing, C.5...Geometry-aware learning processing, D...Correct answer data additional generation unit, D.1...Soft label extraction unit, D.2...Upsampling correct answer data matching unit, D.3...Correct answer depth interpolation unit

Claims

1. a feature point extraction means for extracting feature points from a pair of images of the same subject taken from different viewpoints; a feature point matcher that applies the feature points extracted from the image pairs to a first inference model to detect initial matching pairs; a feature point upsampler that applies the feature points extracted from the image pair and the initial matching pair to a second inference model to upsample the feature points of the image pair; and a correct answer data generating means for generating correct answer data by applying the feature points extracted by the feature point extracting means to a teacher model that has previously learned the relationship between the feature points extracted from a pair of images obtained by photographing the same subject from different viewpoints and the result of feature matching, A feature point upsampling device, characterized in that an inference model of at least one of a feature point matcher and a feature point upsampler is trained using knowledge of the teacher model.

2. The feature point upsampling device according to claim 1 , wherein the intermediate output or predicted score of the teacher model is used as a soft label for the feature point matcher to match feature points.

3. The feature point upsampler a patch dividing means for dividing the initial matching set of image pairs into a plurality of patches; 2. The feature point upsampling device according to claim 1, further comprising: upsampling means for upsampling the feature points for each patch.

4. 4. The feature point upsampling device according to claim 3, wherein the feature point upsampler further comprises patch-based matching means for performing initial matching on a patch-by-patch basis between the pair of images.

5. 5. The feature point upsampling device of claim 4, wherein the feature point upsampler further comprises depth interpolation means for interpolating depths of feature points of initial matching pairs.

6. The feature point upsampling device described in claim 1, characterized in that the correct answer data generation means includes a soft label extraction unit that extracts intermediate information of the teacher model as soft labels, and the feature point matcher uses the soft labels to train the first inference model.

7. the supervised data generating means includes an upsampling supervised data providing means for providing the feature points extracted by the teacher model and a set of matching pairs thereof to the feature point upsampler as supervised data; The feature point upsampling device according to claim 3 , wherein the upsampling means trains the second inference model using the provided ground truth data.

8. the ground truth data generating means includes a ground truth depth interpolation unit that generates depths of feature points extracted by the teacher model and provides the depths to the feature point upsampler as ground truth data; The feature point upsampling device according to claim 1 , wherein the feature point upsampler trains the second inference model using the provided ground truth data.

9. 9. The feature point upsampling device according to claim 1, wherein knowledge of the teacher model is distilled to train an inference model of at least one of a feature point matcher and a feature point upsampler.

10. The computer Extract feature points from pairs of images of the same subject taken from different viewpoints, Applying the feature points extracted from the image pair to a first inference model to detect initial matching pairs; applying the feature points extracted from the image pair and the initial matching pair to a second inference model to upsample the feature points of the image pair; The feature points extracted from the image pair are applied to a training model that has previously learned the relationship between the feature points extracted from the image pair, which are images of the same subject taken from different viewpoints, and the results of feature matching, to generate correct answer data. A feature point upsampling method characterized by training at least one of the first and second inference models using knowledge of the teacher model.

11. A procedure for extracting feature points from a pair of images of the same subject taken from different viewpoints; A step of applying the feature points extracted from the image pair to a first inference model to detect an initial matching pair; applying the feature points extracted from the image pair and the initial matching pair to a second inference model to upsample the feature points of the image pair; A procedure for generating ground truth data by applying feature points extracted from the image pair to a training model that has previously learned the relationship between feature points extracted from an image pair in which the same subject is photographed from different viewpoints and the result of feature matching; A feature point upsampling program that causes a computer to execute the steps of: training at least one of the first and second inference models using knowledge of the teacher model.

Images