System and method for using pyramid and uniqueness matching priors to identify correspondences between images

Abstract

To provide a nerve network for image analysis.SOLUTION: A pair of images includes a first image 110 having a first pixel grid and a second image 120 having a second pixel grid. The first pixel grid 112 is different from the second pixel grid 122 and generates, by a nerve network module 130 having a first set of parametric variables 132, a first feature map 140 on the basis of the first image and generates a second feature map 150 on the basis of the second image; determines a first correlation volume on the basis of the first and second feature maps; iteratively determines a second correlation volume on the basis of the first correlation volume; determines a loss 170 for the first and second feature maps on the basis of the second correlation volume; generates a second set of the parametric variables by minimizing a loss function using the loss; and updates the nerve network module to include the second set of the parametric variables.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to European Patent Application No. EP22305384.4, filed on 28 March 2022, the contents of which are incorporated herein by reference.

[0002] This disclosure relates to neural networks for image analysis, and more specifically, to systems, methods, and computer-readable media for unsupervised learning of neural networks for computing local image descriptors for determining whether a pair of images depict the same scene. [Background technology]

[0003] The background art provided herein is intended to provide a general context for the content of this disclosure. Within the scope described in the background art section, no aspect of this description that is not recognized as prior art at the time of filing, nor as research by the inventors of the present invention, is explicitly or implicitly recognized as prior art contrary to the content of this disclosure.

[0004] One of the enduring challenges in computer-based image analysis is identifying corresponding pixels or portions of two images. Such image analysis problems are at the heart of numerous computer vision tasks, including large-scale visual localization, object sensing, pose estimation, Structure-from-Motion (SfM), 3D reconstruction, and Simultaneous Localization and Mapping (SLAM). All of these tasks involve identifying corresponding portions of two images that depict at least partially the same visual content.

[0005] Two different images representing the same visual content can differ by a wide range of parameters, such as the field of view of the depicted motif, the position of the motif within the image frame, the type of camera, lens, and sensor used to capture the image, lighting and weather conditions, focal length, and / or sharpness. While humans can easily distinguish between images that are remarkably different from each other, and even between parts of an image that show the same features of the depicted object or person, this task is actually somewhat more complex for computers due to differences in geometry, hue, and contrast between images.

[0006] The correspondence between images is a local image descriptor (also called a pixel descriptor) identified and extracted from the image. Local image descriptors can characterize the neighborhood of an image's pixels and provide a computer-processable data structure that allows a computing system to compare the local environments of pixels by determining the correlation between pixels based on these local image descriptors. Local image descriptors can be extracted from an image sparsely (e.g., for only selected keypoints in the image) or densely (e.g., for each individual pixel in the image).

[0007] For the quality of identified correspondences, various implementations of the process of extracting (i.e., computing) local image descriptors from images are performed. Existing learning-based approaches for extracting local image descriptors far outperform standard manual methods. Learning-based approaches are based on training procedures involving annotated training datasets containing numerous image pairs with pixel-level correspondences (e.g., dense ground-truth correspondences). Such correspondences are obtained by considering a set of numerous images for a given landmark and constructing a Structure-from-Motion (SfM) reconstruction. However, such pipelines are prone to failure and can lead to bottlenecks in the range of generateable ground truth data. This limits the potential of available image data to only image pairs from which ground truth labels can be efficiently derived.

[0008] Since a restricted training dataset directly negatively impacts the training results, it is preferable to overcome such limitations in order to maximize the potential of the image pairs available as training datasets. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Korean Registered Patent Publication No. 10-2068279 [Overview of the project] [Problems that the invention aims to solve]

[0010] To overcome the aforementioned shortcomings, a computer-implemented method for learning local descriptors unsupervised is presented. The approach of the method involves implementing two matching priors: local consistency and uniqueness of matching. The former is based on the observation that two adjacent pixels in one image may match to two pixels in another image that form a similar adjacent pair, up to a small deformation. This can generally be true at any scale. In the disclosed example, this prior is efficiently implemented through a pyramidal structure. A pyramidal non-paramatric module extracts higher-level correspondences and systematically applies the local consistency matching prior. The uniqueness prior is based on the observation that one pixel from the first image can correspond to at most one pixel in the second image. This attribute is applied to higher-level correspondences by the uniqueness matching loss. This is naturally propagated to lower-level pixel correspondences thanks to the pyramidal hierarchical structure. Consequently, the combination of the local consistency prior and the uniqueness prior allows for effective learning of local image descriptors unsupervised.

[0011] The resulting training method for a neural network to determine local image descriptors does not require any supervised learning, for example, in the form of correct correspondences between training images. The trained neural network is configured to extract the optimal local image descriptor. The extracted optimal local image descriptor (also called a PUMP descriptor) is configured to identify correspondences between parts of different images, including a pair of images of the same scene that depict the same visual content but have other image mediation variables, such as a difference in time or lighting conditions.

[0012] In one embodiment, a computer-implemented method for unsupervised neural network training for learning local image descriptors is provided. The method is a step of acquiring a pair of images depicting the same scene, the pair of images comprising a first image having a first pixel grid and a second image having a second pixel grid, the first pixel grid being different from the second pixel grid, and applying a neural network having an initial set of parameter variables to the first and second images to generate a feature map F1 for the first image and a feature map F2 for the second image, where F1 comprises a first grid of local image descriptors and F2 comprises a second grid of local image descriptors, each local image descriptor in the first grid of local image descriptors corresponds to each pixel in the first pixel grid and each local image descriptor in the second grid of local image descriptors corresponds to each pixel in the second pixel grid, and generating an initial correlation volume C based on F1 and F2. 0 This is a decision-making step, C 0 This includes local correlations between pixels in a first pixel grid and pixels in a second pixel grid, where each local correlation between pixels in the first pixel grid and pixels in the second pixel grid is determined based on the local image descriptor corresponding to the correlated pixel, and by an iterative pyramid configuration, C 0 Based on high-level correlation volume C L This is a decision-making step, C L This includes aggregated high-level correlations between iteratively constructed high-level patches of the first and second pixel grids, where each patch is a step, which is a rectangular portion of each pixel grid, and C L A step of determining the uniqueness matching loss for F1 and F2 based on, wherein the uniqueness matching loss provides a measure of the deviation of F1 and F2 from the optimal feature map, and C LThe method includes the steps of: low deviation from the optimal feature map if, based on high-level correlations, each high-level patch of the first and second pixel grids has a unique match among the high-level patches of the other pixel grids; and training the neural network by minimizing the loss function based on the uniqueness matching loss to generate an optimized set of mediating variables, thereby generating the neural network to determine the optimal local image descriptor.

[0013] One feature provides a computer-implemented method for identifying correspondences between input image pairs. The method includes the steps of: receiving an input image pair comprising a first input image and a second input image; processing the input image pair via a trained neural network to determine an optimal local image descriptor, wherein the trained neural network generates a local image descriptor for the aforementioned input image pair; extracting correspondences between parts of the input image pair based on the image descriptors, wherein each extracted correspondence maps a part of the first input image to a part of the second input image; determining, based on the extracted correspondences, whether a true match exists between the input image pairs, wherein a true match exists between the input image pairs when corresponding parts of the image pair depict the same scene; and outputting whether a true match exists between the image pairs.

[0014] In one feature, the system includes a processing module, and memory is provided that is linked to the processing module. The memory contains computer-executable instructions such that, when executed by the processing module, the processing module performs a computer-implemented method for unsupervised neural network training for learning local image descriptors and / or a computer-implemented method for identifying correspondences between pairs of input images as described herein.

[0015] In one feature, one or more computer-readable media are provided on which computer-readable instruction words are stored. When executed by a processing module (for example, including one or more processors), the computer-readable instruction words cause the processing module to perform an unsupervised neural network training method for learning local image descriptors, or a computer embodiment method for identifying correspondences between pairs of input images as described herein.

[0016] In one feature, the computer implementation method is a step of obtaining a pair of images depicting the same scene, the pair of images including a first image having a first pixel grid and a second image having a second pixel grid, the first pixel grid being different from the second pixel grid, and a step of generating a first feature map based on the first image and a step of generating a second feature map based on the second image by a neural network module having a first set of parameters, the first feature map including a first grid of image descriptors, the second feature map including a second grid of image descriptors, each local image descriptor of the first grid corresponding to each pixel in the first pixel grid, and each local image descriptor of the second grid corresponding to each pixel in the second pixel grid The method includes: determining a first correlation volume based on first and second feature maps, the first correlation volume including the correlations between (a) pixels in a first pixel grid and (b) pixels in a second pixel grid, where each correlation between pixels in the first and second pixel grids is determined based on an image descriptor corresponding to the correlated pixels; iteratively determining a second correlation volume based on the first correlation volume; determining a loss for the first and second feature maps based on the second correlation volume; generating a second set of mediating variables for a neural network module by minimizing a loss function using the loss; and updating the neural network module to include the second set of mediating variables to generate a trained neural network module.

[0017] In an additional feature, the method further includes the steps of: generating a third feature map based on a third image using a trained neural network module, wherein the third feature map includes a third grid of image descriptors; generating a fourth feature map based on a fourth image, wherein the fourth feature map includes a fourth grid of image descriptors; and identifying a first portion of the third image corresponding to a second portion of the fourth image based on the third and fourth grids.

[0018] In addition, the second image is a composite version of the first image generated by data augmentation, and the step of iteratively determining the second correlation volume includes the step of determining the second correlation volume using an iterative pyramid construction.

[0019] In an additional feature, the method further includes the step of determining a second loss based on a first grid of image descriptors, a second grid of image descriptors, and the ground truth correspondence between the first and second images, and the step of generating a second set of parameters for the neural network module, which includes the step of generating a second set of parameters for the neural network module by minimizing a loss function using the loss and the second loss.

[0020] In the additional features, the step of generating a second set of parameters for the neural network module includes the step of generating a second set of parameters for the neural network module by minimizing a loss function based on the sum of the loss and the second loss.

[0021] In terms of additional features, the sum is a weighted sum.

[0022] In an additional feature, the step of determining a second correlation volume includes the steps of generating a first level correlation volume based on a first level correlation between a first level patch of a first pixel grid and the first level patch of a second pixel grid, and iteratively aggregating the N+1 level correlations of the Nth level correlation volume to the N+1 level correlations between the N+1 level patches of the first pixel grid and the N+1 level patches of the second pixel grid for N between 1 and L-1.

[0023] In the additional features, the N+1 level patches, which include adjacent N level patches in each pixel grid, and the aggregated N level correlations correspond to the adjacent N level patches of the correlated N+1 level patches.

[0024] In an additional feature, the step of generating a first level correlation volume includes determining a first level correlation between a first level patch of a first pixel grid and a first level patch of a second pixel grid by the average sum of the correlations between corresponding pixels in the first level patch of the first pixel grid and the first level patch of the second pixel grid.

[0025] In terms of additional features, each N+1 level patch contains a 2x2N-level patch for each pixel grid.

[0026] In the additional features, the step of determining the Nth level correlation volume includes the step of performing a rectification transformation for each N-level correlation of the Nth level correlation volume.

[0027] In the additional features, the first correlation volume has a first dimension corresponding to the first dimension of the first feature map, and a second dimension corresponding to the second dimension of the first feature map.

[0028] In the additional features, the step of generating a first correlation volume includes the steps of subsampling the first feature map by default factors in the first and second dimensions, and generating a subsampled feature map having a third dimension smaller than the first dimension and a fourth dimension smaller than the second dimension.

[0029] In the additional features, the step of subsampling the first feature map includes dividing the first pixel grid into non-overlapping patches, each patch containing multiple pixels, and determining a descriptor for each patch based on the image descriptors corresponding to the pixels of that patch, wherein the descriptor represents all the pixels of the corresponding patch in the subsampled feature map.

[0030] In the additional features, the step of determining the first correlation volume includes determining the correlation between each patch of the first pixel grid and each pixel of the second pixel grid, where each correlation is determined based on one descriptor representing each patch in a subsampled feature map, and one descriptor in the second feature map corresponds to the correlated pixel in the second pixel grid.

[0031] In the additional features, each patch has a size of 4x4 pixels, the first dimension is four times the third dimension, and the second dimension is four times the fourth dimension.

[0032] In an additional feature, the method further includes the steps of: extracting correspondences between parts of a second pair of images using a second set of trained neural network modules and mediation variables; determining, based on the extracted correspondences, whether the parts of the second pair of images contain the same scene; and outputting an indicator indicating whether the parts of the second pair of images contain the same scene.

[0033] In one feature, the system includes a neural network module configured to generate a first feature map based on a first image of an image pair and a second feature map based on a second image of an image pair, using a trainable parameter, wherein the first feature map includes a first grid of image descriptors and the second feature map includes a second grid of image descriptors, and at least a portion of the first image includes a scene and at least a portion of the second image includes a scene; a correlation module configured to determine a loss based on the first and second feature maps; and a training module configured to train a trainable parameter by minimizing the loss.

[0034] In addition, the training module is configured to train trainable parameters without labels indicating the correspondence between parts of the first and second images.

[0035] In addition, the matching module is configured to, after training, extract the correspondence between feature maps generated by the neural network module based on the received image, determine whether the received image contains the same scene based on the correspondence, and output an indicator indicating whether the received image contains the same scene.

[0036] One feature of a computer implementation method for identifying correspondences between input image pairs includes the steps of: receiving an input image pair including a first input image and a second input image; processing the input image pair using a trained neural network by performing an unsupervised learning method for local image descriptors, wherein the trained neural network generates local image descriptors for the input image pair; extracting correspondences between parts of the input image pair based on the image descriptors, wherein each extracted correspondence maps a part of the first input image to a part of the second input image; determining whether a true match exists between the input image pair based on the extracted correspondences, wherein a true match exists between the input image pair when corresponding parts of the image pair depict the same scene; and outputting whether a true match exists between the image pair. Steps to obtain a pair of training images, the pair of training images comprising a first training image having a first pixel grid and a second training image having a second pixel grid, the first pixel grid being different from the second pixel grid, and applying a neural network having an initial set of mediating variables to the first and second training images to generate a first feature map for the first training image and a second feature map for the second training image, the first feature map comprising a first grid of local image descriptors, the second feature map comprising a second grid of local image descriptors, each local image descriptor in the first grid of local image descriptors corresponding to each pixel in the first pixel grid, and each local image descriptor in the second grid of local image descriptors corresponding to each pixel in the second pixel grid, and determining an initial correlation volume based on the first and second feature maps, the initial correlation volume isThe process includes the steps of: determining local correlations between pixels in a first pixel grid and pixels in a second pixel grid, where each local correlation between pixels in the first and second pixel grids is determined based on a local image descriptor corresponding to the correlated pixel; iteratively determining a second correlation volume based on an initial correlation volume, wherein the second correlation volume includes aggregated high-level correlations between iteratively constructed high-level patches of the first and second pixel grids; determining a uniqueness matching loss for the first and second feature maps based on the second correlation volume, wherein the uniqueness matching loss provides a measure of the deviation of the first and second feature maps from the optimal feature map; and training the neural network by minimizing a loss function based on the uniqueness matching loss to generate an optimized set of mediating variables.

[0037] One feature of a computer implementation method for unsupervised neural network training is a step of acquiring a first pair of images depicting the same scene, the image pair comprising a first image having a first pixel grid and a second image having a second pixel grid, the first pixel grid being different from the second pixel grid, and applying a neural network having an initial set of mediating variables to the first and second images to generate a first feature map for the first image and a second feature map for the second image, the first feature map comprising a first grid of local image descriptors and the second feature map comprising a second grid of local image descriptors, each local image descriptor in the first grid of local image descriptors corresponding to each pixel in the first pixel grid and each local image descriptor in the second grid of local image descriptors corresponding to each pixel in the second pixel grid, and based on the first and second feature maps, an initial correlation coefficient is generated. A step of determining a volume, wherein the initial correlation volume includes local correlations between pixels in a first pixel grid and pixels in a second pixel grid, and each local correlation between pixels in the first pixel grid and pixels in the second pixel grid is determined based on the local image descriptor corresponding to the correlated pixel; a step of iteratively determining a second correlation volume based on the initial correlation volume, wherein the second correlation volume includes aggregated correlations between iteratively constructed patches of the first and second pixel grids; a step of determining a uniqueness matching loss for the first and second feature maps based on the second correlation volume, wherein the uniqueness matching loss provides a measure of the deviation of the first and second feature maps from the optimal feature map; and a step of training a neural network by minimizing a loss function based on the uniqueness matching loss to generate an optimized set of mediating variables, thereby adapted to determine the optimal local image descriptor.The process includes the step of generating a trained neural network.

[0038] In the additional features, the step of iteratively determining a second correlation volume based on a first correlation volume is determined by an iterative pyramid construction, where each patch is a rectangular portion of each pixel grid.

[0039] In the additional features, a low measure of deviation from the optimal feature map occurs when, based on the high-level correlation of the second correlation volume, each patch of the first and second pixel grids has a unique match among the patches of the other pixel grids.

[0040] Additional fields of application of the content of this disclosure are evident from the specific content for carrying out the invention, the claims, and the drawings. The specific content for carrying out the invention and the examples are illustrative, however, and do not limit the scope of the content of this disclosure. [Brief explanation of the drawing]

[0041] The contents of this disclosure should be fully understood from the specific details for carrying out the invention and the accompanying drawings. [Figure 1] This diagram illustrates the process of training a neural network module to determine the correspondence between pairs of images. [Figure 2] This diagram illustrates the process of determining the correspondence between image pairs by applying trained neural network modules to images. [Figure 3] This figure illustrates the correlation module shown in Figure 1. [Figure 4] Figure 3 illustrates the configuration of the repeating pyramid, showing an example of the arrangement from lower-level image patches to upper-level image patches. [Figure 5] This is a functional block diagram illustrating methods for training neural networks. [Figure 6]This is a functional block diagram illustrating a method for determining the correspondence between pairs of images. [Figure 7] This is a series of graphs showing a comparison of experimental sparse matching evaluation results for the HPatches dataset across numerous different technologies. [Figure 8] This diagram illustrates an exemplary system architecture capable of performing the disclosed functions.

[0042] In drawings, reference numerals may be reused to identify similar and / or identical elements. [Modes for carrying out the invention]

[0043] In this specification, unless explicitly specified in the context, a singular expression may include a plural expression, and a plural expression may include a singular expression.

[0044] The first step to solving the aforementioned problem may depend on ground-labeled correspondences between images, but can be provided by a purely self-supervised learning approach, which can generate correspondences as part of the approach with limited effort. A purely self-supervised learning approach can use synthetically generated image pairs in the training step, and the second image is obtained by applying a known transformation such as random homography, hue jittering, or style transfer to the first image.

[0045] However, homography may not be able to model the entire range of possible transformations between actual image pairs. At the same time, weakly supervised learning methods can demonstrate the ability to be trained, for example, from known camera poses. However, this can only be achieved by complex acquisition setups that involve the arrangement of sensors based on various modalities (inertial measurement units (IMUs) or GPS (Global Positioning Systems)) or again rely on SfM reconstruction.

[0046] Unsupervised learning of local descriptors involves extracting features for more images for training, or iterative training of the descriptor at high cost and expensive model fitting, and can include periodic consistency constraint conditions for a number of images.

[0047] Systems and methods for unsupervised neural network training for learning local image descriptors are described herein. A number of examples and specific details are presented to provide a complete understanding of the examples described for purposes of illustration. Examples of claims can include some or all of the features of these examples alone or in combination with other features described hereinafter, and can further include modifications and equivalents of the features and concepts described herein. The illustrated examples are described with reference to drawings in which similar elements and structures are denoted by similar reference numerals.

[0048] The present invention relates to a method, system, and computer-readable medium for unsupervised neural network training for learning an image descriptor adapted to determine corresponding portions from images showing the same visual content. This application achieves unsupervised neural network module training for learning an image descriptor by configuring a neural network such that when applied to a given image H×W of dimension I, the neural network module extracts a very discriminative but strict local image descriptor for each pixel of I. To obtain a neural network module capable of extracting such a local image descriptor, first, the neural network module must be trained. By way of example, the local image descriptor extracted from an image by the trained neural network module is represented by a PUMP (Pyramidal and Uniqueness matching Priors) descriptor.

[0049] FIG. 1 shows a functional block diagram of an exemplary training system for training (or a training step) of a neural network module f θ (130). The neural network module f θ(130) has multiple internally learnable parameters θ(132). The training module 104 trains the neural network module 130 as discussed herein. The training step of the neural network module 130 optimizes the learnable parameters of the neural network module by training the neural network module on a training set of image pairs.

[0050] In supervised learning / training approaches, the expected outcome for each image pair in the training set is known (for example, the corresponding portion of each image pair in the training set is known as the binary ground truth label for each image pair). However, this application includes unsupervised learning / training of the neural network module 130, which means that the learning / training does not involve supervision (for example, in the form of known ground truth correlations between images). The training data for unsupervised learning includes image pairs that depict at least partially the same visual content.

[0051] Training begins with the neural network module 130, which contains an initial value set for the parameter 132. The initial value set can be configured in various ways. For example, the initial values ​​are corrected by the training module 104 during the initial training of the neural network module 130. Another option is to utilize setting the parameter 132 as a default common value for the parameter θ(132). In various implementations, the training module 104 can configure the initial value set as random values ​​for each of the multiple parameter θ(132).

[0052] In the first step of training, the training module 104 inputs two different images 110, 120 of an image pair into the neural network module 130 (which has an initial set of values ​​for the parameter 132). The two images in the pair depict the same visual content in all or at least partially (e.g., the same scene, the same object, the same person, or the same architectural structure). Image I1(110) may include a pixel grid 112 having height H1 and width W1, containing individual pixels 114 of H1×W1. Image I2(120) may include a pixel grid 122 having height H2 and width W2, containing individual pixels 124 of H2×W2. Although the two images depict the same visual content, the pixel grids 112, 122 may be different from each other, for example, because the images are captured under different lighting conditions, different camera positions and field of view, different cameras, and different sensors and lenses.

[0053] Mathematically, neural network module 130 is function f θ :I→F I It is described as follows, where I represents an image.

number

[0054] Therefore, when the neural network module 130 is applied to image I1(110), a feature map F1(140) is generated or extracted from / based on the image. The feature map 140 includes a grid of local image descriptors having height H1 and width W1, each containing individual local image descriptors 144 of H1×W1, where each image descriptor 144 corresponds to a pixel 114 of the pixel grid 112.

number

number

[0055] The correlation module 160 receives feature maps F1(140) and F2(150) and performs a series of steps on the received feature maps to determine the loss 170. The loss 170 can be determined from a single pair of images or as an average over multiple pairs of images. In the latter case, the training module 104 generates a feature map for each image by inputting each image from multiple pairs of images into the neural network module 130 in a manner similar to that described for I1 and I2. Based on the feature map of each image, the correlation module 160 determines the individual loss for each pair of images and determines the average loss for multiple pairs of images from the multiple individual losses for the individual pairs. The specific steps performed by the correlation module 160 are described in detail below with reference to Figure 3.

[0056] The loss represents the value of the loss function (also called the cost function), which is a function that maps the value of one or more variable values ​​(e.g., real numbers) to an event or event, representing a portion of the cost associated with that event or value. Through training, the training module 104 is based on the loss 170 (the value of the loss function), or the neural network module f that minimizes it. θ Modify the learnable parameter θ(132) of (130). Such optimization of the loss function (which converges to a minimum) is performed on the neural network module f θThis is accomplished by backpropagating the so-called "loss gradient," which is obtained by the training module 104 from the partial derivative of the loss function with respect to the learnable parameter θ(132) of (130). Such a loss gradient is used to generate an even lower loss 170 in the next iteration of the training step, by backpropagating the neural network f θ The training module 104 backpropagates to each of the multiple learnable parameters θ(132) in such a way that it is used to modify (or adapt or update) the learnable parameters θ(132) of (130).

[0057] In the next iteration, the neural module 130 generates modified local feature maps F1(140) and F2(150), respectively, containing grids of modified local image descriptors for images I1 and I2, and the feature maps and thereby the local feature descriptors are determined by the neural module 130 using a modified learnable parameter θ(132). Such modified local feature descriptors are further adapted to identify corresponding parts from the images in an image pair, leading to even lower loss 170 and smaller adaptations of the learnable parameter θ(132) in the next backpropagation of the loss gradient until the loss function converges to a minimum loss 170 and no further adjustment / modification of the learnable parameter 132 is required. Once the loss function converges (to a minimum), the neural module 130 can be said to be trained, which can then be applied to the image pair to identify corresponding parts from the images in the image pair.

[0058] Figure 2 illustrates a functional block diagram of a system that performs the process of determining the correspondence between image pairs. A trained neural network module f has multiple internal parameters θ_T. θT(230) may be the result of the training steps described with reference to Figure 1 (the trained neural module 130). The trained neural module 230 receives images 210 and 220 and generates a feature map 240 for image 210 and a feature map 250 for image 220. The process of applying the trained neural module 230 to the images and generating feature maps for the images is the same as the respective processes in the training steps described with reference to Figure 1. Thus, the generated feature maps 240 and 250 each contain a grid of local image descriptors, each containing multiple individual PUMP descriptors, the feature maps of which each feature map corresponds to individual pixels of the image from which they were generated.

[0059] Subsequently, feature maps 240 and 250 are provided to a matching module 260 configured to extract correspondences 270 between parts of images 210 and 220 based on the PUMP descriptors of feature maps 240 and 250. Each correspondence 270 maps a part of image 210 (e.g., a pixel or patch / group of pixels) to a part of image 220 (e.g., a pixel or patch). If the corresponding parts of images 210 and 220 depict the same feature (e.g., an object) of the depicted scene, the correspondence defines a true match.

[0060] The matching module 260 can determine correspondences 270 for images 210 and 220 by using the DeepMatching algorithm, for example, as described in Revaud et al., Hierarchical deformable dense matching, IJCV, 2016, on feature maps 240 and 250, thereby extracting dense or semi-dense correspondences 270 between parts of images 210 and 220, which are included herein in whole. The DeepMatching algorithm can be adapted and used herein by substituting the basic descriptor of the DeepMatching algorithm for the PUMP descriptor of feature maps 240 and 250. While an example of the DeepMatching algorithm can be provided, this application is also applicable to other methods of determining correspondences. In an example where one semi-dense correspondence extraction image of images 210 and 220 is divided into atomic patches (e.g., non-overlapping 4x4 pixel patches), one correspondence 270 is output for each atomic patch.

[0061] In various implementations, the matching module 260 can apply rare keypoint-based matching techniques / algorithms to feature maps 240 and 250 to extract correspondences 270 between keypoints in images 210 and 220. Examples of rare keypoint-based matching techniques include SIFT (Lowe, “Object recognition from local scale invariant features”, ICCV, 1999), ORB (Rublee et al., “ORB: an efficient alternative to SIFT or SURF”, ICCV, 2011), R2D2 (Revaud et al., “R2D2: repeatable and reliable detector and descriptor”, NeurIPS, 2019), or SuperPoint (DeTone et al., “Self-supervised interest point detection and description”, CVPR Workshops, 2018), each of which is included herein by general reference. Regardless of the specific technique / algorithm applied to extract the correspondence between images 210 and 220, the applied technique is applied to the present invention by replacing the base descriptor of each technique with the PUMP descriptor of feature maps 240 and 250. The matching module 260 outputs an indicator indicating whether or not a true match exists between parts of an image. For example, the matching module 260 may display / show a visual indicator on a display or output an auditory indicator via a speaker, and the visual and / or auditory indicators indicate whether or not a true match exists.

[0062] The performance of correspondence extraction, as well as the quality of the determined correspondences, may depend on the PUMP descriptor, which is the result of applying the trained neural network 230 to images 210 and 220. Since the training of the neural network module 130 is performed by backpropagation based on loss 170, the result of the training step, i.e., the trained neural network module 230, and by extension the PUMP descriptor, depends on a specific method for determining loss 170, in particular the step performed by the correlation module 160 to determine loss 170.

[0063] Figure 3 is a functional block diagram showing an example of a correlation module 300 for determining loss during the training steps for the neural network 130. The correlation module 300 may be an example of a correlation module 160 as described with reference to Figure 1. The correlation module 300 receives feature maps F1(140) and F2(150), which are the result of applying the neural network module 130 to images I1 and I2, respectively, as described above with reference to Figure 1.

[0064] In the first selective step, the feature map F1(140) is subsampled 310. Subsampling 310 can mean reducing the size of the feature map F1(140) by replacing multiple entries in the feature map F1(140) with only single values. For example, the grid of local image descriptors 142 of the feature map F1(140) can be divided into non-superimposed rectangular atomic parts having a height of n entries (e.g., n=2, 3, 4, 5, 6, ...) and a width of m entries (e.g., m=2, 3, 4, 5, 6, ...). In some embodiments, the rectangular parts are quadratic equations with n=m, and in a particular example, n=m=4. Each part contains an n×m local image descriptor, each of which is described as a d-dimensional vector. Subsampling 310 replaces the n×m local image descriptors of each atomic part with a single local image descriptor for the atomic part. This is achieved by selecting a single value for each entry in the d-dimensional vector based on the n×m values ​​of each entry in the original local image descriptor of the atomic part. The subsampling module 310 can select a single value from the n×m original values ​​in an appropriate manner. In one example, the maximum value of each n×m original value is selected by the subsampling module 310 for each entry in the d-dimensional vector, as a result of the local image descriptor. In another example, the average value of each n×m original value is selected by the subsampling module 310 for each entry in the d-dimensional vector, as a result of the local image descriptor. In one example, the vector value of the local image descriptor at a specific position within each part is selected by the subsampling module 310 to substitute for each n×m original value. Regardless of the method used to select a single local image descriptor for each part, subsampling the feature map F1(140) generates a reduced feature map F1' containing a reduced grid of local image descriptors, which has dimensions H1' × W1', where H1' = H1 / n and W1' = W1 / m.

[0065] In the next step, the initial 4D (four-dimensional) correlation volume C 0 However, based on feature maps F1(140) and F2(150), the volume module 320 may determine the initial 4D correlation volume C in an example of a feature map F1(140) being subsampled to generate a reduced feature map F1'. 0 This can be determined by the volume module 320 based on the reduced feature map F1' and feature map F2(150). 0 In this case, each entry is a pixel p=(x) of the pixel grid 112 of image 110. p ,y p )(114) and the pixel q=(x) of the pixel grid 122 of image 120. q ,y q The correlation between (124) can be shown. The correlation is C (p,q) This is represented by the initial 4D correlation volume C when the feature map F1(140) is subsampled to produce a reduced feature map F1'. 0 Each of the values ​​in (320) is center p=(x p ,y p In the pixel grid 112 of image 110, where ) is the atomic part (or patch) and in the pixel grid 122 of image 120, pixel q = (x q ,y q ) can be shown to show a correlation between them. Such a correlation can be similarly shown to C (p,q) It can be displayed as follows. For clarity in the following explanation, only p is referred to as pixel 114 of the pixel grid 112 of image 110. However, if feature map F1(140) is subsampled to produce a reduced feature map F1', then p = (x p ,y pIt must be understood that ) can refer to the center of an atomic part (or patch) in the pixel grid 112 of image 110. In the pixel grid 112, each pixel (or part) p(114) corresponds to the local feature descriptor D(p)(144) of the grid of local feature descriptor 142 of feature map F1(140) (or each reduced grid of local feature descriptors of the reduced feature map F1'). Similarly, in the pixel grid 122, each pixel q(124) corresponds to the local feature descriptor D(q)(154) of the grid in local feature descriptor 152 of feature map F2(150). Initial correlation volume C 0 (320) represents all possible pixel-pixel correlation values ​​C 0 Includes (p,q). Correlation value C 0 (p,q) can be determined (for example, by correlation module 318) by calculating similarity values ​​based on D(p) and D(q) using the following equation:

number

[0066] Each similarity value can be a real number representing a measure of the similarity between local image descriptors for D(p) and D(q). An appropriate method is used to calculate the similarity value for a pair of local image descriptors. In one example, each similarity value is determined by the correlation module as the cosine similarity between the pair of local image descriptors for D(p) and D(q), based on the scalar product of the respective d-dimensional vectors representing the local image descriptors. In one example, each value C 0 (p,q) are numbers within the range of 0 and 1. The initial correlation volume 320 can have dimensions H1 × W1 × H2 × W2. If feature map 140 is subsampled to produce a reduced feature map F1', the initial correlation volume C 0(320) can have dimensions H1' × W1' × H2 × W2.

[0067] Initial correlation volume C 0 Once (320) is determined, the iterative pyramid configuration 330 is determined by the initial correlation volume C 0 Based on this, the configuration module 328 performs the task, and high-level correlation volume C L (340) is generated. The iterative pyramid configuration 330 has a higher level of correlation volume C for 1 ≤ l ≤ L. 1 Includes 4D aggregation 334 for iteratively determining. In the first iteration, the correlation volume C at the first level 1 This is determined by the configuration module 328. First level correlation volume C 1 This is the correlation value C, which shows the correlation between the first level patches of pixel grid 112 and pixel grid 122. 1 The patch includes (p,q). Generally, a patch can be a rectangular portion of a pixel grid (e.g., pixel grid 112 or pixel grid 122). For the sake of discussion, the following describes a patch with a width and height of 2 (displayed as patch size 2x2), including a 2x2 pixel or previously iterated 2x2 patch of a pixel grid. However, the size of a particular patch is not limited thereto, and this application is applicable to other patch sizes (e.g., 2x3, 3x2, 3x3, or 4x4). The center of the patch in pixel grid 112 is p=(x p ,y p ) is displayed, and the center of the 122-pixel grid patch is q=(x q ,y q It is represented as ). For consistent notation, a 0-level patch is considered as a pixel in each pixel grid. Using this notation, an l-level patch has a 2x2 children patch of level l-1 for 1≦l≦L. The vector from the center of the l-level patch to the i(l-1)-level children patch of the l-level patch is v i l-1It is represented as follows, where i=0,...,3. For example, each vector is as follows:

number

number

[0068] Aggregation 334 can be performed by the constructor module 328 using a 4D convolution with a fixed rare kernel whose non-zero values ​​encode parent-child relationships in the pyramid. To date, this formula can handle rigid transformations where all child patches strictly conform to the boundaries of the parent patch. Some examples involve extensions of the aggregate that allow a predetermined degree of local deformation. In such examples, at each iteration period, a small local deformation of the child patches is considered at each iteration level by an additional 2D max pooling step 332 previously performed by the constructor module 328 of the 4D aggregate 334. The 2D max pooling can be performed with a kernel fitted by the two dimensions (e.g., 3x3 in size) of the pixel grid 122 of the second image I2(120).

[0069] In one example, the transformation can be considered by adopting aggregation 334, and therefore the aggregation is at position p i l-1 (At this time,

number

number

[0070] In one example, position p i l-1 (l-1)-level patch at position p i l-1 Neighborhood Ω centered on i The deformation can also be taken into account by averaging with respect to the correlation with the (l-1)-level patch at the position.

[0071]

number

[0072] q i l-1 Ω in the neighborhood i The number of (l-1)-level patches is |Ω i It is displayed as |. In one example, the neighbor Ω i The width and height are the default fractions of the width and height of the Children(l-1)-level patch (e.g., 1 / 4, 1 / 2, 1).

[0073] In each iteration, rectification 336 may be selectively performed by the component module 328 to strengthen consistent correlations and discard provisional correlations. Rectification 336 applies power nonlinearity x → max(0,x) to the output correlation values ​​of the 4D aggregate 334 from the end of each iteration. γThis can be embodied in the following. The specific value of γ can be adjusted to suit the situation and should not be analyzed as limiting the present invention. In one example, γ = 1.5.

[0074] Mathematically, the containment field of the parent patch is doubled by the x- and y-dimensions at each pyramidal level, so the size of the overall image is quickly reached, and at this point the aggregation process naturally terminates. Figure 4 illustrates a deformable pyramidal structure performed by the iterative pyramidal configuration 330. Figure 4 illustrates patches at three levels of the iterative pyramidal configuration 330. At level l+2 (the highest level in Figure 4), four neighborhood (l+2)-level patches 412, 414, 416, and 418 are shown. Parent-level correlation volume C (l+2) The correlations are aggregated from the four children's level correlations, each with a small tolerance for deformation across all levels. The (l+2)-level correlation for (l+2)-level patch 412 is aggregated from the (l+1)-level correlations for (l+1)-level patches 422, 424, 426, and 428. Similarly, the (l+1)-level correlation for (l+1)-level patch 422 is aggregated from the l-level correlations for l-level patches 432, 434, 436, and 438.

[0075] Referring again to Figure 3, C 0 The output of the iterative pyramid configuration applied to (320) is the integrated high-level correlation volume C L (340). C corresponds to the dimension of the pixel grid 122 of I2(120). L The third and fourth dimensions of (340) are reduced by a factor of 2^L, where L is the number of iterations (or pyramid levels) in the iteration pyramid configuration 330. The reduction factor may differ from other selections of patch size in the iteration pyramid configuration 330 and should not be analyzed as limiting. In one embodiment, C L (340) can have dimensions H1 × W1 × H2 × W2. Once the selective subsampling step 310 is performed, C L(340) can have dimensions H1' × W1' × H2 × W2.

[0076] Integrated high-level correlation volume C L This shows the correlation between highly deformable high-level patches. High-level patches centered on pixel p can have unique matches in image I2. That is, C L There can only be one q such that (p,q) has a high value, but all other correlations for Q'≠q are close to 0. Such constraints may be impractical for pixel-level descriptors due to repetitive patterns, unstructured general regions, or severe shape changes, but C L This could be a natural attribute of the high-level patch (340). The larger the patch, the easier it is to resolve ambiguities and complex correspondences due to the larger context.

[0077] These considerations lead to the recommendation of a unique matching attribute for all high-level patches. L Based on (340), the loss module 348 determines the uniqueness matching loss 350. The uniqueness matching loss 350 is the output value of the correlation module 300, and shows an example of the loss 170 illustrated in Figure 1. Since each high-level patch is dynamically constructed during test time by a subset of deformable pixel-level correlations, this loss is automatically backpropagated to the optimal pixel correlation that fits the iterative pyramid configuration 330 and is therefore transmitted to the pixel-level local image descriptor.

[0078] Next, we show an exemplary embodiment of the unique matching loss 350. The resulting loss is such that the high-level patch of I1(110) has only one matching high-level patch from I2(120), C L Other examples are within the scope of this application, insofar as they compensate for the uniqueness of matching between high-level patches. For discussion, uniqueness matching loss 350, C L Exemplary examples of this are formally size

number

number

number

[0079] Here, ε is used as a normalization term that deals with the closed region where all correlations are close to 0. The value of ε can be set to suit the situation and should not be considered a restriction. In one example, the value of ε is 0.03. The intrinsic loss can then be expressed as follows:

number

[0080] The expression H1×W1 in equation 6 is replaced by H1'×W1' if a selective subsampling step 310 is performed to generate a reduced feature map F1'. -L Considering the l1-normalized point, the uniqueness loss is close to 1, except for one row (i.e., one high-level patch), C -L It is specifically recommended that all values ​​be close to 0.

[0081] In one example, the asymmetric loss L_U (F1,F2) is averaged over each pair of images in both directions:

number

Number

[0082] The average unsupervised loss L U - can be calculated for all image pairs in a portion of the batch of image pairs. The remaining portion of the batch of image pairs can be synthetically generated via augmentation of the standard data of a single image in a self-supervised manner. In the case of the latter pairs where a dense ground truth relationship is obtained via augmentation, a pixel-wise ranking loss L P (e.g., average precision loss L AP ) is used. The average loss L P - can be determined for all image pairs in the remaining portion of the batch of image pairs. The final loss L can be calculated as a weighted sum of the average losses. In one example, L is

Number

[0083] Figure 5 is a flowchart of a computer embodiment method 500 for unsupervised training of a neural network module for learning local image descriptors. The method is embodied as a training step for a neural network (e.g., neural network 130), as illustrated in Figures 1 and 3. The method begins in step 510, in which a pair of training images are acquired. The step of acquiring training images involves receiving the images from any memory location where they are stored (e.g., a local or remote computing system, a local or remote database, cloud storage, or any other known memory location). Alternatively, the images can be automatically retrieved from the memory location. The images in the acquired image pair contain corresponding parts related to the same features of the scene in which both images are depicted, by depicting the same scene or visual content (e.g., the same object, the same person, the same architectural structure, and / or the same natural structure). No close or rare ground truth correspondences are known for the images in previous image pairs in the training step. The acquired image pair includes a first image (e.g., image 110) having a first pixel grid (e.g., pixel grid 112) and a second image (e.g., image 120) having a second pixel grid (e.g., pixel grid 122). The pixel grids differ in at least one embodiment. The first pixel grid may have different horizontal and / or vertical dimensions with respect to the second pixel grid. Additionally or alternatively, the pixel values ​​of at least a portion of the pixels in the first pixel grid may differ from the pixel values ​​of each pixel in the second pixel grid.

[0084] In step 520, the neural network module receives a first image and generates a feature map F1 (e.g., feature map 140) based on the first image. Additionally, the neural network module receives a second image and generates a feature map F2 (e.g., feature map 150) based on the second image. The neural network module has an initial set of learnable internal parameters (e.g., parameter 132). Feature map F1 contains a first grid of local descriptors (e.g., a grid of local descriptor 142), where each local descriptor in the first grid of local descriptors (e.g., local descriptor 144) corresponds to a pixel in the first pixel grid. Each local descriptor in the first grid of local descriptors is a computer-processable description of the local environment of the corresponding pixel in the first pixel grid. Feature map F2 includes a second grid of local descriptors (e.g., grid 152 of local descriptors), where each local descriptor in the second grid of local descriptors (e.g., local descriptor 154) corresponds to a pixel in the second pixel grid. Each local descriptor in the second grid of local descriptors is a computer-processable description of the local environment of the corresponding pixel in the second pixel grid. Local image descriptors can be represented as d-dimensional vectors, which are extracted from the pixel grid by applying a neural network to each image.

[0085] In step 530, the initial correlation volume C 0 (For example, the initial correlation volume 320) is determined by the correlation module 318 based on F1 and F2. Initial correlation volume C 0includes the local correlation relationship between the pixels of the first pixel grid and the pixels of the second pixel grid. Based on the first corresponding local image descriptor of the first grid of the local image descriptor and the second corresponding local image descriptor of the second grid of the local image descriptor, the respective local correlation relationships are determined for the pixels of the first pixel grid and the pixels of the second pixel grid. The first corresponding local image descriptor corresponds to the pixels of the first pixel grid, and the second corresponding local image descriptor corresponds to the pixels of the second pixel grid. In one example, the correlation relationship C 0 (p,q) between the pixel p of the first pixel grid and the pixel q of the second pixel grid is determined by the similarity between the local image descriptors D(p) and D(q). D(p) corresponds to p and is included in the first grid of the local image descriptor. D(q) corresponds to q and is included in the second grid of the local image descriptor. The similarity between the local image descriptors D(p) and D(q) can be determined by cosine similarity based on the scalar product of the d-dimensional vectors representing D(p) and D(q).

[0086] In step 540, the high-level correlation relationship volume C L (e.g., the high-level correlation relationship volume 340) is determined by the configuration module 328 based on C 0 using an iterative pyramid structure (e.g., the iterative pyramid structure 330). The high-level correlation relationship volume C L includes the aggregated high-level correlation relationships between the repeatedly configured high-level patches of the first and second pixel grids. Each high-level patch of the first pixel grid can be a rectangular portion of the first pixel grid. Each high-level patch of the second pixel grid can be a rectangular portion of the second pixel grid. C L is C 0It can be configured iteratively starting from. The first iteration part is for the first level patch of the first pixel grid and the first level correlation relationship between the first level patches of the second pixel grid. C for the local correlation relationship of 0 aggregates the local correlation relationships of to generate a first level correlation relationship volume C 1 . The first level correlation relationship between the first level patch of the first pixel grid and the first level patch of the second pixel grid can be determined by the average sum of the local correlation relationships between corresponding pixels in the first level patch of the first pixel grid and the first level patch of the second pixel grid. Each of the local correlation relationships from the average sum of the local correlation relationships can have the maximum correlation relationship value among a plurality of adjacent local correlation relationship values.

[0087] For the subsequent (n + 1)-th iteration part where 1 ≦ n < L, C for the (n + 1)-level correlation relationship between the (n + 1)-level patch of the first pixel grid and the (n + 1)-level patch of the second pixel grid n generates a (n + 1)-level correlation relationship volume C (n+1) by aggregating the n-level correlation relationships of. The (n + 1)-level patches can each include adjacent n-level patches of each pixel grid. The aggregated n-level correlation relationships correspond to the adjacent n-level patches of the correlated (n + 1)-level patches. The (n + 1)-level correlation relationship between the (n + 1)-level patch of the first pixel grid and the (n + 1)-level patch of the second pixel grid can be determined by the average sum of the n-level correlation relationships between the corresponding n-level patches of the (n + 1)-level patch of the first pixel grid and the (n + 1)-level patch of the second pixel grid. Each of the n-level correlation relationships from the average sum of the n-level correlation relationships can have the maximum correlation relationship value among a plurality of adjacent n-level correlation relationship values. [[ID=*13]] [[ID=*14]]

[0088] [[ID=*15]] In step 550, C LBased on this, the uniqueness matching loss (e.g., uniqueness matching loss 350) is determined by the loss module 348 for F1 and F2. The uniqueness matching loss provides a measure of the deviation of F1 and F2 from the optimal feature map. L Based on the high-level correlation, if each high-level patch of the first pixel grid has a unique (i.e., only one) matching among the high-level patches of the second pixel grid, and each high-level patch of the second pixel grid has a unique (i.e., only one) matching among the high-level patches of the first pixel grid, then the uniqueness matching loss is configured to show a low deviation from the optimal feature map. Therefore, for each high-level patch of the first and second pixel grids, the C corresponding to each patch is L Only one high-level correlation has a high correlation value, and each high-level patch corresponds to C L If all other high-level correlations have low correlation values, the uniqueness matching loss will have a low value. High correlation values ​​are close to 1, while low correlation values ​​are close to 0.

[0089] In step 560, the neural network module is trained to generate an optimized set of parameters for the neural network module by minimizing a loss function based on the uniqueness matching loss. Minimizing the loss function may involve performing backpropagation based on the loss function to determine a new set of parameters that achieves an even lower uniqueness matching loss in subsequent training iterations. Training the neural network module may further involve generating a trained neural network module configured to determine local image descriptors by iterating the method with the optimized set of parameters for a predetermined number of times, or until the optimized set of parameters converges.

[0090] Figure 6 is a flowchart illustrating an exemplary computer embodiment method 600 for determining corresponding parts of an image using a trained neural network module, the neural network module being trained according to the example in Figure 5. The method begins in step 610, in which the trained neural network module receives a first novel image (e.g., image 210) and generates a first feature map (e.g., feature map 240) containing a first grid of local image descriptors based on the first image.

[0091] In step 620, the trained neural network module receives a second novel image (e.g., image 220) and generates a second feature map (e.g., feature map 250) based on the second image, which includes a second grid of local image descriptors. The first and second novel images can depict the same scene using different pixel grids.

[0092] In step 630, a correspondence (e.g., one of the correspondences 270) between a portion of the first new image and a portion of the second new image is identified based on the first and second grids of the local image descriptor. The identified correspondences correlate with the same portion of the depicted scene. Additional correspondences between images can be determined between keypoints in the images or for each individual pixel in the image (i.e., dense correspondences). Step 630 can be performed by the matching module 260 shown in Figure 2.

[0093] First, details of the training methods used in the series of experiments will be discussed. Next, the diverse datasets and benchmarks used in the experiments will be discussed. Finally, details of the experiments and comparisons with other techniques will be presented.

[0094] The training data consists of 150,000 pairs from the SfM-120k dataset, which includes images from well-known landmarks. These training pairs are constructed using COLMAP and acquired using observational superposition from the SfM model provided by the dataset. This process is performed using SIFT, which does not require supervision and only verifies whether two images depict the same scene, without guiding any matching. Arbitrary crops of size 256x256 are performed on these pairs. To generate composite pairs, images randomly sampled from this dataset and standard data augmentation techniques are applied. In particular, random pixel and hue jittering, random rescaling, rotation, and homography are used.

[0095] Network f θ In an exemplary embodiment, a 5x5 convolution gradient with stride 1 and 128 output channels can be extracted first. Subsequently, a 4x4 non-superimposed gradient patch is embedded in a 512-dimensional feature using a 4x4 kernel convolution with stride 4. A series of pointwise and depthwise convolutions are then applied. The exemplary embodiment uses seven such blocks with depthwise convolutions using a 9x9 kernel. Finally, a final pointwise convolution and PixelShuffle operation are applied to obtain a feature map F with d=128 dimensions.

[0096] The HPatches dataset contains 116 image sequences with varying photometric and time-of-day variations. Each sequence includes a homography-related reference image and five source images for source images taken at different time points or under different illumination conditions.

[0097] The ETH3D dataset includes indoor and outdoor sequences captured using a handheld camera and registered with SfM. Image pairs are generated by sampling frames at fixed intervals. This is used to evaluate robustness to time changes due to the widening of the baseline by increasing intervals.

[0098] Aachen Day-Night v1.1 is a large-scale outdoor visual positioning benchmark. The Day-Night split is considered in particular to measure the generalized performance of embodiments of the present invention, as it is characterized by large time changes and drastic illumination changes due to day / night duality. For this work, the following method is employed: In the first part, a global SfM map is created from database images, and in the second part, query images are localized in relation to this mapping. The computational complexity of a complete match is handled using image lookup with the AP-GeMLM18 global descriptor. The number of image pairs is reduced to the top 20 nearest neighbors during the mapping and query steps. Finally, 20,000 local features for each of these lookedup images are extracted and matched to estimate the global map first, and then the camera pose.

[0099] In relation to dense matching, the performance of the PUMP descriptor (a local image descriptor extracted from an image by applying a trained neural network module 230 to the image) is evaluated using the DeepMatching algorithm in a dense or semi-dense manner. In an exemplary embodiment of the matching module 260, the basic pixel descriptor of the DeepMatching algorithm is replaced by the PUMP descriptor. The rest of the pipeline is kept unchanged, except for a basic providing periodic consistency check which is improved to include the nearest neighbor. The output of the DeepMatching algorithm can be semi-dense rather than dense, as it outputs one correspondence per atomic patch from the first image. If a dense warp field is required, density enhancement techniques are used, such as following the COTR method described in Jiang et al., Cotr: Correspondence transformer for matching across images, ICCV, 2021, which is included throughout this specification and includes linear interpolation matching using Delaunay triangulation.

[0100] In the case of the HPatches dataset, evaluation is performed for all image pairs from the image dataset featuring time changes. Table 1 below shows the semi-dense and fully dense (i.e., interpolated) outputs. Two models are evaluated in the example of this application: one trained only on self-supervised pairs (S), i.e., acquired by data augmentation, and the other including self-supervised and unsupervised pairs (S+U).

[0101] In the absence of interpolation, the self-supervised model (S) can perform slightly better than the model trained with unsupervised pairs (S+U). This is achieved by being trained exclusively with synthetic homography that precisely fits the distribution of the test set. In the dense mode, the unsupervised model (S+U) performs better than the self-supervised model (S), indicating that the unsupervised loss can tolerate the generation of even fewer ideal values ​​(as these greatly damage the Delaunay interpolation) and is therefore more robust. Overall, all two models, with or without interpolation, perform significantly better than other approaches. The two models in the examples of this application also far outperform the recently proposed unsupervised WarpC matching loss described in Truong et al., Warp consistency of unsupervised learning of dense correspondences, ICCV, 2021. The results highlight the previously excellent capacity of pyramid matching for large planar regions without discontinuities.

[0102] [Table 1]

[0103] Table 1 shows the Average End Point Error (AEPE) and Percent of Correct Keypoints (PCK) for various thresholds in the HPatches dataset. The scarcity method returns only a subset of correspondences that it is confident in. The top and second-highest results are shown in bold and underlined, respectively. DM is an abbreviation for DeepMatching, and "Interp." means Interpolation. The examples in this application are evaluated only for combinations of self-supervised pairs (S) and self-supervised and unsupervised training pairs (S+U).

[0104] The examples in this application can be evaluated in more challenging settings using real image pairs from ETH3D, which feature numerous discontinuities in time-varying and optical flow for complex 3D shapes. Since the ground truth is scarce and the output is semi-dense and does not necessarily need to be aligned, only the results of densely interpolated deformations for various frame intervals (e.g., velocities) are presented in Table 2. Models trained with unsupervised pairs outperform self-supervised models by up to 25% (relative gain). This highlights the superior robustness possible against realistic noise in models trained with injected matching friars. Furthermore, they surpass other approaches by obtaining first or second AEPE (Average End Point Error) scores for all velocity intervals. Additionally, the exemplary models in this application are significantly faster than other methods, some of which require only a few minutes per test for specific fine-tuning for each method.

[0105] [Table 2]

[0106] Table 2 shows the Average End Point Error (AEPE) for different speeds on the ETH3D dataset. The top and second-highest results are displayed in bold and underlined, respectively.

[0107] In relation to matching on a scarce keypoint basis, the impact of leveraged matching friars during training on scarce matching settings can be evaluated by re-comparing the performance achieved by the exemplary PUMP(S) and PUMP(S+U) models described herein. The models of this application use an external keypoint detector to select repeatable locations in image scale space in order to generate a dense descriptor map. To make the evaluation as comprehensive as possible, the performance of three standard detectors can be measured: SIFT, R2D2, and SuperPoint. For each detector, the same number of keypoints are extracted at the same location and scale for each example, so that the evaluation is fair and rigorously centered on the descriptor.

[0108] The study of overall descriptor quality can be performed by jointly evaluating two complementary tasks: HPatches keypoint matching and Aachen-Night positioning accuracy. In the case of HPatches, Mean Matching Accuracy (MMA) can be measured. MMA corresponds to the average percentage of correct matches for all image pairs in relation to a specified pixel error threshold. Visual positioning performance can be measured as the percentage of successfully localized queries in relation to specified thresholds for camera position and orientation. Table 3 shows the results for each keypoint detector and each model from the two benchmarks. The results, including the self-supervised model (S), show that the exemplary model in this application far outperforms each keypoint baseline in HPatches. Models trained on unsupervised pairs (S+U), which are not even involved in pixel-level unsupervised training, perform better than the self-supervised model in average and all basic keypoints.

[0109] [Table 3]

[0110] Table 3 shows the ratio of HPatches' Mean Matching Accuracy (MMA) and Aachen-Night's localized queries within three error thresholds using different rare keypoint detectors. The absolute gain indicates the performance improvement when training with unsupervised pairs (S+U) compared to training with self-supervised pairs (S) alone.

[0111] Figure 7 illustrates a performance comparison between the latest technology for HPatches (in which the R2D2 keypoint detector is used to extract keypoints) and the model according to the present invention. Figure 7 shows the matching results for the HPatches dataset in terms of Mean Matching Accuracy (MMA) for various error thresholds. The model according to this application far surpasses the method.

[0112] Although the aforementioned examples were described in the context of method steps, they also serve to describe the corresponding components, modules, or features of the corresponding apparatus or system.

[0113] The methods and embodiments described above can be realized in an architecture such as that shown in Figure 8, which includes a server 800 and one or more computing devices 802 that communicate over a network 804 (wireless and / or wired) such as the Internet for data exchange. Each of the server 800 and computing devices 802 includes a data processor 812a to 812e (812) and memory 813a to 813e (813) such as a disk drive. The computing devices 802 may be any type of computing device, such as an autonomous vehicle 802b, a robot 802c, a computer 802d, or a mobile phone 802e, which communicates with the server 800.

[0114] As an example, Method 500 for unsupervised neural network training for learning local image descriptors is performed by Server 800. Server 800 can provide a trained neural network module (e.g., trained neural network module 230) to a computing device, which can then use the received trained neural network module to perform Method 600 to compute the optimal local image descriptor for determining whether a true match exists between an image pair (e.g., an image stored in memory and an image received by the computing device's camera). In various embodiments, the determination of whether a true match exists is used in applications including visual positioning and mapping, navigation, pose estimation, Structure-from-Motion, three-dimensional (3D) reconstruction, and image retrieval that depends on such descriptors. In one example, one of the computing devices 802 performs both Methods 500 and 600. In an additional example, the autonomous vehicle 802b (or robot 802c) includes optical sensors (e.g., one or more cameras) to acquire a first image of the vehicle's (or robot's) environment. The vehicle (or robot) also stores a second image of the vehicle's (or robot's) environment (including positional information) in memory devices 813b, 813c. The processor 812b (or 812c) then performs method 600 on the first and second images to identify corresponding parts from the images and to determine the vehicle's (or robot's) position within the vehicle's (or robot's) environment by visual positioning. In various embodiments, images from cameras or optical sensors can be compared with a number of images stored in a database to identify objects in the received image and matches between one or more stored images.

[0115] The descriptions above are essentially illustrative, and this disclosure does not limit their application or use. The broad teachings of this disclosure can be embodied in a variety of forms. Therefore, while this disclosure includes specific examples, the true scope of this disclosure is not limited in this way, as other variations become apparent through study of the drawings, specification and claims. It should be understood that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, while each embodiment is described as having certain characteristics, any one or more of these characteristics described in relation to any embodiment of this disclosure may be embodied and / or combined with any characteristics of other embodiments, and such combinations are not explicitly described. In other words, the embodiments described above are not mutually exclusive, and permutations between one or more embodiments are within the scope of this disclosure.

[0116] The spatial and functional relationships between elements (e.g., modules, circuit elements, semiconductor layers, etc.) are described using a variety of terms, including “connected,” “related,” “joined,” “adjacent,” “proximity,” “above,” “on top,” “below,” and “positioned.” When a relationship between first and second elements is described in the foregoing disclosure, unless explicitly referred to as “direct,” that relationship may be a direct relationship in which no other intermediate elements exist between the first and second elements, or an indirect relationship in which one or more intervening elements exist (spatially or functionally) between the first and second elements. As used herein, the phrase “at least one of A, B, and C” must be interpreted as meaning the logic (A OR B OR C) using a non-exclusive logical OR, and not as meaning “at least one of A, at least one of B, and at least one of C.”

[0117] In diagrams, the direction indicated by an arrow generally represents the flow of information (e.g., data or commands) related to the example (of interest). For example, if elements A and B exchange various types of information, but the information sent from element A to element B is related to the example, the arrow will be shown pointing from element A to element B. This unidirectional arrow does not imply that no other information is sent from element B to element A. Furthermore, in response to information transferred from element A to element B, element B may transfer a request for information or an acknowledgment of receipt to element A.

[0118] In this application, the following definitions are included, and the terms “module” or “controller” may be replaced with the term “circuit.” The term “module” may refer to an ASIC (Application Specific Integrated Circuit); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combination logic circuit; an FPGA (Field Programmable Gate Array); a processor circuit (shared, dedicated, or grouped) that executes code; a memory circuit (shared, dedicated, or grouped) that stores code executed by the processor circuit; another suitable hardware component that provides the described function; or a combination of some or all of the above, such as a system-on-a-chip.

[0119] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected by a short-range network (LAN), the Internet, a wide-area network (WAN), or a combination thereof. The functionality of any given module in this disclosure can be distributed across multiple modules connected via interface circuits. For example, multiple modules can allow for load balancing. In other examples, a server module (known as remote or cloud) can perform some functions on behalf of a client module.

[0120] The term "code" may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" includes a single processor circuit that executes some or all of the code from a number of modules. The term "group processor circuit" includes a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more modules. References to multiprocessor circuits include multiprocessor circuits on separate dies, multiprocessor circuits on a single die, multicores of a single processor circuit, multithreads of a single processor circuit, or combinations thereof. The term "shared memory circuit" includes a single memory circuit that stores some or all of the code from a number of modules. The term "group memory circuit" includes a memory circuit that, in combination with additional memory, stores some or all of the code from one or more modules.

[0121] The term “memory circuit” is a subset of the term “computer-readable medium.” As used herein, the term “computer-readable medium” does not include transient electrical or electromagnetic signals propagated through the medium (e.g., on a carrier wave), and therefore the term “computer-readable medium” is considered tangible and non-transient. Non-restrictive examples of non-transient and tangible computer-readable mediums include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random-access memory circuits, dynamic random-access memory circuits), magnetic storage media (e.g., analog or digital magnetic tapes and hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray® discs).

[0122] The apparatus and methods described in this application can be partially or completely embodied by a special-purpose computer generated by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The aforementioned functional blocks, flowchart components, and other elements serve as software specifications that can be converted into a computer program through the routine work of a skilled technician or programmer.

[0123] A computer program includes processor-executable instructions stored in at least one non-temporary, tangible, computer-readable medium. A computer program may also include, or depend upon, stored data. A computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, and the like.

[0124] A computer program may include (i) parsed descriptive text such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, and (v) source code for compilation and execution by a JIT (just-in-time) compiler. For example, source code can be created using the syntax of languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. A computer implementation method for unsupervised neural network training for learning local image descriptors, The aforementioned method, A step of obtaining a first pair of images depicting the same scene, wherein the pair of images includes a first image having a first pixel grid and a second image having a second pixel grid, the first pixel grid being different from the second pixel grid, A neural network having an initial set of mediating variables is applied to the first and second images to obtain a feature map F for the first image. １ , and the feature map F for the second image. ２ A step of generating the F １ This includes the first grid of the local image descriptor, and the F ２ Steps include: a first grid of local image descriptors, each local image descriptor in the first grid of the local image descriptors corresponds to each pixel in the first pixel grid, and each local image descriptor in the second grid of the local image descriptors corresponds to each pixel in the second pixel grid; Said F １ and F ２ Based on this, the initial correlation volume C ０ A step in which the C ０ The steps include: including local correlations between pixels of the first pixel grid and pixels of the second pixel grid, where each local correlation between pixels of the first pixel grid and pixels of the second pixel grid is determined based on the local image descriptor corresponding to the correlated pixels; Based on the iterative pyramid structure, the C ０ determining a high-level correlation volume C based on Ｌ where the C Ｌ includes the aggregated high-level correlation between the repeatedly constructed high-level patches of the first and second pixel grids, and each patch is a rectangular portion of each pixel grid, the step of Said C Ｌ Based on the above F １ and F ２ A step of determining the uniqueness matching loss for F, wherein the uniqueness matching loss is calculated from the optimal feature map. １ and F ２ Provides a measure for the deviation of the C Ｌ Based on the aforementioned high-level correlation, if each of the high-level patches of the first and second pixel grids has a unique match among the high-level patches of the other pixel grids, the deviation from the optimal feature map is low, and the steps are as follows: The steps include training the neural network by minimizing a loss function based on the intrinsic matching loss to generate an optimized set of mediating variables, thereby generating a trained neural network adapted to determine the optimal local image descriptor, Computer implementation methods, including those mentioned above.

2. The steps include applying the trained neural network to a first novel image and generating a third feature map including a third grid of local image descriptors, The steps include applying the trained neural network to a second novel image and generating a fourth feature map including a fourth grid of local image descriptors, wherein the first and second novel images depict the same novel scene using different pixel grids from each other. A step of identifying a correspondence between a portion of the first new image and a portion of the second new image based on the third and fourth grids of the local image descriptor, wherein the corresponding portion correlates with the same portion of the depicted new scene. The computer implementation method according to claim 1, further comprising:

3. The step of obtaining the first pair of images includes the step of obtaining a batch of image pairs, The batch of image pairs includes a first pair of images and a second pair of images, the second pair of images including a source image and an extended image, the extended image being synthesized from the source image by data augmentation, and the correct correspondence between the source image and the extended image being obtained through data augmentation. The aforementioned method, Steps include: applying the neural network having the initial set of parameters to the source image and the extended image, and generating feature maps for the source image and the extended image, wherein each feature map includes each grid of local image descriptors; A step of determining the pixel-by-pixel rank loss based on the grid of the local image descriptor corresponding to the source image, the grid of the local image descriptor corresponding to the extended image, and the correct correspondence relationship between the source image and the extended image. Includes, The step of training the neural network includes minimizing the loss function based on the weighted sum of the uniqueness matching loss and the pixel-specific rank loss, A computer implementation method according to claim 1 or 2, including the method described in claim 1 or 2.

4. Said C Ｌ The steps to make a decision are: Said C ０ The local correlations are aggregated as the first level correlation between the first level patch of the first pixel grid and the first level patch of the second pixel grid, and the first level correlation volume C is obtained. １ The steps to generate, For n between 1 and L-1, C ｎ The n-level correlations are iteratively aggregated into the (n+1)-level correlations between the (n+1)-level patches of the first pixel grid and the (n+1)-level patches of the second pixel grid, resulting in the (n+1)-level correlation volume C. ｎ＋１ A step of generating a (n+1)-level patch, wherein the (n+1)-level patch is composed of adjacent n-level patches of each pixel grid, and the aggregated n-level correlation corresponds to the adjacent n-level patches of the correlated (n+1)-level patch. A computer implementation method according to claim 1 or 2, including the method described in claim 1 or 2.

5. Said C ０ The step of aggregating the local correlations includes determining the first level correlation between the first level patch of the first pixel grid and the first level patch of the second pixel grid by the average sum of the local correlations between the corresponding pixels in the first level patch of the first pixel grid and the first level patch of the second pixel grid, Said C ｎ The step of aggregating the n-level correlations is to determine the (n+1)-level correlation between the (n+1)-level patches of the first pixel grid and the (n+1)-level patches of the second pixel grid by the average sum of the n-level correlations between the corresponding n-level patches of the (n+1)-level patches of the first pixel grid and the (n+1)-level patches of the second pixel grid. The computer implementation method according to claim 4, including the method described in claim 4.

6. Each of the local correlations within the average sum of local correlations is C ０ Among the multiple adjacent local correlation values, the one with the largest correlation value, and each of the n-level correlations within the average sum of the n-level correlations, is C ｎ Among multiple adjacent n-level correlation values, the one with the largest correlation value, or Each of the local correlations within the average sum of local correlations is C ０ Determined by the average correlation value of multiple adjacent local correlation values, each of the n-level correlations within the average sum of n-level correlations is C ｎ From among multiple adjacent n-level correlation values, the average correlation value is used to determine the correlation. The computer implementation method described in claim 5.

7. Each (n+1)-level patch of the first or second pixel grid includes a 2 × 2n-level patch of the respective pixel grid. The computer implementation method described in claim 4.

8. The aforementioned initial correlation volume C ０ F １ The first dimension corresponding to the first dimension of and the F １ It has a second dimension corresponding to the second dimension of, Said C ０ The step of determining the F is by a predetermined factor in the first and second dimensions. １ Subsample the first reduced dimension H １ ', and the second reduced dimension W １ The step includes generating a subsampled feature map having ' The computer implementation method according to claim 1 or 2.

9. Said F １ The step of subsampling is, The first pixel grid is divided into non-overlapping atomic patches, each of which contains multiple pixels. A step of determining a resulting local descriptor for each atomic patch based on the local image descriptor corresponding to the pixels of each patch, wherein the resulting local descriptor represents all the pixels of the atomic patch in the subsampled feature map. The computer implementation method according to claim 8, including the method described in claim 8.

10. Said C ０ The step of determining the correlation includes determining the correlation between each atomic patch of the first pixel grid and each pixel of the second pixel grid, wherein each correlation is the local image descriptor representing each atomic patch in the subsampled feature map and the F corresponding to the correlated pixel of the second pixel grid. ２ Determined based on the aforementioned local image descriptor, The computer implementation method according to claim 9.

11. Each atomic patch has a size of 4x4 pixels, and the F １ The first dimension of the above is 4 × H １ ' and the above F １ The second dimension of the above is 4 × W １ 'is, The computer implementation method according to claim 9.

12. Said C ０ is a four-dimensional tensor, and the above C ０ The third and fourth dimensions are F ２ Corresponding to the first and second dimensions, For each n between 0 and L-1, the C ｎ＋１ is a 4-dimensional tensor, Said C ｎ＋１ The step of generating C ｎ＋１ The third and fourth dimensions of are C ｎ Compared to the third and fourth dimensions, the C is reduced by a predetermined factor. ｎ The step includes performing two-dimensional subsampling, The computer implementation method described in claim 4.

13. For each n between 1 and L, the above C ｎ The step of generating C ｎ The step includes performing a rectification transformation for each of the n-level correlations, The computer implementation method described in claim 4.

14. The aforementioned optimal local image descriptor identifies the correspondence between parts of the first image and parts of the second image that depict scenes having similar visual content but different viewpoints or lighting conditions. The computer implementation method described in claim 4.

15. A computer implementation method for identifying the correspondence between a pair of input images, The aforementioned method, The steps include receiving the input image pair, which includes a first input image and a second input image, A step of processing an input image pair using a trained neural network by performing the unsupervised learning method for local image descriptors according to claim 1 or 2, wherein the trained neural network generates local image descriptors for the input image pair. A step of extracting correspondences between parts of the input image pair based on the local image descriptor, wherein each extracted correspondence maps a part of the first input image to a part of the second input image. A step of determining whether a true match exists between the input image pairs in the extracted correspondence, wherein a true match exists between the input image pairs when the corresponding portions of the image pairs depict the same scene. The steps include outputting whether or not a true match exists between the aforementioned pair of images, Computer implementation methods, including those mentioned above.

16. It is a system, Processing unit and The memory connected to the processing unit, A system comprising, wherein the memory includes computer-executable instructions that, when executed by the processing unit, cause the processing unit to perform the method described in claim 1 or 2.

17. One or more computer-readable media on which computer-readable instruction words are stored, wherein, when executed by a processing unit, the instruction words cause the processing unit to perform the method according to claim 1 or 2.

Images