Map processing device and method thereof

Abstract

A map processing device includes at least one processor configured to perform operations to receive (700) from a mobile device, features of a first type extracted from an image captured by an image sensor movable with the mobile device The operations process (702) the features of the first type through an image recovery operation to generate a reconstructed image approximating the image, and extract (704) features of a second type from the reconstructed image. The operations match (706) correspondences between the features of the second type extracted from the reconstructed image and features of the second type stored in a localization map. The operations determine (708) a pose of the mobile device based on the matched correspondences, and send (710) an indication of the pose to the mobile device.

Description

MAP PROCESSING DEVICE AND METHOD THEREOFTECHNICAL FIELD

[0001] The present disclosure relates to a map processing device and a method by a map processing device.BACKGROUND

[0002] Visual localization operations can include to find the pose of an image or set of images in a map. That map can be constructed using photometry operations, such as Structure-from-Motion or Simultaneous Localization And Mapping (SLAM), and includes a set of 3D points observed across multiple images. Additionally, each point stores a visual descriptor, e.g., scale-invariant feature transform (SIFT) or Self-Supervised Interest Point Detection (SuperPoint), where the visual descriptor (a vector) identifies a point and is used to distinguish between points from different areas of the scene and / or images. The stored descriptors include some metric of the descriptors computed from each image that observed the 3D point, for example the mean or median of all the image descriptors or storing all image descriptors per image point, etc. To solve the visual localization problem for a query image, the operations can start by retrieving local features (point position in the image and visual descriptor) and matching the descriptors against the descriptors from the map. Then, using the resulting correspondences, the operations compute the pose solving the Perspective- n-Point (PnP) problem.

[0003] If the map was built from large sets of images, it can be quite large leading to the matching between map and image points being slow. Matching tends to rely on nearest neighbor or learned methods, e.g., LightGlue, which have high time complexity. So, the most recent visual localization methods, e.g., HLoc in [2], use an additional global descriptor that represents the whole image, e.g., Net VLAD. Visual localization becomes then a hierarchical process where the operations include to: 1) compute a global descriptor for the image, e.g., Net VLAD; 2) compare the global descriptor computed to the global descriptors of the images used to create the map to find the most similar ones; 3) match the local descriptors of the query image to the descriptors of the most similar reference images to find image and map point correspondences; and 4) compute the pose solving the PnP problem.

[0004] Running a visual localization pipeline locally on a device can consume substantial computational resources and require a more complex processing architecture. Featureextraction and matching are computationally expensive and newer methods rely on powerful GPUs. Furthermore, memory storage for a map of medium-large scenes can require from 100Gb to 1Tb depending on the descriptors used. Thus, there has been a drive from industry to move this process to the cloud, e.g., Google Virtual Private Server (VPS), Facebook LiveMaps and Microsoft Azure Spatial Anchors. Given the cloudification of these services, devices need to stream the query images to localize, typically over a wireless connection. This should be avoided due to bandwidth and storage requirements, so most solutions stream the local features instead.

[0005] The previous approaches assume that the same types of local features are employed throughout the entire process, i.e., the local features extracted to build the map are of the same type as the ones extracted in the query images. This assumption allows to easily match the descriptors. However, in practice each visual localization service provider and device manufacturer tend to use their own local features. So, local descriptors of a device sent to the cloud are not necessarily compatible with the descriptors used to build the visual localization system map. A solution to this problem was presented in [1], where a simple multi-layer perceptron (MLP) is proposed to translate between different descriptors. Additionally, they propose an encoder-decoder network for each feature type, whose goal is to project all descriptor types to a shared latent space, where they can be matched. If the map is built with a particular feature type one can use the respective decoder to obtain descriptors compatible with the ones in the map.

[0006] The approach presented in [1] assumes that all descriptors are computed from interest points extracted by a single point detector. This assumption is not realistic since for most local features, a pair of point detector and descriptor are proposed. The detectors rely on different properties of the image, e.g., FAST extracts comers, Difference of Gaussians (DoG) extracts blobs and more recent approaches such as SuperPoint rely on Neural Networks to find interest points. Moreover, each detector is designed to be "consistent" across images with visual overlap in terms of repeatability (ability to accurately detect the same point in the scene across slight viewpoint, illumination, scale, and orientation differences) and reliability (how discriminatively the points can be identified). Descriptors are also tuned to be consistent in the same way and to work on a certain detector, so, if the operations compute a descriptor for an interest point extracted by a point detector other than the one designed for the descriptor, it may lose its ability to distinguish different points. It is a big challenge to make detectors and descriptors consistent with themselves, therefore, trying to solve the matching problem using different detectors / descriptors becomes even more challenging.

[0007] In practice, devices rely on a specific combination of detector and descriptor. To use the solution of [1], the operations would need to use the same detector as the one used by the cloud service, and different services would also need to use the same detector to allow compatibility with the devices. Also, the descriptors the device and the cloud services use need to be compatible with that detector. In summary, the approach described in [1] is both too restrictive to be widely used and would also lead to low performance in practice. This brings the need for a new approach to address interoperability in visual localization where the device and the cloud service rely on different detector and descriptor pairs.

[0008] Moreover, sending the local features may still require a considerable bandwidth while raising some privacy concerns, since some works [3] have demonstrated that images can be recovered by using adversarial attacks on the local features. Some of these concerns may be addressed by transforming descriptors or using descriptor distortion techniques for enhanced privacy, such as NinjaDesc. Such techniques, however, affect the matching performance and can be consequently problematic for estimating the pose of the device even in the homogeneous case, not to mention unpredictable consequences for the unexplored case of heterogeneous visual localization.SUMMARY

[0009] Some embodiments disclosed herein are directed to a map processing device that includes at least one processor configured to perform operations to receive, from a mobile device, features of a first type extracted from an image captured by an image sensor movable with the mobile device. The operations process the features of the first type through an image recovery operation to generate a reconstructed image approximating the image, and extract features of a second type from the reconstructed image. The operations match correspondences between the features of the second type extracted from the reconstructed image and features of the second type stored in a localization map. The operations determine a pose of the mobile device based on the matched correspondences, and send an indication of the pose to the mobile device.

[0010] Some other embodiments are directed to a corresponding method by a map processing device. The method includes receiving, from a mobile device, features of a first type extracted from an image captured by an image sensor movable with the mobile device, and processing the features of the first type through an image recovery operation to generate a reconstructed image approximating the image. The method extracts features of a second type from the reconstructed image, and matches correspondences between the features of thesecond type extracted from the reconstructed image and features of the second type stored in a localization map. The method determines a pose of the mobile device based on the matched correspondences, and sends an indication of the pose to the mobile device.

[0011] As will be explained in further detail below, a potential advantage provided by these and other embodiments is that the pose of the mobile device can be robustly determined in situations where feature heterogeneity occurs between different combinations of mobile devices and map processing devices, e.g., servers. Such feature heterogeneity is compensated for through reconstruction of an image that approximates the image that was captured by the image sensor of the mobile device. The reconstructed image is extracted from the local features sent by the mobile device, and is then used to extract the same type of features which are stored in the localization map used by the map processing device, e.g., for processing through the visual localization pipeline. Some further embodiments are directed to the map processing device generating the reconstructed image using as input distorted and / or transformed versions of the local features which were sent by the mobile device to protect privacy and / or reduce network bandwidth.

[0012] Other map processing device and corresponding methods and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such map processing device, methods, and computer program products be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and / or combination.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0014] Figure 1 illustrates a map processing device which operates to localize a mobile device according to some embodiments of the present disclosure;

[0015] Figure 2 illustrates an example architecture and operational flow through the mobile device and the map processing device of Figure 1 according to some embodiments of the present disclosure;

[0016] Figure 3 illustrates another example architecture and operational flow through the mobile device and the map processing device of Figure 1 for protecting privacy and / or reducing network bandwidth according to some embodiments of the present disclosure;

[0017] Figure 4 illustrates an example architecture and operational flow through the map processing device of Figure 1 for generating the reconstructed image using a convolutional neural network according to some embodiments of the present disclosure;

[0018] Figures 5 and 6 illustrate flowcharts of alternative operational sequences which may be performed by the mobile device and the map processing device of Figure 1 in accordance with some embodiments of the present disclosure; and

[0019] Figure 7 illustrates a flowchart of operations which may be performed by the map processing device of Figure 1 in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0020] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0021] Some embodiments are directed to providing a map processing device that operates to determine a pose of a mobile device. The operations can robustly handle situations where the mobile device is providing features of a first type which have been extracted from an image captured by an image sensor movable with the mobile device, and where the first type of features are different from the features of a second type that are stored in a localization map which is used by the map processing device for determining the pose of the mobile device.

[0022] The map processing device can use visual localization of images captured by the mobile device which are characterized through features of the first type that are used as queries to the map processing device requesting localization of the captured image. Such cooperative functionality between devices can be motivated by the desire for one or more of the cooperating devices to become aware of its pose and possibly also trajectory, e.g., relativeto the localization map and / or to recover from a lost state in case the device’s own localization failed.

[0023] Some further embodiments are now described in context of Figure 1 which illustrates a map processing device 100 configured to localize a mobile device 110 according to some embodiments of the present disclosure.

[0024] Referring to Figure 1, the mobile device 110 includes at least one processor 114 (processor), at least one memory 116 (memory) storing instructions executable by the processor 114 to perform operations, at least one image sensor 122 (image sensor), and at least one wireless transceiver 118. The wireless transceiver 118 is configured to communicate through a radio access network(s) 130 and / or other network(s) 132 with the map processing device 100. The image sensor 122 may, for example, be a monocular camera, stereo cameras, or other sensor that can be configured to capture an image, a series of images, or which can output data from which the image or images can be determined. Although the image sensor 122 is illustrated as being part of the mobile device 110, it may instead be separate therefrom but communicatively connected thereto and moveable with the mobile device 110.

[0025] The map processing device 100 includes at least one processor 102 (processor), at least one memory 104 (memory) storing instructions executable by the processor 102 to perform operations, and at least one network interface 109. The memory 104 includes a localization and mapping module 106 that can perform localization and mapping operations relative to a localization map 108 which may be stored in the memory 104 or otherwise accessible through one or more networks. The map processing device 100 may correspond to another mobile device or a server which may perform localization as a cloud service.

[0026] An example operational scenario includes that the mobile device 110 is to be localized using the localization map 108. Given the constrained bandwidth availability between the mobile device 110 and the map processing device 100 (e.g., through the transceiver 118, radio access network 130, network(s) 132, etc.), the mobile device 110 does not stream the images captured by the image sensor 122 which the device wishes to localize. Instead, the mobile device 110 extracts local features, e.g., SIFT (Scale-invariant feature transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), SuperPoint (end-to-end feature point and descriptor extraction network), from the captured images and sends the features to the map processing device 100.

[0027] Localization is complicated because the type of features extracted by the mobile device 110 from the captured images is different than the type of features stored in the localization map 108.

[0028] For example, the mobile device 110 is constrained to use a first type of local features, e.g., because is common for mobile devices to use a single algorithm for SLAM or other visual localization methods which extracts a single type of features from an image. If the localization map 108 and the map processing device 100 used the first type of local features, the localization operations would be relatively more straightforward. However, in the present scenario the localization map 108 and the map processing device 100 use a second type of features, e.g., the localization map 108 stores features of the second type.

[0029] Various embodiments are directed to robust solutions for performing localization when differences are present in the type of features which are provided by the mobile device 100 for localization and the types of features stored in the localization map used by the map processing device 100.

[0030] Figure 2 illustrates an example architecture and operational flow through the mobile device 110 and the map processing device 100 of Figure 1 according to some embodiments of the present disclosure. Referring to Figure 2, the mobile device 100 obtains an image captured by the image sensor 122, extracts features of a first type (Feature A) from the image, and sends the features to the map processing device 100. The features of the first type may correspond to image interest points extracted from the image and further include corresponding descriptors of the image interest points.

[0031] In the illustrated example, the operations build a tensor with size W x H x Ci, where W and H are the width and height of the image, and Ci is the size of the local feature descriptors. The tensor is set to zeros except in the positions where an interest point (keypoint) was detected, and a local feature was computed. In those positions the tensor is filled with the descriptor values. That tensor is fed into a feature extractor (e.g., UNet), and its output is another tensor of size W x H x C2 which is sent to map processing device 100.

[0032] The localization and mapping module 106 receives the features, which are processed through an image recovery operation to generate a reconstructed image approximating the image. The reconstructed image is then processed through another feature extractor to extract features of a second type. The extracted features (Feature B) are the localized to the features of the second type stored in the localization map.

[0033] Figure 3 illustrates another example architecture and operational flow through the mobile device 110 and the map processing device 100 of Figure 1 for protecting privacyand / or reducing network bandwidth according to some embodiments of the present disclosure. Figure 3 adds an additional operation by the mobile device 110 to compress or distort the features (Feature A) before sending to the map processing device 100. The map processing device 100 also performs a corresponding operation to decompress or undistort the received features. The compression / distortion operation can be configured to protect privacy of the image captured by the image sensor 122 and / or to reduce the communication bandwidth utilized to send the features to the map processing device 100 and / or to reduce the communication bandwidth utilized to send the features to the map processing device 100.

[0034] Figure 5 illustrates a flowchart of operations which may be performed by the mobile device 110 and the map processing device 100 of Figure 1 in accordance with some embodiments of the present disclosure. Figure 6 illustrates which of operations of Figure 5 are performed by the mobile device 110 versus the map processing device 100, and provides some additional operations.

[0035] Referring to Figures 5 and 6, the image sensor 122 captures 500 an image. Images may be captured at a defined frame rate, e.g., 20-30Hz. The mobile device 110 extracts 502 features of the first type from the image. If the mobile device 110 is running its own localization process, then it may process the extracted feature through its own localization process to determine its pose. The mobile device 110 also sends the features to the map processing device 100, which may occur for all features extracted from images or only when a defined rule is satisfied. Example rules may include to send only features extracted from key frame images, only send features when the mobile device's 110 own localization process has an excessive error rate, send only features extracted from images associated with start-up of the mobile device 100 and / or upon entering a new room, or when another determination or condition is satisfied.

[0036] When the mobile device 110 is not running its own localization process, it may select a feature extractor from among a set of feature extractors residing in the mobile device 110.

[0037] The mobile device 110 sends the extracted features without sending the captured images in order to reduce the communication bandwidth requirements.

[0038] In an optional embodiment, a decision 600 is made whether to transform the features before sending. The features may be transformed, for example, to protect privacy of the image and / or to compress the features in order to further reduce the communication bandwidth requirements. When the decision 600 is to transform the features, further operations transform 504 the features by performing distortion of the features to protect privacy and / or performing compression of the features to reduce bandwidth requirements forsending. The decision 600 may be satisfied at all times, i.e., the decision can be eliminated with the features always being transformed, or may be based on, for example, when available network bandwidth is below a threshold value (e.g., network congestion, low bandwidth radio access network 130, etc.). Privacy-preservation operations may be based on the NinjaDesc as described in [5], which impedes or prevents the disclosure of private information.

[0039] Transforming 504 the features may be performed using a descriptor approximation approach, such as using the Bag-of-Word vocabulary (discussed below regarding "Experiment 2") or other defined vocabulary. This approach can include using clusterization of the feature space. Each local descriptor can be approximated with the nearest cluster center in the vocabulary, so each local descriptor is represented with a single integer (e.g., the ID of the vocabulary center in the Bag-of-Word or other vocabulary). The vocabulary is known to (e.g., shared between) both the mobile device 110 and the map processing device 100. The information transmitted in this case is the vocabulary ID (e.g., 1 integer) + the pixel position of the descriptor in the image (e.g., 2 more integers).

[0040] Bandwidth reduction operations may use a dimension reduction approach that is applied to the local feature descriptors. Common descriptors such as SIFT correspond to floating point vectors of size 128 or larger for each interest point and from each image around lk-4k of those points may be extracted. Approaches to reduce the dimension of these vectors can include using Principal Component Analysis or using a multilayer perceptron (MLP) in machine learning processing to project the vectors to a smaller dimension space, as described in [4],

[0041] The operations send 506 the local features (or the transformed local features) to the map processing device 100. Since only the features are sent, the message (e.g., first message) may contain the image parameters, i.e., size of the image, width and height, and the camera intrinsics, e.g., focal length and image principal point. The image size facilitates the map processing device 100 reconstructing (recovering) the image, step 512. The camera intrinsics facilitate the map processing device 100 localizing the image to determine pose of the mobile device 110, step 518.

[0042] If the mobile device 110 is not running its own localization process, then it may send the features of all captured images to localize in the map processing device 100. In contrast, if the mobile device 110 is running its own localization process, which may be a common scenario, the mobile device 110 may be configured to send the features of only selected images. For example, if the mobile device 110 is running SLAM, it can send the keyframes (e.g., image features that are included in its own localization map) to validate or improve onits current pose estimate. Alternatively, the mobile device 110 may identify inconsistency in its pose estimates and start sending image local features for localization until its localization process is able to recover with acceptable consistency.

[0043] The map processing device 100 receives 508 the local features and may perform a decision 602 as to whether the received features were transformed, which may be indicated by parameters in the message. If so, the map processing device 100 performs corresponding operations 510 to revert the transformation of the local features of the first type, e.g., using inverse of the operations described above to recover the cluster centers, reprojection to the original space, undistortion, decompression, etc.

[0044] The map processing device 100 may perform a check to determine 604 whether the type of receive features are the same type as the features stored in the localization map 108. When the feature types are different, the map processing device 100 performs further operations to recover (reconstruct) 512 the image by inverting the received features of the first type. Recovery (reconstruction) 512 of the image includes to process the features of the first type through an image recovery operation to generate a reconstructed image approximating the original image captured by the image sensor 122. The processing may include to process the features of the first type through a convolutional neural network that has been trained to correlate the features of the first type to training images.

[0045] It has been shown in [3] that it is possible to use a Deep Neural Network to reconstruct the original image from the extracted local features with good accuracy, if the keypoints associated to those features cover most of the image and with a sufficient number, e.g., at least 1000 features. For visual localization, the approach may extract more points, e.g., at least 3000 points, for good accuracy.

[0046] An example operational flow which may implement this approach is shown in Figure 4. The operations build a tensor with size W x H x Ci, where W and H are the width and height of the image, and Ci is the size of the local feature descriptors, e.g., 128 in the case of SIFT. The tensor is set to zeros except in the positions where an interest point (keypoint) was detected, and a local feature was computed. In those positions the tensor is filled with the descriptor values. That tensor is fed into a UNet 400 that servers as a feature extractor, and its output is another tensor of size W x H x C2. Finally, that second tensor is fed into a Convolutional Neural Network (CNN) 402 that outputs the reconstructed image, to form a Deep Neural Network (DNN).

[0047] The DNN may be trained using the original images as supervision signals. Additionally, the DNN may be trained in conjunction with a discriminator network 404 thatclassifies the image as real or fake. Then the training process can include operations trying to fool the discriminator and the discriminator trying to find out if the image is real or the one reconstructed by the DNN. This process may be based on techniques used for the training of generative adversarial networks.

[0048] In another embodiment, the interest points positions are rescaled to fit in a tensor with dimensions W2x H2x C15with W2< W and H2< H. This enables reducing the GPU RAM utilization of running inference of the network and thus increase the number of queries that can be served in parallel. Present testing of this approach for W2~ generated theresults presented in Table 1 of "Experiment 1" described below. Even though there is a high deterioration of the performance, it still outperforms other approaches.

[0049] Additionally, denoising (e.g., Diffusion models) or upscaling models (e.g., SRGAN) can be used to improve the image quality, after inverting the smaller image. The need for these methods can be seen in Table 2 of "Experiment 1" described below, where for a different feature to create the localization map, the scaled version provided poorer performance.

[0050] In another embodiment, the DNN can also provide a confidence metric of the quality of the reconstructed / recovered image. If the confidence is below a threshold, the second device or server / cloud can request the first device to extract more features per original image, to improve reconstruction. The confidence metric can be performed per pixel or per image.

[0051] When performed per pixel, the network can predict a confidence map with the same size of the image, where each pixel contains a confidence score of how much it resembles the original image. Then the decision metric can be the average or median confidence.

[0052] When performed per image, the discriminator network can be exploited to classify the recovered image, if the discriminator is able to classify the recovered image correctly, then the map processing device 100 requests more features from the mobile device 110.

[0053] The map processing device 100 extracts 514 (recovers) features of the second type from the reconstructed image approximating the original image captured by the image sensor 122. The second type of features correspond to the type of features stored in the localization map 108.

[0054] In one embodiment, the map processing device 100 extracts 514 local features of the second type. In another embodiment, the map processing device 100 also extracts 516 a global descriptor, e.g., NetVLAD. Extraction 516 of the global descriptor allows speeding- up local feature matching during localization (step 518) by reducing the search space. Theglobal descriptor can be used to retrieve image(s), which was used to build the localization map 108, that is similar to the original image or reconstructed image approximating the original image captured by the image sensor 122. Then the reconstructed image is only matched to the map points that are seen in the retrieved images similar to it. Since local features can be matched with Nearest Neighbors of learned techniques (Neural Networks) that have high time complexity, so matching local features from the image directly to the points in the map may take long time. Using global descriptors in a hierarchical pipeline can substantially reduce the computational load of this step.

[0055] The map processing device 100 then localizes 518 the reconstructed image in the localization map 108. The local and optionally the global features extracted in steps 514 and 516 can be used to find matches between interest point in the image and 3D points in the localization map 108. In some embodiment, local features describe specific regions or patches within an image, for example capturing fine-grained details such as edges, corners, and textures. In contrast, global features describe features summarizing the whole image. The matches can be used to solve the Perspective-n-Point problem to estimate the pose of the mobile device 110 (transporting the image sensor 122) given a set of n 3D points in the world and their corresponding 2D projections in the image, as indicated by the localization map 108. However, the matches usually contain outliers, i.e., incorrect matches, thus a robust estimator, e.g., RANSAC, may be used to estimate the pose. Additionally, the previous estimate can be refined by solving a non-linear least-squares minimization of the point reprojection error with the Levenberg-Marquardt algorithm.

[0056] The localization operation 518 can more generally include to match correspondences between the features of the second type that were extracted and the features of the second type that are stored in a localization map 108, and to then determine a pose of the mobile device 110 based on the matched correspondences.

[0057] The map processing device 100 then sends 520 an indication of the pose to the mobile device 110.

[0058] Figure 7 illustrates a flowchart of operations which may be performed by the map processing device of Figure 1 in accordance with some embodiments of the present disclosure.

[0059] Referring to Figure 7, the operations receive 700, from the mobile device 110, features of a first type extracted from an image captured by an image sensor 122, and process 702 the features of the first type through an image recovery operation to generate a reconstructed image approximating the image. The operations extract 704 features of asecond type from the reconstructed image, and match 706 correspondences between the features of the second type extracted and features of the second type stored in the localization map 108. The operations determine 708 a pose of the mobile device 110 based on the matched correspondences, and send 710 an indication of the pose to the mobile device 110.

[0060] The features of the first type may correspond to image interest points extracted from the image and further comprise corresponding descriptors of the image interest points. The features of the second type may correspond to image interest points extracted from the reconstructed image and further comprise corresponding descriptors of the image interest points.

[0061] The features of the first type may be extracted using a feature detector that is different from the feature detector used to extract the features of the second type.

[0062] In a further embodiment, the features of the first type include local features extracted from the image. The operation to extract 704 features of the second type from the reconstructed image, includes to extract local features and global features of the second type from the reconstructed image. The operation to match 706 correspondences can include to match correspondences between the local features of the second type and local map features of the second type stored in the visual localization map, and to match correspondences between the global features of the second type and global map features of the second type stored in the visual localization map.

[0063] As explained above, some embodiments are further directed to protecting privacy of the image captured by the image sensor 122 and / or to reducing communication bandwidth utilization. Following receipt 700 of the features of the first type and before processing 702 the features of the first type through the image recovery operation, the operations process the features of the first type received from the mobile device 110 through a feature reverse transform operation to reduce distortion and / or compression introduced to the features of the first type by a feature transform operation performed by the mobile device 110 to protect privacy of the image and / or to increase compression level before transmission to the map processing device.

[0064] In a further embodiment, the reverse feature transform operation uses identifiers determined from the features of the first type (e.g., the feature description includes an identifier) to retrieve feature prototypes of the first type from a descriptor repository extracted from modeled images, wherein the feature prototypes of the first type are processed through the image recovery operation to generate the reconstructed image approximating the image.

[0065] In another embodiment, the operation to process 702 the features of the first type through the image recovery operation to generate the reconstructed image approximating the image, includes to determine at least one image parameter that comprises at least one of size of the image, width of the image, and height of the image. The operation further includes to determine at least one intrinsic of the image sensor 122, where the at least one intrinsic comprises at least one of focal length of the image sensor 122, image principal point corresponding to a point on an image plane of the image sensor 122 on which a perspective center of the image is projected, and a distortion parameter of the image sensor 122. The operation then adapts the image recovery operation based on the at least one image parameter and the at least one intrinsic of the image sensor 122.

[0066] In another embodiment, the operation to receive 700 from the mobile device 110 features of the first type extracted from the image captured by the image sensor 122, includes to receive features of the first type extracted from only key images.

[0067] In another embodiment, the map processing device 100 also determines the trajectory of movement of the mobile device 110. The map processing device 100 further operates to, for a sequence of images captured by the image sensor 122, repeat the reception 700 of features of the first type, the processing 702 of the features, the extraction 704 of features from the reconstructed image, the matching 706 of correspondences, and the determining 708 of pose. The operations then determine a trajectory based on the determined poses, and send an indication of the trajectory to the mobile device 110.

[0068] In another embodiment, the operation to process 702 the features of the first type through an image recovery operation to generate a reconstructed image approximating the image, includes to process the features of the first type through a convolutional neural network trained to correlate the features of the first type to training images.

[0069] In another embodiment, the operation to process 702 the features of the first type through the convolutional neural network trained to correlate the features of the first type to training images, includes to scale the coordinates of a point of interest from the features of the first type to generate scaled features with scaled points of interest, where a scaling factor for the scaling is defined based on a relative size of the points of interest from the features of the first type to a size of points of interest from the features of the training images. The features with scaled points of interest are processed through the convolutional neural network.

[0070] In another embodiment, the operation to process 702 the features of the first type through the convolutional neural network trained to correlate the features of the first type tothe training images, includes to determine a confidence metric from the convolutional neural network indicating a similarity between the reconstructed image and an original image.Based on the confidence metric not satisfying a confidence rule, the operation requests more features from the mobile device 110.

[0071] In another embodiment, the map processing device 100 also determines a confidence metric from the convolutional neural network indicating an amount of mismatch when attempting to match correspondences between the features of the second type extracted from the reconstructed image and the features stored in the visual localization map. Based on the confidence metric not satisfying a confidence rule, the operations request more features from the mobile device 110.

[0072] Two experiments are now discussed that were performed in accordance with some presently disclosed embodiments.

[0073] Experiment 1

[0074] A first experiment (Experiment 1) tested operations according to some present embodiments and compared their performance to a well-known visual localization benchmark of Aachen Day and Night, using SuperPoint as the local descriptor in the server / second device [6]; and SIFT and the local descriptor in the device in the heterogeneous case (in the homogeneous case, the mobile device uses the same feature than the map processing device, this is, SuperPoint).

[0075] Operations according to present embodiments enabled localization of images from mobile devices that use features different that the ones used by the visual localization pipeline hosted on map processing device.

[0076] Compared to prior approaches, the present operations provided various advantages which included:• Do not assume that visual descriptors are extracted from points of interest / keypoint detected by the same method.• Allow extraction of a global descriptor from the recovered image and thus speed up the matching by using a hierarchical approach to visual localization, such as HLoc.• Demonstrated that the second type of descriptor can also be a global descriptor and it can be used for coarse visual localization tasks, such as loop closure detection or place recognition, improving the generality of the present operations.• Have better performance on popular visual localization benchmarks, see Table 1 for a performance comparison between the present operations and the prior approach presented in [1]

[0077] Table 1, below, compares the visual localization accuracy on the query images of the Aachen Day and Night visual localization benchmark. The results show the percentage of successfully localized query images under three different thresholds of rotation and translation errors. In this test the first local feature (extracted by the first device) is SIFT and the second local feature (extracted by the second device or server / cloud) is SuperPoint.Method % localized queries: day % localized queries: night<0.25m, <0.5m, <5m, <0.25m, <0.5m, <5m,2° 5° 10° 2° 5° 10°Homogeneous: same feature 90.3 96.4 99,3 77.0 90.6 100.0(SuperPoint)Present operations recovering full 85.1 92.1 96.2 65.4 78.5 88.5 resolution imagesPresent operations recovering images with54 64 8 75 2 7 3 13i18 3of the width and height and then upscales themMethod in [1] 47.5 58.0 72.8 6.3 8.4 18.3

[0078] The computational cost of this solution is not low but allows for performance in lower rates (1-10 Hz) suited for global alignment of the mobile device. The implementation is based on a monolithic UNET+CNN architecture shown in Figure 4 that can be optimized. Images have a resolution around 1600x1200x3 (image resolution times three colors, RGB).

[0079] Table 1 compares the computational costs of the present operations.

[0080] Experiment 2: Second embodiment with operations to preserve image privacy and compress features

[0081] The present operations now use DISK as the local descriptor in the map processing device and LightGlue as the matcher; and SIFT and the local descriptor in the mobile device in the heterogeneous case (in the homogeneous case, the device uses the same feature than the server, this is, DISK).

[0082] In the case of this second embodiment, the operations use an additional descriptor transformation method to improve the bandwidth cost and the privacy. Concretely, approximation of the descriptors in the device uses the cluster centers from a Bag-of-Words (BoW) vocabulary. It is shown in Table 2 below that these operations enable private, low- bandwidth and interoperable visual localization with a small loss of accuracy.

[0083] Table 2 compares the visual localization accuracy on the query images of the Aachen Day and Night visual localization benchmark. The results show the percentage of successfully localized query images under three different thresholds of rotation and translation errors. In this test the first local feature (extracted by the first device) is SIFT and the second local feature (extracted by the second device or server / cloud) is DISK and using LightGlue as matcher.Method % localized queries: day % localized queries: night<0.25m, <0.5m, <5m, <0.25m, <0.5m, <5m,2° 5° 10° 2° 5° 10°Homogeneous: same90.6 99.5 feature (DISK)Present operations recovering full 82.9 90.5 95.3 57.6 74.3 88.0 resolution imagesPresent operations recovering images with % of the width and 15.8 24.9 41.0 0.0 1.6 2.6 height and then upscales themPresent operations recovering full76.9 83.1 35.1 47.6 60.7 resolution images from BoW cluster centers

[0084] The BoW vocabulary is trained for SIFT features, using 10A6 clusters, and occupies 360 MB of hard disk space. Estimating the nearest center takes 8.82 us / feature, approx. 9 ms in an image with 1000 features in the device. Note that the vocabulary should be present both in the mobile device and the map processing device. The mobile device just replaces each local descriptor with the ID of the nearest cluster center (which means the descriptor is represented by a single integer corresponding to the cluster identifier instead of the typical 128-sized floating point SIFT descriptor). From the vocabulary IDs, the map processing device retrieves the corresponding vocabulary cluster centers as input for the image recovery operation (without appreciable extra overhead).

[0085] Note that for this example, the image recovery architecture has not been trained specifically for the BoW vocabulary cluster centers, but instead keeps the same weights for the image recovery architecture as in the full descriptor case (Experiment 1), assuming that these centers are approximations (results are probably to improve if operations fine-tune the architecture for the vocabulary centers). With this, the present operations allow on-demand transformation, in the case of congestion in the network. The mobile device can require more accurate localization from the map processing device by sending the original descriptors or a coarse estimation (lowering the bandwidth) by just sending the cluster IDs.

[0086] Further definitions and embodiments are now explained below.

[0087] In the above description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0088] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As usedherein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0089] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0090] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id Est," may be used to specify a particular item from a more general recitation.

[0091] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and therebycreate means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0092] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0093] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0094] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents and shall not be restricted or limited by the foregoing detailed description.

[0095] A listing of references that are cited herein follows:[1] Dusmanu, M., Miksik, O., Schonberger, J. L., & Pollefeys, M., “Cross-descriptor visual localization and mapping”, 2021 IEEE / CVF International Conference on Computer Vision, pages 6058-6067, IEEE[2] Sarlin, P. E., Cadena, C., Siegwart, R., & Dymczyk, M., “From coarse to fine: Robust hierarchical localization at large scale”, 2019 IEEE / CVF conference on computer vision and pattern recognition, pages 12716-12725, IEEE[3] Pittaluga, F., Koppal, S. J., Kang, S. B., & Sinha, S. N., ’’Revealing scenes by inverting structure from motion reconstructions”, 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 145-154, IEEE[4] Hao Dong, Xieyuanli Chen, Mihai Dusmanu, Viktor Larsson, Marc Pollefeys, Cyrill Stachniss, “Learning-Based Dimensionality Reduction for Computing Compact and Effective Local Feature Descriptors”, arXiv:2209.13586, 2022[5] Tony Ng, Hyo Jin Kim, Vincent Lee, Daniel DeTone, Tsun-Yi Yang, Tianwei Shen, Eddy Ilg, Vassileios Balntas, Krystian Mikolajczyk, Chris Sweeney, “NinjaDesc: Content- Concealing Visual Descriptors via Adversarial Learning”, 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12797-12807, IEEE[6] Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., ... & Pajdla, T., “Benchmarking 6DOF outdoor visual localization in changing conditions”, 2018 IEEE conference on computer vision and pattern recognition, pages 8601-8610, IEEE

Claims

CLAIMS:

1. A map processing device (100) comprising: at least one processor (102) configured to perform operations to: receive from a mobile device (110), features of a first type extracted from an image captured by an image sensor (122) movable with the mobile device (110); process the features of the first type through an image recovery operation to generate a reconstructed image approximating the image; extract features of a second type from the reconstructed image; match correspondences between the features of the second type extracted from the reconstructed image and features of the second type stored in a localization map (108); determine a pose of the mobile device based on the matched correspondences; and send an indication of the pose to the mobile device (110).

2. The map processing device (100) of Claim 1, wherein: the features of the first type comprise first image interest points extracted from the image and further comprise corresponding descriptors of the first image interest points; and the features of the second type comprise second image interest points extracted from the reconstructed image and further comprise corresponding descriptors of the second image interest points.

3. The map processing device (100) of any of Claims 1 to 2, wherein: the features of the first type are extracted using a feature detector that is different from the feature detector used to extract the features of the second type.

4. The map processing device (100) of any of Claims 1 to 3, wherein: the operation to extract features of a second type from the reconstructed image, further comprises to extract local features and global features of the second type from the reconstructed image; and the operation to match correspondences comprises to match correspondences between the local features of the second type and local map features of the second type stored in the visual localization map, and to match correspondences between theglobal features of the second type and global map features of the second type stored in the visual localization map.

5. The map processing device (100) of any of Claims 1 to 4, following receipt of the features of the first type and before processing the features of the first type through the image recovery operation, the operations further comprise to: process the features of the first type received from the mobile device (110) through a reverse feature transform operation to reduce distortion and / or compression introduced to the features of the first type by a feature transform operation performed by the mobile device (110) to protect privacy of the image and / or to increase compression level before transmission to the map processing device.

6. The map processing device (100) of Claim 5, wherein the reverse feature transform operation uses identifiers determined from the features of the first type to retrieve feature prototypes of the first type from a descriptor repository extracted from modeled images, wherein the feature prototypes of the first type are processed through the image recovery operation to generate the reconstructed image approximating the image.

7. The map processing device (100) of any of Claims 1 to 6, wherein the operation to process the features of the first type through the image recovery operation to generate the reconstructed image approximating the image, comprises to: determine at least one image parameter that comprises at least one of size of the image, width of the image, and height of the image; determine at least one intrinsic of the image sensor, wherein the at least one intrinsic comprises at least one of focal length of the image sensor, image principal point corresponding to a point on an image plane of the image sensor on which a perspective center of the image is projected, and a distortion parameter of the image sensor; and adapt the image recovery operation based on the at least one image parameter and the at least one intrinsic of the image sensor.

8. The map processing device (100) of any of Claims 1 to 7, wherein the operation to receive from the mobile device (110) features of the first type extracted from the image captured by the image sensor comprises to:receive features of the first type extracted from only key images.

9. The map processing device (100) of any of Claims 1 to 8, wherein the operations further comprise to: for a sequence of images captured by the image sensor, repeat the reception of features of the first type, the processing of the features, the extraction of features from the reconstructed image, the matching of correspondences, and the determining of the pose; determine a trajectory based on the determined poses; and send an indication of the trajectory to the mobile device (110).

10. The map processing device (100) of any of Claims 1 to 9, wherein the operation to process the features of the first type through the image recovery operation to generate a reconstructed image approximating the image comprises to: process the features of the first type through a convolutional neural network trained to correlate the features of the first type to training images.

11. The map processing device (100) of Claim 10, wherein the operation to process the features of the first type through the convolutional neural network trained to correlate the features of the first type to training images comprises to: scale the coordinates of a point of interest from the features of the first type to generate scaled features with scaled points of interest, wherein a scaling factor for the scaling is defined based on a relative size of the points of interest from the features of the first type to a size of points of interest from the features of the training images; and wherein the features with scaled points of interest are processed through the convolutional neural network.

12. The map processing device (100) of any of Claims 10 to 11, wherein the operation to process the features of the first type through the convolutional neural network trained to correlate the features of the first type to the training images comprises to: obtain a confidence metric from the convolutional neural network indicating a similarity between the reconstructed image and a desired image quality; and based on the confidence metric not satisfying a confidence rule, request more features from the mobile device (110).

13. The map processing device (100) of any of Claims 1 to 12, wherein the operations further comprise to: determine a confidence metric from the convolutional neural network indicating an amount of mismatch when attempting to match correspondences between the features of the second type extracted from the reconstructed image and the features stored in the visual localization map; and based on the confidence metric not satisfying a confidence rule, request more features from the mobile device (110).

14. A method by a map processing device comprising: receiving (700), from a mobile device, features of a first type extracted from an image captured by an image sensor movable with the mobile device; processing (702) the features of the first type through an image recovery operation to generate a reconstructed image approximating the image; extracting (704) features of a second type from the reconstructed image; matching (706) correspondences between the features of the second type extracted from the reconstructed image and features of the second type stored in a localization map; determining (708) a pose of the mobile device based on the matched correspondences; and sending (710) an indication of the pose to the mobile device.

15. The method of Claim 14, wherein: the features of the first type comprise first image interest points extracted from the image and further comprise corresponding descriptors of the first image interest points; and the features of the second type comprise second image interest points extracted from the reconstructed image and further comprise corresponding descriptors of the second image interest points.

16. The method of any of Claims 14 to 15, wherein: the features of the first type are extracted using a feature detector that is different from the feature detector used to extract the features of the second type.

17. The method of any of Claims 14 to 16, wherein: the extracting (704) of features of the second type from the reconstructed image, further comprises extracting local features and global features of the second type from the reconstructed image; and the matching (706) of correspondences comprises matching correspondences between the local features of the second type and local map features of the second type stored in the visual localization map, and matching correspondences between the global features of the second type and global map features of the second type stored in the visual localization map.

18. The method of any of Claims 14 to 17, following the receiving (700) of the features of the first type and before the processing (702) of the features of the first type through the image recovery operation, the method further comprising: processing the features of the first type received from the mobile device through a reverse feature transform operation to reduce distortion and / or compression introduced to the features of the first type by a feature transform operation performed by the mobile device to protect privacy of the image and / or to increase compression level before transmission to the map processing device.

19. The method of Claim 18, wherein the reverse feature transform uses identifiers determined from the features of the first type to retrieve feature prototypes of the first type from a descriptor repository extracted from modeled images, wherein the feature prototypes of the first type are processed through the image recovery operation to generate the reconstructed image approximating the image.

20. The method of any of Claims 14 to 19, wherein the processing (702) of the features of the first type through the image recovery operation to generate the reconstructed image approximating the image, comprises: determining at least one image parameter that comprises at least one of size of the image, width of the image, and height of the image; determining at least one intrinsic of the image sensor, wherein the at least one intrinsic comprises at least one of focal length of the image sensor, image principal point corresponding to a point on an image plane of the image sensor onwhich a perspective center of the image is projected, and a distortion parameter of the image sensor; and adapting the image recovery operation based on the at least one image parameter and the at least one intrinsic of the image sensor.

21. The method of any of Claims 14 to 20, wherein the receiving (700) of features of the first type extracted from the image captured by the image sensor, comprises: receiving features of the first type extracted from only key images.

22. The method of any of Claims 14 to 21 further comprising: for a sequence of images captured by the image sensor, repeating the receiving (700) of features of the first type, the processing (702) of the features, the extracting (704) of features from the reconstructed image, the matching (706) of correspondences, and the determining (708) of pose; determining a trajectory based on the determined poses; and sending an indication of the trajectory to the mobile device (110).

23. The method of any of Claims 14 to 22, wherein the processing (702) of the features of the first type through the image recovery operation to generate a reconstructed image approximating the image, comprises: processing the features of the first type through a convolutional neural network trained to correlate the features of the first type to training images.

24. The method of Claim 23, wherein the processing (702) of the features of the first type through the convolutional neural network trained to correlate the features of the first type to training images, comprises: scaling the coordinates of a point of interest from the features of the first type to generate scaled features with scaled points of interest, wherein a scaling factor for the scaling is defined based on a relative size of the points of interest from the features of the first type to a size of points of interest from the features of the training images; and wherein the features with scaled points of interest are processed through the convolutional neural network.

25. The method of any of Claims 23 to 24, wherein the processing (702) of the features of the first type through the convolutional neural network trained to correlate the features of the first type to the training images, comprises: determining a confidence metric from the convolutional neural network indicating a similarity between the reconstructed image and a desired image quality; and based on the confidence metric not satisfying a confidence rule, requesting more features from the mobile device.

26. The method of any of Claims 14 to 25, further comprising: determining a confidence metric from the convolutional neural network indicating an amount of mismatch when attempting to match correspondences between the features of the second type extracted from the reconstructed image and the features stored in the visual localization map; and based on the confidence metric not satisfying a confidence rule, requesting more features from the mobile device.

Images