Localization by neural network

A neural network-based method for aligning 3D point clouds from multiple sensor modalities improves position estimation accuracy and efficiency by using hierarchical encoding and fusion layers, addressing ambiguity in repetitive structures.

JP2024159616AActive Publication Date: 2024-11-08NOKIA SOLUTIONS & NETWORKS OY

Patent Information

Application Number
JP2024069433
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-26
Filing Date
2024-04-23
Publication Date
2024-11-08
Estimated Expiration
2044-04-23

AI Technical Summary

Technical Problem

Existing position estimation methods, particularly in the context of industrial metaverse applications, face challenges in accurately aligning and registering 3D point clouds from different sensors due to repetitive structures and environmental variations, leading to ambiguity and computational inefficiencies.

Method used

A neural network-based approach that encodes 3D point clouds from multiple modalities (e.g., photographic images, LIDAR, RF fingerprints) using hierarchical encoder layers and fusion layers, followed by attention mechanisms to align feature vectors and estimate rigid transformations, reducing ambiguity and improving computational efficiency.

Benefits of technology

Enhances the accuracy and robustness of position estimation by fusing different sensor modalities, allowing for precise alignment of 3D point clouds even in repetitive environments and reducing computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159616000001_ABST
    Figure 2024159616000001_ABST
Patent Text Reader

Abstract

To provide a device, method, and program for improving localization by a neural network.SOLUTION: A method comprises: encoding a first 3D point cloud of a first coordinate system into a first encoded map comprising first feature center points and first feature vectors; encoding a second 3D point cloud of a second coordinate system into a second encoded map comprising second feature center points and second feature vectors; adapting the first input feature vectors based on the first input feature vectors and the second input feature vectors, to obtain a first joint map; adapting the second input feature vectors based on the first input feature vectors and the second input feature vectors to obtain a second joint map; calculating similarities between the first joint feature vectors and the second joint feature vectors and point correlations based on the similarities; extracting the coordinates of the first joint feature center points and the coordinates of the second joint feature center points if the correlation condition is fulfilled by the similarities; and calculating a transformation between the first coordinate system and the second coordinate system based on the pairs of extracted coordinates.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD This disclosure relates to position estimation.

[0002] Abbreviation 3D-3D AP - Access Point RF - Radio Frequency RGB - Red, Green, Blue RGBD - RGB and Depth SLAM - Simultaneous localization and mapping RSSI - Received Signal Strength VIO - Visual Inertial Odometry RF - Radio Frequency BoW - Bag of Words SVD - Singular Value Decomposition CV - Computer Vision DL - Deep Learning CSI - Channel State Information AoA - Angle of Arrival ToF - Time of Flight BSSID - Basic Service Set Identifier GNSS-Global Navigation Satellite System DoF - degrees of freedom IMU - Inertial Measurement Unit VPS-Visual Positioning System dB - decibel dBm - decibel milliwatt [Background technology]

[0003] Localization of sensors (such as cameras or RF antennas) in an environment is relevant to the industrial metaverse. The most ubiquitous sensor for creating a localization map is a camera. Given a set of camera images, one can try to find the pose of the camera where the images were taken and find the 3D structure of the environment (e.g., represented as a point cloud). This 3D structure, together with metadata, can serve as a visual localization map. This localization map can find the pose of a query image in a world coordinate frame with 6 Degrees of Freedom (6DoF), i.e., translational 3DoF and rotational 3DoF, thus connecting the digital world to the real world. Summary of the Invention

[0004] The aim is to improve upon the prior art.

[0005] According to a first aspect, an apparatus comprising one or more processors and a memory storing instructions, the instructions, when executed by the one or more processors, causes the apparatus to: input a first 3D point cloud of a first modality and a first grid including one or more first cells into a first encoder layer, where coordinates of points of the first 3D point cloud are represented in a first coordinate system; and encoding, by the first encoder layer, the first 3D point cloud of the first modality into a first encoded map of the first modality, where the coordinates of points of the first 3D point cloud are represented in a first coordinate system. the first encoded map includes, for each of the first cells of the first grid, a respective first feature center point and a respective first feature vector, coordinates of the first feature center points being expressed in a first coordinate system; and inputting the first encoded input map into a first matching descriptor layer of a first hierarchical level, the first encoded input map being based on a first encoded map of a first modality and including, for each of the first cells, a respective first joint feature center point and a respective first input feature vector, and the first joint feature center points being expressed in a first coordinate system. inputting a second 3D point cloud of the first modality and a second grid including one or more second cells into a second encoder layer, where coordinates of points of the second 3D point cloud are expressed in the second coordinate system; and encoding, by the second encoder layer, the second 3D point cloud of the first modality into a second encoded map of the first modality, where the second encoded map of the first modality includes, for each of the second cells of the second grid, a respective second feature center point and a second feature center point. and respective second feature vectors, coordinates of the second feature center points being expressed in a second coordinate system; inputting the second encoded input map into a first matching descriptor layer, the second encoded input map being based on the second encoded map of the first modality and comprising, for each of the second cells, respective second joint feature center points and respective second input feature vectors, coordinates of the second joint feature center points being expressed in the second coordinate system; inputting the second encoded input map into a first matching descriptor layer, the second encoded input map being based on the second encoded map of the first modality and comprising, for each of the second cells, respective second joint feature center points and respective second input feature vectors, coordinates of the second joint feature center points being expressed in the second coordinate system;adapting each of the first input feature vectors based optionally on their first feature center points and the second input feature vectors, optionally on their second feature center points, to obtain a first joint map, the first joint map including, for each of the first joint feature center points, a respective first joint feature vector; adapting each of the second input feature vectors based on the first input feature vectors, optionally on their first feature center points and the second input feature vectors, optionally on their second feature center points, to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; adapting each of the first input feature vectors, optionally on their first feature center points and the second input feature vectors, optionally on their second feature center points, to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; calculating, by the first optimal matching layer, for each of the first cells and each of the second cells, a first point correlation between each of the first cells and each of the second cells based on a similarity between the first joint feature vector of each of the first cells and several second joint feature vectors of several second cells and based on a similarity between the second joint feature vector of each of the second cells and several first joint feature vectors of several first cells; and, for each of the first cells and each of the second cells, checking whether at least one of one or more first correlation conditions is satisfied for each of the first cells and each of the second cells, wherein the one or more first correlation conditions include: a first point correlation between each first cell and each second cell is greater than a first correlation threshold for the first tier; or a first point correlation between each first cell and each second cell is among a maximum k values ​​of first point correlations, k being a fixed value; An apparatus is provided that performs the steps of: extracting, for each first cell and each second cell, coordinates of a first joint feature center point of the respective first cell in a first coordinate system and coordinates of a second joint feature center point of the respective second cell in a second coordinate system, and obtaining respective pairs of extracted coordinates of the first hierarchical hierarchy in response to checking whether at least one of one or more first correlation conditions is satisfied for each first cell and each second cell; and calculating an estimated transformation between the first coordinate system and the second coordinate system based on the pairs of extracted coordinates of the first hierarchical hierarchy.

[0006] The first encoder layer, the second encoder layer, and the first matching descriptor layer of the first hierarchy are configured with respective first parameters, and the instructions, when executed by the one or more processors, may further cause the apparatus to perform the step of training the apparatus to determine the first parameters such that a deviation between an estimated transformation between the first coordinate system and the second coordinate system and a known transformation is less than a transformation estimation threshold and such that a first point correlation is increased.

[0007] The first modality may include one of a photographic image, a LIDAR image, and an ultrasound image.

[0008] The first encoder layer may be the same as the second encoder layer.

[0009] The instructions, when executed by the one or more processors, may further cause the apparatus to perform at least the steps of generating a first 3D point cloud based on a sequence of first images of a first modality, where a respective position in the first coordinate system is associated with each of the first images, or generating a second 3D point cloud based on a sequence of second images of the first modality, where a respective position in the second coordinate system is associated with each of the second images.

[0010] The first coded input map may be the first coded map, the first joint feature center points may be the first feature center points, the second coded input map may be the second coded map, and the second joint feature center points may be the second feature center points.

[0011] The instructions, when executed by the one or more processors, further cause the apparatus to input a third 3D point cloud of a second modality and the first grid to a third encoder layer, where coordinates of points of the third 3D point cloud are expressed in the first coordinate system, the second modality being different from the first modality; and encoding, by the third encoder layer, the third 3D point cloud of the second modality into a third encoded map of the second modality, where the third encoded map of the second modality includes, for each of the first cells of the first grid, , respective third feature center points and respective third feature vectors, coordinates of the third feature center points being indicated in a first coordinate system; inputting the third encoded map to a first neural fusion layer; inputting the first encoded map to the first neural fusion layer; and generating a first encoded input map by the first neural fusion layer based on the first encoded map and the third encoded map, wherein for each of the first cells, the respective first joint feature center points are represented by respective first joint feature vectors. Based on the first feature center points of the cells and the third feature center points of the respective first cells, each first input feature vector is calculated based on the first feature vector and the first feature vector, optionally the first feature center points and the third feature vector of each first cell adapted by their first feature center points, optionally the third feature vector and the first feature vector, optionally the third feature center points and the third feature vector of each first cell adapted by their first feature center points, optionally the third feature vector and the first feature vector, inputting the fourth 3D point cloud of the second modality and the second grid into a fourth encoder layer, where coordinates of points of the fourth 3D point cloud are represented in a second coordinate system; and encoding by the fourth encoder layer the fourth 3D point cloud of the second modality into a fourth encoded map of the second modality, where the fourth encoded map of the second modality includes, for each of the second cells of the second grid, a respective fourth feature center point and a respective fourth feature vector, where the coordinates of the fourth feature center point are represented in a second coordinate system.the step of inputting the fourth coded map to a second neural fusion layer, the step of inputting the second coded map to the second neural fusion layer, and the step of generating by the second neural fusion layer a second coded input map based on the second coded map and the fourth coded map, wherein for each of the second cells, the respective second joint feature center points are based on the second feature center points of the respective second cells and the fourth feature center points of the respective second cells, and the respective second input feature vectors are based on the second feature vector and the second feature vector, optionally the second feature center points and the fourth feature vector of the respective second cells adapted by their second feature center points, optionally their fourth feature center points, and the fourth feature vector and the second feature vector, optionally the fourth feature center points and the fourth feature vector of the respective second cells adapted by their second feature center points, optionally their fourth feature center points.

[0012] The second modality may include at least one of a photographic image, an RF fingerprint, a LIDAR image, or an ultrasound image.

[0013] The third encoder layer, the fourth encoder layer, the first neural fusion layer, and the second neural fusion layer are configured with respective second parameters, and the instructions, when executed by the one or more processors, may further cause the apparatus to perform the step of training the apparatus to determine the first parameters and the second parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold and such that the first point correlation is increased.

[0014] The first image may be taken by a first sensor of the first sensor device and the second image may be taken by a second sensor of the second sensor device, and the instructions, when executed by the one or more processors, may further cause the apparatus to perform the steps of: making a plurality of first measurements of a second modality using a third sensor of the first sensor device; making a plurality of second measurements of the second modality using a fourth sensor of the second sensor device; generating a third 3D point cloud of the second modality based on the plurality of first measurements, a position associated with the first image, and a calibration between the first sensor of the first sensor device and the third sensor of the first sensor device; and generating a fourth 3D point cloud of the second modality based on the plurality of second measurements, a position associated with the second image, and a calibration between the second sensor of the second sensor device and the fourth sensor of the second sensor device.

[0015] The third encoder layer may be the same as the fourth encoder layer.

[0016] The first neural fusion layer may be the same as the second neural fusion layer.

[0017] The instructions, when executed by the one or more processors, further cause the apparatus to include inputting the first encoded map of the first modality and a third grid including one or more third cells into a fifth encoder layer, each of the third cells including two or more of the first cells, and the first grid including two or more of the first cells; and encoding, by the fifth encoder layer, the first encoded map of the first modality into a fifth encoded map of the first modality, wherein the fifth encoded map of the first modality is inputting the third encoded input map into a second matching descriptor layer of a second hierarchical level, the third encoded input map being based on the fifth encoded map of the first modality and comprising, for each of the third cells of the third grid, a respective fifth feature center point and a respective fifth feature vector, the coordinates of the third joint feature center points being represented in the first coordinate system; and inputting the third encoded input map into a second matching descriptor layer of a second hierarchical level, the third encoded input map being based on the fifth encoded map of the first modality and comprising, for each of the third cells, a respective third joint feature center point and a respective third input feature vector, the coordinates of the third joint feature center points being represented in the first coordinate system. inputting the second encoded map of the first modality and a fourth grid including one or more fourth cells into a sixth encoder layer, each of the fourth cells including two or more of the second cells, the second grid including two or more of the second cells; and encoding, by the sixth encoder layer, the second encoded map of the first modality into a sixth encoded map of the first modality, the sixth encoded map of the first modality including one or more fourth cells of the fourth grid. inputting the fourth encoded input map into a second matching descriptor layer, the fourth encoded input map being based on the sixth encoded map of the first modality, and comprising, for each of the fourth cells, a respective sixth feature center point and a respective sixth feature vector, the coordinates of the sixth feature center point being expressed in the second coordinate system; and inputting the fourth encoded input map into a second matching descriptor layer, the fourth encoded input map being based on the sixth encoded map of the first modality, and comprising, for each of the fourth cells, a respective fourth joint feature center point and a respective fourth input feature vector, the coordinates of the fourth joint feature center point being expressed in the second coordinate system.adapting each of the third input feature vectors based on the third input feature vector, optionally their third joint feature center points, and the fourth input feature vector, optionally their fourth joint feature center points, by the second matching descriptor layer; obtaining a third joint map, the third joint map including, for each of the third joint feature center points, a respective third joint feature vector; adapting each of the fourth input feature vectors based on the third input feature vector, optionally their third joint feature center points, and the fourth input feature vector, optionally their fourth joint feature center points, by the second matching descriptor layer; obtaining a fourth joint map, the fourth joint map including, for each of the fourth joint feature center points, a respective fourth joint feature vector; calculating, for each of the fourth cells, a similarity between the third joint feature vector of the respective third cell and the fourth joint feature vector of the respective fourth cell; calculating, by the second optimal matching layer, for each of the third cells and each of the fourth cells, a second point correlation between each of the third cells and each of the fourth cells based on the similarity between the third joint feature vector of the respective third cell and the fourth joint feature vector of the fourth cell and based on the similarity between the fourth joint feature vector of the respective fourth cell and the third joint feature vector of the third cell; and checking, for each of the third cells and each of the fourth cells, whether at least one of one or more second correlation conditions for each of the third cells and each of the fourth cells is satisfied, wherein the one or more second correlation conditions include: a second point correlation between each third cell and each fourth cell is greater than a second correlation threshold of the second tier; or the second point correlation between each third cell and each fourth cell is among the maximum l values ​​of the second point correlation, where l is a fixed value.

[0018] The instructions, when executed by the one or more processors, may further cause the apparatus to perform the step of prohibiting the first matching descriptor tier from checking, for each of the first cells and each of the second cells, whether a first point correlation between the first joint feature vector of the respective first cell and the second joint feature vector of the respective second cell is greater than a first tier first correlation threshold in response to checking that at least one of the one or more second correlation conditions for the respective third cell and the respective fourth cell is satisfied.

[0019] The instructions, when executed by the one or more processors, may further cause the apparatus to perform the steps of: extracting, for each of the third cells and each of the fourth cells, coordinates of a third joint feature center point of the respective third cell in the first coordinate system and coordinates of a fourth joint feature center point of the respective fourth cell in the second coordinate system, and obtaining respective pairs of extracted coordinates of the second hierarchical hierarchy in response to checking that at least one of the one or more second correlation conditions for each of the third cells and each of the fourth cells is satisfied; and calculating an estimated transformation between the first coordinate system and the second coordinate system based on the pairs of extracted coordinates of the first hierarchical hierarchy and the pairs of extracted coordinates of the second hierarchical hierarchy.

[0020] The instructions, when executed by the one or more processors, may further cause the apparatus to perform a step of calculating an estimated transformation, such that in the calculating step, extracted coordinate pairs of the first hierarchical layer have different weights than extracted coordinate pairs of the second hierarchical layer.

[0021] A second matching descriptor layer of the second hierarchy is configured with a respective third parameter, and the instructions, when executed by the one or more processors, may further cause the apparatus to perform the step of training the apparatus to determine the first and third parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold, and such that the first point correlation and the second point correlation are increased.

[0022] The fifth encoder layer may be the same as the sixth encoder layer.

[0023] The fifth encoder layer may be the same as the first encoder layer.

[0024] The sixth encoder layer may be the same as the second encoder layer.

[0025] The second matching descriptor layer may be the same as the first matching descriptor layer.

[0026] The second optimal matching layer may be the same as the first optimal matching layer.

[0027] The third encoded input map may be a third encoded map of the first modality and the third joint feature center points may be third feature center points, the fourth encoded input map may be a fourth encoded map of the first modality and the fourth joint feature center points may be fourth feature center points.

[0028] The instructions, when executed by the one or more processors, further cause the apparatus to input the third encoded map of the second modality and the third grid to a seventh encoder layer; and encoding, by the seventh encoder layer, the third encoded map of the second modality into a seventh encoded map of the second modality, the seventh encoded map of the second modality comprising, for each of the third cells of the third grid, a respective seventh feature center point and a respective seventh feature vector, the coordinates of the seventh feature center point being determined by a first a step of inputting the seventh encoded map to a third neural fusion layer; inputting the fifth encoded map to a third neural fusion layer; and generating a third encoded input map by the third neural fusion layer based on the seventh encoded map and the fifth encoded map, wherein for each of the third cells, the respective third joint feature center points are based on the fifth feature center points of the respective third cells and the seventh feature center points of the respective third cells; a third input feature vector based on the fifth feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector, and based on the seventh feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector, inputting the fourth encoded map of the second modality and the fourth grid into an eighth encoder layer; and encoding the fourth encoded map of the second modality into an eighth encoded map of the second modality by the eighth encoder layer, an eighth encoded map of the second modality comprising, for each of the fourth cells of the fourth grid, a respective eighth feature center point and a respective eighth feature vector, the coordinates of the eighth feature center points being represented in the second coordinate system; inputting the eighth encoded map to a fourth neural fusion layer; inputting the sixth encoded map to the fourth neural fusion layer; and generating, by the fourth neural fusion layer, a fourth encoded input map based on the sixth encoded map and the eighth encoded map,For each of the fourth cells, the respective fourth joint feature center points are based on the sixth feature center points of the respective fourth cell and the eighth feature center points of the respective fourth cell, and the respective fourth input feature vectors are based on the sixth feature vector of the respective fourth cell adapted by the sixth feature vector and the eighth feature vector, and based on the eighth feature vector of the respective fourth cell adapted by the sixth feature vector and the eighth feature vector.

[0029] The instructions, executed by the one or more processors, further cause the apparatus to at least: input the first coded input map to a fifth encoder layer and encode, by the fifth encoder layer, the first coded input map and the first coded map of the first modality into a fifth coded map of the first modality; input the first coded input map to a seventh encoder layer and encode, by the seventh encoder layer, the first coded input map and the third coded map of the second modality into a seventh coded map of the second modality. inputting the second coded input map to a sixth encoder layer and encoding, by the sixth encoder layer, the second coded input map and the second coded map of the first modality into a sixth coded map of the first modality; inputting the second coded input map to an eighth encoder layer and encoding, by the eighth encoder layer, the second coded input map and the fourth coded map of the second modality into an eighth coded map of the second modality.

[0030] A second matching descriptor layer of the second hierarchy is configured with a respective third parameter, and the instructions, when executed by the one or more processors, may further cause the apparatus to perform the step of training the apparatus to determine the first, second, and third parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold, and such that the first point correlation and the second point correlation are increased.

[0031] The seventh encoder layer may be the same as the eighth encoder layer.

[0032] The seventh encoder layer may be the same as the third encoder layer.

[0033] The eighth encoder layer may be the same as the fourth encoder layer.

[0034] The third neural fusion layer may be the same as the fourth neural fusion layer.

[0035] The third neural fusion layer may be the same as the first neural fusion layer.

[0036] The fourth neural fusion layer may be the same as the second neural fusion layer.

[0037] The instructions, when executed by the one or more processors, may further cause the apparatus to perform at least one of: re-localizing a second sequence of images in the second coordinate system in a first sequence of images in the first coordinate system based on the estimated transformation; or co-localizing a fourth sequence of images in the second coordinate system with a third sequence of images in the first coordinate system based on the estimated transformation; or looping through a portion of the image sequence with the entire image sequence based on the estimated transformation, where positions of images of the portion of the image sequence are shown in the second coordinate system and images of the entire image sequence are shown in the first coordinate system.

[0038] According to a second aspect of the invention, a method includes the steps of inputting a first 3D point cloud of a first modality and a first grid including one or more first cells into a first encoder layer, where coordinates of points of the first 3D point cloud are expressed in a first coordinate system; and encoding, by the first encoder layer, the first 3D point cloud of the first modality into a first encoded map of the first modality, where the first encoded map of the first modality includes, for each first cell of the first grid, a respective first feature centroid point and a corresponding first feature centroid point. inputting the first encoded input map into a first matching descriptor layer of the first hierarchical layer, the first encoded input map being based on the first encoded map of the first modality and comprising, for each of the first cells, a respective first joint feature center point and a respective first input feature vector, the coordinates of the first joint feature center point being expressed in the first coordinate system; inputting the first encoded input map into a first matching descriptor layer of the first hierarchical layer, the first encoded input map being based on the first encoded map of the first modality and comprising, for each of the first cells, a respective first joint feature center point and a respective first input feature vector, the coordinates of the first joint feature center point being expressed in the first coordinate system; , a second grid including one or more second cells, into a second encoder layer, where coordinates of points of the second 3D point cloud are expressed in a second coordinate system; and encoding, by the second encoder layer, the second 3D point cloud of the first modality into a second encoded map of the first modality, where the second encoded map of the first modality includes, for each of the second cells of the second grid, a respective second feature center point and a respective second feature vector, where the coordinates of the second feature center points are expressed in a second coordinate system. a step of inputting a second coded input map into a first matching descriptor layer, the second coded input map being based on a second coded map of a first modality and comprising, for each of the second cells, a respective second joint feature center point and a respective second input feature vector, the coordinates of the second joint feature center points being represented in a second coordinate system; and a step of inputting the first input feature vectors, optionally their first feature center points and the second input feature vectors,adapting each of the first input feature vectors based on the first feature center points and optionally their second feature center points by a first matching descriptor layer to obtain a first joint map, the first joint map including, for each of the first joint feature center points, a respective first joint feature vector; adapting each of the second input feature vectors based on the first input feature vectors, optionally their first feature center points and the second input feature vectors, optionally their second feature center points by a first matching descriptor layer to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; adapting each of the second input feature vectors based on the first input feature vectors, optionally their first feature center points and the second input feature vectors, optionally their second feature center points by a first optimal matching layer to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; calculating, by the first optimal matching layer, for each of the first cells and each of the second cells, a first point correlation between each of the first cells and each of the second cells based on a similarity between the first joint feature vector of each of the first cells and several second joint feature vectors of several second cells and based on a similarity between the second joint feature vector of each of the second cells and several first joint feature vectors of several first cells; and, for each of the first cells and each of the second cells, checking whether at least one of one or more first correlation conditions is satisfied for each of the first cells and each of the second cells, wherein the one or more first correlation conditions include: a first point correlation between each first cell and each second cell is greater than a first correlation threshold for the first tier; or a first point correlation between each first cell and each second cell is among a maximum k values ​​of first point correlations, k being a fixed value; An apparatus is provided that performs the steps of: extracting, for each first cell and each second cell, coordinates of a first joint feature center point of the respective first cell in a first coordinate system and coordinates of a second joint feature center point of the respective second cell in a second coordinate system, and obtaining respective pairs of extracted coordinates of the first hierarchical hierarchy in response to checking whether at least one of one or more first correlation conditions is satisfied for each first cell and each second cell; and calculating an estimated transformation between the first coordinate system and the second coordinate system based on the pairs of extracted coordinates of the first hierarchical hierarchy.

[0039] The first encoder layer, the second encoder layer, and the first matching descriptor layer of the first hierarchy are configured with respective first parameters, and the method may further include a step of training the apparatus to determine the first parameters such that a deviation between an estimated transformation between the first coordinate system and the second coordinate system and a known transformation is less than a transformation estimation threshold and such that a first point correlation is increased.

[0040] The first modality may include one of a photographic image, a LIDAR image, and an ultrasound image.

[0041] The first encoder layer may be the same as the second encoder layer.

[0042] The method may further include at least a step of generating a first 3D point cloud based on a sequence of first images of a first modality, where a respective position in the first coordinate system is associated with each of the first images, or a step of generating a second 3D point cloud based on a sequence of second images of the first modality, where a respective position in the second coordinate system is associated with each of the second images.

[0043] The first coded input map may be a first coded map and the first joint feature center points may be first feature center points, the second coded input map may be a second coded map and the second joint feature center points may be second feature center points.

[0044] The method further includes inputting a third 3D point cloud of a second modality and the first grid into a third encoder layer, where coordinates of points of the third 3D point cloud are expressed in a first coordinate system, the second modality being different from the first modality; and encoding, by the third encoder layer, the third 3D point cloud of the second modality into a third encoded map of the second modality, where the third encoded map of the second modality includes, for each of the first cells of the first grid, a respective third feature center point and a respective third feature center point. a feature vector of the first cell, the coordinates of the third feature center points being represented in a first coordinate system; inputting the third encoded map to a first neural fusion layer; inputting the first encoded map to the first neural fusion layer; and generating a first encoded input map by the first neural fusion layer based on the first encoded map and the third encoded map, wherein for each of the first cells, the respective first joint feature center points are represented by the first feature center points of the respective first cells and the third joint feature center points of the respective first cells. a step of deriving a feature vector based on the third feature center points, and each first input feature vector being based on the first feature vector and the first feature vector, optionally based on the first feature center points and the third feature vector of each first cell adapted by the first feature center points, optionally based on the third feature vector and the first feature vector, optionally based on the third feature center points and the third feature vector of each first cell adapted by the first feature center points, optionally based on the third feature center points and the first feature vector, optionally based on the third feature center points and the third feature vector of each first cell adapted by the first feature center points, optionally based on the third feature center points; inputting a fourth 3D point cloud of the second modality and the second grid into a fourth encoder layer, where coordinates of points of the fourth 3D point cloud are represented in the second coordinate system; and encoding by the fourth encoder layer the fourth 3D point cloud of the second modality into a fourth encoded map of the second modality, where the fourth encoded map of the second modality includes, for each second cell of the second grid, a respective fourth feature center point and a respective fourth feature vector, where coordinates of the fourth feature center point are represented in the second coordinate system;inputting the fourth encoded map into a second neural fusion layer; inputting the second encoded map into a second neural fusion layer; generating a second encoded input map by the second neural fusion layer based on the second encoded map and the fourth encoded map, where for each of the second cells, the respective second joint feature center points are based on the second feature center points of the respective second cells and the fourth feature center points of the respective second cells; and the respective second input feature vectors are based on the second feature vector and the second feature vector, optionally the second feature center points and the fourth feature vector of the respective second cells adapted by their second feature center points, optionally their fourth feature center points, and the fourth feature vector and the second feature vector, optionally the fourth feature center points and the fourth feature vector of the respective second cells adapted by their second feature center points, optionally their fourth feature center points.

[0045] The second modality may include at least one of a photographic image, an RF fingerprint, a LIDAR image, or an ultrasound image.

[0046] The third encoder layer, the fourth encoder layer, the first neural fusion layer, and the second neural fusion layer are configured with respective second parameters, and the method may further include training the apparatus to determine the first parameters and the second parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold and such that the first point correlation is increased.

[0047] The first image may be taken by a first sensor of the first sensor device and the second image may be taken by a second sensor of the second sensor device, and the method may further include making a plurality of first measurements of a second modality using a third sensor of the first sensor device; making a plurality of second measurements of the second modality using a fourth sensor of the second sensor device; generating a third 3D point cloud of the second modality based on the plurality of first measurements, a position associated with the first image, and a calibration between the first sensor of the first sensor device and the third sensor of the first sensor device; and generating a fourth 3D point cloud of the second modality based on the plurality of second measurements, a position associated with the second image, and a calibration between the second sensor of the second sensor device and the fourth sensor of the second sensor device.

[0048] The third encoder layer may be the same as the fourth encoder layer.

[0049] The first neural fusion layer may be the same as the second neural fusion layer.

[0050] The method further includes inputting the first encoded map of the first modality and a third grid including one or more third cells into a fifth encoder layer, each of the third cells including two or more of the first cells, and the first grid including two or more of the first cells; and encoding, by the fifth encoder layer, the first encoded map of the first modality into a fifth encoded map of the first modality, the fifth encoded map of the first modality including, for each of the third cells of the third grid: inputting the third encoded input map into a second matching descriptor layer of a second hierarchical layer, the third encoded input map being based on the fifth encoded map of the first modality and comprising, for each of the third cells, a respective third joint feature center point and a respective third input feature vector, the coordinates of the third joint feature center point being expressed in the first coordinate system; inputting the second encoded map of the first modality and a fourth grid including one or more fourth cells into a sixth encoder layer, each of the fourth cells including two or more of the second cells, the second grid including two or more of the second cells; encoding the second encoded map of the first modality into a sixth encoded map of the first modality by the sixth encoder layer, the sixth encoded map of the first modality including, for each of the fourth cells of the fourth grid, a respective sixth feature center point and a corresponding one of the sixth feature center points; inputting the fourth encoded input map into a second matching descriptor layer, the fourth encoded input map being based on the sixth encoded map of the first modality and comprising, for each of the fourth cells, a respective fourth joint feature center point and a respective fourth input feature vector, the coordinates of the fourth joint feature center point being indicated in the second coordinate system; inputting the fourth encoded input map into a second matching descriptor layer, the fourth encoded input map being based on the sixth encoded map of the first modality and comprising, for each of the fourth cells, a respective fourth joint feature center point and a respective fourth input feature vector, the coordinates of the fourth joint feature center point being indicated in the second coordinate system;and optionally adapting each of the third input feature vectors based on their third joint feature center points and a fourth input feature vector, optionally based on their fourth joint feature center points; obtaining a third joint map, the third joint map including, for each of the third joint feature center points, a respective third joint feature vector; adapting each of the fourth input feature vectors based on the third input feature vector, optionally based on their third joint feature center points and a fourth input feature vector, optionally based on their fourth joint feature center points; obtaining a fourth joint map, the fourth joint map including, for each of the fourth joint feature center points, a respective fourth joint feature vector; and adapting each of the fourth input feature vectors based on the third input feature vector, optionally based on their third joint feature center points and a fourth input feature vector, optionally based on their fourth joint feature center points; calculating, by the second optimal matching layer, for each of the third cells and each of the fourth cells, a second point correlation between each of the third cells and each of the fourth cells based on the similarity between the third joint feature vector of each of the third cells and the fourth joint feature vector of the fourth cell and based on the similarity between the fourth joint feature vector of each of the fourth cells and the third joint feature vector of the third cell; and checking, for each of the third cells and each of the fourth cells, whether at least one of one or more second correlation conditions for each of the third cells and each of the fourth cells is satisfied, the one or more second correlation conditions being: a second point correlation between each third cell and each fourth cell is greater than a second correlation threshold of the second tier; or The second point correlation between each third cell and each fourth cell is among the maximum l values ​​of the second point correlation, where l is a fixed value.

[0051] The method may further include a step of prohibiting the first matching descriptor layer from checking, for each of the first cells and each of the second cells, whether a first point correlation between the first joint feature vector of the respective first cell and the second joint feature vector of the respective second cell is greater than a first correlation threshold of the first tier in response to checking that at least one of the one or more second correlation conditions for each of the third cell and each of the fourth cell is satisfied.

[0052] The method may further include the steps of extracting, for each of the third cells and each of the fourth cells, coordinates of a third joint feature center point of the respective third cell in the first coordinate system and coordinates of a fourth joint feature center point of the respective fourth cell in the second coordinate system, and obtaining respective pairs of extracted coordinates of the second hierarchy in response to checking that at least one of the one or more second correlation conditions for each of the third cells and each of the fourth cells is satisfied, and calculating an estimated transformation between the first coordinate system and the second coordinate system based on the pairs of extracted coordinates of the first hierarchy and the pairs of extracted coordinates of the second hierarchy.

[0053] The method further includes calculating the estimated transformation such that in the calculating step, extracted coordinate pairs of the first hierarchical layer have different weights than extracted coordinate pairs of the second hierarchical layer.

[0054] A second matching descriptor layer of the second hierarchy is configured with respective third parameters, and the method may further include training the apparatus to determine the first and third parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold and such that the first point correlation and the second point correlation are increased.

[0055] The fifth encoder layer may be the same as the sixth encoder layer.

[0056] The fifth encoder layer may be the same as the first encoder layer.

[0057] The sixth encoder layer may be the same as the second encoder layer.

[0058] The second matching descriptor layer may be the same as the first matching descriptor layer.

[0059] The second optimal matching layer may be the same as the first optimal matching layer.

[0060] The third encoded input map may be a third encoded map of the first modality and the third joint feature center points may be third feature center points, the fourth encoded input map may be a fourth encoded map of the first modality and the fourth joint feature center points may be fourth feature center points.

[0061] The method further includes inputting the third encoded map of the second modality and the third grid to a seventh encoder layer; encoding, by the seventh encoder layer, the third encoded map of the second modality into a seventh encoded map of the second modality, the seventh encoded map of the second modality comprising, for each of the third cells of the third grid, a respective seventh feature center point and a respective seventh feature vector, the coordinates of the seventh feature center point being expressed in the first coordinate system; inputting the fifth encoded map to a third neural fusion layer; and generating a third encoded input map by the third neural fusion layer based on the seventh encoded map and the fifth encoded map, wherein for each of the third cells, the respective third joint feature center points are based on the fifth feature center points of the respective third cells and the seventh feature center points of the respective third cells, and the respective third input feature vectors are based on the fifth feature vector and the seventh feature vector. and based on a fifth feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector; inputting the fourth encoded map of the second modality and the fourth grid to an eighth encoder layer; and encoding the fourth encoded map of the second modality into an eighth encoded map of the second modality by the eighth encoder layer, wherein the eighth encoded map of the second modality is an eighth encoded map of the second modality based on a fifth feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector of each third cell adapted by the fifth feature vector and the seventh feature vector; and for each of the fourth cells, a respective eighth feature center point and a respective eighth feature vector, the coordinates of the eighth feature center point being represented in a second coordinate system; inputting the eighth encoded map to a fourth neural fusion layer; inputting the sixth encoded map to the fourth neural fusion layer; and generating a fourth encoded input map by the fourth neural fusion layer based on the sixth encoded map and the eighth encoded map, wherein for each of the fourth cells, the respective fourth joint feature center points are represented in a second coordinate system;Based on the sixth feature center point of each fourth cell and the eighth feature center point of each fourth cell, and based on the sixth feature vector of each fourth cell adapted by the sixth feature vector and the eighth feature vector, each fourth input feature vector may include a step of: based on the sixth feature vector of each fourth cell adapted by the sixth feature vector and the eighth feature vector, and based on the eighth feature vector of each fourth cell adapted by the sixth feature vector and the eighth feature vector.

[0062] The method further includes inputting the first coded input map to a fifth encoder layer and encoding, by the fifth encoder layer, the first coded input map and the first coded map of the first modality into a fifth coded map of the first modality; inputting the first coded input map to a seventh encoder layer and encoding, by the seventh encoder layer, the first coded input map and the third coded map of the second modality into a seventh coded map of the second modality. , inputting the second coded input map to a sixth encoder layer and encoding, by the sixth encoder layer, the second coded input map and the second coded map of the first modality into a sixth coded map of the first modality; and inputting the second coded input map to an eighth encoder layer and encoding, by the eighth encoder layer, the second coded input map and the fourth coded map of the second modality into an eighth coded map of the second modality.

[0063] A second matching descriptor layer of the second hierarchy is configured with respective third parameters, and the method may further include a step of training the apparatus to determine the first, second, and third parameters such that a deviation between the estimated transformation and the known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold, and such that the first point correlation and the second point correlation are increased.

[0064] The seventh encoder layer may be the same as the eighth encoder layer.

[0065] The seventh encoder layer may be the same as the third encoder layer.

[0066] The eighth encoder layer may be the same as the fourth encoder layer.

[0067] The third neural fusion layer may be the same as the fourth neural fusion layer.

[0068] The third neural fusion layer may be the same as the first neural fusion layer.

[0069] The fourth neural fusion layer may be the same as the second neural fusion layer.

[0070] The method may further include at least re-localizing a second sequence of images in a second coordinate system in a first sequence of images in a first coordinate system based on the estimated transformation, or co-localizing a fourth sequence of images in the second coordinate system with a third sequence of images in the first coordinate system based on the estimated transformation, or loop-closing a portion of the image sequence with the entire image sequence based on the estimated transformation, wherein the positions of the images of the portion of the image sequence are shown in the second coordinate system and the images of the entire image sequence are shown in the first coordinate system.

[0071] The method of the second aspect may be a method of position estimation.

[0072] According to a third aspect of the present invention there is provided a computer program product comprising a set of instructions arranged, when executed on an apparatus, to cause the apparatus to carry out the method according to the second aspect.

[0073] The computer program product may be embodied as a computer readable medium or may be directly loadable into a computer.

[0074] According to some exemplary embodiments, at least one of the following advantages may be achieved. · Robustness in location estimation can be improved because the location estimation is based on a point cloud (which can be generated based on a sequence of images) instead of based on a single image. Robust location estimation can be performed in real time. Large areas can be covered. Ambiguities (e.g. in repetitive regions) can be especially reduced by fusion of different modalities. The accuracy of the location estimation can be improved and especially scaled by hierarchical methods. RF maps can be predicted from the implicit propagation models contained in the data, and thus sensor poses can be predicted even in areas where no measurements have been made before. The process can be camera independent and less susceptible to image distortions. · Training data collection does not require manual effort. In order to achieve high accuracy of rigid transformation estimation even on the coarsest layers, in the encoding layer the 3D positions of the feature vectors are regressed in a data-driven, differentiable and learnable manner.

[0075] It will be understood that any of the above modifications can be applied singly or in combination to the respective aspects to which they refer, unless expressly stated as excluding alternatives.

[0076] Further details, features, objects and advantages will be apparent from the following detailed description of preferred exemplary embodiments, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0077] [Figure 1] FIG. 1 illustrates SLAM building blocks. [Diagram 2] FIG. 2 illustrates a functional structure of an exemplary embodiment. [Diagram 3] FIG. 2 illustrates a functional structure of an exemplary embodiment. [Figure 4] FIG. 2 illustrates a functional structure of an exemplary embodiment. [Diagram 5] FIG. 5 shows a cut-out of the functional structure of the exemplary embodiment of FIG. [Figure 6] FIG. 1 illustrates an apparatus in accordance with an exemplary embodiment. [Figure 7] FIG. 1 illustrates a method according to an exemplary embodiment. [Figure 8] 1 illustrates an apparatus in accordance with an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0078] The present invention has been described in detail below, and the features of the present embodiment can be freely combined and used with reference to the accompanying drawings. However, it should be clearly understood that the description of the specific embodiment is given by way of example only, and is in no way intended to limit the present invention to the disclosed details.

[0079] Furthermore, in some cases, only an apparatus or only a method is described, but it should be understood that the apparatus is configured to perform the corresponding method.

[0080] Some example embodiments aim to provide a transformation between a first map and a second map such that the first map matches the second map. For example, the first map may be a global map, e.g. of buildings, floors, etc., and the second map may be a local map acquired based on images from a sensor, e.g. of a robot. Each of the first map and the second map may be acquired from a respective sequence of images of one or more modalities, such as visual images, LIDAR images, RF measurements, or ultrasound images. At least one of the modalities allows for creating a map based on the respective images, such as the modality of visual images or LIDAR images.

[0081] The second map has its own coordinate system, and typically the coordinate system of the second map is different from that of the first map. A SE(3) transformation is defined between the coordinate frames. The transformation between the coordinates of the second map and the coordinates of the first map may include a translation (represented by a translation vector t with three independent components) and a rotation (represented by a rotation matrix R with three independent components (a rotation matrix is ​​a 3×3 matrix with three degrees of freedom)). Thus, in general, a pose has six independent components (6DoF, also called "6D pose", corresponding to position and orientation). In some exemplary embodiments, a pose may have less than 6DoF if the respective limitations are imposed.

[0082] In particular, one, two, or three of the following operations may be performed by some example embodiments. Relocalization: Matching a local map (second map) with a global map (first map). Co-localization: Matching a second local map (second map) with a first local map (first map). · Loop Closure: A partial local map (second map), obtained from a subset of the images from which the local map is obtained, is aligned with the (global) local map (first map). This procedure is useful to correct drifts in the local map that may result from the accumulation of small odometry errors.

[0083] In the following, several exemplary embodiments for relocation are described, where the first map is a global map and the second map is a local map. The transformation indicates the position and orientation (pose) of the sensor used to acquire the image on which the local map is based in the coordinate system of the global map. The described procedures can be applied correspondingly to colocation and loop closure.

[0084] The 3D structure of the environment may be represented by a point cloud. Here, as an example, we describe the generation of the point cloud from camera images. Each camera image is associated with the position of the camera at the time of capture. Typically, visual inertial odometry allows obtaining the relative transformation between adjacent camera frames, which serves as a good initial value for SLAM state estimation. The output of the SLAM algorithm will further refine the initial relative transformation of the frames and provide depth estimates for a subset of image pixels. Based on the estimated depth information and the relative transformation, a 3D point cloud is constructed that represents the structure of the environment. Relocation estimation can be used when tracking is lost between frames and / or due to drift error correction.

[0085] In the process of creating a localization map and localization algorithm, robustness can be achieved by multimodal sensor fusion such as that used by visual inertial odometry. Here, an inertial measurement unit (IMU) can significantly improve robustness because it is not subject to significant light fluctuations, flare, dynamic environments such as cameras and can measure the angular velocity and acceleration of an agent. In a tightly coupled sensor fusion system, information from the camera image can facilitate the estimation of gravity vector, gyroscope, and accelerometer bias, resulting in significant drift reduction. In some exemplary embodiments, the position of the camera at the time of imaging can be obtained only by visual odometry or inertial odometry, but not by visual inertial odometry.

[0086] The mathematical framework for creating a 3D reconstruction of the environment while determining the 6DoF track of the camera is the Simultaneous Localization and Mapping (SLAM) algorithm or the Structure for Motion (SFM) algorithm. There are several versions of such algorithms.

[0087] Typically, a SLAM algorithm includes two stages. In the first stage (global mapping stage), the SLAM algorithm provides a map suitable for localization purposes. Depending on the number of images used to obtain the map, this stage may be slow or fast. In the next stage, the localization may be used to identify the pose of a new image or image sequence in the acquired map.

[0088] In the global mapping stage, a complex optimization problem called full bundle adjustment is performed to reduce any pose errors and 3D reconstruction errors. An element of the map optimization is loop closure detection. By detecting loops in the mapping process, the accumulated drift and 3D reconstruction errors can be significantly reduced. This non-convex optimization involves a large number of state variables, making this stage very computationally intensive and therefore typically performed offline. Local mapping and position estimation are often performed online, but can also be performed offline.

[0089] [Generating point clouds from a sequence of images] With reference to FIG. 1, a SLAM algorithm (SLAM pipeline) is described as an example of how to generate a point cloud based on a sequence of images.

[0090] The main building blocks of an exemplary SLAM pipeline are tracking, loop closure, local map construction, and global image search.

[0091] In the tracking phase, the goal is to determine the relative odometry between two consecutive frames and to determine whether the new image frame is a keyframe or not. There is no exact definition of a keyframe, since different implementations define different heuristics for keyframe detection. Generally, a keyframe contains new information about the environment that is not available in the previous keyframe. To calculate the odometry between two consecutive frames, feature points (and corresponding feature vectors) are extracted and tracked. If the feature tracking is successful, the relative pose of the two images can be calculated, and depending on the sensor settings, the depth information of the feature points can be estimated.

[0092] If feature tracking fails, a global relocalization procedure is initiated. It aims to find the current pose in the map that exists so far. This task is particularly challenging since the problem complexity (search space) increases as new keyframes are inserted into the keyframe database. To attempt a global image retrieval from the database of keyframes, a global descriptor of the query image is created. In many conventional solutions, a Bag of Words (BoW) algorithm quantizes the image feature descriptors and creates a frequency histogram of the quantized feature vectors. As an alternative, deep learning models can be trained to generate state-of-the-art global image descriptors that can be searched very efficiently. After creating the global descriptor, a similarity matrix is ​​calculated between the global descriptor of the query image and the global descriptors of all stored keyframes. The images with the highest similarity scores are retrieved as candidates for the image matching process, followed by different filtering algorithms that further refine the candidate set. If the retrieval process is successful, the global pose of the query image is calculated and a local map of the environment is searched based on the co-visibility graph. The local map contains keyframes, 3D feature points with local feature descriptors, and depth information. The feature points of the 3D map are searched on the 2D query image, and from these 3D-2D matchings, the camera pose can then be recovered in a Perspective-n-Point (PnP) manner. Tracking continues by analyzing new images.

[0093] If a new keyframe is detected, a loop closure procedure can be initiated. Here, the goal is to find similar keyframes from the past to find if the camera has already visited the same location. The search procedure is similar to the one presented earlier. The output of this block is the loop constraint that is added to the bundle adjustment optimization. The optimization can exploit the fact that along a loop in the pose graph, the overall pose change should be identical.

[0094] The local map construction module is responsible for refining the local map by minimizing the discrepancy between the 3D structure and the camera pose in a bundle adjustment procedure. The discrepancy is formulated as a different residual function. This non-convex optimization is formulated as a least-squares minimization problem. During the optimization, a significant number of state variables are updated: pose, depth values ​​(of the points), sensor extrinsic and intrinsic parameters. A reprojection error function connects the pose, depth, and calibration state variables to the optical flow measurements. If available, an IMU-based residual function may connect adjacent poses and IMU-internal state variables with pre-integrated IMU measurements. Finally, depending on the purpose of the created map, the number of state variables can be reduced, resulting in the local map representing only a limited area of ​​the environment. On the contrary, if every keyframe is retained, the local map will represent a larger area of ​​the environment.

[0095] In the global map optimization component (which can be performed in the background or at an offline stage), the goal is to estimate very accurately (i.) a 3D reconstruction of the environment, (ii.) the camera pose from where the images were captured, and (iii.) descriptors attached to the 3D points that help solve the localization problem. Marginalization or keyframe culling can be performed to limit the computational complexity to a given processing hardware.

[0096] Thus, from a sequence of images, a point cloud is generated, where each point in the point cloud is defined by its coordinates in the respective coordinate system and a feature vector, which describes the properties of the point, such as RGB, reflectance, etc.

[0097] The use of point clouds as input for relocalization (or colocalization or loop closing) has advantages over the use of single images: existing solutions attempt to find the pose of a single query image. However, there are environments where the lack of descriptive features or repetitive patterns leads to high ambiguity. This ambiguity can be resolved (at least in part) by the use of point clouds based on a sequence of captured images and / or the fusion of different modalities.

[0098] Image-based localization problems are known to be difficult to solve because lighting conditions encountered during the day or seasonal changes in the environment result in different artifacts. Creating visual descriptors and models that can generalize to all conditions is particularly challenging. By creating a local map for the structural representation and registering the RF measurements with the camera pose, a better representation can be constructed for localization purposes. After the global map is constructed, different deep learning models can be trained to register the features of the local maps created in different conditions. Nevertheless, it is also possible to train one model that generalizes to all conditions by creating a set of local maps of the same area at different times of the day and / or season.

[0099] Location estimation, image search algorithms, and data-driven training models can be significantly affected by camera lens distortion. When query images are created with a different camera than the keyframe images, only suboptimal accuracy is achieved. Some exemplary embodiments are camera independent, since the camera image sequence is converted to a point cloud.

[0100] As another option, the 3D point cloud may be pre-defined based on planning data that describes, for example, how an environment (such as a building) was planned. However, if the 3D point cloud is based on planning data, there is a risk that the reality of the environment captured by a sensor (such as a camera) will not match the planning data.

[0101] Basic System According to Some Example Embodiments With reference to Fig. 2, this section describes a basic system according to some exemplary embodiments. The basic system comprises two pipelines 1a, 1b, where pipeline 1a is for a global map represented by a global 3D point cloud and pipeline 1b is for a local map represented by a local 3D point cloud. The pipelines 1a, 1b and their parameters may be the same for both point clouds or may be different for the two point clouds. The pipelines 1a, 1b operate independently of each other. The parameters of the layers included in the pipelines are learned in a training process that is further described below. The same applies to the pipelines described for further embodiments below.

[0102] For the estimation (query phase), the 3D point clouds (coordinates and respective feature vectors) are input to each of the pipelines. In addition, a respective 3D grid defining the cells (voxels) is input to each of the pipelines. The 3D grids may be the same or different for pipelines 1a and 1b. The 3D grid for the local map may be shifted and / or rotated with respect to the 3D grid for the global map. Furthermore, the number of (associated) cells may be the same or different for the global and local maps, since these maps may cover volumes of different sizes. Typically, the cells of the grid are cubes, but they may have any shape (e.g., a general rectangular prism or pyramid) that can cover the entire volume. The size of the cells of the 3D grid may be selected depending on the required resolution and the available processing power. The number of cells of the 3D grid may be one, but is typically two or more. The cells of the 3D grid may extend to infinity in one direction (e.g., height direction) such that the cells can be represented by only two coordinates. That is, a 3D grid may be equivalent to a 2D grid.

[0103] The functions of the pipelines 1a and 1b (collectively referred to as 1) will be explained below as representative examples.

[0104] Pipeline 1 includes structural encoder layers ("first and second encoder layers"). The structural encoder layers encode the 3D point cloud in a transformation-invariant manner. This task is challenging because the points of the 3D point cloud are not regularly spaced. Some cells of the 3D grid may not contain any points, while other cells may contain one or more points with their respective feature vectors. There are several suitable encoding techniques, such as (i.) projection networks, where points are projected onto regular 2D and 3D grid structures followed by regular 2D or 3D convolution operators, (ii.) graph convolution networks, (iii.) point-wise multi-layer perceptron networks, or (iv.) point convolution networks.

[0105] The 3D grid can be used for data compression. By applying data compression, the number of points is reduced and becomes better organized structurally. When data compression is performed, the goal is to define (i.) a structural descriptor encoded as a feature vector for each grid cell, and (ii.) regress feature (center) point coordinates of the structural and RF descriptors used in computing the rigid body transformation.

[0106] The point compression operation for one 3D grid configuration can be formulated as follows: (i.) Define a 3D grid with a given grid cell size. (ii.) Group the input points based on grid cells. (iii.) Regress the locations of the output feature centroids and any additional parameters of the selected convolution type. (iv.) For every feature center point based on the feature vectors of the support points (the set of support points is a collection of neighbors of the feature center point, typically defined by a convolution radius or based on a neighbor selection algorithm, and further, the convolution radius may be equal to, greater than, or less than the size of the grid cell), a feature vector is created for the output feature point (i.e., the feature center point). The feature vector is calculated by performing several groups of convolution, normalization, and activation operations.

[0107] Thus, for each cell, the structural encoder layer obtains the coordinates of a feature centre point, i.e. a point representing the cell that has a feature vector representing the cell's features.

[0108] Note that due to the encoding, features in the encoded global map may not have their usual meaning, such as RGB or reflectance: they must be considered as abstract features. The same is true for the output of the other encoding layers described below.

[0109] The first and second encoder layers of pipelines 1a and 1b may have the same or different architectures (e.g., the number of convolutional layers, etc.) and / or the same or different parameters. In some example embodiments, the first and second encoder layers may be the same.

[0110] The output of the structural encoder layer of both pipelines (i.e., their respective encoded maps) is input to the matching descriptor layer. In other words, in this exemplary embodiment, the output of the structural encoder layer can be considered as an encoded input map to the matching descriptor layer. The matching descriptor layer (and other layers following the matching descriptor layer) is common to the global map and the local map, and does not belong to either of the pipelines 1a and 1b that are specific to the global map or the local map.

[0111] In the matching descriptor layer, a function with learnable parameters is applied to each feature vector of the feature center points of each encoded map. The function describes how the feature vector and the position of the other feature center points of each encoded map affect the feature vector of the selected feature center point of the same map. The function may depend on the distance between the respective feature center points. An example of such a mechanism is self-attention. In addition, a further function is applied that describes how the feature vector of each feature center point of one of the encoded maps is affected by the feature vector and the position of the feature center point of one of the other encoded maps. The further function may depend on the distance between the respective feature center points. An example of such a mechanism is mutual attention. The parameters of these functions are learned by a training process that is further described below.

[0112] The result of the matching descriptor layer is a joint global map and a joint local map (the "joint map"), which contains the encoded feature vectors for each feature center point of the two encoded maps.

[0113] The joint map is input to an optimal matching layer, where the similarity between the feature vectors of the feature center points in the joint global map and the feature vectors of the feature center points in the joint local map is calculated. Based on the similarity values ​​as a second step, a soft-assignment matrix is ​​calculated that contains the matching probability for each pair of features. The soft-assignments are found by solving an optimal transportation problem, which is formulated as a sum-maximization problem. We call the values ​​of this soft-assignment matrix point correspondences.

[0114] The soft-assignment matrix has as many columns (or rows) as the number of cells in the encoded global map, as many rows (or columns) as the number of cells in the encoded local map, and one or more dustbin rows and one or more dustbin columns for indicating that a first feature vector does not correspond to any second feature vector. In the dustbin rows and dustbin columns, the initial values ​​are parameters learned during training. The Sinkhorn algorithm can be applied to this initial similarity matrix to obtain the soft-assignment matrix. Top k values ​​(k is a predefined constant) and / or values ​​greater than a threshold (correlation threshold) can be selected from the soft-assignment matrix to identify cells whose feature vectors correspond to each other. The output of the optimal matching layer is a set of corresponding point pairs.

[0115] The transformation layer extracts the coordinates of the feature center points of cells in the encoded global map and the coordinates of the feature center points of the corresponding cells in the encoded local map (i.e., for cells that are assumed to correspond to each other according to the soft assignment matrix) and calculates the transformation between these feature center points (i.e., position t and orientation R). If there are multiple correspondences, they are considered jointly by an algorithm such as a least-squares algorithm. By transformation, each pose in the local coordinate system can be transformed to the corresponding pose in the global coordinate system (and vice versa).

[0116] Modality Fusion According to Some Exemplary Embodiments An exemplary embodiment is described with reference to Fig. 3 in which maps obtained by different modalities are fused, for example images in the visible range (photo images), RF fingerprints, LIDAR images, etc. In the exemplary embodiment of Fig. 3, maps obtained from photo images and RF measurements are fused.

[0117] In an exemplary embodiment, the RF fingerprint is registered to the 6DoF camera pose at the time of image capture to generate a respective global or local map (point cloud). That is, the calibration (i.e., distance and relative orientation) between the camera and the RF antenna for receiving the RF signal is known for each sensor (device), so the pose of the RF fingerprint can be shown in the camera's coordinate system. If the RF measurement is performed at a time when no photographic image was captured, and if the sensor may have moved since the last image capture, the pose of the RF measurement can be obtained by interpolation between the poses at the image captures before and after the RF measurement (or by extrapolation from the poses at the image captures before (or after) the RF measurement).

[0118] From these joint measurements, a joint feature vector is generated by fusing spatial information from the point cloud and the RF fingerprint. If registration fails due to non-overlapping maps or there is not enough information to disambiguate the match, the local map can be expanded by acquiring more images until the match is successful. The RF fingerprint may comprise, for example, the BSSID and / or RSSI values ​​of one or more RF transmitters, such as WiFi APs.

[0119] During the data collection phase, image-based position estimation (in the respective coordinate system) determines the pose of the RF fingerprint measurements. Thus, a respective RF map is obtained that contains RF fingerprints with a specific pose. The RF map can be considered as a 3D point cloud of modality RF measurements. The 3D coordinates are the locations where the measurements were made, and the feature vector is composed by the measured RF fingerprints and the orientation of the measuring device. The RF map can be coded similarly to the structural map.

[0120] As shown in FIG. 3, there is a global map pipeline 2a and a local map pipeline 2b. These pipelines comprise a first and a second (structural) encoder layer, as described with respect to the exemplary embodiment of FIG. 2. In addition, each of the pipelines comprises a respective RF map encoder layer ("third and fourth encoder layers") and a respective neural fusion layer. The RF map encoder layer and / or the neural fusion layer and their parameters may be the same or different for the two pipelines. Unlike FIG. 2, the encoded global map output from the structural encoder layer is input to the neural fusion layer of the respective pipeline, rather than the matching descriptor layer. The pipelines 2a, 2b operate independently of each other.

[0121] The RF map encoder layer corresponds functionally to the structural encoder layer, but instead of features of the points of the point cloud encoded in the structural encoder layer, the RF map encoder layer encodes the RF fingerprints with their poses. The same 3D grid as in the structural encoder layer is used. The resulting encoded RF map contains feature center points of each cell with a respective encoded RF fingerprint, where the feature center points of the encoded global / local RF map may be the same or different from the feature center points of the encoded global / local map. The parameters of the RF map encoder layer are learned by a training process, which is further described below.

[0122] Both the encoded global / local RF maps and the encoded global / local structural maps of each of the pipelines 2a, 2b are input to the respective neural fusion layers. In the neural fusion layers, similar to the matching descriptor layer described with respect to the exemplary embodiment of FIG. 2, a function is applied to each of the feature vectors of all feature center points of the respective encoded maps. The function describes how other points of the respective encoded maps affect a selected point of the map of the same modality. That is, the structural feature vector is updated based on the structural feature vector, and the RF feature vector is updated based on the RF feature vector. The function may depend on the distance between the respective feature center points. An example of such a mechanism is self-attention. Furthermore, a further function (which may be the same as or different from the further function used in the matching descriptor layer) is applied that describes how each point of the encoded map of one modality is affected by a point of the encoded map of the other modality, and vice versa. That is, the RF feature vector is updated based on the structural feature vector, and vice versa. The further function may depend on the distance between the respective feature center points. An example of such a mechanism is cross-attention. The parameters of these functions are learned by a training process, which is described further below.

[0123] In addition, the neural fusion layer calculates a new feature center point ("joint feature center point") for each cell. For example, the new feature center point of a cell may be calculated as the arithmetic mean of the feature center points of the cell in the two encoded maps input to the neural fusion layer. As an alternative, one of the feature center points of one of the modalities may be selected as the new feature center point.

[0124] The result of processing by each neural fusion layer of the pipeline is a respective encoded fused global map, which includes a feature center point and an encoded feature vector for each cell of the 3D grid (where the cell includes at least one point). Thus, the encoded fused global / local map of FIG. 3 corresponds to the encoded global / local map of FIG. 2, but with less ambiguity due to the fusion of different modalities. The encoded fused global map and the encoded fused local map are input to the matching descriptor layer. In other words, in this exemplary embodiment, the output of the neural fusion layer can be considered as an encoded input map to the matching descriptor layer. The processing of further layers is the same as that described with respect to FIG. 2.

[0125] In some exemplary embodiments, not only two modalities but three or more modalities may be fused.

[0126] Hierarchical Model According to Some Example Embodiments Figure 4 shows an example embodiment including modality fusion (as described with respect to Figure 3) and a hierarchical setup with a 3D grid of increasing cell size. Although Figure 4 includes fusion of different modalities (photographic images and RF measurements), in some example embodiments, point clouds of only one modality may be used (as in the example embodiment of Figure 2). Although Figure 4 shows an example of three hierarchies, the number of hierarchies may be two, four or more.

[0127] In each of the tiers, the process is the same as described with respect to Fig. 3. For simplicity, the respective maps shown in Fig. 3 are not shown in Fig. 4. The 3D grids become coarser from tier 1 to tier 3, i.e., the cells of 3D Grid 1 (i.e., tier 1) are smaller than the cells of 3D Grid 2 (i.e., tier 2), which are smaller than the cells of 3D Grid 3 (i.e., tier 3). In the exemplary embodiment of Fig. 4, for the global map, Grid 3 in tier 3 has four cells (Cell 1, Cell 2, Cell 3, Cell 4), while Grid 2 in tier 2 has 16 cells, with each of the four cells of Grid 2 corresponding to one of the cells of Grid 3. Correspondingly, Grid 1 in tier 1 has 64 cells, with each of the four cells of Grid 1 corresponding to one of the cells of Grid 2. For the local map, each of the two cells A, B of Grid 3 of Tier 3 corresponds to each of the four cells of Grid 2 of Tier 2, and each of the eight cells of Grid 2 of Tier 2 corresponds to each of the four cells of Grid 1 of Tier 1. In other exemplary embodiments, the correspondence ratio between the numbers of corresponding cells of different tiers (1:4 in FIG. 4 ) may differ from this value, such as, but not limited to, 1:2, 1:3, 1:5, 1:8, or 1:16.

[0128] The global map pipeline and the local map pipeline may process each independently of the other map pipelines. For each map, the processing of the pipeline (RF map encoder layer, structure encoder layer, neural fusion layer) is performed in order from the lowest layer (finest 3D grid) to the highest layer (coarsest 3D grid). That is, the input to the RF map encoder layer of layer 2 is the output of layer 1 (encoded global / local RF map), the input to the structure encoder layer of layer 2 is the output of layer 1 (encoded global / local map), and so on to higher layers. Furthermore, in some exemplary embodiments, in one or both of the pipelines, the output of the neural fusion layer of the lower layer may be additionally input to the RF map encoder layer and the structure encoder layer of the higher layer. In FIG. 4, this option is indicated by dashed lines.

[0129] In some exemplary embodiments, information about point correspondences in higher hierarchical layers is input to the matching descriptor layer in the lower hierarchical layers. Based on this information, the matching descriptor layer in the lower hierarchical layers can exclude those cells from the calculation of mutual influence (e.g., self-attention mechanism and / or mutual attention mechanism). Thus, although the computational effort can be saved, the calculated matching descriptor may miss important information that was present in the feature vectors of the filtered points.

[0130] Alternatively, every feature point can be used to compute a matching descriptor, but the output feature points are filtered out based on a coarser level of point correspondences. In this case, the matching descriptor may have higher quality, and some computational gain can still be obtained from the optimal matching layer, since the soft allocation matrix is ​​computed among fewer matching descriptors.

[0131] For example, 3D Grid 3 in Layer 3 includes four coarse cells. Layer 3 identifies that two coarse cells in the global map are similar to two coarse cells in the local map, and the other coarse cell in the global map is not similar to any of the coarse cells in the local map, and vice versa. Each of the coarse cells in 3D Grid 3 corresponds to four finer cells in 3D Grid 2 in Layer 2, such that 3D Grid 2 has a total of 16 finer cells. With the knowledge from Layer 3, Layer 2 must look for similarities between the eight finer cells in the global map and the eight finer cells in the local map (64 calculations). Without input from Layer 3, Layer 2 must look for similarities between the 16 finer cells in the global map and the 16 finer cells in the local map, resulting in 256 calculations.

[0132] Correspondingly, the point correspondences obtained in layer 2 are input to a matching descriptor layer in layer 1, which has finer cells. This can also save computational effort in layer 1.

[0133] In some exemplary embodiments, a soft allocation matrix of a single hierarchical level (typically the lowest hierarchical level (hierarchical level 1)) is used by the transform layer to extract each coordinate pair and estimate a transformation based on the extracted coordinate pair. In other exemplary embodiments, soft allocation matrices of multiple hierarchical levels (typically including the lowest hierarchical level (hierarchical level 1)) are used by the transform layer to extract each coordinate pair and estimate a transformation based on the extracted coordinate pair. Each coordinate pair may have a different weight in estimating the transformation. The weight may depend, for example, on the coarseness of the grid (the coarser the grid, the lower the weight) and / or the respective values ​​of the soft allocation matrix (the higher the value, the higher the weight).

[0134] In some exemplary embodiments, one or more of the layers may not include a matching descriptor layer and an optimal matching layer, in which case only the processing in the pipeline is performed.

[0135] 5 shows a cut-out of the functional structure of the exemplary embodiment of FIG. 4, including only layers 1 and 2. The output layer (including the transformation layer) is omitted. In FIG. 5, the terms used in the claims are shown. These terms apply correspondingly to the claims related to the exemplary embodiments of FIG. 2 and FIG. 3.

[0136] [Training stage] Each of the exemplary embodiments of Figures 2-4 must be trained before estimation (= query phase). For training, a global map is compared to a local map for which the ground truth is known. For example, such a local map may be generated by a subset of the images from which the global map was generated, using a mechanism such as SLAM to generate a respective point cloud of the local map. However, the local maps of the training dataset are not limited to point clouds obtained from a subset of images.

[0137] In the training phase, the point clouds of the global and local maps (potentially following different modalities, depending on the embodiment) and the 3D grid are input to the system. Two loss terms can be calculated: The first loss function, L_corr, tries to maximize the point correlation of corresponding point pairs. A second loss function L_geo between the computed transformation and the known transformation.

[0138] For a loss function L, these loss terms can be weighted and combined (e.g., L=α*L_corr m +β*L_geo nwhere α, β, m, n are predefined constant values, e.g. m=n=1; other combinations of L_corr and L_geo are feasible). Through iterations, the parameters of the different layers (structural encoder layer, neural fusion layer (if present), matching descriptor layer) are trained such that the predicted transformation matches the actual transformation (ground truth) sufficiently well. Typically, a number of epochs of iterations are performed until both the training loss function (the error of the model based on the training set used to initially train the model) and the validation loss function are reduced and are smaller than their respective thresholds. Optionally, the training may be stopped after a predefined number of iterations have been performed.

[0139] In the training phase of the exemplary embodiment of Fig. 4, the point correspondences in the higher layer may not be considered in the processing by the matching descriptor layer and the optimal matching layer in the lower layer, i.e., the computational gain (saved computational effort) may be obtained only in the estimation (query) phase.

[0140] Aspects of Some Exemplary Embodiments First, some aspects of some exemplary embodiments will be described from different perspectives below. It should be noted that these aspects are not limiting.

[0141] Some example embodiments may be used to solve problems such as loop closure, relocalization, and colocalization in a SLAM pipeline. Some example embodiments may have one or more of the following highlights: The relocalization, colocalization and loop closure problems are solved by using a sequence of images (instead of a single image) to create a 3D point cloud representation of the environment using 3D reconstruction algorithms. Alternatively, a lidar point cloud can be utilized for example. To increase robustness and make the use of large maps feasible, six DoF poses, called RF fingerprints, are registered with the RSSI values ​​of different APs. Based on the structural information of the environment and the registered RF fingerprints, highly discriminative spatial matching descriptors (previously referred to as joint feature vectors) of different granularities are created in a data-driven manner using fully distinguishable and trainable models. These new descriptors can help solve the correspondence problem between point clouds. By obtaining the exact correspondences, a rigid transformation can be computed that aligns the point clouds. To create the spatial matching descriptor, a fusion layer is used, which creates a fused feature vector (previously called joint feature vector) by an attention graph neural network architecture, e.g., using self-attention and mutual attention to exchange information between the structural and RF codings. Fading and geometric embedding are defined for the attention stage which implicitly encodes the RF propagation characteristics locally, resulting in better RF signal interpolation and extrapolation. In order to achieve high accuracy of rigid transformation estimation even on the coarsest layers, in the encoding layer, the 3D positions of the feature vectors are similarly regressed in a data-driven, differentiable, and learnable manner.

[0142] Some example embodiments accumulate localization information from a sequence of images instead of using a single query image. By capturing the 3D structure of the environment and determining the camera pose, the RF fingerprint can be registered to the 6DoF camera pose. From these joint measurements, a spatial matching descriptor can be created by training a deep learning model that fuses spatial information from the point cloud and the RF fingerprint. Such a spatial matching descriptor can be used to efficiently solve the map-to-map registration problem. If registration fails due to non-overlapping maps or there is not enough information to disambiguate the match, the local map can be expanded by acquiring more images until the match is successful.

[0143] Some exemplary embodiments significantly reduce the computational effort by creating discriminative hierarchical spatial matching descriptors that reduce the search space by first registering matching descriptors at the coarsest level, and continuing the registration process only in areas that belong to successfully registered coarse spatial matching descriptors.

[0144] To create robust and discriminative spatial matching descriptors for feature points, the system of some exemplary embodiments is trainable end-to-end via backpropagation, which means that all equations of all building blocks are distinguishable. Based on the positions of these matching descriptors for different point clouds, a rigid body transformation for point cloud alignment can be estimated. The deep learning model receives as input, for example, a (already optimized) global map including the registered RF fingerprint map, a 3D point cloud describing the structure of the environment, and a bundle-adjusted local map with actual RF fingerprint measurements. These maps are encoded through a series of layers: a structure and RF map encoder layer, a fusion layer, a spatial matching descriptor layer, an optimal matching layer, and a transformation layer.

[0145] In the encoder layer, the input map data passes through a series of convolution, normalization, activation and data compression layers. In some exemplary embodiments, point convolution may be used to encode the point cloud features and the RF map features, although other types of convolution may be used in different embodiments. A mathematical definition of point convolution is given further below. For normalization, several approaches can be applied. For example, data compression may be achieved through a grid division mechanism. A series of grids are defined with increasing grid cell sizes. In the data compression layer, the space is divided by the selected grid and the location of the output feature vector is defined based on the data points that belong to each grid cell. These points are called feature center points. In some embodiments, the output location (feature center point) for each grid cell may be calculated as the arithmetic mean of the grid cell data point locations. The location of the feature vector is very important since the rigid transformation (which aligns the two point clouds) is estimated based on the location of the feature vector. Therefore, several convolution layers are used that regress the location of the output feature vector. In the case of point convolution, once the output position (feature center point) is determined, the supporting data points are selected based on some neighborhood criteria (optional). Typically, a radius is defined around the output position or the kth point closest to the output position is selected. Based on the feature vectors of the supporting points, a feature vector of the output feature point is created. It is important to note that several point convolution iterations can occur between the two compression layers. In this case, the position of the output feature point is the same as the position of the input feature point.

[0146] After every compression layer, the input / output data point connections may be stored. During estimation, the points at the highest / coarsest level are registered first. After the coarse alignment, an initial rigid transformation is estimated based on the 3D positions of the feature points, and is iteratively refined by further extending the aligned points and computing more refined transformations. In this hierarchical registration approach, there is no need to load the entire set of feature points in the global localization map. The point registration drives the loading of the relevant parts of the global map. Moreover, the computational complexity is significantly reduced, since only the relevant parts of the global map are fed to the matching layer of the deep learning model.

[0147] The hierarchical approach is particularly effective when the spatial matching descriptors of feature points are highly discriminative. This can be achieved by converting the RF fingerprints into RF spatial encodings, converting the 3D point clouds into structural encodings, and then fusing these encodings through distinguishable layers. Although 3D structures can be repetitive in industrial environments, the fingerprint map can reduce location ambiguity. Furthermore, by encoding the structural representation with RF signal measurements, RF signal propagation can be better described compared to existing models (Rayleigh fading, Rician fading, etc.) since fading characteristics are implicitly encoded in the network. If the pose of the access point (AP) is known, a geometric encoding of the pose of the RF measurements along with the pose of the AP is also possible.

[0148] The fusion layer may be built on an attention graph neural network architecture, including a self-attention stage and a mutual attention stage. In the self-attention stage, feature points of the same type share information between each other through the attention mechanism, and in the mutual attention stage, the structural coding shares information with the RF spatial coding. In the self-attention stage, an embedding term is introduced that encodes channel characteristics in the form of a path loss exponent, the relative distance and orientation of the current measurement from its corresponding AP. In the mutual attention stage, a distance-based embedding is defined because the positions of the RF spatial coding and the structural coding may be different. The output of the fusion layer is a feature vector with 3D positions that encodes the structural and RF signal information. The exact mathematical formulation is presented further below.

[0149] The fused spatial feature vector is fed into a spatial matching descriptor layer. This is the first layer where the fused spatial feature vectors of the two point clouds to be matched can exchange information. An attention graph neural network architecture is used to drive the information exchange. The output of this layer is called the spatial matching descriptor. Feature points that describe the same region in the global and local maps are expected to have similar descriptors.

[0150] In the optimal mashing layer, a pairwise score matrix is ​​computed as the similarity of the spatial matching descriptors. The pairwise scores can be computed as the dot product or Gaussian correlation between the feature vectors. The point matching problem can then be formulated as an optimal transport problem that maximizes the soft allocation matrix by a differentiable sinkhorn algorithm. Local point correspondences are extracted by mutual top-k selection on the values ​​of the soft allocation matrix.

[0151] In the transform layer, based on point correspondences, the rigid transformation estimation is formulated as a least-squares problem that can be solved in closed form using differentiable weighted SVD. This layer has no learnable parameters, but is differentiable, allowing us to define a loss function between the estimated transformation and the ground truth transformation.

[0152] For soft assignment matrices, a different loss function (loss term) may be used, where the basis assignment matrices can be derived from the ground truth alignment transformation between the point clouds (which is known during the training phase).

[0153] The deep learning model created based on the above layers can solve the problems of relocalization, loop closure, and colocalization. For relocalization, a global map and a local map are given as inputs for the deep learning model. Since the global map can be the result of a SLAM pipeline, the hierarchical fusion features can be pre-computed, so that at the time of estimation, only the local map can be encoded and the structural and RF features can be fused. For loop closure detection and colocalization, the two local maps are the inputs for the deep learning model. In this case, feature points are always extracted from the local map, since it is continuously changing, and the changes can be propagated to the highest level (global map) as well.

[0154] Image-based localization problems are known to be difficult to solve because lighting conditions encountered during the day or seasonal changes in the environment introduce different artifacts. Creating visual descriptors and models that can generalize to all conditions is particularly challenging. Image retrieval approaches to solving localization problems are assumed not to be robust due to the aforementioned issues. By creating a local map for the structural representation and registering the RF measurements with the camera pose, a better representation can be constructed for localization purposes. After the global map is constructed, different deep learning models can be trained to register the features of the local maps created in different conditions. Nevertheless, it is also possible to train one model that generalizes to all conditions by creating a set of local maps of the same area at different times of the day and / or season.

[0155] Location estimation, image retrieval algorithms, and data-driven training models can be heavily affected by camera lens distortion. When a query image is created with a different camera than the keyframe image, only suboptimal accuracy is achieved. Image distortion removal techniques help to alleviate this problem, but the results may still not be optimal. For data-driven algorithms, the data distribution is defined by the images in the training set. To create a robust algorithm, a good method is to collect training images from a wide set of cameras with different distortion effects. This is useful because global maps are usually created with cameras that have similar distortion parameters, but during location estimation, a wide variety of user equipment must be supported. Some exemplary embodiments are camera-independent, since the camera image sequence is converted into a point cloud.

[0156] DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS An exemplary embodiment corresponding to Fig. 4 will be described in detail below. Note that this embodiment is not limiting.

[0157] Map definition The map produced by the SLAM pipeline consists of (i.) 3D map points

number

number

number

number

[0158] Detailed RF measurements are captured by an agent that contains an RF module and a camera. A SLAM map is constructed based on the image sequence captured by the camera and registers the RF measurements to the 6DoF pose of the camera. The radio fingerprint includes the average received signal strength (RSS) from surrounding WiFi access points (APs), base stations, and Bluetooth devices. For a mapped area with K access points, the radio map

number

number

number

number

number

number

number

[0159] Problem definition The goal is to align a local SLAM map within a global SLAM map, or to align two different local SLAM maps, but for simplicity, two point clouds may be considered.

number

number

number

number

number

number

number

[0160] Neural Network Building Blocks Map Encoder Layer Encoding point clouds is difficult because the data is not well structured like image data. Several encoding techniques exist: (i.) projection networks, where points are projected onto regular 2D and 3D grid structures followed by regular 2D or 3D convolution operators, (ii.) graph convolution networks, (iii.) point-wise multi-layer perceptron networks, or (iv.) point convolution networks. The point cloud encoder layer in different embodiments may implement different point cloud encoding strategies. The main goal is to create a representation that encodes the structure of the environment and the measured RF signals in a hierarchical and transformation-invariant manner.

[0161] In this section, the details of the encoder layer are formalized. The encoder layer is applied to the structure map and the RF map in parallel. The encoder layer is defined by a set of convolution, normalization, activation, and data compression operations.

[0162] Convolution: Since point cloud data is unstructured, the convolution operator is redefined. In this embodiment, the following approach to encoding point cloud features and RF map features is adopted, but other embodiments may use different types of convolution.

[0163] point

number

number

number

number

number

[0164] N x is the neighborhood of the center point, which can be defined in several ways. The most common approach is to take the k nearest neighbors of the center point x, or to define a sphere with radius r around the center point, so that the neighborhood set is

number

number

number

number

number

number

number

number

number

[0165] In an embodiment where the encoder layer is formulated by successive point convolution operations, the output position x is the same for all input point positions, so that the number of convolutions evaluated is equal to N.

[0166] Regularization: Regularization layers play an important role in increasing the convergence of training. Several different techniques can be applied.

[0167] Data Compression: For large point clouds, it is impractical to create a model that uses only point convolutions, since the number of convolution evaluations is equal to the number of input points. By applying data compression, the number of points is reduced and becomes better organized structurally. Data compression is achieved through a grid division mechanism. A set of 3D grids is defined with increasing cell sizes. When data compression is performed, the goal is to define: (i.) structural and RF descriptors encoded as feature vectors per grid cell, and (ii.) regress feature (center) point coordinates on the structural and RF descriptors used in computing the rigid body transformation T. At lower hierarchies, the size of the grid cells decreases, and as the coarsest resolution is approached, the size of the grid cells increases.

[0168] A point compression operation for one grid configuration is formulated as follows: (i.) define a 3D grid with a given grid cell size; (ii.) group input points based on grid cells; (iii.) regress the locations of output feature center points and any additional parameters of the selected convolution type; (iv.) define support points for all feature center points (the set of support points is the collection of neighbors of the feature center points, typically defined by a radius of the convolution or based on a neighbor selection algorithm, and further, the radius of the convolution may be equal to, greater than, or smaller than the size of the grid cell); and (vi.) create a structural or RF descriptor for each grid cell based on the points within the cell by performing several groups of convolution, normalization, and activation operations.

[0169] For further data compression, the grid cell size is increased and point cloud encoding continues as above. Structural descriptors and centroid locations are created in a data-driven manner to produce good feature vectors for point registration and pose estimation.

[0170] By using the kernel point convolution presented above, the desired structural and RF descriptors and output feature center points can be created using a two-stage kernel point convolution. In the first stage, grid cell C j Center point of

number

number

number

number

[0171] The encoding of the RF map (i.e., RF measurements are recorded at 3D positions) is done similarly. The positions of the fingerprint measurements may be structurally sparser than the reconstructed point cloud (note also that RF measurement 3D positions are not the same as structural 3D points), so it may be beneficial if the hierarchical encoding occurs in fewer stages, but this is a design choice that may differ in different exemplary embodiments. It is recommended to use the same grid configuration for the layer where the RF features are fused with the structural features. The feature center points of the RF map and the structural map are usually disjoint locations. Therefore, this must be handled in the fusion process. Furthermore, when applying a data compression layer to the RF map, instead of regressing the feature center points, the measurement points are selected as feature center points. An obvious choice could be the measurement points with the most valid RSS measurements in the fingerprint feature vector, or the measurement points that are closest to the arithmetic mean of the measurement positions in the grid cell. The motivation for selecting the measurement points as feature center points is the availability of the RF fingerprint feature vectors used to create the RF embedding in the attention mechanism. Estimation of the RF fingerprint vector at new locations introduces inaccuracies. Center point selected measurement point

number

[0172] The hierarchical features are created by iteratively applying data compression. After defining the feature center points for the new grid configuration, point-to-node grouping is performed. Point-to-node grouping can be implemented based on several heuristics. All points

number

number

number

number

[0173] The point registration process starts at the coarsest level, and the rigid pose transformation is refined in an iterative manner by expanding registered nodes and recomputing the refined pose. More formally, we extract feature center points from the point cloud P.

number

number

number

number

[0174] ·Fusion layer The RF and structural feature fusion architecture can be designed in many ways. In one embodiment, features at a certain level with the same grid cell size are concatenated, followed by several convolutional layers.

[0175] In this exemplary embodiment, a graph neural network architecture with an attention mechanism is used. The attention mechanism is presumed to be able to better capture the propagation characteristics of RF signals in a large environment compared to a group of convolutional layers, since information sharing occurs among all points where valid RSS measurements have been made. By fusing sparse RF coding with structural coding, better prediction of RF signal propagation and a more unique description of any 3D location in the world is expected.

[0176] A series of self-attention and mutual attention modules are employed to share structural information among feature centers. By aggregating this information, a feature vector is created that is expected to have better global context information, which will facilitate the registration problem.

[0177] Structural Feature Self-Attention: In the following, we formalize a single-headed self-attention mechanism for structural feature vectors (in another exemplary embodiment, a multi-headed attention mechanism can be used).

[0178] Structural feature vector of layer l

number

number

number

number

number

number

number

number

number

number

number

number

[0179] The relative distance embedding is done by applying sinusoidal encoding to the coordinates

number

number

number

number

number

number

number

[0180] RF self-attention: An RF embedding mechanism is used to generate a transformation invariant RF propagation embedding.

number

number

number

number

number

number

[0181] Feature Vector

number

number

number

number

[0182] If the AP pose is registered, the RF embedding consists of three terms: Distance (relative position) encoding

number

number

number

[0183] RF embedding term

number

number

number

[0184] Measurement point

number

number

number

[0185] The sinusoidal encoding of the relative orientation between the i-th measurement point and the j-th AP is encoded as follows:

number

number

[0186] In fading embedding, the goal is to implicitly encode the path loss exponent. The log-distance path loss model is formally defined as follows:

number

number

number

number

number

number

number

number

number

number

[0187] The sinusoidal coding for fading embedding is formulated as follows:

number

[0188] Mutual attention of RF and structural features: Output RF feature vectors of the self-attention encoding stage

number

number

[0189] The update of the RF feature vector in the first step is formulated as follows:

number

number

[0190] The update of the structural feature vector in the second step is formulated as follows.

number

number

[0191] The mutual attention stage is the step where the structural and RF information are shared and fused. After several iterations of the self-attention and mutual attention steps, the final fused feature vector

number

[0192] In other exemplary embodiments, the RF center point can be similarly regressed and used as the output of the fusion layer.

[0193] For every iteration in which the attention mechanism is computed, a new trainable projection matrix is ​​used.

[0194] Matching Descriptor Layer The matching descriptor layer is responsible for exchanging information between the fused feature vectors of the two different point clouds P and Q and creating matching descriptors that are used to calculate similarity scores for different regions in the point clouds.

[0195] The fused feature vector of point cloud P at layer l

number

number

number

number

number

number

[0196] In the self-attention step, the feature vector is updated as follows:

number

number

number

number

number

number

[0197] In the mutual attention stage, the attention mechanism is applied as follows.

number

[0198] In the mutual attention stage, weighting factors are calculated based on the mutual attention scores.

number

[0199] Mutual Attention Features

number

[0200] Finally, the attention features are transformed into a projection matrix

number

number

number

number

number

[0201] In other exemplary embodiments, a multi-head architecture may be feasible, in which case the number of projection matrices increases.

[0202] For every iteration, a new trainable projection matrix is ​​used when the attention mechanism is computed.

[0203] Optimal matching layer In the optimal matching layer, the matching descriptor is

number

number

[0204] To be able to mark certain features as not being able to be registered, we use the matrix

number

number

number

number

number

number

[0205] Soft Allocation Matrix A l can be computed in a differentiable way by the Sinkhorn algorithm. The local point correspondences are calculated using the soft assignment matrix A l It is extracted by mutual top-k selection on the values ​​of

[0206] Conversion layer By defining the correspondences between matching feature points, the rigid transformation estimation is formulated as a least-squares problem that can be solved in closed form using differentiable weighted SVD. This layer has no learnable parameters, but because it is differentiable, a loss function can be defined between the estimated transformation and the ground truth transformation.

[0207] The accuracy of the estimated pose can be further improved by expanding the feature points in layer l and computing matching between the expanded features in lower layers. The accuracy is improved because the number of feature points is large and the quantization effect of the center selection process introduces less error. Registration in lower layers is only performed between points whose parent nodes have been previously registered. With this mechanism, fast registration can be achieved even for large maps.

[0208] Loss function Since all layers presented previously are perfectly differentiable, training a deep learning model consists of predicting the rigid transformations at layer l.

number

number

[0209] A possible geometric loss function can be formulated for layer l as follows:

number

[0210] Soft Allocation Matrix A l For each point correspondence, the log-likelihood loss is defined. During the training phase, the ground truth correspondences of layer l are

number

number

number

[0211] The total loss can be formulated as follows:

number

number

[0212] [General embodiment] FIG. 6 shows an apparatus according to an exemplary embodiment. The apparatus may be a location estimation device or an element thereof. FIG. 7 shows a method according to an exemplary embodiment. The apparatus according to FIG. 6 may execute the method of FIG. 7, but is not limited to this method. The method of FIG. 7 may be executed by the apparatus of FIG. 6, but is not limited to being executed by this apparatus.

[0213] The apparatus comprises first to fourth input means 110, 130, 140, 160, first and second encoding means 120, 150, first and second adapting means 170, 180, first, second and third calculating means 190, 200, 230, checking means 210, and extracting means 220. The first to fourth input means 110, 130, 140, 160, first and second encoding means 120, 150, first and second adapting means 170, 180, first, second and third calculating means 190, 200, 230, checking means 210, and extracting means 220 may be the first to fourth input means, first and second encoding means, first and second adapting means, first, second and third calculating means, checking means, and extracting means, respectively. The first to fourth input means 110, 130, 140, 160, the first and second encoding means 120, 150, the first and second adapting means 170, 180, the first, second and third calculating means 190, 200, 230, the checking means 210 and the extracting means 220 may be first to fourth input parts, first and second encoders, first and second adapters, first, second and third calculators, checkers and extractors, respectively. The first to fourth input means 110, 130, 140, 160, the first and second encoding means 120, 150, the first and second adaptation means 170, 180, the first, second and third calculation means 190, 200, 230, the check means 200 and the extraction means 210 may be first to fourth input processors, first and second encoding processors, first and second adaptation processors, first, second and third calculation processors, check processors and extraction processors, respectively.

[0214] The first input means 110, the first encoding means 120, and the second input means 130 constitute a first pipeline. The third input means 140, the second encoding means 150, and the fourth input means 160 constitute a second pipeline. The work of the first pipeline may be performed independently of the work of the second pipeline, and the work of the second pipeline may be performed independently of the work of the first pipeline. The work of the first pipeline may be performed in any time sequence relative to the work of the second pipeline. The work of the first pipeline may be performed fully or partially in parallel with the work of the second pipeline.

[0215] A first input means 110 inputs a first 3D point cloud of a first modality and a first grid to a first encoder layer (S110). The first grid includes one or more first cells. Coordinates of points of the first 3D point cloud are expressed in a first coordinate system.

[0216] The first encoding means 120 encodes the first 3D point cloud of the first modality into a first encoded map of the first modality (S120) by a first encoder layer. The first encoded map of the first modality includes a respective first feature center point and a respective first feature vector for each of the first cells of the first grid. The coordinates of the first feature center points are represented in a first coordinate system.

[0217] The second input means 130 inputs a first coded input map to the first matching descriptor layer of the first hierarchy (S130). The first coded input map is based on the first coded map of the first modality. The first coded input map comprises, for each of the first cells, a respective first joint feature center point and a respective first input feature vector. The coordinates of the first joint feature center point are shown in a first coordinate system.

[0218] A third input means 140 inputs (S140) a second 3D point cloud of the first modality and a second grid including one or more second cells to the second encoder layer. Coordinates of points of the second 3D point cloud are represented in a second coordinate system. The second coordinate system may or may not be different from the first coordinate system.

[0219] The second encoding means 150 encodes (S150) the second 3D point cloud of the first modality into a second encoded map of the first modality by a second encoder layer. The second encoded map of the first modality comprises, for each of the second cells of the second grid, a respective second feature center point and a respective second feature vector. The coordinates of the second feature center points are represented in a second coordinate system.

[0220] A fourth input means 160 inputs a second encoded input map to the first matching descriptor layer (S160). The second encoded input map is based on the second encoded map of the first modality. The second encoded input map comprises, for each of the second cells, a respective second joint feature center point and a respective second input feature vector. The coordinates of the second joint feature center point are shown in a second coordinate system.

[0221] After the two pipeline activities S110 to S130 and S140 to S160 have been performed, activities S170 to S230 may follow.

[0222] The first adaptation means 170 adapts each of the first input feature vectors based on the first input feature vector and the second input feature vector by the first matching descriptor layer to obtain a first joint map (S170). Optionally, the first matching descriptor layer may adapt each of the first input feature vectors based on the first input feature vectors with their first feature center points and / or based on the second input feature vectors with their second feature center points. The first joint map includes a respective first joint feature vector for each of the first joint feature center points.

[0223] The second adaptation means 180 adapts each of the second input feature vectors based on the first input feature vector and the second input feature vector by the first matching descriptor layer to obtain a second joint map (S180). Optionally, the first matching descriptor layer may adapt each of the second input feature vectors based on the first input feature vectors with their first feature center points and / or based on the second input feature vectors with their second feature center points. The second joint map includes a respective second joint feature vector for each of the second joint feature center points.

[0224] The first calculation means 190 calculates, for each of the first cells and each of the second cells, a similarity between the first joint feature vector of each of the first cells and the second joint feature vector of each of the second cells by a first optimal matching layer of the first hierarchy (S190).

[0225] The second calculation means 200 calculates a first point correlation between each of the first cells and each of the second cells by the first optimal matching layer (S200). The first point correlation is calculated based on the similarity between the first joint feature vector of each of the first cells and the second joint feature vector of the second cell, and based on the similarity between the second joint feature vector of each of the second cells and the first joint feature vector of the first cell.

[0226] The checking means 210 checks whether at least one of one or more first correlation conditions is satisfied for each of the first cells and each of the second cells (S210). The one or more correlation conditions include: a first point correlation between each first cell and each second cell is greater than a first correlation threshold for the first tier; or The first point correlation between each first cell and each second cell is among the largest k values ​​of first point correlations, where k is a fixed value.

[0227] When the extraction means 220 checks that at least one of the one or more first correlation conditions for each of the first cells and each of the second cells is satisfied (S210=yes), it extracts, for each of the first cells and each of the second cells, the coordinates of the first joint feature center point of each of the first cells in the first coordinate system and the coordinates of the second joint feature center point of each of the second cells in the second coordinate system, and obtains each pair of extracted coordinates (S220). When the extraction means 220 checks that at least one of the one or more first correlation conditions for each of the first cells and each of the second cells is not satisfied (S210=no), it may not extract, for each of the first cells and each of the second cells, the coordinates of the first joint feature center point of each of the first cells in the first coordinate system and the coordinates of the second joint feature center point of each of the second cells in the second coordinate system.

[0228] Based on the coordinate pairs extracted in S220, the second calculation means 230 calculates an estimated transformation between the first coordinate system and the second coordinate system (S230).

[0229] 8 shows an apparatus according to an exemplary embodiment. The apparatus comprises at least one processor 810 and at least one memory 820 storing instructions that, when executed by the at least one processor 810, cause the apparatus to perform at least the method according to FIG.

[0230] Unless otherwise stated or clear from the context, a statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described herein may be based on different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described herein may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described herein may be deployed in a cloud.

[0231] Thus, based on the above description, it will be apparent that examples provide, for example, a neural network or components thereof, apparatus embodying the same, methods for controlling and / or operating the same, computer programs for controlling and / or operating the same, and media carrying such computer programs and forming computer program products.

[0232] Implementations of any of the blocks, apparatus, systems, techniques, or methods described above include, by way of non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or any combination thereof. Each of the entities described herein may be embodied within a cloud.

[0233] It should be understood that the foregoing are what are presently considered to be preferred exemplary embodiments. It should be noted, however, that the description of the preferred exemplary embodiments is given by way of example only, and that various modifications may be made without departing from the scope of the present invention as defined by the appended claims.

[0234] The terms "first X" and "second X" include the option that the "first X" is the same as the "second X" and the option that the "first X" is different from the "second X" unless otherwise specified. As used herein, "at least one of" is a similar phrase to "a list of two or more elements" and "at least one of ", where a list of two or more elements is joined by "and" or "or", it means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

Claims

1. 1. An apparatus comprising one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to: inputting a first 3D point cloud of a first modality and a first grid including one or more first cells into a first encoder layer, where coordinates of points of the first 3D point cloud are expressed in a first coordinate system; encoding, by the first encoder layer, the first 3D point cloud of the first modality into a first encoded map of the first modality, the first encoded map of the first modality including, for each of the first cells of the first grid, a respective first feature center point and a respective first feature vector, the coordinates of the first feature center points being expressed in the first coordinate system; inputting a first coded input map into a first matching descriptor layer of a first hierarchy, the first coded input map being based on the first coded map of the first modality and comprising, for each of the first cells, a respective first joint feature center point and a respective first input feature vector, the coordinates of the first joint feature center points being represented in the first coordinate system; inputting a second 3D point cloud of the first modality and a second grid including one or more second cells into a second encoder layer, the coordinates of points of the second 3D point cloud being expressed in a second coordinate system; encoding, by the second encoder layer, the second 3D point cloud of the first modality into a second encoded map of the first modality, the second encoded map of the first modality including, for each of the second cells of the second grid, a respective second feature center point and a respective second feature vector, the coordinates of the second feature center points being expressed in the second coordinate system; inputting a second coded input map into the first matching descriptor layer, the second coded input map being based on the second coded map of the first modality and comprising, for each of the second cells, a respective second joint feature center point and a respective second input feature vector, the coordinates of the second joint feature center points being represented in the second coordinate system; adapting, by the first matching descriptor layer, each of the first input feature vectors based on the first input feature vectors, optionally their first feature center points, and the second input feature vectors, optionally their second feature center points, to obtain a first joint map, the first joint map including, for each of the first joint feature center points, a respective first joint feature vector; adapting, by the first matching descriptor layer, each of the second input feature vectors based on the first input feature vectors, optionally their first feature center points, and the second input feature vectors, optionally their second feature center points, to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; calculating, for each of the first cells and each of the second cells, a similarity between the first joint feature vector of the respective first cell and the second joint feature vector of the respective second cell by a first optimal matching layer; calculating, by the first optimal matching layer, for each of the first cells and each of the second cells, a first point correlation between the respective first cell and the respective second cell based on the similarity between the first joint feature vector of the respective first cell and the number of second joint feature vectors of the number of second cells, and based on the similarity between the second joint feature vector of the respective second cell and the number of first joint feature vectors of the number of first cells; checking, for each of the first cells and each of the second cells, whether at least one of one or more first correlation conditions is satisfied for the respective first cell and the respective second cell, the one or more first correlation conditions comprising: the first point correlation between each of the first cells and each of the second cells is greater than a first correlation threshold for the first tier; or the first point correlation between each of the first cells and each of the second cells is among a maximum k value of the first point correlation, where k is a fixed value; extracting, for each of the first cells and each of the second cells, the coordinates of the first joint feature center point of the respective first cell in the first coordinate system and the coordinates of the second joint feature center point of the respective second cell in the second coordinate system, and obtaining, for each of the first cells and each of the second cells, a respective pair of extracted coordinates of the first hierarchy in response to checking whether at least one of the one or more first correlation conditions is satisfied; calculating an estimated transformation between the first coordinate system and the second coordinate system based on the extracted coordinate pairs of the first layer; An apparatus for executing the above-mentioned procedure.

2. the first encoder layer, the second encoder layer, and the first matching descriptor layer of the first hierarchy are configured with respective first parameters, and the instructions, when executed by the one or more processors, further cause the apparatus to: training the device to determine the first parameters such that a deviation between the estimated transformation and a known transformation between the first coordinate system and the second coordinate system is less than a transformation estimation threshold and such that the first point correlation is increased; 2. The apparatus according to claim 1, further comprising:

3. the first modality includes one of a photographic image, a LIDAR image, and an ultrasound image; 3. Apparatus according to claim 1 or 2.

4. 3. The apparatus of claim 1 or 2, wherein the first encoder layer is the same as the second encoder layer.

5. The instructions, when executed by the one or more processors, further cause the apparatus to: generating the first 3D point cloud based on a sequence of first images of the first modality, wherein a respective position in the first coordinate system is associated with each of the first images; or generating the second 3D point cloud based on a sequence of second images of the first modality, wherein a respective position in the second coordinate system is associated with each of the second images; 3. The apparatus according to claim 1, further comprising:

6. 2. The apparatus of claim 1 , wherein the first coded input map is the first coded map, the first joint feature center points are the first feature center points, the second coded input map is the second coded map, and the second joint feature center points are the second feature center points.

7. The instructions, when executed by the one or more processors, further cause the apparatus to: inputting a third 3D point cloud of a second modality and the first grid into a third encoder layer, the coordinates of points of the third 3D point cloud being expressed in the first coordinate system, the second modality being different from the first modality; encoding, by the third encoder layer, the third 3D point cloud of the second modality into a third encoded map of the second modality, the third encoded map of the second modality comprising, for each of the first cells of the first grid, a respective third feature center point and a respective third feature vector, the coordinates of the third feature center points being expressed in the first coordinate system; inputting the third encoded map into a first neural fusion layer; inputting the first encoded map into the first neural fusion layer; generating, by the first neural fusion layer, the first coded input map based on the first coded map and the third coded map, wherein for each of the first cells, the respective first joint feature center points are based on the first feature center points of the respective first cell and the third feature center points of the respective first cell, the respective first input feature vectors are based on the first feature vector and the first feature vector, optionally the first feature center points and the third feature vector of the respective first cell adapted by their first feature center points, optionally their third feature center points, and the third feature vector and the first feature vector, optionally the third feature center points and the third feature vector of the respective first cell adapted by their first feature center points, optionally their third feature center points; inputting a fourth 3D point cloud of the second modality and the second grid into a fourth encoder layer, where the coordinates of points of the fourth 3D point cloud are represented in the second coordinate system; encoding, by the fourth encoder layer, the fourth 3D point cloud of the second modality into a fourth encoded map of the second modality, the fourth encoded map of the second modality including, for each of the second cells of the second grid, a respective fourth feature center point and a respective fourth feature vector, the coordinates of the fourth feature center points being expressed in the second coordinate system; inputting the fourth encoded map into a second neural fusion layer; inputting the second encoded map into the second neural fusion layer; generating, by the second neural fusion layer, the second coded input map based on the second coded map and the fourth coded map, wherein for each of the second cells, the respective second joint feature center points are based on the second feature center points of the respective second cell and the fourth feature center points of the respective second cell, the respective second input feature vectors are based on the second feature vector and the second feature vector, optionally the second feature center points and the fourth feature vector of the respective second cell adapted by their second feature center points, optionally their fourth feature center points, and the fourth feature vector and the second feature vector, optionally the fourth feature center points and the fourth feature vector of the respective second cell adapted by their second feature center points, optionally their fourth feature center points; 7. The apparatus of claim 1, 2 or 6, further comprising:

8. the second modality includes at least one of a photographic image, an RF fingerprint, a LIDAR image, or an ultrasound image; 8. The apparatus according to claim 7 .

9. inputting a first 3D point cloud of a first modality and a first grid including one or more first cells into a first encoder layer, where coordinates of points of the first 3D point cloud are expressed in a first coordinate system; encoding, by the first encoder layer, the first 3D point cloud of the first modality into a first encoded map of the first modality, the first encoded map of the first modality including, for each of the first cells of the first grid, a respective first feature center point and a respective first feature vector, the coordinates of the first feature center points being expressed in the first coordinate system; inputting a first coded input map into a first matching descriptor layer of a first hierarchy, the first coded input map being based on the first coded map of the first modality and comprising, for each of the first cells, a respective first joint feature center point and a respective first input feature vector, the coordinates of the first joint feature center points being represented in the first coordinate system; inputting a second 3D point cloud of the first modality and a second grid including one or more second cells into a second encoder layer, the coordinates of points of the second 3D point cloud being expressed in a second coordinate system; encoding, by the second encoder layer, the second 3D point cloud of the first modality into a second encoded map of the first modality, the second encoded map of the first modality including, for each of the second cells of the second grid, a respective second feature center point and a respective second feature vector, the coordinates of the second feature center points being expressed in the second coordinate system; inputting a second coded input map into the first matching descriptor layer, the second coded input map being based on the second coded map of the first modality and comprising, for each of the second cells, a respective second joint feature center point and a respective second input feature vector, the coordinates of the second joint feature center points being represented in the second coordinate system; adapting, by the first matching descriptor layer, each of the first input feature vectors based on the first input feature vectors, optionally their first feature center points, and the second input feature vectors, optionally their second feature center points, to obtain a first joint map, the first joint map including, for each of the first joint feature center points, a respective first joint feature vector; adapting, by the first matching descriptor layer, each of the second input feature vectors based on the first input feature vectors, optionally their first feature center points, and the second input feature vectors, optionally their second feature center points, to obtain a second joint map, the second joint map including, for each of the second joint feature center points, a respective second joint feature vector; calculating, for each of the first cells and each of the second cells, a similarity between the first joint feature vector of the respective first cell and the second joint feature vector of the respective second cell by a first optimal matching layer; calculating, by the first optimal matching layer, for each of the first cells and each of the second cells, a first point correlation between the respective first cell and the respective second cell based on the similarity between the first joint feature vector of the respective first cell and the several second joint feature vectors of the several second cells, and based on the similarity between the second joint feature vector of the respective second cell and the several first joint feature vectors of the several first cells; checking, for each of the first cells and each of the second cells, whether at least one of one or more first correlation conditions is satisfied for the respective first cell and the respective second cell, the one or more first correlation conditions comprising: the first point correlation between each of the first cells and each of the second cells is greater than a first correlation threshold for the first tier; or the first point correlation between each of the first cells and each of the second cells is among a maximum k value of the first point correlation, where k is a fixed value; extracting, for each of the first cells and each of the second cells, the coordinates of the first joint feature center point of the respective first cell in the first coordinate system and the coordinates of the second joint feature center point of the respective second cell in the second coordinate system, and obtaining, for each of the first cells and each of the second cells, a respective pair of extracted coordinates of the first hierarchy in response to checking whether at least one of the one or more first correlation conditions is satisfied; calculating an estimated transformation between the first coordinate system and the second coordinate system based on the extracted coordinate pairs of the first layer; 23. A method comprising:

10. A computer program product comprising a set of instructions configured, when executed on an apparatus, to cause said apparatus to perform the method of claim 9.

Citation Information

Patent Citations

  • Method and device for classifying objects

    JP2021536634A

  • Depth estimation using a neural network

    US20220335638A1

Cited By

  • Method for finding similar model based on point cloud

    KR102986669B1