Training a neural network

Abstract

A processing device trains a neural network Operations obtain (500) sets of images of real- world scenes. For each of the real-world scenes, operations obtain (502) a 3D scene model including a 3D point cloud and information indicating poses of cameras. Operations create (504) sub-sets of the set of images based on closeness of camera poses. For each of the sub-sets, operations generate (506) a 3D point cloud. Operations determine (506) pairs of the sub-sets of the selected images that satisfy an overlap rule. For each pair of sub- sets (510), the operations: determine a set of ground-truth matches between points of the 3D point clouds based on covisibility of points; and establish, using the neural network, matches between points of the 3D point clouds. The operations compute (512) an error based on comparison of the matches between points of the 3D point clouds; and 2) the ground-truth matches. The operations update (514) parameters of the neural network based on the error.

Description

TRAINING A NEURAL NETWORKTECHNICAL FIELD

[0001] The present disclosure relates to a processing device for training a neural network, a method by a processing device for training a neural network, and a corresponding computer program product.BACKGROUND

[0002] Mobile processing devices, such as robots, vehicles, and smartphones, can capture images of real-world scenes. To build 3D point clouds (PCs), also called maps, from images, most approaches rely on Structure-from -Motion (SfM) algorithms, e.g., as implemented by the COLMAP software, or Simultaneous Localization and Mapping (SLAM) algorithms, e.g., as implemented by the ORB-SLAM3 software. The resulting map contains a set of 3D points and the poses in a common coordinate system of cameras where images have been captured. To enable reuse of these point clouds for further tasks (e.g., Visual Localization algorithms), it is common to store a visual feature descriptor (e.g., scale-invariant feature transform (SIFT)) per map point, where the visual feature descriptor characterizes the local appearance of the points in the images.

[0003] An approach to match point clouds created by SfM and / or SLAM is to exploit the visual descriptors of point clouds. For example, some approaches compare and match individually the visual descriptors of a pair of point clouds, so the distance between the descriptors of two matched points in each point cloud should be small, while it should be high for the descriptors of the remaining points. An example of such approach is presented in [1, 2], Another approach that relies on descriptors is presented in [3], where an assumption on repetitive structure is used to fix the relative scale and height between the point clouds. Then possible connection points are found by matching SIFT features, then using the so-called Manhattan world assumption to find the possible transformations, and finally choosing the best one based on loop-closure and symmetry alignment criteria.

[0004] Since the result of SfM and / or SLAM is a 3D model / point cloud, those can also be merged by exploiting solely geometric cues, e.g., the three-dimensional location of the points in the models (thus avoiding the use of visual features). Indeed, this is tightly related to the 3D registration problem, where the goal is to find matches between two geometric point clouds, i.e., point clouds without visual descriptors, typically gathered through LiDAR orRGBD sensors, which are subsequently employed to estimate the pose transform (and, in some cases, the scale) aligning them. To find these correspondences, existing methods employ geometric features that characterize the spatial distribution of the points in the point neighborhood, such as Fast Point Feature Histograms FPFH (handcrafted), or as PREDATOR (learned). However, 3D registration is only considered in the case of relatively dense (i.e., more points per unit of volume) point clouds (e.g., PointNetLK) or single scans from RGB-D sensors as in RoITr [4], which are also denser than SfM / SLAM point clouds. This means that point cloud matching only using geometric information has not been explored in the context of point clouds created by Structure-from-Motion (SfM) and / or Simultaneous Localization and Mapping (SLAM), which are relatively sparse point clouds.

[0005] Regarding the density of point clouds, and the comparison of point clouds with respect to their density, if any prior information on the metric scale is available, e.g., from GPS or other positioning data, the density can be calculated as #points / volume. If the point clouds are normalized, so the points coordinates are between -1 and 1, one can also compute #points / volume (with a normalized volume), do voxelization and count #points per voxel, or measure the average / median distance between points.

[0006] Most approaches for matching point clouds created by SfM and / or SLAM discussed above rely on images and / or visual feature descriptors (e.g., SIFT) to find matches across map points from the different models. Moreover, they also consider a homogeneous SfM / SLAM process, i.e., the same algorithm being used for generating the different models. When relaxing the later assumption, previous approaches may not work in practice since different SfM / SLAM approaches tend to use different feature extractors, which yield incompatible descriptors (i.e., they are not distinctive enough to provide feasible matching, or are not comparable). Also, there is no guarantee on keypoint repeatability, i.e., different extractors will output different sets of keypoints. Furthermore, different point clouds may be built from different camera types, e.g., perspective, fisheye, etc, which can also lead to the use of specific feature extractors.

[0007] Additional disadvantages of using images and / or visual feature descriptors are:• Storage space: storing the original images represents a substantial increase in the space requirements of the 3D model. On the other hand, storing a visual descriptor per map point leads to a size increase of two orders of magnitude for integer descriptors and three others orders of magnitude for floating point descriptors in comparison to storing only the 3D points for the model.• Reduced privacy: images may contain personal data, objects and / or people that raise privacy concerns, requiring an additional anonymization process that adds extra computational complexity. When considering visual descriptors, they are susceptible to feature inversion attacks that can recover detailed images that resemble the original ones.• Scalability: traditional visual descriptor matching relies on finding the k nearest neighbors and then using ratio test to filter non-distinctive matches. Nearest neighbors have at worst quadratic time complexity and at best O (n log (log (n))). Moreover, it is shown to have poor performance, since most times descriptors are not distinctive enough to provide good matches. Recently, learned approaches have been employed to improve matching accuracy, but their runtime is higher than Nearest Neighbors.

[0008] Considering geometric-based matching approaches (which do not rely on visual descriptors) for point clouds created by depth sensors such as LiDAR and / or RGB-D sensors, they do not generalize well to SfM point clouds because these approaches have only been tailored to match dense point clouds (i.e., have more points per unit of volume). This phenomenon is depicted in Table 1 discussed below, where the performance of RoITr (a state-of-the-art geometric-based method) in (sparse) SfM point clouds is demonstrated to be low. The reasons for the lack of generalization are:• The representation in terms of discriminative geometric descriptors (these are equivalent to the visual descriptors computed from images) does not capture enough detail and / or context. This is because they rely on smaller neighborhoods to capture distinguishable characteristics of the local structure represented by the points, which is not sufficient for sparse point clouds (eventually, they would require bigger neighborhoods; while this can work it would lose accuracy; the matching would be more coarse in the sense that it would be matching bigger regions, so their centers are less likely to represent the same point; the bigger the region, the easier it is to find regions that look similar leading to false matches).• These models are trained using RGB-D / LiDAR data, so the approaches never dealt with point clouds from SfM / SLAM while learning the matching task, making it harder for them to be able to understand and correctly interpret the context / geometry around each point.SUMMARY

[0009] Some embodiments disclosed herein are directed to a processing device for training a neural network. The processing device includes at least one processor performing operations. The operations obtain sets of images of real-world scenes. For each of the real-world scenes, the operations obtain a 3D scene model of the real-world scene, where the scene model includes a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and information indicating poses of cameras when the images have been captured. The operations create sub-sets of the set of images of the real- world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set. For each of the sub-sets of selected images, the operations generate a 3D point cloud from the images in the sub-set. The operations determine pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets. For each pair of sub-sets, the operations: determine a set of ground-truth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair; and establish, using the neural network, matches between points of the 3D point clouds of the pair. The operations compute an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches. The operations update parameters of the neural network based on the error.

[0010] Some other embodiments are directed to a corresponding method by a processing device for training a neural network. The method obtains sets of images of real-world scenes. For each of the real -world scenes, the method obtains a 3D scene model of the real -world scene, where the scene model includes a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and information indicating poses of cameras when the images have been captured. The method creates subsets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set. For each of the sub-sets of selected images, the method generates a 3D point cloud from the images in the sub-set. The method determines pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets. For each pair of sub-sets, the method: determines a set of ground-truth matches between points of the 3D point clouds ofthe pair based on covisibility of points between the point clouds of the pair; and establishes, using the neural network, matches between points of the 3D point clouds of the pair. The method computes an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches. The method updates parameters of the neural network based on the error.

[0011] Some other embodiments are directed to a corresponding computer program product comprising a non-transitory computer readable medium storing instructions executable by at least one processor of a processing device to perform operations. The operations obtain sets of images of real-world scenes. For each of the real-world scenes, the operations obtain a 3D scene model of the real-world scene, where the scene model includes a 3D point cloud generated from the images of the real-world scene and representing structure of the real- world scene, and information indicating poses of cameras when the images have been captured. The operations create sub-sets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set. For each of the sub-sets of selected images, the operations generate a 3D point cloud from the images in the sub-set. The operations determine pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets. For each pair of sub-sets, the operations: determine a set of groundtruth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair; and establish, using a neural network, matches between points of the 3D point clouds of the pair. The operations compute an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches. The operations update parameters of the neural network based on the error.

[0012] As will be explained in further detail below, a potential advantage which may be provided by various embodiments is that the operations reduce memory utilization by storing just the 3D location of the points and neural network parameters without necessarily storing the visual descriptors and images. The operations may, in some instances, save up to three orders of magnitude in how much disk space, or other mass memory space, is used. The operations may similarly reduce the communication bandwidth requirements when streaming the point clouds, owing to their reduced size. Faster execution time may be provided as will be described in more detail below. Reduced privacy concerns can be achieved since the point clouds do not contain images nor visual descriptors.

[0013] Other processing devices, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such processing devices, methods, and computer program products be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and / or combination.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0015] Figure 1 illustrates poses of cameras when capturing images of a real-world scene;

[0016] Figure 2 illustrates example trajectories between the camera poses of Figure 1 corresponding to operations to create sub-sets of images of the real-world scene in accordance with some embodiments;

[0017] Figure 3 illustrates example operations by a neural network to match point clouds in accordance with some embodiments;

[0018] Figure 4 illustrates example application of the neural network to process points clouds to establish matches between points and compute poses of the devices, in accordance with some embodiments;

[0019] Figure 5 illustrates a flowchart of operations by a processing device for training a neural network in accordance with some embodiments; and

[0020] Figure 6 illustrates components of a mobile processing device and components of a network processing device which can operate in accordance with some embodiments.DETAILED DESCRIPTION

[0021] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0022] Embodiments of the present disclosure are directed to training a neural network that produces point matches from geometric cues, in such a manner that enables the neural network to perform matching of point clouds created by Structure-from-Motion (SfM) and / or Simultaneous Localization and Mapping (SLAM) approaches. For example, the generation of sub-sets of images based on a criterium allows for better generalization of the neural network when training.

[0023] Initially, a general overview of operations is provided which can be performed by a processing device for training a neural network in accordance with some embodiments. The operations are discussed with regard to the flowchart of Figure 5.

[0024] Referring to Figure 5, the operations obtain 500 sets of images of real-world scenes. For each of the real-world scenes, the operations obtain 502 a 3D scene model of the real- world scene. The scene model includes a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and includes information indicating poses of cameras when the images have been captured. The operations create 504 sub-sets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set. For each of the sub-sets of selected images, the operations generate 506 a 3D point cloud from the images in the sub-set. The operations determine 508 pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets. For each pair of sub-sets, the operations determine 510 a set of ground-truth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair, and the operations establish 510, using the neural network, matches between points of the 3D point clouds of the pair. The operations compute 512 an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches. The operations update 514 parameters of the neural network based on the error.

[0025] Some of the operational steps illustrated in Figure 5 may be optional. The order of some of the operational steps may be changed unless required to be in the illustrated order based on the output of one operational step necessarily providing input to a subsequent operational step.

[0026] The operation to create 504 the sub-sets may include to select an image from the set of images of the real-world scene as a subsequent image which has a smallest distance between camera poses of the selected subsequent image and the precedingly selected image.

[0027] The operation to establish 510, using the neural network, matches between the points of the 3D point clouds of the pair, may include to output for each established match, a pair of points with one of the points obtained from the 3D point cloud of one of the pair and the other one of the points obtained from the other 3D point cloud of the other one of the pair.

[0028] The operations may further include to generate the 3D point cloud and the information indicating poses of cameras when the images have been captured using a Structure from Motion (SfM) or a Simultaneous Localization And Mapping (SLAM) algorithm. Accordingly, the neural network may be trained using geometric information to match point clouds created by SfM and / or SLAM approaches.

[0029] When creating 504 the sub-sets of the set of images of the real-world scene, the operations may randomly select a first image and select subsequent images which are close in terms of the respective camera poses (translation and / or rotation of the camera between the locations where the images were captured). Subsequent images can also be selected further based on a number of covisible points (points in the real-world scene which have been captured in both images).

[0030] It is noted that the computation 512 of the error and the updating 514 of the neural network parameters are usually performed in a training phase. The images are thereby only used to generate the point clouds and establish the ground truth matches.

[0031] A potential advantage which may be provided by various embodiments is that the operations reduce memory utilization by storing just the 3D location of the points and neural network parameters without necessarily storing the visual descriptors and images. The operations may similarly reduce the communication bandwidth requirements when streaming the point clouds. Faster execution time may be provided by using the neural network for inferences rather than running computations for Nearest Neighbors, learned visual descriptor matchers or global image descriptors (i.e., NetVLAD) from hierarchical pipelines (e.g., hloc). Reduced privacy concerns can be achieved since the point clouds do not contain images nor visual descriptors.

[0032] A comparison is now made between some operations according to some embodiments to prior known point-cloud matching approaches:• Present operations can provide better generalization (“generalization” here means being able to perform well on previously unseen data, i.e., not the training data) to SfM and / or SLAM data and to sparse reconstructions from videos with low variation of viewpoint (see Table 1 and 2 where the neural network trained according to some present operations is named as SfMReger and shows improved performancecompared with a state-of-the-art 3D registration method called RoiTr). The comparison uses metrics: the number of matches, IR - inlier ratio (percentage), i.e., percentage of correct matches and FMR - Feature Matching Recall, which is the fraction of point cloud pairs whose IR is larger than a threshold (e.g., 5%).Table 1 below compares an approach in RoiTr [4] to present operations according to some embodiments, evaluated on the dataset generated by the present operations and 3DMatch.SE(3) Sim(3)Method IR FMR IR FMRRoiTr 1| (3DMntch’) 3.1) 12.6 1.6 7.0SfMReger (3DMatch) 8.6 29.8 5.1 20.3SfMReger (3DMatdh + MegaDepth) 48.7 95.1 44.1* 92.5*Table 2 below lists results of SfM Registration on the Cambridge dataset. Comparison between the state-of-the-art approach RoiTr [4] and the neural network trained with the proposed preprocessing method SfMReger.Great Court Kings College Old Hospital Shop Facade St Mary’s ChurchMethod IR Matches IR. Matches IR Matches IR Matches IR MatchesRoiTr [4] (3DMatch) 0 285 0 447 0 321 0.3 959 1.5 407SfMReger (3DMatch) 0 269 1.6 385 0 544 0.5 558 2.8 567SfMReger (3DMatch + MegaDepth) 69.4 4551 54.0 3246 28.0 1501 39.1 1342 77.8 5168

[0033] The operations to train a neural network that uses geometric information to match two point clouds created from SfM and / or SLAM are now described in further detail in accordance with some embodiments. The two point clouds may, e.g., originate from different devices, have been derived by different methods (SfM vs SLAM) or by using different algorithms implementing the same method (SfM or SLAM), or sensor data acquired from sensors with different characteristics or of different type.

[0034] The operations can include one or more of the following steps, which can be implemented in an order other than listed below:1) Obtain sets of (either ordered or unordered) images of different environments / scenes, the respective poses where the images were captured, and a point cloud created by triangulating points from the images, e.g., the point clouds from SfM and SLAM aregenerated from images, so the 3D points are obtained by triangulating the corresponding points from the images. Since the cameras are calibrated, a model is provided with a known direction of the light for each pixel (e.g., given by a line crossing the origin of the image coordinate system), and since the relative position of the cameras to each other is known, the operations may estimate the location of the 3D points by computing the intersection of lines from corresponding points. It is noted that in some embodiments, only using ordered images (typically from SLAM) may not suffice to sufficiently generalize to SfM scenes (typically consisting of disordered images) and vice-versa. According to some more general embodiments, operations obtain the information indicating poses of cameras when the images have been captured, and generate the 3D point cloud based on triangulating points between the images and using the poses of cameras where the images have been captured. For example, the operations to obtain the information indicating the poses, can include to obtain a set of unordered images from a scene, and then compute the poses from the images and generate the point cloud, e.g., simultaneously. a) The images may be self-acquired or publicly available, e.g., from MegaDepth [5], The number of images can play an important role in the success of the training. For example, MegaDepth has around 150K images across 196 different locations. For training the neural network, example test operations were performed using 110 scenes out of the 196 scenes of Megadepth, where each scene has between 1000 and 3000 images and for each scene the test operations created 10 subsets each with 100 to 200 images.2) For each scene, operations create sub-sets of images according to the closeness of the cameras in some space. In some embodiments, the operations determine closeness of camera poses based on camera translation and / or camera rotation between poses when the images were captured. a) for each scene do i) While (unaggregated images > 0) do(1) Random sample an image from the set of unaggregated images, and initialize a new image trajectory with it(2) Randomly sample a length for the trajectory Ti(3) While (trajectory length < Ti) do(a) Find the image in the set of unaggregated images that has the minimum distance to the image according to the selected criterium. This criterium can be:(i) A displacement or distance between camera poses where the images have been captured, i.e., the norm of the difference between the camera poses (the camera’s translation between capturing the two images) in the coordinate system of the point cloud of the whole scene(ii) A weighted sum of the displacement in both translation and rotation(iii)A number of covisible points (in the point cloud of the whole scene) between the images. Most available datasets contain that information already. For example, COLMAP outputs for each 3D point the images which capture the point.(b) Add that image to the trajectory. b) Figure 1 illustrates poses of cameras when capturing images of a real-world scene. Figure 2 illustrates example trajectories 200, 210, 220 between the camera poses of Figure 1 corresponding to operations to create sub-sets of images of the real-world scene in accordance with some embodiments. ) Operations generate a point cloud for each sub-set of images. In some embodiments, the operation to create 504 the sub-sets are operative to create the sub-sets to include ones of the images selected based on a number of covisible points in both the selected subsequent image and the precedingly selected image for the sub-set. In some embodiments, for each of the sub-sets of selected images, a 3D point cloud is generated 506 from the images in the sub-set through operations that include to extract features from the images using a feature extractor, match the features extracted from the images in the one of the sub-sets, determine locations of the matched features based on triangulating the matched features, and generate the 3D point cloud based on the determined locations of the matched features. a) Since most datasets provide a ground truth point cloud (a point cloud for the whole scene), we can obtain the sub-set point clouds by checking the points covisibility, i.e., which images see each point in the cloud, and adding to the point cloud the points that are viewed by at least two images in the sub-set. b) Alternatively, to increase robustness to point cloud pairs that were created by different methods, we can create new point clouds: by extracting keypoints from the images, using any feature extractor (e.g., SIFT, Superpoint, ORB) and finding matching keypoints between the images (using for example Nearest Neighbors, or SuperGlue, LightGlue) in the same sub-set, and use the ground truth poses of the images to triangulate the matched keypoints. In this case, feature extractors are only used tocreate the PC, but the visual descriptors which are created during the process are not used for matching the PCs. i) Even though the feature extractor is the same for all images in the same sub-set, different ones can be used for each sub-set. ii) Notice that, even though we use the images to generate the partial reconstructions, one PC for each sub-set of images, the images are not used anywhere else, including in supervising the training. ) Operations define point cloud pairs. In some embodiments, the operation to create 504 sub-sets of the set of images of the real-world scene, includes responsive to a pair of the sub-sets of the selected images having an amount of overlap that exceeds a threshold value, perform operations to: sample random points in a pair of the 3D point clouds generated from the images in the pair of the sub-sets; and for each of the sampled random points, generate a unit vector that defines a normal plane containing the sampled random point, and modify the pair of the 3D point clouds to reduce overlap by removing points from the pair of the 3D point clouds located within a threshold distance of a first side of the normal plane, and keeping points in the pair of the 3D point clouds located within the threshold distance of a second side of the normal plane opposite to the first side. a) PCs of the same location / scene are expected to have some degree of overlap (shared points which will likely generate correspondences), so an option is to set as pairs all possible combinations of sub-sets of each location. b) Alternatively, a situation can occur where there is no overlap. So, the pairs can be defined by measuring the overlap between the point clouds using the ground truth poses and admitting only pairs that have an overlap above a threshold a . Example operations to compute overlap are explained below. c) If all pairs have too much overlap this can lead the neural network to not be able to generalize to match pairs with low overlap. Then, pairs with high overlap (e.g., > 80%) are selected and their overlap is synthetically reduced by sampling random points in both point clouds, generate a random unit vector that defines the normal of a plane containing the point and remove / keep points only on the positive / negative side of the plane. d) Given the ground truth poses, ground truth correspondences are established between each pair. Two points are considered to match if the distance between the two points, after transforming one of points from the coordinate system of its PC to the coordinatesystem of the other PC, is below a threshold ft. In our experiments we set ft = 0.05 m. i) Ground truth pairs can also be obtained using the covisibility of 3D points in the ground truth point cloud created from all images of that location. Since the images in each trajectory / sub-set are known, the operations can see which 3D points are seen by images in the two trajectories of the pair. Due to noise in the images and / or error in the feature extraction the error of matches found this way can still be above , so those matches should also go through the threshold check. ) Using the geometric information contained in the point clouds, the neural network establishes matches between each pair of point clouds of the same scene, which will likely have some overlap. Operations for possible neural network configurations are described in further detail below. ) Operations compare the established matches to the ground truth matches to compute an error. In some embodiments, operations to compute 512 the error can include to compute an assignment matrix having entries determined based on the comparison of: 1) the points of the 3D point clouds established to match by the neural network; and 2) the ground truth matches, and compute the error based on the entries of the assignment matrix. a) The network outputs data identifying matches by computing an assignment matrix where, in some embodiments, the entries are 1 if the points match or 0 if they do not. However, a binary matrix may not be sufficiently differentiable (which can be a requirement to update the network, see next step). Thus, in some other embodiments the operations relax the binary matrix to use a matrix having entries with a range of values such as entriesG [0,1]. Example operations for computing this matrix are described further below. b) The neural network preferably operates to find correct matches, so the entries of the matrix corresponding to the correct matches should be as close to one as possible. Thus, a goal is to maximize the assignment values. Since the operations can be defining a minimization, the operations can be based on considering the minimization of the symmetric assignment values. c) An issue with only maximizing the assignment of true matches is that we are enforcing those values to be different than the incorrect matches. An option to prevent that is to simultaneously maximize the scores of correct matches and minimize thescore of wrong matches. This can be addressed using loss function such as the triplet loss or the circle loss.7) Operations update 514 the parameters of the neural network based on the error. In some embodiments, operations to update parameters of the neural network based on the error include to backpropagate the error through the neural network to obtain a gradient of the error with respect to the parameters of the neural network, and update the parameters of the neural network based on the gradient of the error. a) The error is backpropagated through the network to obtain the gradient of the error with respect to the model parameters. b) An optimizer is then used to update the neural network parameters. Example of optimizers are Stochastic Gradient Descent and Adam optimizer.

[0035] Steps 2, 3, and 4 can either be performed once as a pre-processing step, such that in some embodiments the operations perform steps 2, 3, and 4, once and then perform steps 5, 6, and 7, multiple times for a certain number of epochs (in Machine Learning this refers to processing all the data / pairs once) or until the error is below a threshold e. Alternatively, to have different trajectories in each training run, steps 2, 3, and 4, can be performed online at the cost of additional computational time to train the neural network.

[0036] Note that the pre-processing steps 2,3,4 and the training steps 5,6,7 may be repeated in the case of unseen data if the performance of the neural network is not sufficient. This process, called retraining or fine-tuning, might help the Neural Network to adapt better to the peculiarities of certain types of data (new descriptors, unseen structures, dynamic scenes).

[0037] Operations that can be performed to define image distance or closeness include:• The distance of the translation components of the images poses d(7), 7}) =\\ti — tj || , where ttrepresents the translation of the pose Tt= [ , Rttherotation, and || || represents a norm (e.g., L1(L2)• The distance on the rotation component of the images can be defined for example. z as d(T i, Tj) = acos• The weights if both distances are used can be defined either from trial and error, or by checking the ratio between the averages of both metrics for all images. For example, if the rotation has small variation and the translation varies more, the operations should give more weight to selecting based on the translation, since we can get two images with very similar rotation but that are very far apart.

[0038] Operations that can be performed to create sub-sets of initial images and length include:• Trajectories / sub-sets of the same environment / scene can all start from the same image, they can start from the image closest or farthest from the trajectories that have already been finished. Meaning selecting the first image randomly is an optional step, and in a test case it was defined that way since in the experiments it showed to be able to generalize well to unseen locations.• The length of the trajectories can be fixed. However, having variable length of the trajectories helps the model to generalize in matching point clouds of different sizes (i.e., size of the 3D bounding box of the point cloud).

[0039] Operations that can be performed to define trajectory -like sub-sets include:• A simpler and inferior approach than what is described in steps 2 and 3 above is to randomly sample a set of 3D points from the reconstruction of the scene and create a partial reconstruction from the images in which they are visible.• Testing has determined that while the simpler approach works well when registering reconstructions from collections of random images, it does not generalize as well to reconstructions from real camera trajectories. It is believed that point clouds generated from random images capture more information from the scene (it is seen from different viewpoints) and show little occlusion (most parts of the scene are visible when randomly sampling), so they are easier to register. On the other hand, sequential reconstructions (typical from sequential data and SLAM) usually capture the scene partially, tending to focus on specific parts of the environment, where occlusions tend to be more present, and visual information is not evenly distributed. Hence point clouds generated randomly exhibit large overlap, while trajectories often have significantly lower overlap.

[0040] Operations that can be performed to compute overlap between point clouds include:• Overlap is defined directionally, meaning from one point cloud to the other. Considering two point clouds A and B the overlap of A to B is defined as the percentage of points in A that are close enough (i.e., below a threshold a) to its nearest neighbor in B. The percentage is then given as overlap G4 -> B). overlap (B -> A~).

[0041] An example architecture configuration for the neural network which is focused on networks for 3D registration is now described. The neural network is not limited to thisexample and can be instantiated with a different architecture. The example neural network is described with reference to Figure 3 which illustrates example operations by the neural network to match point clouds in accordance with some embodiments.• The input has two point clouds of size (IV, 3) and (M, 3), respectively.• Both point clouds are processed by an encoder to obtain a discriminative vector per point. These vectors characterize each point allow to distinguish among them, such that similar points (in terms of the structure in a small vicinity of the points) if those vectors are similar and different if the distance between those vectors is high.Examples of encoders are: o PointNet o KPConv-FPN o PPFTrans• A similarity matrix or a score matrix is built based on a distance between the vectors, this matrix as size N*M. The distance between the vectors, which are of the same size, can be calculated as the Euclidean distance.• An assignment is established based on the similarity / score matrix. Examples on how to obtain this assignment are: o Pose the assignment problem as an optimal transport and use the Sinkhom algorithm to solve it. o Define a soft assignment matrix using a double sofmax function similar to LightGlue.• Even though computing the score matrix directly on the vectors from the encoder can work, it can fail to capture global context from the point clouds. To improve the neural network matches the discriminative vectors can first be glued together with a Transformer. o This process can be similar to the image feature matching of SuperGlue, LightGlue. o Conceptually, there are three fully connected graphs, one containing all the points from the first point cloud as nodes, one with all the points from the second point cloud, and one containing all points from both clouds. Then a process of self-attention is used in the first two graphs separately, their nodes are used to create the third graph and a process of cross-attention is used to pass messages between the point clouds.o The improved vectors are then used to compute the similarity / score matrix.• Additionally, the previous step can be further refined by taking the top-k matches (mt, rrij). One experiment set k=255, which is a result from tests with different values. o A local Transformer is used on the matches to obtain refined the discriminative vectors further, by gluing together the local features on the neighborhood around each match. o Again, there are three types of graphs. The first takes the 64 closest points to each point inthe second the 64 closest points to each point in m7, so there are 255 graphs for each point cloud representing the neighborhoods of each of the top matches, each of which goes to a self-attention transformer. The third type of graphs are constructed after matching the local graphs in each point cloud and going through a cross-attention process. o From these refined features the operations construct a local similarity matrix, which is used in the assignment step.• The final matches can consist of the most confident matches.

[0042] Figure 4 illustrates example application of the neural network to process points clouds to establish matches between points and compute poses of the devices, in accordance with some embodiments. Given point clouds, e.g., from two different devices or from the same device, the neural network finds matches between them that can be used to compute the pose between the devices. Optionally, the pose can be used to merge the point clouds.

[0043] Referring to Figure 4, after training the neural network, during inference the neural network estimates the matching points between the input point clouds. Two possible applications of these results are described below:1. The matches are used to solve for the pose, i.e., the relative translation / rotation between the two point clouds. This can be equivalent to solving the 3D registration problem. Possible algorithms to get the pose are: RANSAC, Iterative Closest Point(however, this needs a good initial guess), Fast Global Registration, or Maximal Cliques. a. Optionally the pose can be further refined by using: i. ICP, the previous step provides a good approximation of the solution which allows for faster convergence and increase robustness. ii. Nonlinear Least squares, using for example the Levenberg-Marquardt algorithm to solve it.2. Optionally, the relative pose or / and the matches can be used to compute the merged point cloud. The matches can be used to merge the points of one cloud to the other after by taking the mean, median or selecting one of the points to keep, and this process can be further refined if applying the relative pose. The non-matched points can be added directly after applying the relative pose.

[0044] Figure 6 illustrates components of a mobile processing device and components of a network processing device which can operate in accordance with some embodiments.

[0045] Referring to Figure 6, operations of a processing device as described herein may be performed by components of the illustrated mobile processing device 600 alone or in combination with a network processing device 640.

[0046] The mobile processing device 600 includes at least one camera 612 ("camera"), at least one processor 610 ("processor"), at least one memory 620 ("memory") storing program code executable by the processor 610, and a wireless transceiver 614 to communicate with a radio access network 630. Memory 620 may include a map processing module 622, a 3D scene model 624, a neural network training module 626, and a neural network 630 having instructions executable by the processor 610 to operate according to one or more embodiments disclosed herein. The neural network 630 may be a software-based implementation or a hardware circuit separate from the memory 620.

[0047] The network processing device 640 includes at least one processor 650 ("processor"), at least one memory 660 ("memory") storing program code executable by the processor 650, and a network interface 680 to communicate with the mobile processing device 600.Memory 660 may include a map processing module 662, a 3D scene model 664, a neural network training module 666, and a neural network 670 having instructions executable by the processor 650 to operate according to one or more embodiments disclosed herein. The neural network 670 may be a software-based implementation or a hardware circuit separate from the memory 660.

[0048] Further definitions and embodiments are now explained below.

[0049] In the above description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0050] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0051] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0052] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item and is notintended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id Est," may be used to specify a particular item from a more general recitation.

[0053] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0054] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0055] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / orblocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0056] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents and shall not be restricted or limited by the foregoing detailed description. REFERENCES[1] A. Locher, M. Havlena, and L. Van Gool, "Progressive structure from motion", Computer Vision - ECCV 2018, Springer, 2018, doi: 10.1007 / 978-3-030-01225-0_2 P[2] M. Havlena, A. Torii, and T. Pajdla, "Efficient structure from motion by graph optimization", Computer Vision - ECCV 2010, Springer, 2010, doi: 10.1007 / 978-3-642- 15552-9_8[3] A. Cohen, T. Sattler, and M. Pollefeys, "Merging the unmatchable: Stitching visually disconnected SfM models", 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, 2015, doi : 10.1109 / ICC V.2015.246[4] H. Yu, Z. Qin, J. Hou, M. Saleh, D. Li, B. Busam, and S. Ilic, "Rotation-invariant transformer for point cloud matching", 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5384-5393, IEEE, 2023, doi : 10.1109 / CVPR52729.2023.00521[5] Z. Li and N. Snavely, "Megadepth: Learning Single- View Depth Prediction from Internet Photos", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2041-2050, IEEE, 2018, doi: 10.1109 / CVPR.2018.00218

Claims

CLAIMS:

1. A processing device (800) for training a neural network, the processing device comprising at least one processor (810) operative to: obtain sets of images of real-world scenes; for each of the real-world scenes: obtain a three-dimensional, 3D, scene model of the real-world scene, the scene model comprising a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and information indicating poses of cameras when the images have been captured; create sub-sets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set; for each of the sub-sets of selected images, generate a 3D point cloud from the images in the sub-set; determine pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets; for each pair of sub-sets: determine a set of ground-truth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair, and establish, using the neural network, matches between points of the 3D point clouds of the pair; compute an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches; and update parameters of the neural network based on the error.

2. The processing device of any of Claim 1, wherein to create the sub-sets, the at least one processor is further operative to: select an image from the set of images of the real-world scene as a subsequent image which has a smallest distance between camera poses of the selected subsequent image and the precedingly selected image.

3. The processing device of any of Claims 1 to 2, wherein to establish, using the neural network, matches between the points of the 3D point clouds of the pair, the at least one processor is further operative to: output for each established match, a pair of points with one of the points obtained from the 3D point cloud of one of the pair and the other one of the points obtained from the other 3D point cloud of the other one of the pair.

4. The processing device of any of Claims 1 to 3, wherein the at least one processor is further operative to: generate the 3D point cloud and the information indicating poses of cameras when the images have been captured using a Structure from Motion, SfM, or a Simultaneous Localization and Mapping, SLAM, algorithm.

5. The processing device of any of Claims 1 to 4, wherein the at least one processor is further operative to: obtain the information indicating poses of cameras when the images have been captured; and generate the 3D point cloud based on triangulating points between the images and using the poses of cameras where the images have been captured.

6. The processing device of any of Claims 1 to 5, wherein the at least one processor is further operative to determine closeness of camera poses based on camera translation and / or camera rotation between poses when the images were captured.

7. The processing device of any of Claims 1 to 6, wherein to create the sub-sets the at least one processor is further operative to: create the sub-sets to include ones of the images selected based on a number of covisible points in both the selected subsequent image and the precedingly selected image for the sub-set.

8. The processing device of any of Claims 1 to 7, wherein, for each of the sub-sets of selected images, to generate a 3D point cloud from the images in the sub-set, the at least one processor is further operative to: extract features from the images using a feature extractor;match the features extracted from the images in the one of the sub-sets; determine locations of the matched features based on triangulating the matched features; and generate the 3D point cloud based on the determined locations of the matched features.

9. The processing device of any of Claims 1 to 8, wherein to create sub-sets of the set of images of the real-world scene, the at least one processor is further operative to: responsive to a pair of the sub-sets of the selected images having an amount of overlap that exceeds a threshold value, performing operations to: sample random points in a pair of the 3D point clouds generated from the images in the pair of the sub-sets; and for each of the sampled random points, generate a unit vector that defines a normal plane containing the sampled random point, and modify the pair of the 3D point clouds to reduce overlap by removing points from the pair of the 3D point clouds located within a threshold distance of a first side of the normal plane, and keeping points in the pair of the 3D point clouds located within the threshold distance of a second side of the normal plane opposite to the first side, wherein the neural network establishes matches between points of the modified pair of the 3D point clouds.

10. The processing device of any of Claims 1 to 9, wherein to compute the error, the at least one processor is further operative to: compute an assignment matrix having entries determined based on the comparison of: 1) the points of the 3D point clouds established to match by the neural network; and 2) the ground truth matches; and compute the error based on the entries of the assignment matrix.

11. The processing device of any of Claims 1 to 10, wherein to update parameters of the neural network based on the error, the at least one processor is further operative to: backpropagate the error through the neural network to obtain a gradient of the error with respect to the parameters of the neural network; and update the parameters of the neural network based on the gradient of the error.

12. The processing device of any of Claims 1 to 11, wherein the at least one processor is further operative to: perform a defined number of iterations or until the error is below an error threshold, the operations to establish, using the neural network, matches between points of the 3D point clouds of the pair, compute an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches, and update parameters of the neural network based on the error .

13. A method by a processing device for training a neural network, the method comprising: obtaining (500) sets of images of real-world scenes; for each of the real-world scenes: obtaining (502) a three-dimensional, 3D, scene model of the real-world scene, the scene model comprising a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and information indicating poses of cameras when the images have been captured; creating (504) sub-sets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set; for each of the sub-sets of selected images, generating (506) a 3D point cloud from the images in the sub-set; determining (508) pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets; for each pair of sub-sets: determining (510) a set of ground-truth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair, and establishing (510), using the neural network, matches between points of the 3D point clouds of the pair;computing (512) an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches; and updating (514) parameters of the neural network based on the error.

14. The method of Claim 13, further comprising performing the operations of any of Claims 2 through 12.

15. A computer program product comprising a non-transitory computer readable medium storing instructions executable by at least one processor of a processing device to perform operations to: obtain sets of images of real-world scenes; for each of the real-world scenes: obtain a three-dimensional, 3D, scene model of the real-world scene, the scene model comprising a 3D point cloud generated from the images of the real-world scene and representing structure of the real-world scene, and information indicating poses of cameras when the images have been captured; create sub-sets of the set of images of the real-world scene, the images of each sub-set being selected based on closeness of camera poses between images subsequently selected for the sub-set; for each of the sub-sets of selected images, generate a 3D point cloud from the images in the sub-set; determine pairs of the sub-sets of the selected images that satisfy an overlap rule based on the camera poses of the images of the sub-sets and / or the 3D point clouds generated from the images of the sub-sets; for each pair of sub-sets: determine a set of ground-truth matches between points of the 3D point clouds of the pair based on covisibility of points between the point clouds of the pair, and establish, using a neural network, matches between points of the 3D point clouds of the pair; compute an error based on comparison of: 1) the matches between points of the 3D point clouds established by the neural network; and 2) the ground-truth matches; and update parameters of the neural network based on the error.

16. The computer program product of Claim 15, wherein the operations further comprise any of Claims 2 through 12.

Images