Systems and methods for determining locations of vehicles along a pathway

Abstract

A computer-implemented method includes receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, including: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured.

Description

SYSTEMS AND METHODS FOR DETERMINING LOCATIONS OF VEHICLES ALONG A PATHWAYCross Reference To Related Applications

[0001] The present application claims priority to U.S. Provisional Patent Application No. 63 / 639,318, filed April 26, 2024, entitled “TRACKSIGHT THEORY AND IMPLEMENTATION,” the disclosure thereof incorporated by reference herein in its entirety.

[0002] The present application is related to U.S. Patent Application No. (TBA, attorney Docket No. 00212-001001), filed April 25, 2025, entitled “SYSTEMS AND METHODS FOR DETERMINING LOCATIONS OF VEHICLES ALONG A PATHWAY USING ACCELERATED KERNELIZED ADAPTIVE ZERNIKE MOMENTS (AKAZE).

[0003] The present application is related to U.S. Patent Application No. (TBA, attorney Docket No. 00212-001002), filed April 25, 2025, entitled “SYSTEMS AND METHODS FOR DETERMINING LOCATIONS OF VEHICLES ALONG A PATHWAY USING A FAST FOURIER TRANSFORM (FFT)”.

[0004] The present application is related to U.S. Patent Application No. (TBA, attorney Docket No. 00212-001003), filed April 25, 2025, entitled “SYSTEMS AND METHODS FOR DETERMINING LOCATIONS OF VEHICLES ALONG A PATHWAY USING ORIENTED FAST AND ROTATED BRIEF (ORB)”.Description of Related Art

[0005] The disclosed technology relates generally to transportation intelligence, and more specifically to systems and methods for identification of transit location.Summary

[0006] In general, one aspect disclosed features a system, comprising: one or more hardware processors; and one or more non-transitory machine-readable storage media encoded with instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

[0007] Embodiments of the system may include one or more of the following features. In some embodiments, generating the fingerprints comprises: creating a nonlinear scale space using the floating-point numbers; and detecting features in the nonlinear scale space. In some embodiments, creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images. In some embodiments, generating the fingerprints further comprises: creating masks for the first2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks. In some embodiments, generating the fingerprints further comprises: refining the retained features. In some embodiments, generating the fingerprints further comprises: creating descriptors for the refined features; and comparing the descriptors. In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

[0008] In general, one aspect disclosed features one or more non-transitory machine-readable storage media encoded with instructions that, when executed by one or more hardware processors of a computing system, cause the computing system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2Dimages to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

[0009] Embodiments of the one or more non-transitory machine-readable storage media may include one or more of the following features. In some embodiments, generating the fingerprints comprises: creating a nonlinear scale space using the floating- point numbers; and detecting features in the nonlinear scale space. In some embodiments, creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images. In some embodiments, generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks. In some embodiments, generating the fingerprints further comprises: refining the retained features. In some embodiments, generating the fingerprints further comprises: creating descriptors for the refined features; and comparing the descriptors. In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

[0010] In general, one aspect disclosed features a computer-implemented method comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

[0011] Embodiments of the computer-implemented method may include one or more of the following features. In some embodiments, generating the fingerprints comprises: creating a nonlinear scale space using the floating-point numbers; and detecting features in the nonlinear scale space. In some embodiments, creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images. In some embodiments, generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; retaining the features that are within the eroded masks; and refining the retained features. In some embodiments, generating the fingerprints further comprises: creating descriptors for the refined features; and comparing the descriptors.In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

[0012] In general, one aspect disclosed features a system, comprising: one or more hardware processors; and one or more non-transitory machine-readable storage media encoded with instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

[0013] Embodiments of the system may include one or more of the following features. In some embodiments, generating fingerprints from the floating-point numbersusing a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array. In some embodiments, generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array. In some embodiments, comparing the fingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images. In some embodiments, comparing the third 2D images comprises: calculating Euclidean distances between corresponding values in the third 2D images; and calculating standard deviations of the Euclidean distances. In some embodiments, selecting one of the second 2D images based on the comparing comprises: selecting the second 2D image having the lowest standard deviation. In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

[0014] In general, one aspect disclosed features one or more non-transitory machine-readable storage media encoded with instructions that, when executed by oneor more hardware processors of a computing system, cause the computing system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

[0015] Embodiments of the one or more non-transitory machine-readable storage media may include one or more of the following features. In some embodiments, generating fingerprints from the floating-point numbers using a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array. In some embodiments, generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array. In some embodiments, comparing thefingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images. In some embodiments, comparing the third 2D images comprises: calculating Euclidean distances between corresponding values in the third 2D images; and calculating standard deviations of the Euclidean distances. In some embodiments, comparing the third 2D images comprises: selecting the second 2D image having the lowest standard deviation. In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

[0016] In general, one aspect disclosed features a computer-implemented method comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints fromthe floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

[0017] Embodiments of the computer-implemented method may include one or more of the following features. In some embodiments, generating fingerprints from the floating-point numbers using a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array. In some embodiments, generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array. In some embodiments, comparing the fingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images. In some embodiments, comparing the third 2D images comprises: calculating Euclidean distances between corresponding values in the third 2D images; calculating standard deviations of the Euclidean distances; and selecting the second 2D image having the lowest standard deviation. In some embodiments, the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping theresulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.Brief Description of the Drawings

[0018] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0019] The present disclosure, in accordance with one or more various embodiments, is described in detail with reference to the following figures. The figures are provided for purposes of illustration only and merely depict typical or example embodiments.

[0020] FIG. 1 depicts an example of registered survey point clouds near Penn Station, NYC.

[0021] FIG. 2 depicts a “cosine error” adjustment.

[0022] FIG. 3 is a flowchart illustrating a process 300 for pre-processing the LiDAR and RADAR data according to some embodiments of the disclosed technologies.

[0023] FIG. 4 provides a graphical illustration of operations for converting from Cartesian space into spherical coordinates.

[0024] FIG. 5 is an example grayscale image with captured LiDAR data plotted as points in the image according to some embodiments of the disclosed technologies.

[0025] FIG. 6 illustrates an example progression of the smudge dilation process according to some embodiments of the disclosed technologies.

[0026] FIG. 7 presents an example of an image where the points of FIG. 5 have been “smudge” dilated to create coherent areas.

[0027] FIG. 8 graphically illustrates redistribution of a tile’s histogram values above a clip limit 802. This process may be repeated until all values are below the clip limit.

[0028] FIG. 9 illustrates nearest neighbor, linear, and bilinear interpolation of histogram values between adjacent tiles.

[0029] FIG. 10 presents an example of an image normalized using the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm.

[0030] FIG. 11 is a flowchart illustrating a process for a processing pipeline for creating Channel A fingerprints according to some embodiments of the disclosed technologies.

[0031] FIG. 12 presents an example mask according to some embodiments of the disclosed technologies.

[0032] FIG. 13 presents an example eroded mask according to some embodiments of the disclosed technologies.

[0033] FIG. 14 presents sample images for each octave and sublevel of the nonlinear scale space according to some embodiments of the disclosed technologies.

[0034] FIG. 15 presents sample images of det H of the nonlinear scale space according to some embodiments of the disclosed technologies.

[0035] FIG. 16 is a flowchart illustrating a process for a processing pipeline for creating the nonlinear scale space according to some embodiments of the disclosed technologies.

[0036] FIG. 17 presents an example Scharr convolution with a 3x3 horizontal kernel, mapped to grayscale.

[0037] FIG. 18 presents an example Scharr convolution with a 3x3 vertical kernel, mapped to grayscale.

[0038] FIG. 19 presents an example second partial derivative with respect to x.

[0039] FIG. 20 presents an example second partial derivative with respect to y.

[0040] FIG. 21 presents an example second partial derivative with respect to both x and y.

[0041] FIG. 22 presents an example of a Gaussian blur with o=1.6 with 0.99 threshold, mapped to grayscale.

[0042] FIG. 23 presents an example of a determinant of a Hessian matrix for an image, according to some embodiments of the disclosed technologies.

[0043] FIG. 24 presents an example of the determinant of a Hessian matrix of an image plotted in 3D as a height map, according to some embodiments of the disclosed technologies.

[0044] FIG. 25 presents an example of a magnitude of the two orthogonal Scharr3x3 convolutions, mapped to grayscale, according to some embodiments of the disclosed technologies.

[0045] FIG. 26 presents a typical histogram of magnitude of gradients.

[0046] FIG. 27 illustrates a diffusivity function, according to some embodiments of the disclosed technologies.

[0047] FIG. 28 presents a sample Charbonnier diffusivity map.

[0048] FIG. 29 is a flowchart illustrating a process for diffusion according to some embodiments of the disclosed technologies.

[0049] FIGS. 30A,B,C,D present sequential applications of Charbonnier diffusion to an example image for T=0.149961 ,T=0.961474, T=0.115968, and T=0.27221 1 , respectively, according to some embodiments of the disclosed technologies.

[0050] FIG. 31 presents a sample 24x24 area of an image, according to some embodiments of the disclosed technologies.

[0051] FIG. 32 presents a determinant of the Hessian, according to some embodiments of the disclosed technologies.

[0052] FIG. 33 presents an example first level (2x2) of a descriptor, according to some embodiments of the disclosed technologies.

[0053] FIG. 34 illustrates calculating similarity of two small example descriptors, according to some embodiments of the disclosed technologies.

[0054] FIG. 35 illustrates paths of static features due to forward / backward motion.Note distortion due to spherical to planar mapping.

[0055] FIG. 36 is a flowchart illustrating a process 3600 for a processing pipeline for creating Channel B fingerprints according to some embodiments of the disclosed technologies.

[0056] FIG. 37 is a normalized grayscale image of the components of an example 64x16 fingerprint (real on the left, imaginary on the right), for illustration purposes only.

[0057] FIG. 38 graphically illustrates the Channel B matching process.

[0058] FIG. 39 is a flowchart illustrating a process 3900 for a processing pipeline for creating Channel C fingerprints according to some embodiments of the disclosed technologies.

[0059] FIG. 40 is an image with highlighting for a center pixel (green) and 16 surrounding pixels (red) as defined by Bresenham.

[0060] FIG. 41 illustrates example pairs of points used to sample a 31x31 region.

[0061] FIG. 42 is a high-level block diagram of a vehicle-borne system for determining locations of a vehicle along a pathway according to some embodiments of the disclosed technologies.

[0062] FIG. 43 shows the Superdroid HK-1000 robot platform in stock (i.e., unmodified) condition.

[0063] FIG. 44 shows the Superdroid HK-1000 robot platform as modified.

[0064] FIG. 45 shows results for Sequential Scan Matching Mode.

[0065] FIG. 46 shows results for Fingerprint matching mode.

[0066] FIG. 47 is a block diagram of a system used when a survey is taken, creating the fingerprint databases, according to some embodiments of the disclosed technologies.

[0067] FIG. 48 is a block diagram of the system during production use, according to some embodiments of the disclosed technologies.

[0068] FIG. 50 illustrates the MSOP Header Structure.

[0069] FIG. 51 illustrates the MSOP Data Block Structure.

[0070] FIG. 52 depicts a block diagram of an example computer system in which embodiments described herein may be implemented.

[0071] The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed.Detailed Description

[0072] Introduction

[0073] Embodiments of the present disclosure provide systems and methods for locating vehicles along pathways using LiDAR technology. For clarity and conciseness, in the following description, the vehicles and pathways are sometimes described as trains and tracks. However, the disclosed technology may be used to locate any vehicle along any pathway. Furthermore, the disclosed technology is not limited to LiDAR, and may be used with any technology that captures 3D data in the form of distance and reflectivity.

[0074] According to the disclosed embodiments, a LiDAR unit mounted on the train captures a 3D image, and uses the 3D image to determine a location of the train alongits track. In conventional approaches, the captured 3D image is compared to reference 3D images previously captured at known locations along the track. However, the computations required to quickly compare 3D images are so intensive that the power required by the computer performing the comparisons exceeds the power available on a standard subway train.

[0075] A high-quality LiDAR can produce nearly a million points per second. FIG.1 depicts an example of registered survey point clouds near Penn Station, NYC. A 3D model of an entire railway can result in trillions of points. The mathematics to correlate 3D points is complex and time-consuming. In order to reduce the computation, the survey model is typically decimated and partitioned, and some out-of-band input (such as a crude position) is used to reduce the search space. Even so, the amount of computation is daunting even with extremely powerful computers that would be unsuitable for rail use due to power and heat budgets.

[0076] The disclosed embodiments take a different approach. In the disclosed embodiments, the 3D images are converted to 2D images, and the 2D images are compared. The power required to compare 2D images is well within the capacity of a standard subway train. Accordingly, a computer performing the techniques described herein may be located on the train.

[0077] Embodiments of the disclosed technology may feature additional advantages. In particular, the disclosed location systems require no wayside equipment. That is, all of the equipment required to determine the location of the train may be locatedon or inside the train. As wayside equipment involves significant installation and maintenance costs, this feature greatly reduces the costs of operating a location system.

[0078] The disclosed technologies employ an approach that does not use a single 3D model, but rather uses the original point clouds captured during a survey. When a new point cloud is captured, the system tries to find the best match against all of the previously captured point clouds.

[0079] For a new point cloud c, a set of survey clouds C = { co, ci, C2, ... cn}, and a correlation function corr(a,b), the best matching point cloud cmsatisfies:

[0080] cm<- max corr(c, cx) cxEC

[0081] Several algorithms exist for this correlation function, most commonly involving least-squares fitting. However, as stated, this approach is not significantly better, as the correlation function is still too computationally expensive to be useful as-is.

[0082] Embodiments of the disclosed technologies provide a useful solution by transforming the point clouds into something simpler, and creating a correlation function that can use those transformed point clouds to produce a result proportional to the original correlation function on the original point clouds.

[0083] The disclosed technologies feature a hash function hash(x) and a corresponding correlation function hashcorr(a,b) such that:

[0085] Both the hash and hash correlation function may be usefully less compute- intensive than the original functions. The survey clouds may be hashed before runtime.During runtime, the current point cloud may be hashed only once. Once the hashing process is completed, the point clouds from which they were created may be discarded.

[0086] The disclosed systems may be built around hash functions that may have multiple diverse sets of hash and correlation functions which produce similar results, and which can be cross-checked for a vital implementation. The current disclosure describes three such sets of hash and correlation functions, referred to herein as channels A, B, and C.

[0087] The hashed point clouds are referred to herein as “fingerprints”. Each fingerprint encapsulates enough information for a correlation function to make a match. Unlike more conventional hash functions typically used in computing (such as MD5), which match with exact equality, the resulting fingerprints are “soft” in that a correlation function will result in a quality-of-match measurement. Because of the nature of sensors (which naturally have noise and bias), the nature of train vehicles (which wobble unpredictably), and the variability of environmental conditions (night vs day, rain, etc.), the fingerprints reduce sensitivity to these factors.

[0088] Sensors

[0089] The disclosed technologies use a sensor fusion approach. The primary sensor may be a Light Detection and Ranging (LiDAR) sensor, supported by one or more secondary sensors. The disclosed systems can technically support a variety of different types of LiDAR devices. The scan density, scan rate and timing information should meet a sufficiently high level of quality to create useful fingerprints.

[0090] LiDAR

[0091] Some embodiments employ a high-quality solid state LiDAR sensor as the primary sensor. For example, the LiDAR unit may be implemented using the RoboSense RS-M1-AN2 automotive-grade MEMS LiDAR, manufactured by RoboSense / Suteng Innovation Technology Co., Ltd of Shenzhen, China. This device has been integrated into over 50 models of automotive vehicles, and has been used by Piper Networks, Inc. for rail solutions.

[0092] Unlike a mechanical LiDAR, the RS-M1 has no moving parts. It consists of five MEMS sensors providing a 120° by 25° field of view, with an average angular resolution of 0.2°, and an accuracy of ± 5cm up to 200m. The RS-M1 produces 787,500 points / sec at a scan rate of 10Hz. The RS-M1 also provides the intensity of the signal return on a per-point basis, and provides microsecond resolution of timing.

[0093] As a single RS-M1 scan takes 100ms, for a fast moving vehicle, there can be a significant difference in the position of the vehicle between when a scan begins and when it ends, which can result in the “smearing” of the point cloud data. The timing information produced by the LiDAR, in conjunction with knowledge of the velocity of the vehicle, can be used to “correct” the point cloud data to use a single frame of reference of time.

[0094] One advantage of a rail system is that the possible movement of the vehicle is highly constrained. A general adjustment of point cloud data would require information about all three axes of translation as well as all three axes of rotation. However, for rail purposes, only the direction of movement is required, as any other movements and rotations can be safely ignored or addressed in the fingerprint.

[0095] LiDAR units address the night / day problem suffered by visible light cameras, since they produce their own light sources as lasers in the infrared spectrum. However, visible light direct from the sun may have some light components overlapping the IR spectrum used by the LiDAR, and so can result in some transient artifacts. Also, as multiple LiDARs shining on the same objects (or pointing at each other) can sometimes cause similar artifacts, some LiDARs support the ability to desynchronize their scans with other LiDARs devices.

[0096] Every 100ms, the RS-M1 produces a single scan. Multi-byte values are provided in network byte order.

[0097] Each scan is reported over User Datagram Protocol (UDP) on a 1000Base- T Ethernet connection as 630 1210-byte packets.

[0098] Each packet contains an absolute 10-byte ps resolution timestamp, and 25 blocks of data. The timestamp is provided as 6 bytes of seconds from epoch and 4 bytes of microseconds.

[0099] Each block contains an 8-bit ps resolution time offset from the packet timestamp, and 5 channels of data (one for each MEMS sensor).

[0100] Each channel contains a single point in spherical coordinates, consisting of 16-bit pitch with 0.01 ° resolution, 16-bit yaw with 0.01° of resolution, 16-bit distance with 5mm resolution, and an 8-bit intensity value. All channels in a block are captured simultaneously.

[0101] Each channel contains a single point in spherical coordinates, consisting of 16-bit pitch with 0.01 ° resolution, 16-bit yaw with 0.01° of resolution, 16-bitdistance with 5mm resolution, and an 8-bit intensity value. All channels in a block are captured simultaneously.

[0102] For yaw and pitch, the LiDAR produces an integer value in multiples of 0.01 degrees, offset by +32768, so as to be stored in a single 16-bit unsigned integer.

[0103] For instance, an angle of 0 degrees would be represented as 32768, a value of +20 degrees would be represented by 32768+20*100 = 34768, and a value of -45 degrees would be represented as 32768-45*100 = 28268.

[0104] These values in turn may be converted to radians by multiplying by pi / 180. These equations combine the correction of the 32768 offset, the conversion from hundredths of degrees to unit degrees and the conversion from degrees to radians.

[0105] For distance, the values provided by the LiDAR are in multiples of 5mm, so to get the distance in millimeters, the value is multiplied by 5.

[0106]

[0107]

[0108]

[0109] In total, each scan consists of 78,750 points (630 packets * 25 blocks * 5 channels) sent in 762,300 bytes over UDP, for an average of 9.68 bytes per point.

[0110] Documentation for the LiDAR data protocol is described in Section A below.

[0111] Doppler RADAR

[0112] As noted above, LiDAR data can be corrected for vehicle movement with knowledge of the vehicle velocity. Some embodiments use a narrow-beam Doppler radar system to determine vehicle velocity.

[0113] The Doppler radar system may be implemented using an OPS243- C Frequency Modulated Continuous Wave (FMCW) and Doppler Radio Detection and Ranging (RADAR) system, manufactured by OmniPreSense Corporation of San Jose, California, United States. The OPS243-C provides both ranging and velocity information. The OPS243-C uses a 20°x24° beam operating on the 24.00-24.25 GHz band, and reports velocity with up to 0.5% accuracy with a 60m reporting range. The interface is over RS-232 or USB serial at 9600 baud.

[0114] It should be understood that the disclosed systems do not require the use of this specific RADAR unit, or any RADAR unit. The systems just require some device to report velocity with enough precision and rate to correct the LiDAR data feed. This device is expected to be mounted on the end of a vehicle, angled to detect the movement relative to the ground: in the case of the train, the ties and ballast. Because of the angled mounting, a “cosine error” adjustment may be made to the device’s reading to compute the actual linear velocity of the vehicle.

[0115] A “cosine error” adjustment is described with reference to FIG. 2. FIG. 2 depicts a RADAR 102, an object of interest 104, and a line of sight from the RADAR 102 to the object of interest 104. The horizontal distance to the object of interest 104 is R. The vertical distance to the object of interest 104 is d. The line of sight 106 is depressed below horizontal by an angle 0.

[0116] Referring to FIG. 2, for the angle 0 and the measured velocity vM, the actual velocity vAmay be computed by:

[0118] If the angle 9 is not known, but instead the distance R and the height d are known, then 0 may be computed by:

[0119]

[0120] Since angle 0 does not change, the value is a constant and canbe computed once, reducing the adjustment to a single multiplication.

[0121] Fingerprints

[0122] Point cloud data as captured by the LiDAR is basically in 5 dimensions: 3 spherical coordinates (pitch, yaw, distance), 1 dimension of intensity, and 1 dimension in time. The disclosed methods may reduce this data to a 2-dimensional 8 bits-per-pixel (bpp) gray-scale image, then use 2D image processing techniques to further reduce the image into the fingerprints. Three diverse fingerprinting methods are disclosed herein. Two (referred to herein as “Channel A” and “Channel C”) may use feature extraction to create a set of scale-invariant “key points” and “descriptors” in the image, and may use a 2D key point matching algorithm to compute image similarity. The other (“Channel B”) may transform the 2D image into the frequency domain, may use a band- pass filter to encapsulate the important frequencies, and then may use a statistical method to compute similarity.

[0123] The disclosed implementations may exploit the separable nature of many image processing algorithms by using threads to spread the compute load across multiple CPU cores, and may take advantage of accelerated compute subsystems such as GPUs. The disclosed implementations may also take advantage of the specific use cases employed; many image processing algorithms may be highly optimized by reducing their generalized code to only that specifically used by the particular implementation. Also, exploiting the quantization and range of data produced by the sensors, lookup tables of previously computed values may be used extensively, trading memory for speed. In addition, some portions of certain algorithms do not require rigorous precision, so simplified versions of various functions may be employed. Examples of such simplified versions include Taylor Series or Newton’s Method approximations, or even some scurrilous hacks such as the infamous Quake III fast inverse square root function in the cases where the precise version is unacceptably slow. More information may be found in Section B below.

[0124] In preparation for eventual vital certification, the code may be written in MISRA C-2012, with minimal documented exceptions for cases where speed trumps strict compliance, while still maintaining safety.

[0125] Portions of the disclosed algorithms may also be implemented for computation on a graphics processing unit (GPU). Initial versions have been developed in OpenGL Shading Language (GLSL), a language similar to C. While not an official MISRA-supported language, care is taken to follow the spirit of MISRA. In practice, thenatural development patterns to create functional GPU software have many requirements consistent with creating safe code. More information may be found in Section C below.

[0126] Unlike most rail computing systems, some embodiments heavily use floating-point math, which makes them more sensitive to processor architecture. The code may be written to support both single (32 bit) and double (64 bit) precision floating point, the choice of which to use may be made at build-time.

[0127] The disclosed methods may statically allocate a set of arrays of various dimensions and data types for the use of the processing pipeline.

[0128] Pre-processing

[0129] FIG. 3 is a flowchart illustrating a process 300 for pre-processing the LiDAR and RADAR data according to some embodiments of the disclosed technologies. Process 300 may be used for all three channels A, B, and C.

[0130] The elements of the process 300 are presented in one arrangement.However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 300 may include other elements in addition to those presented. For example, the process 300 may include error-handling functions if exceptions occur, and the like.

[0131] Packet Capture

[0132] Referring again to FIG. 3, the process 300 may include a capture thread 302 and a main thread 304. In the capture thread 302, the LiDAR data 306 may be captured in packets, at 308. In the example of the RS-M1 LiDAR device, a network interface board provides a 1000Base-T Ethernet interface. The RS-M1 device has a fixedInternet Protocol (IP) address of 192.168.1.200, and sends UDP packets to IP address192.168.1.102. Each packet contains 1210 bytes of captured LiDAR data.

[0133] In the main thread 304, RADAR data 310 is captured, and used to determine the velocity of the vehicle, at 312.

[0134] The packets of captured LiDAR data may be passed to the main thread 304 by a queue 310. The queue 310 may be implemented as a circular queue able to hold three scans (630*3 packets) worth of data. The queue 310 may launch a thread whose sole purpose is to wait for incoming UDP packets and add them the tail of the queue 310. The queue 310 may not add packets if the queue 310 is full. The queue 310 may be protected by mutex.

[0135] Packet Decoding

[0136] The main thread 304 may have a loop that locks the mutex on the packet queue 310, and if the queue 310 contains any packets, removes the head element. The mutex is then released.

[0137] The process 300 may include packet decoding, at 316. The system may include a packet decoder having a 1024x256x8bpp image buffer allocated, which is filled in as a grayscale bitmap by the decoder.

[0138] The decoder may check the packet sequence number. If the packet sequence number is “1”, then the image may be cleared to 0 (black), and the ps portion of the timestamp may be captured for use as a base timestamp to adjust point locations. The base timestamp may be subtracted (adjusting for overflow) from the current packettimestamp to determine how much time has elapsed since the first packet of the scan was captured.

[0139] Each of the 25 blocks in the packet may then be processed, adding each block’s ps time offset from the accumulated packet timestamp to compute the amount of time from the start of the scan the block was captured.

[0140] Each of the 5 channels, each containing a single point, may then be processed. If any of the pitch, yaw and distance values are out of range (for example, either 0 or 65535), that indicates a lack of signal return, and that point may be ignored.

[0141] If the determined velocity of the vehicle is non-zero, then the point may be adjusted. To adjust the point, the raw distance may be converted to millimeters, and the sine and cosine of the pitch and yaw may be looked up in precomputed tables and used to convert the spherical coordinates into Cartesian space. The resulting X axis points along the rail, the Y axis to the left, and the Z axis up. Since the vehicle is traveling along the rail, the X value may be modified by the time from the first packet multiplied by the velocity of the vehicle, to move the point to where it was at the first packet capture. If the device is facing the direction of travel, the point is moved away; otherwise it is moved closer. In general, forward movement results in the points moving closer to the center of the image.

[0142] For a device pointing away from the direction of travel, operating at a sufficiently high speed could result in an adjusted point resolving behind the vehicle, so that point may be ignored.

[0143] The point may then be converted from Cartesian space back into spherical coordinates. For example, a fast arctangent function may be used. This collapses the time dimension into the three spherical dimensions.

[0144] FIG. 4 provides a graphical illustration of these operations, where:

[0145] 0m= measured yaw, 0a= actual yaw

[0146] ( / >m= measured pitch, f>a= actual pitch

[0147] dm— measured distance, da— actual distance

[0148] At = time of f set from start of scan, v = velocity along x axis

[0152] Note that only motion along the vehicle pathway (e.g., track) may be considered. While the vehicle is subject to additional transformations, they are de minimis over the typical time period. The yaw and pitch values may be converted to image space coordinates using a precomputed table, and checked to make sure they are within the bounds of the image.

[0153] A periodic triangle wave function of some configured period may be applied to the direction, which may be modulated by a power function based on the intensity, then mapped to a subset of the range [0...255]. This collapses the distance and intensity dimensions into a single dimension, where:

[0157] The gray value gray is then plotted in the image at the mapped pitch and yaw coordinates, according to:

[0158] x «- (0 ■ scale) + centerx

[0159] y <- (0 ■ scale) + centery

[0160] M[x,y] «- gray

[0161] Note that when adjusting for motion, it is possible for more than one point to resolve to the same pixel coordinates. A depth buffer may be employed to ensure the closest point is chosen for any given pixel. Once all of the points in the packet have been plotted, the packet’s sequence number may be checked. The packet decoding function returns true if the packet’s sequence number is 630 (that is, the last packet index for the scan), indicating that the scan is complete and ready for subsequent processing.

[0162] Smudge Dilation

[0163] FIG. 5 is an example grayscale image with captured LiDAR data plotted as points in the image according to some embodiments of the disclosed technologies.

[0164] Referring again to FIG. 3, the process 300 may include smudge dilation, at 318. The plotted scan (that is, the grayscale image) may be sparse in somesections, and may contain holes in other sections. The disclosed technologies may use a novel dilation algorithm to spread the non-zero pixel values into the zero pixel values. A similar result may be obtained by performing a Voronoi partitioning of the image around the non-zero pixels and filling each zero pixel in each region with the associated non-zero pixel. However, that method is too slow and would extend to the edges of the bitmap. Instead, the disclosed implementations may perform a fixed number of passes expanding each non-zero pixel into zero-pixel space in a general radial direction. The order in which the pixels for a given radius are smudged may be shuffled to reduce artifacts from appearing due to the algorithm. The amount of smudging may be chosen to blend the valid points within the scan together, not necessarily to extend to the edge of the bitmap. While the passes themselves are not separable, the processing of pixels within each pass is. In some embodiments, this function is threaded.

[0165] FIG. 6 illustrates an example progression of the smudge dilation process according to some embodiments of the disclosed technologies.

[0166] FIG. 7 presents an example of an image where the points of FIG. 5 have been “smudge” dilated to create coherent areas.

[0167] Bilateral Filter

[0168] Referring again to FIG. 3, the process 300 may include bilateral filtering, at 320. Here a small bilateral filter may be applied to the image. In some embodiments, a small bilateral filter may be applied to the entire image. This type of non- linear filter tends to smooth out low-contrast portions of the image while preserving high contrast elements such as edges.

[0169] The general concept is that the intensity of each pixel is replaced with a weighted average of the intensities of nearby pixels. The weighting may be based on not only the distance to each neighbor, but also on the difference in intensity. The weighting function may be the Gaussian function. When fixed kernel sizes are used for the filters, precomputed tables may be used for the complex portions of the algorithm, which significantly improves the performance over the general case:

[0172] I is the original image

[0173] ! filtered js thefHteredjmage

[0174] x are the coordinates of a pixel in the image

[0175] fl is the window centered around x, such that xf6 fl is another pixel

[0176] fris the range kernel for smoothing differences in intensities

[0177] gsis the spatial kernel for smoothing differences in coordinates

[0178] Using Gaussian weighting functions, this approach can be simplified.For a pixel at (ij), the total weighting function for a neighboring pixel at (kJ ) can be computed with:(i-fc)

[0179] w(i,j, k, r) = exp

[0180] where °d and aii' are the smoothing parameters for distance and intensity, respectively. Once all the weights from all the neighbors for a given pixel are calculated, they may then be normalized and the final pixel value computed:

[0182] In some embodiments, the system subsamples the local pixels to closely approximate the weighting. A Poisson-disk pattern may be used to determine which pixels are subsampled. These embodiments improve processing speed at the cost of some accuracy.

[0183] Contrast Limited Adaptive Histogram Equalization (CLAHE)

[0184] Referring again to FIG. 3, the process 300 may include Contrast Limited Adaptive Histogram Equalization (CLAHE), at 322. Here, in order to bring out more detail in the image, the image may be equalized. However, applying the same equalization to the entire image will result in fading out fine detail in sections. Instead, a CLAHE method may be used to normalize areas of the image separately and then interpolate them together again.

[0185] The image may be broken into tiles. For example, the image may be broken into 256 tiles. A histogram of the gray values may be created for each tile. The histogram may be clipped within a given range. The out-of-range values may be evenly redistributed within the histogram. The histogram may then be converted into a lookup table to convert the original gray value of a pixel into a new gray value.

[0186] FIG. 8 graphically illustrates redistribution of a tile’s histogram values above a clip limit 802. This process may be repeated until all values are below the clip limit.

[0187] This process results in individual histograms for each tile. Applying these histograms at this point would result in the tiles being disjoint because the same initial pixel value in adjacent tiles would likely end up with different values when mapped through the histogram. Instead, for each pixel in the image, the mapped values from the nearby tiles’ histograms are collected, and a new value for the pixel is computed by weighting the values based on the distance to these tiles, which smooths out the boundaries between tiles. FIG. 9 illustrates nearest neighbor (red), linear (green) and bilinear (blue) interpolation of histogram values between adjacent tiles.

[0188] FIG. 10 presents an example of an image normalized using the CLAHE algorithm.

[0189] CHANNEL A - IMAGE FEATURES FINGERPRINT (AKAZE)

[0190] Some implementations use the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm to compile a list of “key points” and associated “descriptors” for a particular image, which together comprise the fingerprint for Channel A. A matching algorithm may quickly compare two fingerprints and create a list of potentially matching key points. This list may be further filtered by checking for plausibility based upon the relative positions of the key points.

[0191] AKAZE is a form of nonlinear diffusion scale-space analysis combined with a response to a scale-normalized determinant of the Hessian to locate keypoints, which are found at the local minima and maxima of the response. For each key point, a descriptor is created to encapsulate the orientation of the local gradient, which allows rotation-invariant matching.

[0192] FIG. 11 is a flowchart illustrating a process 1100 for a processing pipeline for creating Channel A fingerprints according to some embodiments of the disclosed technologies.

[0193] The elements of the process 1100 are presented in one arrangement. However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 1100 may include other elements in addition to those presented. For example, the process 1100 may include error-handling functions if exceptions occur, and the like.

[0194] The process begins with the grayscale image output by the CLAHE process (FIG. 3, at 322). The grayscale image may be converted to floating point values, at 1102. The grayscale image may be converted to floating point values with the source range of [0 ... 255] mapped to [0.0 ...1.0] into a “real” array of the same dimensions.

[0195] Mask

[0196] Referring again to FIG. 11 , the process 1100 may include creating a mask, and insetting (i.e., “eroding”) the mask, at 1104 and 1106. This process ensures the identification of key points is the result of artifacts within the image, not the edges of the image. Here, the image may be “eroded” by a few pixels. Key points that are not within the mask may be discarded.

[0197] In some embodiments, the mask is modified to explicitly exclude portions of the image that are covered by the LiDAR, but that may not be helpful for the purpose of matching. For instance, in a rail embodiment, the railway roadbed itself may be excluded, as it’s very similar at all locations and thus may not add value for matching. Another example opposite to the railbed example concerns platforms at stations, where the arrangement of people would be different every time. In general, candidates for exclusion fall into two basic categories: features that are practically identical in a large subset of fingerprints and thus not useful for differentiation (e.g., the rail bed), or ones where the features change radically at the same location, creating "noise" that are also not useful for matching (e.g., the people on the platforms).

[0198] The mask may be created by checking whether the gray value is above or below a specified threshold. FIG. 12 presents an example mask according to some embodiments of the disclosed technologies.

[0199] Erosion

[0200] Erosion is a morphological image processing function that “erodes” pixels of a certain range of values. In some embodiments, a structuring element is passed across the image; a determination of coverage is made to determine if the pixel under the center of the structuring element should be removed. The structuring element may be a small circle or similar shape.

[0201] For Euclidean space E, and a binary image A in E, with structuring element B:

[0202] A © B = {z e E\BZc >1}

[0203] Bz= {b + z\b E B], Vz E E

[0204] In some embodiments, the actual code may be highly optimized by using a circular kernel, and instead of using a binary mask, computing the square of the Euclidean distance, short circuiting as soon as a black pixel is found within the circle.

[0205] This function is separable. In some embodiments, this function is threaded.

[0206] This process keeps the key point from being too close to the edges of the image or areas where the LiDAR may not have received a return signal, which could be erroneously treated as a physical object.

[0207] FIG. 13 presents an example eroded mask according to some embodiments of the disclosed technologies.

[0208] Creating the Nonlinear Scale Space

[0209] Referring again to FIG. 11 , the process 1100 may include creating a nonlinear scale space, at 1108. AKAZE uses a scale-space tree to apply various operations at different scales in order to produce the key features that are later abstracted. The tree has two dimensions: “octave”, in which the image is halved in dimension as the octave number increases, and “sublevel”, in which various blurs and diffusion steps are performed to isolate the features. For instance, the tree might have four octaves and four sublevels, for a total of 16 images. The images in octave 0 are the size of the original image, images in octave 1 are half the size dimensionally (i.e., a quarter the area), images in octave 2 are quarter the size dimensionally, etc.

[0210] As the tree is built, each image is blurred, its first and second partial derivatives are computed, a Hessian matrix is created, the determinant of the Hessian is computed, and a non-linear diffusion is performed. Each image then provides the starting point for the next image in the tree.

[0211] FIG. 14 presents sample images for each octave and sublevel of the nonlinear scale space according to some embodiments of the disclosed technologies.

[0212] FIG. 15 presents sample images of det H of the nonlinear scale space according to some embodiments of the disclosed technologies.

[0213] FIG. 16 is a flowchart illustrating a process 1600 for a processing pipeline for creating the nonlinear scale space according to some embodiments of the disclosed technologies.

[0214] The elements of the process 1600 are presented in one arrangement. However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 1600 may include other elements in addition to those presented. For example, the process 1600 may include error-handling functions if exceptions occur, and the like.

[0215] Scharr Operator

[0216] The AKAZE algorithm uses the Scharr operator to compute first- order partial derivatives of the images, which is part of the process of feature extraction, as derivatives can be used to locate edges. The Sobel operator may be used for this, but the Scharr operator improves rotational symmetry.

[0217] The Scharr operator uses two kernels, one for horizontal, and one for vertical, which are then convolved with the image. The 3x3 normalized convolution kernels for x and y first-order derivatives are given by:

[0218]

[0219]

[0220] The horizontal kernel effectively creates the first order partial derivative of the “function” f represented by the image with respect andsimilarly, the vertical kernel creates the first order partial derivative with respect to y (i.e., Some embodiments have an optimized function that minimizes the number of math operations required to perform this specific convolution with various kernel sizes. When creating the nonlinear scale space, kernels of larger sizes may be used. For examples, kernels of size 5x5, 7x7, and 9x9 may be used.

[0221] FIG. 17 presents an example Scharr convolution with a 3x3 horizontal kernel, mapped to grayscale.

[0222] FIG. 18 presents an example Scharr convolution with a 3x3 vertical kernel, mapped to grayscale.

[0223] In some embodiments, second-order derivatives

[0224] which are the components of a Hessian matrix of two variables, are created by combining various combinations of applications of the Scharr operator.

[0225] Note that because the x and y values are uncorrelated, the middle two second derivatives are actually the same and so can be computed just once, so the resulting Hessian matrix is symmetric across the main diagonal. This feature may be exploited for an efficient specific computation of a determinant.

[0226] FIG. 19 presents an example second partial derivative with respect to x.

[0227] FIG. 20 presents an example second partial derivative with respect to y.

[0228] FIG. 21 presents an example second partial derivative with respect to both x and y.

[0229] Gaussian Blur

[0230] When creating the scale-space, AKAZE also makes use of 2D Gaussian blurs. Referring again to FIG. 16, the process 1600 may include Gaussian blurring according to some embodiments of the disclosed technologies, at 1602. The 2D Gaussian function is of the form:

[0232] where x and y are the relative distances from the origin. As this function extends to all values of x and y, in theory, every pixel in an image affects the blur for any other pixel, but in practice, far away pixels have negligible effect. Someembodiments generate kernels that allow control over the threshold of minimum contribution by integrating the Gaussian function to find a kernel size for which the integration reaches the threshold.

[0233] The 2D Gaussian blur is separable, so it can be computed by performing a blur on the rows, then the columns, and can be threaded. In some embodiments, this function is threaded.

[0234] FIG. 22 presents an example of a Gaussian blur with a = 1.6 with 0.99 threshold, mapped to grayscale.

[0235] Gradient Contrast Factor

[0236] Referring again to FIG. 16, the process 1600 may include computing a Gradient Contrast Factor, according to some embodiments of the disclosed technologies, at 1604. The contrast factor may be computed according to the AKAZE algorithm. Using the first partial derivative images that together specify a gradient, the magnitude of the gradient at each point is calculated, with the maximum value recorded.

[0237] FIG. 25 presents an example of a magnitude of the two orthogonal Scharr 3x3 convolutions, mapped to grayscale, according to some embodiments of the disclosed technologies.

[0238] The maximum value may then be used to scale the magnitudes and sort them into a collection of bins of equal spacing. For example, the number of bins may be 300. A scan of the resulting histogram table may be used to determine the bin index at which the cumulative number of items in the bins reaches a percentile threshold. For example, the percentile threshold may be 70%. The contrast factor may then becomputed by multiplying this index by the maximum magnitude value and dividing by the number of bins.

[0239] FIG. 26 presents a typical histogram of the magnitude of the gradients. Here the max value of 6.863, an index of 66 of 300, and a computed contrast of 1.510.

[0240] Determinant of the Hessian Response

[0241] Referring again to FIG. 16, the process 1600 may include computing the determinant of the Hessian response, according to some embodiments of the disclosed technologies, at 1606. The Hessian matrix is a square matrix of second order partial derivatives of a scalar function of multiple variables. In the general case, the Hessian Hffor a function f(x-i, X2, ... xn) is:

[0242]

[0243] In the disclosed embodiments, the “function” is the output of the Gaussian blur, and the variables are x and y, so the Hessian matrix is of dimensions 2x2, and the determinant is trivial to calculate:

[0246] Using the second order partial derivatives created by applying the Scharr operator in various combinations, and computing the determinants, results in an image where that begins to identify the important “key points” in the image.

[0247] FIG. 23 presents an example of a determinant of a Hessian matrix for an image, according to some embodiments of the disclosed technologies.

[0248] FIG. 24 presents an example of the determinant of a Hessian matrix of an image plotted in 3D as a height map, according to some embodiments of the disclosed technologies.

[0249] Diffusion

[0250] Referring again to FIG. 16, the process 1600 may include computing and applying diffusivity, according to some embodiments of the disclosed technologies, at 1608 and 1610, respectively. Creating the scale space involves some smoothing of the image, through diffusion. A simple diffusion would use the Gaussian, but would not preserve some features such as edges, which would be blurred with a simple Gaussian. Instead, some embodiments use the AKAZE nonlinear diffusion function to control how the diffusion occurs. Also, usually diffusion is performed in even increments of “time”. Instead, some embodiments use the AKAZE technique called “Fast Explicit Diffusion” (FED), which uses fewer non-uniform time steps to obtain an identical result.

[0251] For a signal f(x), a nonlinear diffusion u(x,t) may be created with a diffusivity function #(|x|):

[0253] In some embodiments, the common “Charbonnier” diffusivity function is used:

[0255] FIG. 27 illustrates a diffusivity function 2700, according to some embodiments of the disclosed technologies. For each pixel, the diffusivity function 2700 fetches the values of the corresponding value in the first partial derivatives ,then combines them, at 2702, as the “length” of the vector

[0257] This parameter "x" is sent to the diffusivity function 2700. In the case of Charbonnier, the diffusivity function 2700 internally uses x2, so the code avoids computing the square root since it's just going to square it anyway. The outputs of the diffusivity function are the diffusivity map 2704.

[0258] FIG. 28 presents a sample Charbonnier diffusivity map.

[0259] Since this function is constant in time for a given value of A (which is the gradient contrast factor, reduced by 25% for each increasing octave), it can be computed once per octave / sublevel in the scale space and applied for each FED time step T:

[0261] FIG. 29 is a flowchart illustrating a process 2900 for diffusion according to some embodiments of the disclosed technologies.

[0262] The elements of the process 2900 are presented in one arrangement. However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 2900 may include other elements in addition to those presented. For example, the process 2900 may include error-handling functions if exceptions occur, and the like.

[0263] Referring to FIG. 29, the process 2900 is repeated for each time step T. A diffusion step 2904 may be computed, at 2902. The inputs to the computation may include an input image 2906, the diffusivity image 2908, and the time step T.

[0264] The process 2900 may include applying the diffusion step 2904 by computing an output image 2910 based on the input image 2906 and the diffusion step 2904, at 2912.

[0265] FIGS. 30A,B,C,D present sequential applications of Charbonnier diffusion to an example image for T = 0.149961, T = 0.961474, T = 0.115968, and T = 0.272211, respectively, according to some embodiments of the disclosed technologies.

[0266] Referring again to FIG. 16, the process 1600 may include downscaling the image, according to some embodiments of the disclosed technologies, at 1612, respectively. The “downscale” image is the image for the next octave at sublevel 0. For instance, referring again to FIG. 14, the image for Octave 2, Sublevel 0 is the downscale image of the image for Octave 1 , Sublevel 0. Note that for a given octave’simage, the process creates the sublevel images, then reduces the size of the octave image by 50%, and creates the sublevels from *that* image, etc. Basically, the down- sampling is represented by the sequence of left-to right images in FIG. 14.

[0267] Feature Detection

[0268] Referring again to FIG. 11 , the process 1100 may include feature detection, according to some embodiments of the disclosed technologies, at 1110. Once the images of determinants of the Hessian for each octave / sublevel have been created, they may then be scanned for local extrema, defined as a pixel where (a) the value is above a certain threshold and (b) where all the surrounding eight pixels have lower values. A “size” correlating with the pixel’s scale, and the value, may be attached to this point. Then the point may be added to a list of candidates. Previously found points may then be compared with this new point to make sure it’s not a “duplicate”, defined as another point where the new point is closer than the other point’s “size”.

[0269] FIG. 31 presents a sample 24x24 area of an image, according to some embodiments of the disclosed technologies.

[0270] FIG. 32 presents a determinant of the Hessian, according to some embodiments of the disclosed technologies.

[0271] Once all the extrema in all the levels have been collected, then another scan may take place to weed out any similar points in higher levels. These are the candidate key points.

[0272] Applying the Mask

[0273] Referring again to FIG. 11 , the process 1100 may include applying the mask, according to some embodiments of the disclosed technologies, at 1112. Once the key points have been collected, they may be filtered against the mask so only the important elements in the image remain, without the boundary edges or holes in the image. This may be accomplished by looking up the pixel in the eroded mask corresponding to the key point and verifying that it’s within the mask.

[0274] Refining the Features

[0275] Referring again to FIG. 11 , the process 1100 may include refining the features, according to some embodiments of the disclosed technologies, at 1114. The first pass of key point collection yields integer coordinates. Another pass may be run through the key points, collecting first and second order partial derivatives of the determinant of the Hessian for the nine pixels around the point to create a small Hessian matrix and vector, which creates a pair of linear equations with two free variables. These equations may be solved for their variable values, which creates a subpixel offset in both x and y from the integral point position to a subpixel point position, which is then added to the original point location.

[0276] For a key point (x, y) and a function f= det H,

[0277]

[0278]

[0279]

[0280]

[0282] Solving these two linear equations for dx and dy (see Section D) yields:

[0283]

[0284]

[0285] The refined point position is given by:

[0286]

[0287]

[0288] Creating the Descriptors

[0289] Referring again to FIG. 11 , the process 1100 may include creating the descriptors, according to some embodiments of the disclosed technologies, at 1116. Having just a list of key points alone is not enough to correlate points between two images; additional information should be collected about the area surrounding each key point. The disclosed embodiments may use the Modified Local Difference Binary (MLDB) descriptor to encode this information. Because image rotation is not a material factor for the application, the descriptors are the “upright” version, which are not rotation invariant, and are quicker to compute.

[0290] A small area around each key point is analyzed using the original image and the first partial derivatives in x and y as “channels”. The area is broken into 2x2, 3x3 and 4x4 grids, and all of the values in each subarea are averaged. Each of the averages are compared against each other, and a bit is set for each comparison.

[0291] For instance, with a descriptor area size of 24x24, the grid is split into a 2x2 grid of 12x12 pixel subareas. The 144 values in each subarea are averaged, to create a total of four values, one for each quadrant (NW, NE, SW, and SE). Each quadrant is then compared uniquely with every other quadrant , i.e.. (NW>NE), (NW>SW), (NW>SE), (NE>SW), (NE>SE), (SW>SE) resulting in 6 bits.

[0292] This may be done for each of the three images (original,for a total of 6*3 = 18 bits. In the bit vector, the 3 different sets of bits are interleaved.

[0293] The general formula for the number of combinations of n items from a list of m items is:[100294] J

[0295]

[0296]

[0297]

[0298]

[0299] FIG. 33 presents an example first level (2x2) of a descriptor, according to some embodiments of the disclosed technologies.

[0300] The process may be repeated for a 3x3 grid of 8x8 pixel subareas, for a total of 36*3 = 108 bits, and for a 4x4 grid of 6x6 pixels, for a total of 120*3 = 360 bits, for a grand total of 486 bits, which can be encapsulated in 61 bytes. This may be padded to 64 bytes, which can be represented as sixteen 32-bit unsigned integers, eight 64-bit unsigned integers, or four 128-bit integers. These formats may be desired for the optimal use of special CPU instructions and / or word alignment. For instance, many processors support a machine instruction for counting the number of bits set in various sizes of integers, and many C compilers support intrinsics to generate these instructions.

[0301] For an analysis resulting in 500 key points, the storage required to store the descriptions would be about 30.5Kb, vs. 256K for the original image, or 726K for the raw LiDAR scans, an approximately 95.8% reduction.

[0302] Comparing Descriptors

[0303] Comparing descriptors may be done by computing the Hamming distance of the bit vectors. This can be done very quickly by performing a bitwise “exclusive or” (XOR) of the two bit vectors, and counting the number of 1 bits remaining. Counting the number of set bits in a byte can be done with a lookup in a precomputed table, or using a special intrinsic if the native architecture and compiler support it. This results in a “similarity” value in the range 0 to 486, where 0 is a perfect match, and 486 is no match.

[0304] FIG. 34 illustrates calculating similarity of two small example descriptors, according to some embodiments of the disclosed technologies.

[0305] The number of set bits n can then be mapped to a quality value q between 0.0 (no match), and 1.0 (perfect match), with:(486-n) q ~ 486

[0307] Matching Channel A Fingerprints

[0308] To compute the match between two images, every key point in one image is matched with every key point in the other image by comparing the descriptors. Typically, the two best matches for a given key point are recorded and the qualities compared using a Lowe’s ratio; for example, the quality of the second best match should be less than 70% of the best match for the first best match to be considered to be “The Match”. If a sufficient number of key point matches of sufficient quality are found, then it is a candidate. In some embodiments, the best matching image (which may be filtered by Lowe’s ratio) is considered to be the definitive match, pending cross-checking with the other channels.

[0309] In some embodiments, geometric locality and perspective may be exploited. Since motion is almost exclusively forward and backward along a track, two correctly matching key points between two images should be very close to the same line radiating from a point typically near the center of the image (the ’’vanishing point”), accounting for distortion of the spherical to planar mapping.

[0310] FIG. 35 illustrates paths of static features due to forward / backward motion. Note distortion due to spherical to planar mapping.

[0311] This can be implemented by calculating the angle of each keypoint around the vanishing point, and comparing the angles of candidate matches to see if they are similar. Note that the actual position of the vanishing point is determined by how the LiDAR is oriented on the vehicle. Ideally the vanishing point is at or very close to the center of the image. Otherwise, any offset needs to be accounted for.

[0312] CHANNEL B - FREQUENCY DOMAIN FINGERPRINT

[0313] FIG. 36 is a flowchart illustrating a process 3600 for a processing pipeline for creating Channel B fingerprints according to some embodiments of the disclosed technologies.

[0314] The elements of the process 3600 are presented in one arrangement. However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 3600 may include other elements in addition to those presented. For example, the process 3600 may include error-handling functions if exceptions occur, and the like.

[0315] Conversion to Real / lmaginary Floating Point

[0316] The process 3600 begins with the grayscale image output by the CLAHE process (FIG. 3, at 322). The grayscale image may be converted to floating point values, at 3602. The grayscale image may be converted to floating point values with thesource range of [0 ... 255] mapped to [0.0 ...1.0] into a “real” array of the same dimensions. A parallel “imaginary” array may be initialized with all values set to 0.0.

[0317] Fast Fourier Transform

[0318] Referring again to FIG. 36, the process 3600 may include performing a Fast Fourier Transform (FFT) on the real and imaginary arrays, at 3604. The FFT may be performed using an in-place Cooley-Tukey algorithm with bit-reversal permutations. Optimizations may be obtained from the use of power-of-two dimensions and table lookups for the necessary trigonometric functions.

[0319] Since the 2D FFT is separable, it may be computed in two passes. On the first pass, a 1 D FFT may be performed on each row. A second pass may perform the same 1 D FFT on each column. Alternatively, the first pass may be performed on the columns, and the second pass on the rows. As each row and column are also computed independently, each pass may be threaded.

[0320] Capturing Low Frequency Components

[0321] The process 3600 may include capturing low frequency components, at 3606. The results of the FFT are the real and imaginary components of the frequencies. The low frequency components can be found in the four corners. Some embodiments copy the four corners into new smaller arrays, shuffling them diagonally, to create an array with the low frequencies in the center. Note that the use of only the corners can be taken into consideration in the FFT algorithm. Therefore, it is not necessary to perform the FFT on all of the columns in the second pass, only those that cover the corners.

[0322] The resulting pair of arrays comprises the Channel B fingerprint. For example, a fingerprint of dimensions 64x16 for each array component combined has 2048 floating point values, for a total of 8192 bytes (approximately 1% of the original scan data size). Tests against real captured data may be used to determine the optimal dimensions, trading size / speed for accuracy.

[0323] FIG. 37 is a normalized grayscale image of the components of an example 64x16 fingerprint (real on the left, imaginary on the right), for illustration purposes only.

[0324] Matching Channel B Fingerprints

[0325] For each frequency component, there are two floating point values: real and imaginary. In some embodiments, these values are treated these as 2D Cartesian coordinates. To match two images, the Euclidean distance between the corresponding values in the two fingerprints may be computed. That is, two values at the same location from the real and imaginary arrays for given fingerprint may be treated as a "point" in 2D Cartesian space. The same "point" may be extracted from the same location in the other fingerprint, and the distance between those two "points" may be calculated. Then the standard deviation of these distances may be computed. This standard deviation is roughly proportional to the entropy of the difference of the images. The lower the standard deviation, the closer the match.

[0326] In practical use, the higher frequencies are more important for differentiating between very similar fingerprints. Therefore, in some embodiments, thematching skips over the pixels near the centers, which then acts as a high-pass filter on the original image.

[0327] FIG. 38 graphically illustrates the Channel B matching process, where:

[0328]

[0329]

[0330]

[0331] Fast Lookup

[0332] While comparison of a single fingerprint with another fingerprint is fast, it would take considerable time to compare a fingerprint with a large database of fingerprints. By taking advantage of the meaning of the components of the fingerprint, it’s possible to partition the database in order to drill down to a smaller subset of more likely candidates. The components near the center of the fingerprint reflect a crude representation of the original image. And the further away from the center, the more the components reflect fine detail. Some embodiments take components from the center, in order of increasing detail (for instance, spiraling out from the center point), and reduce the components to a reduced range of values. Then all of the fingerprints are sorted into a tree that can be quickly navigated to identify plausible matches.

[0333] The values of the components are floating point values roughly evenly distributed around zero. Some embodiments reduce each component to a single bit: for example, “1” if the value is greater that or equal to zero, and “0” otherwise. Then the fingerprint database may be partitioned into two groups: one with the center component reduced to “0”, and the other with the center component reduced to ‘“1”. Each group is then split into two subgroups using the reduced value of the next component, and so on until the subgroups are of a manageable size, or the component is too detailed. In practice, this simple binary system would have a weakness where an important component’s float value is close to 0, and an actual match might have the component also very close to zero, and thus actually close to the scanned value, but with the opposite sign.

[0334] To address this weakness, some embodiments reduce to three possible values, “near zero”, “positive not near zero”, and “negative not near zero”, resulting in a ternary tree rather than a binary tree. For a given component value, more than one branch of the tree might be searched depending upon its distance from zero, and secondarily the sign.

[0335] In some embodiments, the “near zero” boundary value is set to “1”; thus the “positive not near zero” branch would contain fingerprints with the reduced component value greater than 1 inclusive, the “negative not near zero” branch would have the fingerprint with the reduced component value less than -1 inclusive, and the “near zero” branch would contain the fingerprints with the reduced component value between -1 and 1.

[0336] Now when a new component is to be matched, the value would determine which branch or branches might be searched; a “buffer” zone may be set around the “near zero” boundaries. For example, buffer may be set to 0.5. Thus, a reduced component value between -0.5 and +0.5 might only check the “near zero” branch. A value greater than 1.5 might check only the “positive not near zero” branch. Values between 0.5 and 1.5 might check both the “positive not near zero” and “near zero” branches. The values for the “near zero” boundary and “buffer” may be determined by analysis of actual survey data when the tree is created.

[0337] CHANNEL C - IMAGE FEATURES FINGERPRINT (ORB)

[0338] FIG. 39 is a flowchart illustrating a process 3900 for a processing pipeline for creating Channel C fingerprints according to some embodiments of the disclosed technologies.

[0339] The elements of the process 3900 are presented in one arrangement. However, it should be understood that one or more elements of the process may be performed in a different order, in parallel, omitted entirely, and the like. Furthermore, the process 3900 may include other elements in addition to those presented. For example, the process 3900 may include error-handling functions if exceptions occur, and the like.

[0340] Conversion to Real / lmaginary Floating Point

[0341] ORB is a keypoint fingerprint system much like AKAZE, except that it uses a completely different method for identifying keypoints and creating descriptors.Like AKAZE, it ultimately produces a binary descriptor and uses a Hamming distance for matching.

[0342] ORB (“Oriented Fast and Rotated Brief’) combines two algorithms: “Features from accelerated segment test” (FAST), a corner-based test for the identification of keypoints, and “Binary Robust Independent Elementary Features” (BRIEF) for the creation of descriptors around those keypoints.

[0343] Some embodiments use a simplified version of BRIEF that does not account for rotation, similar to the “upright” MLDB descriptors used by AKAZE, and for the same reason: the vehicle does not materially rotate.

[0344] The process begins with the 8-bit grayscale image output by the CLAHE process (FIG. 3, at 322). Referring to FIG. 39, the process 3900 may include creating a mask, and insetting (i.e., “eroding”) the mask, at 3902 and 3904. These processes are similar to those processes for Channel A, but the erosion is by a greater amount (e.g., 15 pixels) to account for the size of the area around each keypoint used for the BRIEF descriptor (i.e., 31x31 pixels).

[0345] FAST Feature Detection

[0346] The process 3900 may include FAST feature detection, at 3906. To detect corners, FAST walks through each pixel, and compares the value of that pixel with a group of 16 pixels defined by a Bresenham (i.e., pixel-aligned) circle of radius 3.4 pixels. The pixel is considered to be a corner if a continuous sequence of at least nine of the 16 pixels are all of lower or higher intensity (beyond a specified threshold) than the candidate center pixel.

[0347] FIG. 40 is an image with highlighting for a center pixel (green) and16 surrounding pixels (red) as defined by Bresenham.

[0348] Note that due to the size of the circle, pixels next to each other can frequently be considered as the same corner. FAST has a scoring algorithm for the “quality” of the corner, based on the divergence of the values of circle pixels from the threshold, so it chooses the pixel with the best score in any 3x3 neighborhood to represent that corner. When a corner is identified, it is then compared with the eroded mask so that corners of the actual scan area itself are eliminated from consideration. The number of keypoints captured for any given image can be controlled by the value of the threshold. There are several optimization opportunities exploiting the geometry of the circle to quickly eliminate the candidacy of a point without sampling all of the circle points. For instance, if the values of two pixels directly opposite each other are both within the threshold of the center pixel, then the center cannot be a corner.

[0349] By processing a large number of domain-representative images using this method, some embodiments also build a ternary tree of the circle pixels to determine an optimal ordering of pixels to check to even more quickly reach a decision.

[0350] Creating BRIEF Descriptors

[0351] Referring again to FIG. 39, the process 3900 may include creating BRIEF descriptors, at 3908. To create the descriptors, a Gaussian blur with a fairly large kernel size (~4.0) may be applied to the image. For each keypoint, a 31x31 pixel region of the blurred image surrounding the keypoint may be sampled. The relative intensities of 256 pairs of points within the region may be compared. The pairs of points themselvesare pre-chosen. The points for each pair are selected from a Gaussian distribution based on the dimensions of the region. Each comparison is reduced to a bit, where a value of “1” indicates the intensity of the pixel at the second point is greater than the intensity of the pixel at the first point, and a value of “0” indicates otherwise). Processing all the points results in a descriptor of 32 8-bit values.

[0352] FIG. 41 illustrates example pairs of points used to sample a 31x31 region.

[0353] Comparing BRIEF Descriptors

[0354] Referring again to FIG. 39, the process 3900 may include comparing BRIEF descriptors, at 3910. The comparison of descriptors may use the same Hamming distance metric as the Channel A (AKAZE) descriptors, except the descriptors are 256 bits (32 bytes) long instead of 486 bits long.

[0355] Matching Channel C Fingerprints

[0356] The comparison of fingerprints may be performed using the same method as Channel A (AKAZE) fingerprints.

[0357] Machine Learning

[0358] In some embodiments, the disclosed technologies may include the use of one or more trained machine learning models at one or more points in the described processes. Any machine learning models may be used. For example, the machine learning models and techniques may include classifiers, decision trees, neural networks, gradient boosting, and similar machine learning models and techniques.

[0359] The machine learning models may be trained previously according to historical correspondences between inputs and corresponding outputs. Once the machine learning models have been trained, new inputs may be applied to the trained machine learning model as inputs. In response, the machine learning models may provide the desired outputs.

[0360] Some embodiments include the training of the machine learning models. The training may be supervised, unsupervised, or a combination thereof, and may continue between operations for the lifetime of the system. The training may include creating a training set that includes the input parameters and corresponding assessments described above.

[0361] The training may include one or more second stages. A second stage may follow the training and use of the trained machine learning models, and may include creating a second training set, and training the trained machine learning models using the second training set. The second training set may include the inputs applied to the machine learning models, and the corresponding outputs generated by the machine learning models, during actual use of the machine learning models.

[0362] The second training stage may include identifying erroneous assessments generated by the machine learning model, and adding the identified erroneous assessments to the second training set. Creating the second training set may also include adding the inputs corresponding to the identified erroneous assessments to the second training set.

[0363] As a particular example, the machine learning models may be used to populate the fields of the repair estimate data structure. In this example, the training data set(s) may include correspondences between field values and field identifiers of the repair estimate data structure.

[0364] It should be noted that the terms "optimize," "optimal" and the like as used herein can be used to mean making or achieving performance as effective or perfect as possible. However, as one of ordinary skill in the art reading this document will recognize, perfection cannot always be achieved. Accordingly, these terms can also encompass making or achieving performance as effective as possible under the given circumstances, or making or achieving performance better than that which can be achieved with other settings or parameters.

[0365] Context Diagram

[0366] FIG. 42 is a high-level block diagram of a vehicle-borne system for determining locations of a vehicle along a pathway according to some embodiments of the disclosed technologies. Referring to FIG. 42, the system includes a vehicle 4202 traveling on a vehicle pathway 4204. A LiDAR device 4206 is disposed at one end of the vehicle 4202 with a view of the vehicle pathway 4204. A RADAR device 4208 is disposed at one end of the vehicle 4202 with a view of the vehicle pathway 4204. The devices 4206, 4208 may be disposed at the front of the vehicle 4202 with a view of the vehicle pathway 4204 ahead of the vehicle 4202, at the rear of the vehicle 4202 with a view of the vehicle pathway 4204 behind the vehicle 4202, or both, or any combination thereof.

[0367] A computer system 4210 may be disposed on the vehicle 4202. Data collected by the devices 4206, 4208 may be fed to the computer system 4210 by one or more communications channels 4212. The computer system 4210 may be implemented as described with reference to FIG. 52 below.

[0368] Development / Prototyping

[0369] Test Robot (“Robobug”)

[0370] The robot platform used for testing is built around a Superdroid HK1000-V1 DM4-E (Model 60511) 4 Wheel Drive, All Terrain, Enclosed Robot Platform. It has four independent wheelchair motors with optical encoders, two RoboteQ MDC2460 Motor Controllers, two 12.8V 42 Ah LiFePO4 batteries, an NVIDIA Jetson computer and Wifi Radio. It weighs approximately 190 lbs., can carry a payload of up to 200 lbs., and can reach a top speed of over 5mph.

[0371] FIG. 43 shows the Superdroid HK-1000 robot platform in stock (i.e. , unmodified) condition.

[0372] For testing purposes, the NVIDIA Jetson was bypassed in order to control the robot with an Arduino, and several sensors were added:

[0373] • Four ultrasonic sensors for obstruction detection (two forward, two back);

[0374] • Two Lightware SF-11 single-beam laser range finders for ground truth for surveys and validation

[0375] • OmniPreSense OPS243-C FMCW / Doppler RADAR;

[0376] • RoboteQ MGSW16000 Magnetic Line Following Sensor withGyroscope;

[0377] • Raspberry Pi Zero W with OLED display for remote commands; and

[0378] • Kill switch.

[0379] In addition, a motor-controlled rotating platform was added to hold the RS-M1 LiDAR with its interface board, and an SBC built around a Raspberry Pi 4 compute module.

[0380] FIG. 44 shows the Superdroid HK-1000 robot platform as modified.

[0381] A Microsoft Windows Surface tablet with a custom application is used to send high-level commands to the robot over Wifi to the Raspberry Pi Zero W.

[0382] The robot is programmed to operate autonomously by following a magnetic strip. When it reaches an end of the strip, it performs a 180° turn in place, reacquires the magnetic strip and proceeds to the other end. It is also programmed to rotate the carousel holding the LiDAR, so that it always faces the same direction independent of the direction of travel.

[0383] The robot is programmed to halt and produce an audible alert if it detects an obstruction with the ultrasonic sensors, and will resume operation when the obstacle is removed.

[0384] Test Application

[0385] The test application was implemented in C++ on top of the Qt application framework. The application uses a modified version of the data pipeline as described above, since it is distributed between the robot and a laptop; the robot captures the raw packet data and encodes in a compressed format for transmission to the laptop. The robot can, of course, run the entire pipeline, but that’s inconvenient for development; in fact, the same Qt application is run on both devices, but is run headless on the robot with the functionality limited by a build-time switch.

[0386] On the robot, the packets are captured, and an “image” is plotted much like the MISRA pipeline but with each pixel encoded in a three channel image as RGB, with the raw 16-bit distance encoded across two channels, and the intensity in the third channel. The image data is then compressed, broken into UDP-sized packets, and transmitted as sequenced packets to the laptop. The laptop captures the packets, concatenates the payloads together, decompresses them, and formats the data back into a form suitable for the rest of the pipeline.

[0387] The robot also transmits its movement status and the readings from the laser range finders over UDP.

[0388] The test system is much more general than the MISRA pipeline - it splits the algorithm into more, but simpler, steps, does not have most of the optimizations, and uses off-the shelf libraries, such as OpenCV, to implement some of the functionality. It also supports a Ul for visualization of the results.

[0389] The laptop’s application operates in two modes: Sequential Scan Matching Mode and Fingerprint matching mode.

[0390] Sequential Scan Matching Mode

[0391] In Sequential Scan Matching Mode, the application continuously captures scans and compares them with a scan that is captured every 20th frame. As more scans come in, the new scans generally diverge from the saved scan as the robot moves or the environment changes. This divergence should be manifested as lower match values in the matching algorithms in both channels.

[0392] FIG. 45 shows results for Sequential Scan Matching Mode. In the below image, the left column shows, in order, the diffusion image, the CLAHE image, the matching of Channel A data showing the correspondence of individual key points between the two images, and the magnitude of Channel B matching (white because of a high quality match). The upper graph on the right shows the history of Channel A matching - the low point indicates a poor match.

[0393] The lower graph shows the history of Channel B matches, in this case, a high value indicates a poor match.

[0394] Note that both channels reported the same poor match at the same time, on the same set of images, indicating that the two channels agree.

[0395] Fingerprint matching mode

[0396] In Fingerprint matching mode, the application captures frames annotated with “ground truth” offsets generated from the laser range finders. If the survey database does not contain a scan from that location, the fingerprints are saved (this is the “survey” mode). If the database *does* contain a scan from that location, the fingerprints are compared with all database entries and the resulting “best match” is indicated (this is the “positioning” mode).

[0397] FIG. 46 shows results for Fingerprint matching mode. In the drawing, the text data block shows the behavior of Channel A: the computed offset using Fingerprint matching, the measured offset produced by the laser range finder, and the difference, all in meters. In this case, the computed position is 21 cm from the actual position, within the 50 cm expected by a survey taken with 1m spacing.

[0398] The upper graph shows the history of the quality of “best matches”. The second graph plots the history of the difference between the actual and measured positions; green indicates values within the requirements. The dead spots in these two graphs are when the robot is turning around, and positioning is not performed.

[0399] The bottom graph shows how well the current position matched against *every* position in the survey database for both Channel A (green) and Channel B (blue). Note that both peak at the same location.

[0400] The brown part of the graph is a rough estimate of the complexity of the fingerprints, and can be used to indicate where additional geometry for dazzling would be helpful to improve fingerprint quality. In this case the section of the left with low values is a relatively featureless wall with a file cabinet - at least within the LiDAR’s viewpoint.

[0401] The black gap on the right is a position that has not yet been captured by the survey mode.

[0402] At any time, the “Reset Database” button can be pressed, and Test Application will resurvey the area.

[0403] Deployment

[0404] Deployment of the system requires a survey step to capture the fingerprints for the railway, and the runtime to position the vehicle in production use.

[0405] Survey

[0406] A survey involves capturing a set of point clouds annotated by metadata of the absolute position, so that the fingerprints can be generated and stored in a database.

[0407] The survey may be conducted in different ways.

[0408] • A “live” survey, where the vehicle captures and converts the point clouds into fingerprints in real-time and stores them in a database. The database is then optimized off line.

[0409] • A registered point cloud with absolute coordinates may be used to create synthetic point clouds from a given position, which may in turn be converted to fingerprints. However, the point cloud would have to have intensity data consistent with the LiDAR used at runtime.

[0410] • Network capture of live scans with metadata may be converted offline. This may be done by a geometry train. However, it would be highly preferable for the LiDAR sensor to be positioned as close as possible to the position it would be on the target vehicle.

[0411] FIG. 47 is a block diagram of a system used when a survey is taken, creating the fingerprint databases, according to some embodiments of the disclosed technologies. LiDAR scans are captured and processed into fingerprints. The other sensors are used to annotate the fingerprints with the physical locations where they were captured.

[0412] FIG. 48 is a block diagram of the system during production use, according to some embodiments of the disclosed technologies. This system may use the fingerprint databases created by FIG. 47, producing new fingerprints using the same algorithms and matching them against the fingerprint databases. When the matching fingerprint is found, the annotations may be used to then report the physical location where the matching fingerprint was captured during the survey.

[0413] Section A: Robosense M1 LiDAR protocol

[0414] Table 1 and FIG. 49 illustrate the MSOP Packet Structure.

[0415] Table 1

[0416] FIG. 50 illustrates the MSOP Header Structure.

[0417] FIG. 51 illustrates the MSOP Data Block Structure.

[0418] Section B: Threading Architecture

[0419] Several of the disclosed functions are separable, which permits the use of CPU threads for performance. Various embodiments implement a very similar architecture across the various functions for which this has been exploited.

[0420] The main thread may break up the task into smaller subtasks, for instance, a row, column, or tile. It adds the relevant information for a subtask (i.e.. which row, column or tile) to a mutex-protected queue, and records the number of subtasks needed to be completed. It also keeps a mutex-protected counter of the number of subtasks actually completed.

[0421] The first time a function is called, a one-time initialization may be performed. A fixed number of threads (typically configured at build-time, based on the number of CPU cores) are spawned to watch the queue for subtasks. When a subtaskbecomes available, it is removed from the queue, the subtask is performed, and the count of completed subtasks is incremented.

[0422] When the subtask completed counter matches the counter of subtasks expected, the function is complete.

[0423] The mutexes are initialized in a way such that the subtask threads are actively blocked when there is no work available, by having the main thread lock the mutex on the subtask queue until a subtask has been added.

[0424] Section C: GPU Architecture

[0425] Embodiments that have compatible GPU hardware may exploit the high-performance, parallel computing behavior of “compute shaders”. Similar to the threaded CPU architecture, the main task is split up into small subtasks that can be performed in parallel, these subtasks and associated data are uploaded into the GPU memory, the subtasks are performed, the CPU is notified when they are completed, and the results downloaded from the GPU.

[0426] Some embodiments use the Vulkan API to execute compute shaders implemented in OpenGL Shading Language (GLSL). GLSL is a C-like language specifically tailored for the data types, parallelism, and protocols used by GPUs. GLSL is typically compiled into a portable format called Standard Portable Intermediate Representation (SPIR-V), which creates a requirement that the build machine must support the local compilation of GLSL to SPIR-V in addition to its other build requirements.

[0427] A tradeoff for any given function is the time it takes to load and offload data between the CPU and GPU, and the amount of actual GPU resources available for a system.

[0428] Section D: Closed form solution for linear systems of two variables

[0429] When refining the positions of keypoints, the AKAZE algorithm requires solving a system of two linear equations of two free variables. All published AKAZE implementations use a generic linear system solver to do this, such as provided by the OpenCV and Eigen math libraries. Calling those functions incurs some overhead. However, since this is a simple system, a closed form solution is available and much faster than the iterative methods used by these libraries; note that some libraries check for this simple case and short circuit using a closed form, but the function call overhead is still present. The method documented here can be placed inline. The problem is to solve the following pair of linear equations for free variables x and y with constants a, b, c, d, e, and f:

[0430] ax + by = c

[0431] dx + ey = f

[0432] Each of these equations can be interpreted geometrically as an infinite straight line in 2D Cartesian space, with the solution as the point of intersection. This is a common secondary school algebra homework problem, and solutions typicallyexploit finding common factors of simple integer constants. In our case, these constants are fairly uncorrelated real numbers.

[0433] These equations can be represented as an augmented matrix:

[0434]

[0435] Normally this would be fed into a generic algorithm transforming it into row echelon form using Gaussian elimination to get:

[0436]

[0437] Instead, we can split it into three square matrices, one for each combination of two columns:

[00438]

[0439] Then one can compute:

[0440]

[0441] In a more direct form:[L0v0V4-T4-r2«.]j

[0443] Note that the equations have no solution if det A -> (ae-bd) is zero; geometrically, this corresponds to the lines being parallel, thus either having nointersection, or being the same line. This case would need to be checked explicitly in code before performing the division.

[0444] Example Computer System

[0445] FIG. 52 depicts a block diagram of an example computer system 5200 in which embodiments described herein may be implemented. The computer system 5200 includes a bus 5202 or other communication mechanism for communicating information, one or more hardware processors 5204 coupled with bus 5202 for processing information. Hardware processor(s) 5204 may be, for example, one or more general purpose microprocessors.

[0446] The computer system 5200 also includes a main memory 5206, such as a random access memory (RAM), cache and / or other dynamic storage devices, coupled to bus 5202 for storing information and instructions to be executed by processor 5204. Main memory 5206 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 5204. Such instructions, when stored in storage media accessible to processor 5204, render computer system 5200 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0447] The computer system 5200 further includes a read only memory (ROM) 5208 or other static storage device coupled to bus 5202 for storing static information and instructions for processor 5204. A storage device 5210, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), etc., is provided and coupled to bus 5202 for storing information and instructions.

[0448] The computer system 5200 may be coupled via bus 5202 to a display 5212, such as a liquid crystal display (LCD) (or touch screen), for displaying information to a computer user. An input device 5214, including alphanumeric and other keys, is coupled to bus 5202 for communicating information and command selections to processor 5204. Another type of user input device is cursor control 5216, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 5204 and for controlling cursor movement on display 5212. In some embodiments, the same direction information and command selections as cursor control may be implemented via receiving touches on a touch screen without a cursor.

[0449] The computing system 5200 may include a user interface module to implement a GUI that may be stored in a mass storage device as executable software codes that are executed by the computing device(s). This and other modules may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0450] In general, the word “component,” “engine,” “system,” “database,” data store,” and the like, as used herein, can refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, C or C++. A software component may be compiled and linked into an executable program, installed in adynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software components may be callable from other components or from themselves, and / or may be invoked in response to detected events or interrupts. Software components configured for execution on computing devices may be provided or encoded on a computer readable or machine readable medium, such as a compact disc, digital video disc, flash drive, magnetic disc, or any other tangible medium, or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression or decryption prior to execution). Such software code may be stored, partially or fully, on a memory device of the executing computing device, for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware components may be comprised of connected logic units, such as gates and flip-flops, and / or may be comprised of programmable units, such as programmable gate arrays or processors.

[0451] The computer system 5200 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic which in combination with the computer system causes or programs computer system 5200 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 5200 in response to processor(s) 5204 executing one or more sequences of one or more instructions contained in main memory 5206. Such instructions may be read into main memory 5206 from another storage medium, such as storage device 5210. Execution of the sequences of instructions contained in main memory 5206 causes processor(s) 5204 to perform theprocess steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0452] The term “non-transitory media,” and similar terms, as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such non-transitory media may comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 5210. Volatile media includes dynamic memory, such as main memory 5206. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, and networked versions of the same.

[0453] Non-transitory media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between non-transitory media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 5202. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0454] The computer system 5200 also includes a communication interface 5218 coupled to bus 5202. Network interface 5218 provides a two-way data communication coupling to one or more network links that are connected to one or more local networks. For example, communication interface 5218 may be an integratedservices digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, network interface 5218 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or a WAN component to communicate with a WAN). Wireless links may also be implemented. In any such implementation, network interface 5218 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0455] A network link typically provides data communication through one or more networks to other data devices. For example, a network link may provide a connection through local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet.” Local network and Internet both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link and through communication interface 5218, which carry the digital data to and from computer system 5200, are example forms of transmission media.

[0456] The computer system 5200 can send messages and receive data, including program code, through the network(s), network link and communication interface 5218. In the Internet example, a server might transmit a requested code for an application program through the Internet, the ISP, the local network and the communication interface 5218.

[0457] The received code may be executed by processor 5204 as it is received, and / or stored in storage device 5210, or other non-volatile storage for later execution.

[0458] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code components executed by one or more computer systems or computer processors comprising computer hardware. For example, a method bay be referred to as a "computer-implemented" method. The one or more computer systems or computer processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). The processes and algorithms may be implemented partially or wholly in application-specific circuitry. The various features and processes described above may be used independently of one another, or may be combined in various ways. Different combinations and sub- combinations are intended to fall within the scope of this disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate, or may be performed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The performance of certain of the operations or processes may be distributed among computer systems or computers processors, not only residing within a single machine, but deployed across a number of machines.

[0459] As used herein, a circuit might be implemented utilizing any form of hardware, or a combination of hardware and software. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routines or other mechanisms might be implemented to make up a circuit. In implementation, the various circuits described herein might be implemented as discrete circuits or the functions and features described can be shared in part or in total among one or more circuits. Even though various features or elements of functionality may be individually described or claimed as separate circuits, these features and functionality can be shared among one or more common circuits, and such description shall not require or imply that separate circuits are required to implement such features or functionality. Where a circuit is implemented in whole or in part using software, such software can be implemented to operate with a computing or processing system capable of carrying out the functionality described with respect thereto, such as computer system 5200.

[0460] As used herein, the term “or” may be construed in either an inclusive or exclusive sense. Moreover, the description of resources, operations, or structures in the singular shall not be read to exclude the plural. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps.

[0461] Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open ended as opposed tolimiting. Adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known,” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.

[0462] The foregoing description of the present disclosure has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Many modifications and variations will be apparent to the practitioner skilled in the art. The modifications and variations include any relevant combination of the disclosed features. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical application, thereby enabling others skilled in the art to understand the disclosure for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the disclosure be defined by the following claims and their equivalents.

Claims

AMENDED CLAIMS received by the International Bureau on 10 October 2025 (10.10.2025)

1. A system, comprising: one or more hardware processors; and one or more non-transitory machine-readable storage media encoded with instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

2. The system of claim 1 , wherein generating the fingerprints comprises: creating a nonlinear scale space using the floating-point numbers; and detecting features in the nonlinear scale space.

3. The system of claim 2, wherein creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images.

4. The system of claim 3, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks.

5. The system of claim 4, wherein generating the fingerprints further comprises: refining the retained features.

6. The system of claim 5, wherein generating the fingerprints further comprises: creating descriptors for the refined features; and comparing the descriptors.

7. The system of claim 1 , wherein: the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

8. One or more non-transitory machine-readable storage media encoded with instructions that, when executed by one or more hardware processors of a computing system, cause the computing system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway,selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

9. The one or more non-transitory machine-readable storage media of claim 8, wherein generating the fingerprints comprises: creating a nonlinear scale space using the floating-point numbers; and detecting features in the nonlinear scale space.

10. The one or more non-transitory machine-readable storage media of claim 9, wherein creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images.

11. The one or more non-transitory machine-readable storage media of claim 10, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks.

12. The one or more non-transitory machine-readable storage media of claim 11 , wherein generating the fingerprints further comprises: refining the retained features.

13. The one or more non-transitory machine-readable storage media of claim 12, wherein generating the fingerprints further comprises:creating descriptors for the refined features; and comparing the descriptors.

14. The one or more non-transitory machine-readable storage media of claim 8, wherein: the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

15. A computer-implemented method comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using the Accelerated Kernelized Adaptive Zernike moments (AKAZE) feature extraction algorithm, and comparing the fingerprints.

16. The computer-implemented method of claim 15, wherein generating the fingerprints comprises: creating a nonlinear scale space using the floating-point numbers; and detecting features in the nonlinear scale space.

17. The computer-implemented method of claim 16, wherein creating a nonlinear scale space comprises: applying a 2D Gaussian blur to the first and second 2D images.

18. The computer-implemented method of claim 17, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; retaining the features that are within the eroded masks; and refining the retained features.

19. The computer-implemented method of claim 18, wherein generating the fingerprints further comprises: creating descriptors for the refined features; and comparing the descriptors.

20. The computer-implemented method of claim 15, wherein: the LiDAR data comprises a point cloud of points each having a distance value, an intensity value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values and intensity values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

21. A system, comprising: one or more hardware processors; and one or more non-transitory machine-readable storage media encoded with instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

22. The system of claim 21 , wherein generating fingerprints from the floating-point numbers using a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array.

23. The system of claim 22, wherein generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array.

24. The system of claim 23, wherein comparing the fingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images.

25. The system of claim 24, wherein comparing the third 2D images comprises: calculating Euclidean distances between corresponding values in the third 2D images; and calculating standard deviations of the Euclidean distances.

26. The system of claim 25, wherein selecting one of the second 2D images based on the comparing comprises: selecting the second 2D image having the lowest standard deviation.

27. The system of claim 21 , wherein: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

28. One or more non-transitory machine-readable storage media encoded with instructions that, when executed by one or more hardware processors of a computing system, cause the computing system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, anddetermining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers, generating fingerprints from the floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

29. The one or more non-transitory machine-readable storage media of claim 28, wherein generating fingerprints from the floating-point numbers using a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array.

30. The one or more non-transitory machine-readable storage media of claim 29, wherein generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array.

31. The one or more non-transitory machine-readable storage media of claim 30, wherein comparing the fingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images.

32. The one or more non-transitory machine-readable storage media of claim 31 , wherein comparing the third 2D images comprises:calculating Euclidean distances between corresponding values in the third 2D images; and calculating standard deviations of the Euclidean distances.

33. The one or more non-transitory machine-readable storage media of claim 31 , wherein comparing the third 2D images comprises: selecting the second 2D image having the lowest standard deviation.

34. The one or more non-transitory machine-readable storage media of claim 28, wherein: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

35. A computer-implemented method comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected second 2D image was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: converting pixel values of the first 2D image and the multiple second 2D images to floating-point numbers,generating fingerprints from the floating-point numbers using a Fast Fourier Transform (FFT), and comparing the fingerprints.

36. The computer-implemented method of claim 35, wherein generating fingerprints from the floating-point numbers using a FFT comprises: generating first arrays of the floating-point numbers; generating second arrays having the same dimensions as the first arrays, and having all values set to zero; generating a third array by performing a FFT on the first array; and generating a fourth array by performing a FFT on the second array.

37. The computer-implemented method of claim 36, wherein generating the fingerprints comprises: generating a fifth array by copying corners of the third array and shuffling them diagonally such that the lower frequencies are in the center of the fifth array; and generating a sixth array by copying corners of the fourth array and shuffling them diagonally such that the lower frequencies are in the center of the array.

38. The computer-implemented method of claim 37, wherein comparing the fingerprints comprises: treating corresponding values in corresponding fifth and sixth arrays as Cartesian coordinates of third 2D images; and comparing the third 2D images.

39. The computer-implemented method of claim 38, wherein comparing the third 2D images comprises: calculating Euclidean distances between corresponding values in the third 2D images; calculating standard deviations of the Euclidean distances; and selecting the second 2D image having the lowest standard deviation.

40. The computer-implemented method of claim 35, wherein: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises:converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

41. A system, comprising: one or more hardware processors; and one or more non-transitory machine-readable storage media encoded with instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: generating fingerprints from [[the]] pixel values of the first 2D image and the multiple second 2D images using the Oriented Fast and Rotated Brief (ORB) feature extraction algorithm, and comparing the fingerprints.

42. The system of claim 41 , wherein generating the fingerprints comprises: detecting features using the floating-point numbers.

43. The system of claim 42, wherein generating the fingerprints comprises: detecting the features using the floating-point numbers and the Features From Accelerated Segment Test (FAST).

44. The system of claim 42, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks.

45. The system of claim 44, wherein generating the fingerprints further comprises: creating descriptors for the retained features; and comparing the descriptors.

46. The system of claim 44, wherein generating the fingerprints further comprises: creating the descriptors for the retained features using the Binary Robust Independent Elementary Features” (BRIEF) algorithm.

47. The system of claim 41 , wherein generating the fingerprints further comprises: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

48. One or more non-transitory machine-readable storage media encoded with instructions that, when executed by one or more hardware processors of a computing system, cause the computing system to perform operations comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway,selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: generating fingerprints from [[the]] pixel values of the first 2D image and the multiple second 2D images using the Oriented Fast and Rotated Brief (ORB) feature extraction algorithm, and comparing the fingerprints.

49. The one or more non-transitory machine-readable storage media of claim 48 [[49]], wherein generating the fingerprints comprises: detecting features using the floating-point numbers.

50. The one or more non-transitory machine-readable storage media of claim 49 [[50]], wherein generating the fingerprints comprises: detecting the features using the floating-point numbers and the Features From Accelerated Segment Test (FAST).

51. The one or more non-transitory machine-readable storage media of claim 50, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks.

52. The one or more non-transitory machine-readable storage media of claim 51 , wherein generating the fingerprints further comprises: creating descriptors for the retained features; and comparing the descriptors.

53. The one or more non-transitory machine-readable storage media of claim 51 , wherein generating the fingerprints further comprises: creating the descriptors for the retained features using the Binary Robust Independent Elementary Features” (BRIEF) algorithm.

54. The one or more non-transitory machine-readable storage media of claim 48, wherein generating the fingerprints further comprises: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

55. A computer-implemented method comprising: receiving a 3D image comprising LiDAR data, the 3D image captured from a vehicle on a pathway; transforming the 3D image into a first 2D image; and determining a location of the vehicle along the pathway, comprising: comparing the first 2D image to multiple second 2D images each captured at a respective known location along the pathway, selecting one or more of the second 2D images based on the comparing, and determining the location of the vehicle along the pathway based on the known location where the selected one or more of the second 2D images was captured; wherein comparing the first 2D image to the multiple second 2D images comprises: generating fingerprints from [[the]] pixel values of the first 2D image and the multiple second 2D images using the Oriented Fast and Rotated Brief (ORB) feature extraction algorithm, and comparing the fingerprints.

56. The computer-implemented method of claim 55, wherein generating the fingerprints comprises: detecting features using the floating-point numbers.

57. The computer-implemented method of claim 56, wherein generating the fingerprints comprises:detecting the features using the floating-point numbers and the Features From Accelerated Segment Test (FAST).

58. The computer-implemented method of claim 57, wherein generating the fingerprints further comprises: creating masks for the first 2D image and the multiple second 2D images; eroding edges of the mask by multiple pixels; and retaining the features that are within the eroded masks.

59. The computer-implemented method of claim 58, wherein generating the fingerprints further comprises: creating descriptors for the retained features; and comparing the descriptors.

60. The computer-implemented method of claim 55, wherein generating the fingerprints further comprises: the LiDAR data comprises a point cloud of points each having a distance value, a yaw value, and a pitch value; and transforming the 3D image into a first 2D image comprises: converting the yaw values and the pitch values to image space coordinates, applying a periodic triangle wave function to the distance values, and mapping the resulting values to gray values, and plotting the gray values in the 2D image at the corresponding image space coordinates.

Images