A map containing covariance across multiresolution voxels

Abstract

Techniques for representing a scene or map based on statistical data of captured environmental data are described herein. In some cases, the data (e.g., covariance data, mean data, etc.) may be stored as a multi-resolution voxel space including multiple semantic layers. In some cases, each semantic layer may include multiple voxel grids with different resolutions. Voxel spaces at multiple resolutions may be merged to generate a combined scene based on the covariance of voxels detected at one or more resolutions.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 16 / 722,598, filed December 20, 2019, and entitled "MAPS COMPRISING COVARIANCES IN MULTI-RESOLUTION VOXELS," which is hereby incorporated by reference in its entirety.

[0002] Data may be captured in an environment and represented as a map of the environment. Often, the map may be used by a vehicle traveling through the environment, and the map may be used for a variety of purposes. In some cases, the environment may be represented as a two-dimensional map, while in other cases, the environment may be represented as a three-dimensional map. Furthermore, surfaces within the environment are often represented using multiple polygons or triangles.

[0003] The detailed description is set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features. [Brief description of the drawings]

[0004] [Figure 1] 1 is an exemplary diagram illustrating an example architecture of a multi-resolution voxel space as described herein; [Diagram 2] FIG. 2 is an exemplary pictorial diagram illustrating exemplary resolutions of semantic layers of a multi-resolution voxel space as described herein. [Diagram 3] FIG. 3 is an example diagram illustrating a first resolution for the multi-resolution voxel space of FIG. 2 described herein. [Figure 4]FIG. 3 is an example diagram illustrating a second resolution for the multi-resolution voxel space of FIG. 2 as described herein. [Diagram 5] FIG. 3 is an example diagram illustrating a third resolution for the multi-resolution voxel space of FIG. 2 as described herein. [Figure 6] FIG. 2 is an example process flow diagram illustrating an example data flow of a system configured to align data representing a physical environment with map data as described herein. [Figure 7] FIG. 1 is an example flow diagram illustrating an example process associated with generating a multi-resolution voxel space as described herein. [Figure 8] FIG. 11 is another flow diagram illustrating an example process for aligning a target multi-resolution voxel space with a reference multi-resolution voxel space as described herein. [Figure 9] FIG. 1 is a block diagram of an example system for implementing the multi-resolution voxel space alignment system described herein. [Figure 10] FIG. 5 is a pictorial diagram of the example multi-resolution voxel spaces of FIGS. 2-4 compared to a point cloud representation of the captured data described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Techniques described herein are directed to determining and / or using map data including a multi-resolution voxel space including voxels that store spatial means, covariances, and weights of point distributions of data representing a physical environment. The map data may include multiple voxel grids or layers that represent the physical environment at different resolutions or physical distances. As an example, each voxel layer may represent the physical environment at twice the resolution of the proceeding layer. That is, voxels in a first layer may represent a first volume (e.g., 10 cm×10 cm×10 cm), while voxels in a second layer may represent a second volume (e.g., 20 cm×20 cm×20 cm). Data associated with voxels in the multi-resolution voxel space may be represented as multiple covariance ellipsoids. A representation of the covariance ellipsoid may be generated based on the calculated mean and covariance values ​​of the data points associated with each voxel. In some cases, the voxel data may be associated with semantic information, such as, for example, classification and / or segmentation information, and data associated with a particular classification may be associated with a unique multi-resolution voxel space associated with the particular classification. In the example just described, each voxel covariance semantic layer may include data points associated with a unique semantic class (e.g., trees, vehicles, buildings, etc.) as a covariance ellipsoid.

[0006] In some cases, map data represented by a multi-resolution voxel space may be generated from data points representing a physical environment, such as the output of a lidar (light detection and ranging) system. As an example, a system may receive a plurality of lidar points or lidar data represented as a point cloud. The system may assign or otherwise associate the lidar points with voxels of a voxel grid having a first resolution (e.g., the voxel grid of the multi-resolution voxel space has the largest number of voxels) based at least in part on a local frame of reference of a vehicle (e.g., from which the system captures the lidar points). The system may determine statistical data associated with each voxel, such as, for example, a mean, centroid, covariance, and the like, of the aggregated data. The system may then merge or otherwise combine voxels (or data associated with voxels) of lower resolution grids when generating a higher level voxel grid. For example, voxels (or data associated with voxels) within a three-dimensional neighborhood (e.g., the number of voxels in the x, y, and z directions associated with the physical space of a higher level voxel) in a lower resolution grid may be merged when forming the next higher level voxel grid. In one particular example, voxels within a neighborhood are merged by taking a weighted sum of the individual Gaussians for each voxel of the lower resolution grid. In some cases, merging voxels at a lower resolution to form a higher resolution grid is not only computationally inexpensive, but also allows the lower resolution grid to help localize the lidar data with a reference frame.

[0007] In some implementations, the system may utilize the multi-resolution voxel space to align multiple scans of a physical environment to generate maps and scenes of the physical environment, as well as aid in localization of a vehicle within the map or scene. As an example, once a multi-resolution voxel space (e.g., a target multi-resolution voxel space) is generated for a specific scan or data set representing the physical environment, the system may align the generated multi-resolution voxel space with a multi-resolution voxel space representing the scene (e.g., a reference multi-resolution voxel space). In some cases, the alignment may be performed by substantially simultaneously finding correspondence between voxels in each resolution of the reference multi-resolution voxel space and the target multi-resolution voxel space. For example, for each voxel of a specific resolution of the target multi-resolution voxel space, the system may search among voxels within a threshold distance or within a threshold number of voxels (e.g., a neighborhood of voxels) that include the average target point in the corresponding specific resolution of the reference multi-resolution voxel space for the occupied voxel. In examples involving semantic layers, for each voxel of a specific resolution of each semantic layer in the target multiresolution voxel space, the system may search for a neighborhood of voxels that contain the average target point in the specific resolution of the corresponding semantic layer in the reference multiresolution voxel space.

[0008] Of the identified voxels in the neighborhood of the reference multiresolution voxel space, the system may select a voxel having a centroid close to the voxel of the target multiresolution voxel space. The system may then average the distribution of the selected voxels in the reference multiresolution voxel space with the voxels of the target covariance stack. The system may then perform a principal component analysis on the combined covariance matrix and select an eigenvalue (e.g., the smallest eigenvalue) as the matching normal vector for the two voxels. The system may then determine a residual (or error, etc.) for each of the matched voxels, which may, in at least some examples, be based at least in part on the matched normal vectors, and subsequently perform an optimization on all the residuals. The optimization may minimize the distance between the pair of voxel centroids. In this way, the merged voxel representing the two voxels may be located in a grid of locations that accurately represents both the covariances (e.g., of associated data) and weights of the original voxels. Of course, there are applications that do not require the above voxel merging: as a non-limiting example, a relative transformation between two voxel spaces may generally be used for localization without combining (merging) voxels.

[0009] During alignment, even when layers may be merged substantially simultaneously, the coarser resolution (e.g., a resolution corresponding to larger voxels) may result in matching before the finer resolution. Thus, matching at the coarser resolution may help bring the two multi-resolution voxel spaces into closer alignment so that the finer resolution can begin matching and complete the alignment process. In some cases, by merging the captured sensor data into a multi-resolution voxel space representing the environment, the vehicle may be able to initialize or localize a position within the environment more accurately and / or more quickly than a system utilizing traditional map data including polygons and / or meshes. Additionally, storing voxels in a multi-resolution voxel space may improve processing speed and throughput by storing data in a more easily indexable / searchable manner. For example, if the coarse resolution is acceptable for the actual task, the coarse layer may be loaded into memory to reduce the amount of data accessed and processed for the desired operation.

[0010] In some cases, a multi-resolution voxel space may represent an environment more accurately than conventional systems because each layer of the space provides a different resolution of detail about the environment. Thus, in some situations, having access to a more detailed representation of the physical environment may improve the overall safety of an autonomous vehicle.

[0011] FIG. 1 is an exemplary diagram illustrating an exemplary architecture 100 of a multi-resolution voxel space 102 described herein. In the current example, the multi-resolution voxel space 102 is formed from multiple semantic layers, illustrated as semantic layers 104, 106, and 108. Each of the semantic layers 104-108 may represent data for a specific semantic class or type. As an example, the first semantic layer 104 may include data representing trees, while the second semantic layer 106 may include data representing buildings. Thus, the multi-resolution voxel space 102 including the multiple semantic layers 104-108 may represent data from each semantic layer 104-108 as a full picture or map of a physical environment, illustrated below with respect to FIGS. 2-5. In some cases, some applications may require identification or awareness of only specific semantic classes, while other applications may require a detailed understanding of the entire physical environment. Segmenting the multi-resolution voxel space 102 into semantic layers 104-108 may improve processing speed in some applications by allowing each application to process only the appropriate class or type of data that corresponds to its environment.

[0012] Additionally, each of the semantic layers 104-108 may include one or more voxel grids, exemplified as voxel covariance grids 110, 112, and 114. Each of the voxel covariance grids 110-114 represents the same semantic data of the corresponding semantic layer 104-108, but at a different resolution. As an example, a first voxel covariance grid of the plurality of grids 110 may have voxels having a size of approximately 25 centimeters, while a second voxel covariance grid of the plurality of grids 110 has voxels having a size of approximately 16 meters. Thus, each voxel covariance grid of each of the plurality of grids 110-114 may have a different resolution or coarseness to aid in alignment and processing of the represented data of the physical environment. For example, some applications may require only a coarse, general understanding of the physical environment, while other applications require a detailed understanding of the physical environment, and each application may process a voxel grid at a desired or appropriate resolution to improve processing speed in some applications.

[0013] In some examples, such as those illustrated below with respect to Figures 2-5, data associated with voxels of a voxel covariance grid 110-114 of a multiresolution voxel space 102 may be represented by voxels that store covariance matrices, means, and weights that represent point distributions. In some cases, the voxels of the grids 110-114 may be visually represented as covariance ellipsoids. The covariance ellipsoids may be based at least in part on shape parameters of eigenvalue ratios of each voxel.

[0014] In the illustrated example, three semantic layers 104-108 and three sets of voxel covariance grids 110-114 are shown. However, it should be understood that the multi-resolution voxel space 102 may include any number of semantic layers, and each semantic layer may include any number of voxel covariance grids. In some implementations, the number of voxel covariance grids for each semantic layer may be the same, while in other implementations, the number of voxel covariance grids in each semantic layer may differ. As an example, some semantic classes, such as foliage (or pedestrians), may require more additional fine resolution voxel covariance grids than other semantic classes, such as buildings, and thus a semantic layer representing a pedestrian class may include more voxel covariance grids than a semantic layer representing a building class.

[0015] FIG. 2 is an exemplary pictorial diagram 200 illustrating exemplary resolutions 202, 204, and 206 of the semantic layer of a multi-resolution voxel space 208 described herein. In the current example, the resolutions are shown in two dimensions for illustrative purposes only, and it should be understood that any number of dimensions may be used (e.g., three dimensions representing a real-world three-dimensional physical space, etc.). In the current example, voxels in a first neighborhood 210 of a first resolution 202 are combined to form a voxel 212 of a second resolution 204. Similarly, voxels in a first neighborhood 214 of a second resolution 204 are combined to form a voxel 216 of a third resolution 206. The third resolution voxel 216, described below, may be formed based on a weighted sum of individual Gaussians from each of the voxels 218 and 220 in the neighborhood 214 to generate a single higher resolution voxel. It should be appreciated that determining a weighted sum of individual Gaussian distributions is computationally inexpensive in terms of processing resources and time, and therefore constructing the multi-resolution voxel space 208 may be done faster and with fewer processing resources than conventional systems.

[0016] In the current example, a two-dimensional 2x2 neighborhood is shown. However, it should be understood that the multi-resolution voxel space can be formed as a three-dimensional voxel grid representing the physical space, and that the neighborhood can have a variety of uniform sizes, e.g., 2x2x2, 3x3x3, 5x5x5, or non-uniform sizes, e.g., 2x3x4, 4x3x4, 5x1x3, etc. In one particular example, the neighborhood can have a 2x2x2 voxel size, with each higher resolution layer having half the number of voxels as the progressing lower layer.

[0017] 3-5 are exemplary diagrams illustrating multiple resolutions 202, 204, and 206 for the multi-resolution voxel space 208 of FIG. 2 described herein. In the current example, each of the semantic layers of the multi-resolution voxel space 208 is shown to generate a picture or map of the physical environment. By way of example, the multi-resolution voxel space 208 may be formed by merging or aligning multiple lidar scans of the physical environment captured by an autonomous vehicle. In the current example, the multi-resolution voxel space 208 may be zoomed in or out to show the physical environment at different resolutions 202, 204, and 206. By way of example, the resolution 202 shows voxels at a first or finest resolution. Thus, the resolution 202 of the multi-resolution voxel space 208 includes more voxels than each of the resolutions 204 or 206, and also includes the most detailed representation of the physical environment. Each progressing resolution 204 or 206 represents the physical environment in terms of voxels of the succeeding coarser resolution. By way of example, each voxel in the multiresolution voxel space of resolution 202 may represent an area of ​​25 cm, while each voxel in the multiresolution voxel space of resolution 206 may represent an area of ​​16 m.

[0018] In some cases, voxels associated with a unique semantic layer may be colored or textured to visually distinguish voxels associated with the two semantic layers from one another when viewing the multi-resolution voxel space 208. Additionally, it should be noted that because the data associated with each voxel is represented as an ellipsoid of covariance having a shape based at least in part on the voxel's eigenvalue ratios, shape parameters, and spatial statistics, the data illustrated in Figures 2-5 have shapes that substantially represent the real shapes of the corresponding objects.

[0019] In some examples, each higher resolution 300-500 of the multi-resolution voxel space 102 may have half the number of voxels as the preceding lower level resolution 200-400. As an example, if resolution 300 has voxels approximately 4 meters in size, then the voxels of resolution 400 may be approximately 8 meters in size (e.g., twice the size of the voxels of resolution 300). However, in other examples, the size and / or number of voxels of each resolution 200-500 may have other mathematical and / or arbitrary relationships.

[0020] In the current example, different semantic classes are indicated based on differences in the patterns or colors of the ellipses. For instance, ellipsoid 302 may correspond to foliage, ellipsoid 304 may correspond to a wall, structure or building, and ellipsoid 306 may correspond to ground covering, such as grass.

[0021] 6-8 are flow diagrams illustrating an example process associated with the multi-resolution voxel space of FIGS. 1-5. The process is illustrated as a collection of blocks in a logical flow diagram, which represent a sequence of some or all of the operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable mediums that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types.

[0022] The order of the described operations should not be construed as a limitation. Any of the described blocks can be combined in any order and / or in parallel to perform processes or alternative processes, and not all blocks need to be executed. For purposes of discussion, the processes herein are described with reference to the frameworks, architectures, and environments described in the examples herein, but the processes may be implemented in a wide variety of other frameworks, architectures, or environments.

[0023] 6 is an example process flow diagram 600 illustrating an example data flow for a system configured to align data representing a physical environment with a scene as described herein. In the illustrated example, the system may be configured such that the scene, as well as the data representing the environment, is stored as a multi-resolution voxel space. As discussed above, the multi-resolution voxel space may have multiple semantic layers, each of which includes multiple voxel grids representing voxels as ellipsoids of covariance at different resolutions.

[0024] In one particular example, a sensor system 602, such as a lidar, radar, sonar, infrared, camera, or other image capture device, may capture data representative of the physical environment surrounding the system. In some cases, the captured data may be a plurality of data points 604, such as a point cloud generated from the output of a lidar scan. In the example just described, the data points 604 may be received by a multi-resolution voxel space generation component 606.

[0025] The multi-resolution voxel space generation component 606 may be configured to generate a target multi-resolution voxel space 608 from the data points 604. In some cases, the multi-resolution voxel space generation component 606 may process the data points via classification and / or segmentation techniques. By way of example, the multi-resolution voxel space generation component 606 may assign types or classes to the data points to identify and classify the data points 604 with semantic labels using one or more neural networks (e.g., deep neural networks, convolutional neural networks, etc.), regression techniques, etc. In some cases, the semantic labels may include classes or entity types, such as, for example, vehicle, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc. In additional and / or alternative examples, the semantic labels may include one or more characteristics associated with the data points 604. For example, characteristics may include, but are not limited to, x position (global and / or local position), y position (global and / or local position), z position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), velocity of the entity, acceleration of the entity, rate of change of velocity and / or acceleration, range (magnitude) of the entity, etc.

[0026] In some examples, generating the target multi-resolution voxel space 608 may include associating data associated with static objects (e.g., buildings, trees, foliage, etc.) into the target multi-resolution voxel space 608 while filtering data associated with dynamic objects (e.g., representing pedestrians, vehicles, etc.).

[0027] In an alternative implementation, the data points 604 may be output by a perception pipeline or component with a semantic label attached. By way of example, the data points 604 may be received as part of a state representation of a scattered object output by a perception component, details of which are described in U.S. Application Serial No. 16 / 549,694, which is incorporated by reference herein in its entirety.

[0028] In the current example, the multi-resolution voxel space generation component 606 may assign the semantically labeled data points 604 to a semantic layer of the target multi-resolution voxel space 608 with a corresponding semantic label (e.g., tree, building, pedestrian, etc.). As an example, the multi-resolution voxel space generation component 606 may project the data points 604 to a common reference frame and then multiplex the data points 604 in the common reference frame into an appropriate point cloud associated with a corresponding semantic class. For each point cloud, the multi-resolution voxel space generation component 606 may then assign each data point 604 to a voxel of the finest resolution voxel grid (e.g., base voxel grid) of each semantic layer. In some particular cases, the multi-resolution voxel space may be a single layer that stores multiple statistics including the semantic class of each of the voxels.

[0029] Once each of the data points 604 for the corresponding group has been assigned to a voxel, the multi-resolution voxel space generation component 606 may calculate spatial statistics (e.g., spatial mean, covariance, and weight or number of data points 604 assigned to a voxel) for each voxel of the finest resolution grid of the semantic layer. In one particular example, the spatial statistics of a particular voxel may be calculated using Welford's Online Algorithm.

[0030] Upon completion of the base or finest resolution voxel grid of the semantic layer, the multi-resolution voxel space generation component 606 may iteratively or recursively generate each of the next higher resolution voxel grids of the semantic layer. As an example, the multi-resolution voxel space generation component 606 may utilize the preceding lower resolution grid (fixated by the base or finest resolution grid) and merge data associated with voxels within a 2x2x2 neighborhood to form the next higher level voxel grid. In one particular example, voxels within the neighborhood of the lower resolution voxel grid are merged by taking a weighted sum of the individual Gaussian distributions of each voxel within the neighborhood. Thus, the voxel grids in the semantic layer of the multi-resolution voxel space may form a multi-resolution pyramid, as described in more detail above with respect to Figures 1-5, with each higher resolution grid containing fewer voxels than the preceding lower resolution grid. In one particular example, each preceding lower resolution grid in a semantic layer may have four times as many voxels as the next higher resolution grid.

[0031] Once the target multi-resolution voxel space 608 is generated from the data points 604, the target multi-resolution voxel space 608 is aligned with a reference multi-resolution voxel space 610 (e.g., a multi-resolution voxel space representing a scene). By way of example, in the illustrated example, the multi-resolution voxel space alignment component 612 may align the newly generated target multi-resolution voxel space 608 with the reference multi-resolution voxel space 610, or may be to determine a transformation between the target multi-resolution voxel space 608 and the reference multi-resolution voxel space 610. To align the target multi-resolution voxel space 608 with the reference multi-resolution voxel space 610, the multi-resolution voxel space alignment component 612 may, for each semantic layer and each resolution of the target multi-resolution voxel space 608, substantially simultaneously take each voxel and determine the average target point in the corresponding resolution and semantic layer of the reference multi-resolution voxel space 610. The multiresolution voxel space alignment component 612 may then determine a 2x2x2 neighborhood of the voxel grid of the corresponding resolution and semantic layer of the reference multiresolution voxel space 610 and identify whether the neighboring voxels are occupied. The multiresolution voxel space alignment component 612 may then select a voxel from the target multiresolution voxel space 608 that has a closest centroid to the voxel, and average the distribution of the selected voxel and the distribution of the voxel from the target voxel. The multiresolution voxel space alignment component 612 may then perform a principal component analysis on the combined covariance matrix of the selected voxel and a voxel from the target, and more closely align the target multiresolution voxel space 608 with the reference multiresolution voxel space 610 by selecting the minimum eigenvalue as the matched normal vector for the two voxels.In some cases, optimization may be performed on the matched voxels to improve the overall alignment between the reference and target multi-resolution voxel spaces and / or to determine a relative transformation (e.g., used for localization), including, but not limited to, non-linear optimization (e.g., non-linear least-squares optimization). As an example, gradient descent techniques may be utilized, such as the Gauss-Newton technique described below.

[0032] In the Gauss-Newton technique, matching between a first voxel i in the target multi-resolution voxel space 608 and a second voxel j in the reference multi-resolution voxel space 610, the matched residual may be calculated as follows:

[0033]

number

[0034] however,

[0035]

number

[0036] is the matched normal vector, and μ i , is the mean of voxel i, and λ0 is the smallest eigenvalue of the matching covariance matrix. As mentioned above, the matching normal vector is calculated from the smallest eigenvector of the weighted sum of the corresponding voxel covariance matrices. The weight of each residual, z ij is reweighted according to the M-estimator framework (e.g., using a Cauchy loss function). Then, the Jacobian of the matching error ij with respect to the transformation between the reference grid and the target grid

[0037]

number

[0038] is given by J ij =[Rn ij xμ j ×Rn ij ] T The multi-resolution voxel space alignment component 612 may then compute, for each matching ij, the total gradient and an approximate Hessian as follows:

[0039]

number

[0040]

number

[0041] The Gauss-Newton optimization is calculated as follows: HδT=-g Furthermore, the multi-resolution voxel space alignment component 612 is SO(3)×R 3 and the updated alignment transformation is given by: x n+1 = [exp(δR)R n δp+p n ] T where exp() is the SO(3) exponential map. It should be understood that the transformation given above may be applied to the entire multi-resolution voxel space in further iterations of the optimization, and the final iteration may involve a transformation between two voxel spaces.

[0042] The alignment process may continue iterating after each adjustment of the target multi-resolution voxel space 608 until the two multi-resolution voxel spaces 608 and 610 are aligned within a tolerance or threshold or until a predetermined number of iterations (e.g., voxel merging) are completed. In this manner, during alignment, a coarser resolution (e.g., a resolution corresponding to larger voxels) may result in earlier matching that brings the two multi-resolution voxel spaces 608 and 610 into closer alignment such that a finer resolution may begin to match beyond a tolerance or threshold and complete the alignment process. However, in some implementations, operations may be performed across all layers and / or semantic classes substantially simultaneously with a single data transformation determined to align some or all of the various voxel spaces.

[0043] In one particular example, the multi-resolution voxel space alignment component 612 may utilize only the highest or coarsest resolution of each semantic layer in the first iteration to initialize the alignment prior to additional iterations. In some cases, each additional iteration may introduce another finer resolution to the alignment process. The fully aligned multi-resolution voxel space 614 may then be output by the multi-resolution voxel space alignment component 612 and used as the next reference multi-resolution voxel space 610.

[0044] 7 is an example flow diagram illustrating an example process 700 associated with constructing a multi-resolution voxel space described herein. As discussed above, a multi-resolution voxel space may include multiple voxel grids or voxel layers that represent a physical environment at different resolutions or physical distances. By way of example, each voxel layer may represent a physical environment at twice the resolution of the proceeding layer (e.g., 1 foot, 2 feet, 4 feet, etc.). In some cases, a multi-resolution voxel space may be separated into multiple semantic layers, each of which includes multiple voxel grids of different resolutions.

[0045] At 702, the multi-resolution voxel space generation component may receive data representing a physical environment. For example, the multi-resolution voxel space may be generated from data points representing the physical environment, such as, for example, the output of a lidar system. In other examples, the data may include the output of a radar, sonar, infrared, camera, or other image / data capture device. In some examples, the multi-resolution voxel space generation component may assign a semantic class to each data point. By way of example, in one particular example, the assignment of semantic classes to data points is described in U.S. Application Serial No. 15 / 820,245, which is incorporated by reference herein in its entirety.

[0046] At 704, a multi-resolution voxel space generation component generates a semantic point cloud from the data representing the physical environment. For example, the multi-resolution voxel space generation component may project data points from the data representing the physical environment into a common frame.

[0047] As an example, the multi-resolution voxel space generation component or another component may apply classification and / or segmentation techniques to the data points to assign semantic classes. In some examples, one or more neural networks (e.g., deep neural networks, convolutional neural networks, etc.), regression techniques, etc. may be used to identify and classify the data points by semantic classes. In some cases, the semantic classes may include classes or entity types, such as, for example, vehicles, pedestrians, cyclists, animals, buildings, trees, road surfaces, curbs, sidewalks, unknown, etc.

[0048] At 706, the multi-resolution voxel space generation component may generate a voxel covariance grid for each semantic class for a first resolution of the multi-resolution voxel space. In some examples, the multi-resolution voxel space generation component may assign data points to corresponding voxels of a matching semantic layer of the multi-resolution voxel space to generate each of the grids at the first resolution. Once the data points are assigned to the voxels of the semantic layer, the multi-resolution voxel space generation component may determine voxel spatial statistics, such as, for example, the mean and covariance for each voxel. In some cases, the multi-resolution voxel space generation component may start with the finest resolution layer and generate each next coarser layer when forming the multi-resolution voxel space.

[0049] At 708, the multi-resolution voxel space generation component determines whether there are additional resolutions to generate. For example, the multi-resolution voxel space generation component may determine whether the resolution is greater than a resolution threshold and / or whether the number of layers is greater than a layer threshold. If there are additional resolutions, process 700 proceeds to 710. However, if there are no additional resolutions to generate, process 700 proceeds to 712.

[0050] At 710, the multi-resolution voxel space generation component may generate a voxel covariance grid for each semantic class for the next higher resolution. Each higher resolution grid may be formed based at least in part on the lower resolution grid by merging voxels from the lower grid. As an example, the multi-resolution voxel space generation component may take a neighborhood (e.g., a 2x2x2 group) of voxels from a lower resolution grid in a semantic layer and compute a weighted sum of individual distributions (e.g., Gaussian distributions) from each of the voxels in the neighborhood to generate a single higher resolution voxel. In this way, each higher resolution has fewer voxels than the lower resolution grid, and the multi-resolution voxel space may form a multi-resolution voxel pyramid.

[0051] At 712, the multi-resolution voxel space generation component may smooth the resulting multi-resolution voxel space. For example, the multi-resolution voxel space generation component may convolve voxels of the multi-resolution voxel space with a Gaussian kernel to reduce noise in the voxel normal estimates. In addition, when lidar is used to collect data representing a physical environment, the multi-resolution voxel space generation component may remove voxels when they correspond to observations below a threshold (e.g., when a single lidar beam is observed) because these data points may have erroneously determined normals and poor statistics.

[0052] At 714, the multi-resolution voxel space generation component may reduce voxels having weights greater than the maximum weight to the maximum weight and remove voxels having weights less than the minimum weight. In some cases, by applying maximum and minimum weight ranges to voxels, the multi-resolution voxel space may maintain a more uniform sample density and prevent voxels close to the system (e.g., an autonomous vehicle) from causing disruptions to the alignment process described below with respect to FIG. 8. In other examples, the multi-resolution voxel space may include a lookup table stored as a hash of each voxel's location in the three-dimensional space and then indexing the hash (e.g., using voxel hashing) to provide quick memory access. In this way, only the desired portion of the multi-resolution voxel space is loaded into memory and accessing the multi-resolution voxel space may be performed using fewer processing resources.

[0053] 8 is another flow diagram illustrating an example process 800 for aligning a target multi-resolution voxel space with a reference multi-resolution voxel space described herein. As an example, the multi-resolution voxel space alignment component described above may utilize the multi-resolution voxel space structure to align multiple scans of a physical environment, such as a target multi-resolution voxel space and a reference multi-resolution voxel space. As an example, once a multi-resolution voxel space (e.g., a target multi-resolution voxel space) is generated for a particular scan or data set representing a physical environment, the multi-resolution voxel space alignment component may align the multi-resolution voxel space with a multi-resolution voxel space (e.g., a reference multi-resolution voxel space) representing a scene.

[0054] At 802, a multi-resolution voxel space alignment component may receive a target multi-resolution voxel space to be aligned with a reference multi-resolution voxel space representing a scene. In some cases, the reference multi-resolution voxel space may be maintained by the system and updated with each new scan of the environment to predetermine object detection and tracking.

[0055] At 804, the multi-resolution voxel space alignment component may determine voxel correspondence between the target multi-resolution voxel space and the reference multi-resolution voxel space. In some examples, the correspondence may be per semantic layer and per resolution. Furthermore, the correspondence may be determined for each resolution of each semantic layer substantially simultaneously. As an example, at 804, the multi-resolution voxel space alignment component may search for each voxel of a unique resolution in the target multi-resolution voxel space a 2x2x2 neighborhood of a voxel that contains the mean target point in a corresponding unique resolution of the reference multi-resolution voxel space. The multi-resolution voxel space alignment component may then select a voxel from the 2x2x2 neighborhood that has a closest centroid to the voxel in the target multi-resolution voxel space.

[0056] At 806, the multi-resolution voxel space alignment component may reweight the corresponding voxels. As an example, the multi-resolution voxel space alignment component may calculate a weighted average of the data contained in the two corresponding voxels (e.g., the target voxel and the selected voxel). As an example, a combined covariance may be calculated. Once the aggregate covariance is determined, the multi-resolution voxel space alignment component may perform a principal component analysis (such as an eigenvalue decomposition) on the combined covariance matrix of the two corresponding voxels and select the smallest eigenvalue as the matched normal vector. A residual (or error) for each voxel may be calculated proportional to the difference between the matched normal vectors and / or the means (or centroids) of the corresponding voxels, and an optimization over the transformation between the two frames may be performed to minimize the residual. In this way, during alignment, a coarser resolution (e.g., a resolution corresponding to larger voxels) may result in matching before a finer resolution. In this way, matching at the coarser resolution brings the two multi-resolution voxel spaces into closer alignment so that the finer resolution can begin to do the matching described above with respect to FIG. 6 and complete the alignment process.

[0057] At 808, the multi-resolution voxel space alignment component may determine whether a number of iterations have been completed. For example, the system may include a maximum number of iterations of the alignment process and may cap or limit the processing time associated with aligning two multi-resolution voxel spaces. If the number of iterations has been completed, process 800 proceeds to 812; otherwise, process 800 proceeds to 810.

[0058] At 810, the multi-resolution voxel space alignment component may determine whether the reweighted average for the combined multi-resolution voxel space is below a tolerance threshold. If the reweighted average is below the tolerance threshold, the process 800 proceeds to 812, otherwise the process 800 returns to 804. The system may include a tolerance that sets a requirement for how well the two multi-resolution voxel spaces should be aligned for a particular use. As an example, in some applications only a coarse understanding of the physical environment may be required, while in other cases, such as autonomous vehicles, a more accurate detailed understanding may be required.

[0059] At 812, the multiresolution voxel space alignment component may reduce and / or determine an amount of uncertainty in the alignment between the target multiresolution voxel space and the reference multiresolution voxel space. For example, after optimization (e.g., the nonlinear optimization described above), the multiresolution voxel space alignment component may propagate measurement noise to the aligned voxels. In one particular example, discussed in more detail below, the multiresolution voxel space alignment component may determine a model of the alignment uncertainty according to a Gaussian distribution with zero mean and zero covariance. By way of example, the multiresolution voxel space alignment component may model the alignment uncertainty according to a normal random variable x~N(μ, n) such that each step is calculated as follows: x ,Σ x ) may be modeled. x=Cz where C=(J T WJ) ―1 J T W (J and W represent the determined identical Jacobian and weights, and C represents the weighted pseudo-Hamiltonian), z~N(0, δ z 2 I) is the residual.

[0060] The covariance of x may then be determined by propagating the residual noise and expanding as follows: Σ x =CΣ z C T Σ x =(J T WJ) ―1 J T σ z 2 W 2 J(J T WJ) ―1 In some cases, the residual noise is calculated incrementally and then the matrix J T WJ and σ z 2 J T W 2 J may be collected for each voxel. In one particular example, the multi-resolution voxel spatial alignment component may further include:

[0061]

number

[0062] We may also derive the isotropic variance of,and,for each voxel, the average is,

[0063]

number

[0064] where σ p 2 is the isotropic Gaussian noise at each point observation, and W i are the voxel weights. The covariance of the residuals may then be calculated as: σ z 2 I=E[zz T ] E[zz T The terms containing ] vanish due to the independence assumed for the voxel means. Hence,

[0065]

number

[0066] however,

[0067]

number

[0068] The multiresolution voxel space alignment component may then further regularize the resulting covariance matrix by adding an assumed contribution from a nominal number of outliers and by clamping or limiting the diagonal to a minimum value.

[0069] At 814, the multi-resolution voxel space alignment component may output an aligned multi-resolution voxel space (which may include measurement uncertainties). As an example, the aligned multi-resolution voxel space may be provided to another system, such as, for example, a planning or perception system of an autonomous vehicle. In other cases, the aligned multi-resolution voxel space may be sent to a remote system or device, such as, for example, a cloud-based computing system, over one or more networks. In other examples, the multi-resolution voxel space alignment component may output localization or transformation data between a target multi-resolution voxel space associated with the position of the vehicle with respect to the physical environment and a reference multi-resolution voxel space. In some examples, the reference multi-resolution voxel space may be pre-generated by the cloud-based computing system and sent to the vehicle before the vehicle begins to move. In some cases, the cloud-based system may update the reference multi-resolution voxel space with data collected during operation from multiple vehicles (e.g., a merge target multi-resolution voxel space). Additionally, in some instances, the vehicle may be equipped to update the reference multi-resolution voxel space in an offline manner (e.g., when parked or otherwise not in a nimble driving situation).

[0070] 9 illustrates an example system for implementing techniques described herein according to aspects of the disclosure. In some examples, the system may include one or more features, processing resources, components, and / or functionality of the aspects described herein with reference to FIGS. 1-8. As noted above, in some aspects, the system may include an autonomous vehicle.

[0071] 9 is a block diagram of an example system 900 for implementing the multi-resolution voxel space alignment system described herein. In the presently described aspect, the system 900 is a vehicle 902 that may include a vehicle computing device 904, one or more sensor systems 906, one or more communication connections 908, and one or more drive systems 91010.

[0072] The vehicle computing device 904 may include one or more processors 912 (or processing resources) and a computer-readable medium 914 communicatively coupled to the one or more processors 912. In the illustrated example, the vehicle 902 is an autonomous vehicle, however, the vehicle 902 could be any other type of vehicle or any other system (e.g., a robotic system, a camera-enabled smartphone, etc.). In the illustrated example, the computer-readable medium 914 of the vehicle computing device 904 stores a multi-resolution voxel space generation component 916, a multi-resolution voxel space alignment component 918, a planning component 920, a perception component 922, as well as other systems associated with the autonomous vehicle. Additionally, the computer-readable medium 914 may store sensor data 924 and a multi-resolution voxel space 926. It should be understood that in some implementations, data stored on computer-readable media may also be accessible to the vehicle 902 in addition to or instead of the system (e.g., stored on or otherwise accessible on other computer-readable media separate from the vehicle 902).

[0073] The multi-resolution voxel space generation component 916 may generate a multi-resolution voxel space from data points representing a physical environment, such as the output of a lidar system. In some cases, the multi-resolution voxel space generation component 916 may receive multiple lidar points, or lidar data represented as a point cloud. The multi-resolution voxel space generation component 916 may assign the lidar points to voxels of a first base resolution voxel grid. The multi-resolution voxel space generation component 916 may then merge voxels of lower resolution grids when generating a higher level voxel grid. For example, the multi-resolution voxel space generation component 916 may merge voxels within a neighborhood (e.g., a 2x2x2 neighborhood, etc.) in a lower resolution grid when forming a next higher level voxel grid.

[0074] In one particular example, the multi-resolution voxel space generation component 1016 may generate the multi-resolution voxel space as a mappable contiguous block of memory accessible via a collision-free hash table with pointers implemented as offsets that allow the blocks to be moved or relocated in memory. In some cases, the memory block may be represented as a tile with a header, an index (e.g., a hash table), and a voxel array. The index may be separated by layer and / or resolution. The voxel array may include a single array or multiple arrays arranged by resolution (e.g., a first resolution grid of a first semantic layer, a first resolution grid of a second semantic layer, a first resolution grid of a third semantic layer, ...). In the voxel array, each element may be a voxel and a key to the spatial location of the voxel. In some cases, the header may include a stack identifier, a version number, a number of resolutions, a number of semantic labels, a total number of layers, an offset, etc. The index may be a sparse hash table that relates hash values ​​to offsets within the memory block, and may also include a salt value used to salt the input to the particular table just mentioned, and a prime value used in the first round of the modulus calculation.

[0075] In some examples, the multi-resolution voxel space alignment component 1018 may align two multi-resolution voxel spaces (e.g., a target multi-resolution voxel space and a reference multi-resolution voxel space). In some cases, the multi-resolution voxel space alignment component 918 may find correspondence between voxels in the reference multi-resolution voxel space and the target multi-resolution voxel space. For each voxel of a specific resolution in the target multi-resolution voxel space, the multi-resolution voxel space alignment component 918 may find the correspondence by searching a three-dimensional (e.g., 2x2x2, 3x3x3, 5x5x5, etc.) neighborhood of the voxel that contains the mean target point in the corresponding specific resolution of the reference multi-resolution voxel space. Of the identified voxels in the neighborhood, the multi-resolution voxel space alignment component 918 may select a voxel that has a centroid close to the voxel of the target multi-resolution voxel space. The multi-resolution voxel space alignment component 918 may then average the distribution of the selected voxel in the reference multi-resolution voxel space with the voxels in the target covariance stack. The multi-resolution voxel space alignment component 1018 may then perform principal component analysis on the combined covariance matrix and select the smallest eigenvalue as the matched normal vector for the two voxels.

[0076] The planning component 920 may determine a path for the vehicle 902 to follow to move through and traverse a physical environment. For example, the planning component 920 may determine various routes and trajectories and various levels of detail. For example, the planning component 920 may determine a path of travel from a current location to a goal location. For purposes of this discussion, a route may be a sequence of waypoints to travel between two locations.

[0077] In some implementations, the prediction component 922 may be configured to estimate current and / or predict future, characteristics, or states of an object (e.g., a vehicle, pedestrian, animal, etc.), such as, for example, attitude, speed, trajectory, velocity, yaw, yaw rate, roll, roll rate, pitch, pitch rate, position, acceleration, or other characteristics, based at least in part on the multi-resolution voxel space 926 output by the multi-resolution voxel space generation component 916 and the multi-resolution voxel space alignment component 918.

[0078] Additionally, the vehicle 902 may also include communication connection(s) 908 that enable communication between the vehicle 902 and other local or remote computing device(s). By way of example, the communication connection(s) 908 may facilitate communication of the vehicle 902 with other local computing device(s) and / or with drive system(s) 910. Additionally, the communication connection(s) 908 may enable the vehicle 902 to communicate with other nearby computer device(s) (e.g., other nearby vehicles, traffic signals, etc.). Additionally, the communication connection(s) 908 may also enable the vehicle 902 to communicate with remote teleoperated computing devices, or other remote services.

[0079] The communication connection(s) 908 may include physical and / or logical interfaces for connecting the vehicle computing device 904 to another computing device (e.g., computing device(s) 930) and / or to a network, such as network(s) 928. For example, the communication connection(s) 908 may enable Wi-Fi-based communications, such as, for example, frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as BLUETOOTH, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or via any suitable wired or wireless communications protocol that enables each computing device to interface with other computing device(s). In some examples, the communication connection 908 of the vehicle 902 may transmit or send the multi-resolution voxel space 926 to the computing device(s) 930.

[0080] In at least one example, the sensor system(s) 906 can include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), and one or more time of flight (ToF) sensors, etc. The sensor system(s) 906 can include multiple instances of each of the just mentioned or other types of sensors. As an example, the lidar sensors can include individual lidar sensors located at the corners, front, rear, sides, and / or top of the vehicle 902. As another example, the camera sensors can include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 902. The sensor system(s) 906 may provide input to the vehicle computing device 904. Additionally or alternatively, the sensor system(s) 906 can send sensor data over one or more networks 928 to one or more computing device(s) 930 at a specific frequency, at predetermined intervals, in near real-time, etc.

[0081] In at least one example, the vehicle 902 can include one or more drive systems 910. In some examples, the vehicle 902 can have a single drive module 910. In at least one example, if the vehicle 902 has multiple drive systems 910, the individual drive systems 910 can be located at opposing ends of the vehicle 902 (e.g., at the front and rear, etc.). In at least one example, the drive system(s) 910 can include one or more sensor systems 906 that detect the state of the drive system(s) 910 and / or the state of the surroundings of the vehicle 902, as described above. By way of example and not limitation, the sensor system(s) 906 can include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors that acoustically detect objects around the drive system, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system(s) 910. In some cases, the sensor system(s) 906 in the drive system(s) 910 may overlap or supplement a corresponding system in the vehicle 902.

[0082] In at least one example, the components described herein may process the sensor data 924 as described above and may send their respective outputs over one or more network(s) 928 to one or more computing device(s) 930. In at least one example, the components described herein may send their respective outputs to the one or more computing device(s) 930 at a particular frequency, at a predetermined fixed time, in near real-time, etc.

[0083] In some examples, the vehicle 902 can send sensor data to one or more computing device(s) 930 via the network(s) 928. In some examples, the vehicle 902 can send raw sensor data 924 or processed multi-resolution voxel space 926 to the computing device(s) 930. In other examples, the vehicle 902 can send processed sensor data 924 and / or a representation of the sensor data (e.g., an object perception track) to the computing device(s) 930. In some examples, the vehicle 902 can send the sensor data 924 to the computing device(s) 930 at a characteristic frequency, after a pre-defined period of time, in near real-time, etc. In some cases, the vehicle 902 can send the sensor data (raw or processed) to the computing device(s) 930.

[0084] The computing system(s) 930 may include a processor(s) 932 and a computer-readable medium 934 that stores a multi-resolution voxel space generation component 936, a multi-resolution voxel space alignment component 938, as well as sensor data 940 and a multi-resolution voxel space 942 received from the vehicles 902. In some examples, the multi-resolution voxel space generation component 936 and the multi-resolution voxel space alignment component 938 may be configured to generate a multi-resolution voxel space 942 or align multi-resolution voxel spaces 942 generated from data captured by multiple vehicles 902 to form a more complete scene of various physical environments and / or connect various scenes together as a signal augmented physical environment. In some cases, the multi-resolution voxel space generation component 936 and / or the multi-resolution voxel space alignment component 938 may be configured to generate one or more models from the sensor data 924 that may be used for machine learning and / or future code testing.

[0085] The processor(s) 912 of the vehicle 902 and the processor(s) 932 of the computing device(s) 930 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) 912 and 932 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts the electronic data into registers and / or other electronic data that may be stored in a computer-readable medium. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0086] The computer readable media 914 and 934 are examples of non-transitory computer readable media. The computer readable media 914 and 934 can store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and the functions attributed to the various systems. In various implementations, the computer readable media can be implemented using any suitable computer readable media technology, such as SRAM (Static RAM), SDRAM (Synchronous DRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architecture, systems, and individual elements described herein can include many other logical, programmatic, and physical components, and those shown in the accompanying drawings are merely examples relevant to the description of this specification.

[0087] As can be appreciated, the components described herein are illustrated as separate for purposes of illustration, however, operations performed by various components can be combined or performed by any other components.

[0088] 9 is illustrated as a distributed system, it should be noted that in alternative examples, components of the vehicle 902 may be associated with the computing device(s) 930 and / or components of the computing device(s) 930 may be associated with the vehicle 902. That is, the vehicle 902 may perform one or more functions associated with the computing device(s) 930, and vice versa.

[0089] 10 is a pictorial illustration 1000 of an example of a multi-resolution voxel space as described herein, such as the multi-resolution voxel space 208 of FIGS. 2-4 compared to a point cloud representation 1008 of captured data. Both the illustrated multi-resolution voxel space 208 and the point cloud representation 1008 correspond to real-world physical locations or spaces.

[0090] Example clauses A. A system including a lidar sensor, one or more processors, and one or more non-transitory computer readable media storing instructions executable by the one or more processors, the instructions, when executed, causing a system to receive data representing a physical environment from the lidar sensor, determine a first semantic class associated with a first portion of the data, determine a second semantic class associated with a second portion of the data, associate the first portion of the data with a voxel of a first voxel of a first voxel grid, the first voxel grid being associated with a first semantic layer of a target multi-resolution voxel space, and determine a second semantic class associated with a second voxel of a second voxel grid. and associating a first set of adjacent voxels of the first voxel grid with a second voxel grid associated with a second semantic layer of the target multi-resolution voxel space, the first voxel grid and the second voxel grid being associated with a first resolution; merging a first set of adjacent voxels of the first voxel grid to form voxels of a third voxel grid associated with the first semantic layer, the third voxel grid being associated with a second resolution lower than the first resolution; and merging a second set of adjacent voxels of the second voxel grid to form voxels of a fourth voxel grid associated with the second semantic layer, the fourth voxel grid being associated with the second resolution.

[0091] B. The system of paragraph A, wherein the operations further include merging a third set of adjacent voxels of the third voxel grid to form voxels of a fifth voxel grid associated with the first semantic layer, the fifth voxel grid having a third resolution lower than the second resolution, and merging a fourth set of adjacent voxels of the fourth voxel grid to form voxels of a sixth voxel grid associated with the second semantic layer, the sixth voxel grid having a third resolution.

[0092] C. The system of term A, wherein associating the first portion of the data includes determining that a number of observations in the first portion of the data is greater than or equal to a threshold number of observations, determining a mean of the first portion of the data, determining a covariance of the first portion of the data, and associating the mean and covariance with the first voxel.

[0093] D. The system of paragraph C, wherein the operations further include receiving a reference multi-resolution voxel space; determining a voxel correspondence between a target voxel in the target multi-resolution voxel space and a reference voxel in the reference multi-resolution voxel space, the target voxel and the reference voxel comprising the same resolution; determining weighted statistics of combined voxels representing the target voxel and the reference voxel; determining a transformation between the reference multi-resolution voxel space and the target multi-resolution voxel space based at least in part on the weighted average covariance; and controlling the autonomous vehicle based at least in part on the transformation.

[0094] E. A method including receiving sensor data from a sensor; associating at least a first portion of the sensor data with a first voxel of a first voxel grid of a multi-resolution voxel space, the first voxel being associated with a first semantic classification and a first resolution; associating at least a second portion of the sensor data with a second voxel of a second voxel grid of the multi-resolution voxel space, the second voxel being associated with the first semantic classification and the first resolution; determining a third voxel associated with a second resolution lower than the first resolution based at least in part on the first voxel and the second voxel, the third voxel being associated with the first semantic classification; and controlling an autonomous vehicle based at least in part on the multi-resolution voxel space.

[0095] F. The method of clause E, further including determining a first semantic classification associated with the first portion of the data, determining a second semantic classification associated with a third portion of the data, and associating the third portion of the data with a third voxel of the multi-resolution voxel space based at least in part on the second semantic classification.

[0096] G. The method of paragraph E, wherein correlating the first portion of the data includes determining a first mean value for the first portion of the data, determining a first covariance for the first portion of the data, associating the first mean and first covariance with the first voxel, determining a second mean value for the second portion of the data, determining a second covariance for the second portion of the data, and associating the second mean and second covariance with the second voxel.

[0097] H. The method of term E, wherein determining the third voxel includes determining a weighted average of the first mean of the first voxel and the second mean of the second voxel, determining a weighted average of the first covariance of the first voxel and the second covariance of the second voxel, and associating the weighted average of the first mean and the second mean and the weighted average of the first covariance and the second covariance with the third voxel.

[0098] I. The method of clause E, further including receiving a reference multi-resolution voxel space; determining a voxel correspondence between a first voxel and a reference voxel in the reference multi-resolution voxel space, the reference voxel having a first resolution; determining weighted statistics of combined voxels representing the first voxel and the reference voxel; and determining a transformation between the multi-resolution voxel space and the reference multi-resolution voxel space based at least in part on the weighted average statistics, wherein controlling the autonomous vehicle is based at least in part on the transformation.

[0099] J. The method of section I, wherein the voxel correspondence is based on a distance between at least a first centroid associated with a reference multi-resolution voxel space and a second centroid associated with a target multi-resolution voxel space.

[0100] K. Weighted statistics include weighted covariances, method of section I.

[0101] L. The method of paragraph I, wherein determining the transformation further includes determining a measurement uncertainty based at least in part on modeling the alignment as a Gaussian distribution.

[0102] M. The method of section I, wherein determining the transformation includes determining a weighted average of the covariance of the first voxel and the covariance of the reference voxel, and determining a minimum eigenvector of the weighted average.

[0103] N. The method of paragraph E, wherein the first voxel and the second voxel are neighbors within the first resolution.

[0104] O. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations including receiving sensor data from a sensor associated with the vehicle; associating a first portion of the data with a first voxel of a first grid of a voxel space, the first portion of the data having a first semantic class; determining a first weighted statistic associated with the first portion of the data; associating a second portion of the data with a second voxel of the first grid of the voxel space, the second portion of the data having the first semantic class; determining a third weighted statistic associated with a third voxel of a second grid of the voxel space based at least in part on the first weighted statistic and the second weighted statistic, the first grid being associated with a first resolution having fewer voxels than a second resolution associated with the second grid; and controlling the vehicle based at least in part on the voxel space.

[0105] P. A non-transitory computer-readable medium of term O, the operations including associating a first portion of the data and a second portion of the data with a first semantic layer in voxel space, the first semantic layer corresponding to a first semantic class, associating a third portion of the data with a third voxel of a first grid in voxel space, the third portion of the data having a second semantic class, and associating the third portion of the data with a second semantic layer in voxel space, the second semantic layer corresponding to the second semantic class.

[0106] Q. A non-transient computer-readable medium of term O, the first semantic class of which includes pedestrians, vehicles, buildings, animals, or foliage.

[0107] R. A non-transitory computer readable medium of a term O, wherein the first weighted statistic includes a first mean and a first covariance of a first portion of the data and the second weighted statistic includes a second mean and a second covariance of a second portion of the data.

[0108] S. A non-transitory computer readable medium of term O, wherein a third weighted statistic is determined based at least in part on determining a weighted average of the first mean and the second mean, determining a weighted average of the first covariance and the second covariance, and associating the weighted average of the first mean and the second mean and the weighted average of the first covariance and the second covariance with a third voxel.

[0109] T. The non-transitory computer-readable medium of claim O, wherein the operations further include determining a location of the vehicle within the physical environment based at least in part on the voxel space and the multi-resolution voxel space.

[0110] U. A system including one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions which, when executed, cause the system to perform operations including receiving data, associating the data with a target multi-resolution voxel space, receiving a reference multi-resolution voxel space, determining a target voxel in the target multi-resolution voxel space associated with a reference voxel in the reference multi-resolution voxel space, the target voxel and the reference voxel associated with a same resolution, determining a weighted statistical value associated with combined voxels representing the target voxel and the reference voxel, determining a transformation based at least in part on the weighted statistical value, and controlling an autonomous vehicle based at least in part on the transformation.

[0111] V. Weighted statistics is the weighted covariance matrix of the system of terms U.

[0112] W. The operations further include performing a principal component analysis on the weighted mean covariance matrix and determining a minimum eigenvector of the principal component analysis, wherein determining a transformation of the system of terms U is further based on the minimum eigenvector.

[0113] X. A system of terms U, wherein determining a target voxel in a target multiresolution voxel space associated with a reference voxel in a reference multiresolution voxel space and determining a weighted statistical value associated with a combined voxel representing the target voxel and the reference voxel is performed iteratively for a plurality of pairs of voxels, each pair of voxels including a voxel in the target multiresolution voxel space and a voxel in the reference multiresolution voxel space.

[0114] Y. A method comprising: receiving map data including a first voxel space, the first voxel space having a first tier associated with a first resolution and a second tier associated with a second resolution different from the first resolution; receiving sensor data from a sensor associated with a vehicle, associating the sensor data with a second voxel space, the second voxel space having a first tier associated with the first resolution and a second tier associated with the second resolution; determining first aggregated voxel data based at least in part on the first voxel space and the second voxel space; determining a transformation between the first voxel space and the second voxel space based at least in part on the first aggregated voxel data; and determining a location of the vehicle in a physical environment based at least in part on the transformation.

[0115] The method of term Y, wherein determining the first aggregated voxel data includes identifying, relative to a first voxel in the first voxel space, a set of voxels in a second voxel space having a centroid within a specified distance of the centroid of the first voxel, selecting a second voxel of the set of voxels, the second voxel having a centroid closest to the centroid of the first voxel, and determining a weighted average of the covariance of the first voxel and the covariance of the reference voxel.

[0116] AA. The method of claim 1, wherein the first voxel and the second voxel comprise the same semantic class.

[0117] AB. The method of term Z, wherein determining the first aggregated voxel data further includes determining a minimum eigenvector of the weighted average and determining a normal vector representing the first aggregated voxel data based at least in part on the minimum eigenvector.

[0118] AC. The method of paragraph AB, wherein reweighting the first aggregated voxel data includes applying an m-estimator framework.

[0119] AD. The method of term AB, wherein determining the transformation includes determining a residual based at least in part on the minimum eigenvalue, and determining one or more of a rotation or a translation between the target voxel space and the reference voxel space based at least in part on the residual.

[0120] AE. The method of paragraph AD, further comprising determining an uncertainty associated with the alignment based at least in part on modeling the distribution.

[0121] AF. Transformation is a method of the term Z that indicates the difference in one or more of position or orientation between the first voxel space and the second voxel space.

[0122] AG. The method of claim Z, wherein the vehicle is an autonomous vehicle, and the method further includes controlling the autonomous vehicle based at least in part on a location of the autonomous vehicle in the physical environment.

[0123] AH. When executed, the method has one or more processors receive a target multi-resolution voxel space; receive a reference multi-resolution voxel space; determine that a first target voxel in the target multi-resolution voxel space is associated with a first reference voxel in the reference multi-resolution voxel space, the first target voxel and the first reference voxel sharing a first resolution; determine first weighted statistics of the first target voxel and the first reference voxel; and determine that a second target voxel in the target multi-resolution voxel space is associated with a first reference voxel in the reference multi-resolution voxel space, the first target voxel and the first reference voxel sharing a first resolution. a second combined voxel representing the second target voxel and the second reference voxel sharing a second resolution, the second resolution being different from the first resolution; determining second weighted statistics of second combined voxels representing the second target voxel and the second reference voxel; and determining a transformation between the target multi-resolution voxel space and the reference multi-resolution voxel space based at least in part on the first weighted statistics and the second weighted statistics.

[0124] AI. A non-transitory computer-readable medium of term AH, wherein the target multi-resolution voxel space includes a first set of voxels associated with a first classification and a second set of voxels associated with a second classification.

[0125] AJ. The non-transitory computer-readable medium of claim AH, wherein determining that a first target voxel in the target multi-resolution voxel space is associated with a first reference voxel in the reference multi-resolution voxel space includes identifying a set of voxels in the reference multi-resolution voxel space having a centroid within a specified distance of a centroid of the first target voxel relative to the first target voxel, and determining the first reference voxel from the set of voxels based on a distance of the centroid of the first reference voxel to the centroid of the first target voxel.

[0126] AK. The non-transitory computer-readable medium of term AH, further including determining that the first target voxel and the first reference voxel correspond based at least in part on a distance between a centroid of the first target voxel and a centroid of the first reference voxel.

[0127] AL. A non-transitory computer readable medium for a term AH, wherein the first weighted statistic is a weighted average covariance.

[0128] A non-transitory computer-readable medium of the term AH, wherein determining the transformation includes performing a principal component analysis on the first weighted statistics, determining a minimum eigenvalue of the principal component analysis, determining a residual based at least in part on the minimum eigenvalue, and determining as the transformation, one or more of a rotation or a translation between the target multi-resolution map and the reference multi-resolution map that optimizes the residual.

[0129] AN. A non-transitory computer-readable medium of the term AM, further comprising applying one or more of a gradient descent technique or a nonlinear optimization technique that minimizes a value based at least in part on the residual, wherein the transformation comprises one or more of a translation or a rotation.

[0130] Although the exemplary clauses described above are described with respect to certain specific implementations, it should be understood in the context of this document that the contents of the exemplary clauses can be implemented via methods, devices, systems, computer-readable media, and / or other implementations. In addition, any of the examples A-AN may be implemented alone or in combination with any other one or more of the examples A-AN.

[0131] termination As can be understood, the components described herein are described as separate for purposes of illustration. However, operations performed by various components can be combined or performed by any other components. Furthermore, it should be understood that components or steps described with respect to one example or implementation may be used in conjunction with components or steps of other examples. For example, the components and instructions of FIG. 9 may utilize the processes and flows of FIGS. 1-8.

[0132] Although one or more examples of the techniques described herein have been described, various alternatives, additions, permutations and equivalents fall within the scope of the techniques described herein.

[0133] In describing the examples, reference is made to the accompanying drawings, which form a part hereof, which show, by way of example, specific examples of the claimed subject matter. It is understood that other examples can be used and that modifications or alternatives, such as structural changes, can be made. The above examples, modifications, or alternatives do not necessarily depart from the scope of the intended and claimed subject matter. Although steps herein may be given in a certain order, in some cases the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. Furthermore, the disclosed procedures could be performed in a different order. Additionally, the various calculations described herein need not be performed in the disclosed order, and other examples using alternative orderings for the calculations could be implemented without difficulty. In addition to being reordered, in some cases the calculations could also be decomposed into sub-calculations having the same results.

Claims

1. receiving map data including a first voxel space, the first voxel space having a first layer associated with a first resolution and a second layer associated with a second resolution different from the first resolution; receiving sensor data from a sensor associated with the vehicle; Associating the sensor data with a second voxel space, the second voxel space including a first layer associated with the first resolution and a second layer associated with the second resolution; and determining first aggregated voxel data based at least in part on the first voxel space and the second voxel space; determining a transformation between the first voxel space and the second voxel space based at least in part on the first aggregated voxel data; determining a location of the vehicle in a physical environment based at least in part on the transformation; and Equipped with Determining the first aggregated voxel data includes: identifying, relative to a first voxel in the first voxel space, a set of voxels in the second voxel space having a centroid within a specified distance of a centroid of the first voxel; selecting a second voxel from the set of voxels, the second voxel having a centroid closest to the centroid of the first voxel; determining a weighted average of the covariance of the first voxel and the covariance of a reference voxel; A method comprising:

2. The method of claim 1 , wherein the first voxel and the second voxel comprise a same semantic class.

3. Determining the first aggregated voxel data includes: determining a minimum eigenvector of the weighted average; determining a normal vector representing the first aggregated voxel data based at least in part on the minimum eigenvector; and The method according to claim 1 or 2, further comprising:

4. 4. The method of claim 1, wherein reweighting the first aggregated voxel data comprises applying an m-estimator framework.

5. Determining the transformation comprises: determining a residual based at least in part on the minimum eigenvalue; determining one or more of a rotation or a translation between the target voxel space and the reference voxel space based at least in part on the residual; 4. The method according to claim 1, further comprising:

6. 6. The method of claim 1, further comprising determining an uncertainty associated with the alignment based at least in part on modeling the distribution.

7. 7. The method of claim 1, wherein the transformation indicates a difference in one or more of position or orientation between the first and second voxel spaces.

8. the vehicle is an autonomous vehicle, and the method comprises: Controlling the autonomous vehicle based at least in part on the location of the autonomous vehicle in the physical environment. The method according to any one of claims 1 to 7, further comprising:

9. A computer program comprising coded instructions which, when executed on a computer, implements the method according to any one of claims 1 to 8.

10. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors; the instructions, when executed, cause the system to: Receiving a target multi-resolution voxel space; receiving a reference multi-resolution voxel space; determining that a first target voxel in the target multi-resolution voxel space is associated with a first reference voxel in the reference multi-resolution voxel space, the first target voxel and the first reference voxel sharing a first resolution; determining first weighted statistics of the first target voxel and the first reference voxel; determining that a second target voxel in the target multi-resolution voxel space is associated with a second reference voxel in the reference multi-resolution voxel space, the second target voxel and the second reference voxel sharing a second resolution, the second resolution being different from the first resolution; determining a second weighted statistic of second combined voxels representing the second target voxel and the second reference voxel; determining a transformation between the target multi-resolution voxel space and a reference multi-resolution voxel space based at least in part on the first weighted statistics and the second weighted statistics; A system for causing a computer to perform an operation including the steps of:

11. 11. The system of claim 10, wherein the target multi-resolution voxel space includes a first set of voxels associated with a first classification and a second set of voxels associated with a second classification.

12. Determining that the first target voxel in the target multi-resolution voxel space is associated with the first reference voxel in the reference multi-resolution voxel space includes: identifying a set of voxels in the reference multi-resolution voxel space having a centroid relative to a first target voxel within a specified distance of a centroid of the first target voxel; determining the first reference voxel from the set of voxels based on a distance of a centroid of the first reference voxel to the centroid of the first target voxel; 12. The system according to claim 10 or 11, comprising:

13. 13. The system of claim 10, wherein the operations further include determining that the first target voxel and the first reference voxel correspond based at least in part on a distance between a centroid of the first target voxel and a centroid of the first reference voxel.

14. Determining the transformation comprises: performing a principal component analysis on the first weighted statistics; determining a minimum eigenvalue of the principal component analysis; determining a residual based at least in part on the smallest eigenvalue; and determining, as said transformation, one or more of a rotation or a translation between the target multi-resolution map and the reference multi-resolution map that optimizes said residual; Including, The operations further include applying one or more of a gradient descent technique or a nonlinear optimization technique to minimize a value based at least in part on the residual, and the transformation includes one or more of a translation or a rotation.

14. The system according to any one of claims 10 to 13.

Images