Method for converting 3D point clouds to confidence-scored 3D grids for height mapping in stereo imaging systems

Abstract

Systems and methods related to converting 3D point clouds to confidence-scored 3D grids for height mapping in stereo imaging systems are disclosed herein. A height map may be generated for a physical surface from a point cloud. The point cloud may be captured using a stereo camera system, where the point cloud comprises a list of three-dimensional coordinates for a set of points on the physical surface. A grid map may be generated from the point cloud, where the grid map comprises a set of centroid points with confidence scores for a set of voxels of the grid map. The height map may be generated from the grid map, where the height map comprises a set of height values derived using the set of confidence scores. The height map may have increased accuracy, for example by removing outliers and reducing noise.

Description

Method for Converting 3D Point Clouds to Confidence-Scored 3D Grids for Height Mapping in Stereo Imaging SystemsCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 640,869, filed April 30, 2024.BACKGROUND

[0002] The present description relates to 3D occupancy grid prediction, 3D change detection, object volume estimation, height mapping, and occupancy assessment, such as for instance using multi-camera systems. It relates to the conversion of 3D point cloud data outputted by a stereo camera into a spatial grid map or height map that will be more usable to calculate volumes and occupancy. Applications include logistics, urban planning, and surveillance, where precise space optimization, inventory management, and monitoring are crucial.SUMMARY

[0003] This disclosure relates to methods for converting 3D point clouds to confidence-scored 3D grids for height mapping in stereo imaging systems. The methods offer a solution to create a height map or grid map from a 3D point cloud and add a confidence value to each cell of the grid map or value of the height map. The confidence values can be based on all the confidence values of the points from the point cloud that will fall into the cell of the grid map or that contribute to the generation of the value of the height map.

[0004] Some approaches to transforming grid maps into height maps may be prone to errors. For example, searching the highest cell that is filled by point cloud data points in a column of cells and labeling the highest cell as the height of (x,y) coordinate of the height map may fall victim to outliers and thus errors, as outliers of 3D points tend to overestimate heights significantly. Instead of determining the height of a column by only determining the first occupied (e.g., highest) cell in a column, the height of the column can be determined by using the estimated centroid with its own estimated confidence from one or multiple contiguous cells in a column. For instance, the height value of each column of cells can be computed by the weighted averaged confidence value of each cell, using the confidence calculated previously.Hence, rather than considering only the maximum cell filled by points, the height of a coordinate may be a computed value based on all cells of the column. This allows for the reduction of outliers of 3D points that are inherent to stereo depth estimation. A benefit of this method is a noise reduction effect, as voxels with low confidence may be eliminated. Another way to calculate the height of a column, and incurring these benefits, may include determining whether one or more point cloud data points in a voxel have a combined confidence score above a threshold.

[0005] A stereo camera system may make a point cloud from data obtained when measuring a region or volume of interest (e.g., an environment, object, etc.). The region or volume of interest may, accordingly, be in the field of view of the stereo camera system. The stereo camera system may sample the region of interest and may divide it into voxels. The voxels may correspond to any size or shape of physical space (e.g., 1cm3). The voxels can have the same size along x, y, and z directions (e.g., cubes). Alternatively, the voxels can have different sizes along x, y, z directions (e.g., parallelepipeds).

[0006] The point cloud may be converted to a height map. To convert the point cloud to a height map, the stereo camera system may first convert the point cloud into a grid map; and then convert the grid map into a height map. The point cloud and the grid map may be divided into voxels. Multiple point cloud data points may be measured for each voxel and each point cloud data point may have an associated confidence score (a point cloud confidence score). A subset of the 3D point cloud data points that are inside a first voxel of the point cloud may be averaged to create a centroid point (x, y, z coordinate) for the corresponding voxel of the grid map. The point cloud confidence scores corresponding to the subset of the 3D point cloud data points inside the first voxel of the point cloud may also be averaged to create a confidence score for the centroid point of the corresponding voxel of the grid map. This process may be repeated for every voxel of the point cloud to convert the point cloud to the grid map. The confidence scores may indicate the level of certainty for that point (point cloud confidence scores for point cloud data points and centroid point confidence scores for centroid points).

[0007] The centroid confidence scores may be determined on a per voxel basis. For each voxel (or cell), the centroid point coordinates can be computed by using an average of all the pointcloud data points that belong to this voxel. The average can be determined with a standard average, a weighted average, a median, or / and combined with thresholds (based on the confidence value of each point), or any other statistical averaging method. For instance, the centroid point coordinates for each voxel can be a standard average value of the point cloud data points included in the voxel. Alternatively, the centroid point coordinates can be determined with an average value of the point cloud data points weighted by the confidence value of each data point. Alternatively, the centroid point coordinates can be a median. The confidence score for the centroid point can be an average or median value of the confidence values of the subset of the 3D point cloud corresponding to that voxel.

[0008] The grid map may be converted to a height map. A column of the grid map may include all the voxels (centroid points and confidence scores) sharing the same horizontal plane (x,y) coordinates of the grid map. Within a column, there may be both empty and filled voxels. An empty voxel may not have any centroid point (nor associated confidence score). In specific embodiments, the empty voxel may not have any point cloud data points. In specific embodiments, the empty voxel may have one or more point cloud data points but the combined confidence scores of these point cloud data points may be below a threshold value such that the system disregards these point cloud data points and considers the voxel to be empty. A filled voxel may have a centroid point (and associated confidence score). In specific embodiments, the filled voxel may have one or more corresponding point cloud data points. In specific embodiments, the combined confidence scores of the one or more corresponding point cloud data points may be above a threshold value. It may be determined that one or more voxels at the top of the column may be empty while one or more voxels lower than those voxels may be filled. The highest filled voxel may be detected.

[0009] The height of the corresponding (x,y) coordinate of the voxel may be determined based on the height of the highest centroid (corresponding to the highest filled voxel) or an average of the heights of a plurality of highest centroid points. Whether the highest centroid determines the height of the height map or whether the plurality of highest centroid determines the height may depend on the physical surface the coordinate of the height map refers to. For example, if the column of interest corresponds to the middle of a physical object, only one centroid pointwill be present on the top surface, the inner part of the physical object being occulted by the top surface. Conversely, if the column of interest corresponds to an edge or side of the physical object, a plurality of centroid points (in a plurality of voxels) will be present on a vertical surface. In order to reduce the noise in these situations, the height value of the column may be determined using an average of the heights of topmost centroid points of the column. The average value can be a standard average, a weighted average (using the confidence scores) or a median. When multiple centroid points are combined, an average confidence score can be determined. The average confidence score may allow for employing further noise reduction based on the height map values, for example by applying a noise reducing filter.

[0010] The height map (with confidence scores) may be converted to a filtered height map with reduced noise. In specific embodiments, a bilateral filter can be used to reduce noise. The bilateral filter may use the previously computed height value and the previously computed confidence score value of each cell to create a smooth height map output. The output height for the cell may be a function of the input height for the cell, the confidence scores, and a normalization factor. In specific embodiments, a gaussian function may be used as a coefficient factor in the filter and the sigma value of the gaussian function may be based on the confidence scores previously calculated for each cell. The height value of a coordinate in the filtered height map may be influenced mainly by coordinates that are close spatially to, and that have a height similar to, that coordinate.

[0011] In specific embodiments of the invention, a method for generating a height map for a physical surface from a point cloud is provided. The method comprises capturing the point cloud using a stereo camera system. The point cloud comprises a list of three-dimensional coordinates for a set of points on the physical surface. The method further comprises generating a grid map from the point cloud. The grid map comprises a set of centroid points for a set of voxels of the grid map, the centroid points are centroids of the set of points in each voxel of the set of voxels, and the set of centroid points are associated with a set of confidence scores. The method further comprises generating the height map from the grid map. The height map comprises a set of height values derived using the set of confidence scores.

[0012] In specific embodiments of the invention, a stereo camera system is provided. The stereo camera system comprises: a pair of cameras, one or more processors, and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the stereo camera system to conduct a method for generating a height map for a physical surface from a point cloud. The method comprises capturing the point cloud using the stereo camera system. The point cloud comprises a list of three- dimensional coordinates for a set of points on the physical surface. The method also comprises generating a grid map from the point cloud. The grid map comprises a set of centroid points for a set of voxels of the grid map, the centroid points are centroids of the set of points in each voxel of the set of voxels, and the set of centroid points are associated with a set of confidence scores. The method further comprises generating the height map from the grid map. The height map comprises a set of height values derived using the set of confidence scores.

[0013] In specific embodiments of the invention, one or more non-transitory computer- readable media are provided. The one or more non-transitory computer-readable media store instructions, which when executed by one or more processors cause the one or more processors to conduct a method for generating a height map for a physical surface from a point cloud. The method comprises capturing the point cloud using a stereo camera system. The point cloud comprises a list of three-dimensional coordinates for a set of points on the physical surface. The method further comprises generating a grid map from the point cloud. The grid map comprises a set of centroid points for a set of voxels of the grid map, the centroid points are centroids of the set of points in each voxel of the set of voxels, and the set of centroid points are associated with a set of confidence scores. The method further comprises generating the height map from the grid map. The height map comprises a set of height values derived using the set of confidence scores.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings illustrate various embodiments of systems, methods, and embodiments of various other aspects of the disclosure. A person with ordinary skills in the art will appreciate that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one example of the boundaries. It may be that in someexamples one element may be designed as multiple elements or that multiple elements may be designed as one element. In some examples, an element shown as an internal component of one element may be implemented as an external component in another, and vice versa. Furthermore, elements may not be drawn to scale. Non-limiting and non-exhaustive descriptions are described with reference to the following drawings. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating principles.

[0015] Figure 1 provides an example of a stereo camera system including a stereo camera capturing a volume of interest and generating a height map in accordance with specific embodiments of the inventions disclosed herein.

[0016] Figure 2 provides an example of converting a point cloud into a grid map in accordance with specific embodiments of the inventions disclosed herein.

[0017] Figure 3 provides an example of transforming a grid map column into a height map column in accordance with specific embodiments of the inventions disclosed herein.

[0018] Figure 4 provides an example of converting a height map into a filtered height map with reduced noise in accordance with specific embodiments of the inventions disclosed herein.

[0019] Figure 5 provides an example of a method for generating a height map for a physical surface from a point cloud in accordance with specific embodiments of the inventions disclosed herein.

[0020] Figure 6 provides examples of determining whether a voxel is filled in accordance with specific embodiments of the inventions disclosed herein.DETAILED DESCRIPTION

[0021] Reference will now be made in detail to implementations and embodiments of various aspects and variations of systems and methods described herein. Although several exemplary variations of the systems and methods are described herein, other variations of the systems and methods may include aspects of the systems and methods described herein combined in any suitable manner having combinations of all or some of the aspects described.

[0022] Different systems and methods for converting 3D point clouds to confidence-scored 3D grids for height mapping in stereo imaging systems in accordance with the summary above are described in detail in this disclosure. The methods and systems disclosed in this section arenonlimiting embodiments of the invention, are provided for explanatory purposes only, and should not be used to constrict the full scope of the invention. It is to be understood that the disclosed embodiments may or may not overlap with each other. Thus, part of one embodiment, or specific embodiments thereof, may or may not fall within the ambit of another, or specific embodiments thereof, and vice versa. Different embodiments from different aspects may be combined or practiced separately. Many different combinations and sub-combinations of the representative embodiments shown within the broad framework of this invention, that may be apparent to those skilled in the art but not explicitly shown or described, should not be construed as precluded.

[0023] The methods disclosed herein may create a height map or grid map from a 3D point cloud and add a confidence value to each cell of the grid map or value of the height map. The confidence values can be based on all the confidence of the points from the point cloud that will fall into the cell of the grid map or that contribute to the generation of the value of the height map. The height maps created with these methods may have reduced noise and increased accuracy.

[0024] A method for generating a height map from a point cloud is disclosed. The method may include receiving or capturing the point cloud. The point cloud can be from a region of interest such as a physical area under surveillance. The point cloud can be obtained from the measurements of a set of stereo cameras or imagers that are directed towards the region of interest. The point cloud may include a list of three-dimensional (3D) coordinates for a set of points. The set of points may be on a physical surface (e.g., an object, the ground, and any physical thing). The method can also include generating a grid map from the point cloud. The grid map can include a set of centroid points and an associated set of confidence scores for a set of voxels. The centroid point may be the centroids of the set of points in each voxel of the set of voxels. The method can also include generating a height map from the grid map. The height map can comprise a set of height values derived using the set of centroid points and the associated set of confidence scores.

[0025] In specific embodiments, generating the grid map comprises sampling the region of interest using a set of voxels, and determining the centroid points by averaging the 3Dcoordinates for the set of points on a per voxel basis with regards to the set of voxels. For example, the region of interest could be divided into 1 cm3voxels and the grid map could be a set of centroids that correspond with the voxels and have values that are set by the average of the set of points that fell within each voxel.

[0026] In specific embodiments, the point cloud includes a set of confidence scores for the set of points and the generating of the grid map uses these points. Generating the grid map could comprise determining the associated set of confidence scores using the set of confidence scores by averaging the set of confidence scores for the set of points on a per voxel basis with regards to the set of voxels. For example, the confidence scores for all the points within a voxel could be averaged to determine a confidence score for the centroid associated with that voxel in the grid map.

[0027] In specific embodiments, generating the height map further comprises conducting a noise reduction filtering on the grid map using the associated set of confidence scores. For example, the height map value could be derived by filtering out centroids (or point cloud data points) from the grid map that had low confidence values. The height map could be derived using the highest centroid in a column of the grid map and the confidence value thereof. In situations in which multiple centroids and associated confidence values were part of the grid map (e.g., along the edge of an object) multiple (e.g., some or all) of the values for the column could be considered in determining the height value for that column using noise reduction filtering and the associated confidence values.

[0028] Figure 1 illustrates an example of a stereo camera system 111, including a stereo camera, capturing volume of interest 101 to generate height map 104. Volume of interest 101 may accordingly be in the field of view of the stereo camera. Stereo camera system 111 may include one or more processors, one or more memories, one or more network connections, and other components not shown. The stereo camera of stereo camera system 111 may capture point cloud data points 112 to create point cloud 102. Each point cloud data point 112 may be associated with a point cloud confidence score 122. Point cloud data points 112 of each voxel may be combined (e.g., averaged) into centroid points 113. Point cloud confidence scores 122 of each voxel may be combined (e.g., averaged) into centroid confidence scores 123. Centroidpoints 113 and centroid confidence scores 123 of each column of grid map 103 may be used to calculate height map 104. Volume of interest 101, point cloud 102, and grid map 103 may be 3D. Height map 104 may be 2D, 2.5D, or 3D. A confidence score may also be referred to as a confidence value.

[0029] Volume of interest 101 may be divided into (e.g., sampled as) voxels (or cells) of known dimensions. For example, a voxel may be 1 cm3. In specific embodiments, the voxels may have the same size along x, y, and z directions (e.g., cubes). Alternatively, the voxels may have different sizes along x, y, z directions (e.g., parallelepipeds). Volume of interest 101 shows eight voxels for simplicity; a volume (or region) of interest may be divided into any quantity of voxels and may correspond to any size of physical space capturable by stereo camera system 111. Volume of interest 101 may also be called a region of interest, among other labels.

[0030] Point cloud 102 may be acquired from the stereo camera imaging and measuring aspects of volume of interest 101. Each point cloud data point 112 may be projected into a specific voxel in volume of interest 101. Any number of point cloud data points 112 may be within a voxel and different voxels may have different quantities of point cloud data points 112. In specific examples, some voxels may have no point cloud data points 112. In this case, the voxel may be "empty" or "unfilled." If a voxel includes at least one point cloud data point 112, the then voxel may be "filled." In specific embodiments, the confidence score 122 of the point cloud data point 112 must be above a threshold in order for the voxel to be "filled." In specific embodiments, a combination of confidence scores 122 of all point cloud data points 112 within a voxel must be above a threshold in order for the voxel to be "filled."

[0031] Stereo camera system 111 may convert point cloud 102 into grid map 103. Point cloud102 and grid map 103 may be divided into voxels such that the locations and dimensions of the voxels of point cloud 102 are the same as the voxels of grid map 103. The voxels of grid map103 may also be called grid cells. Multiple point cloud data points 112 may be measured for each voxel and each point cloud data point 112 may have an associated point cloud confidence score 122. A subset of point cloud data points that are inside a voxel of point cloud 102 may be averaged to create a centroid point (x, y, z coordinate) for the corresponding voxel of grid map 103. The point cloud confidence scores corresponding to the subset of point cloud data pointsinside the voxel may also be averaged to create a centroid confidence score for the corresponding centroid point of grid map 103. This process may be repeated for every voxel of point cloud 102 to convert point cloud 102 into grid map 103. Point cloud confidence scores 122 may indicate the level of certainty for the corresponding point cloud data points 112. Centroid confidence scores 123 may indicate the level of certainty for the corresponding centroid points 113.

[0032] Centroid confidence scores 123 may be determined on a per voxel basis. For each voxel (or cell), the coordinates of the centroid point 113 can be computed by using an average of all the point cloud data points 112 that belong to this voxel. The average can be determined with a standard average, a weighted average, a median, or / and combined with thresholds (based on the confidence value of each point), or any other statistical averaging method. For instance, the coordinates of centroid point 113 for each voxel can be a standard average value of the point cloud data points 112 included in the voxel. Alternatively, the coordinates of a centroid point 113 can be determined with an average value of the point cloud data points 112 weighted by the point cloud confidence score 122 of each point cloud data point 112. Alternatively, the coordinates of centroid point 113 can be a median of each point cloud data point 112 int the voxel. Centroid confidence score 123 for centroid point 113 can be an average or median value of the point cloud confidence scores 122 of the subset of the point cloud data points corresponding to that voxel.

[0033] Stereo camera system 111 may convert grid map 103 to height map 104. A column of grid map 103 may include all the voxels (with associated centroid points 113 and centroid confidence scores 123) sharing the same horizontal plane (x,y) coordinates of grid map 103. That is, a column of voxels of grid map 103 may correspond to an (x,y) coordinate of height map 104. Within a column, there may be both empty and filled voxels. An empty voxel may not have any centroid point 113 (nor associated centroid confidence score 123). In specific embodiments, the empty voxel may not have any point cloud data points 112 (nor associated point cloud confidence scores 122). In specific embodiments, the empty voxel may have one or more point cloud data points 112 but the combined point cloud confidence scores 122 may be below a threshold value such that the system disregards these point cloud data points 112 andconsiders the voxel to be empty. A filled voxel may have a centroid point 113 (and associated confidence score 123). In specific embodiments, the filled voxel may have one or more corresponding point cloud data points 112 (and associated point cloud confidence scores 122). In specific embodiments, the combined point cloud confidence scores 122 of the one or more corresponding point cloud data points 112 may be above a threshold value. Stereo camera system 111 may determine that one or more voxels at the top of the column may be empty while one or more voxels lower than those voxels may be filled. The highest filled voxel may be detected; the highest centroid points 113 and associated centroid confidence scores 123 of each column of grid map 103 are shown in height map 104. The height of the corresponding (x,y) coordinate (e.g., height value 131) may be determined based on the height of the highest centroid (corresponding to the highest filled voxel) or an average of the heights of a plurality of highest centroid points. Whether the highest centroid determines height value 131 of the height map or whether the plurality of highest centroid determines height value 131 may depend on the physical surface the coordinate of the height map refers to. In specific embodiments, height map 104 (with centroid confidence scores 123) may be converted to a filtered height map with reduced noise. Filtering height map 104 may include conducting a noise reduction filtering on the grid map using a noise reduction filter. The noise reduction filter may include a coefficient factor that uses a gaussian function based on the set of centroid confidence scores 123.

[0034] Figure 2 illustrates an example of converting point cloud 201 into grid map 202. The upper-right-front voxel (bolded) of point cloud 201 may correspond to point cloud voxel 205; and the upper-right-front voxel (bolded) of grid map 202 may correspond to grid map voxel 206. Grid map voxel 206 may correspond to (e.g., be derived from) point cloud voxel 205 and they each may represent the same physical space. Point cloud 201 and grid map 202 contain eight voxels each for the sake of illustration but may contain any number of voxels. A voxel may also be called a cell. Point cloud 201 may be 3D and may be captured using a stereo camera system that may sample the region of interest using the set of voxels. Point cloud 201 may comprise a list of 3D coordinates for a set of points on a physical surface. These 3D coordinates may be point cloud data points 212. Point cloud 201 may be made from dataobtained when measuring a region of interest (e.g., an environment, object, etc.), or rather physical surfaces within the region of interest. Point cloud data points 232 may be a subset of the set of point cloud data points 212. Point cloud confidence scores 242 may be a subset of the set of point cloud confidence scores 222. Centroid 233 may be a subset of the set of centroids 213. Centroid confidence score 243 may be a subset of set of centroid confidence scores 223.

[0035] The conversion of point cloud 201 to grid map 202 can be done by dividing the world into voxels of known dimensions (e.g., each voxel may be 1cm x 1cm x 1cm). Then each point cloud data point 212 of point cloud 201 can be projected into a corresponding voxel or cell of grid map 202. The voxels of grid map 202 may be the same dimensions as, and correspond directly to, the voxels of point cloud 201. Point cloud data may include point cloud data points 212 with point cloud confidence scores 222. The point cloud data can be utilized to determine the conversion (e.g., projection) of point cloud 201 to grid map 202. Point cloud 201 can be converted to grid map 202 by dividing the environment of point cloud 201 into voxels, such as point cloud voxel 205, of a known dimension, and then each point 212 of point cloud 201 within a voxel can be averaged with the other points 212 in that voxel to make centroids 213.Centroids 213 can be projected into a specific grid cell to form grid map 202. Each grid cell of grid map 202 may include a centroid 213 and an associated confidence score 223. For example, grid map voxel 206 may include centroid 233 and centroid confidence score 243.

[0036] Point cloud data points 212 of a voxel, along with the point cloud confidence scores 222, may be combined (e.g., averaged) to form centroids 213 of grid map voxels and centroid confidence scores 223. For example, points 232 of point cloud voxel 205 may be combined (e.g., averaged) to form centroid 233 of grid map voxel 206; and the point cloud confidence scores 222 of point cloud voxel 205 may be combined (e.g., averaged) to form centroid confidence score 243.

[0037] A point cloud may act as an intermediate step to calculate a height map. There inputs for each cloud point voxel may be: (i) a subset of the 3D point cloud, of all points inside the voxel of interest; and (ii) a confidence value for each point. The outputs for each cloud point voxel may be: (i) a centroid point with (x, y, z) coordinates; and (ii) a confidence score for thiscentroid point, indicating the level of certainty for that centroid point. In the example of Fig. 2, the inputs to point cloud voxel 205 are (i) 3D point cloud data points 232; and (ii) point cloud confidence scores 242; and the outputs of point cloud voxel 205 are (i) centroid 233 with an (x, y, z) coordinate; and (ii) centroid confidence score 243, indicating the level of certainty for centroid 233.

[0038] In the example of Fig. 2, calculating centroid confidence score 243 uses the point cloud confidence scores 242, which is a subset of point cloud confidence scores 222. Point cloud confidence scores 242 may be associated with a point cloud data points 232, which may be a subset of point cloud data points 212 that are included in point cloud voxel 205. Calculating centroid 233 uses point cloud data points 232, which is a subset of point cloud points 212. Centroid 233 may be computed using a standard average, a weighted average, or a median of the point cloud data points 232.

[0039] Centroids may be determined on a per voxel basis. For each voxel (or cell), the centroid point coordinates can be computed by using an average of all the points (e.g., 3D coordinates of a list of 3D coordinates) that belong to this voxel (e.g., on a per voxel basis). The average can be determined with a standard average, a weighted average, a median, or / and combined with thresholds (e.g., based on the confidence value of each point), or any other statistical averaging method. For instance, the centroid point coordinates for each voxel can be a standard average value of the point cloud coordinates included in the voxel. Alternatively, the centroid point coordinates can be determined with an average value of the point cloud coordinates weighted by the confidence value of each point. As another example, the centroid point coordinates can be a median, which may refrain from using the confidence value, but may eliminate outliers. Point cloud data points 212 with point cloud confidence scores that satisfy a threshold may be removed from the subset of point cloud data points used to calculate centroids for a voxel. For example, if a point cloud confidence score is too uncertain, the corresponding point cloud data point may be eliminated before calculating the average of the subset of point cloud data points. Centroid confidence scores (e.g., values) are determined on a per voxel basis with regards to the set of voxels. The confidence score for the centroid point can be an average value of the confidence values of the subset of the 3D point cloud or a median value of the confidencevalues of the subset of the 3D point cloud. The grid map, then, may include centroid points with associated confidence scores for each grid map voxel.

[0040] Figure 3 illustrates an example of transforming grid map column 301 into height map column 302. The stereo camera system may convert a grid map to a height map by converting each voxel column of the grid map to a height at an (x,y) coordinate of a height map. Grid map column 301 may include all the voxels (with associated centroids 321, 331, 341, and 351 and centroid confidence scores 322, 332, 342, and 352) sharing the same horizontal plane (x,y) coordinates of the grid map. Within a column, there may be both empty voxels and filled voxels. An empty voxel may not have any centroid point (nor associated centroid confidence score). In specific embodiments, the empty voxel may not have any point cloud data points (nor associated point cloud confidence scores). In specific embodiments, the empty voxel may have one or more point cloud data points but the combined confidence scores of these point cloud data points may be below a threshold value such that the system disregards these point cloud data points and considers the voxel to be empty. A filled voxel may have a centroid point (and associated confidence score). In specific embodiments, the filled voxel may have one or more corresponding point cloud data points (and associated point cloud confidence scores). In specific embodiments, the combined confidence scores of the one or more corresponding point cloud data points may be above a threshold value. In the example of Fig. 3, voxel 310 is empty and voxels 320, 330, 340, and 350 are filled. Although grid map column 301 is shown with five voxels, a grid map column may include any number of voxels. Similarly, although a single grid map column and a single height map column are shown, the associated grid map and height map may include any number of columns.

[0041] The stereo camera system may determine that one or more voxels at the top of the column may be empty while one or more voxels lower than those voxels may be filled. The highest filled voxel may be detected. In the example of Fig. 3, voxel 310 is the highest voxel and is empty; accordingly, voxel 320 is the highest filled voxel. The height of the corresponding (x,y) coordinate of the height map column may be determined based on the height of the highest centroid (centroid 321) of that column or an average of the heights of a plurality of highest centroid points of that column. Whether the highest centroid determines the height of theheight map column or whether the plurality of highest centroid determines the height may depend on the physical surface the coordinate of the height map refers to. For example, if a height map column corresponds to the middle of a physical object, only one centroid point will be present on the top surface, the inner part of the physical object being occulted by the top surface. Conversely, if the height map column corresponds to an edge of the physical object, a plurality of centroid points (in a plurality of voxels) will be present on a vertical surface. In order to reduce the noise in these situations, the height value of the column may be determined using an average of the heights of topmost centroid points of the column. The average value can be a standard average, a weighted average (using the confidence scores) or a median. When multiple centroid points are combined, an average confidence score can be determined. The average confidence score may allow for employing further noise reduction based on the height map values, for example by applying a noise reducing filter.

[0042] The confidence scores of the grid map centroids allow higher accuracy and reduced noise in the conversion to the height map. Without confidences scores (e.g., by only searching the highest voxel that is filled by points in a column of voxels in the grid map), the corresponding height map may be prone to errors, as outliers of 3D points tend to overestimate the heights significantly. Instead of determining the height by only determining the first (e.g., highest) occupied voxel in a column, the height of the column can be determined by using the estimated centroid with its own estimated confidence from one or multiple contiguous voxels in the column. For instance, the height value of each column of cells can be computed by the weighted averaged confidence value of each cell, using the confidence calculated previously. For example, the height value of height map column 302 may be computed by the weighted averaged confidence value of centroid confidence scores 322, 332, 342, and 352 (corresponding to each filled voxel 320, 330, 340, and 350, respectively). Hence, rather than considering only the maximum voxel filled by points, the height map may be a computed value based on multiple (e.g., all, the top four) filled voxels of the column. This allows for the reduction of outliers of 3D points that are inherent to the stereo depth estimation.

[0043] In specific embodiments, a certain quantity of voxels in a grid map column may be considered to calculate the height of the corresponding height map column. For example, thetop three voxels may be averaged together, rather than every voxel in the column. In the example of Fig. 3, centroid 321 (with height 323), centroid 331 (with height 333), and centroid341 (with height 343) may be averaged (e.g., weighted by their confidence scores 322, 332, and342 respectively) to determine height 303 of height map column 302. In this example, centroid 351 and centroid confidence score 352 may not be considered in the calculation of the height of height map column 302, as voxel 350 is not one of the top three filled voxels. The averaging of the top three filled voxels is merely an example, and the height of the height map column may be calculated using information from any quantity of voxels. In specific embodiments, height 303 of height map column 302 may be determined based on height 323 of centroid 321 in the highest filled voxel 320 and not on other centroids. In specific embodiments, determining the highest filled voxel may include determining that the highest filled voxel has a centroid point or one or more point cloud data points having an associated confidence score above a threshold and that at least one voxel above the highest filled voxel does not include a centroid point or one or more point cloud data points with an associated confidence score above the threshold.

[0044] A grid map may act as an intermediate step for calculating a height map. The input of the grid map may include centroid points with (x, y, z) coordinates and associated confidence scores. The output of the grid map may be a height map with a height value for each column (x,y) and an associated confidence score. For example, grid map column 301 may input centroids 321, 331, 341, and 351 with heights 323, 333, 343, and 353 and associated confidence scores 322, 332, 342, and 352. Grid map column 301 may output height 303 of height map column 302 and height confidence score 304.

[0045] A column of a grid map includes all the voxels (centroid points and confidence scores) sharing the same horizontal plane (x,y) coordinates. Within a column, there are some empty and / or filled voxels. An empty voxel doesn't have any centroid point. A filled voxel has a centroid point (and associated confidence score). For instance, voxel 310 is empty, and voxels 320, 330, 340 and 350 are filled (having the centroids 321, 331, 341, and 351 respectively).

[0046] A filled voxel can be identified by either looking at the confidence score of a single point in the point cloud or the confidence score of many points summed together and comparingthese to a threshold. Both tests can be used alone or in combination. For example, if a single point in the voxel has a 90% confidence then the voxel may be considered to be filled (e.g., if the threshold is 90% or lower). The single point may be a centroid or may be a point cloud data point. Alternatively, if there are three separate points (e.g., point cloud data points) within a voxel that each have 30% confidence scores, then the voxels may be considered to be filled (e.g., if the threshold is 90% or lower).

[0047] The first filled voxel may be determined by analyzing (e.g., reading data from) voxels in the column, starting from the top of the column of interest. Once a non-empty voxel is detected, the height value can be determined based on the height of the highest centroid (e.g., height 323) or an average of the heights of a plurality of highest centroids (e.g., heights 323 and 333). If grid map column 301 corresponds to an edge of the physical object, a plurality of centroid points (in a plurality of voxels) will be present on a vertical surface. It has been observed that the noise in this situation tends to be higher. In order to reduce the noise, it is possible to determine the height value by using an average of the heights of the topmost centroid points (e.g., centroids 321 and 331 or the topmost centroids of multiple nearby grid map columns). The average value can be a standard average, a weighted average (using the confidence scores) or a median. By calculating the height of the height map column using one or more centroid confidence scores, there may be a noise reduction effect. For example, voxels with low confidence may be eliminated. For example, some heights may be eliminated before calculating the average value if the confidence scores of the heights are too uncertain (e.g., satisfy a threshold). When multiple centroid points are combined, an average confidence score can be determined (e.g., confidence score 304 of height 303).

[0048] Each column of the height map may be calculated using the associated grid map column. In specific embodiments, each column of the height map may be calculated using the associated grid map column as well as other grid map columns close to the associated grid map column. The height map (with confidence scores) may be converted to a filtered height map with reduced noise. In specific embodiments, a bilateral filter can be used to reduce noise. The bilateral filter may use the previously computed height value and the previously computed confidence score value of each cell to create a smooth height map output. The output heightfor the cell may be a function of the input height for the cell, the confidence scores, and a normalization factor. In specific embodiments, a gaussian function may be used as a coefficient factor in the filter and the sigma value of the gaussian function may be based on the confidence scores previously calculated for each cell. The height value of a coordinate in the filtered height map may be influenced mainly by coordinates that are close spatially to, and that have a height similar to, that coordinate.

[0049] Figure 4 illustrates height map 400 converted to filtered height map 450. A stereo camera system may convert height map 400 (with confidence scores) to filtered height map 450 with reduced noise. Height map 400 and filtered height map 450 may be 2D maps (x,y coordinates) with height values (z coordinate). In addition, the confidence score associated with the centroid point may be advantageously provided along with height map 400, filtered height map 450, or both. The confidence scores may allow for employing further noise reduction based on the height map values by applying a noise reducing filter. Height map 400 may be referred to as an input height map; filtered height map 450 may be referred to as an output height map. In specific embodiments, a bilateral filter can be used to reduce noise. The bilateral filter may use the previously computed height value and the previously computed confidence score value of each cell or column to create a smooth height map output for filtered height map 450. The output height for the cell may be a function of the input height for the cell, the confidence scores of the cell, confidence scores of surrounding cells, and a normalization factor.

[0050] A bilateral filter, a filter in x and y directions, can be used to convert height map 400 into filtered height map 450. This filter may use the previously computed height value and the previously computed confidence score value of each cell to create a smooth height map output. An example formula for the filter is as follows in Equations 1 and 2:wherewhere H(s) is the output height for the cell, H(p) is the input height for the cell, fcis a coefficient factor directly based on the confidence scores and k(s) is the normalization factor. The conversion of height map 400 may use a gaussian function for fcin the spatial domain where the sigma value may be provided by the confidence scores previously calculated for each cell, and a Gaussian for g in the intensity / height domain. Therefore, the value at a pixel s may be influenced mainly by pixels (e.g., voxels, cells) that are close spatially and that have a similar height.

[0051] Figure 5 provides an example of method 500 for generating a height map for a physical surface from a point cloud in accordance with specific embodiments of the inventions disclosed herein. A physical surface may be the surface of any physical thing such as an object or the ground. The physical surface may have any orientation (e.g., a vertical, horizontal, or diagonal surface) and any geometry (e.g., straight, curved, irregular, textured, etc.). Method 500 may be implemented by a system such as a stereo camera system including a pair of cameras (or other imagers), one or more processors, and one or more non-transitory computer-readable media. The one or more non-transitory computer-readable media may store instructions that, when executed by the one or more processors, cause the stereo camera system to conduct method 500. Method 500 may be implemented by a system including means for performing the steps of method 500. Steps, or portions of steps, of method 500 may be duplicated, omitted, rearranged, or otherwise deviate from the form shown. Additional steps may be added to method 500. Steps, or portions of steps, of method 500 may be performed in series or parallel.

[0052] At step 502, the point cloud can be captured using a stereo camera system. The point cloud may comprise a list of 3D coordinates for a set of points of the physical surface. In specific embodiments, the point cloud may be from a region of interest. In specific embodiments, the point cloud may be divided into voxels that correspond to or match up with a set of voxels of the grid map. In specific embodiments, the set of points in the point cloud (e.g., point cloud data points) may be associated with a set of point cloud confidence scores.

[0053] In specific embodiments, at step 504, the set of confidence scores (e.g., centroid confidence scores) may be calculated using the set of point cloud confidence scores. In specificembodiments, step 504 may be a part of step 502 (e.g., capturing the point cloud), a part of step 512 (e.g., generating a grid map), or a separate step.

[0054] In specific embodiments and as part of calculating the set of confidence scores using the set of point cloud confidence scores, at step 506, a confidence score for a voxel in the set of voxels may be calculated. The confidence score may be calculated using a subset of point cloud confidence scores. The subset of point cloud confidence scores may be associated with a subset of points included in the voxel. The set of points may include the subset of points. The set of point cloud confidence scores may include the subset of point cloud confidence scores.

[0055] In specific embodiments and as part of calculating the confidence score for the voxel, at step 508, an average or a median of the subset of point cloud confidence scores may be calculated.

[0056] In specific embodiments, at step 510, the set of confidence scores may be determined using the set of point cloud confidence scores by averaging the set of confidence scores (e.g., point cloud confidence scores) for the set of points on a per voxel basis with regards to the set of voxels. Methods of calculating a single confidence score may be repeated (e.g., parallelized) for multiple confidence scores. In specific embodiments, step 518 may be a part of step 506 (e.g., calculating the set of confidence scores), a part of step 512 (e.g., generating a grid map), or a separate step.

[0057] At step 512, a grid map may be generated from the point cloud. The grid map may comprise a set of centroid points for a set of voxels of the grid map. The centroid points may be centroids of the set of points in each voxel of the set of voxels. The set of centroid points may be associated with a set of confidence scores (e.g., calculated at step 512).

[0058] In specific embodiments and as part of generating the grid map, at step 514, the region of interest (e.g., associated with the point cloud) may be sampled using the set of voxels.

[0059] In specific embodiments and as part of generating the grid map, at step 516, a centroid for a voxel may be computed by using a subset of points (e.g., point cloud data points) that is included in the voxel. The set of points may include the subset of points. The centroid may be computed using a standard average, a weighted average, or a median of the subset of points.

[0060] In specific embodiments and as part of generating the grid map, at step 518, the set of centroid points may be determined by averaging the 3D coordinates of the list of 3D coordinates for the set of points on a per voxel basis with regards to the set of voxels. Methods of calculating a single centroid point may be repeated (e.g., parallelized) for multiple centroid points.

[0061] At step 520, the height map may be generated from the grid map. The height map may comprise a set of height values derived using the set of confidence scores.

[0062] In specific embodiments and as part of generating the height map from the grid map, at step 522, a highest filled voxel of a column of voxels may be determined. In specific embodiments, either step 524 or step 526 may be performed as part of step 522.

[0063] In specific embodiments and as part of determining the highest filled voxel of a column of voxels, at step 524, at least one of the following may be determined to be above a threshold: (i) a confidence score from the set of point cloud confidence scores associated with a point that is in the highest filled voxel and is from the set of points; and (ii) a sum of a subset of confidence scores from the set of point cloud confidence scores that are associated with a subset of points that are in the highest filled voxel and are from the set of points. The set of points in the point cloud may be associated with a set of point cloud confidence scores.

[0064] In specific embodiments, and as part of determining the highest filled voxel, at step 526, the highest filled voxel may be determined to have a centroid point having an associated confidence score above a threshold. At least one voxel above the highest filled voxel may be determined to not include a centroid point with an associated confidence score above the threshold. In specific embodiments, the highest filled voxel may include a centroid point. Voxels above (e.g., in the positive z-direction) the highest filled voxel may not include centroid points. In specific embodiments, voxels below (e.g., in the negative z-direction) the highest filled voxel may include centroid points.

[0065] In specific embodiments and as part of generating the height map from the grid map, at step 528, the height value of the column of voxels may be calculated based on a height of the highest filled voxel.

[0066] In specific embodiments and as part of generating the height map from the grid map, at step 530, the height value of the column of voxels may be calculated based on a plurality of heights of a plurality of voxels in the column. The plurality of heights may include a height of the highest filled voxel. In specific embodiments, the heights of the plurality of voxels may be based on their associated centroids. In specific embodiments, either step 528 or step 530 may be performed based on the geometry of the physical surface.

[0067] In specific embodiments and as part of generating the height map, at step 532, a noise reduction filtering may be conducted on the grid map using the set of confidence scores (e.g., calculated at step 504).

[0068] In specific embodiments and as part of conducting the noise reduction filtering, at step 534, a noise reduction filter may be used. In specific embodiments, the noise reduction filter may be a bilateral filter. In specific embodiments, the noise reduction filter may filter in the x and y directions.

[0069] In specific embodiments and as part of conducting the noise reduction filtering, at step 536, a gaussian function may be used for a coefficient factor of the noise reduction filter. The gaussian function may be based on the set of confidence scores. In specific embodiments, the noise filtering uses a normalization factor. In specific embodiments, the value of the height of a column of voxels in the height map may be based on voxels that are close to the column of voxels and that have a similar height. Method 500 may result in a height map with improved accuracy and reduced noise.

[0070] Figure 6 illustrates examples of determining whether a voxel is filled in accordance with specific embodiments of the inventions disclosed herein. In specific embodiments, a voxel can be determined to be filled based on the confidence score of a single point in the point cloud. In specific embodiments, a voxel can be determined to be filled based on the confidence score of many points summed together. Both methods can be used alone or in combination; both the confidence score of a single point and the confidence scores of multiple points may be compared to respective thresholds and the voxel may be determined to be filled if at least one of the thresholds is satisfied. In specific embodiments, the two thresholds are the same; in specific embodiments, the two thresholds are different.

[0071] Determining a highest filled voxel may include determining that a confidence score or a sum of confidence scores is above a threshold. For example, a highest filled voxel may be determined by determining that a confidence score from a set of point cloud confidence scores associated with a point (e.g., a point cloud data point) that is in the supposed highest filled voxel is above a threshold. As another example, a highest filled voxel may be determined by determining that a sum of a subset of confidence scores from the set of point cloud confidence scores that are associated with a subset of points that are in the supposed highest filled voxel are above a threshold. In specific embodiments, determining the highest filled voxel may include determining that the highest filled voxel has a centroid point (e.g., rather than a point cloud data point) having an associated confidence score above a threshold and that at least one voxel above the highest filled voxel does not include a centroid point with an associated confidence score above the threshold.

[0072] Point cloud voxel 600 may include data point 601 with confidence score 602, data point 603 with confidence score 604, and data point 605 with confidence score 606. Confidence score 602 may be 40%, confidence score 604 may be 20% and confidence score 606 may be 30%. The combined (e.g., summed) confidence score for the voxel may be 90%. The threshold for a filled voxel may be 90%, such that the combined confidence score meets the threshold. Accordingly, point cloud voxel 600 may be determined to be filled.

[0073] Point cloud voxel 610 may include centroid point 611 with confidence score 612. Confidence score 612 may be 92%. The threshold for a filled voxel may be 90%, such that confidence score 612 meets or exceeds the threshold. Accordingly, point cloud voxel 610 may be determined to be filled.

[0074] Point cloud voxel 620 may include data point 621 with confidence score 622, data point 623 with confidence score 624, and data point 625 with confidence score 626. Confidence score 622 may be 40%, confidence score 624 may be 17% and confidence score 626 may be 25%. The combined (e.g., summed) confidence score for the voxel may be 82%. The threshold for a filled voxel may be 90%, such that the combined confidence score does not meet (e.g., or exceed) the threshold. Accordingly, point cloud voxel 620 may be determined to be empty (e.g., not filled).

[0075] Point cloud voxel 630 may include data point 631 with confidence score 632. Data point 631 may be the only point cloud data point in the voxel. Confidence score 632 may be 75%. The threshold for a filled voxel may be 90%, such that confidence score 632 does not meet (e.g., or exceed) the threshold. Accordingly, point cloud voxel 630 may be determined to be empty (e.g., not filled).

[0076] Point cloud voxel 640 may include centroid point 641 with confidence score 642. Confidence score 642 may be 70%. The threshold for a filled voxel may be 90%, such that confidence score 642 does not meet (e.g., or exceed) the threshold. Accordingly, point cloud voxel 640 may be determined to be empty (e.g., not filled).

[0077] The "filled" status of point cloud voxels 600, 620, and 630 may be determined without (e.g., or prior to) determining voxel centroids or centroid confidence values. That is, in specific embodiments, it is not necessary to determine centroids or centroid confidence values in order to determine whether a voxel is filled or empty. In specific embodiments, a height map may be generated from a grid map without determining centroids or centroid confidence values. For example, determining a highest filled voxel of a column of voxels, calculating a height value of the column of voxels, and conducting a noise reduction filtering on the grid map may be performed without determining centroids or centroid confidence values. Although the confidence scores are added together to form the combined confidence score to compare to the threshold in the example of Fig. 6, the confidence scores may be combined in other ways. For example, the combined confidence score can be determined with a standard average, a weighted average, a median, or / and combined with other thresholds, or any other statistical averaging method.

[0078] At least one processor in accordance with this disclosure can include at least one non- transitory computer readable media. The at least one processor could comprise at least one computational node in a network of computational nodes. The media could include cache memories on the processor. The media can also include shared memories that are not associated with a unique computational node. The media could be a shared memory, could be a shared random-access memory, and could be, for example, a DDR DRAM. The shared memory can be accessed by multiple channels. The non-transitory computer readable mediacan store data required for the execution of any of the methods disclosed herein, the instruction data disclosed herein, and / or the operand data disclosed herein. The computer- readable media can also store instructions which, when executed by the system, cause the system to execute the methods disclosed herein. The concept of executing instructions is used herein to describe the operation of a device conducting any logic or data movement operation, even if the "instructions" are specified entirely in hardware (e.g., an AND gate executes an "and" instruction). The term is not meant to impute the ability to be programmable to a device.

[0079] While the specification has been described in detail with respect to specific embodiments of the invention, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily conceive of alterations to, variations of, and equivalents to these embodiments. Any of the method steps discussed above can be conducted by a processor operating with a computer-readable non-transitory medium storing instructions for those method steps. The computer-readable medium may be memory within a personal user device or a network accessible memory. Although examples in the disclosure were generally directed to cameras, any imager may be used. These and other modifications and variations to the present invention may be practiced by those skilled in the art, without departing from the scope of the present invention, which is more particularly set forth in the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A method (500) for generating a height map (104; 400; 450) for a physical surface from a point cloud (102; 201), the method (500) comprising: capturing the point cloud (102; 201) using a stereo camera system (111), wherein the point cloud (102; 201) comprises a list of three-dimensional coordinates for a set of points (112; 212) on the physical surface; generating a grid map (103; 202) from the point cloud (102; 201), wherein the grid map (103; 202) comprises a set of centroid points (113; 213) for a set of voxels (206; 310; 320) of the grid map (103; 202), the centroid points (113; 213) are centroids of the set of points (112; 212) in each voxel of the set of voxels (206; 310; 320), and the set of centroid points (113; 213) are associated with a set of confidence scores (123; 223); and generating the height map (104; 400; 450) from the grid map (103; 202), wherein the height map (104; 400; 450) comprises a set of height values (131; 303) derived using the set of confidence scores (123; 223).

2. The method (500) of claim 1, wherein: the set of points (112; 212) in the point cloud (102; 201) are associated with a set of point cloud confidence scores (122; 222); and the method (500) further comprises calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222).

3. The method (500) of claim 2, wherein calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222) comprises: calculating a confidence score (123; 223) for a voxel in the set of voxels (206; 310; 320) using a subset of point cloud confidence scores (242), the subset of point cloud confidence scores (242) being associated with a subset of points (232) included in the voxel (206; 310; 320), wherein the set of points (112; 212) includes the subset of points (232) and the set ofpoint cloud confidence scores (122; 222) includes the subset of point cloud confidence scores (242).

4. The method (500) of claim 3, wherein calculating the confidence score (123; 223) for the voxel (206; 310; 320) comprises: calculating an average or a median of the subset of point cloud confidence scores (242).

5. The method (500) of claim 1, wherein generating the grid map (103; 202) comprises: computing a centroid for a voxel (206; 310; 320) by using a subset of points (232) that is included in the voxel (205; 206; 310; 320), the set of points (112; 212) including the subset of points (232), wherein the centroid (233) is computed using a standard average, a weighted average, or a median of the subset of points (232).

6. The method (500) of claim 1, wherein: the set of points (112; 212) in the point cloud (102; 201) are associated with a set of point cloud confidence scores (122; 222); generating the height map (104; 400; 450) from the grid map (103; 202) comprises determining a highest filled voxel (320) of a column of voxels (310; 320; 330; 340; 350); and determining the highest filled voxel (320) comprises determining that at least one of the following is above a threshold: (i) a confidence score (123; 223) from the set of point cloud confidence scores (123; 223) associated with a point (321; 631) that is in the highest filled voxel (320) and is from the set of points (112; 212); and (ii) a sum of a subset of confidence scores (242) from the set of point cloud confidence scores (122; 222) that are associated with a subset of points (232) that are in the highest filled voxel (320) and are from the set of points (112; 212).

7. The method (500) of claim 1, wherein: generating the height map (104; 400; 450) from the grid map (103; 202) comprises determining a highest filled voxel (320) of a column of voxels (301); anddetermining the highest filled voxel (320) comprises determining that the highest filled voxel (320) has a centroid point (113; 321) having an associated confidence score (322) above a threshold and that at least one voxel (310) above the highest filled voxel (320) does not include a centroid point with an associated confidence score above the threshold.

8. The method (500) of claim 7, wherein generating the height map (104; 400; 450) from the grid map (103; 202) further comprises: calculating a height value (131; 303) of the column of voxels (301) based on a height (323) of the highest filled voxel (320).

9. The method (500) of claim 7, wherein generating the height map (104; 400; 450) from the grid map (103; 202) further comprises: calculating a height value (131; 303) of the column of voxels (301) based on a plurality of heights (323; 333; 343; 353) of a plurality of voxels (310; 320; 330; 340; 350) in the column (301), the plurality of heights (323; 333; 343; 353) including a height (323) of the highest filled voxel (320).

10. The method (500) of claim 1, wherein the point cloud (102; 201) is from a region of interest (101) and generating the grid map (103; 202) comprises: sampling the region of interest (101) using the set of voxels (205; 206; 310; 320); and determining the set of centroid points (113; 213) by averaging the three-dimensional coordinates of the list of three-dimensional coordinates for the set of points (112; 212) on a per voxel basis with regards to the set of voxels (205; 206; 310; 320).

11. The method (500) of claim 10, wherein the point cloud (102; 201) includes a set of point cloud confidence scores (122; 222) for the set of points (112; 212) and generating the grid map (103; 202) further comprises: determining the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222) by averaging the set of point cloud confidence scores (122;222) for the set of points (112; 212) on a per voxel basis with regards to the set of voxels (205; 206; 310; 320).

12. The method (500) of claim 1, wherein generating the height map (450) further comprises: conducting a noise reduction filtering on the grid map (103; 202) using the set of confidence scores (123; 223).

13. The method (500) of claim 12, wherein conducting the noise reduction filtering comprises: using a noise reduction filter; and using a gaussian function for a coefficient factor of the noise reduction filter, the gaussian function being based on the set of confidence scores (123; 223).

14. A stereo camera system (111) comprising: a pair of cameras; one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the stereo camera system (111) to conduct a method (500) for generating a height map (104; 400; 450) for a physical surface from a point cloud (102; 201), the method (500) comprising: capturing the point cloud (102; 201) using the stereo camera system (111), wherein the point cloud (102; 201) comprises a list of three-dimensional coordinates for a set of points (112; 212) on the physical surface; generating a grid map (103; 202) from the point cloud (102; 201), wherein the grid map (103; 202) comprises a set of centroid points (113; 213) for a set of voxels (206;310; 320) of the grid map (103; 202), the centroid points (113; 213) are centroids of theset of points in each voxel of the set of voxels (206; 310; 320), and the set of centroid points (113; 213) are associated with a set of confidence scores (123; 223); and generating the height map (104; 400; 450) from the grid map (103; 202), wherein the height map (104; 400; 450) comprises a set of height values (131; 303) derived using the set of confidence scores (123; 223).

15. The stereo camera system (111) of claim 14, wherein: the set of points (112; 212) in the point cloud (102; 201) are associated with a set of point cloud confidence scores (122; 222); and the method (500) further comprises calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222).

16. The stereo camera system (111) of claim 15, wherein calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222) comprises: calculating a confidence score (123; 223) for a voxel in the set of voxels (206; 310; 320) using a subset of point cloud confidence scores (242), the subset of point cloud confidence scores (242) being associated with a subset of points (232) included in the voxel (205; 206; 310; 320), wherein the set of points (112; 212) includes the subset of points (232) and the set of point cloud confidence scores (122; 222) includes the subset of point cloud confidence scores (242).

17. The stereo camera system (111) of claim 16, wherein calculating the confidence score (123; 223) for the voxel (206; 310; 320) comprises: calculating an average or a median of the subset of point cloud confidence scores (242).

18. The stereo camera system (111) of claim 14, wherein generating the grid map (103; 202) comprises: computing a centroid for a voxel (206; 310; 320) by using a subset of points (232) that is included in the voxel (205; 206; 310; 320), the set of points (112; 212) including the subsetof points (232), wherein the centroid (233) is computed using a standard average, a weighted average, or a median of the subset of points (242).

19. The stereo camera system (111) of claim 14, wherein: generating the height map (104; 400; 450) from the grid map (103; 202) comprises determining a highest filled voxel (320) of a column of voxels (301); and determining the highest filled voxel (320) comprises determining that the highest filled voxel (320) has a centroid point (321) having an associated confidence score (322) above a threshold and that at least one voxel (310) above the highest filled voxel (320) does not include a centroid point with an associated confidence score above the threshold.

20. The stereo camera system (111) of claim 19, wherein generating the height map (104;400; 450) from the grid map (103; 202) further comprises: calculating a height value (131; 303) of the column of voxels (301) based on a height (323) of the highest filled voxel (320).

21. The stereo camera system (111) of claim 19, wherein generating the height map (104;400; 450) from the grid map (103; 202) further comprises: calculating a height value (131; 303) of the column of voxels (301) based on a plurality of heights (323; 333; 343; 353) of a plurality of voxels (310; 320; 330; 340; 350) in the column (301), the plurality of heights (323; 333; 343; 353) including a height (323) of the highest filled voxel (320).

22. The stereo camera system (111) of claim 14, wherein the point cloud (102; 201) is from a region of interest (101) and generating the grid map (103; 202) comprises: sampling the region of interest (101) using the set of voxels (205; 206; 310; 320); and determining the set of centroid points (113; 213) by averaging the three-dimensional coordinates of the list of three-dimensional coordinates for the set of points (112; 212) on a per voxel basis with regards to the set of voxels (205; 206; 310; 320).

23. The stereo camera system (111) of claim 22, wherein the point cloud (102; 201) includes a set of point cloud confidence scores (122; 222) for the set of points (112; 212) and generating the grid map (103; 202) further comprises: determining the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222) by averaging the set of point cloud confidence scores (122; 222) for the set of points (112; 212) on a per voxel basis with regards to the set of voxels (205; 206; 310; 320).

24. The stereo camera system (111) of claim 14, wherein generating the height map (450) further comprises: conducting a noise reduction filtering on the grid map (103; 202) using the set of confidence scores (123; 223).

25. The stereo camera system (111) of claim 24, wherein conducting the noise reduction filtering comprises: using a noise reduction filter; and using a gaussian function for a coefficient factor of the noise reduction filter, the gaussian function being based on the set of confidence scores (123; 223).

26. One or more non-transitory computer-readable media storing instructions, which when executed by one or more processors cause the one or more processors to conduct a method (500) for generating a height map (104; 400; 450) for a physical surface from a point cloud (102; 201), the method (500) comprising: capturing the point cloud (102; 201) using a stereo camera system (111), wherein the point cloud (102; 201) comprises a list of three-dimensional coordinates for a set of points (112; 212) on the physical surface;generating a grid map (103; 202) from the point cloud (102; 201), wherein the grid map (103; 202) comprises a set of centroid points (113; 213) for a set of voxels (206; 310; 320) of the grid map (103; 202), the centroid points (113; 213) are centroids of the set of points (112; 212) in each voxel of the set of voxels (206; 310; 320), and the set of centroid points (113; 213) are associated with a set of confidence scores (123; 223); and generating the height map (104; 400; 450) from the grid map (103; 202), wherein the height map (104; 400; 450) comprises a set of height values (131; 303) derived using the set of confidence scores (123; 223).

27. The one or more non-transitory computer-readable media of claim 26, wherein: the set of points (112; 212) in the point cloud (102; 201) are associated with a set of point cloud confidence scores (122; 222); and the method (500) further comprises calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores (122; 222).

28. The one or more non-transitory computer-readable media of claim 27, wherein calculating the set of confidence scores (123; 223) using the set of point cloud confidence scores comprises (122; 222): calculating a confidence score (123; 223) for a voxel (205; 206) in the set of voxels (205; 206; 310; 320) using a subset of point cloud confidence scores (242), the subset of point cloud confidence scores (242) being associated with a subset of points (232) included in the voxel (205; 206), wherein the set of points (112; 212) includes the subset of points (232) and the set of point cloud confidence scores (122; 222) includes the subset of point cloud confidence scores (242).

29. The one or more non-transitory computer-readable media of claim 28, wherein calculating the confidence score (123; 223) for the voxel (205; 206) comprises: calculating an average or a median of the subset of point cloud confidence scores (242).

30. The one or more non-transitory computer-readable media of claim 26, wherein generating the grid map (103; 202) comprises: computing a centroid for a voxel (206; 310; 320) by using a subset of points (232) that is included in the voxel (205; 206), the set of points (112; 212) including the subset of points (232), wherein the centroid (233) is computed using a standard average, a weighted average, or a median of the subset of points (232).

Images