Systems and methods for air-to-ground registration

Abstract

The present invention relates to systems and methods for aerial-to-ground registration. Systems and methods for registering aerial and ground data include locating rigid features such as walls in both the aerial and ground data, registering the ground rigid data to the aerial rigid data, and transforming the ground data using the transformation from the registration, including breaking the data into sectors and aligning the sectors. Distortions in the ground data are adjusted.

Description

[0001] This application is a divisional application of patent application No. 202180043623.5 (PCT / US2021 / 070718), filed on December 19, 2022, with an international filing date of June 16, 2021, entitled "System and Method for Air-to-Ground Registration".

[0002] Cross-referencing of related applications

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 040,789, filed on June 18, 2020, entitled “SYSTEM AND METHOD FOR AERIAL TOGROUND REGISTRATION (Agent No. AA249)”. Background Technology

[0004] A point cloud is a set of points collected in 3D space from sensors such as LiDAR sensors. Point cloud registration is the process of aligning two or more 3D point clouds of the same scene, thus enabling the combination of data from different sources to create a richer dataset than if a single sensor were used. A 3D scene can be created during the registration process. A typical registration process involves: filtering sensor data, determining the location of points within the dataset, and aligning / stitching data from different parts of the dataset together. Common registration types include rigid, affine, and non-rigid. The chosen registration type determines which transformation is applied to the points in the point cloud. For example, rigid registration applies the same transformation to all points in the point cloud, while non-rigid registration applies a transformation to points based on a displacement field. Common rigid registration methods include Normal Distribution Transform (NDT) and Iterative Closest Point (ICP), while common general-purpose registration methods include Coherent Point Shift (CPD).

[0005] Registering ground-to-aerial data can present numerous challenges. Camera viewpoints and imaging conditions may differ. Aerial images are typically captured from greater distances and at different angles compared to ground images. The direction of sunlight can affect the appearance of landmarks in aerial images, making feature matching challenging. Automatically matching and reconstructing aerial and ground images without any human intervention can be even more challenging. Some successes have been achieved in matching cropped aerial photographs with composite views from ground images. Georegistration of ground-level models can be performed using image data. GPS and text labels can be used to roughly estimate locations. Edges in aerial imagery can be mapped to a two-dimensional projection of ground information, but there are issues with the number of edges needed to map these edges in the ground information. Some studies use geolocated ground images with street view imagery. Each of these processes suffers from location error problems. Given the significant geographical and temporal variations, high-precision feature matching can be extremely time-consuming and unsuitable for the rapid decision-making processes of autonomous vehicles. Summary of the Invention

[0006] The system and method taught in this paper can be used to combine aerial and ground data to create a dataset suitable for autonomous vehicles. Aerial data can be considered more reliable than ground data for reasons including, but not limited to, the possibility that GPS providing position data to ground data sensors may be blocked. Therefore, by locating the same features in both aerial and ground data, the location of the aerial data features can be used to correct the ground data. Thus, the input to the process of this teaching is two point clouds, one aerial and one ground, and the output from the process of this teaching is a unified dataset of aerial and ground data.

[0007] The aerial-to-ground approach taught in this paper can include, but is not limited to, collecting both ground point cloud data and aerial point cloud data from the same geographic area. Ground point cloud data can have a higher point density than aerial point cloud data, but optimally, the point densities of the aerial and ground datasets should be consistent. Filters, such as, but not limited to, traditional voxel grid filters, can be applied to both datasets to achieve consistent point density. Regardless of the filter chosen, the point cloud provided to the filter can be divided into sectors. If a sector does not contain at least a pre-selected threshold number of points, that sector can be discarded. The pre-selected threshold can be dependent on the filter and / or on the application. Traditional voxel grid filters can provide a point cloud with fewer points than the original point cloud, representing the point cloud as a whole. A voxel is a unit of graphical information that defines a point in 3D space and represents a single sample on a regularly spaced 3D grid. A voxel can include a single data point or multiple data points, such as color and opacity. Voxel grid filters can spatially average the points in the point cloud by dividing the point cloud into a regular voxel grid and combining the points within a voxel into an output point. The assembly point can be determined by methods such as locating the centroid or spatial average of the voxels, or the geometric center of the voxels. The choice of assembly method can be based on, for example, the relative importance of accuracy to computational intensity.

[0008] The air-to-ground approach can include surface localization in ground PC data. In some configurations, surface points can be determined by removing elevated features from the point cloud, thus leaving the surface points intact. In other configurations, conventional progressive morphological filters can be used to determine surface points.

[0009] Point cloud data can include deformations, such as warping, which can cause registration failures. To address this, surface data can be decomposed into sectors that are small enough to linearly approximate deformations like warping, but large enough to keep the number of intersections between sectors requiring smoothing to a minimum. If sectors are too small, each sector may not contain any surface points at all. Sectors can range in size from 75m x 75m to 125m x 125m. Once the sectors are determined, alignment can be performed. In some configurations, sector alignment can be achieved using a traditional Normal Distribution Transform (NDT) process. Zaganidis et al., Semantic-assisted 3D Normal Distributions Transform for scan registration in environments with limited structure, Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), pp. 4064-4069, 9 / 2017. Other alignment processes can also be used, especially when the point cloud data distribution is sparse. If NDT is used, the first step in alignment is to compute the NDT for each sector. NDT can transform surface data into a set of normal distributions representing the local structure of points and can register distributions rather than matching individual scan points across sectors. To determine the distance between sectors, the minimum distance to / from each point in the first sector to / from points in neighboring sectors is computed. If the minimum distance is below a pre-selected threshold, the average distance across all measurements is determined. Sectors are considered aligned when the average distance is less than a pre-selected threshold in the range of approximately 0.5 to 1.5 m, for example, 0.85 m. When the average distance is greater than or equal to the pre-selected threshold, each sector is assigned Universal Lateral Mercator (UTM) coordinates, and an NDT transform is calculated for all corners of the sector data. Some sectors may be aligned with each other. To determine which sectors are still not aligned with their neighbors, the distance between a sector edge and its neighbor's edge can be determined. If the distance exceeds the pre-selected threshold, the sector is not aligned. The pre-selected threshold can include a range of 0.5 m to 1.5 m, for example, 0.85 m. These sectors can be sorted according to the number of their aligned neighbors. By using the NDT transform associated with sectors that are nearly aligned with their unaligned neighbors, the sector with the most aligned neighbors can be fully aligned with its remaining (or more) unaligned neighbors(s).If more than one NDT transform can be used, the average of the NDT transforms is taken and used. The sorting and alignment can be re-performed until no sectors remain to be aligned. This process can result in aligned sectors that include filtered surface data from which wall features can be extracted.

[0010] To extract wall features, surface points within aligned sectors can undergo a feature determination process, such as, but not limited to, the traditional convex hull process. The convex hull process can begin by selecting the point in the dataset with the smallest angle between the point and the vertical axis extending through the selected starting point. A line is drawn between the starting point and the selected point. The next point in the dataset with the next smallest angle between the starting point and the line between the selected and the starting point is selected, and a line is drawn connecting the selected point and the next point. This process is followed until the desired number of polygons are formed. Extracting wall features from the polygons can include comparing the angles formed by the potential walls with the horizontal axis. Polygons that slope too far towards the ground are not included in the polygon set representing the walls.

[0011] The filtered aerial point cloud data can include convex shapes that represent roof and surface features. Roof and surface features can be determined using, for example, but not limited to, traditional convex hull algorithms. Edges of roof and surface features can be connected to infer wall features. Registration between walls found in ground data and walls found in aerial data can produce transformations that can be applied to ground point cloud data, thus forming a unified aerial / ground point cloud dataset.

[0012] A method for registering 3D ground point cloud (PC) data to 3D aerial PC data may include, but is not limited to, locating surfaces in the 3D ground PC data; dividing the 3D ground PC data with the located surfaces into members of a sector set; aligning the members with each other; locating similar features in the 3D aerial PC data and the aligned members in the sector set; registering the similar features in the 3D ground PC data with the 3D aerial PC data; and creating a transformation based on the registration. The aligned sectors may optionally include: determining the coordinates and coordinate transformations of each corner of each member of the sector set; and determining a first member among the members based on the coordinates. The first member may be adjacent to but not aligned with a second member among the members. The aligned sectors may include: applying the coordinate transformation associated with the first member to the coordinates of the second member. The method may optionally include: applying a combination of multiple coordinate transformations to the second member based at least on a third member among the members. The method may optionally include: filtering the 3D aerial PC data and the 3D ground PC data. The method may optionally include using a voxel filter to filter the 3D aerial PC data and the 3D ground PC data.

[0013] Sector alignment can optionally include: determining whether members are adjacent based at least on the distance between selected data points among the members; determining whether members are aligned based at least on the coordinates of each corner of each member; determining a coordinate transformation for each corner of each member; and applying the coordinate transformation associated with the first member to the second member of the member, at least based on whether the first member and the second member are adjacent and not aligned. Determining whether members are aligned can optionally include: determining the distance between the corners of the members based on the coordinates; and determining a relationship between the corner distances in adjacent corners of the members. The relationship can be associated with a line connecting the corners. If the relationship produces a value less than or equal to a third threshold, the method can optionally mark the members as aligned. If the relationship produces a value greater than the third threshold, the method can optionally include: evaluating the alignment between all adjacent members; and sorting the members based on the number of alignments of the members that are also adjacent to the member. The method may optionally include: if the fifth member is unaligned, and if the second corner of the fifth member is to be used for alignment with the third corner that is being aligned with the first member, then averaging the coordinate transformations of the first corner of the fourth member that is unaligned with the fifth member and the coordinate transformations of the second corner of the fifth member; and applying the average transformation to the fourth member. Each member may optionally include a range of substantially 50 to 150 square meters.

[0014] A method for locating walls in aerial point cloud (PC) data and terrestrial PC data may include, but is not limited to: filtering the aerial PC data and the terrestrial PC data; creating sectors of a pre-selected size within the filtered aerial PC data and terrestrial PC data; aligning the sectors in the terrestrial PC data; identifying possible walls within the aligned sectors; identifying possible roofs and possible surfaces in the aerial PC data; forming polygons around the possible walls, the possible roofs, and the possible surfaces; and forming walls from the possible roofs and the possible surfaces. The method may optionally include downsampling the terrestrial PC data. The method may optionally include applying a voxel filter to the terrestrial PC data.

[0015] A system for point cloud registration from terrestrial PC data to aerial PC data may include, but is not limited to: a surface mesh processor that creates a surface mesh from the terrestrial PC data, including surfaces located in the terrestrial PC data; a sector processor that decomposes the surface mesh into sectors, the sector processor aligning the sectors; a ground rigid feature processor that locates ground rigid features within the aligned sectors; an aerial rigid feature processor that locates aerial rigid features in the aerial PC data; a registration process that determines a transformation by registering the ground rigid features to the aerial rigid features; and a transformation process that transforms the terrestrial PC data according to the transformation. The ground rigid features may optionally include walls. The aerial rigid features may optionally include roofs and rigid surfaces. The system may optionally include filters that filter the aerial PC data and the terrestrial PC data. The system may optionally include voxel filters that filter the aerial PC data and the terrestrial PC data.

[0016] The sector processor may optionally include: computer code that executes instructions for: (a) if more of the sectors need to be aligned, (b) selecting candidate adjacent sectors; (c) determining the minimum distance between points in the candidate adjacent sectors; (d) if the minimum distance is less than or equal to a first threshold, marking the candidate adjacent sectors as aligned and returning to (b); (e) if the minimum distance is greater than the first threshold, taking the average distance between all points in the candidate adjacent sectors; (f) if the average distance is less than or equal to a second threshold, marking the candidate adjacent sectors as aligned and returning to (b); (g) if the average distance is greater than the second threshold, (1) determining the coordinates of each corner of each potentially misaligned sector; and (2) determining a transformation for each corner of each potentially misaligned sector; (h) determining the distance between the corners of the potentially misaligned sectors based on the coordinates; (i) determining a relationship between the corner distances in adjacent corners, the relationship being associated with a line connecting the corners; (j) if the average distance is greater than the second threshold, (i) determining the average distance between points in the candidate adjacent sectors. If the average value is less than or equal to the third threshold, the candidate adjacent sectors are marked as aligned and returned to (b); (k) if the average value is greater than the third threshold, the alignment between all adjacent sectors is evaluated; (l) the sectors are sorted based on the number of aligned neighboring sectors to form a sorted list of sectors; (m) the sector with the largest number of aligned neighbors, but not all aligned neighbors, is selected from the sorted list of sectors; (n) the transformation from the selected sector is applied to the unaligned neighbors at the adjacent corners; (o) if the unaligned sector... If the sector has another unaligned neighbor, and if the corner of the unaligned neighbor will be used for alignment with the same corner that is being aligned with the selected sector, (1) the transformation of the same corner is averaged; and (2) the averaged transformation is applied to the unaligned sector; (p) if the unaligned sector does not have another unaligned neighbor, and if there are more sectors to align, then return to (b); and (q) if one of the corners of the unaligned neighbor is not used for alignment with the same corner that is being aligned with the selected sector, then continue at (o). The relationship can optionally include averaging. The sector processor can optionally include identifying a value representing the distance between two potentially unaligned sectors that may be misaligned as the relationship. The sector can optionally include a range of substantially 50 to 150 square meters.

[0017] A system for locating walls in aerial point cloud (PC) data and terrestrial PC data can include, but is not limited to: a filter processor that filters the aerial PC data and the terrestrial PC data; a sector processor that creates sectors of pre-selected size within the filtered aerial PC data and terrestrial PC data, the sector processor aligning the sectors in the terrestrial PC data; a terrestrial wall processor that identifies possible walls in the terrestrial PC data; an aerial wall processor that identifies possible roofs and possible surfaces in the aerial PC data; a polygon processor that forms polygons around the possible walls, the possible roofs, and the possible surfaces; and a wall processor that forms walls from the roofs and surfaces. The filter processor can optionally include downsampling of the terrestrial PC data. The filter processor can optionally include a voxel filter that filters the terrestrial PC data.

[0018] A method for registering 3D ground point cloud (PC) data to 3D aerial PC data may include, but is not limited to: creating a surface mesh from the ground 3D PC data, including surfaces located in the ground 3D PC data; dividing the surface mesh into sectors; aligning the sectors; locating features within the aligned sectors; locating the features in the 3D aerial PC data; and determining a transformation by registering the features in the aligned sectors to the features in the aerial PC data. Attached Figure Description

[0019] This teaching will be more readily understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0020] Figure 1 This is a diagram illustrating the data collection process used in this teaching;

[0021] Figure 1A It is used to use from Figure 1 The flowchart of the data teaching process;

[0022] Figure 2 This is a diagram illustrating the filtering process in this teaching;

[0023] Figure 3 This is a diagram illustrating the surface locator process of this teaching;

[0024] Figures 4A-4C This is a diagram illustrating the sector alignment process in this teaching;

[0025] Figure 4D-4F Is Figures 4A-4C The flowchart of the method described in the document;

[0026] Figure 5 This is a diagram illustrating the process of extracting walls from ground PC data in this tutorial;

[0027] Figure 6 This is a diagram illustrating the process of extracting walls from aerial PC data in this tutorial;

[0028] Figure 7 This is a schematic block diagram of the system for air-to-ground registration as described in this teaching; and

[0029] Figure 8 This is a schematic block diagram of the system for locating walls in aerial and ground data, as taught in this course. Detailed Implementation

[0030] The systems and methods taught in this paper enable the use of airborne sensor data to augment ground-based sensor data when compiling scenarios that can be used for autonomous vehicle navigation.

[0031] Now for reference Figure 1 Topographic data of a geographic area can be acquired from aerial sensor 401 and ground-based sensor 501, and can be combined to form a richer dataset than the datasets that the aerial or ground-based sensors can generate individually. Aerial sensor data can include, but is not limited to, aerial point cloud data 701. Ground-based data can include, but is not limited to, ground-based point cloud data 601. Aerial data 701 can be collected by airborne equipment, which can include, but is not limited to, unmanned aerial vehicles, such as drones and all types of manned aircraft. Airborne equipment can include aerial sensor 401, which can include, but is not limited to, LiDAR technology. The LiDAR system can include the sensor itself as well as high-precision satellite positioning systems and high-precision sensors, such as inertial measurement units, to determine the sensor's orientation in space. These systems work together to enable the processing of raw aerial sensor data. Ground-based device 501 can collect ground-based data 601. Ground-based device 501 can be mounted on any ground-based structure, such as a building, vehicle, or tree. Ground-based device 501 can include the LiDAR system discussed with respect to the aerial device. The positioning system associated with the ground-based device 501 may be obstructed, which could affect the quality of the ground-based data 601.

[0032] Continue to refer to Figure 1Regarding the combination of aerial data 601 and ground data 701, and considering that ground-based data 601 may have positional issues, the positions of points in the datasets must be corrected relative to each other. The ground truth positions of fixed objects can be used to transform the entire ground-based point cloud to correct any problems related to point position. Fixed objects that can be located by both ground-based and aerial sensors can include, but are not limited to, walls. Fixed objects can include buildings, and buildings can include walls. Any fixed object can be used as a ground truth. The process taught herein can be extended to include the use of any kind of fixed object as a ground truth. The examples provided herein (where buildings and walls are used as ground truths) are for illustrative purposes only. Aerial data 701 can include, for example, roof-like structures 102 and surface data 104. Ground data 601 can include, for example, wall-like structures 106 and surface data 108. Roof-like structures 102 and surface data 104 can be concatenated to extract wall data from aerial data 701. Wall-like structures appearing in both datasets can be used to register aerial and ground data.

[0033] Now for reference Figure 1A The process of this teaching for combining aerial point cloud data 701 from aerial data database 701A and ground-based point cloud data 601 from ground data database 601A can include filtering the aerial point cloud data 701 and the ground-based point cloud data 601 to create a dataset that can be effectively manipulated and combined. Ground data database 601A and aerial data database 701A can represent any data source, such as data received directly from sensors, or data or combinations stored in a repository for later processing. Ground-based data can be smoothed to address distortions, such as warping, in the data. The perimeter of potentially interesting surfaces can be found from the dataset, and filtering criteria can be applied to the surfaces to identify desired features. Transformations used to register features found in the ground data with the same features found in the aerial data can be used to register the ground-based dataset 601 to the aerial dataset 701, thereby forming a consistent georeferenced composite dataset.

[0034] Continue to refer to Figure 1AThe filtered point cloud data can include, but is not limited to, filters 201 that can be used, for example, to reduce the number of points considered. For example, the type of filter selected can be controlled by the size of the dataset and the specific features to be considered as of interest. In one aspect, a conventional voxel filter as described herein can be used to filter the data, producing filtered aerial data 711 and filtered ground-based data 605. Surface locator 211 can locate surfaces 137 in the filtered ground-based data 605. Sector alignment 209 can divide the ground PC data 601 with the located surfaces 137 into sectors 137. Sector alignment 209 can align the sectors with each other to form aligned sectors 138 / 139. Polygon processor 213A can locate aerial features 1101 from the filtered aerial data 711. The feature perimeters of both the aligned sectors 138 / 139 and the aerial features 1101 can be established by, for example, but not limited to, conventional convex hull procedures performed in, for example, feature processor 212 and connection processor 215. The connection processor 215 can connect interconnected features in the airborne PC data 701 and form connected features 1118. For example, the roof of a building visible in the airborne PC data 701 and nearby surfaces can be connected to form other features, such as the walls of a building. The feature register 739 can use the ground feature 1102 and the connected features 1118 to register features in the ground PC data with similar features in the airborne PC data. For example, if the connected feature 1118 is a wall, walls in similar locations can be located in the ground PC data 601. The transformation process 727 can determine the transformation from the registration process, and the ground data transformation 743 can apply the transformation to the ground data.

[0035] Now for reference Figure 2Regarding filtering, given a point cloud 601, in one aspect, a regular voxel filter can return a point cloud 605A with fewer points than the original point cloud 601. The filtered point cloud 605A can represent the original point cloud 601 as a whole. A voxel filter can downsample the data from the gridded point cloud 605 by selecting points 602 in each "cell" (voxel) 604 of the gridded point cloud 605 to represent the remaining points in cell 604. The set of points located within the boundaries of voxels 604 can be combined to represent point 602. The method of representing points in voxels 604 with a single point can take any number of forms. For example, it is possible to compute the centroid or spatial mean of the point distribution, or to select points at the geometric center of voxels 604 to represent points in voxels 604. For example, a specific method can be chosen based on whether accuracy or computational speed is considered more important to a particular application. When using a voxel filter, the dataset can be decomposed into cells of a pre-selected size, such as, but not limited to, 0.25m. If you need to reduce the number of points in the dataset, you can use any downsampling filter.

[0036] Now for reference Figure 3 The surface locator 211 is capable of locating surfaces within the filtered ground data 605. In one aspect, the progressive morphology filter can include a process of classifying non-surface points from surface points. This process can begin by forming a regularly spaced grid 136 from the point cloud data and locating cells 131 within the grid. Within each cell 131, points with the minimum elevation are selected. A sliding filter window 133 can be used to acquire a subset of the minimum elevation points identified as the minimum point array 134. Another sliding filter window can be applied to the minimum point array 134 to create a point array 137 with the maximum elevation within the minimum point array 134. The elevations of the points in the maximum elevation array 137 can be compared to a threshold, and these points can be identified as surface points or non-surface points based on their relationship to the threshold. The progressive morphology filter progressively increases the window size of the sliding filter window and uses various pre-selected elevation difference thresholds, enabling the removal of buildings, vehicles, and vegetation from the dataset while preserving surface data. See Zhang et al., A Progressive Morphological Filter for Removing Nonground Measurements from Airborne LIDAR Data, IEEE Transactions on Geoscience and Remote Sensing, 41:4, April, 2003.

[0037] Now for reference Figure 4A Surface data isolated according to the process described herein or any similar process can be decomposed into sectors of a pre-selected size that can then be aligned. Sector alignment 209 can be used to address deformations in the dataset, such as, but not limited to, distortions. Therefore, sector alignment 209 can perform an approximation equivalent to distortions in the data. In one aspect, the approximation can be linear, non-linear, or a combination thereof. As discussed herein, the surface in point sector 137 can be decomposed into sectors of a size suitable for undergoing the alignment process. In some configurations, the sector size can be selected based on characteristics such as the point density of the dataset. In one aspect, the sector size can include, but is not limited to, 100m x 100m.

[0038] Continue to refer to Figure 4A It can calculate coordinates (such as, but not limited to, Universal Transverse Mercator (UTM)) for each corner of each sector 138 / 139, and can also calculate coordinate transformation (such as, but not limited to, Normal Distribution Transform (NDT)) values ​​for each corner. It can align sectors by applying the coordinate transformation calculated for the corners of the first sector to a second sector that is not aligned with the first sector. If a corner of the second sector that is not aligned with the first sector is also not aligned with a third sector at one of the transformation points, a new transformation is calculated for that corner, which is a function of the coordinate transformations of the third sector and the adjacent points of the first sector. In the example, to determine which sectors are adjacent and aligned, it is possible to base the results on any two points D in adjacent sectors 138 / 139. A -D D The distance between adjacent sectors is used to evaluate sectors. For example, if the shortest distance between any two points in adjacent sectors 138 / 139 is less than a first pre-selected threshold D... T1 If so, then sector 138 / 139 alignment is assumed. In some configurations, a pre-selected threshold D is used. T1 This can include, but is not limited to, 0.85m. If the shortest distance D... A Greater than D T1 It can take the distance D of all points. A -D D Average D AVG If D AVG Less than the second pre-selected threshold D T2 Sectors 138 / 139 can be considered aligned. On the other hand, if D AVG Greater than or equal to the second pre-selected threshold D T2 If this is the case, then sectors 138 / 139 can be considered unaligned. Note that any coordinate system and any coordinate transformation can be used to align sectors.

[0039] Now for reference Figure 4B Continuing with this example, we can calculate sector 138 / 139 ( Figure 4A The distance D1 / D2 between the corners of ) can be calculated, and the relationship between distances D1 / D2 can be calculated. For example, the average value D3 of distances D1 / D2 can be taken. Other relationships are also possible. If the third pre-selected threshold D4 is greater than or equal to the average value D3, sector 138 / 139 ( Figure 4A ) can be considered aligned. On the other hand, if the third preselected threshold D4 is less than the average value D3, then sector 138 / 139 is considered aligned. Figure 4A (Unaligned). An example of eight sectors AH in various alignment states is shown. The average distance between sectors AH is evaluated relative to a third pre-selected threshold D4, ranging from 801 to 819. Table 821 presents the sector states related to neighbor alignment. The contents of Table 821 are listed in Table 823, which shows sector AH and the total number of its aligned neighbors compared to the total number of its neighbors. Table 825 is a sorted version of Table 823, ordered from the least needed to the most needed sectors in sector AH. For example, sectors F and G are aligned with all their neighbors, while sectors A and E are not aligned with their neighbors. Sectors B, H, C, D, A, and E need to be aligned with at least one neighbor. The process first selects the sector that needs the fewest aligned neighbors, in this case, sector B.

[0040] Now for reference Figure 4C Sector B has three neighbors: sectors E, F, and H, two of which, F and H, are aligned with sector B. To align sector E with sector B, transformations NDT1 and NDT2 can be applied to the corner of sector E adjacent to the corner of sector B. For example, transformation NDT1, associated with a point at the end of sector B on line 861, can be used to transform a point in sector E at the end of sector E on line 861. If sector E has no other neighbors, transformation NDT2, associated with a point at the end of sector B on line 863, can be used to transform a point in sector E at the end of sector E on line 863. However, sector E has a neighboring sector C that is not aligned with sector E. Transformation NDT4 is associated with the point C adjacent to the point of sector E at the end of line 865. To complete the alignment of sector E, the relationship between NDT2 and NDT4 can be developed and applied to the point of sector E at the ends of lines 863 and 865. This relationship can include, but is not limited to, averaging NDT2 and NDT4 and applying the averaging transformation to sector E. Repeat this process until all sectors are aligned.

[0041] Now for reference Figure 4DThe method 250 for aligning sectors may include, but is not limited to, selecting 251 candidate adjacent sectors and determining 253 the minimum distance between points in adjacent sectors. If 255 the minimum distance is less than or equal to a first threshold, then method 250 may include marking the candidate adjacent sectors 257 as aligned and returning 251 more candidate adjacent sectors. If 255 the minimum distance is greater than the first threshold, then method 250 may include taking 259 the average distance between all points in adjacent sectors. If 261 the average distance is less than or equal to a second threshold, then method 250 may include marking the candidate adjacent sectors 257 as aligned and returning 251 more candidate adjacent sectors. If 261 the average distance is greater than the second threshold, then method 250 may include determining 263 the coordinates of each corner of each potentially misaligned sector.

[0042] Now for reference Figure 4E Method 250 can include determining 265 a transformation for each corner of each potentially misaligned sector. Method 250 can include determining 267 the distance between corners of potentially misaligned sectors based on coordinates, and determining 269 a relationship between corner distances in adjacent corners, associated with a line connecting the corners. This relationship can include, but is not limited to, averaging, weighted averaging, or any other method used to identify values ​​representing the distance between two potentially misaligned sectors. If 271 the average is less than or equal to a third threshold, method 250 can include marking 257 candidate adjacent sectors as aligned and returning 251 more candidate adjacent sectors. If 271 the average is greater than the third threshold, method 250 can include evaluating 273 the alignment between all adjacent sectors and sorting the sectors based on the number of aligned neighbors 275. Method 250 can include selecting 277 sectors with the most, but not all, aligned neighbors from the sorted list of sectors.

[0043] Now for reference Figure 4FMethod 250 can include applying a transformation from the selected sector 279 to its unaligned neighbor at an adjacent corner. If the unaligned sector 281 has another unaligned neighbor, and if the corner of the unaligned neighbor 283 will be used for alignment with the same corner being aligned with the selected sector, then method 250 can include averaging the transformation of the same corner 285 and applying the average transformation to the unaligned sector. If 287 there are more sectors to align, then method 250 can include returning to step 251. If 287 there are no more sectors to align, then method 250 can include ending the process. If the unaligned sector 281 has no other unaligned neighbor, and if 287 has more sectors to align, then method 250 can include returning to the beginning of method 250 by selecting candidate adjacent sectors that 251 has not yet aligned. If the corner of the unaligned neighbor 283 is not used for alignment with the same corner being aligned with the selected sector, then method 250 can include continuing method 250 at step 281.

[0044] Now for reference Figure 5 This includes the previously determined aligned surface point sector 138 (see...). Figure 3 The dataset can include features that may include walls 1001 / 1003 / 1005. To determine whether any wall-like features are included in the dataset, conventional polygon forming algorithms, such as, but not limited to, convex hull as described herein, can be used. For example, using a convex hull algorithm, original points 1017 / 1027 / 1037 can be selected from a set of points that can represent features. Points located relatively close to the original point 1017, such as points 1011 / 1013 / 1015, can also be examined. Any point associated with the minimum angle on the vertical axis is selected as the first corner of the polygon to be generated by the analysis. The next point is selected based on the next minimum angle. The first corner is connected to the next point. The next point is selected based on the next minimum angle and connected to the previously determined point. Finally, the last selected point is connected to the original point to form the polygon. The angle 1001A formed by the polygon and the x-axis 1043 can be compared with a pre-selected threshold. If the absolute value of the angle 1001A is greater than the threshold angle, the polygon is classified as a wall feature. If the angle 1001A is less than or equal to the absolute value of the threshold, the polygon is classified as a non-wall feature. Figure 5 Examples of other possible wall features are shown below. The threshold angle can include, but is not limited to, |45°| relative to the horizontal.

[0045] Now for reference Figure 6Filter 201 can be applied to aerial data 701 to create a uniform point mesh. For example, but not limited to, voxel filters can be applied. Voxels 1105 can be evaluated as discussed herein to determine a representative point 1103 for each voxel, creating a mesh 1107. Convex hull processes, or any processes capable of determining shape from point cloud data, can be applied to the filtered aerial data 711 to refine possible aerial features 1101. Figure 1A Examples, but not limited to, roof and surface features 1109 / 1111 / 1113. Angles 1115 / 1117 / 1119 relative to the horizontal axis can be evaluated relative to a threshold angle, and shapes with angles exceeding the threshold angle are not used for further analysis because these shapes are unlikely to be roof or ground surfaces. For example, if the angle 1115 of shape 1109 relative to the horizontal axis 1116 is determined to be greater than the absolute value of the threshold angle, shape 1109 will be eliminated from further analysis. The same applies to shapes 1111 / 1113 and angles 1117 / 1119. In some configurations, the threshold angle can include |15°| relative to the horizontal. Features that rise relative to each other can represent roofs. For example, feature 1117, which rises relative to feature 1113, can be used to estimate the connection feature 1118 by connecting the corners of feature 1117 to surface points from surface features such as feature 1113. Figure 1A The location of ) such as, but not limited to, walls. Located on the ground, data 601 ( Figure 1A Wall features in the data can be registered to wall features in the aerial data 701. Any conventional method can be used, such as the Generalized Iterative Closest Point (ICP) algorithm, normal distribution transformation, fast point feature histogram, and 4-point consensus set. The transformation determined by the registration can be applied to the ground data 601. Figure 1A ).

[0046] Now for reference Figure 7A system 1200 for combining aerial point cloud data 701 and ground-based point cloud data 601 can include a surface ground processor 1201 for creating a surface mesh 1217 from the ground PC data 701. The surface mesh 1217 can include surfaces located within the ground PC data 601. The surface ground processor 1201 can receive the ground PC data 601 and provide the surface mesh 1217 to a sector processor 1203. The sector processor 1203 can decompose the surface mesh 1217 into sectors 1219. The sectors 1219 can be large enough to include a representative number of points, but not so large that processing time becomes an issue. Therefore, the sector size 1219 can vary with the density of the ground PC data 601. The sector processor 1203 can align unaligned sectors within the surface mesh 1217 and can provide the aligned sectors 1219 to a ground feature processor 1215. Ground feature processor 1205 is capable of locating ground rigid feature 1223 within aligned sector 1219 and providing ground rigid feature 1223 to registration processor 1209. Airborne rigid feature processor 1207 is capable of receiving airborne PC data 701, locating airborne rigid feature 1215, and providing airborne rigid feature 1215 to registration process 1209. Registration process 1209 is capable of registering ground rigid feature 1223 to airborne rigid feature 1215, determining transformation 1213 from the process, and providing transformation 1213 to transformation processor 1211. Transformation processor 1211 is capable of applying transformation 1213 to ground PC data 601 and providing ground PC data transformation 602 to autonomous processor 1212. In some configurations, ground rigid features may include walls. In some configurations, airborne rigid features may include roofs and surfaces. System 1200 may optionally include filters for filtering airborne PC data 701 and ground PC data 701. The filter may optionally include a voxel filter.

[0047] Continue to refer to Figure 7The sector processor 1203 may optionally include computer code performing the following steps: (a) if there are more sectors to align, (b) selecting candidate adjacent sectors, and (c) determining the minimum distance between points in adjacent sectors. (d) if the minimum distance is less than or equal to a first threshold, the sector processor 1203 may include marking the candidate adjacent sectors as aligned and returning to (b). (e) if the minimum distance is greater than the first threshold, the sector processor 1203 may include taking the average distance between all points in adjacent sectors. (f) if the average distance is less than or equal to a second threshold, the sector processor 1203 may include marking the candidate adjacent sectors as aligned and returning to (b). (g) if the average distance is greater than the second threshold, (1) the sector processor 1203 may include determining the coordinates of each corner of each potentially misaligned sector and (2) determining the transformation of each corner of each potentially misaligned sector. (h) the sector processor 1203 may include determining the distance between corners of potentially misaligned sectors based on coordinates, and (i) determining the relationship between corner distances in adjacent corners. This relationship can be associated with the line connecting the corners. (j) If the average is less than or equal to the third threshold, the sector processor 1203 can include marking the candidate adjacent sectors as aligned and returning to (b). (k) If the average is greater than the third threshold, the alignment between all adjacent sectors is evaluated, (l) the sectors are sorted based on the number of aligned neighbors, (m) the sector with the largest number of aligned neighbors, but not all aligned neighbors, is selected from the sorted list of sectors, and (n) a transformation from the selected sector is applied to its unaligned neighbors at adjacent corners. (o) If the unaligned sector has another unaligned neighbor, and if the corner of the unaligned neighbor will be used for alignment with the same corner that is being aligned with the selected sector, the sector processor 1203 can include (1) averaging the transformation of the same corner, and (2) applying the average transformation to the unaligned sector. (p) If the unaligned sector does not have another unaligned neighbor, and if there are more sectors to align, the sector processor 1203 can include returning to (b). (q) If the corner of an unaligned neighbor is not used for alignment with the same corner that is being aligned with the selected sector, then sector processor 1203 may include a continuing sector processor at (o). This relationship may optionally include averaging. Sector processor 1203 may optionally include identifying a value representing the distance between two potentially unaligned sectors as a relationship. Sector 1218 may optionally include a range of approximately 50 to 150 square meters.

[0048] Now for reference Figure 8A system 1300 for locating walls in aerial point cloud (PC) data 701 and ground PC data 601 can include, but is not limited to: a filter processor 1301 that filters the aerial PC data 701 and ground PC data 601; and a sector processor 1303 that creates sectors 1317 of pre-selected size within the filtered aerial PC data 1315 and ground PC data 1313. The sector processor 1303 is capable of aligning ground sectors 1313 in the ground PC data, and a ground wall processor 1305 is capable of identifying possible walls 1319 in the aligned ground PC sectors 1313. The system 1300 can include: an aerial wall processor 1311 that identifies possible roofs and possible surfaces 1321 in the aerial PC data; a polygon processor 1309 that forms polygons 1323 around possible walls 1319 and roofs / surfaces 1321; and a wall processor 1307 that forms walls 1323 based on roofs and surfaces and provides them to an autonomous device processor 1212. The filter processor 1301 may optionally include downsampled ground PC data 601. The filter processor 1301 may optionally include a voxel filter for filtering the ground PC data 601.

[0049] The configuration described herein relates to a computer system for carrying out the methods discussed herein, and a computer-readable medium containing programs for carrying out these methods. It is capable of storing raw data and results for future retrieval and processing, printing, displaying, transferring to another computer, and / or elsewhere. Communication links can be wired or wireless, for example, using cellular communication systems, military communication systems, and satellite communication systems. Certain parts of the system can run on a computer with a variable number of CPUs. Other alternative computer platforms can be used.

[0050] This configuration also applies to software used to carry out the methods discussed herein, and to computer-readable media storing the software used to carry out these methods. The various modules described herein can be completed on the same CPU or on different computers. In accordance with regulations, this configuration has been described using language more or less specific to structural and methodological features. However, it should be understood that this configuration is not limited to the specific features shown and described, as the apparatus disclosed herein includes preferred forms that enable this configuration.

[0051] The method can be implemented electronically, either wholly or partially. Signals identifying actions taken by system components and other publicly disclosed configurations can be propagated through at least one real-time communication network. Control and data information can be executed electronically and stored on at least one computer-readable medium. The system can be implemented to execute on at least one computer node in at least one real-time communication network. Common forms of at least one computer-readable medium can include, for example, but not limited to, floppy disks, hard disks, magnetic tape or any other magnetic media, optical disc read-only memory or any other optical media, punched cards, paper tape or any other physical media with a perforated pattern, random access memory, programmable read-only memory and erasable programmable read-only memory (EPROM), flash memory EPROM or any other memory chip or cartridge or any other medium that a computer can read. Furthermore, at least one computer-readable medium can contain any form of graphics subject to appropriate licensing if necessary, including but not limited to Graphics Exchange Format (GIF), Joint Group of Image Experts (JPEG), Portable Web Graphics (PNG), Scalable Vector Graphics (SVG), and Tagged Image File Format (TIFF).

[0052] While the teachings have been described above with respect to specific configurations, it should be understood that they are not limited to these disclosed configurations. Many modifications and other configurations will occur to those skilled in the art, which will be covered by this disclosure and the appended claims. The scope of the teachings should be determined by the proper interpretation and construction of the appended claims and their legal equivalents, as understood by those skilled in the art based on the disclosure in this specification and the accompanying drawings.

Claims

1. A method for locating a wall in aerial point cloud data and ground point cloud data, comprising: Filter the aerial point cloud data and the ground point cloud data; Create sectors of a pre-selected size within the filtered aerial point cloud data and the ground point cloud data; Align the sectors in the ground point cloud data; Identify possible walls within the aligned sectors; Identify possible rooftops and surfaces in the aerial point cloud data; A polygon is formed around the possible walls, the possible roof, and the possible surface; as well as Walls are formed from the possible roof and the possible surface. Aligning the sectors includes: For each corner of each sector, calculate the coordinates and coordinate transformation; The coordinate transformation calculated for the corner of the first sector is applied to the second sector, which is not aligned with the first sector.

2. The method according to claim 1, further comprising: The ground point cloud data is downsampled.

3. The method according to claim 1, further comprising: A voxel filter is applied to the ground point cloud data.

4. A system for point cloud registration from ground point cloud data to aerial point cloud data, comprising: A surface mesh processor configured to create a surface mesh from the ground point cloud data, including surfaces located in the ground point cloud data; A sector processor configured to decompose the surface mesh into sectors, the sector processor aligning the sectors; A ground rigid feature processor configured to locate ground rigid features within the aligned sector; An aerial rigid feature processor, configured to locate aerial rigid features in the aerial point cloud data; The registration process is configured to determine the transformation by registering the ground rigid feature to the air rigid feature; as well as The transformation process is configured to transform the ground point cloud data according to the transformation. The sector processor is configured as follows: (a) If there are more of the aforementioned sectors to align, (b) Select candidate adjacent sectors; (c) Determine the minimum distance between points in the candidate adjacent sectors; (d) If the minimum distance is less than or equal to the first threshold, then mark the candidate adjacent sectors as aligned and return to (b); (e) If the minimum distance is greater than the first threshold, take the average distance between all points in the candidate adjacent sectors; (f) If the average distance is less than or equal to the second threshold, then mark the candidate adjacent sectors as aligned and return to (b); (g) If the average distance is greater than the second threshold, (1) Determine the coordinates of each corner of each potentially misaligned sector; and (2) Determine the transformation for each corner of each of the potentially misaligned sectors; (h) Determine the distance between the corners of the potentially misaligned sectors based on the coordinates; (i) Determine the relationship between the corner distances in adjacent corners, the relationship being associated with a line connecting the corners; (j) If the average distance is less than or equal to the third threshold, then mark the candidate adjacent sectors as aligned and return to (b); (k) If the average distance is greater than the third threshold, then evaluate the alignment between all adjacent sectors; (l) Sort the sectors based on the number of aligned neighbors to form a sorted list of sectors; (m) Select the sector from the sorted list of sectors that has the largest number of aligned neighbors, but not all of them; (n) Apply the transformation from the selected sector to the unaligned neighbor at the adjacent corner; (o) If an unaligned sector has another unaligned neighbor, and if the corner of said unaligned neighbor will be used for alignment with the same corner that is being aligned with the selected sector, (1) Averaging the transformations of the same corners; and (2) Apply the averaged transformation to the unaligned sector; (p) If the unaligned sector has no other unaligned neighbor, and if there are more sectors to align, then return to (b); and (q) If one of the corners of the unaligned neighbor is not used for alignment with the same corner that is being aligned with the selected sector, then continue at (o).

5. The system according to claim 4, wherein, The rigid features of the ground include walls.

6. The system according to claim 4, wherein, The rigid features in the air include the roof and rigid surfaces.

7. The system according to claim 4, further comprising: A filter configured to filter the aerial point cloud data and the ground point cloud data.

8. The system according to claim 4, further comprising: A voxel filter, configured to filter the aerial point cloud data and the ground point cloud data.

9. The system according to claim 4, wherein, The relationship includes: average.

10. The system according to claim 4, wherein, The sector processor is configured to: The value representing the distance between two potentially misaligned sectors is identified as the relationship.

11. The system according to claim 4, wherein, The sector includes an area of ​​50 to 150 square meters.

12. A system for locating a wall in aerial point cloud data and ground point cloud data, comprising: A filter processor configured to filter the aerial point cloud data and the ground point cloud data; A sector processor configured to create sectors of a pre-selected size within filtered aerial point cloud data and ground point cloud data, the sector processor aligning the sectors in the ground point cloud data. A ground wall processor configured to identify possible walls in the ground point cloud data; An aerial wall processor, configured to identify possible roofs and surfaces in the aerial point cloud data; A polygon processor configured to form polygons around the possible walls, the possible roof, and the possible surfaces; as well as Wall processor, the wall processor being configured to form walls from the roof and surface, Aligning the sectors includes: For each corner of each sector, calculate the coordinates and coordinate transformation; The coordinate transformation calculated for the corner of the first sector is applied to the second sector, which is not aligned with the first sector.

13. The system according to claim 12, wherein, The filter processor is configured to: The ground point cloud data is downsampled.

14. The system according to claim 12, wherein, The filter processor is configured to: A voxel filter for filtering the ground point cloud data.

15. A method for registering 3D ground point cloud data to 3D aerial point cloud data, comprising: Create a surface mesh from ground 3D point cloud data, including surfaces located in the ground 3D point cloud data; The surface mesh is decomposed into sectors; Align the sectors; Locate features within aligned sectors; Locate the feature in the 3D aerial point cloud data; as well as The transformation is determined by registering the features within the aligned sectors to the features within the aerial point cloud data. Aligning the sectors includes: (a) If there are more of the aforementioned sectors to align, (b) Select candidate adjacent sectors; (c) Determine the minimum distance between points in the candidate adjacent sectors; (d) If the minimum distance is less than or equal to the first threshold, then mark the candidate adjacent sectors as aligned and return to (b); (e) If the minimum distance is greater than the first threshold, take the average distance between all points in the candidate adjacent sectors; (f) If the average distance is less than or equal to the second threshold, then mark the candidate adjacent sectors as aligned and return to (b); (g) If the average distance is greater than the second threshold, (1) Determine the coordinates of each corner of each potentially misaligned sector; and (2) Determine the transformation for each corner of each of the potentially misaligned sectors; (h) Determine the distance between the corners of the potentially misaligned sectors based on the coordinates; (i) Determine the relationship between the corner distances in adjacent corners, the relationship being associated with a line connecting the corners; (j) If the average distance is less than or equal to the third threshold, then mark the candidate adjacent sectors as aligned and return to (b); (k) If the average distance is greater than the third threshold, then evaluate the alignment between all adjacent sectors; (l) Sort the sectors based on the number of aligned neighbors to form a sorted list of sectors; (m) Select the sector from the sorted list of sectors that has the largest number of aligned neighbors, but not all of them; (n) Apply the transformation from the selected sector to the unaligned neighbor at the adjacent corner; (o) If an unaligned sector has another unaligned neighbor, and if the corner of said unaligned neighbor will be used for alignment with the same corner that is being aligned with the selected sector, (1) Averaging the transformations of the same corners; and (2) Apply the averaged transformation to the unaligned sector; (p) If the unaligned sector has no other unaligned neighbor, and if there are more sectors to align, then return to (b); and (q) If one of the corners of the unaligned neighbor is not used for alignment with the same corner that is being aligned with the selected sector, then continue at (o).

Images