Automated annotation of lidar point group executed by computer
Patent Information
- Application Number
- JP2024074708
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-05
- Filing Date
- 2024-05-02
- Publication Date
- 2024-12-03
Smart Images

Figure 2024167163000001 
Figure 2024167163000002 
Figure 2024167163000003
Abstract
Description
[Technical field]
[0001]
[0001] The present disclosure relates generally to the field of light detection and ranging (LiDAR), and more particularly to applying classification labels to LiDAR point clouds. [Background technology]
[0002]
[0002] Light detection and ranging (LiDAR) uses laser light to measure distance by targeting an object or surface with a laser and measuring the time it takes for the reflected light to return to a receiver. Lidar scans can be performed to generate highly accurate and detailed three-dimensional (3D) measurements of an environment, including various features such as the ground, vegetation, buildings, and other objects. Lidar scans generate a 3D point cloud of the scanned environment. The speed and accuracy of Lidar allows for the creation of detailed large-area maps that were previously only possible with time-consuming and expensive ground surveyors.
[0003]
[0003] Lidar point clouds are commonly used to generate labeled maps of airport environments. The assignment of classification labels to the lidar point clouds is traditionally performed by humans. However, such human annotation of lidar point clouds representing airport environments can have challenges. As an example, a human annotator, even if experienced, may make labeling errors that generate mislabeled lidar point clouds that misrepresent the airport environment. As another example, the time it takes a human to annotate a lidar point cloud of an airport environment can be significant, such that the resulting human-annotated lidar point cloud of the airport environment may be outdated by the time it is complete. In particular, dynamic objects may change position during the time it takes a human annotator to generate a human-annotated lidar point cloud, such that the point cloud does not accurately represent the current state of the airport environment. Summary of the Invention
[0004]
[0004] An embodiment is disclosed that relates to an approach for automatically applying classification labels to a Light Detection and Ranging (LiDAR) point cloud of an airport in a manner that is faster and more accurate than human annotation of the LiDAR point cloud. In one embodiment, a computing device includes one or more processors configured to execute instructions stored in a memory to: (1) receive LiDAR data of an airport environment, the LiDAR data including a LiDAR point cloud including a plurality of points in the airport environment; (2) partition the LiDAR point cloud into ground and non-ground regions; (3) automatically cluster various points in the non-ground region into a plurality of non-ground segments via a trained machine learning model; (4) automatically apply classification labels selected from the airport domain to the plurality of non-ground segments via the trained machine learning model; and (5) output a global, labeled LiDAR point cloud including the plurality of non-ground segments and points in the ground region with their respective classification labels.
[0005]
[0005] The features, functions, and advantages discussed may be realized individually in various embodiments or may be combined in yet other embodiments, further details of which can be understood with reference to the following description and drawings. [Brief description of the drawings]
[0006] [Figure 1]
[0006] An exemplary scenario is shown in which an aircraft scans an airport environment with a Light Detection and Ranging (LiDAR) device to obtain LIDAR data of the airport environment. [Diagram 2]
[0007] 2 illustrates an example lidar point cloud of the airport environment of FIG. 1. [Diagram 3]
[0008] 1 illustrates a schematic diagram of an example computing system configured to automatically apply classification labels to lidar point clouds to generate a global, labeled lidar point cloud. [Figure 4]
[0009] 1 illustrates generally an exemplary process flow for generating an overview, labeled lidar point cloud. [Diagram 5]
[0010] 5-6 show a flowchart of an exemplary computer-implemented method for automatically applying classification labels to lidar point clouds to generate an overview, labeled lidar point cloud. [Figure 6] 5-6 show a flowchart of an exemplary computer-implemented method for automatically applying classification labels to lidar point clouds to generate an overview, labeled lidar point cloud. [Figure 7]
[0011] 4 illustrates a schematic diagram of an exemplary computing system in which the computing system of FIG. 3 may be embodied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007]
[0012] The present description is directed to a computer-implemented approach for automatically assigning classification labels to a Light Detection and Ranging (LiDAR) point cloud of an airport in a manner that is faster and more accurate than human annotation of LiDAR point clouds. In one embodiment, LiDAR data of an airport environment is received, the LiDAR data including a LiDAR point cloud having a plurality of points representing the airport environment. The LiDAR point cloud is partitioned into ground and non-ground regions. Various points in the non-ground region are automatically clustered into a plurality of non-ground segments via a trained machine learning model. The plurality of non-ground segments are automatically assigned classification labels selected from the airport domain via a trained machine learning model. A global labeled LiDAR point cloud is output. The global labeled LiDAR point cloud includes the plurality of non-ground segments and points in the ground region that are assigned respective classification labels.
[0008]
[0013] The computer-implemented approach performs unsupervised, global segmentation of lidar point clouds of an airport environment using automated / self-learning techniques that enable automatic labeling of lidar point clouds without human involvement. The computer-implemented approach intelligently segments the lidar point cloud into ground and non-ground regions to reduce the total number of points that need to be analyzed by a trained machine learning model and to reduce the time and processing resources required to generate the global, labeled lidar point cloud.
[0009]
[0014] The trained machine learning model is trained using airport-specific domain knowledge. The airport-specific domain knowledge enables the trained machine learning model to automatically cluster points in the non-ground area into segments that correspond to objects typically found in an airport environment without human intervention. Training the machine learning model based on the airport-specific domain knowledge enables the trained machine learning model to identify airport-specific objects in the lidar point cloud with increased accuracy compared to human annotators or generically trained machine learning models. Furthermore, the trained machine learning model utilizes the airport-specific domain knowledge to automatically assign airport-specific classification labels (e.g., aircraft, truck, terminal, skybridge, tower, etc.) to multiple non-ground area segments. These labels provide a higher classification granularity compared to more general labels (e.g., dynamic and non-dynamic object labels).
[0010]
[0015] In some embodiments, the global, labeled lidar point cloud generated using the computer-implemented approach may be used by a navigation system of an autonomous aircraft. Moreover, in particular, the computer-implemented approach may generate the global, labeled lidar point cloud at a sufficient rate such that information embedded within the global, labeled lidar point cloud is still accurate enough for the navigation system to operate the autonomous aircraft based on such information.
[0011]
[0016] FIG. 1 illustrates an exemplary scenario in which an aircraft 100 scans an airport environment 102 with a lidar device 104 to obtain lidar data for the airport environment 102. In particular, the lidar device 104 includes an emitter configured to emit multiple pulses of laser light, such as representative pulse 106, toward the airport environment 102. The pulses emitted by the lidar device 104 reflect off surfaces within the airport environment 102 and are received by a sensor in the lidar device 104. Each pulse generates an individual point that represents a portion of the airport environment 102. The lidar device 104 aggregates these points to generate a lidar point cloud 200 (shown in FIG. 2) that collectively represents the airport environment 102. The number of pulses emitted / received by the lidar device 104 to generate the lidar point cloud can be varied from that shown in FIG. 1 without undue experimentation depending on the particular application. As an example, a lidar point cloud of an airport environment may include in the range of 2-10 or more points per square meter of the airport environment.
[0012]
[0017] Aircraft 100 may perform multiple lidar scans of airport environment 102 from different positions to aggregate points that contribute to lidar point cloud 200 (shown in FIG. 2 ) throughout different phases of a mission. For example, aircraft 100 may scan airport environment 102 with lidar device 104 while taxiing, taking off, approaching, and / or landing, depending on the mission. By performing multiple lidar scans from different positions, different portions of airport environment 102 are exposed to lidar device 104 in different positions and captured in lidar point cloud 200 (shown in FIG. 2 ) to obtain an at-a-glance representation of airport environment 102.
[0013]
[0018] Various vehicles carrying lidar devices may perform scans of the airport environment 102 during designated time windows to collect lidar data (e.g., points in the lidar point cloud 200 shown in FIG. 2). The lidar scans may be performed within the designated time windows to increase the likelihood that dynamic objects (e.g., vehicles) in the spatial environment 102 remain in the same location captured in the lidar point cloud. Different aircraft may perform lidar scans throughout different phases of their respective missions to collect lidar data. Various ground vehicles (e.g., tractor vehicles, staircase vehicles, belt loader vehicles) may perform lidar scans to collect lidar data. Fixed lidar devices may be positioned around the airport environment 102 to collect lidar data. Lidar data generated from multiple lidar scans from various lidar devices (mounted on vehicles or fixed) may be aggregated to generate the lidar point cloud 200 (shown in FIG. 2). The lidar point cloud 200 (shown in FIG. 2) may be generated in any suitable manner.
[0014]
[0019] FIG. 2 illustrates a lidar point cloud 200 of the airport environment 102 of FIG. 1. Although not specifically depicted in FIG. 2, it will be understood by those skilled in the art that the lidar point cloud 200 typically includes a very large collection of individual points plotted in three-dimensional (3D) space. Each point, such as point 202, included in the lidar point cloud 200 includes Cartesian coordinates (X, Y, and Z) and an intensity value (i). The intensity value (i) represents the recorded return intensity of the laser beam. In one embodiment, the intensity value (i) may be between 1 and 256. The intensity value (i) varies with the composition of the surface reflecting the laser beam. For example, a low number may indicate low reflectivity and a high number indicates high reflectivity. The scale of the intensity value (i) may depend on the lidar sensor making the measurement. In one embodiment, the intensity scale includes decimals between 0 and 1. In another embodiment, the intensity scale includes values between 0 and 255. The return intensity of the laser beam may also be affected by the angle of arrival (scan angle), distance, surface composition, roughness, and amount of moisture in the atmosphere when the lidar scan is performed. This means that features at the nadir of the lidar device will typically have higher intensity than the same features along the edge (further tilted) since the latter will have less returning energy. The intensity values of the lidar point cloud 200 may be used for various analyses of the lidar point cloud 200. For example, analyses of the intensity values may be performed for data registration, feature extraction, classification, surface analysis, and segmentation, among other types of analyses.
[0015]
[0020] 3 illustrates generally an example computing system 300 configured to execute a processing pipeline 301 configured to automatically annotate a lidar point cloud, such as the lidar point cloud 200 illustrated in FIG. 2. The processing pipeline 301 is configured to receive lidar data 302. The lidar data 302 may be received from one or more lidar devices performing one or more lidar scans of an airport environment, such as the airport environment 102 illustrated in FIG. 1. Note that a lidar scan may include emitting laser light pulses into the airport environment via an emitter of the lidar device and receiving return pulses reflected from airport-related objects in the airport environment via a sensor of the lidar device.
[0016]
[0021] In some embodiments, the lidar data 302 is received from a single lidar device. In some embodiments, the lidar device is a lidar device mounted on an aircraft. In some embodiments, the lidar data 302 is aggregated from multiple lidar scans performed by the lidar device. In other embodiments, the lidar data 302 is received from multiple lidar devices. In some embodiments, multiple lidar devices are mounted on various vehicles within an airport, such as aircraft and ground vehicles. In some embodiments, at least a portion of the lidar data 302 is received from one or more fixed lidar devices located within the airport environment. In some embodiments, the lidar data 302 is aggregated from multiple lidar scans performed by multiple lidar devices.
[0017]
[0022] The lidar data 302 is aggregated by processing pipeline 301 to generate a lidar point cloud 304. The lidar point cloud 304 includes a plurality of points within the airport environment. In particular, each point included in the lidar point cloud 304 includes an X, Y, and Z coordinate and an intensity value (i) that represents the reflectivity of a surface within the airport environment.
[0018]
[0023] The processing pipeline 301 includes segmentation logic 306. The segmentation logic 306 is configured to receive the lidar point cloud 304 and output a segmented lidar point cloud 308 including a ground region 310 and a non-ground region 312. The ground region 310 includes points corresponding to the ground surface within an airport environment. The ground region 310 may include airport related objects or features at various surface levels, such as runways, roads, walkways, fields, water surfaces, etc. The non-ground region 312 includes points corresponding to objects or features that extend above or above ground level, such as vehicles, terminals, hangars, air traffic control towers, etc. The segmentation logic 306 may be configured to segment the lidar point cloud 304 using any suitable segmentation algorithm or technique. In one embodiment, the segmentation logic 306 segments the lidar point cloud 304 using a plane fitting algorithm.
[0019]
[0024] The processing pipeline 301 includes a trained machine learning model 314 configured to receive the segmented lidar point cloud 308 as an input. Additionally, the trained machine learning model 314 is configured to receive an airport domain 316 as an input. The airport domain 316 includes a library of classification labels 318 specific to an airport environment. The library of classification labels 318 includes various types of classification labels, including dynamic semantic labels 320 for movable objects (e.g., aircraft and ground vehicles) and non-dynamic semantic labels 322 for fixed objects (e.g., terminals, hangars, air traffic control towers). It should be noted that such classification labels indicate objects as being dynamic or non-dynamic. Additionally, the classification labels indicate a particular class of object (e.g., aircraft, ground vehicles, terminals, hangars, air traffic control towers) associated with an airport.
[0020]
[0025] The trained machine learning model 314 utilizes the airport domain 316 to perform automatic clustering and annotation operations specific to the airport environment. The trained machine learning model 314 is configured to automatically cluster various points in the non-ground region 312 into multiple non-ground segments 324 (e.g., 1, 2, N). The trained machine learning model 314 typically does not perform clustering of points in the ground region 310 to reduce resource consumption and the time required to perform the clustering operations.
[0021]
[0026] The non-ground segments 324 represent airport-related objects typically found in an airport environment. The airport domain 316 informs the trained machine learning model 314 of information about the characteristics of the airport-related objects. The trained machine learning model 314 can then recognize and use that information to make clustering decisions about the non-ground segments 324.
[0022]
[0027] The trained machine learning model 314 is configured to automatically assign classification labels 326 to a plurality of non-ground segments 324 selected from a library of classification labels 318 for the airport domain 316. In some examples, the non-ground segments 324 corresponding to movable objects may be assigned dynamic semantic labels 320. In some examples, the non-ground segments 324 corresponding to fixed objects may be assigned non-dynamic semantic labels 322.
[0023]
[0028] In some implementations, the trained machine learning model 314 is configured to automatically assign one or more classification labels 326 to the plurality of non-ground segments 324, and one or more confidences corresponding to the one or more classification labels. The confidence indicates a level of correlation between the non-ground segment and a particular classification label. For example, a higher confidence indicates a stronger correlation with a particular classification label, and vice versa. For example, a confidence of greater than 90% may indicate a strong correlation that the non-ground segment is, in fact, a particular type of classification. It should be noted that the confidence may be evaluated based on any suitable threshold. In one example, the non-ground segment may be classified as an aircraft with a confidence of 95% and as a refueling truck with a confidence of 5%. In this example, the non-ground segment has a strong correlation with the aircraft and a weak correlation with the refueling truck. In some implementations, the non-ground segment may be assigned multiple classification labels based at least on the highest determined confidence corresponding to one classification label being less than a threshold. The trained machine learning model 314 may be configured to automatically assign any suitable number of classification labels and corresponding confidence levels to the plurality of non-ground segments 324.
[0024]
[0029] The trained machine learning model 314 is configured to output a global, labeled lidar point cloud 328 that includes a number of non-ground segments 324 and points in the ground domain 310 that are assigned respective classification labels 326. The points in the ground domain 310 are added back into the global, labeled lidar point cloud 328, thereby providing a comprehensive representation of the entire airport environment, including airport-related objects that are accurately identified and classified as non-ground segments 324.
[0025]
[0030] In some implementations, the trained machine learning model 314 is configured to generate a plurality of masks 330 corresponding to the plurality of non-ground segments 324. In particular, each mask 330 forms a solid surface of a non-ground segment (e.g., an airport-related object) that collectively generates a solid representation of the non-ground segment. The trained machine learning model 314 may be configured to texture map the plurality of masks 330 onto the plurality of non-ground segments 324 in the panoramic, labeled lidar point cloud 328 to make it easier to visually recognize objects in the panoramic, labeled lidar point cloud 328.
[0026]
[0031] In some implementations, the trained machine learning model 314 is configured to provide the plurality of non-ground segments 324 with an individual identifier 330. By assigning an individual identifier 330 to each non-ground segment 324, portions of the plurality of non-ground segments 324 may be tracked via their respective individual identifiers 330 across a panoramic, labeled lidar point cloud output by the processing pipeline 301 over a specified period of time. Such position tracking may allow dynamic non-ground segment behaviors or tendencies to be learned and applied to retrain the trained machine learning model 314 to improve classification accuracy and / or learn additional features.
[0027]
[0032] In some implementations, the mask generation function and / or the annotation function of individual identifiers may be performed by other logic in the processing pipeline 301 instead of the trained machine learning model 314.
[0028]
[0033] The trained machine learning model 314 may take a variety of forms. In some implementations, the trained machine learning model 314 is a neural network. In some implementations, the neural network is a convolutional neural network. In some such implementations, the convolutional neural network is a dynamic graph convolutional neural network. In some implementations, the neural network is a fully connected neural network. In some implementations, the neural network is a one-stage type neural network, such as a single-shot detection neural network. In some implementations, the neural network is a two-stage type neural network, such as a feature pyramid neural network or a region proposal neural network. The trained machine learning model 314 may be any suitable type of neural network trained to classify non-ground segments with airport-based classification labels using any suitable object detection and / or localization techniques.
[0029]
[0034] In some implementations, the processing pipeline 301 includes labeling logic 334 configured to receive one or more images 336 of the airport environment captured by one or more cameras. For example, an aircraft equipped with a lidar device may also be equipped with an RGB camera, an infrared camera, a night vision camera, and / or other types of cameras that output images of the airport environment. The labeling logic 334 is configured to receive a global, labeled lidar point cloud 328. The labeling logic 334 is configured to automatically assign classification labels 326 to a plurality of objects in the one or more images 336 that correspond to a plurality of non-ground segments 324 in the global, labeled lidar point cloud 328. In such implementations, the classification information in the global, labeled lidar point cloud 328 can be utilized to automatically label the image(s) without any human involvement and in a manner that is faster and more accurate than human annotation of the lidar point cloud. Such labeled images can be beneficially used for a variety of applications, such as input for automated aircraft navigation systems or as training data for training machine learning models to automatically label images of airport environments.
[0030]
[0035] In some implementations, the global labeled lidar point cloud 328 is included in the set of training data 340. In some such implementations, the computing system 300 includes a model trainer 342. The model trainer 342 is configured to retrain the trained machine learning model 314 based at least on the set of training data 340. For example, the set of training data 340 may include a plurality of global labeled lidar point clouds 328 generated over a specified period of time. Such retraining of the trained machine learning model 314 enables the trained machine learning model 314 to self-learn relevant features directly from the global labeled lidar point cloud 328 to improve the accuracy of clustering and classification at each retraining cycle without the need for human involvement in the annotation process. The model trainer 342 may retrain the trained machine learning model 314 based at least on any suitable number of global labeled lidar point clouds 328. The model trainer 342 may retrain the trained machine learning model 314 in any suitable manner and according to any suitable frequency. In some embodiments, the frequency at which the model is retrained depends on the residual error of the model, and retraining is performed when new data is processed and new / corner cases are discovered.
[0031]
[0036] FIG. 4 illustrates an example process flow 400. According to this flow 400, the processing pipeline 301 shown in FIG. 3 generates a global, labeled lidar point cloud 328. At 402, the lidar point cloud is received as an input. At 404, the lidar point cloud is segmented via segmentation logic 306 into a ground region 406 of points corresponding to the ground and a non-ground region 408 of points corresponding to points above the ground. At 410, points in the non-ground region 408 are clustered into multiple non-ground segments via trained machine learning model 314. At 412, the non-ground segments are labeled with classification labels selected from the airport domain via trained machine learning model 314. At 414, the global, labeled lidar point cloud 328 is output. At a glance, the labeled lidar point cloud 328 includes multiple non-ground segments and points in the ground region with respective classification labels.
[0032]
[0037] 5-6 show a flowchart of an exemplary computer-implemented method 500 for automatically applying classification labels to a lidar point cloud to generate an overview, labeled lidar point cloud. For example, the computer-implemented method 500 may be performed by the computing system 300 shown in FIG. 3 or another computing system.
[0033]
[0038] 5, at 502, a computer-implemented method 500 includes receiving lidar data of an airport environment. The lidar data includes a lidar point cloud that includes a plurality of points within the airport environment. Each point included in the lidar point cloud includes an X, Y, and Z coordinate and an intensity value (i).
[0034]
[0039] In some embodiments, at least a portion of the lidar data may be received at 502 from a lidar device mounted on an aircraft, such as the lidar device 104 mounted on the aircraft 100 shown in FIG. 1 .
[0035]
[0040] In some implementations, at 506, the lidar point cloud may include points within the airport environment aggregated from multiple scans of one or more lidar devices. In some implementations, the multiple scans may be performed by the same lidar device, such as a lidar device mounted on an aircraft. In other implementations, the multiple scans may be performed by different lidar devices, such as lidar devices mounted on different vehicles.
[0036]
[0041] At 508, the computer-implemented method 500 includes segmenting the lidar point cloud into ground and non-ground regions. Any suitable segmentation technique may be used to segment the lidar point cloud. In one example, the lidar point cloud is segmented by employing a plane fitting segmentation technique applied to the ground-registered points within the airport environment.
[0037]
[0042] At 510, the computer-implemented method 500 includes automatically clustering, via the trained machine learning model, various points in the non-ground region into a plurality of non-ground segments. The non-ground segments may correspond to various objects in an airport environment, such as buildings, vehicles, trees, and other objects. For example, the trained machine learning model may be trained on a set of training data including a lidar point cloud of an airport environment, including various airport-related objects (e.g., aircraft, ground vehicles, terminals, air traffic control towers, runways) at various points in the lidar point cloud.
[0038]
[0043] In some embodiments, at 512, the computer-implemented method 500 may include generating a plurality of masks corresponding to the plurality of non-ground segments, each of the plurality of masks connecting groups of points in the non-ground segments to form a plurality of surfaces to more easily visualize identified objects in the lidar point cloud.
[0039]
[0044] At 514, the computer-implemented method 500 includes automatically assigning, via the trained machine learning model, classification labels selected from an airport domain to the plurality of non-ground segments. The airport domain includes a library of classification labels corresponding to objects related to airports.
[0040]
[0045] In some implementations, the classification labels include dynamic semantic labels for movable objects and non-dynamic semantic labels for fixed objects at 516. These semantic classification labels identify the objects as dynamic or non-dynamic, as well as identifying the objects as part of a more specific airport-related class, such as an aircraft, ground vehicle, terminal, air traffic control tower, runway, or another class of airport-related object.
[0041]
[0046] In some implementations, at 518, the computer-implemented method 500 may include assigning an individual identifier to each of the multiple non-ground segments via the trained machine learning model. Such individual identifiers enable the positions of the multiple non-ground segments (e.g., dynamic airport-related objects such as aircraft or ground vehicles) to be tracked across multiple, panoramic, labeled lidar point clouds generated over a specified time period.
[0042]
[0047] 6, at 520, the computer-implemented method 500 includes outputting a global, labeled lidar point cloud including a plurality of non-ground segments and points in the ground area with respective classification labels. The global, labeled lidar point cloud provides a comprehensive representation of the airport environment, including precisely identified and classified airport related objects.
[0043]
[0048] In some implementations, the global, labeled lidar point cloud includes multiple non-ground segments with individual identifiers at 522. The individual identifiers allow the multiple non-ground segments to be tracked across the multiple global, labeled lidar point clouds generated over a specified time period.
[0044]
[0049] In some implementations, the global, labeled lidar point cloud includes masks aligned to the non-ground segments at 524. The masks form surfaces of airport-related objects that allow the airport-related objects to be more easily recognized.
[0045]
[0050] In some implementations, at 526, the computer-implemented method 500 may include receiving one or more images of the airport environment captured by one or more cameras. For example, an aircraft equipped with a lidar device may also be equipped with an RGB camera, an infrared camera, a night vision camera, and / or other types of cameras that provide images of the airport environment.
[0046]
[0051] In some implementations, at 528, the computer-implemented method 500 may include automatically assigning classification labels to a plurality of objects in one or more images that correspond to a plurality of non-ground segments in the globally labeled lidar point cloud. In such implementations, the classification information in the globally labeled lidar point cloud may be utilized to automatically label an image(s) without any human involvement and in a manner that is faster and more accurate than human annotation of the lidar point cloud. Such labeled images may be beneficially used for a variety of applications, such as input for automated aircraft navigation systems and training data for training machine learning models to automatically label images of airport environments.
[0047]
[0052] In some implementations, at 530, the computer-implemented method 500 may include retraining the trained machine learning model based at least on a set of training data including the global, labeled lidar point clouds. For example, the set of training data may include multiple global, labeled lidar point clouds generated over a specified time period. Such retraining of the trained machine learning model enables the trained machine learning model to self-learn relevant features (e.g., aircraft, ground vehicles, terminals, hangars, air traffic control towers, and other airport-related objects) directly from the global, labeled lidar point clouds to improve clustering and classification accuracy with each retraining cycle without the need for human involvement in the annotation process.
[0048]
[0053] The above systems and methods can be employed to perform unsupervised segmentation, clustering, and annotation of points in a lidar point cloud using trained machine learning models that generate a global, labeled lidar point cloud that is faster and has higher classification accuracy than human-annotated lidar point clouds.
[0049]
[0054] In some embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0050]
[0055] Figure 7 illustrates generally one exemplary implementation of a computing system 700 that may implement one or more of the methods and processes described above. The computing system 700 may embody the computing system 300 illustrated in Figure 3. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, networked computing devices, mobile computing devices (e.g., smartphones), wearable computing devices such as smart watches or head-mounted augmented reality devices, embedded computing devices, Internet of Things (IoT) computing devices, and / or other computing devices.
[0051]
[0056] Computing system 700 includes a logical processor 702, a volatile memory 704, and a non-volatile storage device 706. Computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communications subsystem 712, and / or other components not shown in FIG.
[0052]
[0057] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions, which may be part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve a technical effect, and / or otherwise arrive at a desired result.
[0053]
[0058] A logical processor may include one or more physical processors (e.g., hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of logical processor 702 may be single-core or multi-core, and instructions executed by the processors may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed across two or more separate devices. These devices may be remotely located and / or configured for coordinated processing. Aspects of a logical processor may be virtualized and executed by remotely accessible networked computing devices configured as a cloud computing configuration. In such cases, those skilled in the art will recognize without undue experimentation that these virtualized aspects are executed on different physical logical processors of different machines.
[0054]
[0059] Non-volatile storage device 706 includes one or more physical devices configured to hold instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device 706 may be transformed, for example to hold various data.
[0055]
[0060] The non-volatile storage device 706 may include removable and / or built-in physical devices. The non-volatile storage device 706 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory), and / or magnetic memory (e.g., hard disk drives, floppy disk drives, tape drives, MRAM), and / or other mass storage device technologies. The non-volatile storage device 706 may include non-volatile, dynamic, static, read / write, serial access, location addressable, file addressable, and / or content addressable devices. Those skilled in the art will recognize without undue experimentation that the non-volatile storage device 706 is configured to retain instructions even when power to the non-volatile storage device 706 is interrupted or lost.
[0056]
[0061] Volatile memory 704 may include physical devices including random access memory (RAM). Volatile memory 704 is typically utilized by logic processor 702 to temporarily store information during the processing of software instructions. Those skilled in the art will appreciate without undue experimentation that volatile memory 704 typically does not continue to store instructions if power to volatile memory 704 is interrupted or discontinued.
[0057]
[0062] The logical processor 702, volatile memory 704, and non-volatile storage device 706 may be integrated together into one or more hardware logic components, which may include, for example, a field programmable gate array (FPGA), a program specific integrated circuit / application specific integrated circuit (PASIC / ASIC), a program specific standard product / application specific standard product (PSSP / ASSP), a system on a chip (SOC), and a complex programmable logic device (CPLD).
[0058]
[0063] As used herein, the terms "module," "program," and "engine" may describe an aspect of the computing system 700 that is typically implemented in software by a processor to perform a particular function using portions of volatile memory, the function including a transformation operation that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via the logical processor 702 that executes instructions held by the non-volatile storage device 706 using portions of the volatile memory 704. Those skilled in the art will appreciate without undue experimentation that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. As used herein, the terms "module," "program," and "engine" may encompass individual or groupings of executable files, data files, libraries, drivers, scripts, database records, etc.
[0059]
[0064] When included, the display subsystem 708 can be used to present a visual representation of the data held by the non-volatile storage device 706. This visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 can likewise be transformed to visually represent the changes in the underlying data. The display subsystem 708 can include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 702, the volatile memory 704, and / or the non-volatile storage device 706 in a shared enclosure, or such display devices may be peripheral display devices.
[0060]
[0065] When included, the input subsystem 710 may include or interact with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some implementations, the input subsystem may include or interact with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the delivery and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition, infrared cameras, color cameras, stereo cameras, and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition, as well as electric field sensing components for assessing brain activity, and / or any other suitable sensors.
[0061]
[0066] If included, the communications subsystem 712 may be configured to communicatively couple the various computing devices described herein with each other and / or with other devices. The communications subsystem 712 may include wired and / or wireless communication devices compatible with one or more of a variety of communication protocols. As some examples, the communications subsystem may be configured for communication over a wireless telephone network, or a wired or wireless local or wide area network (e.g., HDMI over a Wi-Fi connection). In some implementations, the communications subsystem may enable the computing system 700 to send and / or receive messages to and from other devices over a network, such as the Internet.
[0062]
[0067] In one embodiment, a computing device includes one or more processors configured to execute instructions stored in a memory to receive Laser Detection and Ranging (LiDAR) data of an airport environment, the LiDAR data including a LiDAR point cloud including a plurality of points within the airport environment, partition the LiDAR point cloud into a ground region and a non-ground region, automatically cluster various points within the non-ground region into a plurality of non-ground segments via a trained machine learning model, automatically assign classification labels selected from an airport domain to the plurality of non-ground segments via the trained machine learning model, and output a labeled LiDAR point cloud including the plurality of non-ground segments and points within the ground region with the respective classification labels. In this and / or other embodiments, the LiDAR point cloud may include points within the airport environment aggregated from multiple scans of one or more LiDAR devices. In this and / or other embodiments, at least a portion of the LiDAR data may be received from one or more LiDAR devices mounted on an aircraft. In this and / or other embodiments, the airport domain may include a library of classification labels corresponding to objects associated with an airport. In this and / or other embodiments, each point included in the lidar point cloud may include an X, Y, and Z coordinate and an intensity value. In this and / or other embodiments, the classification labels may include a dynamic semantic label for movable objects and a non-dynamic semantic label for fixed objects. In this and / or other embodiments, the one or more processors are configured to execute instructions stored in the memory to generate a plurality of masks corresponding to the plurality of non-ground segments, and the labeled lidar point cloud includes the plurality of masks. In this and / or other embodiments, the one or more processors may be configured to assign an individual identifier to each of the plurality of non-ground segments.In this and / or other embodiments, one or more processors may be configured to execute instructions stored in memory to receive one or more images of an airport environment captured by one or more cameras and automatically assign classification labels to objects in the one or more images that correspond to non-ground segments in the labeled lidar point cloud. In this and / or other embodiments, the labeled lidar point cloud may be included in a set of training data, and the one or more processors may be configured to execute instructions stored in memory to retrain a trained machine learning model based at least on the set of training data.
[0063]
[0068] In another example, a computer-implemented method includes receiving lidar data of an airport environment, the lidar data including a lidar point cloud including a plurality of points within the airport environment, partitioning the lidar point cloud into a ground region and a non-ground region, automatically clustering various points within the non-ground region into a plurality of non-ground segments via a trained machine learning model, automatically assigning classification labels selected from the airport domain to the plurality of non-ground segments via the trained machine learning model, and outputting a labeled lidar point cloud including the plurality of non-ground segments and points within the ground region with the respective classification labels. In this and / or other examples, the computer-implemented method further includes generating a plurality of masks corresponding to the plurality of non-ground segments, the labeled lidar point cloud including the plurality of masks. In this and / or other examples, the computer-implemented method further includes assigning an individual identifier to each of the plurality of non-ground segments via the trained machine learning model. In this and / or other examples, the computer-implemented method further includes receiving one or more images of an airport environment captured by one or more cameras, and automatically assigning classification labels to objects in the one or more images corresponding to non-ground segments in the labeled lidar point cloud. In this and / or other examples, the labeled lidar point cloud may be included in a set of training data, and the computer-implemented method may further include retraining the trained machine learning model based at least on the set of training data. In this and / or other examples, the lidar point cloud may include points in the airport environment aggregated from multiple scans of one or more lidar devices. In this and / or other examples, at least a portion of the lidar data may be received from one or more lidar devices mounted on an aircraft. In this and / or other examples, the airport domain may include a library of classification labels corresponding to objects associated with an airport.In this and / or other embodiments, the classification labels may include dynamic semantic labels for movable objects and non-dynamic semantic labels for fixed objects.
[0064]
[0069] In yet another embodiment, a computing device comprises one or more processors configured to execute instructions stored in a memory to: receive lidar data of an airport environment, the lidar data including a lidar point cloud including a plurality of points within the airport environment, partition the lidar point cloud into a ground region and a non-ground region, automatically cluster various points within the non-ground region into a plurality of non-ground segments via a trained machine learning model, automatically assign classification labels selected from the airport domain to the plurality of non-ground segments via the trained machine learning model, output a labeled lidar point cloud including the plurality of non-ground segments and points in the ground region with the respective classification labels, the labeled lidar point cloud being within a set of training data, and retrain the trained machine learning model based at least on the set of training data including the labeled lidar point cloud.
[0065]
[0070] Those skilled in the art will appreciate without undue experimentation that the configurations and / or approaches described herein are exemplary in nature and that these specific implementations or examples should not be considered in a limiting sense, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, simultaneously, or omitted. Similarly, the order of the processes described above may be altered.
[0066]
[0071] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various process, system, and configurations disclosed herein, as well as other features, functions, operations, and / or properties, and any and all equivalents thereof. [Explanation of symbols]
[0067] 100: Aircraft 102:Airport environment 104: Lidar device 106: Representative Pulse 200: Lidar point cloud 202:Points 300: Computing Systems 301: Processing Pipeline 302: Receive rider data 304: Lidar point cloud 306: Segmentation Logic 308: Segmented Lidar Point Cloud 310: Ground area 312: Non-terrestrial area 314: Trained machine learning models 316:Airport domain 318: Classification labels 320; Dynamic Semantic Labels 322: Non-dynamic semantic labels 324: Non-terrestrial segments 326: Classification labels 328: A labelled lidar point cloud with a complete overview 330: Mask 334: Labeling Logic Images with a ratio of 336:1 or more 340: Training data 342: Model trainer 400: Process Flow 406: Ground area 408: Non-terrestrial area
Claims
1. A computing device (300) comprising one or more processors (702), the one or more processors (702) executing instructions stored in a memory (704) to: receiving laser imaging detection and ranging (LiDAR) data (302) of an airport environment (200), the LiDAR data including a LiDAR point cloud (304) including a plurality of points (202) within the airport environment (200); partitioning the lidar point cloud into ground regions (310) and non-ground regions (312); automatically clustering various points within the non-ground region into a plurality of non-ground segments (324) via a trained machine learning model (314); automatically assigning classification labels (326) selected from an airport domain (316) to the non-ground segments (324) via the trained machine learning model (314); and and outputting a labeled lidar point cloud (328) including the plurality of non-ground segments (324) and the points within the ground region (310) with respective classification labels (326).
2. The computing device (300) of claim 1 , wherein the LIDAR point cloud comprises points within the airport environment aggregated from multiple scans of one or more LIDAR devices.
3. The computing device (300) of claim 1 , wherein at least a portion of the lidar data is received from one or more lidar devices (104) mounted on an aircraft.
4. The computing device of claim 1 , wherein the airport domain includes a library of classification labels corresponding to airport-related objects.
5. The computing device of claim 1 , wherein each point included in the lidar point cloud includes an X, Y, and Z coordinate and an intensity value.
6. 2. The computing device (300) of claim 1, wherein the classification labels (318) include dynamic semantic labels (320) for movable objects and non-dynamic semantic labels (322) for fixed objects.
7. 2. The computing device (300) of claim 1, wherein the one or more processors (702) are configured to execute instructions stored in a memory (704) to generate a plurality of masks (332) corresponding to the plurality of non-ground segments (324), and the labeled lidar point cloud (328) includes the plurality of masks (332).
8. The computing device (300) of claim 1, wherein the one or more processors (702) are configured to provide an individual identifier (330) to each of the plurality of non-ground segments (324).
9. The one or more processors (702) execute instructions stored in a memory (704) to receiving one or more images (336) of the airport environment (200) captured by one or more cameras; and 2. The computing device (300) of claim 1, configured to automatically assign classification labels (326) to a plurality of objects in the one or more images (336) that correspond to the plurality of non-ground segments (324) in the labeled lidar point cloud (328).
10. 2. The computing device (300) of claim 1, wherein the labeled lidar point cloud (328) is included in a set of training data (340), and the one or more processors (702) are configured to execute instructions stored in a memory (704) to retrain the trained machine learning model (314) based at least on the set of training data (340).
11. receiving (502) lidar data (302) of an airport environment (200), the lidar data (302) including a lidar point cloud (304) including a plurality of points (202) within the airport environment (200); Segmenting (508) the lidar point cloud (304) into ground regions (310) and non-ground regions (312); automatically clustering (510) various points within the non-ground region (312) into a plurality of non-ground segments (324) via a trained machine learning model (314); automatically assigning (514) classification labels (326) selected from an airport domain (316) to the non-ground segments (324) via the trained machine learning model (314); and A computer-implemented method (500) comprising outputting (502) a labeled lidar point cloud (328) including the plurality of non-ground segments (324) and the points within the ground region (310) with respective classification labels (326).
12. 12. The computer-implemented method (500) of claim 11, further comprising generating (512) a plurality of masks (332) corresponding to the plurality of non-ground segments (324), wherein the labeled lidar point cloud (328) includes the plurality of masks (332).
13. 12. The computer-implemented method (500) of claim 11, further comprising assigning an individual identifier (330) to each of the plurality of non-ground segments (324) via the trained machine learning model (314).
14. receiving (526) one or more images (336) of the airport environment (200) captured by one or more cameras; and 12. The computer-implemented method (500) of claim 11, further comprising automatically assigning classification labels (326) to a plurality of objects in the one or more images (336) that correspond to the plurality of non-ground segments (324) in the labeled lidar point cloud (328).
15. 12. The computer-implemented method (500) of claim 11, wherein the labeled lidar point cloud (328) is included within a set of training data (340), and the computer-implemented method (500) further comprises retraining (530) the trained machine learning model (314) based at least on the set of training data (340).
16. 12. The computer-implemented method (500) of claim 11, wherein the lidar point cloud (304) comprises points (202) within the airport environment (200) aggregated from multiple scans of one or more lidar devices (104).
17. 12. The computer-implemented method of claim 11, wherein at least a portion of the lidar data is received from one or more lidar devices mounted on an aircraft.
18. 12. The computer-implemented method of claim 11, wherein the airport domain includes a library of classification labels corresponding to airport-related objects.
19. 12. The computer-implemented method (500) of claim 11, wherein the classification labels (318) include dynamic semantic labels (320) for movable objects and non-dynamic semantic labels (322) for fixed objects.
20. A computing device (300) comprising one or more processors (702), the one or more processors (702) executing instructions stored in a memory (704) to: receiving lidar data (302) of an airport environment (200), the lidar data (302) including a lidar point cloud (304) including a plurality of points (202) within the airport environment (200); partitioning the lidar point cloud (304) into ground regions (310) and non-ground regions (312); automatically clustering various points within the non-ground region (312) into a plurality of non-ground segments (324) via a trained machine learning model (314); automatically assigning classification labels (326) selected from an airport domain (316) to the non-ground segments (324) via the trained machine learning model (314); outputting a labeled lidar point cloud (328) including the non-ground segments (324) and the points in the ground region (310) with respective classification labels (326), the labeled lidar point cloud (328) being within a set of training data (340); and retraining the trained machine learning model (314) based at least on the set of training data (340) that includes the labeled lidar point cloud (328).
