A method of and system for classifying a light detection and ranging point

Abstract

A method of classifying a light detection and ranging, lidar, point is disclosed, wherein the lidar point is associated with one or more features in an area being surveyed. The method is performed by a processor and comprises: training two or more models based on respective training datasets, each dataset being associated with respective classes of one of the two or more models; calculating, using the trained two or more models, corresponding probabilities of the lidar point being associated with the respective classes of the two or more models; and classifying the lidar point to a class of one of the two or more models by combining the calculated corresponding probabilities. Unlocking insights from Geo-Data, the present invention further relates to improvements in sustainability and environmental developments: together we create a safe and liveable world.

Description

A METHOD OF AND SYSTEM FOR CLASSIFYING A LIGHT DETECTION ANDRANGING POINTFIELD OF THE INVENTION

[0001] The present specification relates broadly, but not exclusively, to methods and systems of classifying a light detection and ranging, lidar, data point. Unlocking insights from Geo-Data, the present invention further relates to improvements in sustainability and environmental developments: together we create a safe and liveable world.BACKGROUND OF THE INVENTION

[0002] Light detection and ranging, lidar, is an optical remote-sensing technique that uses light in the form of a pulsed laser to sample a surface of the earth. Distances to the earth can be measured in three axes. The measurements can be analyzed and processed to generate precise three-dimensional information about surface features of the earth. The surface features can then be processed and a lidar point can be classified. Lidar can be a cost-effective alternative to traditional surveying techniques such as photogrammetry.

[0003] A typical method of classifying a lidar data point comprises collecting a training dataset and training a model based on the collected training dataset. If a new class is desired, the collected training dataset needs to be updated or a new training dataset needs to be collected. Thereafter, the model needs to be re-trained based on the updated training dataset or new training dataset. This process may incur relatively high costs and may require a relatively long timeline.

[0004] A need therefore exists to provide a method of classifying a lidar data point that seeks to address at least some of the above problems. Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.BRIEF SUMMARY OF THE INVENTION

[0005] In one aspect of the invention there is provided a method of classifying a light detection and ranging, lidar, point, wherein the lidar point is associated with one or more features in an area being surveyed, the method is performed by a processor and comprises:

[0006] - training two or more models based on respective training datasets, each dataset being associated with respective classes of one of the two or more models;

[0007] - calculating, using the trained two or more models, corresponding probabilities of the lidar point being associated with the respective classes of the two or more models; and

[0008] - classifying the lidar point to a class of one of the two or more models by combining the calculated corresponding probabilities.

[0009] The present disclosure is based on the inventors’ insight that using multiple models trained on specific datasets can help to achieve higher accuracy in classification. Each model is specialized in recognizing specific features, which will lead to more accurate predictions.

[0010] Having multiple models provides robustness against uncertainties or variations in the data. If one model fails to accurately classify a point, other models can compensate for it, leading to more reliable results.

[0011] Besides, the method of the present disclosure allows for flexibility in handling complex classification tasks involving multiple classes. Each model can focus on learning distinct features of the data, making it easier to manage and interpret the results.

[0012] Overall, the method of the present disclosure can be adapted to different surveying environments and applications by training models with relevant datasets. This adaptability makes it suitable for a wide range of Lidar-based classification tasks.

[0013] In an example of the present disclosure, a first set of probabilities of the lidar point being associated with the respective classes of a first model is calculated based on a first model of the two or more models, the first model being an existing model; a second set of probabilities of the lidar point being associated with the respective classes of a second model is calculated based on a second model of the two or more models, the second model being a model trained after the lidar point is classified based on the first set of probabilities; the step of classifying comprises reclassifying the lidar point by combining the first set of probabilities and the second set of probabilities.

[0014] This method is especially advantageous in that it provides a sequential process where the lidar point classification is refined or adapted iteratively by using multiple models.

[0015] The lidar point is originally classified using an existing model, referred to as the first model. This model has already been trained on a specific dataset and provides the first set of probabilities indicating the likelihood of the lidar point belonging to each class.

[0016] After the initial classification, a second model, which could be a more specialized or refined version, is trained based on classes that can be the same or different than those classes of the first model. This second model is trained to further improve the classification accuracy, potentially focusing on correcting any misclassifications made by the first model.

[0017] The lidar point is then reclassified by combining the probabilities obtained from both the first and second models. This combination could involve various techniques such as weighted averaging or using a more complex fusion mechanism to generate a more accurate and refined classification result.

[0018] By using an iterative process of classification and refinement, the method improves the accuracy of classification. Moreover, this approach allows for adaptation to evolving data or changing environmental conditions. As new data becomes available or as the understanding of the survey area improves, new models can be trained and integrated into the classification process to enhance accuracy and adaptability.

[0019] By leveraging the strengths of both existing and newly trained models, this approach can lead to enhanced performance in classification tasks. The combination of different models can capture a broader range of features and nuances in the data, resulting in more accurate and robust classification results.

[0020] Overall, this iterative approach to lidar point classification offers advantages in terms of accuracy, adaptability, and performance, making it well-suited for applications requiring precise and reliable classification results in dynamic environments.

[0021] In an embodiment, the two or more models are binary or multi-class models.

[0022] In an embodiment, the second model may be a binary model.

[0023] When the second model is a binary model, it can be designed to address specific misclassifications or unsatisfying classification from the first model. By targeting only the classes that require correction, the second model can efficiently improve the accuracy of the overall classification process without the need of training the whole first model completely or unnecessarily reclassifying all classes.

[0024] Binary models can provide precise corrections for misclassified lidar points. Instead of providing probabilities for multiple classes, they specifically indicate whether a point should be reclassified or not. This precision ensures that only the necessary adjustments are made, minimizing the risk of overcorrection or introducing new errors.

[0025] Moreover, the second binary model can also identify new features or patterns in the lidar data that were not initially captured by the first model. By focusing on detecting new features not captured by the initial classification, the second model can uncover previously unrecognized features, leading to a more comprehensive understanding of the surveyed area.

[0026] The use of a binary model for correction allows for efficient iterative improvement of the classification process. Instead of retraining the entire model from scratch or performing exhaustive reclassifications, the second model specifically targets areas for improvement identified by the first model, leading to faster convergence towards a more accurate classification result.

[0027] Binary models can easily adapt to changing environmental conditions or evolving datasets. As new data becomes available or as the survey area undergoes changes, the binary model can quickly identify and address any discrepancies or new features, ensuring that the classification remains accurate and up-to-date.

[0028] In an embodiment, classes associated with the second model may be subclasses of classes associated with the first model.

[0029] The method of the present disclosure can be conveniently used to refine existing classifications decided by the first model. As an example, when the first model provides classifications including road, vehicle, buildings, the class vehicle can be refined to differentiate different vehicle types which are present in the area under survey.

[0030] In an embodiment, classes associated with the second model may be independent of classes of the first model.

[0031] When classes associated with the second model are independent of classes of the first model, it means that the second model is designed to address different aspects or features of the Lidar data that are not explicitly captured by the first model's classification scheme.

[0032] The independence of classes between the two models allows the second model to focus on identifying and classifying features or phenomena in the lidar data that were not considered in the initial classification by the first model. For example, the first model may focus on classifying natural terrain features such as vegetation and terrain elevation, while the second model may focus on identifying man-made structures or specific types of vegetation that were not originally included in the first model's classification scheme.

[0033] By having independent classes, the second model can provide a more comprehensive coverage of the surveyed area, capturing a wider range of features and phenomena that may be present. This comprehensive coverage enhances the overall understanding of the environment being surveyed and provides valuable insights for variousapplications such as urban planning, environmental monitoring, and infrastructure management.

[0034] The independent classes of the second model can complement the classification results of the first model by providing additional information or correcting misclassifications. This is more advantages when more than one new models are trained to cover features which were not identified (in enough detail) by the first model.

[0035] In a sense, the independence of classes between the two models makes the classification framework more adaptable to diverse environments and applications. As the surveyed area or the requirements of the classification task change, new classes can be introduced in the second model to address specific features or phenomena relevant to the new environment or application, without requiring significant modifications to the first model.

[0036] In an embodiment, the lidar point is classified to a class with a highest probability of the probabilities of the lidar point being associated with the respective classes.

[0037] After calculating the probabilities of the lidar point belonging to each class, the classification algorithm selects the class with the highest probability as the predicted class for the point. This is a straightforward and intuitive approach where the class with the highest likelihood is chosen as the most probable classification for the point.

[0038] Selecting the class with the highest probability tends to lead to higher accuracy in classification. The class with the highest probability represents the model's best estimate of the true class for the Lidar point, based on the available data and the model's learning. This approach is simple and easy to understand, making it accessible for users without deep knowledge of the underlying classification algorithms or methodologies.

[0039] By focusing on the class with the highest probability, computational resources are directed towards making the most confident classification decisions. This can lead to faster inference times and more efficient processing, especially in real-time applications or large- scale surveys.

[0040] In an embodiment, the lidar point is classified to a class with a larger distance to an associated threshold value of a respective class, if two or more probabilities have a same highest value.

[0041] When two or more probabilities have the same highest value for a Lidar point, classifying the point to a class with a larger distance to an associated threshold value of a respective class means selecting the class that has a greater margin or distance from a predetermined threshold value. This means that even though multiple classes have equally highprobabilities, the class that is farther away from its threshold is considered more confidently predicted by the model.

[0042] By considering the margin between the probabilities and their respective thresholds, this approach accounts for the uncertainty inherent in the classification process. Choosing the class with a larger margin ensures a more confident classification decision.

[0043] The use of threshold values allows for optimization of decision boundaries between classes. By adjusting the thresholds, the classification model can be fine-tuned to prioritize certain classes or minimize misclassifications in specific scenarios.

[0044] In an embodiment, reclassifying the lidar point comprises assigning the lidar point to a class not comprised in the first model.

[0045] When reclassifying a lidar point involves assigning the point to a class not comprised in the first model, it means that the initial classification process did not include this specific class, but through further analysis or refinement, the point is identified as belonging to a previously unrecognized class.

[0046] By identifying and including new classes in the classification process, the accuracy and comprehensiveness of the classification results are enhanced. This ensures that all relevant classes and features present in the lidar data are appropriately recognized and classified, leading to a more accurate representation of the surveyed area.

[0047] In an embodiment, reclassifying the lidar point comprises assigning the lidar point to a class which is a subclass of a class comprised in the first model.

[0048] When reclassifying a lidar point involves assigning the point to a subclass of a class comprised in the first model, it means that the initial classification process identified a broader class in the first model, but through further analysis or refinement, the point is identified as belonging to a more specific subclass within that broader class.

[0049] By assigning Lidar points to more specific subclasses, the reclassification process improves the precision and granularity of the classification results. This allows for a more detailed understanding of the surveyed area, as points are classified into more specific categories that capture finer variations in the landscape or environment.

[0050] Classifying lidar points into subclasses provides a more interpretable and actionable classification result. Subclasses often correspond to meaningful distinctions in the environment, such as different types of vegetation, terrain features, or man-made structures, allowing for more informed decision-making and analysis.

[0051] The identification of subclasses enables more refined analysis and planning in various applications such as urban planning, environmental management, and infrastructuredevelopment. Decision-makers can better understand the composition and distribution of specific features within broader classes, leading to more targeted interventions and strategies.

[0052] In an embodiment, the method may further comprise constructing training datasets such that each class of a model is presented by a substantially equal number of samples from unbalanced training datasets.

[0053] For the models to classify the lidar point accurately and precisely, the selection of the training datasets plays an important role.

[0054] In many real-world scenarios, datasets used for training machine learning models may be imbalanced, meaning some classes or features have significantly fewer samples than others. This can lead to biased models that perform poorly on minority classes.

[0055] To address this issue, the training datasets are constructed or augmented in such a way that each class in the model is represented by a substantially equal number of samples from unbalanced training datasets. This can be achieved by oversampling minority classes, undersampling majority classes, or using other techniques such as data augmentation to create synthetic samples.

[0056] Balancing the training datasets ensures that the model is trained on a representative set of samples from each class, leading to more robust and accurate predictions, especially for minority classes. This helps mitigate biases and ensures that the model is capable of generalizing well to all classes in the classification task.

[0057] In an embodiment, the step of constructing comprises:

[0058] - clustering features in a feature space;

[0059] - sampling training dataset from each class equally from each cluster.

[0060] Following the method of the present disclosure, initially, the features extracted from the lidar data are clustered in a feature space. Clustering involves grouping similar data points together based on their feature representations. This is typically done using clustering algorithms such as k-means, hierarchical clustering, or density-based clustering.

[0061] Once the features are clustered, the training dataset is sampled from each class equally from each cluster. This means that for each class, an equal number of samples are selected from each cluster to ensure balanced representation. This can involve randomly selecting samples from each cluster or using other sampling techniques to ensure an equal distribution.

[0062] By sampling training data equally from each cluster for each class, the constructed training datasets ensure balanced representation of features across different clusters andclasses. This helps prevent biases that may arise from uneven distributions of features in the original datasets.

[0063] Balanced training datasets facilitate better generalization of machine learning models. By ensuring that each class has an equal representation across different clusters, the model learns to recognize and classify features consistently across different contexts or environments.

[0064] In an embodiment, each class may be represented by multiple clusters.

[0065] To get comparable accuracy from challenging / unbalanced binary models, selective training data from different clusters in the feature space is used. As an example, to detect building / not building, for the purpose of describing the feature space boundaries, one would want examples of wall and roof points in the building class. In the not building class, one would like as many examples of everything else, such as, Ground / Vehicles / Powerlines etc.

[0066] The method of the present disclosure therefore allows the training dataset to include substantially equal number of samples representing all features to be included, which helps to have better trained models.

[0067] A second aspect of the present disclosure provides a device for classifying a light detection and ranging, lidar, point, wherein the lidar point is associated with one or more features in an area being surveyed, the device comprising a processor for performing the method according to the first aspect of the present disclosure.

[0068] A third aspect of the present disclosure provides a computer program product, comprising a computer readable storage medium storing instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to the first aspect of the present disclosure.

[0069] The above mentioned and other features and advantages of the disclosure will be best understood from the following description referring to the attached drawing. In the drawings, like reference numerals denote identical parts or parts performing an identical or comparable function or operation.BRIEF DESCRIPTION OF THE DRAWINGSIn order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplaryembodiments of the disclosure and are therefore not to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0070] Figure 1 is a flow chart illustrating an example of a method of classifying a light detection and ranging, lidar, point, according to an embodiment of the present disclousre.

[0071] Figure 2 is a schematic representation illustrating an example of a feature space.

[0072] Figure 3 is a schematic representation illustrating an example of a multi-class model applied to the feature space of Figure 2.

[0073] Figure 4 is a schematic representation illustrating an example of the multi-class model of Figure 3 with additional classes.

[0074] Figure 5 is a schematic representation illustrating an example of a retrained multiclass model of Figure 4.

[0075] Figure 6 is a schematic representation illustrating an example of a method of training an additional model for classifying a lidar point, according to the present disclosure.

[0076] Figure 7 is a schematic representation illustrating an example of a model of the method of classifying the lidar point of Figure 4 trained based on a poorly sampled training dataset.

[0077] Figure 8 is a schematic representation illustrating an example of the model of Figure7 applied to a feature space.

[0078] Figure 9 shows an image of lidar data having ground classification.

[0079] Figure 10 shows the image of lidar data of Figure 9 having building and vegetation classifications.

[0080] Figure 11 shows a schematic diagram of a computer system suitable for use in executing at least some steps of the method of classifying the lidar point.

[0081] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale. For example, the dimensions of some of the elements in the illustrations, block diagrams or flowcharts may be exaggerated in respect to other elements to help to improve understanding of the present embodiments.DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0082] Embodiments contemplated by the present disclosure will now be described in more detail with reference to the accompanying drawings. The disclosed subject matter should notbe construed as limited to only the embodiments set forth herein. Rather, the illustrated embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0083] Some portions of the description which follows are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consi stent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0084] Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions utilizing terms such as “receiving”, “scanning”, “calculating”, “determining”, “replacing”, “generating”, “initializing”, “outputting”, or the like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission or display devices.

[0085] The present specification also discloses apparatus for performing the operations of the methods. Such apparatus may be specially constructed for the required purposes, or may comprise a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various machines may be used with programs in accordance with the teachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be appropriate. The structure of a computer suitable for executing the various methods / processes described herein will appear from the description below.

[0086] In addition, the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be usedto implement the teachings of the specification contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows.

[0087] Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially. Such a computer program may be stored on any computer readable medium. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer. The computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. The computer program when loaded and executed on such a computer effectively results in an apparatus that implements the steps of the preferred method.

[0088] The phrases “lidar point” and “lidar data point” are used in the present disclosure interchangeably to refer to a data record collected using a lidar sensor, which may be visualized as a point on an image created by processing raw lidar data.

[0089] As mentioned above, a typical method of classifying a light detection and ranging, lidar, point comprises collecting a training dataset and training a model based on the collected training dataset. If a new class is desired, the collected training dataset needs to be updated or a new training dataset needs to be collected. Thereafter, the model needs to be re-trained based on the updated training dataset or new training dataset. This process may incur relatively high costs and may require a relatively long timeline.

[0090] The technical concept of reducing multi-class classification problems to one versus rest and one versus one approaches is known in the art. An simple example is give as follows. If one wishes to classify if something is an apple or a watermelon, the feature of weight can be used. A model can be, if > 0.5kg, it is watermelon, otherwise it is apple, this is a multiclass classifier. Alternatively, two binary models may be used to perform the classification. According to one model, it is an apple: if < 0.5kg, else not apple. The other mode is, it is watermelon: if > 2kg, else not watermelon. The approach is primarily used to allow the utilisation of binary model types that cannot be used for multi-class classification tasks.

[0091] Embodiments of the invention provide a method of classifying lidar points which advantageously allows a user to include one or more additional classes to improve classification in specific areas, without impacting on existing results in other areas. In other words, classification of lidar points can be extended to include additional classes without having to update existing training datasets, update the entire model and / or retrain the model. Thisbeneficially reduces human correction, decreases costs and reduces delivery time of completed projects to clients.

[0092] Figure 1 is a flow chart 100 illustrating an example of a method of classifying a light detection and ranging, lidar, point, according to an embodiment of the present disclsoure. The lidar point is associated with one or more features in an area to be / being surveyed. At step 102, two or more models are trained based on respective training datasets, each dataset is associated with respective classes of one of the two or more models. At step 104, corresponding probabilities of the lidar point being associated with the respective classes are calculated using the trained two or more models. At step 106, the lidar point is to a class of one of the two or more models by combining the calculated corresponding probabilities.

[0093] As an example, two models are trained. Each model is trained using its respective training dataset. For example, Model 1 is trained using a dataset that includes classes for vegetation, buildings, and roads, while Model 2 is trained using a dataset that includes classes for trees, vehicles, and pedestrians.

[0094] As many models as necessary can be trained, depending on the interested features to be identified and extracted.

[0095] Once the models are trained, they are used to calculate the probabilities of a certain lidar point belonging to each respective class.

[0096] For instance, Model 1 calculates the probabilities of the lidar point being vegetation, building, or road, while Model 2 calculates the probabilities of the lidar point being a tree, vehicle, or pedestrian.

[0097] Thereafter, the lidar point is classified by combining the calculated probabilities from both models. This could involve various methods such as highest probability or using a voting mechanism.

[0098] For example, if Model 1 assigns probabilities of 0.7 for vegetation, 0.2 for building, and 0.1 for road, and Model 1 assigns probabilities of 0.8 for tree, 0.1 for vehicle, and 0.1 for pedestrian, we might combine these probabilities to determine the final classification. An easier way is to select the highest probability, in this case, the lidar point is classified as tree.

[0099] The machine learning models that can be used are described as follows. The one or models can be multi-class models. As an example, Random Forest is a popular ensemble learning method that can be used for multi-class classification. It builds multiple decision trees and combines their predictions to classify Lidar points into various classes.

[0100] As another example, Support Vector Machine, SVM, is a powerful supervised learning algorithm that can be used for multi-class classification. It works by finding the hyperplane that best separates different classes in the feature space.

[0101] Still an example can be neural networks. Deep learning models such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) can be used for multi-class Lidar point classification. These models can automatically learn complex patterns and features from Lidar data, leading to highly accurate classifications.

[0102] The models that can be used in the present disclosure also include binary models. As an example, Logistic Regression is a binary classification algorithm that can be extended to handle multi-class classification using techniques like one-vs-rest or one-vs-one. In the context of Lidar point classification, logistic regression can be used to classify points into binary classes such as "vegetation" vs. "non-vegetation".

[0103] As another example, Decision Trees can be used as binary classifiers by splitting the feature space into two branches at each node. In the context of lidar point classification, decision trees can be used to classify points into binary classes such as "obstacle" vs. "nonobstacle".

[0104] Binary neural networks are neural networks that use binary activation functions and weights. They are efficient for binary classification tasks and can be trained to classify lidar points into binary classes such as "road" vs. "non-road" or "building" vs. "non-building".

[0105] Figure 2 is a schematic representation 200 illustrating an example of a feature space. As depicted in Figure 2, each class is represented by a different shape. Four classes respectively represented by dots, triangles, stars and squares are shown in Figure 2.

[0106] Figure 3 is a schematic representation 300 illustrating an example of a multi-class model applied to the feature space of Figure 2. The multi-class model may define the boundaries 302, 304, 306, 308 between the classes as shown in Figure 2.

[0107] Figure 4 is a schematic representation 400 illustrating an example of the multi-class model of Figure 3 with additional classes 402, 404, where the class 402 is represented by diamonds and the class 404 is represented by pentagon. As can be seen in Figure 4, the multiclass model is unable to classify the additional classes 402, 404. Hence, the multi-class model may need to be retrained to classify the additional classes 402, 404.

[0108] A conventional way of classifying additional classes 402, 404 is to have the model retrained to cover all classes present, which is timing and resources consuming.

[0109] Figure 5 is a schematic representation 500 illustrating an example of a retrained multi-class model of Figure 4. As can be seen in Figure 5, the retrained multi-class model isable to classify the additional classes 402, 404 with re-defined boundaries 502, 504, 506, 508, 510, 512, 514, 516 between the classes. The training dataset used to retrain the multi-class model need to cover all classes and any detected edge case.

[0110] If a user discovers another example of a given class that has not been correctly determined by the current model, the user will need to retrain the entire multi-class model. That is, the user inputs the training dataset to the multi-class model. The trained multi-class model may be able to determine classes that are represented in that training dataset used.[OHl] However, if data from another area is to be classified, and such has classes which were not in the original trained morel, then the classification can’t class the classes present in the other area. As an example, if the other area has large boulders, if such a class is not considered in the original training dataset, the ground classification would be wrong for the boulders.

[0112] As will be discussed later, embodiments of the invention advantageously allow modification of the existing models instead of modifying or retraining the entire model.

[0113] The method of the present disclosure layering models, which can be multi-class models or binary models, and makes decisions based on probability outputs from each model. For the example of Figure 4, for the purpose of classifying the new class 402, a binary model covering class / non-class 402 can be trained, which is illustrated as Figure 6, in which the new model 800 is shown as covering class 802 (402 of Figure 4).

[0114] In a sense, a first set of probabilities of the lidar point being associated with the respective classes may be calculated based on a first model of the two or more models as described with reference to Figure 1. The first model is an existing model and can be a model as illustrated in Figure 3. The first model may be a multi-class model depicted in Figure 3. The first model can also be a binary model.

[0115] A second set of probabilities of the lidar point being associated with the respective classes may be calculated based on a second model of the two or more models, the second model is a model trained after the lidar points are classified based on the first set of probabilities. The second model can be a model as illustrated in Figure 6.

[0116] In Figure 6, the second model is trained to determine an additional class 802 represented by diamond shapes. Referring to Figure 1, the step of classifying the lidar point 106 may comprise reclassifying the lidar point by combining the first set of probabilities and the second set of probabilities. In other words, reclassifying the lidar point can allow the class of interest to be identified. Advantageously, the existing first model need not be retrained todetermine additional classes. The method of classifying the lidar point can be iterated on the existing model, to refine and / or add additional classes to the existing model.

[0117] For the lidar data as shown in Figure 4, by combining the trained model of Figure 3 and Figure 6, the new class 802 of lidar data point 402 is determined.

[0118] In some implementations, the two or more models may be binary or multi-class models. The second model may be a binary model. Classes associated with the second model may be subclasses of classes associated with the first model. Classes associated with the second model may be independent of classes of the first model.

[0119] For example, the existing first model may be a binary model of vehicle / non-vehicle and the second model may be a binary model of car / non-car. Classifying the lidar point based on the first model and the second model may allow the user to identify three classes, namely vehicle and car, vehicle not car, and non-vehicle.

[0120] As another example, the existing first model may be a multi-class model including vegetation, building, ground, and the second model may be a binary model of powerline / non- powerline. By combining the two models, powerlines can be identified, which may overlap with existing classes.

[0121] In a sense, reclassifying the lidar point may comprise assigning the lidar point to a class not comprised in the first model. In some other implementations, reclassifying the lidar point may comprise assigning the lidar point to a class which is a subclass of a class comprised in the first model.

[0122] It will be understood by those skilled in the art that a plurality of additional models may be trained to cover different features of interest. By combining one or more existing models and the newly trained models, it is possible to classify more features, or refine the original classification.

[0123] As for determining the classification of the lidar point by combining two or more models, the lidar point may be classified to a class with a highest probability of the probabilities of the lidar point being associated with the respective classes. According to another embodiment, the lidar point may be classified to a class with a larger distance to an associated threshold value, if two or more probabilities have a same highest value.

[0124] In order to effectively train the binary models on the features that are included and not included in a particular class, appropriate selection of training datasets may be required. In other words, it may be required to sample both regions effectively in order to derive appropriate boundaries.

[0125] This is because underlying classification models performing the classification operate in the feature space and effectively seek to divide up a region into the different classes. As an example, the binary model is trained to classify Ground vs Other. If there is a basic scene with only flat ground points (at a height of 0m) and some noise scattered above the ground at a height of 5m, feeding those examples to the model and the feature of height into a SVM model, it will divide up the region to maximise the margin between Ground / other and put the border at 2.5m. However, if there is noise / other things that go down to 1.0m they would be miss classed as Ground. Therefore, other examples need to be included in the training dataset to have the model trained properly.

[0126] Figure 7 is a schematic representation 900 illustrating an example of a model of the method of classifying the lidar point of Figure 4 based on a poorly sampled training dataset. As can be seen in Figure 7, an amount of data in the training dataset is relatively limited.

[0127] Figure 8 is a schematic representation 1000 illustrating an example of the model of Figure 7 applied to a feature space. As can be seen in Figure 8, if the training dataset for the model of diamond shape / non-diamond shape is sampled poorly, the region boundary 1002 may incorrectly include some of the non-diamond shape region.

[0128] According to one embodiment, the method of classifying the lidar point may further comprise constructing training datasets such that each class of a model is presented by a substantially equal number of samples from unbalanced training datasets.

[0129] Constructing the training datasets may comprise the step of clustering features in a feature space. Constructing the training datasets may further comprise the step of sampling training dataset from each class equally from each cluster. Each class may be represented by multiple clusters.

[0130] Referring to Figure 8, for example, the clusters for the class represented by nondiamond shapes can represent the classes represented by circle shapes, triangle shapes, star shapes, pentagon shapes and square shapes. Sampling from each of the clusters may represent an underlying data or boundary between the class represented by diamond shapes and nondiamond shapes.

[0131] According to the method of the present disclosure, the problem of unbalanced / challenging classes is resolved by clustering in the feature space, multiple clusters may be present for each class. Training samples for each class is selected equally from each cluster.

[0132] As an example, when a machine learning models with as many as for example forty features is trained, surrounding area of points is considered in terms of how linear it is / howplanar it is etc. In the feature space points that are similar (and belong to the same class) generally lie together. When looking at the 'other' class for binary it often contains many things.

[0133] As an example, for a Ground / non-ground model, in the 'not ground' points there are buildings / vehicles / powerlines etc. These things are not represented evenly, there are not the same number of points on the powerlines as for buildings. By grouping the points in the feature space and then sampling from these evenly, it helps to get a better training set that gives better samples to learn from.

[0134] As a result, even though there are few powerline points and many building points, equal samples are selected of each. Idea is that the clusters in non-ground would represent the classes buildings / vehicles / powerlines etc and sampling from each of these will best represent the underlying data / boundary between ground / non-ground.

[0135] Exemplary steps of constructing the training dataset can be as follows:

[0136] Feature Extraction: Extract relevant features from the lidar point cloud data. These features could include characteristics such as point intensity, elevation, point density, curvature, and neighbourhood attributes (e.g., surrounding area, linearity, planarity).

[0137] Clustering in Feature Space: Utilize clustering algorithms (e.g., k-means, hierarchical clustering) to cluster the lidar points based on their extracted features. Each cluster represents a group of points that are similar in feature space.

[0138] Equal Sampling from Each Cluster: For each cluster e.g., ground, non-ground including buildings / vehicles / powerlines, evenly sample points from each cluster in the feature space. This can be done by iterating through each cluster and class combination, determine the number of points to sample from each cluster based on the desired balance, and randomly select points from each cluster within the specified quota for that cluster. The sampled points from all clusters are then combined to form the balanced training dataset.

[0139] Thereafter labels may be assigned to the sampled points based on their known class e.g., ground or non-ground, which is known to those skilled in the art and will not be elaborated herein.

[0140] As can be contemplated by those skilled in the art, the training dataset may also be shuffled to ensure randomness and prevent any biases during training.

[0141] Thereafter, machine learning models such as Random Forest, SVM, Neural Networks can be trained using the balanced training dataset.

[0142] Embodiments of the invention provide an automated point classification method that can provide a generic scalable cloud-based pipeline combining a high level of accuracy and configurable output based on selection of different models. The models may consist ofindividual building blocks that can be run in combination. The models may be extendable and individually editable. Each model can output the probability of classification of the lidar point, then combined to determine the final classification. For a larger configuration flexibility, a plurality of binary models (i.e. class / non-class models) may be used as opposed to using multiclass models. An example of a multi-class model is a network which outputs different classes for powerlines, poles and towers.

[0143] To obtain a relatively higher accuracy from unbalanced binary models, a technique of selective training data from different clusters in the feature space may be used. For example, the user may wish to detect building / non-building. In order to define the feature space boundaries, the user may use examples of walls and roof points in a building class. In the nonbuilding class, the user may use many examples of everything else (e.g. ground, vehicles, powerlines, etc.). However, some objects may not have as many examples as compared to others. For example, there may be many ground points but very few points on powerlines. Hence, if the user samples randomly, he / she may not get an example of the powerline point and / or the powerline point may be incorrectly included in the building class.

[0144] In some implementations, specific class dependent cleanup steps may be used. The cleanup steps may provide clustering or fitting operations. For example, planes may be fitted for buildings and derived geometries can be used to restrict final classifications or probabilities.

[0145] Figure 9 shows an image of lidar data having ground classification 1200. Figure 10 shows the image of lidar data 1200 of Figure 9 having building 1302 and vegetation 1304 classifications 1300.

[0146] According to one embodiment, a system is provided. The system comprises at least one processor. The system also comprises a non-transitory computer-readable storage medium coupled to the at least one processor and storing programming instructions for execution by the at least one processor, the programming instructions instruct the at least one processor to perform the method of classifying a lidar point.

[0147] Figure 11 shows a schematic diagram of a computer system suitable for use in executing at least some steps of the method of classifying the lidar point.

[0148] The following description of the computer system / computing device 1400 is provided by way of example only and is not intended to be limiting.

[0149] As shown in Figure 11, the example computing device 1400 includes a processor 1404 for executing software routines. Although a single processor is shown for the sake of clarity, the computing device 1400 may also include a multi -processor system. The processor 1404 is connected to a communication infrastructure 1406 for communication with othercomponents of the computing device 1400. The communication infrastructure 1406 may include, for example, a communications bus, cross-bar, or network.

[0150] The computing device 1400 further includes a main memory 1408, such as a random access memory (RAM), and a secondary memory 1410. The secondary memory 1410 may include, for example, a hard disk drive 1412 and / or a removable storage drive 1414, which may include a magnetic tape drive, an optical disk drive, or the like. The removable storage drive 1414 reads from and / or writes to a removable storage unit 1418 in a well-known manner. The removable storage unit 1418 may include a magnetic tape, optical disk, or the like, which is read by and written to by removable storage drive 1414. As will be appreciated by persons skilled in the relevant art(s), the removable storage unit 1418 includes a computer readable storage medium having stored therein computer executable program code instructions and / or data.

[0151] In an alternative embodiment, the secondary memory 1410 may additionally or alternatively include other similar devices for allowing computer programs or other instructions to be loaded into the computing device 1400. Such devices can include, for example, a removable storage unit 1422 and an interface 1420. Examples of a removable storage unit 1422 and interface 1420 include a removable memory chip (such as an EPROM or PROM) and associated socket, and other removable storage units 1422 and interfaces 1420 which allow software and data to be transferred from the removable storage unit 1422 to the computer system 1400.

[0152] The computing device 1400 also includes at least one communication interface 1424. The communication interface 1424 allows software and data to be transferred between computing device 1400 and external devices via a communication path 1426. In various embodiments, the communication interface 1424 permits data to be transferred between the computing device 1400 and a data communication network, such as a public data or private data communication network. The communication interface 1424 may be used to exchange data between different computing devices 1400 which such computing devices 1400 form part of an interconnected computer network. Examples of a communication interface 1424 can include a modem, a network interface (such as an Ethernet card), a communication port, an antenna with associated circuitry and the like. The communication interface 1424 may be wired or may be wireless. Software and data transferred via the communication interface 1424 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface 1424. These signals are provided to the communication interface via the communication path 1426.

[0153] Optionally, the computing device 1400 further includes a display interface 1402 which performs operations for rendering images to an associated display 1430 and an audio interface 1432 for performing operations for playing audio content via associated speaker(s) 1434.

[0154] As used herein, the term "computer program product" may refer, in part, to removable storage unit 1418, removable storage unit 1422, a hard disk installed in hard disk drive 1412, or a carrier wave carrying software over communication path 1426 (wireless link or cable) to communication interface 1424. Computer readable storage media refers to any non- transitory tangible storage medium that provides recorded instructions and / or data to the computing device 1400 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray TM Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device 1400. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computing device 1400 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0155] The computer programs (also called computer program code) are stored in main memory 1408 and / or secondary memory 1410. Computer programs can also be received via the communication interface 1424. Such computer programs, when executed, enable the computing device 1400 to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 1404 to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system 1400.

[0156] Software may be stored in a computer program product and loaded into the computing device 1400 using the removable storage drive 1414, the hard disk drive 1412, or the interface 1420. Alternatively, the computer program product may be downloaded to the computer system 1400 over the communications path 1426. The software, when executed by the processor 1404, causes the computing device 1400 to perform functions of embodiments described herein.

[0157] It is to be understood that the embodiment of Figure 11 is presented merely by way of example. Therefore, in some embodiments one or more features of the computing device1400 may be omitted. Also, in some embodiments, one or more features of the computing device 1400 may be combined together. Additionally, in some embodiments, one or more features of the computing device 1400 may be split into one or more component parts.

[0158] According to one embodiment, a computer-readable storage medium having a computer program stored thereon is provided. When the computer program is executed in a computer, the computer is caused to execute the method of classifying a lidar point.

[0159] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the embodiments. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

[0160] The invention has been described by reference to certain embodiments discussed above. It will be recognized that these embodiments are susceptible to various modifications and alternative forms well known to those of skill in the art.

[0161] Further modifications in addition to those described above may be made to the structures and techniques described herein without departing from the spirit and scope of the invention. Accordingly, although specific embodiments have been described, these are examples only and are not limiting upon the scope of the invention.

Claims

CLAIMS1. A method of classifying a light detection and ranging, lidar, point, wherein the lidar point is associated with one or more features in an area being surveyed, the method being performed by a processor and comprising: training two or more models based on respective training datasets, each dataset being associated with respective classes of one of the two or more models; calculating, using the trained two or more models, corresponding probabilities of the lidar point being associated with the respective classes of the two or more models; and classifying the lidar point to a class of one of the two or more models by combining the calculated corresponding probabilities.

2. The method of claim 1, wherein: a first set of probabilities of the lidar point being associated with the respective classes of a first model is calculated based on a first model of the two or more models, the first model being an existing model; a second set of probabilities of the lidar point being associated with the respective classes of a second model is calculated based on a second model of the two or more models, the second model being a model trained after the lidar point is classified based on the first set of probabilities; the step of classifying comprises reclassifying the lidar point by combining the first set of probabilities and the second set of probabilities.

3. The method of claim 1 or 2, wherein the two or more models are binary or multi-class models.

4. The method of claim 2 or 3, wherein the second model is a binary model.

5. The method of any one of the preceding claims, wherein classes of the second model are subclasses of classes of the first model.

6. The method of any of the preceding claims 1-4, wherein classes of the second model are independent of classes of the first model.

7. The method of any one of the preceding claims, wherein the lidar point is classified to a class with a highest probability of the probabilities of the lidar point being associated with the respective classes.

8. The method of any of the preceding claimsl to 6, wherein the lidar point is classified to a class with a larger distance to an associated threshold value of a respective class, if two or more probabilities have a same highest value.

9. The method of any of the preceding claims 2 to 6, wherein reclassifying the lidar point comprises assigning the lidar point to a class not comprised in the first model.

10. The method of any of the preceding claims 2 to 6, wherein reclassifying the lidar point comprises assigning the lidar point to a class which is a subclass of a class comprised in the first model.

11. The method of any one of the preceding claims, further comprising constructing training datasets such that each class of a model is presented by a substantially equal number of samples from unbalanced training datasets.

12. The classification method of claim 11, the step of constructing comprises: clustering features in a feature space; sampling training dataset from each class equally from each cluster.

13. The classification method of claim 11 or 12, wherein each class is represented by multiple clusters.

14. A device for classifying a light detection and ranging, lidar, point, wherein the lidar point is associated with one or more features in an area being surveyed, the device comprising a processor for performing the method according to any of the previous claims 1 to 13.

15. A computer program product, comprising a computer readable storage medium storing instructions which, when executed on at least one processor, cause the at least one processor to carry out the method according to any of the claims 1 to 13.

Images