Lidar malfunction detection method and apparatus, and data generation method and apparatus therefor

Abstract

To provide a LiDAR malfunction detection method and apparatus, and a data generation method and apparatus therefor.SOLUTION: A LiDAR malfunction detection method includes: acquiring point cloud data from a LiDAR; determining whether or not the number of points contained in the point cloud data is smaller than a point-number threshold; and, when the number of points is greater than or equal to the point-number threshold, determining whether or not the LiDAR has a defect by using a LiDAR malfunction detection model based on the point cloud data. The LiDAR malfunction detection model is a pre-trained model based on a normal dataset and at least one LiDAR abnormal dataset, and the normal dataset includes preset statistical values and point cloud data acquired from the LiDAR in a normal state.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates to a lidar anomaly detection method and apparatus, and a data generation method and apparatus therefor. [Background technology]

[0002] The following description merely provides background information related to the present embodiment and may not constitute prior art.

[0003] Conventionally, methods used to detect malfunctions in LiDAR (Light Detection and Ranging) include analyzing received laser signals to determine the LiDAR's operating status, and calculating the distance between multiple sensors installed in fixed positions on a vehicle and a single target object, and then using the error in the distance to determine whether the LiDAR is malfunctioning.

[0004] The above existing methods detect LIDAR abnormalities using test patterns or targets in specific environments, and therefore have the limitation that they can only detect LIDAR abnormalities in specific environments.

[0005] The above-described existing methods detect a rider abnormality based on the rider's movement, and therefore, are limited in that they are based on the assumption that the rider is in operation. Therefore, when the rider is not in operation, the existing methods cannot detect a rider abnormality. Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure primarily aims to provide a method and apparatus for detecting LIDAR anomalies using artificial intelligence technology, and a data generation method and apparatus therefor. Specifically, the present disclosure primarily aims to provide a method and apparatus for detecting LIDAR anomalies using artificial intelligence technology, and a data generation method and apparatus therefor, which generate LIDAR-abnormal datasets for each type based on normal datasets, train a LIDAR anomaly detection model based on the normal datasets and the LIDAR-abnormal datasets, and determine whether a LIDAR defect exists using the trained LIDAR anomaly detection model, thereby enabling the detection of LIDAR anomalies in various environments.

[0007] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0008] According to one aspect of the present disclosure, there is provided a method for detecting a lidar anomaly, the method including: acquiring point cloud data from the lidar; determining whether a number of points included in the point cloud data is less than a point count threshold; and, if the number of points is greater than or equal to the point count threshold, determining whether the lidar has a defect based on the point cloud data using a lidar anomaly detection model, wherein the lidar anomaly detection model is a model pre-trained based on a normal dataset and at least one or more lidar-abnormal datasets, the normal dataset including preset statistics and point cloud data acquired from the lidar in a normal state, and the lidar-abnormal dataset including point cloud data generated based on the normal dataset.

[0009] According to one aspect of the present disclosure, there is provided a method for generating data used to train a lidar anomaly detection model, the method including the steps of acquiring a normal dataset and generating at least one or more lidar-abnormal datasets based on the normal dataset.

[0010] According to one aspect of the present disclosure, there is provided a method for detecting a LIDAR abnormality, including: acquiring point cloud data from the LIDAR; determining whether a number of points included in the point cloud data is less than a point number threshold; and, when the number of points is greater than or equal to the point number threshold, not increasing a timer error accumulation number if a time taken to receive the point cloud data is less than the point cloud data reception time threshold, and increasing a timer error accumulation number by 1 if the time taken to receive the point cloud data is greater than or equal to the point cloud data reception time threshold; determining whether the timer error accumulation number is greater than a timer error accumulation number threshold; and, when the timer error accumulation number is greater than the timer error accumulation number threshold, determining that there is a defect in the step of acquiring point cloud data from the LIDAR, rather than in the LIDAR. [Effects of the Invention]

[0011] According to one embodiment of the present disclosure, LIDAR abnormality data sets are generated by classifying them into multiple types, thereby making it possible to detect LIDAR abnormalities in various environments.

[0012] According to one embodiment of the present disclosure, LIDAR-abnormal data sets are generated by classifying them into various types, and a LIDAR anomaly detection model is trained based on the LIDAR-abnormal data sets thus created, thereby improving the accuracy of LIDAR anomaly detection using the LIDAR anomaly detection model.

[0013] According to one embodiment of the present disclosure, even if actual data does not exist, a LIDAR-abnormal dataset generated based on a normal dataset can be used for model training, thereby making it possible to detect LIDAR abnormalities even in situations where the LIDAR is not operating.

[0014] According to one embodiment of the present disclosure, even if actual data does not exist, costs can be reduced by using a lidar-abnormal dataset generated based on a normal dataset for model training.

[0015] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0016] [Figure 1a] FIG. 1a is a flowchart illustrating a LiDAR anomaly detection method according to one embodiment of the present disclosure. [Figure 1b] FIG. 1b is a flowchart illustrating a LiDAR anomaly detection method according to one embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a class structure of training data according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is an exemplary diagram illustrating normal data according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a flowchart illustrating a method for generating first type LIDAR-abnormal data of the learning data according to one embodiment of the present disclosure. [Figure 5a] FIG. 5a is an exemplary diagram showing normal data according to one embodiment of the present disclosure. [Figure 5b] FIG. 5b is an exemplary diagram illustrating the first type of LIDAR-abnormal data according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a flowchart illustrating a method for generating second type LIDAR-abnormal data from the learning data according to an embodiment of the present disclosure. [Figure 7a]FIG. 7a is an example diagram showing normal data according to one embodiment of the present disclosure. [Figure 7b] FIG. 7b is an exemplary diagram illustrating a second type of LIDAR-abnormal data according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart illustrating a method for generating third type LIDAR-abnormal data among the learning data according to an embodiment of the present disclosure. [Figure 9a] FIG. 9a is an example diagram showing normal data according to one embodiment of the present disclosure. [Figure 9b] FIG. 9b is an exemplary diagram illustrating a third type of LIDAR-abnormal data according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a flowchart illustrating a preprocessing method according to an embodiment of the present disclosure. [Figure 11a] FIG. 11a is an exemplary diagram illustrating a preprocessing result for normal data according to an embodiment of the present disclosure. [Figure 11b] FIG. 11b is an exemplary diagram illustrating a preprocessing result for the first type of LIDAR-abnormal data according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart that schematically illustrates a lidar anomaly detection algorithm according to one embodiment of the present disclosure. [Figure 13] FIG. 13 is a flowchart illustrating a LIDAR anomaly detection algorithm according to one embodiment of the present disclosure. [Figure 14] FIG. 14 is a flowchart illustrating a LIDAR anomaly detection algorithm according to one embodiment of the present disclosure. [Figure 15] FIG. 15 is a block diagram that schematically illustrates an example computing device that may be used to implement the apparatus and methods described in this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] Some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. When assigning reference numerals to components in each drawing, please note that the same reference numerals are used to designate the same components even if they appear in different drawings. In describing the present disclosure, if a detailed description of related known structures or functions is deemed to obscure the gist of the present disclosure, such a detailed description will be omitted.

[0018] In describing components of the embodiments of the present disclosure, reference numerals such as 1, 2, i), ii), a), b), etc. may be used. Such reference numerals are used only to distinguish the components from other components, and do not limit the essence or order or sequence of the components. When a part in this specification "includes" or "has" a certain component, this does not mean that other components are excluded, but that other components may also be included, unless explicitly stated to the contrary.

[0019] The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0020] In this disclosure, the terms fault, breakdown, and malfunction may be used with similar meanings. Therefore, a method and apparatus for detecting a LIDAR abnormality may be expressed as a method and apparatus for detecting a LIDAR malfunction or a method and apparatus for detecting a LIDAR defect. In this disclosure, the terms detect and diagnose may be used with similar meanings. Therefore, a method and apparatus for detecting a LIDAR abnormality may be expressed as a method and apparatus for diagnosing a LIDAR abnormality. Furthermore, a method and apparatus for detecting a LIDAR abnormality may be expressed as a method and apparatus for diagnosing a LIDAR malfunction, etc.

[0021] FIG. 1a is a flowchart illustrating a LiDAR anomaly detection method according to one embodiment of the present disclosure.

[0022] FIG. 1b is a flowchart illustrating a LiDAR anomaly detection method according to one embodiment of the present disclosure.

[0023] 1a, a LIDAR anomaly detection device according to an embodiment of the present disclosure collects point cloud data (S110), generates training data based on the collected data (S130), preprocesses the training data (S150), and trains a LIDAR anomaly detection model based on the preprocessed training data (S170). The step of preprocessing the training data (S150) may be omitted. When the step of preprocessing the training data is omitted, the step of training a LIDAR anomaly detection model based on the preprocessed training data may be the step of training a LIDAR anomaly detection model based on the generated training data.

[0024] 1b, a LIDAR anomaly detection device according to an embodiment of the present disclosure collects point cloud data (S120), preprocesses the data (S140), and uses the trained LIDAR anomaly detection model to infer and determine whether there is a defect in the LIDAR (S160). The step of preprocessing the data (S140) may be omitted.

[0025] The point cloud data may be acquired using a lidar and may include normal data, which is described in more detail below with reference to FIG.

[0026] FIG. 2 is a diagram illustrating a class structure of training data according to an embodiment of the present disclosure.

[0027] The training data 200 may include a normal dataset 210 and an abnormal dataset. The abnormal dataset may be classified into at least one type of dataset based on the type of LIDAR fault that actually occurs. LIDAR fault types may be classified into three types. The abnormal dataset may include a first type LIDAR-abnormal dataset, a second type LIDAR-abnormal dataset, and a third type LIDAR-abnormal dataset. The first type LIDAR-abnormal dataset may be a channel-fault type LIDAR-abnormal dataset 220. The second type LIDAR-abnormal dataset may be an occlusion type LIDAR-abnormal dataset 230. The third type LIDAR-abnormal dataset may be a partial occlusion type LIDAR-abnormal dataset 240. That is, the abnormal dataset may include the channel-fault type LIDAR-abnormal dataset 220, the occlusion type LIDAR-abnormal dataset 230, and the partial occlusion type LIDAR-abnormal dataset 240.

[0028] The channel failure type LIDAR-abnormal data set 220 may be data obtained by combining channel failure type LIDAR-abnormal data. The channel failure type LIDAR-abnormal data may be data indicating that all or part of a LIDAR channel is lost. The channel failure type LIDAR-abnormal data may be synthetic point cloud data that simulates point cloud data collected using a LIDAR in a situation where part of the LIDAR channel is lost, i.e., actual point cloud data. Therefore, the channel failure type LIDAR-abnormal data set 220 may be data obtained by combining point cloud data collected using a LIDAR in a situation where part of the LIDAR channel is lost, i.e., synthetic point cloud data that simulates actual point cloud data. At least one or more point cloud data may be data collected at different times.

[0029] The occlusion-type LIDAR-abnormal data set 230 may be data obtained by combining occlusion-type data. The occlusion-type LIDAR-abnormal data may be data indicating a case where point cloud data is not reflected and lost over the entire yaw angle range of 0 to 360° based on the z-axis of the LIDAR. The occlusion-type LIDAR-abnormal data may be point cloud data collected using a LIDAR in a situation where point cloud data is not reflected and lost over the entire yaw angle range of 0 to 360° based on the z-axis of the LIDAR, i.e., synthesized point cloud data that replicates actual point cloud data. Therefore, the occlusion-type LIDAR-abnormal data set 230 may be data obtained by combining point cloud data collected using a LIDAR in a situation where point cloud data is not reflected and lost over the entire yaw angle range of 0 to 360° based on the z-axis of the LIDAR, i.e., synthesized point cloud data that replicates actual point cloud data. At least one or more point cloud data may be data collected at different times.

[0030] The partial occlusion type LIDAR-abnormal data set 240 may be data obtained by combining partial occlusion type data. The partial occlusion type LIDAR-abnormal data may be data indicating a case where point cloud data is not reflected and lost at a portion of a yaw angle of 0 to 360° based on the z-axis of the LIDAR. The partial occlusion type LIDAR-abnormal data may be point cloud data collected using a LIDAR in a situation where point cloud data is not reflected and lost at a portion of a yaw angle of 0 to 360° based on the z-axis of the LIDAR, i.e., synthesized point cloud data that replicates actual point cloud data. Therefore, the partial occlusion type LIDAR-abnormal data set 240 may be data obtained by combining point cloud data collected using a LIDAR in a situation where point cloud data is not reflected and lost at a portion of a yaw angle of 0 to 360° based on the z-axis of the LIDAR, i.e., synthesized point cloud data that replicates actual point cloud data. At least one or more point cloud data may be data collected at different times.

[0031] FIG. 3 is an exemplary diagram illustrating normal data according to an embodiment of the present disclosure.

[0032] The normal data may be point cloud data collected using a lidar in a normal state, for example, a lidar without a malfunction or a lidar without a fault. The normal data set may be data obtained by combining at least one or more point cloud data collected using a lidar in a normal state, for example, a lidar without a malfunction or a lidar without a fault. The at least one or more point cloud data may be data collected at different times.

[0033] FIG. 4 is a flowchart illustrating a method for generating first type LIDAR-abnormal data of the learning data according to one embodiment of the present disclosure.

[0034] The first type of LIDAR-abnormal data may be channel failure type LIDAR-abnormal data.

[0035] A LIDAR anomaly detection device according to an embodiment of the present disclosure retrieves point cloud data (S410). The point cloud data may be normal data. The normal data may be one of at least one point cloud data included in the normal data set.

[0036] A LIDAR anomaly detection device according to an embodiment of the present disclosure classifies points into a first group and a second group based on a preset rule (S420). The points may be points included in the point cloud data retrieved (fetched) in step S410. The first group and the second group may be spatially separated.

[0037] The preset rule may be a rule that, assuming a situation in which the lidar is installed on top of a vehicle, classifies points within the z-axis range from "ground level - (lidar height - vehicle height)" to "ground level - (lidar height + vehicle height)" into a first group, and points within other z-axis ranges into a second group. The preset rule may vary depending on various factors, such as which lidar channel collected the point cloud data. The preset rule may be determined experimentally under specific conditions.

[0038] The LIDAR anomaly detection device according to an embodiment of the present disclosure applies different clustering techniques to the first group and the second group (S430).

[0039] The clustering technique applied to the first group may be a K-means clustering technique, and the clustering technique applied to the second group may be a DBSCAN clustering technique.

[0040] For the first group, for example, points within the z-axis range from "ground level - (z value of the LIDAR installation point - total vehicle height)" to "ground level - (z value of the LIDAR installation point + total vehicle height)," points are often distributed at different heights depending on the LIDAR channel, so applying the K-means clustering technique, a distance-based clustering technique that clusters adjacent points, may be effective.

[0041] For the second group, for example, points that are not included in the z-axis range from "ground level ~ (z value of the LIDAR installation point - total vehicle height)" to "ground level ~ (z value of the LIDAR installation point + total vehicle height)," it may be effective to apply the DBSCAN clustering technique, a density-based clustering technique. i) For points with z-values ​​lower than "ground level ~ (z value of the LIDAR installation point - total vehicle height)," the z-axis intervals for each LIDAR channel are not distributed at different heights and all have z-values ​​of 0. ii) For points with z-values ​​higher than "ground level ~ (z value of the LIDAR installation point + total vehicle height)," the z-axis intervals for each LIDAR channel become sparse. This may make it effective to apply the DBSCAN clustering technique, a density-based clustering technique.

[0042] The lidar may be installed at the highest point on the roof of the vehicle. In this case, the z value of the point where the lidar is installed may be the same as the overall height of the vehicle. In this case, "ground ~ (z value of the point where the lidar is installed - overall height of the vehicle)" may be z = 0. "ground ~ (z value of the point where the lidar is installed + overall height of the vehicle)" may be z = (overall height of the vehicle) * 2. The overall height of the vehicle may be used interchangeably with the height of the vehicle.

[0043] The first and second clusters may be generated by applying different clustering techniques to the first and second groups. The first cluster may be the result of applying a K-means clustering technique to the first group. The second cluster may be the result of applying a DBSCAN clustering technique to the second group.

[0044] The LIDAR anomaly detection device according to an embodiment of the present disclosure generates an entire cluster set (S440). The entire cluster set may be data obtained by merging the first cluster and the second cluster.

[0045] A LIDAR anomaly detection device according to an embodiment of the present disclosure randomly removes clusters based on a preset ratio (S450). The clusters may be clusters included in the overall cluster set generated as a result of S440. Generally, when a LIDAR is defective, the number of points included in point cloud data measured by the LIDAR is reduced. Therefore, the method according to the present disclosure can replicate point cloud data that may be measured when a LIDAR is defective by randomly removing clusters, i.e., a large number of points.

[0046] The preset ratio may be a ratio set based on the number of points of the data with the smallest number of points among all the data, i.e., all the normal data.

[0047] A LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a first type of LIDAR abnormal data by randomly removing clusters.

[0048] The LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether steps S410 through S450 have been applied to all data (S460). The data may be normal data. The normal data may be point cloud data included in the normal data set. That is, the LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether steps S410 through S450 have been applied to all normal point cloud data included in the normal data set.

[0049] If steps S410 through S450 have not been applied to even one data item, the method according to an embodiment of the present disclosure returns to step S410. The LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a first type LIDAR-abnormal data set by applying steps S410 through S450 to all data items.

[0050] FIG. 5a is an exemplary diagram showing normal data according to one embodiment of the present disclosure.

[0051] FIG. 5b is an exemplary diagram illustrating the first type of LIDAR-abnormal data according to one embodiment of the present disclosure.

[0052] A LIDAR anomaly detection device according to an embodiment of the present disclosure can generate first-type LIDAR-abnormal data as illustrated in Fig. 5b based on normal data as illustrated in Fig. 5a by performing steps S410 to S460. The first-type LIDAR-abnormal data as illustrated in Fig. 5b can be synthetic data that replicates actual data.

[0053] FIG. 6 is a flowchart illustrating a method for generating second type LIDAR-abnormal data from the learning data according to an embodiment of the present disclosure.

[0054] The second type of LIDAR-abnormal data may be occlusion type LIDAR-abnormal data.

[0055] A LIDAR anomaly detection device according to an embodiment of the present disclosure retrieves point cloud data (S610). The point cloud data may be normal data. The normal data may be one of at least one point cloud data included in the normal data set.

[0056] A LIDAR anomaly detection device according to an embodiment of the present disclosure randomly removes points based on a preset ratio (S620). The points may be points included in the point cloud data retrieved in step S610. Generally, if there is a LIDAR defect, the number of points included in the point cloud data measured by the LIDAR decreases. Therefore, the method according to the present disclosure can replicate point cloud data that may be measured if there is a LIDAR defect by randomly removing points.

[0057] The preset ratio may be a ratio set based on the number of points of the data with the smallest number of points among all the data, i.e., all the normal data.

[0058] A LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a second type of LIDAR abnormal data by randomly removing points.

[0059] The LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether steps S610 and S620 have been applied to all data (S630). The data may be normal data. The normal data may be point cloud data included in the normal dataset. That is, the LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether steps S610 and S620 have been applied to all normal point cloud data included in the normal dataset.

[0060] If steps S610 and S620 have not been applied to even one data item, the method according to an embodiment of the present disclosure returns to step S610. The LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a second type LIDAR-abnormal data set by applying steps S610 and S620 to all data items.

[0061] FIG. 7a is an example diagram showing normal data according to one embodiment of the present disclosure.

[0062] FIG. 7b is an exemplary diagram illustrating a second type of LIDAR-abnormal data according to an embodiment of the present disclosure.

[0063] A LIDAR anomaly detection device according to an embodiment of the present disclosure can generate second-type LIDAR-abnormal data as illustrated in Fig. 7b based on normal data as illustrated in Fig. 7a by performing steps S610 to S630. The second-type LIDAR-abnormal data as illustrated in Fig. 7b can be synthetic data that replicates actual data.

[0064] FIG. 8 is a flowchart illustrating a method for generating third type LIDAR abnormal data according to an embodiment of the present disclosure.

[0065] The third type of LIDAR-abnormal data may be partial occlusion type LIDAR-abnormal data.

[0066] A LIDAR anomaly detection device according to an embodiment of the present disclosure retrieves point cloud data (S810). The point cloud data may be normal data. The normal data may be one of at least one point cloud data included in the normal data set.

[0067] The LIDAR anomaly detection device according to one embodiment of the present disclosure converts the coordinate system of the point cloud data from a Cartesian coordinate system to a cylindrical coordinate system (S820).

[0068] A LIDAR anomaly detection device according to an embodiment of the present disclosure randomly removes points based on a preset rule and a preset ratio (S830). Generally, if there is a LIDAR defect, the number of points included in the point cloud data measured by the LIDAR decreases. Therefore, the method according to the present disclosure can replicate point cloud data that may be measured if there is a LIDAR defect by randomly removing points.

[0069] The preset ratio may be a ratio set based on the number of points of the data with the smallest number of points among all the data, i.e., all the normal data.

[0070] The preset rule may be a rule for classifying points into various subgroups. For example, the preset rule may be a rule for classifying points into various subgroups by setting an angle range and a z-axis range. Specifically, the angle range may be one of 12 sections obtained by dividing a yaw angle of 0 to 360° based on the z-axis of the LIDAR, with each section being 30°. The z-axis range may be one of 10 sections obtained by dividing the z-axis into 0.1 units, assuming that the z-axis length in the point cloud data is 0 to 1. Based on these angle ranges and z-axis ranges, points may be classified into a total of 120 subgroups. A LIDAR anomaly detection device according to an embodiment of the present disclosure removes points from the first subgroup to the 120th subgroup based on a preset ratio.

[0071] A LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a third type of LIDAR abnormal data by removing a point and then merging all subgroups containing one data, for example, points included in the first subgroup through the 120th subgroup.

[0072] The LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether steps S810 through S830 have been applied to all data (S840). The data may be normal data. The normal data may be point cloud data included in the normal data set. That is, the LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether steps S810 through S830 have been applied to all normal point cloud data included in the normal data set.

[0073] If steps S810 to S830 have not been applied to even one data item, the method according to an embodiment of the present disclosure returns to step S810. The LIDAR anomaly detection device according to an embodiment of the present disclosure can generate a third type of LIDAR-abnormal data set by applying steps S810 to S830 to all data items.

[0074] FIG. 9a is an example diagram showing normal data according to one embodiment of the present disclosure.

[0075] FIG. 9b is an exemplary diagram illustrating a third type of LIDAR-abnormal data according to an embodiment of the present disclosure.

[0076] A LIDAR anomaly detection device according to an embodiment of the present disclosure may perform steps S810 to S840 to generate first-type LIDAR-abnormal data as illustrated in Fig. 9b based on normal data as illustrated in Fig. 9a. The first-type LIDAR-abnormal data as illustrated in Fig. 9b may be synthetic data that replicates actual data.

[0077] FIG. 10 is a flowchart illustrating a preprocessing method according to an embodiment of the present disclosure.

[0078] The LIDAR anomaly detection device according to an embodiment of the present disclosure retrieves point cloud data (S1010). The LIDAR anomaly detection device according to an embodiment of the present disclosure may acquire the point cloud data and the number of selected points. The point cloud data may include at least one point.

[0079] A LIDAR anomaly detection device according to an embodiment of the present disclosure calculates the distance from the origin to each point (S1020). The LIDAR anomaly detection device according to an embodiment of the present disclosure may calculate the distance from the origin in a LIDAR coordinate system to each point. The origin in the LIDAR coordinate system may refer to the location where the LIDAR is installed. The LIDAR anomaly detection device according to an embodiment of the present disclosure may store the calculated results in an array.

[0080] The LIDAR anomaly detection device according to an embodiment of the present disclosure may sort the calculation results (S1030). The LIDAR anomaly detection device according to an embodiment of the present disclosure may sort the values ​​stored in the array in ascending order.

[0081] A LIDAR anomaly detection device according to an embodiment of the present disclosure may extract a certain number of points (S1040). The LIDAR anomaly detection device according to an embodiment of the present disclosure may select a specific number of values ​​from the arranged values ​​and store them in another array. The specific number may be the number of selected points. The LIDAR anomaly detection device according to an embodiment of the present disclosure may select points corresponding to values ​​included in the results stored in the other array. The LIDAR anomaly detection device according to an embodiment of the present disclosure may reconstruct point cloud data including the selected points.

[0082] A LIDAR anomaly detection device according to one embodiment of the present disclosure can extract and sample only a portion of points close to the LIDAR installation position from the entire points of the point cloud data by performing steps S1010 to S1040.

[0083] FIG. 11a is an exemplary diagram illustrating a preprocessing result for normal data according to an embodiment of the present disclosure.

[0084] A LIDAR anomaly detection device according to an embodiment of the present disclosure may perform steps S1010 to S1040 to generate a pre-processing result for normal data as illustrated in Fig. 11a. The pre-processing result may be point cloud data.

[0085] FIG. 11b is an exemplary diagram illustrating a preprocessing result for the first type of LIDAR-abnormal data according to an embodiment of the present disclosure.

[0086] A LIDAR anomaly detection device according to an embodiment of the present disclosure may perform steps S1010 to S1040 to generate a pre-processing result for the first type of LIDAR-abnormal data as illustrated in Fig. 11b. The first type of LIDAR-abnormal data may be generated based on the normal data by the steps of Fig. 4. The pre-processing result may be point cloud data.

[0087] 1a and 1b, the preprocessing (S140, S150) steps may be performed before the model training (S170) step or before the model inference (S160) step. Specifically, the preprocessing (S150) step may be performed before the model training (S170) step, and the preprocessing (S140) step may be performed before the model inference (S160) step. The preprocessing (S140) step and the preprocessing (S150) step may be essentially the same step, but may have different preprocessing targets and purposes.

[0088] FIG. 12 is a flowchart that schematically illustrates a lidar anomaly detection algorithm according to one embodiment of the present disclosure.

[0089] The LIDAR anomaly detection device according to an embodiment of the present disclosure retrieves point cloud data (S1210).

[0090] A LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether the number of points in the retrieved point cloud data is less than a point number threshold (S1220). Generally, if there is a LIDAR defect, the number of points included in the point cloud data measured by the LIDAR decreases. Therefore, if the number of points in the point cloud data is less than a preset threshold, it is recommended to suspect a LIDAR anomaly.

[0091] If the number of points is greater than or equal to the point number threshold, the LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether there is a defect in the step of acquiring point cloud data (S1230). An algorithm for determining whether there is a defect in the step of acquiring point cloud data will be described in detail below with reference to FIG. 13.

[0092] If it is determined that there is no defect in the step of acquiring point cloud data, the LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether there is a defect in the LIDAR (S1240). The LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether there is a defect in the LIDAR using a pre-trained LIDAR anomaly detection model. The step of determining whether there is a defect in the LIDAR using the LIDAR anomaly detection model (S1240) may be the same step as the model inference step (S160) of FIG. 1b.

[0093] The LIDAR anomaly detection model according to an embodiment of the present disclosure may be an artificial intelligence model trained using learning data including a normal data set and a type-specific LIDAR-abnormal data set. The artificial intelligence model may be implemented as a 3D-GCN (Graph Convolution Network). The LIDAR anomaly detection device according to an embodiment of the present disclosure detects LIDAR anomalies based on the density of point cloud data. Therefore, the relationship between points included in the point cloud data is important for the LIDAR anomaly detection device according to an embodiment of the present disclosure to detect LIDAR anomalies. The 3D-GCN uses a GCN to learn by taking into account the distance between adjacent points, and therefore can learn the relationship between points. Therefore, the 3D-GCN is a neural network suitable for implementing the LIDAR anomaly detection model according to an embodiment of the present disclosure.

[0094] The LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether a LIDAR defect exists using a pre-trained LIDAR anomaly detection model and may additionally verify the results inferred by the LIDAR anomaly detection model. The process by which the LIDAR anomaly detection device according to an embodiment of the present disclosure verifies the results inferred by the LIDAR anomaly detection model will be described in detail below with reference to FIG. 14.

[0095] If the LIDAR abnormality detection device according to an embodiment of the present disclosure determines that the LIDAR has a defect, it outputs the result (S1250). The result may be output via an output interface. If the LIDAR abnormality detection device according to an embodiment of the present disclosure is a device for detecting an abnormality in a LIDAR used in an autonomous vehicle, the LIDAR abnormality detection device may output the result by transmitting a signal to another device for control of the autonomous vehicle.

[0096] A rider abnormality detection device according to one embodiment of the present disclosure can detect a rider abnormality by performing steps S1210 to S1240, and if a rider abnormality is detected, can output a result by performing step S1250.

[0097] FIG. 13 is a flowchart illustrating a LIDAR anomaly detection algorithm according to one embodiment of the present disclosure.

[0098] Referring to FIG. 13, a LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether the time (Duration) required to receive the point cloud data is less than a point cloud data reception time threshold (D_threshold) if the number of points in the called point cloud data is greater than or equal to the point number threshold (S1310).

[0099] A LIDAR abnormality detection device according to an embodiment of the present disclosure sets a flag (Timer Fault Flag) for determining timer error accumulation to true (S1320) if the time required to receive point cloud data is greater than or equal to the point cloud data reception time threshold.

[0100] A lidar abnormality detection device according to an embodiment of the present disclosure sets a flag (Timer Fault Flag) for determining timer error accumulation to false (S1330) if the time required to receive point cloud data is less than the point cloud data reception time threshold.

[0101] The LIDAR abnormality sensing device according to an embodiment of the present disclosure determines whether a timer fault flag for determining timer error accumulation is true (S1340).

[0102] A rider abnormality detection device according to an embodiment of the present disclosure increases a timer error accumulation count (Timer Counter) by 1 if a timer fault flag for determining timer error accumulation is determined to be true (S1350).

[0103] A rider abnormality detection device according to one embodiment of the present disclosure does not increment the timer error accumulation count (Timer Counter) if the flag for determining the timer error accumulation (Timer Fault Flag) is determined to be false, i.e., not true (S1360).

[0104] The LIDAR abnormality sensing device according to an embodiment of the present disclosure determines whether the timer error accumulation number (Timer Counter) is greater than the timer error accumulation number threshold (T_threshold) (S1370).

[0105] A LIDAR abnormality detection device according to an embodiment of the present disclosure determines a timer fault when a timer error accumulation count (Timer Counter) is greater than a timer error accumulation count threshold (T_threshold). In other words, it determines that there is a fault in the step of acquiring a point cloud from a LIDAR, not in the LIDAR. Examples of faults in the step of acquiring a point cloud from a LIDAR include when the power switch of the LIDAR abnormality detection device is off, when an electrical wire connected to the LIDAR abnormality detection device is disconnected, when a program executed by the LIDAR abnormality detection device is terminated unexpectedly, and when an association device included in the LIDAR abnormality detection device is overloaded.

[0106] A LIDAR abnormality detection device according to an embodiment of the present disclosure determines that there is no defect in the process of acquiring the point cloud if the timer error accumulation number (Timer Counter) is less than or equal to the timer error accumulation number threshold (T_threshold), and performs step S1240.

[0107] FIG. 14 is a flowchart illustrating a LIDAR anomaly detection algorithm according to one embodiment of the present disclosure.

[0108] 14, a LIDAR anomaly detection device according to an embodiment of the present disclosure may determine whether a LIDAR defect exists using a LIDAR anomaly detection model in step S1410, and then verify the results inferred by the LIDAR anomaly detection model in steps S1420 through S1480. That is, the LIDAR anomaly detection device according to an embodiment of the present disclosure may additionally perform steps S1420 through S1480 to verify the results inferred by the LIDAR anomaly detection model.

[0109] A LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether a LIDAR has a defect using a pre-trained LIDAR anomaly detection model (S1410). The step of determining whether a LIDAR has a defect using the LIDAR anomaly detection model (S1410) may be the same step as the step of determining whether a LIDAR has a defect using the LIDAR anomaly detection model of FIG. 12 (S1240) and the model inference step of FIG. 1b (S160).

[0110] When it is determined that there is a defect in the LIDAR using the LIDAR anomaly detection model, the LIDAR anomaly detection device according to an embodiment of the present disclosure determines whether the LIDAR anomaly detection model confidence (LIDAR Fault Confidence) is less than a LIDAR anomaly detection model confidence threshold (C_threshold) (S1420).

[0111] Confidence is an indicator of the accuracy of the results inferred by an artificial intelligence model. Confidence may be a confidence score. Confidence may be a value between 0 and 1. Confidence may be generated by an artificial intelligence model.

[0112] A LIDAR abnormality detection device according to an embodiment of the present disclosure sets a flag for determining LIDAR error accumulation (LIDAR Fault Flag) to true if the LIDAR abnormality detection model confidence is greater than or equal to the LIDAR abnormality detection model confidence threshold (S1430).

[0113] A LIDAR abnormality detection device according to an embodiment of the present disclosure sets a flag for determining LIDAR error accumulation (LIDAR Fault Flag) to false (S1440) when the LIDAR abnormality detection model confidence is smaller than the LIDAR abnormality detection model confidence threshold.

[0114] The LIDAR abnormality sensing device according to an embodiment of the present disclosure determines whether a flag for determining LIDAR error accumulation (LIDAR Fault Flag) is true (S1450).

[0115] If it is determined that a flag for determining LIDAR error accumulation (LIDAR Fault Flag) is true, the LIDAR abnormality detection device according to an embodiment of the present disclosure increases the LIDAR error accumulation count (LIDAR Counter) by 1 (S1460).

[0116] A LIDAR abnormality detection device according to an embodiment of the present disclosure does not increase the LIDAR error accumulation count (LIDAR Counter) if the flag for determining LIDAR error accumulation (LIDAR Fault Flag) is determined to be false, i.e., not true (S1470).

[0117] The LIDAR abnormality detection device according to an embodiment of the present disclosure determines whether the LIDAR error accumulation number (LIDAR Counter) is greater than the LIDAR error accumulation number threshold (L_threshold) (S1480).

[0118] If the LIDAR error accumulation count (LIDAR Counter) is greater than the LIDAR error accumulation count threshold (L_threshold), the LIDAR anomaly detection device according to an embodiment of the present disclosure determines that a LIDAR error has occurred (LIDAR Fault) and performs step S1250. That is, it ultimately determines that there is a LIDAR defect, i.e., it determines that the result inferred by the LIDAR anomaly detection model is correct.

[0119] FIG. 15 is a block diagram that schematically illustrates an example computing device that may be used to implement the apparatus and methods described in this disclosure.

[0120] Computing device 150 may include some or all of memory 1500, processor 1520, storage 1540, input / output interface 1560, and communication interface 1580. Computing device 150 may be a stationary computing device such as a desktop computer, a server, or the like, as well as a mobile computing device such as a laptop computer, a smartphone, or the like. Computing device 150 may include any specialized hardware accelerator capable of processing calculations for artificial intelligence models in an efficient manner. For example, computing device 150 may include a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0121] Memory 1500 may store a program that causes processor 1520 to perform methods or operations according to various embodiments of the present disclosure. For example, the program may include multiple instructions executable by processor 1520, and the execution of the multiple instructions by processor 1520 may perform the above-described methods or operations. Memory 1500 may be a single memory or multiple memories. In this case, information required to perform the methods or operations according to various embodiments of the present disclosure may be stored in a single memory or may be divided among multiple memories. When memory 1500 is configured with multiple memories, the multiple memories may be physically separated. Memory 1500 may include at least one of volatile memory and non-volatile memory. Volatile memory may include static random access memory (SRAM) or dynamic random access memory (DRAM), and non-volatile memory may include flash memory.

[0122] Processor 1520 may include at least one core capable of executing at least one instruction. Processor 1520 may execute instructions stored in memory 1500. Processor 1520 may be a single processor or multiple processors.

[0123] Storage 1540 maintains stored data even when power supplied to computing device 150 is interrupted. For example, storage 1540 may include a non-volatile memory or a storage medium such as a magnetic tape, an optical disk, or a magnetic disk. A program stored in storage 1540 may be loaded into memory 1500 before being executed by processor 1520. Storage 1540 stores files written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory 1500. Storage 1540 may store data to be processed by processor 1520 and / or data processed by processor 1520.

[0124] The input / output interface 1560 may provide an interface with input devices such as a keyboard, a mouse, etc. and / or output devices such as a display device, a printer, etc. A user can trigger the execution of a program by the processor 1520 via the input device and / or check the processing results of the processor 1520 via the output device.

[0125] The communication interface 1580 may provide access to an external network. The computing device 150 may communicate with other devices via the communication interface 1580.

[0126] Each component of the device or method according to the present invention may be implemented in hardware or software, or a combination of hardware and software. Furthermore, the function of each component may be implemented in software, and a microprocessor may execute the software function corresponding to each component.

[0127] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include those embodied in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special-purpose processor or a general-purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. The computer program (also known as a program, software, software application, or code) includes instructions for the programmable processor and is stored on a "computer-readable storage medium."

[0128] The computer-readable recording medium includes any type of recording device that stores data readable by a computer system. Such computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, or storage device, or may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across computer systems connected to a network, so that the computer-readable code is stored and executed in a distributed manner.

[0129] Although the flowcharts / timing diagrams in this specification describe steps being performed sequentially, this is merely an illustrative example of the technical concept of one embodiment of the present disclosure. In other words, a person skilled in the art to which an embodiment of the present disclosure pertains would be able to make various modifications and variations by changing the order of steps described in the flowcharts / timing diagrams or performing one or more steps in parallel, without departing from the essential characteristics of one embodiment of the present disclosure. The flowcharts / timing diagrams are not limited to a chronological order.

[0130] The above description merely exemplifies the technical concept of the present embodiment, and various modifications and variations may be made by a person skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiment is intended to illustrate, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by such an embodiment. The scope of protection of the present embodiment should be interpreted by the claims, and all technical concepts within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

Claims

1. 1. A method for detecting abnormalities in a lidar, comprising: acquiring point cloud data from the lidar; determining whether the number of points included in the point cloud data is less than a point number threshold; and If the number of points is greater than or equal to the point number threshold, determining whether the LIDAR has a defect based on the point cloud data using a LIDAR anomaly detection model; The LIDAR anomaly detection model is a pre-trained model based on a normal dataset and at least one or more LIDAR-abnormal datasets; The normal data set includes point cloud data acquired from the lidar under a predetermined statistical value and a normal condition; The method, wherein the LIDAR-non-normal dataset includes point cloud data generated based on the normal dataset.

2. 1. A method for generating data used to train a lidar anomaly detection model, comprising: acquiring a normal data set; and generating at least one or more LIDAR-abnormal data sets based on the normal data set; The at least one or more LIDAR-abnormal data sets include a first type LIDAR-abnormal data set; The first type LIDAR-abnormal data set includes: Obtaining point cloud data for each data set from the normal data set; Classifying points included in the point cloud data into a first group and a second group based on a predetermined rule, and the first group and the second group may be spatially separated; applying a first clustering technique to the first group to generate at least one or more first clusters and a second clustering technique to the second group to generate at least one or more second clusters; generating a total cluster set by merging the first cluster and the second cluster; and The method includes data generated according to a step of randomly removing clusters included in the total set of clusters based on a predetermined ratio.

3. 1. A method for generating data used to train a lidar anomaly detection model, comprising: acquiring a normal data set; and generating at least one or more LIDAR-abnormal data sets based on the normal data set; The at least one or more LIDAR-abnormal data sets include a second type LIDAR-abnormal data set; The second type LIDAR-abnormal data set includes: Obtaining point cloud data for each data set from the normal data set; and The method includes data generated according to a step of randomly removing points included in the point cloud data based on a predetermined ratio.

4. 1. A method for generating data used to train a lidar anomaly detection model, comprising: acquiring a normal data set; and generating at least one or more LIDAR-abnormal data sets based on the normal data set; The at least one or more LIDAR-abnormal data sets include a third type LIDAR-abnormal data set; The third type LIDAR-abnormal data set includes: Obtaining point cloud data for each data set from the normal data set; transforming the coordinate system of the point cloud data from a Cartesian coordinate system to a cylindrical coordinate system; and The method includes data generated according to a step of randomly removing points included in the point cloud data based on a predetermined rule and a predetermined ratio.

5. further determining whether the lidar abnormality detection model confidence is less than a confidence threshold; not increasing the accumulated number of rider errors when the rider abnormality detection model confidence is less than a confidence threshold, and increasing the accumulated number of rider errors by 1 when the rider abnormality detection model confidence is greater than or equal to the confidence threshold; determining whether the cumulative number of lidar errors is greater than a threshold; and determining that the LIDAR is defective if the accumulated LIDAR error number is greater than the accumulated LIDAR error number threshold; The method of claim 1 , comprising:

6. further comprising pre-processing the point cloud data; The preprocessing step includes: Calculating a distance from the origin of a lidar coordinate system to each of the points included in the point cloud data; sorting the points in order of shortest distance based on the calculation results; and extracting and sampling a certain number of points based on the alignment result; The method of claim 1 , comprising:

7. further comprising pre-processing point cloud data included in the normal data set and the LIDAR-unnormal data set; The pre-processing step is Calculating a distance from the origin of a lidar coordinate system to each of the points included in the point cloud data; sorting the points in order of shortest distance based on the calculation results; and extracting and sampling a certain number of points based on the alignment result; 6. The method of claim 3, comprising:

8. 1. A method for detecting abnormalities in a lidar, comprising: acquiring point cloud data from the lidar; determining whether the number of points included in the point cloud data is less than a point number threshold; If the number of points is greater than or equal to the point number threshold, determining whether the time required to receive the point cloud data is less than a point cloud data reception time threshold; not incrementing a timer error accumulation number when the time required to receive the point cloud data is less than the point cloud data reception time threshold, and incrementing a timer error accumulation number by 1 when the time required to receive the point cloud data is greater than or equal to the point cloud data reception time threshold; determining whether the timer error accumulation number is greater than a timer error accumulation number threshold; and If the timer error accumulation number is greater than the timer error accumulation number threshold, determining that there is a defect not in the LIDAR but in the step of acquiring point cloud data from the LIDAR; A method comprising:

9. A device for detecting abnormalities in a rider, at least one memory for storing instructions and at least one processor; The at least one processor executes the instructions to: acquiring point cloud data from the lidar; determining whether the number of points included in the point cloud data is less than a point number threshold; and If the number of points is greater than or equal to the point number threshold, determining whether the LIDAR has a defect based on the point cloud data using a LIDAR anomaly detection model; and The LIDAR anomaly detection model is a pre-trained model based on a normal dataset and at least one or more LIDAR-abnormal datasets; The normal data set includes point cloud data acquired from the lidar under a predetermined statistical value and a normal condition; The lidar-non-normal dataset includes point cloud data generated based on the normal dataset.

10. 1. An apparatus for generating data used to train a lidar anomaly detection model, comprising: at least one memory for storing instructions and at least one processor; The at least one processor executes the instructions to An apparatus for carrying out the steps of the method according to any one of claims 3 to 5.

11. A device for detecting abnormalities in a rider, at least one memory for storing instructions and at least one processor; The at least one processor executes the instructions to: acquiring point cloud data from the lidar; determining whether the number of points included in the point cloud data is less than a point number threshold; If the number of points is greater than or equal to the point number threshold, determining whether the time required to receive the point cloud data is less than a point cloud data reception time threshold; not incrementing a timer error accumulation number when the time required to receive the point cloud data is less than the point cloud data reception time threshold, and incrementing a timer error accumulation number by 1 when the time required to receive the point cloud data is greater than or equal to the point cloud data reception time threshold; determining whether the timer error accumulation number is greater than a timer error accumulation number threshold; and If the timer error accumulation number is greater than the timer error accumulation number threshold, determining that there is a defect not in the LIDAR but in the step of acquiring point cloud data from the LIDAR; A device that carries out the above.

Images