System for removing noise of lidar sensor generated owing to bad weather meteorological environment, and its operating method

Abstract

To provide a system for removing noise of a LiDAR sensor generated owing to a bad weather meteorological environment, and its operating method.SOLUTION: In a LiDAR sensor noise removal system, a processor executes instructions including: a step S210 of receiving input of a first point cloud including noise from a LiDAR sensor; a step S220 of generating a two-dimensional first distance image and a two-dimensional first reflectivity image based upon the first point cloud; a step S230 of inputting the first distance image and the first reflectivity image to a previously learnt machine learning model to generate a two-dimensional first noise boundary surface image; and a step S240 of using the first noise boundary surface image to remove the noise included in the first point cloud.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a method for effectively removing noise generated in bad weather by a lidar sensor that is corely used in an autonomous vehicle, and a system for executing the same.

[0002] Specifically, in meteorological conditions of bad weather such as snow, rain, and fog, a large amount of fine floating particles are generated in the atmosphere. These fine floating particles obstruct the path of the laser beam of the lidar sensor and act as noise. The present invention relates to a lidar sensor noise removal system and an operation method thereof for effectively removing such noise.

Background Art

[0003] The method for removing noise of a vehicle lidar sensor can be roughly divided into a statistical approach method and a method using learning.

[0004] The statistical approach method is a method that utilizes the statistical characteristics of the distribution of point clouds. Typically, there are the Dynamic Radius Outlier Removal (DROR, [1]) and Dynamic Statistical Outlier Removal (DSOR, [2]) methods.

[0005] DROR is the first research result in this field and is based on the number of neighboring points of each point. DROR constructs a K-d tree and determines outliers based on the number of neighbors obtained using a dynamic search radius, utilizing the characteristic that the density of the target points is inversely proportional to the distance. The search radius of each location is dynamically adjusted according to the distance.

[0006] DSOR also depends on the search for the nearest neighbor, but calculates the average and variance of the relative distances based on the fixed number of neighbors of each point. That is, DSOR estimates the local density around each point and filters based on this.

[0007] As a learning-based method, WeatherNet ([3]) is the most representative algorithm. WeatherNet uses a LiDAR semantic segmentation method to reduce point cloud noise under different weather conditions and constructs training data to generate a semantic segmentation network.

[0008] 4DenoiseNet ([4]) presented a noise classification method that uses temporally continuous point cloud data and a semantic segmentation network. A significant amount of virtual synthetic data is used in 4DenoiseNet to train the semantic segmentation network.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0010]

Non-Patent Document 1

Non-Patent Document 2

[0011] A LiDAR sensor is a sensor that measures the distance to an object using the time difference between the transmission and reception of a laser beam, and is the most typical sensor used in recent autonomous vehicles. A LiDAR sensor generates a three-dimensional point cloud from the distance to an object and the emission angle of the laser beam. However, in adverse weather conditions, fine suspended matter in the air such as rain, snow, and fog obstructs the laser beam, induces noise, and significantly degrades the quality of the point cloud.

[0012] The object of the present invention is to determine whether each point of the point cloud measured from a lidar sensor is noise due to fine suspended matter or an actual object, remove the noise, and provide a lidar sensor noise removal system and its operation method that can provide a clean point cloud to the recognition system of an autonomous driving vehicle.

[0013] Also, the technology for removing noise is a technology corresponding to preprocessing in the recognition system of an autonomous driving vehicle. In order to provide more system resources by the main autonomous driving core technology, it is also an important object of the present invention to embody a technology that operates quickly with few resources. The ultimate object of the present invention is to enable an autonomous driving vehicle to drive more safely in bad weather meteorological conditions through the present invention.

[0014] The object of the present invention is not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

Means for Solving the Problems

[0015] The operation method of the lidar sensor noise removal system according to the first embodiment of the present invention includes the steps of: the system receiving an input of a first point cloud containing noise from a lidar sensor; the system generating a two-dimensional first distance image and a two-dimensional first reflectivity image based on the first point cloud; the system inputting the first distance image and the first reflectivity image into a pre-learned machine learning model to generate a two-dimensional first noise boundary image; and the system using the first noise boundary image to remove the noise contained in the first point cloud.

[0016] In the first embodiment of the present invention, the step of generating the first distance image and the first reflectivity image may include the system estimating the reflectivity for each point included in the first point cloud by using the impulse response of a single beam output of the lidar sensor, the coordinates of the first point cloud, and an adaptive medium variable reflecting the backscattering effect, and generating the first reflectivity image based on the reflectivity.

[0017] In the first embodiment of the present invention, the step of generating the first distance image and the first reflectivity image may include the system increasing the reflectivity for points in the first point cloud having a negative z - coordinate by applying a predetermined weighting value.

[0018] In the first embodiment of the present invention, the operation method may further include the system generating a second distance image and a second reflectivity image based on a pre - collected second point cloud in which noise points are labeled with noise labels; the system removing the noise points labeled with the noise labels in the second distance image to generate a third distance image, and multiplying the third distance image by a constant of 1 or less to generate a second noise boundary image; and the system using the second distance image, the second reflectivity image, and the second noise boundary image as training data to train the machine learning model.

[0019] In the first embodiment of the present invention, the step of generating the second noise boundary image

[0020] may further include the system correcting the second noise boundary image such that the points of the noise boundary corresponding to the noise points having a greater distance than the noise boundary appearing in the second noise boundary image are at a greater distance than the noise points having a greater distance if there are such noise points.

[0021] In the first embodiment of the present invention, the step of generating the second noise boundary surface image may include the system performing edge-preserving smoothing on the second noise boundary surface image.

[0022] In the first embodiment of the present invention, the machine learning model may be a generative model.

[0023] In the first embodiment of the present invention, the step of training the machine learning model includes the system training an encoder included in the machine learning model using the second distance image and the second reflectivity image as training data; and the system fixing the trained encoder and training a decoder included in the machine learning model using the second distance image, the second reflectivity image, and the second noise boundary surface image as training data.

[0024] In the first embodiment of the present invention, the step of training the decoder may include the system calculating a reconstruction loss between the output of the decoder and the second noise boundary surface image and training the decoder so that the reconstruction loss decreases.

[0025] In the first embodiment of the present invention, the operation method may further include the system generating the noise points and the noise labels assigned to the noise points using a pre-constructed weather simulator.

[0026] In the first embodiment of the present invention, the weather simulator may include any one or a combination of LISA (LiDAR light scattering augmentation), SnowSim (LiDAR snowfall simulation), and FogSim (Fog simulation on real LiDAR point clouds).

[0027] The operation method of the lidar sensor noise removal system according to the second embodiment of the present invention can include a method of training a machine learning model so that the lidar sensor noise removal system generates a noise boundary image used to select noise included in the point cloud generated by the lidar sensor. The method of training the machine learning model includes: a step in which the system generates a second distance image and a second reflectance image based on a pre-collected second point cloud in which noise labels are assigned to noise points; a step in which the system removes the noise points to which the noise labels are assigned in the second distance image to generate a third distance image, and multiplies the third distance image by a constant of 1 or less to generate a second noise boundary image; and a step in which the system uses the second distance image, the second reflectance image, and the second noise boundary image as training data to train the machine learning model.

[0028] In the second embodiment of the present invention, the step of generating the second noise boundary image can further include the system correcting the second noise boundary image so that points of the noise boundary corresponding to the noise points having a greater distance than the noise boundary appearing in the second noise boundary image have a greater distance than the noise points having a greater distance.

[0029] In the second embodiment of the present invention, the step of generating the second noise boundary image can include the system performing edge-preserving smoothing on the second noise boundary image.

[0030] In the second embodiment of the present invention, the machine learning model can be a generative model.

[0031] In the second embodiment of the present invention, the step of training the machine learning model includes: the system training an encoder included in the machine learning model using the second distance image and the second reflectivity image as training data; and the system fixing the trained encoder and training the decoder included in the machine learning model such that the reconstruction loss between the output of the decoder and the second noise boundary image decreases, using the second distance image, the second reflectivity image, and the second noise boundary image as training data.

[0032] The lidar sensor noise removal system according to the third embodiment of the present invention includes a memory storing computer-readable instructions; and at least one processor configured to execute the instructions.

[0033] By executing the instructions, the at least one processor is configured to receive an input of a first point cloud including noise from a lidar sensor, generate a first distance image and a first reflectivity image based on the first point cloud, input the first distance image and the first reflectivity image into a pre-trained machine learning model to generate a first noise boundary image, and use the first noise boundary image to remove the noise included in the first point cloud.

[0034] In the third embodiment of the present invention, the at least one processor may be configured to estimate the reflectivity for each point included in the first point cloud using the impulse response of a single beam output of the lidar sensor, the coordinates of the first point cloud, and an adaptive medium variable reflecting the backscattering effect, and generate the first reflectivity image based on the reflectivity.

[0035] In the third embodiment of the present invention, the at least one processor may be configured to increase the reflectivity for points in the first point cloud having a negative z coordinate by applying a predetermined weighting value.

[0036] In the third embodiment of the present invention, the at least one processor generates a second distance image and a second reflectivity image based on a pre-collected second point cloud in which noise labels are assigned to noise points, removes the noise points with the noise labels in the second distance image to generate a third distance image, multiplies the third distance image by a constant of 1 or less to generate a second noise boundary image, and may be configured to train the machine learning model using the second distance image, the second reflectivity image, and the second noise boundary image as training data.

Advantages of the Invention

[0037] The advantages of the present invention are as follows. 1. The present invention proposes a new method of generating and discriminating a noise boundary surface, which is a region where noise exists, instead of discriminating individual points to remove noise from the point cloud of a lidar. 2. The present invention represents the noise boundary surface as a two-dimensional image instead of a three-dimensional space, and can simplify the network structure because it does not require a detailed judgment compared to individual point processing, so it has a very fast execution speed compared to the existing technology. This is a very important factor in the application of autonomous vehicles. 3. The present invention proposes a method of numerically interpreting a method of restoring reflectivity to determine a noise boundary surface. The present invention shows much improved performance in classifying noise compared to the received intensity which is the input of the sensor. 4. The present invention proposes a method for generating training data and model training for a generative model that generates a noise boundary surface. The present invention provides a method for efficiently training a network by performing unsupervised and supervised learning in parallel even without a lot of labeled data.

[0038] The effects that can be obtained by the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those with ordinary knowledge in the technical field to which the present invention belongs from the following description.

Brief Description of the Drawings

[0039]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5a

Figure 5b

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0040] The list of references of the present invention is as follows: [1] to

[10] . Each reference or the methodology proposed in each reference in this specification can be referred to by the numbers assigned to each document as follows. The entire content of reference [5] is included in this specification as a reference. [1] Charron, Nicholas, Stephen Phillips, and Steven L. Waslander. "De-noising of lidar point clouds corrupted by snowfall." 2018 15th Conference on Computer and Robot Vision (CRV). IEEE, 2018. [2] Kurup, Akhil, and Jeremy Bos. "Dsor: A scalable statistical filter for removing falling snow from lidar point clouds in severe winter weather." arXiv preprint arXiv:2109.07078 (2021). [3] Heinzler, Robin, et al. "Cnn-based lidar point cloud de-noising in adverse weather." IEEE Robotics and Automation Letters 5.2 (2020): 2514-2521. [4] Seppanen, Alvari, Risto Ojala, and Kari Tammi. "4denoisenet: Adverse weather denoising from adjacent point clouds." IEEE Robotics and Automation Letters 8.1 (2022): 456-463. [5] Han, Seung-Jun, et al. "RGOR: De-noising of LiDAR point clouds with reflectance restoration in adverse weather." 2023 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, 2023. [6] He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." IEEE transactions on pattern analysis and machine intelligence 35.6 (2012): 1397-1409. [7] Kilic, Velat, et al., "Lidar light scattering augmentation (lisa): Physics-based simulation of adverse weather conditions for 3d object detection," arXiv preprint arXiv:2107.07004 (2021). [8] Hahner, Martin, et al., "Lidar snowfall simulation for robust 3d object detection," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022. [9] Hahner, Martin, et al., "Fog simulation on real LiDAR point clouds for 3D object detection in adverse weather," Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021.

[10] Kingma, Diederik P., and Max Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114 (2013).

[0041] The present invention relates to a system and an operating method for removing noise of a lidar sensor generated due to adverse weather conditions. This specification describes a method of generating and removing a noise-occurring region using a generative model, rather than a classification method through semantic segmentation of each point. The present invention has features that operations are processed in a two-dimensional region instead of a three-dimensional region, and a noise-existing region is roughly generated instead of classification for individual points. Therefore, the present invention has an advantage that operations are very quickly performed because it can be executed by a simple network.

[0042] The advantages and features of the present invention, and the method for achieving them, will become clear by referring to the embodiments described in detail below together with the attached drawings. However, the present invention is not limited to the embodiments disclosed below and can be embodied in various different forms. However, these embodiments are provided to make the disclosure of the present invention complete and to fully inform those with ordinary knowledge in the technical field to which the present invention pertains of the scope of the invention. The present invention is only defined by the scope of the claims. On the other hand, the terms used in this specification are for the purpose of explaining the embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated otherwise in the context. The terms "comprises" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other constituent elements, steps, operations, and / or elements.

[0043] Terms such as first, second, etc. can be used to describe various components, but the components should not be limited by these terms. These terms can be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the rights of the present invention, the first component can be named the second component, and similarly, the second component can be named the first component.

[0044] When it is mentioned that a certain component is "connected to" or "attached to" another component, it should be understood that it may be directly connected or attached to the other component, or there may be other components in between. On the contrary, when it is mentioned that a certain component is "directly connected to" or "directly attached to" another component, it should be understood that there are no other components in between. Other expressions for explaining the relationship between components, such as "between", "immediately between", or "adjacent to", "directly adjacent to", etc. should be interpreted in the same way.

[0045] In describing the present invention, when it is determined that a specific description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

[0046] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate an overall understanding in describing the present invention, the same reference numerals will be used for the same means regardless of the drawing numbers.

[0047] 1. Motivation of the Invention

[0048] FIG. 1 is a drawing showing the driving principle of a lidar sensor and the generation principle of noise. To explain the motivation of the present invention, with reference to FIG. 1, the principle of noise generation in the operation process of the lidar sensor will be described.

[0049] As shown in FIG. 1(a), after the laser beam of the lidar sensor is emitted from the transmitter of the sensor, it passes through the atmosphere, is reflected by an object, and returns to the receiver of the sensor. In this process, when fine particles such as snow, rain, and fog are present in the atmosphere, these fine particles can be detected as noise.

[0050] FIG. 1(b) shows the received power measured by an actual sensor. When the received power generated by fine particles is greater than the received power generated by an object, the fine particles are detected. The received power of the fine particles is determined by factors such as the size of the fine particles, scattering characteristics, degree of overlap with the laser beam, and distance from the light source. The received power (magnitude of the received power) is referred to as the received intensity, which is a value that can be obtained by an actual sensor. Since the laser beam has a cone shape, the energy of the laser beam is greater and the overlap is greater closer to the light source. Therefore, fine particles near the sensor are mainly detected, and fine particles near an object with a wide reflection surface are hardly detected.

[0051] On the one hand, as shown in Fig. 1(c), the reflectance of fine particles and objects shows quite different characteristics from the received power measured by the sensor. Even if the received power of the fine particles is strong and detected as noise, since the fine particles (particles) are small in size, their reflectance actually has a very small characteristic. Since fine particles such as snow, rain, and fog are non-metallic substances, their reflectance is lower than that of metallic substances. For example, most of the suspended substances in the air are moisture, and the reflectance of moisture is relatively low compared to the structures beside the road and other vehicles driving on the road.

[0052] As described above, if noise is generated only near the sensor and the boundary surface between the noise and the object can be known based on the characteristic that the actual reflectance is very small, the noise can be easily removed by assuming the points inside this boundary surface as noise and removing them.

[0053] In order to provide means for implementing such a concept, the present invention provides a method for estimating reflectance, a method for generating learning data for learning a generative model for generating a boundary surface between noise and an object, and a method for learning the generative model.

[0054] Since the present invention adopts a method of roughly generating an area where noise exists instead of classifying individual points, it can be implemented with a lightweight network and has very fast execution performance.

[0055] 2. Outline of the Invention

[0056] Fig. 2 is a drawing for explaining a lidar sensor noise removal method according to an embodiment of the present invention. The overall process proposed by the present invention is shown in Fig. 2.

[0057] The method may be included in an operation method of a lidar sensor noise removal system according to an embodiment of the present invention. That is, the lidar sensor noise removal method may be performed by a lidar sensor noise removal system 1000.

[0058] (a) of FIG. 2 shows a point cloud with noise measured by a lidar sensor. For example, the noise may be generated by bad weather conditions.

[0059] (b) of FIG. 2 shows a point cloud with restored reflectance. The lidar sensor noise removal method restores the reflectance of each point and substitutes the received intensity of the sensor with the restored reflectance in order to better classify fine suspended matter in the point cloud. (b) of FIG. 2 is an image that differently represents the hue of each pixel according to the value of the reflectance corresponding to each pixel. The reflectance has a value between 0 and 1. For example, when following the JET color code, the pixel corresponding to a reflectance of 0 is represented in blue, and the pixel corresponding to a reflectance of 1 is represented in red.

[0060] (c) of FIG. 2 shows a generative model for noise boundary determination. The generative model is a model that determines the boundary of the region where noise is formed within the point cloud.

[0061] (d) of FIG. 2 shows the noise boundary surface. The method uses a range image, which is a two-dimensional representation of the three-dimensional point cloud, for fast processing of the point cloud and the boundary surface. That is, the method generates the noise boundary surface using a two-dimensional image.

[0062] (e) of FIG. 2 shows a point cloud with noise removed. The method finally obtains a clean point cloud by removing the points inside the noise boundary surface.

[0063] 3. Reflectance Restoration Method

[0064] (b) of FIG. 2 has been described as a point cloud with restored reflectance. The reflectance restoration method according to the present invention will be described below.

[0065] The lidar sensor measures two scalar values for the strongest reflection, namely the distance r calculated over time and the intensity μ, which is the magnitude of the received power (P(r)). The received power P(r) is a function of distance or time, and a simple mathematical model for the received power of a single laser beam is as shown in Equation 1.

[0066]

Equation

[0067] In Equation 1, K is a constant representing the performance of the lidar sensor. K can be set differently depending on the wavelength and intensity of the sensor's laser. G(r) is a geometric element, and G(r) = O(r) / r 2 is defined as. Here, O(r) ∈ [0, 1] is the degree of overlap between the beam and the laser detector. O(r) converges to 1 when the proximity sensing performance is above a certain distance. And β(r) represents the backscatter effect and is affected by the particle size and laser wavelength. The last term in Equation 1, T(r), means transmittance and can be expressed as T(r) = exp(-2αr). Here, α is the extinction coefficient and generally has a small value. Therefore, at a short distance, the transmittance T(r) approaches 1.

[0068] The impulse response R(r) of a single beam, which is the output of an actual lidar sensor, can be shown as in Equation 2.

[0069]

Equation

[0070] In Equation 2, ρ is the reflectivity of the object and δ is the Dirac delta function. r t indicates the distance at time t.

[0071] The reflectance (restored reflectance) ρ can be calculated as shown in Equation 3 based on Equation 1 and Equation 2.

[0072]

Number

[0073] When the distance of the object is r, the energy measured by the impulse response R(r) can be expressed as a scalar value of intensity μ, and the distance can be defined as r = (x 2 + y 2 + z 2 ) 1 / 2 In Equation 3, K is a constant. Since G(r) = 1 / r for distances greater than a certain distance (e.g., several tens of cm) and T(r) = 1 can be assumed for short distances (e.g., several tens of m), the reflectance ρ in Equation 3 can be simplified as shown in Equation 4. 2 In Equation 4, the intensity μ is generally a quantized positive value. When the value of the intensity μ is 0, a small constant must be added to the intensity μ to normalize it so that it is not ignored. γ = Kβ(r) is a constant related to the performance of the lidar sensor and the size of the particles. γ is an adaptive medium variable (hyperparameter) that varies depending on the type of sensor and particles (such as fog, rain, snow).

[0074]

Number

[0075]

[0076] ​The point cloud collected by the lidar sensor mounted on the vehicle has the range of negative values of the z-axis coordinate restricted to the road surface. That is, since the origin of the coordinates (x, y, z) of the points included in the point cloud is the position of the lidar sensor (which can be installed on the roof of the vehicle, for example), among the points with a negative z-axis coordinate, the point with the largest absolute value of the z-axis coordinate is the point on the road surface. There is data only up to the road surface in the downward direction (-z) with respect to the lidar sensor. Therefore, the efficiency can be improved by introducing a weighting value (κ) to the z-axis as in Mathematical Formula 5.

[0077]

Number

[0078] JPEG2025094928000007.jpg33170

[0079] 4. Learning Data Generation Method

[0080] In the present invention, a learned generative model is used to generate a noise boundary surface (or a noise boundary surface video). For the learning of the generative model for generating the noise boundary surface, the distance video TD1, the reflectance video TD3, and the corresponding noise boundary surface video TD5 are used as learning data. A method for generating the learning data will be described below.

[0081] 4.1 Distance Video and Reflectance Video

[0082] Generating a three-dimensional point cloud using a generative model is a very difficult problem. Therefore, in order to simplify the problem, it is preferable to convert the three-dimensional point cloud into a two-dimensional range image (TD1). First, the azimuth angle and elevation angle with respect to the coordinates of each point included in each three-dimensional point cloud are obtained, and the two-dimensional range image TD1 for each point is obtained by quantizing the azimuth angle and the elevation angle so as to correspond to the horizontal and vertical axes of the image respectively. Then, the distance of the point corresponding to the coordinates of the range image TD1 is expressed in hue code to obtain the two-dimensional range image TD1. The distance between the lidar sensor and the point (i.e., the distance of the point) r, the azimuth angle θ az and the elevation angle θ el are defined as in Mathematical Formulas 6 to 8.

[0083]

Equation

[0084]

Equation

[0085]

Equation

[0086] The lidar sensor provides the azimuth angle (θ az ) and the elevation angle (θ el ) together with the distance r and intensity μ information for each measured point. Therefore, if the data collected by the lidar sensor is used directly, the range image TD1 can be obtained more easily.

[0087] Figure 3 is an exemplary drawing of learning data, which is an example of learning data for a situation where snow is falling.

[0088] Fig. 3(a) is an illustration of a distance image (TD1, range image), where the distance r is represented by a hue code to enhance visibility, and only the ±π / 2 region of the azimuth angle (θ az ) is shown. The actual azimuth angle region of the distance image TD1 is ±π.

[0089] JPEG2025094928000011.jpg79164

[0090] Therefore, the present invention uses the distance image TD1 and the reflectance image TD3 as the first training data. The distance image TD1 and the reflectance image TD3 can be automatically generated as long as data is collected through a lidar sensor. Therefore, a large number of distance images TD1 and reflectance images TD3 can be generated and used for the encoder learning of the generative model.

[0091] Incidentally, in the illustration of Fig. 3, the actual data of (a) to (e) are all single-channel real numbers. When creating this data, angular data was mapped to the same image coordinates for generation. The illustration of Fig. 3 represents such actual data in an image, and each image is represented by a hue code that matches the range and characteristics of each data.

[0092] In the images TD1 and TD5 of Fig. 3(a) and (e), the distance is represented by color. In Fig. 3(a) and (e), assuming that the respective distance data are mapped to the same image coordinates, the hue code used in "Stereo Evaluation" of the "KITTI Vision Benchmark Suite" (https: / / www.cvlibs.net / datasets / kitti / eval_stereo_flow.php?benchmark=stereo) was used. The hues are arranged in the order of white, green, red, and blue from near to far distances.

[0093] Fig. 3(b) is a reflectance intensity image (intensity image, TD2) represented in grayscale. The value of each pixel is represented by 8 bits (0 to 255).

[0094] The reflectance corresponding to each pixel in the reflectance image TD3 of (c) in FIG. 3 has a value between 0 and 1. In the example of (c) in FIG. 3, the JET color code is used, and it is represented as blue if close to 0 and red if close to 1.

[0095] What corresponds to each pixel in the noise label image TD4 of (d) in FIG. 3 is binary data indicating the presence or absence of noise. The noise label image TD4 is represented as white (not noise) if it is 0 and red (noise) if it is 1.

[0096] 4.2 Noise boundary image

[0097] The second training data required for generating the noise boundary surface in the present invention is the noise boundary surface image (TD5, noise region boundary image) illustrated in (e) of FIG. 3.

[0098] FIG. 4 is a flowchart for explaining a method for generating the noise boundary surface image TD5 according to an embodiment of the present invention. The method can be performed by the lidar sensor noise removal system 1000.

[0099] First, the lidar sensor noise removal system 1000 receives an input of point cloud (S110). Then, the lidar sensor noise removal system 1000 removes the points labeled as noise in the point cloud and generates a 2D distance image using the azimuth, elevation angle, and distance of the remaining points (S120). Next, the lidar sensor noise removal system 1000 fills the areas vacated by the removal of the points labeled as noise with the values of the surrounding points (S130). That is, the values of the points corresponding to the vacant areas are determined through interpolation using the surrounding points. The range of the surrounding points of the vacant points follows the settings. And the lidar sensor noise removal system 1000 corrects the distance image generated in step S130 with a margin so that no non-noise points are included inside the noise boundary surface (S140). Specifically, the lidar sensor noise removal system 1000 can correct the distance image by multiplying the distance image generated in step S130 by a correction constant of 1 or less. For example, the correction constant can be selected from values ranging from 0.8 to 0.9. Thereafter, the lidar sensor noise removal system 1000 projects the points corresponding to the noise onto the corrected distance image to check whether interference occurs (S150). If interference occurs despite the correction (for example, if there are noise points at a distance farther than the corrected distance image), the distance image is re-corrected with the intermediate value between the value before correction and the noise in the distance image (S160). At this time, the range of the re-correction includes up to the vicinity of the point where interference occurred (interference point), and the range of the vicinity of the interference point is determined by the settings. If it is confirmed in step S150 that no interference occurs, edge-preserving smoothing (or edge-aware smoothing) is performed on the corrected distance image so that the boundary of the distance image is not damaged, and a noise boundary surface image TD5 is generated. For this purpose, the lidar sensor noise removal system 1000 can use a guided filter ([6]).

[0100] 4.3 Use of Weather Simulator

[0101] To generate a noise boundary image, noise labels are required. Creating noise labels for a large amount of three-dimensional points sufficient for training a deep learning network is a very difficult problem. Therefore, it is preferable to use a weather simulator for generating noise labels. As a weather simulator for a lidar sensor, any one or a combination of the weather simulators of LISA ([7]), SnowSim ([8]), and FogSim ([9]) can be used. LISA is suitable for generating noise caused by rain, SnowSim is suitable for generating noise caused by snow, and FogSim is suitable for generating noise caused by fog.

[0102] The lidar sensor noise removal system 1000 can apply a weather simulator to pre-collected lidar sensor data to automatically generate synthetic data and respective labels for various weather conditions and use them as training data. Also, since the simulator alone cannot perfectly simulate the actual situation, it is preferable to add data labels collected in the actual situation.

[0103] 5. Learning of the Generative Model

[0104] The lidar sensor noise removal system 1000 according to an embodiment of the present invention uses a generative model to generate a noise boundary image based on a distance image and a reflectivity image. To increase the calculation speed, it is preferable to use a simple and efficient variational autoencoder (VAE) model as the generative model (see

[10] ).

[0105] As shown in FIGS. 5a and 5b, a generative model generally has a structure in which an encoder converts input data into a latent value, and a decoder restores the latent value to the original data. At this time, if the latent value has a known distribution, even if an unlearned input comes in, the output value can be stably generated. Since the latent value of the variational autoencoder (VAE) follows a normal distribution, it is suitable as the structure of the generative model for generating noisy boundary surface images in the present invention.

[0106] The present invention provides the following two-stage learning method for learning a generative model for generating a noisy boundary surface.

[0107] 5.1 Encoder learning

[0108] The first stage of the generative model learning method according to the present invention is a stage of learning the entire generative model using only the distance image TD1 and the reflectance image TD3 as shown in FIG. 5(a). In this stage, the learning targets are the encoder E1 and the decoder D1. However, since the decoder used in the final generative model is the decoder D2 learned in the second stage, the object to be substantially learned in this stage is the encoder E1.

[0109] Since the distance image TD1 and the reflectivity image TD3 are created only from lidar sensor data without learning labels, it can be provided only by the operation of collecting data with a lidar sensor. That is, the lidar sensor noise removal system 1000 can learn a generative model by an unsupervised learning method based on a lot of data collected through the lidar sensor. The lidar sensor noise removal system 1000 inputs the distance image TD1 and the reflectivity image TD3 into the encoder E1, and the decoder D1 generates the distance image TD1 and the reflectivity image TD3 again based on the latent values generated by the encoder E1. The lidar sensor noise removal system 1000 learns the generative model (encoder E1, decoder D1) so that the reconstruction loss between the input (TD1 and TD3) and the output (TD1 and TD3) of the generative model decreases.

[0110] At this stage, the lidar sensor noise removal system 1000 can utilize the weather simulator and the data collected under various weather conditions. Through this, the encoder E1 of the generative model can be learned to well reproduce the input data with latent values.

[0111] 5.2 Decoder Learning

[0112] The second stage of the generative model learning method according to the present invention is the stage of learning so that the generative model generates a noise boundary surface. The learning target at this stage is the decoder D2. That is, at this stage, the lidar sensor noise removal system 1000 constructs a generative model that combines a newly learned decoder D2 with the pre-learned encoder E1 without using the decoder D1 learned in the first stage.

[0113] On the other hand, since the execution speed of the weather simulator is very slow, it is difficult to generate as much data as to learn the entire generative model even using the simulator, and there is a possibility that an overfitting problem may occur if the ratio of the data generated by the simulator is higher than the actual data in the learning data. A generative model that mimics lidar sensor data using a lot of data was obtained in the first stage.

[0114] In the second stage, the lidar sensor noise removal system 1000 trains only the decoder D2 that generates the noise boundary image using relatively less data than in the first stage. As shown in FIG. 5(b), the lidar sensor noise removal system 1000 trains only the new decoder D2 while keeping (fixing) the encoder E1 trained in the first stage as it is. At this time, the lidar sensor noise removal system 1000 inputs the distance image TD1 and the reflectivity image TD3 to the encoder E1 in the same manner as in the first stage. Then, the lidar sensor noise removal system 1000 uses the noise boundary image TD5 corresponding to the distance image TD1 and the reflectivity image TD3 as a label to train the decoder D2 so that the reconstruction loss between the output of the generative model and the noise boundary image TD5 decreases.

[0115] 6. Noise Removal Method

[0116] Finally, a method for removing noise from lidar sensor data in real time in an autonomous vehicle will be described. First, the lidar sensor noise removal system 1000 generates a distance image and a reflectivity image based on lidar sensor data collected in real time, and inputs the distance image and the reflectivity image into a pre-trained generative model to generate a noise boundary image. Then, among the points included in the point cloud collected through the lidar sensor, the lidar sensor noise removal system 1000 regards the points closer to the noise boundary appearing in the noise boundary image as noise and deletes them. Through such a method, the lidar sensor noise removal system 1000 can remove noise points from the point cloud collected through the lidar sensor and obtain a clean point cloud.

[0117] 7. Lidar Sensor Noise Removal System and Its Operation Method

[0118] FIG. 6 is a block diagram showing the configuration of a lidar sensor noise removal system according to an embodiment of the present invention. The lidar sensor noise removal system according to an embodiment of the present invention can be implemented in the form of the computer system of FIG. 6.

[0119] Referring to FIG. 6, the lidar sensor noise removal system 1000 can include at least one of at least one processor 1010, a memory 1030, an input interface device 1050, an output interface device 1060, and a storage device 1040 that communicate through a bus 1070. The lidar sensor noise removal system 1000 can also further include a communication device 1020 coupled to a network.

[0120] The lidar sensor noise removal system 1000 illustrated in FIG. 6 is according to one embodiment, and the components of the lidar sensor noise removal system 1000 according to the present invention are not limited to the embodiment illustrated in FIG. 6 and can be added, changed, or deleted as necessary.

[0121] The processor 1010 can be a central processing unit (CPU) or a semiconductor device that executes computer-readable instructions stored in the memory 1030 or the storage device 1040. The memory 1030 and the storage device 1040 can include various forms of volatile or non-volatile storage media. For example, the memory 1030 can include a ROM (read only memory) and a RAM (random access memory). In the embodiments described herein, the memory 1030 can be located inside or outside the processor 1010, and the memory 1030 can be coupled to the processor 1010 through various means already known. The memory 1030 is various forms of volatile or non-volatile storage media. For example, the memory 1030 can include a read-only memory (ROM) or a random access memory (RAM).

[0122] Accordingly, embodiments of the present invention can be implemented in a method embodied in a computer or in a non-transitory computer-readable medium storing computer-executable instructions. In one embodiment, when executed by a processor 1010, the computer-readable instructions can perform a method according to at least one aspect described herein.

[0123] The communication device 1020 can transmit or receive a wired signal or a wireless signal.

[0124] Also, the operation method of the rider sensor noise removal system according to an embodiment of the present invention can be embodied in a program instruction form executable through various computer means and can be recorded on a computer-readable medium.

[0125] The computer-readable medium can include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable medium can be those specially designed and configured for embodiments of the present invention or those available to ordinary technicians in the field of computer software. The computer-readable recording medium can include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium can be a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical medium such as a CD-ROM, a DVD, a magneto-optical medium such as a floptical disk, a ROM, a RAM, a flash memory, etc. The program instructions can include not only machine language code made by a compiler but also high-level language code executable by a computer through an interpreter or the like.

[0126] The processor 1010 executes computer-readable instructions stored in the memory 1030 or the storage device 1040, and by executing the instructions, receives an input of a first point cloud containing noise from a lidar sensor, generates a first distance image and a first reflectivity image based on the first point cloud, inputs the first distance image and the first reflectivity image into a pre-trained machine learning model to generate a first noise boundary image, and is configured to remove the noise included in the first point cloud using the first noise boundary image.

[0127] The processor 1010 may be configured to estimate the reflectivity for each point included in the first point cloud using the impulse response of the single-beam output of the lidar sensor, the coordinates of the first point cloud, and an adaptive medium variable reflecting the backscattering effect, and generate the first reflectivity image based on the reflectivity.

[0128] The processor 1010 may be configured to increase the reflectivity for points in the first point cloud having a negative z coordinate by applying a predetermined weighting value.

[0129] The processor 1010 generates a second distance image and a second reflectivity image based on a pre-collected second point cloud to which noise labels are assigned to noise points, removes the noise points to which the noise labels are assigned in the second distance image to generate a third distance image, multiplies the third distance image by a constant of 1 or less to generate a second noise boundary image, and may be configured to use the second distance image, the second reflectivity image, and the second noise boundary image as training data to train the machine learning model.

[0130] The machine learning model may be a variational autoencoder (VAE)-based generative model.

[0131] FIG. 7 and FIG. 8 are flowcharts for explaining an operation method of a lidar sensor noise removal system according to an embodiment of the present invention. Specifically, FIG. 7 is a flowchart regarding a lidar sensor noise removal method included in the operation method, and FIG. 8 is a flowchart regarding a learning method of a generative model included in the operation method.

[0132] Referring to FIG. 7, a lidar sensor noise removal method according to an embodiment of the present invention is composed of steps S210 to S240. The lidar sensor noise removal method illustrated in FIG. 7 is according to an embodiment, and the steps of the lidar sensor noise removal method according to the present invention are not limited to the embodiment illustrated in FIG. 7 and may be added, changed, or deleted as necessary.

[0133] Step S210 is a step of receiving an input of point cloud.

[0134] The processor 1010 receives an input of a first point cloud including noise from the lidar sensor.

[0135] Step S220 is a step of generating a reflectivity image.

[0136] The processor 1010 generates a two-dimensional first distance image TD1 and a two-dimensional first reflectivity image TD3 based on the first point cloud. Since the distance image and the reflectivity image have been described above, the description is omitted.

[0137] In this step, the processor 1010 can estimate the reflectivity for each point included in the first point cloud by using the impulse response of a single beam output of the lidar sensor, the coordinates of the first point cloud, and an adaptive medium variable reflecting the backscattering effect, and generate the first reflectivity image TD3 based on the reflectivity.

[0138] In the process of generating the first distance image TD1 and the first reflectivity image TD3, the processor 1010 can increase the reflectivity for points with a negative z coordinate in the first point cloud by applying a predetermined weighting value.

[0139] Step S230 is a step of generating a noise boundary image.

[0140] The processor 1010 inputs the first distance image TD1 and the first reflectance image TD3 into a pre-trained machine learning model (e.g., a variational autoencoder model (VAE) which is one of the generative models) to generate a two-dimensional first noise boundary image TD5. Since the noise boundary image and its generation method have been described above with reference to FIG. 4 and the like, the description is omitted.

[0141] Step S240 is a step of removing noise included in the point cloud.

[0142] The processor 1010 uses the first noise boundary image TD5 to remove the noise included in the first point cloud.

[0143] Hereinafter, with reference to FIG. 8, a learning method of a generative model for generating a noise boundary image will be described. The generative model is a model based on a machine learning model. As described above, the learning method is included in the operation method of the lidar sensor noise removal system 1000.

[0144] The learning method of the generative model illustrated in FIG. 8 may be executed prior to the lidar sensor noise removal method illustrated in FIG. 7 or may be executed in parallel with the lidar sensor noise removal method.

[0145] Referring to FIG. 8, the learning method of the generative model according to an embodiment of the present invention is composed of steps S310 to S330. The learning method of the generative model illustrated in FIG. 8 is according to an embodiment, and the steps of the learning method of the generative model according to the present invention are not limited to the embodiment illustrated in FIG. 8 and may be added, changed or deleted as necessary.

[0146] Step S310 is a step of generating a distance image and a reflectance image from learning data for training a machine learning model.

[0147] The processor 1010 generates a second distance image TD1 and a second reflectivity image TD3 based on a pre-collected second point cloud in which noise labels are assigned to noise points. Since the distance image and the reflectivity image have been described above, the description is omitted.

[0148] Before executing the step S310, the processor 1010 can generate the noise points and the noise labels assigned to the noise points by using a pre-constructed weather simulator. The weather simulator can include any one or a combination of LISA (LiDAR light scattering augmentation), SnowSim (LiDAR snowfall simulation), and FogSim (Fog simulation on real LiDAR point clouds).

[0149] The step S320 is a step of generating a noise boundary image from the training data for training the machine learning model.

[0150] The processor 1010 removes the noise points with noise labels in the second distance image TD1 to generate a third distance image, and multiplies the third distance image by a constant of 1 or less to generate a second noise boundary image TD5.

[0151] When there are noise points whose distances are greater than the noise boundary appearing in the second noise boundary image TD5, the processor 1010 can correct the second noise boundary image TD5 so that the points of the noise boundary corresponding to the noise points with greater distances are farther than the noise points with greater distances.

[0152] The processor 1010 can perform edge-preserving smoothing on the generated or corrected second noise boundary image TD5.

[0153] The step S330 is a step of training a machine learning model by using the training data generated in the previous step.

[0154] The processor 1010 uses the second distance image TD1, the second reflectivity image TD3, and the second noise boundary image TD5 as training data to train a machine learning model. The machine learning model can be a variational autoencoder (VAE)-based generative model.

[0155] At this stage, the processor 1010 uses the second distance image TD1 and the second reflectivity image TD3 as training data to train the encoder (E1) and the decoder D1 included in the machine learning model. Then, the processor 1010 replaces the decoder D1 with a new decoder D2 in the machine learning model. Then, the processor 1010 fixes the already trained encoder E1 in the machine learning model, and uses the second distance image TD1, the second reflectivity image TD3, and the second noise boundary image TD5 as training data to train the decoder (D2) newly included in the machine learning model so that the reconstruction loss between the output of the decoder D2 and the second noise boundary image TD5 decreases.

[0156] The operation method of the lidar sensor noise removal system described above has been described with reference to the flowcharts presented in FIGS. 7 and 8. For simplicity of explanation, the operation method has been illustrated and described in a series of blocks, but the present invention is not limited to the order of the blocks, and some blocks may occur in a different order or simultaneously from those illustrated and described herein, and various other branches, flow paths, and orders of blocks that achieve the same or similar results can be implemented. Also, not all blocks illustrated for the implementation of the operation method may be required.

[0157] On the one hand, in the description with reference to FIGS. 7 to 8, each step may be further divided into additional steps or combined into fewer steps according to the embodiments of the present invention. Also, some steps may be omitted as necessary, and the order between steps may be changed. Also, even for other omitted content, the content of FIGS. 1 to 5b (1. Motivation of the Invention to 6. Noise Removal Method) may be applied to the content of FIGS. 6 to 8. Also, the content of FIGS. 6 to 8 may be applied to the content of FIGS. 1 to 5b (1. Motivation of the Invention to 6. Noise Removal Method).

[0158] In the foregoing, the present invention has been described with reference to preferred embodiments thereof. However, it will be understood by those skilled in the relevant art that the present invention can be variously modified and changed without departing from the spirit and scope of the present invention described in the following claims.

Description of Reference Numerals

[0159] 1000: Lidar Sensor Noise Removal System 1010: Processor 1020: Communication Device 1030: Memory 1040: Storage Device 1050: Input Interface Device 1060: Output Interface Device 1070: Bus

Claims

1. 1. A method of operating a lidar sensor noise reduction system, comprising: (a) receiving, by the LIDAR sensor noise reduction system, an input of a first noisy point cloud from a LIDAR sensor; (b) the lidar sensor noise reduction system generating a first two-dimensional range image and a first two-dimensional reflectance image based on the first point cloud; (c) the lidar sensor noise reduction system inputs the first range image and the first reflectance image into a pre-trained machine learning model to generate a two-dimensional first noise interface image; and (d) removing noise contained in the first point cloud by using the first noise boundary surface image.

2. The (b) is 2. The method of claim 1, wherein the lidar sensor noise reduction system estimates a reflectivity for each point included in the first point cloud using an impulse response of a single beam output of the lidar sensor, coordinates of the first point cloud, and adaptive parameters reflecting a backscattering effect, and generates the first reflectivity image based on the reflectivity.

3. The (b) is 3. The method of claim 2, wherein the lidar sensor noise reduction system includes increasing the reflectivity for points in the first point cloud that have negative z coordinates by applying a predetermined weighting value.

4. (e) the lidar sensor de-noising system generating a second range image and a second reflectance image based on a second pre-collected point cloud in which noise points are assigned noise labels; (f) the lidar sensor noise reduction system removes the noise points assigned with the noise labels from the second distance image to generate a third distance image, and multiplies the third distance image by a constant less than or equal to 1 to generate a second noise boundary image; and 2. The method of claim 1, further comprising: (g) training the machine learning model using the second distance image, the second reflectivity image, and the second noise boundary image as training data.

5. The (f) is 5. The method of claim 4, wherein the lidar sensor noise reduction system further comprises, when there is a noise point that is greater in distance than the noise boundary surface that appears in the second noise boundary surface image, correcting the second noise boundary surface image so that a point on the noise boundary surface that corresponds to the noise point that is greater in distance has a greater distance than the noise point that is greater in distance.

6. The (f) is 5. The method of claim 4, wherein the lidar sensor noise reduction system comprises performing edge-preserving smoothing on the second noise boundary image.

7. The machine learning model is The method of operation of the LIDAR sensor noise reduction system of claim 4, which is a generative model.

8. The (g) is (h) the LIDAR sensor noise reduction system uses the second distance image and the second reflectance image as learning data to train an encoder included in the machine learning model; and 8. The method of claim 7, further comprising: (i) fixing the trained encoder, and training a decoder included in the machine learning model using the second range image, the second reflectivity image, and the second noise boundary surface image as training data.

9. The above (i) is 9. The method of claim 8, wherein the lidar sensor denoising system includes calculating a reconstruction loss between an output of the decoder and the second noise boundary image, and training the decoder to reduce the reconstruction loss.

10. 5. The method of claim 4, further comprising: (j) the lidar sensor noise reduction system generating the noise points and the noise labels assigned to the noise points using a pre-built weather simulator.

11. The weather simulator includes:

11. The method of operating the lidar sensor noise reduction system of claim 10, comprising: LISA (LiDAR light scattering augmentation), SnowSim (LiDAR snowfall simulation), and FogSim (Fog simulation on real LiDAR point clouds).

12. 1. A method of training a machine learning model to generate a noise interface image, the noise interface image being used by a lidar sensor denoising system to filter noise contained in a point cloud generated by the lidar sensor, the method comprising: (k) generating a second range image and a second reflectance image based on a second pre-collected point cloud in which noise points are labeled with noise labels by the lidar sensor de-noising system; (l) the lidar sensor noise reduction system generating a third distance image by removing the noise points assigned with the noise labels from the second distance image, and generating a second noise boundary image by multiplying the third distance image by a constant less than or equal to 1; and (m) a method of operating a lidar sensor noise reduction system, further comprising: a step of training the machine learning model using the second distance image, the second reflectivity image, and the second noise boundary surface image as training data.

13. The (l) is 13. The method of claim 12, wherein the lidar sensor noise reduction system further comprises, when there is a noise point that is greater in distance than the noise boundary surface that appears in the second noise boundary surface image, correcting the second noise boundary surface image so that a point on the noise boundary surface that corresponds to the noise point that is greater in distance has a greater distance than the noise point that is greater in distance.

14. The (l) is 13. The method of claim 12, wherein the lidar sensor noise reduction system comprises performing edge-preserving smoothing on the second noise boundary image.

15. The machine learning model is The method of operation of the LIDAR sensor noise reduction system of claim 12, which is a generative model.

16. The (m) is (n) the lidar sensor noise reduction system uses the second distance image and the second reflectance image as learning data to train an encoder included in the machine learning model; and 16. The method of claim 15, further comprising: (o) fixing the trained encoder, and training the decoder using the second range image, the second reflectivity image, and the second noise boundary image as training data, so that a reconstruction loss between an output of a decoder included in the machine learning model and the second noise boundary image is reduced.

17. A memory storing computer readable instructions; and at least one processor configured to execute the instructions; The at least one processor executes the instructions to: receiving a noisy first point cloud from a lidar sensor; generating a first distance image and a first reflectance image based on the first point cloud; inputting the first distance image and the first reflectance image into a pre-trained machine learning model to generate a first noise boundary image; A lidar sensor denoising system configured to remove noise contained in the first point cloud using the first noise boundary surface image.

18. The at least one processor:

18. The LIDAR sensor noise reduction system of claim 17, configured to estimate a reflectivity for each point included in the first point cloud using an impulse response of a single beam output of the LIDAR sensor, coordinates of the first point cloud, and adaptive parameters reflecting a backscattering effect, and generate the first reflectivity image based on the reflectivity.

19. The at least one processor:

20. The lidar sensor noise reduction system of claim 18, configured to increase reflectivity for points in the first point cloud that have negative z coordinates by applying a predetermined weighting value.

20. The at least one processor: generating a second distance image and a second reflectance image based on a second point cloud previously collected in which noise labels are assigned to the noise points; generating a third distance image by removing the noise points to which the noise labels are assigned from the second distance image; and generating a second noise boundary image by multiplying the third distance image by a constant equal to or less than 1; 20. The lidar sensor noise reduction system of claim 17, configured to train the machine learning model using the second range image, the second reflectivity image, and the second noise interface image as training data.

Images