Method and system for preprocessing point cloud data for autonomous vehicle

Abstract

Provided are a method and system for processing point cloud data for an autonomous vehicle. The point cloud data processing method according to an embodiment of the present invention: generates point cloud data; preprocesses the generated point cloud data; and converts the preprocessed point cloud data into structured data. Therefore, according to embodiments of the present invention, various Lidar instruments and new applications can be easily developed via a general-purpose and flexible point cloud data preprocessing system.

Description

Point cloud data preprocessing method and system for autonomous vehicles

[0001] The present invention relates to data preprocessing technology, and more particularly, to a method and system for preprocessing point cloud data acquired from a lidar sensor mounted on an autonomous vehicle.

[0002] In autonomous driving of existing vehicles and mobility robots, lidar sensors are widely used alongside cameras to ensure stable object detection and recognition. However, unlike camera data, lidar sensor data is random, with data locations varying each time they are acquired. Furthermore, valid data is scattered throughout the system. Processing with a processor (CPU, GPU) requires access to a large amount of memory and space for data processing, and processing 3D data requires significant computational resources.

[0003] Furthermore, autonomous driving requires the use of numerous deep learning models, which require extensive preprocessing steps such as calibration, downsampling, and voxelization to create the necessary point cloud data. If each application were to process these separately on the GPU or CPU, the system would become complex, resulting in excessive power consumption and memory usage.

[0004] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a lightweight embedded system capable of efficiently and quickly preprocessing lidar-based point cloud data required for stable object detection / recognition of an autonomous vehicle.

[0005] A method for processing point cloud data according to one embodiment of the present invention for achieving the above purpose includes: a step of generating point cloud data; a step of preprocessing the generated point cloud data; and a step of converting the preprocessed point cloud data into structured data.

[0006] The transformation step may be to project the preprocessed point cloud data from 3D space to 2D space and convert it into a 2D BEV (Bird Eye View) image.

[0007] The transformation step may be to convert the preprocessed point cloud data into a 2D grid-based feature map.

[0008] The transformation step may be to transform the preprocessed point cloud data into pillars.

[0009] It may be to compress the preprocessed point cloud data into a set voxel unit and convert it into a voxel map.

[0010] The preprocessing step may include a step of calibrating point cloud data to image data; a step of range filtering the calibrated point cloud data; and a step of randomly sampling the range filtered point cloud data.

[0011] The transformed structured data can be input into a 2D CNN and processed.

[0012] A 2D CNN can be a CNN for performing any one of object detection, object recognition, and image segmentation.

[0013] Point cloud data is lidar sensor-based point cloud data, and the preprocessing and conversion steps are performed within an embedded system, and the embedded system can be mounted on an autonomous vehicle.

[0014] According to another aspect of the present invention, a point cloud data processing system is provided, comprising: a sensor for generating point cloud data; a preprocessing unit for preprocessing point cloud data generated by the sensor; and a conversion unit for converting the point cloud data preprocessed by the preprocessing unit into structured data.

[0015] According to another aspect of the present invention, a method for preprocessing point cloud data is provided, characterized by including a step of preprocessing point cloud data; and a step of converting the preprocessed point cloud data into structured data.

[0016] According to another aspect of the present invention, a point cloud data preprocessing system is provided, characterized by including a preprocessing unit for preprocessing point cloud data; and a conversion unit for converting the preprocessed point cloud data into structured data.

[0017] As described above, according to embodiments of the present invention, it is possible to efficiently and quickly preprocess lidar-based point cloud data required for stable object detection / recognition in a lightweight embedded system mounted on an autonomous vehicle.

[0018] Additionally, according to embodiments of the present invention, the development of various lidar equipment and new applications is facilitated through a universal and flexible point cloud data preprocessing system.

[0019] Figure 1. Block diagram of a lidar-based point cloud system.

[0020] Figure 2. Preprocessing process of lidar-based point cloud data

[0021] Figure 3. Structured data conversion process of lidar-based point cloud data.

[0022] Figure 4. Data flow of a lidar-based point cloud deep learning application.

[0023] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0024] For autonomous driving of mobility vehicles, various computer vision technologies such as object classification, recognition, and image segmentation are required. Recently, these problems are being solved through network learning / inference using deep learning-based technologies.

[0025] Because lidar-based point cloud data possesses three-dimensional information compared to image data, a 3D CNN deep learning model is applied instead of a 2D CNN, resulting in increased computational load. Furthermore, point cloud data locations change randomly each time they are acquired, and the density of valid data is often scattered rather than dense.

[0026] In order to process this with a typical NPU computing device, data must be accessed randomly and a large amount of data must be stored compared to the actual valid data, which requires a lot of memory space.

[0027] To overcome these challenges of point cloud computation, numerous preprocessing steps are required. These include point data gathering / scattering, voxelization, random sampling, and range filtering.

[0028] In an embodiment of the present invention, an integrated system capable of efficiently processing such point cloud data is proposed, thereby effectively reducing the amount of data processing applied to NPU and GPU.

[0029] It is a high-efficiency, high-speed lidar data processing technology applicable to lightweight systems. It can be applied to a wide range of applications by adjusting the precision according to the type of lidar sensor, and can be built as an integrated preprocessing system that processes lidar-based point cloud input.

[0030] FIG. 1 is a diagram illustrating the configuration of a lidar-based point cloud data processing system according to an embodiment of the present invention. As illustrated, the lidar-based point cloud data processing system according to an embodiment of the present invention is configured to include a lidar sensor (110), a point cloud processor (120), an autonomous driving application (130), a data bus (140), a memory (150), and a computing device (160).

[0031] The lidar sensor (110) generates spatial information about the surroundings as point cloud data by shooting a laser pulse into the surrounding space and measuring the time it takes for it to be reflected and return, thereby measuring the position coordinates of the reflector.

[0032] The point cloud processor (120) performs necessary preprocessing on point cloud data generated by the lidar sensor (110) and then converts the preprocessed point cloud data into structured data.

[0033] The autonomous driving application (130) uses point cloud data output from the point cloud processor (120) to perform object detection, object recognition / classification, and image segmentation required for autonomous driving.

[0034] The memory (150) provides storage space required for the components that constitute the point cloud data processing system, and the computing device (160) controls the overall operation of the point cloud data processing system and executes the autonomous driving application (130).

[0035] The data bus (140) supports data communication between components that make up the point cloud data processing system.

[0036] Meanwhile, the structure and processing method of the NPU or GPU, which is a computing device (160), mainly consists of an image-centric 2D CNN structure. However, the 2D CNN structure is not suitable for processing 3D point clouds, and results in inefficient memory access and calculations.

[0037] Accordingly, the point cloud processor (120) functions to process 3D point cloud data while using the NPU structure as is without changing it, thereby enabling application processing utilizing lidar sensor-based point cloud data in an embedded system without changing the design of the NPU.

[0038] The detailed functions of the point cloud processor (120) that performs such functions are described in detail below. The point cloud processor (120) is composed of a lidar sensor preprocessor (121) and a structured data converter (122), and the functions of each component are shown in FIGS. 2 and 3, respectively.

[0039] Figure 2 is a diagram illustrating the functions of the lidar sensor preprocessor (121). Raw data from the lidar sensor (110) typically contains a large number of points due to its high noise content and coverage of data in areas wider than the required ROI range. Processing a large number of points requires a correspondingly large amount of computation, necessitating a process of filtering out unnecessary data. This process can significantly reduce the number of points.

[0040] To this end, the lidar sensor preprocessor (121) performs calibration between point cloud data and image data, that is, calibration between the lidar sensor (110) and a camera (not shown) that serves as the reference for the integrated coordinate system.

[0041] The following rider sensor preprocessor (121) performs range filtering on the calibrated point cloud data and random sampling on the range filtered point cloud data.

[0042] Figure 3 is a diagram illustrating the function of a structured data converter (122). 3D point cloud data has the characteristic of being randomly scattered, and by converting this data into a structured form, data processing can be efficiently performed at the back end of the system, i.e., the autonomous driving application (130).

[0043] To this end, the structured data converter (122) converts the point cloud data preprocessed by the rider sensor preprocessor (121) into structured data. Specifically, as illustrated in FIG. 3,

[0044] 1) Project the preprocessed point cloud data from 3D space to 2D space and convert it into a 2D BEV (Bird Eye View) image.

[0045] 2) Projecting the preprocessed point cloud data into 2D space through Pillar transformation and then converting it into a 2D grid-unit feature map, or

[0046] 3) The preprocessed point cloud data is compressed into a set voxel unit and converted into a voxel map.

[0047] Utilizing structured data, autonomous driving applications (130) can be implemented using a 2D CNN as a CNN backbone network system. The series of processes for this are illustrated in Figure 4.

[0048] FIG. 4 shows that point cloud data generated from a lidar sensor (110) is preprocessed in a lidar sensor preprocessor (121), then converted into a 2D BEV image, feature map, and voxel map in a structured data converter (122), and then processed by a 2D CNN backbone of an autonomous driving application (130) to be utilized for object detection, object recognition / classification, object information detection, and image segmentation.

[0049] So far, preferred embodiments of point cloud data preprocessing methods and systems for embedded systems mounted on autonomous vehicles have been described in detail.

[0050] In an embodiment of the present invention, a lidar sensor-based point cloud preprocessing system for a low-spec embedded system is proposed, and a system configuration that integrates point cloud preprocessing required for autonomous driving deep learning technology is presented.

[0051] Through a lightweight embedded system capable of efficiently and quickly preprocessing lidar-based point cloud data required for stable object detection / recognition of autonomous vehicles, efficient memory access and use, and high-speed processing that takes into account the unique data characteristics of point clouds, can be expected.

[0052] By streamlining and accelerating the point cloud preprocessing process, it can be applied to lightweight systems, and in particular, it can be used for various lidar equipment and new applications through the implementation of a universal and flexible data preprocessing device.

[0053] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0054] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. Step for generating point cloud data; Step of preprocessing the generated point cloud data; A method for processing point cloud data, comprising: a step of converting preprocessed point cloud data into structured data.

2. In claim 1, The conversion step is, A point cloud data processing method characterized by projecting preprocessed point cloud data from 3D space to 2D space and converting it into a 2D BEV (Bird Eye View) image.

3. In claim 1, The conversion step is, A point cloud data processing method characterized by converting preprocessed point cloud data into a 2D grid-unit feature map.

4. In claim 3, The conversion step is, A point cloud data processing method characterized by performing pillar transformation on preprocessed point cloud data.

5. In claim 1, A point cloud data processing method characterized by converting preprocessed point cloud data into a voxel map by compressing it into a set voxel unit.

6. In claim 1, The preprocessing step is, Step of calibrating point cloud data to image data; Step of Range Filtering the Calibrated Point Cloud Data; A point cloud data processing method, characterized by including a step of randomly sampling range-filtered point cloud data.

7. In claim 1, The converted structured data is, A method for processing point cloud data, characterized in that it is input and processed by a 2D CNN.

8. In claim 6, 2D CNN is, A method for processing point cloud data, characterized in that it is a CNN for performing any one of object detection, object recognition, and image segmentation.

9. In claim 7, Point cloud data is, It is a point cloud data based on a lidar sensor, The preprocessing and transformation steps are: It is performed within an embedded system, Embedded systems, A point cloud data processing method characterized by being mounted on an autonomous vehicle.

10. Sensors that generate point cloud data; A preprocessing unit that preprocesses point cloud data generated from a sensor; A point cloud data processing system, characterized by including a conversion unit that converts point cloud data preprocessed in a preprocessing unit into structured data.

11. Step of preprocessing point cloud data; and A point cloud data preprocessing method, characterized by including a step of converting preprocessed point cloud data into structured data.

12. A preprocessing unit that preprocesses point cloud data; and A point cloud data preprocessing system, characterized by including a conversion unit that converts preprocessed point cloud data into structured data.

Images