Method for accelerating deep learning preprocessing of three-dimensional point cloud data

Abstract

A method for accelerating deep learning preprocessing of three-dimensional point cloud data is provided. The method for deep learning preprocessing of three-dimensional point cloud data, according to an embodiment of the present invention, comprises: acquiring three-dimensional point cloud data; classifying three-dimensional point cloud data with common spatial locality into different spaces with respect to the acquired three-dimensional point cloud data; and preprocessing the classified three-dimensional point cloud data in parallel. Therefore, deep learning preprocessing can be performed at high speed even with limited hardware resources in vehicle autonomous driving, an edge end device, and the like by performing deep learning preprocessing acceleration through efficient parallelization of three-dimensional point cloud data acquired from a LiDAR device.

Description

Accelerated preprocessing of 3D point cloud data using deep learning

[0001] The present invention relates to image processing and system SoC (System on Chip), and more particularly, to a method for accelerating deep learning inference preprocessing using 3D point cloud data in a system utilizing a LiDAR (Light Detection And Ranging) device.

[0002] Many deep learning inference techniques using 3D point cloud data obtained using LiDAR devices have been proposed for use in fields such as object classification, object detection, and image segmentation.

[0003] These deep learning inference methods include methods that directly process 3D point cloud data for each point and methods that divide the 3D space into grids and process them. The method that directly processes each point requires a significant amount of processing, processing time, and massive hardware resources, while the method that divides the grid and processes it requires relatively little processing, short processing time, and small hardware resources.

[0004] Therefore, grid-based processing is considered a practical solution for real-time utilization in autonomous vehicles and small edge devices. However, even with this approach, conventional deep learning inference preprocessing methods for the large volume of 3D point cloud data acquired from LiDAR do not efficiently parallelize, posing significant limitations in achieving high-speed, real-time processing with limited hardware resources (especially in small edge devices).

[0005] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method for accelerating deep learning preprocessing through efficient parallelization of 3D point cloud data acquired from a lidar device, as a means for enabling high-speed deep learning preprocessing even with limited hardware resources in autonomous vehicle driving, edge devices, etc.

[0006] A 3D cloud data preprocessing method according to one embodiment of the present invention for achieving the above object includes the steps of: acquiring 3D point cloud data; classifying 3D point cloud data having common spatial locality among the acquired 3D point cloud data into different spaces; and preprocessing the classified 3D cloud data in parallel.

[0007] The classification step may be intended to reduce the comparison target of 3D cloud data in the preprocessing step.

[0008] The classification step may be performed on a plane unit of a specific direction for the 3D point cloud data.

[0009] The classification step may be to classify the 3D point cloud data into a different space from other adjacent 3D point cloud data on the plane.

[0010] The classification step may be to regularly classify 3D point cloud data on a plane.

[0011] The classification step may be to classify 3D point cloud data on a plane based on a uniform pattern.

[0012] The classification step may be to classify 3D point cloud data of {even, even} coordinates into a first space, 3D point cloud data of {odd, even} coordinates into a second space, 3D point cloud data of {even, odd} coordinates into a third space, and 3D point cloud data of {odd, odd} coordinates into a fourth space.

[0013] The 3D cloud data preprocessing method according to the present invention further includes a step of dividing the acquired 3D point cloud data into pillar units; and the classification step may be to classify 3D point cloud data having a common spatial locality with respect to the 3D point cloud data divided in the division step into different spaces.

[0014] Preprocessing can be preprocessing for deep learning inference.

[0015] According to another aspect of the present invention, a computing device is provided, comprising: a memory for acquiring 3D point cloud data; a preprocessing unit for classifying 3D point cloud data having common spatial locality among 3D point cloud data stored in the memory into different spaces, and preprocessing the classified 3D cloud data in parallel.

[0016] According to another aspect of the present invention, a deep learning operation method is provided, characterized by including the steps of: classifying 3D point cloud data having common spatial locality into different spaces; preprocessing the classified 3D cloud data in parallel; and performing a deep learning operation using the preprocessed 3D cloud data as input.

[0017] According to another aspect of the present invention, a deep learning operation device is provided, characterized by including: a preprocessing unit that classifies 3D point cloud data having a common spatial locality into different spaces for 3D point cloud data and preprocesses the classified 3D cloud data in parallel; and a deep learning accelerator that performs a deep learning operation using the preprocessed 3D cloud data as input.

[0018] As described above, according to embodiments of the present invention, deep learning preprocessing is accelerated through efficient parallelization of 3D point cloud data acquired from a lidar device, thereby enabling high-speed deep learning preprocessing even with limited hardware resources in autonomous vehicle driving, edge devices, etc.

[0019] Figure 1. Example of a lidar device from Velodyne, a leading company in the lidar equipment industry.

[0020] Figure 2. Example of urbanization of 3D point cloud data acquired from a lidar device.

[0021] Figure 3. Example of a Pillar-based method, one of the deep learning processing methods for 3D point cloud data.

[0022] Figure 4. Example of Bayer filter pattern of image sensor

[0023] Figure 5. Based on the Bayer pattern structure, the XY plane grid is divided into even / odd coordinates for each X / Y coordinate, and an example of regular division into Bayer0 (Red), Bayer1 (Green), Bayer2 (Yellow), and Bayer3 (Blue)

[0024] Figure 6. A method and structure in which four independent operations are performed in parallel by arranging memory (SLAM) for each Bayer allocation space and comparing and processing only in the same space.

[0025] Figure 7 is a configuration of a deep learning operation device according to another embodiment of the present invention.

[0026] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0027] From autonomous vehicles to small edge devices like street delivery robots and drones, the processing of 3D point cloud data using LiDAR has significant utility in various fields, such as mapping terrain or recognizing objects on roads and sidewalks using expensive, high-resolution and inexpensive, low-resolution LiDAR devices (Fig. 1). In particular, autonomous driving is directly related to human life, not only to the driver but also to pedestrians and cyclists on the road. Therefore, processing must be performed very quickly and in real time, even at high vehicle speeds, to ensure safety.

[0028] The 3D point cloud data obtained through these LiDAR devices is not image-based data obtained through image sensors, but rather is a composite of individual laser points fired from the LiDAR, striking objects, and returning, forming a cloud (Figure 2). While this urbanized result doesn't reveal the sharp and distinct shape or background of an object like traditional image-based results, it does allow for a vague understanding of the object's shape.

[0029] Deep learning technology can be used on this 3D point cloud data for applications such as object classification, object recognition, and image segmentation. A practical approach for real-world applications is to divide the 3D space into pillars (vertical columns), as shown in Figure 3, and process each pillar individually. This approach is efficient in terms of processing capacity, processing speed, and required hardware resources.

[0030] This Pillar-based deep learning preprocessing method sequentially calculates Pillar X / Y coordinates for each 3D point, assigns the corresponding Pillar, and updates the data by comparing it with the point data of the previously assigned Pillar. However, this method means that as the number of objects to be compared increases, the next point data must wait for the processing time taken by one point, which slows down the processing speed. This is critical for autonomous driving applications that require real-time processing in moving vehicles. Therefore, efficient parallel processing is essential.

[0031] 3D point cloud data inherently possesses the characteristic of spatial locality. As shown in Figure 2, laser points emitted from a LiDAR device strike objects and return to the LiDAR, providing a vague image of the object's shape. In other words, point data that strikes an object and returns are often spatially and temporally clustered with points located around the object.

[0032] Point data sequentially received from a lidar device (or accessed via external memory such as DRAM) are likely to be located in similar spatial locations. Leveraging this advantage to effectively parallelize data can significantly increase processing speed.

[0033] In order to utilize the spatial locality characteristics of such 3D point cloud data, a parallel processing method and structure inspired by the Bayer filter pattern commonly used in the image sensor of Fig. 4 are proposed.

[0034] FIG. 5 is a diagram illustrating a data classification method for a 3D point cloud data deep learning preprocessing method according to one embodiment of the present invention.

[0035] First, the 3D point cloud data acquired through the lidar device is divided into Pillar units, and then, as shown in Fig. 5, the Pillar column is formed, which divides the XY plane Grid into even / odd coordinates for the X / Y coordinates, respectively. The coordinates of {X_even, Y_even} are classified into the red Bayer 0 space, the coordinates of {X_odd, Y_even} are classified into the green Bayer 1 space, the coordinates of {X_even, Y_odd} are classified into the yellow Bayer 2 space, and finally, the coordinates corresponding to {X_odd, Y_odd} are classified into the blue Bayer 3 space.

[0036] This classification aims to reduce the number of 3D cloud data comparison targets during the deep learning preprocessing stage. It categorizes 3D point cloud data into a different space than adjacent 3D point cloud data on a plane. In other words, 3D point cloud data with common spatial locality are classified into different spaces.

[0037] The above classification method involves dividing 3D point cloud data on a plane into four independent spaces based on a uniform pattern. However, it is not necessarily limited to the order described above. Furthermore, it is not limited to four divisions, and can include more divisions.

[0038] Deep learning preprocessing is then performed in parallel on the classified 3D cloud data. In this case, processing is performed on only a quarter of the data allocated to it, eliminating the need to compare and process the entire XY plane grid. This significantly improves single-point processing speed. Furthermore, parallel processing is possible for four partitioned areas simultaneously, further enhancing the effectiveness.

[0039] In addition, as shown in Fig. 6, when the example of the entire XY plane Grid coordinates from 0 to 9 is divided into each Bayer area, each can be independently expressed as a value from 0 to 4, so the amount of data required to express it is also reduced. This reduces the memory capacity by this amount compared to the conventional sequential processing of the entire XY plane Grid range.

[0040] Each Bayer region has a memory called SLAM (Spatial Locality-Aware Memory) that utilizes the spatial locality characteristics of 3D point clouds, and appropriately divides the memory space for the entire XY plane grid range. There is also a function called SLAM MU, and SLAM is also allocated here, and it plays a role in supplementing the overflow portion of the memory of the four-part region and buffering and managing it.

[0041] FIG. 7 is a diagram illustrating the configuration of a deep learning computing device according to another embodiment of the present invention. As illustrated, the deep learning computing device according to the embodiment of the present invention is configured to include a memory (110), an MCU (120), a preprocessing unit (130), and a deep learning accelerator (140).

[0042] The MCU (120) receives 3D point cloud data acquired by the lidar device from an external memory and stores it in the memory (110). The preprocessing unit (130) performs pillar segmentation and classification according to the Bayer filter pattern on the 3D point cloud data stored in the memory (110), and then performs preprocessing for deep learning inference.

[0043] The deep learning accelerator (140) inputs 3D point cloud data preprocessed by the preprocessing unit (130) into the deep learning network to perform inference (object recognition, object classification, etc.).

[0044] So far, we have described in detail preferred embodiments of a method for accelerating deep learning preprocessing of 3D point cloud data.

[0045] In the above embodiment, a method and structure are presented that enable processing at a significantly improved speed through parallel processing that utilizes the characteristics of spatial locality of 3D point cloud data while using the same or even slightly smaller memory capacity than that used in conventional sequential processing.

[0046] Specifically, it is an efficient parallel processing technology that utilizes the characteristics of 3D point cloud data that can be acquired through a lidar device, and it is possible to improve the processing speed through effective parallel processing that utilizes the characteristics of 3D point cloud data, enabling real-time processing application in the field of autonomous driving vehicles that put human lives at risk, and it is possible to efficiently implement it in a small edge device with the same memory capacity as the conventional sequential processing method.

[0047] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0048] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

Step 1: Acquire 3D point cloud data; A step of classifying 3D point cloud data having common spatial locality into different spaces for the acquired 3D point cloud data; A three-dimensional cloud data preprocessing method, characterized by including a step of parallel preprocessing of classified three-dimensional cloud data.

2. In claim 1, The classification step is, A three-dimensional cloud data preprocessing method characterized by reducing the comparison target of three-dimensional cloud data in the preprocessing stage.

3. In claim 1, The classification step is, A three-dimensional cloud data preprocessing method characterized in that it is performed on a plane unit in a specific direction for three-dimensional point cloud data.

4. In claim 3, The classification step is, A 3D cloud data preprocessing method characterized by classifying 3D point cloud data into a different space from other adjacent 3D point cloud data on a plane.

5. In claim 4, The classification step is, A three-dimensional cloud data preprocessing method characterized by regularly classifying three-dimensional point cloud data on a plane.

6. In claim 5, The classification step is, A 3D cloud data preprocessing method characterized by classifying 3D point cloud data on a plane based on a uniform pattern.

7. In claim 6, The classification step is, 3D point cloud data of {even, even} coordinates are classified into the first space, 3D point cloud data of {odd, even} coordinates are classified into the second space. 3D point cloud data of {even, odd} coordinates are classified into the third space, A 3D cloud data preprocessing method characterized in that 3D point cloud data of {odd, odd} coordinates are classified into a fourth space.

8. In claim 1, A step of dividing the acquired 3D point cloud data into pillar units is further included; The classification step is, A 3D cloud data preprocessing method characterized by classifying 3D point cloud data having common spatial locality into different spaces in the 3D point cloud data divided in the 3D point cloud data division step.

9. In claim 1, Preprocessing is, A three-dimensional cloud data preprocessing method characterized as being a preprocessing for deep learning inference. Memory for acquiring 10.3D point cloud data; A computing device characterized by including a preprocessing unit that classifies 3D point cloud data having common spatial locality among 3D point cloud data stored in memory into different spaces and preprocesses the classified 3D cloud data in parallel.

11. A step of classifying 3D point cloud data having common spatial locality into different spaces for 3D point cloud data; A step of parallel preprocessing of classified 3D cloud data; A deep learning operation method, characterized by including a step of performing a deep learning operation using preprocessed three-dimensional cloud data as input. 12.3 A preprocessing unit that classifies 3D point cloud data with common spatial locality into different spaces and preprocesses the classified 3D cloud data in parallel; A deep learning operation device characterized by including a deep learning accelerator that performs deep learning operation using preprocessed three-dimensional cloud data as input.

Images