Hardware device and method for parallel pre-processing operation of 3D object detection algorithm

Abstract

A hardware device and a method for SoC real-time digital signal processing of a PointPillars algorithm are provided. The hardware device according to an embodiment of the present invention comprises: a parallel pre-processing processor for receiving data of a point cloud data format as an input and performing pre-processing operations of a PointPillars algorithm; and a memory for storing output data of the pre-processing processor.

Description

Hardware device and method for parallel preprocessing operations of 3D object detection algorithms

[0001] The present invention relates to a hardware device and method for parallel preprocessing operations of a 3D object detection algorithm, and more particularly, to a hardware device and method for SoC real-time digital signal processing of the PointPillars algorithm, which is one of the 3D object detection algorithms.

[0002] Recently, research is being conducted on 3D object detection algorithms using LiDAR sensors.

[0003] These 3D object detection algorithms are implemented in software and utilize the graphics processing unit (GPU) of a PC graphics card. The parallel architecture of the GPU allows for parallel processing of the repetitive processing of the many operations of the 3D object detection algorithm, enabling rapid 3D object detection.

[0004] If the core is used instead of the graphics card, the 3D object detection algorithm will perform all operations through sequential statements, which will result in slow operations and make real-time data signal processing impossible.

[0005] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a hardware device and method capable of processing (operating) digital signals in parallel for SoC real-time digital signal processing of the PointPillars algorithm.

[0006] According to one embodiment of the present invention for achieving the above purpose, a hardware device for parallel preprocessing operation of a 3D object detection algorithm includes a parallel preprocessing operator that receives data in the form of point cloud data as input and performs preprocessing operation of a point pillars algorithm; and a memory that stores output data of the preprocessing operator.

[0007] And the parallel preprocessing operator can extract multiple features using a 32-bit floating point operator.

[0008] Additionally, when multiple features are extracted, the parallel preprocessing operator can perform quantization of the extracted multiple features into 8-bit integers.

[0009] And the parallel preprocessing operator may include a standard deviation operator, a distance to center operator, and a quantization operator.

[0010] Additionally, multiple features may include data on each coordinate, standard deviation, and distance from the center.

[0011] And the distance calculator from the center can calculate the distance for each coordinate based on the center value of the pillar and produce a distance value from the center.

[0012] Additionally, the quantization operator can apply the output data of the distance operator to the center as input data.

[0013] And the quantization operator is composed of a plurality of FPUs (Floating Point Units), and the plurality of FPUs are divided into a first FPU group for performing a standard deviation operation, a second FPU group for performing a distance operation from a center, a third FPU group for performing a quantization operation, and a fourth FPU group for performing the operation variably, and one or more FPUs belonging to the fourth FPU group can variably support performing at least one operation among the standard deviation operation, the distance operation from a center, and the quantization operation depending on the operation resource status of each FPU group.

[0014] Additionally, the parallel preprocessing operator can repeatedly perform preprocessing operations until all assigned preprocessing operations are completed.

[0015] Meanwhile, a parallel preprocessing operation method of a 3D object detection algorithm according to another embodiment of the present invention includes a step in which a parallel preprocessing operator receives data in the form of point cloud data as input and performs a preprocessing operation of a point pillars algorithm; and a step in which output data of the preprocessing operator is stored in a memory.

[0016] And according to another embodiment of the present invention, a hardware device for parallel preprocessing operations of a 3D object detection algorithm includes: a parallel preprocessing operator that performs preprocessing operations of a PointPillars algorithm, repeatedly performing preprocessing operations until all assigned preprocessing operations are completed; and a memory that stores output data of the preprocessing operator each time a preprocessing operation is performed.

[0017] In addition, according to another embodiment of the present invention, a parallel preprocessing operation method of a 3D object detection algorithm includes a step in which a parallel preprocessing operator performs a preprocessing operation of a PointPillars algorithm; a step in which output data of the preprocessing operator is stored in a memory; and a step in which the parallel preprocessing operator determines whether all assigned preprocessing operations have been completed each time a preprocessing operation task is performed; and the parallel preprocessing operator repeatedly performs the step of performing the preprocessing operations until all assigned preprocessing operations have been completed.

[0018] As described above, according to embodiments of the present invention, by processing (operating) digital signals in parallel to improve signal processing speed, it is possible to contribute to performing digital signal processing of the PointPillars algorithm in real time.

[0019] FIG. 1 is a diagram provided for describing the configuration of a hardware device for parallel preprocessing operations of a 3D object detection algorithm according to one embodiment of the present invention.

[0020] Figure 2 is a drawing provided for the description of the preprocessing operation of the PointPillars algorithm.

[0021] FIG. 3 is a diagram provided for explaining input data and output data of a parallel preprocessing operator according to one embodiment of the present invention;

[0022] FIG. 4 is a drawing provided for the description of a standard deviation operator and a quantization operator according to one embodiment of the present invention;

[0023] FIG. 5 is a drawing provided for the description of a standard deviation calculator according to one embodiment of the present invention;

[0024] FIG. 6 is a drawing provided for the description of a distance calculator and a quantization calculator according to one embodiment of the present invention;

[0025] Figure 7 is a drawing provided for the description of a distance calculator with a center according to one embodiment of the present invention;

[0026] FIG. 8 is a drawing provided to explain the pipeline structure of a parallel preprocessing operator according to one embodiment of the present invention.

[0027] Figure 9 is a drawing illustrating the appearance of a parallel preprocessing operator without FPU minimization.

[0028] FIG. 10 is a drawing illustrating an FPU-minimized parallel preprocessing operator according to one embodiment of the present invention; and

[0029] Figure 11 is a flowchart provided to explain a parallel preprocessing operation method using a parallel preprocessing operator according to one embodiment of the present invention.

[0030]

[0031] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0032] FIG. 1 is a diagram provided for explaining the configuration of a hardware device for parallel preprocessing operations of a 3D object detection algorithm according to one embodiment of the present invention, FIG. 2 is a diagram provided for explaining the preprocessing operations of a PointPillars algorithm, and FIG. 3 is a diagram provided for explaining input data and output data of a parallel preprocessing operator (110) according to one embodiment of the present invention.

[0033] A hardware device (hereinafter collectively referred to as a “hardware device”) for parallel preprocessing operations of a 3D object detection algorithm according to the present embodiment may include a parallel preprocessing operator (110) and a memory (120) for storing output data of the preprocessing operator in order to process the preprocessing operations of the PointPillars algorithm in parallel.

[0034] The parallel preprocessing operator (110) may mean the preprocessing operation of the PointPillars algorithm from the Point cloud stage to the Stacked Pillars stage.

[0035] The Stacked Pillars step is a procedure for accumulating data that structures a three-dimensional space into two-dimensional grid columns. Each column is vertically aligned, and the data within the column can represent coordinate information including height information of point clouds.

[0036] At the Stacked Pillars stage, a [D, P, N] tensor can be output, where D represents the dimension, P represents the numbering of the generated Pillars, and N represents the number of points within each pillar.

[0037] The number of points in a pillar is limited to 32. If there are more points, 32 points can be randomly extracted and used. Zero-padding can be used if the number of points in a pillar is less than 32. In other words, zero-padding means filling empty spaces with zero data.

[0038] And here, Dimension is 10-dimensional data (x, y, z, r, xc, yc, zc, xp, yp, zp), r represents reflectance rate, c represents the distance from the center coordinate of the point within the pillar, and p represents the distance from the center coordinate of the pillar.

[0039] The parallel preprocessing operator (110) is implemented with a parallel structure by having multiple FPUs (Floating Point Units) provided inside, and is designed to have a parallel pipeline structure, so that it can receive data in the form of point cloud data as input and perform preprocessing operations of the PointPillars algorithm in parallel.

[0040] Specifically, the parallel preprocessing operator (110) can extract multiple features using a 32-bit floating point operator, and when multiple features are extracted, the extracted multiple features (e.g., 10) can be quantized into 8-bit integers.

[0041] That is, the format of the output data of the preprocessing operator can be an 8-bit integer that is output after quantization.

[0042] And at this time, multiple features may include data on each coordinate, standard deviation, and distance from the center.

[0043] FIG. 4 is a drawing provided for the explanation of a standard deviation operator (111) and a quantization operator (113) according to one embodiment of the present invention, and FIG. 5 is a drawing provided for the explanation of a standard deviation operator (111) according to one embodiment of the present invention.

[0044] The parallel preprocessing operator (110) may include a standard deviation operator (111), a distance operator (112) to the center, and a quantization operator (113) to perform preprocessing operations of the PointPillars algorithm in parallel by receiving data in the form of Point Cloud Data as input as described above.

[0045] The parallel preprocessing operator (110) can obtain data for each coordinate and standard deviation by using a standard deviation operator (111) and multiple quantization operators (113).

[0046] Specifically, the parallel preprocessing operator (110) uses a 32-bit Floating Point Unit, inputs input data to a standard deviation operator (111) for standard deviation operation, and applies the output data of the standard deviation operator (111) to a first quantization operator (113) to obtain data on the standard deviation as output data in the form of an 8-bit integer.

[0047] At this time, the standard deviation calculator (111) can be designed in a way that calculates the average value of each coordinate and performs subtraction.

[0048] In addition, the parallel preprocessing operator (110) can obtain data for each coordinate as output data in the form of an 8-bit integer by applying input data to a second quantization operator (113) connected in parallel with the first quantization operator (113).

[0049] Here, when the input data of the parallel preprocessing operator (110) is X, Y, and Z, the data for each coordinate and standard deviation obtained as output data by the parallel preprocessing operator (110) can be expressed as XC, YC, and ZC.

[0050] That is, when X1 is input as the first input data, Y1 is input as the second input data, and Z1 is input as the third input data, the parallel preprocessing operator (110) can obtain XC1, YC1, and ZC1, which include data for each coordinate and standard deviation, as output data.

[0051] FIG. 6 is a drawing provided for the explanation of a distance calculator (112) and a quantization calculator (113) from a center according to one embodiment of the present invention, and FIG. 7 is a drawing provided for the explanation of a distance calculator (112) from a center according to one embodiment of the present invention.

[0052] The distance calculator (112) from the center can calculate the distance for each coordinate based on the center value of the pillar and produce a distance value from the center.

[0053] Specifically, the distance calculator (112) from the center can calculate the center value of the Pillar for each coordinate to produce a value, and can output each coordinate value after a subtraction operation from the center value of the Pillar for each coordinate. Here, X1, Y1, and Z1 represent data of 32 points of the first Pillar, and X2, Y2, and Z2 represent 32 points of the second Pillar.

[0054] The quantization operator (113) can perform quantization by applying the output data of the distance operator (112) to the center as input data.

[0055] When the input data of the parallel preprocessing operator (110) are X, Y, and Z, the data on the distance from the center of each coordinate obtained as output data by the parallel preprocessing operator (110) can be expressed as XP, YP, and ZP.

[0056] FIG. 8 is a drawing provided to explain the pipeline structure of a parallel preprocessing operator (110) according to one embodiment of the present invention, FIG. 9 is a drawing exemplifying the appearance of a parallel preprocessing operator (110) without FPU minimization, and FIG. 10 is a drawing exemplifying the appearance of a parallel preprocessing operator (110) with FPU minimization according to one embodiment of the present invention.

[0057] In Fig. 8, operation 1 is a standard deviation operator (111), operation 2 is a distance operator from the center (112), and operation 3 is a quantization operator (113).

[0058] In order to make it a little easier to understand the parallel preprocessing operator (110) having a pipeline structure according to the present embodiment, when X1, Y1, Z1, X2, and Y2 are sequentially used as input data, Y0, Z0, X1, Y1, and Z1 can be obtained as output data.

[0059] A parallel preprocessing operator (110) designed with such a pipeline structure can basically perform one of the four basic arithmetic operations through multiple FPUs provided inside.

[0060] At this time, the parallel preprocessing operator (110) is configured so that the multiple FPUs provided inside perform only the operations assigned to them among the standard deviation operation, the distance operation from the center, and the quantization operation in order to extract features as illustrated in FIG. 9. In this case, approximately 250 to 300 FPUs are required.

[0061] However, the parallel preprocessing operator (110) according to the present embodiment can reduce the number of FPUs required to extract features by variably adjusting some of the multiple FPUs provided internally.

[0062] For example, a plurality of FPUs provided inside the quantization operator (113) can be divided into a first FPU group whose role is fixed to perform a standard deviation operation, a second FPU group whose role is fixed to perform a distance operation from the center, a third FPU group whose role is fixed to perform a quantization operation, and a fourth FPU group whose role is variably adjusted to perform a variably operation.

[0063] That is, one or more FPUs belonging to the 4th FPU group can variably support performing at least one operation among a standard deviation operation, a distance operation from the center, and a quantization operation, depending on the operational resource status of each FPU group.

[0064] At this time, if one FPU belonging to the 4th FPU group supports the standard deviation operation, another FPU can support the distance operation from the center, and as the operation resource status of each FPU group changes, the FPU that supported the standard deviation operation can be used to support the distance operation from the center.

[0065] The parallel preprocessing operator (110) in which the number of FPUs is reduced in the above manner can output 10-dimensional data, namely X1, XC1, XP1, Y1, YC1, YP1, Z1, ZC1, and ZP1, as output data when X1 is input as the first input data, Y1 is input as the second input data, and Z1 is input as the third input data, as illustrated in FIG. 10.

[0066] At this time, XC1, YC1, and ZC1 may include data on each coordinate and standard deviation, and XP1, YP1, and ZP1 may include data on distance information from the center distance of each coordinate.

[0067] And, the parallel preprocessing operator (110) with the number of FPUs reduced in the above manner can repeatedly perform preprocessing operation tasks until all assigned preprocessing operations are completed.

[0068] FIG. 11 is a flowchart provided to explain a parallel preprocessing operation method using a parallel preprocessing operator (110) according to one embodiment of the present invention.

[0069] Referring to FIG. 11, a parallel preprocessing operator (110) with a reduced number of FPUs performs a preprocessing operation of a PointPillars algorithm (S1110), stores the output data of the preprocessing operator in a memory (120) each time a preprocessing operation task is performed (S1120), determines whether all assigned preprocessing operations are completed (S1130), and terminates the task when the assigned preprocessing operations are completed (S1130-Yes), and performs the preprocessing operation task again when the assigned preprocessing operations are not completed (S1130-No).

[0070] Through this, the parallel preprocessing operator (110) with a reduced number of FPUs can repeatedly perform preprocessing operations until all assigned preprocessing operations are completed.

[0071] Here, the parallel preprocessing operator (110) with a reduced number of FPUs can determine whether all preprocessing operations have been completed for one-shot (lidar) sensor data in which one sample data exists for each class.

[0072] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0073] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. A parallel preprocessing operator that receives data in the form of Point Cloud Data as input and performs preprocessing operations of the Point Pillars algorithm; and A hardware device for parallel preprocessing operations of a 3D object detection algorithm, comprising a memory for storing output data of a preprocessing operator.

2. In claim 1, Parallel preprocessing operators, A hardware device for parallel preprocessing operations of a 3D object detection algorithm characterized by extracting multiple features using a 32-bit floating point operator.

3. In claim 2, Parallel preprocessing operators, A hardware device for parallel preprocessing operations of a 3D object detection algorithm, characterized in that when multiple features are extracted, quantization is performed on the extracted multiple features into 8-bit integers.

4. In claim 3, Parallel preprocessing operators, A hardware device for parallel preprocessing operations of a 3D object detection algorithm, characterized by including a standard deviation operator, a distance to center operator, and a quantization operator.

5. In claim 4, Multiple features include: A hardware device for parallel preprocessing operations of a 3D object detection algorithm, characterized in that it includes data on each coordinate, standard deviation, and distance from the center.

6. In claim 4, The distance calculator from the center is, A hardware device for parallel preprocessing operations of a 3D object detection algorithm characterized by calculating the distance for each coordinate based on the center value of the pillar and calculating the distance value from the center.

7. In claim 4, The quantization operator is, A hardware device for parallel preprocessing operations of a 3D object detection algorithm characterized by applying output data of a distance calculator to the center as input data.

8. In claim 4, The quantization operator is, It consists of multiple FPUs (Floating Point Units), Multiple FPUs, It is divided into a first FPU group for performing standard deviation operations, a second FPU group for performing distance operations from the center, a third FPU group for performing quantization operations, and a fourth FPU group for performing operations variably. One or more FPUs belonging to the 4th FPU group, A hardware device for parallel preprocessing operations of a 3D object detection algorithm, characterized in that it variably supports performing at least one operation among a standard deviation operation, a distance operation from a center operation, and a quantization operation depending on the operational resource status of each FPU group.

9. In claim 1, Parallel preprocessing operators, A hardware device for parallel preprocessing operations of a 3D object detection algorithm, characterized in that preprocessing operations are repeatedly performed until all assigned preprocessing operations are completed.

10. A step in which a parallel preprocessing operator receives data in the form of point cloud data as input and performs a preprocessing operation of the PointPillars algorithm; and A parallel preprocessing operation method of a 3D object detection algorithm, comprising a step of storing output data of a preprocessing operator in memory.

11. A parallel preprocessing operator that performs preprocessing operations of the PointPillars algorithm, repeatedly performing preprocessing operations until all assigned preprocessing operations are completed; and A hardware device for parallel preprocessing operations of a 3D object detection algorithm, comprising a memory for storing output data of a preprocessing operator whenever a preprocessing operation is performed.

12. A step in which a parallel preprocessing operator performs a preprocessing operation of the PointPillars algorithm; A step in which the output data of the preprocessing operator is stored in memory; and The parallel preprocessing operator includes a step of determining whether all assigned preprocessing operations have been completed each time a preprocessing operation task is performed; Parallel preprocessing operators, A parallel preprocessing operation method for a 3D object detection algorithm, characterized in that a step of performing a preprocessing operation is repeatedly performed until all assigned preprocessing operations are completed.

Images