Hardware architecture, computing method and device for 3D point cloud neural network algorithm

Abstract

The application relates to the technical field of point cloud neural network, in particular to a hardware architecture, a calculation method and equipment for a 3D point cloud neural network algorithm, wherein the hardware architecture comprises: off-chip storage; a mapping module, the mapping module is provided with distance filtering technology and / or output priority mapping calculation technology to reduce the off-chip memory access amount of the off-chip storage when calculating the mapping operation of the 3D point cloud neural network algorithm, and a mapping relationship between input and output is generated according to the mapping operation; and a calculation module, the calculation module is provided with an elastic array architecture, the elastic array architecture is adjusted according to different scales of calculation tasks in the 3D point cloud neural network algorithm, and the weight and the corresponding input feature are taken out according to the mapping relationship to perform matrix operation to obtain the output feature. Thus, the problems in the prior art that the off-chip memory access amount of the point cloud neural network accelerator is large, the utilization rate of the calculation unit is low, the processing speed of the accelerator is slow, the expansibility and flexibility are poor, and the actual needs cannot be met are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud neural network, and particularly relates to a hardware architecture, a computing method and a device for a 3D point cloud neural network algorithm. BACKGROUND

[0002] With the continuous development of 3D point cloud neural network, the 3D point cloud neural network has become an important method for processing sparse 3D point cloud data in the fields of automatic driving, robots, virtual reality and the like. In the related art, point cloud neural network algorithms can be roughly divided into two categories: point-based and voxel-based, and the basic operations can be abstracted into two categories: mapping operation and matrix operation.

[0003] The mapping operation constructs the relationship between the sparse input data and the output data. However, in the farthest point sampling of the point-based network, sampling one point in each iteration needs to access all input points and perform distance calculation, which has a large amount of redundant memory access; in the voxel-based network, when calculating the sparse convolution mapping relationship, all input voxels need to be repeatedly accessed multiple times, which also has a large amount of redundant off-chip memory access. The sparse convolution software acceleration library in the related art is only applicable to voxel-based networks and cannot effectively accelerate other types of networks, and the acceleration performance is limited; for matrix operation, the related art point cloud neural network accelerator has poor scalability and low utilization of computing units. SUMMARY

[0004] The present application provides a hardware architecture, a computing method and an electronic device for a 3D point cloud neural network algorithm to solve the problems in the related art that the off-chip memory access amount of the point cloud neural network accelerator is large, the utilization rate of the computing unit is low, which leads to slow processing speed of the accelerator, poor scalability and flexibility, and thus cannot meet the actual use needs.

[0005] The first aspect of the present application provides a hardware architecture for a 3D point cloud neural network algorithm, comprising an off-chip memory; a mapping module, the mapping module is provided with a distance filtering technology and / or an output priority mapping calculation technology to reduce the off-chip memory access amount of the off-chip memory during the mapping operation of the 3D point cloud neural network algorithm, and generate a mapping relationship between the input and the output according to the mapping operation; a computing module, the computing module is provided with an elastic array architecture, the elastic array architecture is adjusted according to the computing task size in the 3D point cloud neural network algorithm, and the weight and the corresponding input feature are taken out according to the mapping relationship to perform matrix operation to obtain the output feature.

[0006] Optionally, the mapping module comprises a distance calculation unit, a sorting unit, a maximum calculation unit, a merging unit, a splicing unit, a coincidence detection unit, a grid filtering unit, a loading unit, a grid cache, a sorting unit cache, a merging unit cache and a mapping relationship cache, wherein the sorting unit, the merging unit and the splicing unit jointly constitute a unit for implementing arbitrary-length data sorting, the coincidence detection unit is used for detecting a coincidence position in sparse convolution mapping calculation, and when the convolution kernel has symmetry and a mapping of a non-convolution kernel center position is detected, an opposite mapping is generated according to the current mapping; the grid filtering unit is used for calculating the shortest distance between a center point and a grid and filtering out a grid with a distance greater than a filtering threshold; the grid cache is used for storing information of a grid for grouping when a point-based network is processed, and storing coordinates of a voxel when a voxel-based network is processed; the sorting unit cache and the merging unit cache are respectively used for storing data to be processed by the sorting unit and the merging unit; and the mapping relationship cache is used for storing a generated mapping relationship table.

[0007] Optionally, the 3D point cloud neural network algorithm comprises a point-based network and a voxel-based network, wherein the mapping module filters off-chip data access that does not meet an access condition by using a distance filtering technology to reduce off-chip memory access of the off-chip storage; and the mapping module multiplexes on-chip data by using an output-priority mapping calculation technology to reduce off-chip memory access of the off-chip storage.

[0008] Optionally, the distance filtering technology is a technology for limiting off-chip data access that does not meet an access condition in a mapping operation by using a geometric distance of a point.

[0009] Optionally, the distance filtering technology comprises: dividing points in an input point cloud into grids according to coordinates and sorting the points according to grid positions; and filtering, at each iteration sampling, a grid with a distance greater than a filtering threshold from a grid in which a sampling point is located.

[0010] Optionally, the dividing points in an input point cloud into grids according to coordinates and sorting the points according to grid positions comprises: dividing a target space into a plurality of grids according to a grid size set in advance; and sorting all points according to coordinates of a grid in which each point in a point cloud is located and writing the points back to the off-chip storage, and retaining, in an on-chip cache, a position index of a non-empty grid and a minimum serial number of a point in the grid.

[0011] Optionally, the output-priority mapping calculation technology is a technology for multiplexing on-chip data by cutting an output voxel into the on-chip cache.

[0012] Optionally, the output priority mapping calculation technology comprises: sorting the voxel data in the off-chip storage in a coordinate order; multiplexing the voxels at the edges of the convolution kernel, and calculating the mapping of the center and one side of the convolution kernel according to the symmetry of the convolution kernel, and obtaining the mapping of the other side by inverting the mapping relationship according to the symmetry; and writing the mapping result into different addresses of the cache according to the input point cloud weight.

[0013] Optionally, the elastic array architecture comprises a plurality of sub-pulsating arrays, a multiplexer between the sub-pulsating arrays, an input feature on-chip cache, an output feature on-chip cache and a weight on-chip cache, and the multiplexer is used to adjust the topological relationship of the plurality of sub-pulsating arrays according to the size of the calculation task in the 3D point cloud neural network algorithm.

[0014] The second aspect embodiment of the present application provides a calculation method of the hardware architecture for the 3D point cloud neural network algorithm, wherein the method is calculated by using the hardware architecture for the 3D point cloud neural network algorithm as described in the above embodiments, and the method comprises the following steps: the mapping module is provided with a distance filtering technology and / or an output priority mapping calculation technology to reduce the off-chip memory access amount of the off-chip storage when calculating the mapping operation of the 3D point cloud neural network algorithm, and the mapping relationship between the input and the output is generated according to the mapping operation; the calculation module is provided with an elastic array architecture, the elastic array architecture is dynamically adjusted according to the size of the calculation task in the 3D point cloud neural network algorithm, and the weight and the corresponding input feature are taken out according to the mapping relationship to perform matrix operation to obtain the output feature.

[0015] The third aspect embodiment of the present application provides an electronic device comprising the hardware architecture for the 3D point cloud neural network algorithm as described in the above embodiments.

[0016] Therefore, the present application has at least the following beneficial effects:

[0017] The embodiments of the present application can use the distance filtering technology for the point-based network to filter out unnecessary off-chip data access, thereby reducing the off-chip memory access amount; at the same time, the output priority mapping calculation technology is used for the voxel-based network to increase the on-chip data multiplexing and improve the data multiplexing rate, thereby reducing the off-chip memory access amount; and the elastic array architecture can be used to flexibly adapt to different sizes of calculation tasks in the point cloud network by different sub-array combinations, so that the utilization rate of the calculation unit is still high when the size of the calculation unit increases, thereby improving the flexibility and scalability of the accelerator and meeting the actual use needs. Therefore, the technical problems in the related art that the off-chip memory access amount of the point cloud neural network accelerator is large, the utilization rate of the calculation unit is low, the processing speed of the accelerator is slow, the scalability and flexibility are poor, and the actual use needs cannot be met are solved.

[0018] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the references to the figures, in which:

[0020] Figure 1 An example diagram of a core mapping operator in a point cloud neural network algorithm of related technologies of embodiments of the present application;

[0021] Figure 2 A schematic diagram of an output-first mapping computation method of embodiments of the present application;

[0022] Figure 3 A schematic diagram of utilization of a systolic array computation unit of related technologies of embodiments of the present application;

[0023] Figure 4 A hardware architecture diagram for a 3D point cloud neural network algorithm of embodiments of the present application;

[0024] Figure 5 A schematic diagram of an efficient point cloud neural network accelerator framework of embodiments of the present application;

[0025] Figure 6 A schematic diagram of an architecture of a mapping unit of embodiments of the present application;

[0026] Figure 7 A schematic diagram of a distance filtering pre-processing stage of embodiments of the present application;

[0027] Figure 8 A schematic diagram of a distance filtering running stage of embodiments of the present application;

[0028] Figure 9 A schematic diagram of a hardware architecture of a flexible array of embodiments of the present application;

[0029] Figure 10 A schematic diagram of an end-to-end speedup on different tasks of embodiments of the present application;

[0030] Figure 11 A schematic diagram of computation unit utilization on different tasks of embodiments of the present application;

[0031] Figure 12 A schematic diagram of a relationship between subarray size and performance in a flexible array architecture of embodiments of the present application;

[0032] Figure 13 A schematic diagram of the effect of a distance filtering technique when using different grid sizes of embodiments of the present application;

[0033] Figure 14 Time proportion overhead diagram for the ranking preprocessing introduced by the embodiments of the present application;

[0034] Figure 15 Computing method flowchart for the hardware architecture of the 3D point cloud neural network algorithm of the embodiments of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0036] With the continuous development of 3D point cloud neural networks, 3D point cloud neural networks have become an important method for processing sparse 3D point cloud data in the fields of autonomous driving, robots, virtual reality, etc. At the same time, compared with 2D convolutional neural networks, 3D point cloud neural networks have significant precision advantages in segmentation, detection, tracking and other tasks.

[0037] In related art, the current mainstream point cloud neural network algorithm can be roughly divided into two categories: point-based and voxel-based; the basic operation in point cloud neural network can be abstracted into two categories: mapping operation and matrix operation.

[0038] Among them, the mapping operation establishes the relationship between the sparse input data and the output data, but the mapping operation is quite different in voxel-based networks and point-based networks. The point-based network usually first samples the input point cloud using FPS (Farthest Point Sampling) or other sampling operators to obtain the center point, and then uses Ball Query or kNN (k-Nearest Neighbor) method to search the neighbor points of the center point and aggregate the features, as shown in (a); the voxel-based method first voxelizes the input point cloud according to the coordinates, converts it into a sparse voxel in space, and establishes the input-output position mapping relationship for sparse convolution array operation through the positions of the input voxel and the output voxel, as shown in (b). Figure 1 Figure 1

[0039] ​​As the importance of 3D point cloud neural networks is increasing, further acceleration methods for point cloud neural networks based on GPUs (Graphic Processing Units), FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) can be used.

[0040] Among them, the GPU-based method is mostly to write a high-performance GPU operator library for the core operators in the point cloud network, such as SpConv, MinkowskiEngine, TorchSparse, etc. These works can use different data streams to take advantage of the large-scale parallelism of GPUs to achieve high-performance computing of sparse convolution; however, sparse convolution is only applicable to voxel-based networks and cannot effectively accelerate other types of networks, and the acceleration performance of the pure software solution is limited. The FPGA-based acceleration method is usually to accelerate the kNN, sub-manifold sparse convolution, etc. operators of the point cloud network, or to accelerate simple point-based networks such as PointNet while using FPGA to process sensor data. However, in the FPGA-based accelerator, there is a lack of a general point cloud network accelerator; the ASIC-based accelerator usually has higher performance than the GPU and FPGA-based accelerators, such as Mesorasi, PointX, Crescent and PointAcc, etc. Among them, PointAcc is the first general point cloud accelerator, which proposes a sorting-based sparse convolution kernel mapping calculation method and realizes general point cloud neural network acceleration by abstracting operators for various point cloud networks; PointAcc can accelerate mainstream point cloud neural network algorithms based on points, voxels, etc.

[0041] However, PointAcc still has defects, and its mapping operation has a large amount of off-chip memory access and low utilization of matrix operation calculation units:

[0042] (1) In the farthest point sampling, sphere query, kNN, sparse convolution mapping calculation, etc. mapping operators, the existing implementation has a large amount of redundant off-chip data access and low efficiency. For the farthest point sampling of point-based networks (such as PointNet++), each sampled point needs to access all input points and perform distance calculation, resulting in a large amount of redundant memory access. Experiments show that PointNet++ algorithm needs to access an average of 983 points for each sampled point. For voxel-based networks, the existing implementation needs to repeatedly access all input voxels multiple times when calculating the sparse convolution mapping relationship, also having a large amount of redundant off-chip memory access.

[0043] (2) For matrix operations, the scalability of existing point cloud neural network accelerators is poor, and the utilization rate of computing units is low. There is still a huge gap in computing power between the point cloud neural network accelerator and the commercial GPU in the related art. For example, the point cloud accelerator PointAcc has only 8TOPS of computing power, which is significantly lower than the general-purpose processor (such as NVIDIA Xavier with 30TOPS). As the number of parameters and the amount of computation of point cloud neural network models continue to increase, the demand for the number of tasks processed simultaneously, real-time performance, and throughput continues to increase, and the scalability of computing units in the accelerator is becoming increasingly important, thus putting higher requirements on the processing platform. However, the utilization rate of computing units in existing accelerators is low, and the scalability is poor. When the size of the computing unit array is directly expanded to increase the computing power, the actual performance cannot increase linearly with the computing power, because the utilization rate of the computing unit decreases with the increase of the computing power. Experiments show that when the peak computing power of PointAcc is increased from 8TOPS to 32TOPS, the utilization rate of the computing unit will decrease from 40.2% to 16.4%.

[0044] For example, in the related art, (1) the farthest point sampling is usually applied in point-based networks to realize the downsampling of point cloud data, while reducing the number of points and retaining key points. The distance between a certain point p i and a center point set containing m points is defined as the minimum value of the Euclidean distance from point p i to all points in , that is:

[0045]

[0046] where d(p i , p j ) is the Euclidean distance between points p i and p j , that is:

[0047] d(p i , p j ) = ||p i -p j ||2 (2)

[0048] The farthest point sampling algorithm iteratively samples the point with the maximum distance from the current center point set, as shown in the following formula:

[0049]

[0050] where is the center point set that has been sampled in the mth iteration; in each iteration, the point with the maximum distance will be added to the center point set, i.e.:

[0051]

[0052] The farthest point sampling calculation method can be completed by maintaining a table T that records the minimum distance of all points to the current center point set (m) . After updating the center point set each time using formula (4), the minimum distance table T (m) needs to be updated; in each iteration, the point with the largest distance in T (m) is taken out and added to the center point set , and the table T (m) is updated; therefore, each element in the table T (m) can be recorded as According to the definition, there is

[0053]

[0054] However, the update of the minimum distance table T (m) in each iteration requires a large amount of off-chip memory access and distance calculation. The method in the related art updates the minimum distance table T by traversing all points p i in the input point set and calculating their distances from Therefore, when sampling M center points from N points using this method, the complexity is O(MxN). In the PointNet++ based on point cloud neural network, the farthest point sampling calculation alone accounts for 36.68% of the total running time.

[0055] (2) Sparse convolution is a basic operator of a point cloud neural network based on voxels. Due to the sparsity of the input and output data of sparse convolution, mapping calculation is required before feature calculation, that is, the corresponding relationship between the input voxels, output voxels and weights is found to form a mapping relationship table of input-output-weights.

[0056] The point cloud neural network accelerator PointAcc in the related art uses a weight-first method to calculate the sparse convolution mapping, that is, only the mapping relationship between one position of the convolution kernel and the input-output is calculated each time. Figure 2 (a) is a 2D schematic diagram of calculating the mapping relationship of the submanifold sparse convolution using this method. The specific mapping calculation method is to find the intersection of the input feature map after offset according to the position of the convolution kernel and the output feature map, as shown in Figure 2 (c). In order to calculate the mapping relationship between the weight at the (-1, -1) position and the input P and the output Q, first add an offset of (1, 1) to the input feature map, and then overlap and compare with the output feature map, the input P i and the output Qj The overlapping position means that there is P at this position i Q j W -1,-1 The mapping relationship, that is, the mapping relationship can be generated and stored in the mapping table.

[0057] However, the weight-priority convolution kernel mapping method needs to traverse all inputs when calculating each position of the weight. Due to the limitation of on-chip cache capacity, all voxel data cannot be saved at one time, the on-chip data reuse rate is low, and high off-chip memory access will be caused. Assuming that the size of the convolution kernel is KxKxK, the number of non-zero voxels is N, and the number of voxels of off-chip memory access is (K 3 xN).

[0058] (3) The systolic array is the mainstream computing unit used in existing accelerators. The systolic array multiplies the input features and the weight data to obtain the output feature data. In actual point cloud neural networks, the computing unit is used to process both sparse convolution and ordinary dense convolution. Therefore, the input features can be taken out according to the mapping relationship table generated by the mapping unit, or can be directly taken out by the subscript according to the rules of dense convolution.

[0059] Figure 3 (a) is a schematic diagram of the systolic array using "weight stationary" data flow calculation. The left side of the systolic array is the input feature data, the weight is stored in each computing unit, and the output data below is the partial sum obtained by multiplying the input features and the weight and then accumulating the multiplication; after the products of all input channels are accumulated into the partial sum, the calculation of the output feature of the corresponding output channel is completed.

[0060] However, when the scale of the systolic array expands, due to the limited number of input, output channels and mapping relationships, the systolic array can not be fully utilized. For example Figure 3 (a) shows that when the input or output channels are not enough to fill the systolic array, the remaining part of the array will be idle, resulting in the deterioration of throughput and delay. Experiments show that when the array of PointAcc is expanded to four times the original, the utilization rate decreases by 2.58 times.

[0061] As shown in Figure 3 (a) (b), by dividing a 4x4 systolic array into four 2x2 sub-arrays, the number of working units can be increased from 6 to 12 through weight replication, and the utilization rate can be increased from 37.5% to 75%. This improvement is mainly due to the fact that different sub-arrays can accept different input data and calculate their respective partial sum results Figure 3 (b) left and right array), and the partial sums can be accumulated to obtain the final result Figure 3(b) the array above and below.

[0062] In view of the problems in the prior art mentioned in the background, the off-chip memory amount of the point cloud neural network accelerator is large, the utilization rate of the computing unit is low, the processing speed of the accelerator is slow, the scalability and flexibility are poor, and the actual use needs cannot be met. To solve the above problems, the present application provides a hardware architecture for 3D point cloud neural network algorithm.

[0063] The hardware architecture for 3D point cloud neural network algorithm, the computing method and the electronic device of the embodiments of the present application are described below with reference to the accompanying drawings. Specifically, Figure 4 A schematic diagram of the hardware architecture for 3D point cloud neural network algorithm provided by the embodiments of the present application.

[0064] As Figure 4 shown, the hardware architecture for 3D point cloud neural network algorithm 10 includes off-chip memory 100, mapping module 200 and computing module 300.

[0065] The mapping module 200 is provided with distance filtering technology and / or output priority mapping calculation technology to reduce the off-chip memory amount of the off-chip memory during the mapping operation of the 3D point cloud neural network algorithm, and to generate the mapping relationship between the input and the output according to the mapping operation; the computing module 200 is provided with an elastic array architecture, which adjusts the elastic array architecture according to the different scale of the computing task in the 3D point cloud neural network algorithm, and takes out the weight and the corresponding input feature to perform matrix operation to obtain the output feature according to the mapping relationship.

[0066] As Figure 5 can be understood, as

[0067] In the embodiment of the present application, the mapping module 200 comprises a distance calculation unit, a sorting unit, a maximum calculation unit, a merging unit, a splicing unit, a coincidence detection unit, a grid filtering unit, a loading unit, a grid cache, a sorting unit cache, a merging unit cache and a mapping relationship cache. The sorting unit, the merging unit and the splicing unit together constitute a unit capable of sorting data of any length. The coincidence detection unit is used to detect the coincidence position in sparse convolution mapping calculation, and when the convolution kernel has symmetry and a mapping of a non-convolution kernel center position is detected, an opposite mapping is generated according to the current mapping. The grid filtering unit is used to calculate the shortest distance between a center point and a grid, and filter out grids with a distance greater than a filtering threshold. The grid cache is used to store the information of the grids used for grouping when processing a point-based network, and store the coordinates of voxels when processing a voxel-based network. The sorting unit cache and the merging unit cache are respectively used to store the data to be processed by the sorting unit and the merging unit. The mapping relationship cache is used to store the generated mapping relationship table.

[0068] It can be understood that the function of the mapping module 200 is to calculate the mapping relationship between sparse input and output data. The embodiment of the present application can be designed in hardware, and the distance filtering and output priority mapping calculation are realized through the above-mentioned calculation unit, sorting unit, maximum calculation unit, merging unit, coincidence detection unit, grid cache, sorting unit cache, merging unit cache and mapping relationship cache, so as to optimize the mapping operation in the point-based and voxel-based networks respectively, thereby reducing the off-chip memory access amount of mapping calculation.

[0069] Specifically, as shown in FIG. 2, the sorting unit and the merging unit are both implemented based on double sorting. Figure 6 The distance calculation unit, the sorting unit, the maximum calculation unit, the merging unit and the coincidence detection unit are designed in parallel to improve the performance as much as possible. The sorting unit, the merging unit and the splicing unit together constitute a unit capable of sorting data of any length. The coincidence detection unit is used to detect the coincidence position in sparse convolution mapping calculation, and when the convolution kernel has symmetry and a mapping of a non-convolution kernel center position is detected, an opposite mapping is generated according to the current mapping. The grid filtering unit calculates the shortest distance between a certain center point and a grid, and filters out grids with a distance too large to reduce off-chip data access.

[0070] The mapping unit has four caches: a grid cache, a sorting unit cache, a merging unit cache and a mapping relationship cache. The grid cache has two functions: storing the information of the grids used for grouping when processing a point-based network, and storing the coordinates of voxels when processing a voxel-based network. The sorting unit cache and the merging unit cache are respectively used to store the data to be processed by the sorting unit and the merging unit. The mapping relationship cache is used to store the generated mapping relationship table. In order to improve the utilization rate as much as possible, double buffering design is used for the related caches.

[0071] In the embodiment of the present application, the 3D point cloud neural network algorithm includes a point-based network and a voxel-based network, wherein the mapping module filters off-chip data access that does not meet the access condition through a distance filtering technique to reduce the off-chip memory access amount of off-chip storage; the mapping module reuses on-chip data through an output-priority mapping calculation technique to reduce the off-chip memory access amount of off-chip storage.

[0072] It can be understood that the embodiment of the present application can design an efficient mapping module, which can use the distance filtering technique and the output-priority mapping calculation technique to support the mapping calculation of the point-based network and the voxel-based network, so as to reduce the off-chip memory access amount of off-chip storage and meet the actual use needs; wherein the distance filtering process and the output-priority mapping calculation process of the embodiment of the present application can be specifically as follows:

[0073] (1) Distance filtering

[0074] In the embodiment of the present application, the update process of the minimum distance table can be technically simplified.

[0075] Firstly, according to the definition of formula (3) in the above embodiment, all values in the minimum distance table T (m) are less than that is:

[0076]

[0077] Secondly, in any iteration (for example, it can be set as the mth iteration), if the distance between the point p i and the center point sampled in this round is greater than or equal to the ith element in the table then will not be updated, that is:

[0078]

[0079] Based on formula (6) (7), formula (8) can be obtained; formula (8) is a key formula for optimizing the farthest point sampling:

[0080]

[0081] Formula (8) shows that the minimum distance of the node with a large distance from the center point sampled in this round does not need to be updated, so the memory access amount and the calculation amount can be reduced by filtering out the points with a too far distance .

[0082] ​According to the above principle, the embodiment of the present application can use a distance filtering technology to quickly filter out points far from the center point; wherein the distance filtering technology can include two stages: a preprocessing stage and a runtime stage, which can be as shown in Figure 7 and Figure 8 .

[0083] In the embodiment of the present application, the distance filtering technology includes: dividing the points in the input point cloud into grids according to coordinates, and sorting them according to grid positions; in each iteration sampling, filtering the grids whose distance from the sampling point is greater than the filtering threshold.

[0084] It can be understood that the distance filtering technology is a technology that uses the geometric distance of points to limit the access of off-chip data that does not meet the access conditions in the mapping operation; wherein the specific scheme of the distance filtering technology in the embodiment of the present application can be as follows:

[0085] I. In the preprocessing stage, the points in the input point cloud are divided into grids according to coordinates, and sorted according to grid positions:

[0086] In the embodiment of the present application, the points in the input point cloud are divided into grids according to the pre-set grid size according to coordinates, and sorted according to grid positions, including: dividing the target space into multiple grids according to the grid size set in advance; sorting all points according to the coordinates of the grid where each point in the point cloud is located and writing back to off-chip storage, and retaining the position index of non-empty grids and the minimum sequence number of points in the grid in on-chip cache.

[0087] It can be understood that in the preprocessing stage, the distance filtering technology can group and sort the input point cloud data according to the grid, enhancing data locality.

[0088] For example, the entire space is uniformly divided into multiple grids according to the grid size set in advance, and for 3D point cloud, each grid is a three-dimensional cube (different dimensions can have different lengths); when in two-dimensional space, as shown in Figure 7 , the grid can be a two-dimensional rectangle. Then sort all points according to the coordinates of the grid where each point is located and write back to off-chip storage, so that the points in the same grid will be stored in adjacent positions off-chip, thereby improving data locality; wherein the non-empty grid can be as shown by the dark grid in Figure 7 . Only the position index of non-empty grids and the minimum sequence number of points in the grid are retained in on-chip cache to facilitate off-chip access to specific data. Wherein grouping by grid only needs to be done once, because the points sampled after grouping still maintain the grouped state, so subsequent furthest point sampling or neighbor query operations can be performed.

[0089] II. During runtime, in each iteration of sampling, filter out grids whose distance from the sampling point is greater than the filtering threshold:

[0090] During runtime, the distance filtering technology in this application embodiment filters the grid based on distance.

[0091] Specifically, in the farthest point sampling operator, if a grid is located at a sampling point selected in the current iteration... The nearest distance is greater than The entire grid will be filtered out, and data for all points within that grid will no longer be accessed. The correctness of the calculation after filtering is guaranteed by equation (8). In calculating each non-empty grid... When determining the closest distance, the spatial coordinate range of the grid can be obtained based on the grid position index stored in the on-chip cache and the preset grid size, and the relationship between the grid and the target grid can be calculated based on the coordinate range. The minimum distance.

[0092] Therefore, the grid and its coordinates can be calculated without accessing points within the grid. The closest distance, and and That is, the minimum distance table T (m) The table contains the maximum distance and the point corresponding to the maximum distance, so no additional calculation is needed; it is based solely on the existing T. (m) The table can be used to obtain the center point. With filter threshold

[0093] For example, Figure 8 An example of group filtering is shown in... Figure 8 In a certain iteration, the sampling center point selected is point ⑧. The distance filtering technique first calculates the minimum distance between point ⑧ and all non-empty grids (dark grids), then compares the minimum distance between each grid and the minimum distance with a filtering threshold. Figure 8 The radius R in the diagram. Distance filtering removes all grid cells with a minimum distance greater than R, only accessing the remaining grid cells, specifically the dark grid cells that intersect with the circle centered at point ⑧ and with radius R. Figure 8 In this context, the grid with position indices (1,3), (2,3), and (3,3) is used. Then, based on the starting index of the grid's midpoint, the data at the midpoint in off-chip memory is accessed. Figure 8 In the middle, the coordinates of points ⑦, ⑧, and ⑨ are accessed, the distance to point ⑧ is calculated, and the minimum distance table T is updated. (m) .

[0094] In point-based algorithms, after sampling the center point, a ball query or kNN operator can be used to perform a neighbor query on the center point, aggregate the features of the neighbor points, and concatenate them as the features of the center point to achieve feature extraction and aggregation.

[0095] Specifically, in the ball query operator, for a given center point, the ball query algorithm returns the index of k neighbor points contained in the sphere with the center point as the center and the maximum query distance as the radius. If k neighbor points cannot be found within the given query distance, the neighbor points that have been queried are used to fill to k. The kNN operator is similar to the ball query operator, but the difference is that kNN does not set a maximum query distance and returns the index of the k neighbor points closest to the center point.

[0096] Since the calculation process of the ball query and kNN also only needs the information of the points near the center point, the distance filtering technology can also be used for optimization in the embodiments of the present application.

[0097] For example, Figure 8 The ball query operator can be regarded as an example of a neighbor query operator with a filtering threshold, and the point 7 and the radius R represent the center point and the filtering threshold, respectively. In the ball query operator, the filtering threshold is the maximum query distance; in the kNN operator, since there is no maximum query distance, the filtering threshold is set to a function that increases with the number of iterations, gradually increasing the threshold R of the distance filtering until k neighbor points are found. Thus, the distance filtering technology of the embodiments of the present application can effectively reduce the off-chip data access amount and the calculation amount of the calculation distance in the mapping operator based on the point network.

[0098] (2) Output-priority mapping calculation method

[0099] In the embodiments of the present application, the output-priority mapping calculation technology includes: sorting the voxel data in the off-chip storage in the order of coordinates; multiplexing the voxels at the edge of the convolution kernel and calculating the mapping of the center and one side of the convolution kernel according to the symmetry of the convolution kernel, and obtaining the mapping of the other side by inverting the mapping relationship according to the symmetry; and writing the mapping result into different addresses of the cache according to the input point cloud weight.

[0100] The output-priority mapping calculation technology is a technology for multiplexing on-chip data by cutting the output voxels into the on-chip cache.

[0101] It can be understood that the embodiments of the present application can use an output-priority mapping calculation method, such as Figure 2 As shown in (b), the output voxel row is cut to ensure that the cut voxels can be completely placed in the on-chip cache, thereby improving the on-chip data multiplexing rate, and thus the on-chip data multiplexing can be maximized.

[0102] Specifically, when calculating the input-output-weight mapping relationship of all the output rows, no additional off-chip data access is needed. Finally, all the voxels are only loaded once.

[0103] In addition, since the sub-manifold sparse convolution does not change the positions of the non-zero voxels in the input and output, the mapping calculation can be reduced by half by using the characteristics of the symmetric convolution kernel. For example, as shown in Figure 2 (b) If the mapping P i -Q j -W (k,l) exists, then according to the symmetry, the reverse mapping P j -Q i -W (-k,-l) also exists. Taking a 3x3 convolution kernel as an example, according to the symmetry, only the mapping relationships of the center and the four positions on the right side need to be calculated, and the mapping relationships of the four positions on the left side can be derived according to the mapping of the symmetric positions on the right side.

[0104] Figure 2 (b) also shows a specific tile strategy for the output-priority convolution kernel mapping calculation, in which the input voxels are arranged in the coordinate order in the off-chip memory.

[0105] For example, assuming that 4 voxels can be accommodated in the on-chip cache, the on-chip cache is filled as much as possible during the splitting process. The voxels at the edge of the convolution kernel, for example Figure 2 P1, P2, and P3 in (b) at the two tile boundary rows can be reused in the next tile and will not be loaded repeatedly. Figure 2 (d) shows the process of calculating Figure 2 the mapping relationship of tile 0 in (b). According to the symmetry of the convolution kernel, only the mapping of the center and the four positions on the right side need to be calculated, and the mapping of the four positions on the left side of the convolution kernel is obtained by inverting the mapping relationship according to the symmetry. The final mapping result is written into different addresses of the cache according to the position of the weight in the mapping result.

[0106] In the embodiment of the present application, the computing module 300 is further applied to: the elastic array architecture includes a plurality of sub-arrays, a multiplexer between the sub-arrays, an input feature on-chip cache, an output feature on-chip cache, and a weight on-chip cache, wherein each sub-array is a systolic array; and the multiplexer is called to adjust the topological relationship of the plurality of sub-systolic arrays according to the size of the calculation task in the 3D point cloud neural network algorithm.

[0107] It can be understood that, according to the principle in (b) in the above embodiment Figure 3 , the elastic array architecture as shown in (b) can be designed, which is composed of a plurality of N*N sub-systolic arrays and uses a weight fixed data stream for calculation. Figure 9

[0108] ​Through the multiplexer between the sub-arrays, the architecture runtime can flexibly adjust the topology according to the task characteristics, forming a large array or multiple sub-arrays. By dividing a single large array into multiple small arrays, the topology between the arrays is dynamically adjusted, and the array of a suitable size is allocated for different scale computing tasks, thereby improving the utilization rate of the computing unit.

[0109] When the input channel number of the current task is large, the vertical direction sub-array can be connected in a top-down manner through the multiplexer. The cascaded sub-arrays form a longer impulse array in the vertical direction through the accumulation of partial sums; when the input channel number of the current task is small, the vertical direction sub-array is connected with the input feature cache through the multiplexer, and different weight inputs are used for parallel operation. N can be regarded as the smallest unit of array flexibility. In the case of the same computing power, the array flexibility increases with the decrease of N, but the cost is that the connection overhead between the sub-arrays and between the sub-arrays and the cache will increase, so there is a trade-off between array flexibility and hardware overhead.

[0110] The hardware architecture of the 3D point cloud neural network algorithm of the embodiment of the application will be described below according to a specific embodiment, as follows:

[0111] One embodiment of the application can use four different point cloud neural networks for evaluation, and the task types include classification, detection and segmentation, as shown in Table 1.

[0112] Table 1

[0113] Task Dataset Network model Network category Notation Classification ModelNet40 PointNet++ Point-based PN Detection KITTI CenterPoint Voxel-based CP Segmentation S3DIS MinkowskiUNet Voxel-based MU(i) Segmentation SemanticKITTI MinkowskiUNet Voxel-based MU(o)

[0114] These data sets contain objects (ModelNet40), indoor scenes (S3DIS) and outdoor scenes (KITTI, SemanticKITTI), and the number of points in the input point cloud varies from 1000 to 124,000 points. For the voxel-based network among them, the number of input voxels varies from 15,000 to 94,000. One embodiment of the application can be implemented using Verilog, synthesized under TSMC 65nm technology, and evaluated for performance through a cycle-accurate simulator.

[0115] One embodiment of the application can use the point cloud accelerator PointAcc as the evaluation baseline design, and in the experiment, 8TOPS and 32TOPS two computing power configurations are used to evaluate the application and PointAcc, respectively. The specific hardware design parameters are shown in Table 2.

[0116] Table 2

[0117]

[0118] Compared with PointAcc, the design proposed in one embodiment of the present application has obvious end-to-end speedup on each task, as shown in Figure 10 The reason for performance improvement is the increase of computing unit utilization and the reduction of off-chip memory access amount. When the peak computing power is 8TOPS, the performance of one embodiment of the present application is 1.23-2.92 times that of PointAcc; when the peak computing power is expanded to 32TOPS, the performance of one embodiment of the present application is 1.73-3.97 times that of PointAcc. The reason for greater performance improvement at higher peak computing power is the significant increase of computing unit utilization. Figure 11 The utilization is shown to be improved. Even if the peak computing power is expanded to 32TOPS, the design proposed in one embodiment of the present application still maintains a high utilization of more than 57.01%, while the utilization of PointAcc in the worst case is only 16.37%.

[0119] Figure 12 The relationship between the subarray size and performance in the elastic array design is shown. Under the condition of the same total computing capacity, more smaller subarrays can provide higher flexibility, and thus higher performance, but at the cost of higher hardware overhead. This also confirms the effectiveness of the elastic array design of one embodiment of the present application. Finally, one embodiment of the present application selects a design scheme with a subarray size of 16x16 to balance performance and hardware overhead. In the design with a peak computing power of 32TOPS, one embodiment of the present application uses 8x8 16x16-sized subarrays, which can achieve a performance improvement of 3.76 times compared with using only one 128x128 array (i.e., PointAcc).

[0120] Figure 13 The effect of the distance filtering technique when using different grid sizes under the ModelNet40 dataset is shown. The experimental results show that the performance is best when the grid size is 0.26, which can reduce the off-chip memory access data amount by an average of 6.52 times, and thus bring an end-to-end speedup of 1.22 times.

[0121] Since the present application needs to introduce preprocessing, the distance filtering technique first needs to sort the input points, and the output priority mapping calculation needs to sort the input voxels by coordinates, thus introducing additional preprocessing overhead. Experiments show that the preprocessing overhead accounts for less than 2.43% of the total running time, as shown in Figure 14

[0122] ​Therefore, one embodiment of the present application proposes an efficient 3D point cloud neural network accelerator, which can reduce off-chip memory access by filtering out unnecessary data access and increasing on-chip data multiplexing, and improve the utilization of computing units by introducing a flexible computing array architecture. Compared with the point cloud accelerator PointAcc in the related art, the above embodiment respectively realizes an end-to-end speed improvement of up to 1.76 times and 3.97 times for point-based networks and voxel-based networks.

[0123] In summary, the hardware architecture for 3D point cloud neural network algorithm according to the embodiment of the present application can use distance filtering technology to filter out unnecessary off-chip data access for point-based networks, thereby reducing off-chip memory access. Meanwhile, for voxel-based networks, the output-first mapping computing technology can be used to increase on-chip data multiplexing and improve data multiplexing rate, thereby reducing off-chip memory access. In addition, the flexible array architecture can be used to flexibly adapt to different sizes of computing tasks in the point cloud network by different sub-array combinations, so that the utilization of computing units remains high when the size of computing units increases, thereby improving the utilization of computing units, enhancing the flexibility and scalability of the accelerator, and meeting the actual use needs.

[0124] Secondly, the computing method of the hardware architecture for 3D point cloud neural network algorithm according to the embodiment of the present application is described with reference to the accompanying drawings, Figure 15 The flowchart of the computing method of the hardware architecture for 3D point cloud neural network algorithm provided by the embodiment of the present application is shown in the figure.

[0125] As Figure 15 shown, the computing method of the hardware architecture for 3D point cloud neural network algorithm uses the hardware architecture for 3D point cloud neural network algorithm according to the above embodiment for computing, wherein the method comprises the following steps:

[0126] In step S101, the distance filtering technology and / or the output-first mapping computing technology are set in the mapping module to reduce the off-chip memory access of the off-chip storage when computing the mapping operation of the 3D point cloud neural network algorithm, and the mapping relationship between the input and the output is generated according to the mapping operation.

[0127] In step S102, the flexible array architecture is set in the computing module, the flexible array architecture is dynamically adjusted according to the size of the computing task in the 3D point cloud neural network algorithm, and the output feature is obtained by matrix operation on the weight and the corresponding input feature according to the mapping relationship.

[0128] It should be noted that the above description of the hardware architecture for 3D point cloud neural network algorithm is also applicable to the embodiment of the computing method of the hardware architecture for 3D point cloud neural network algorithm, which will not be described here.

[0129] According to the computing method of the hardware architecture of the neural network algorithm for the 3D point cloud in the embodiments of the present application, for the point-based network, a distance filtering technology can be used to filter out unnecessary off-chip data access, thereby reducing the off-chip memory access amount; for the voxel-based network, an output priority mapping computing technology can be used to increase on-chip data multiplexing and improve data multiplexing rate, thereby reducing the off-chip memory access amount; and the embodiments of the present application can also use an elastic array architecture to flexibly adapt to different scale computing tasks in the point cloud network through different sub-array combinations, so that the utilization rate of the computing unit is still high when the scale of the computing unit is increased, thereby improving the utilization rate of the computing unit and improving the flexibility and scalability of the accelerator, and meeting the actual use needs.

[0130] The embodiments of the present application also provide an electronic device comprising the hardware architecture of the neural network algorithm for the 3D point cloud in the above embodiments.

[0131] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0132] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0133] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium. The program, when executed, includes one of the steps of the method embodiment or a combination thereof.

Claims

1. A hardware architecture for a 3D point cloud neural network algorithm, characterized in that, The application relates to a 3D point cloud neural network algorithm computing device, which comprises the following parts: off-chip storage; a mapping module provided with a distance filtering technology and / or an output priority mapping calculation technology to reduce off-chip access of the off-chip storage during mapping operation of the 3D point cloud neural network algorithm, the 3D point cloud neural network algorithm comprising a point-based network and a voxel-based network, wherein the mapping module filters off-chip data access that does not meet an access condition through the distance filtering technology to reduce off-chip access of the off-chip storage; the mapping module multiplexes on-chip data through the output priority mapping calculation technology to reduce off-chip access of the off-chip storage, the distance filtering technology is a technology for limiting off-chip data access that does not meet an access condition in a mapping operation by using the geometric distance of points, and the distance filtering technology comprises the following steps: points in an input point cloud are divided into grids according to coordinates, and the points are sorted according to grid positions; during each iteration sampling, grids with a distance greater than a filtering threshold from a grid where a sampling point is located are filtered; the output priority mapping calculation technology is a technology for multiplexing on-chip data by putting output voxel segmentation into an on-chip cache; a mapping relationship between input and output is generated according to the mapping operation; a computing module provided with a flexible array architecture, the flexible array architecture being adjusted according to the scale of a computing task in the 3D point cloud neural network algorithm, and weight and corresponding input features being taken out according to the mapping relationship to perform matrix operation to obtain output features.

2. The hardware architecture for 3D point cloud neural network algorithm according to claim 1, wherein, The mapping module comprises a distance calculation unit, a sorting unit, a maximum calculation unit, a merging unit, a splicing unit, a coincidence detection unit, a grid filtering unit, a loading unit, a grid cache, a sorting unit cache, a merging unit cache and a mapping relationship cache, wherein the sorting unit, the merging unit and the splicing unit jointly form a unit capable of realizing sorting of data of any length, the coincidence detection unit is used for detecting a coincidence position in sparse convolution mapping calculation, and when a convolution kernel has symmetry and a mapping of a non-convolution kernel center position is detected, a reverse mapping is generated according to the current mapping; the grid filtering unit is used for calculating the shortest distance between a center point and a grid, and filtering out grids with a distance greater than a filtering threshold; the grid cache is used for storing information of grids used for grouping when a point-based network is processed, and storing coordinates of voxels when a voxel-based network is processed; the sorting unit cache and the merging unit cache are respectively used for storing data to be processed by the sorting unit and the merging unit; and the mapping relationship cache is used for storing a generated mapping relationship table.

3. The hardware architecture for 3D point cloud neural network algorithm of claim 1, wherein, The points in the input point cloud are divided into grids according to coordinates, and the points are sorted according to grid positions, which comprises the following steps: a target space is divided into multiple grids according to a grid size set in advance; all points are sorted according to coordinates of grids where the points are located and written back to the off-chip storage, and the position index of non-empty grids and the minimum serial number of points in the grids are reserved in the on-chip cache.

4. The hardware architecture for 3D point cloud neural network algorithm of claim 1, wherein, The elastic array architecture comprises a plurality of sub-pulsating arrays, a plurality of selectors between the sub-pulsating arrays, an input feature on-chip cache, an output feature on-chip cache and a weight on-chip cache, and the plurality of selectors are used to adjust the topological relationship of the plurality of sub-pulsating arrays according to the size of the calculation task in the 3D point cloud neural network algorithm.

5. The hardware architecture for 3D point cloud neural network algorithm of claim 1, wherein, The output-priority mapping calculation technology comprises: sorting the voxel data in the off-chip storage in a coordinate order; multiplexing the voxels at the edge of the convolution kernel, and calculating the mapping of the center and one side position of the convolution kernel according to the symmetry of the convolution kernel, and obtaining the mapping of the other side through symmetry inversion; writing the mapping result into different addresses of the cache according to the input point cloud weight.

6. A computing method of a hardware architecture for a 3D point cloud neural network algorithm, characterized in that, The method uses the hardware architecture for the 3D point cloud neural network algorithm according to any one of claims 1-5 for calculation, wherein the method comprises the following steps: The mapping module is provided with a distance filtering technology and / or an output-priority mapping calculation technology to reduce the off-chip memory access amount of the off-chip storage when calculating the mapping operation of the 3D point cloud neural network algorithm, and the 3D point cloud neural network algorithm comprises a point-based network and a voxel-based network, wherein the mapping module filters the off-chip data access that does not meet the access condition through the distance filtering technology to reduce the off-chip memory access amount of the off-chip storage; the mapping module multiplexes the on-chip data through the output-priority mapping calculation technology to reduce the off-chip memory access amount of the off-chip storage, the distance filtering technology is a technology for limiting the off-chip data access that does not meet the access condition in the mapping operation by using the geometric distance of the point, and the distance filtering technology comprises: dividing the points in the input point cloud into grids according to coordinates, and sorting the grids according to positions; filtering the grids whose distance from the grid where the sampling point is located is greater than the filtering threshold at each iteration sampling; the output-priority mapping calculation technology is a technology for multiplexing on-chip data by cutting the output voxels into on-chip caches; and a mapping relationship between the input and the output is generated according to the mapping operation; The calculation module is provided with an elastic array architecture, and the elastic array architecture is dynamically adjusted according to the size of the calculation task in the 3D point cloud neural network algorithm, and the output feature is obtained by performing matrix operation on the weight and the corresponding input feature according to the mapping relationship.

7. An electronic device, comprising: The hardware architecture for the 3D point cloud neural network algorithm.

Images