A 3D Point Cloud Data Processing Method and Accelerator Based on Neural Network

Abstract

The present invention provides a method and an accelerator for processing 3D point cloud data based on a neural network. The method includes: inputting 3D point cloud data into the neural network to output the target category represented by the point cloud. The neural network includes a feature extraction module LCEM, an LBR block, a max pooling layer, a downsampling layer, and a fully connected layer. The feature extraction module LCEM fuses the features corresponding to the part of the points closest to the sampling points through a convolutional layer. The LBR block is used for feature extraction. The max pooling layer and the downsampling layer output a single value to reduce the input size of the neural network. The fully connected layer is used to output the probabilities of the target corresponding to each category and output the category with the highest probability. The present invention is used to process 3D point cloud classification problems, aggregates neighborhood features through learnable weights, and can still maintain the same accuracy under the condition of compressing network parameters by more than 30 times compared with Pointnet.

Description

Technical Field

[0001] The present invention relates to a method for processing 3D point cloud data based on a neural network and an accelerator. Background Art

[0002] 3D point cloud is a series of scattered points in a 3D space, and each point contains spatial coordinates and feature values. Compared with a planar image, 3D point cloud has more depth information and can better restore the shape and position information of an object. Therefore, 3D point cloud data is widely used in fields such as autonomous driving and virtual reality. A neural network is an important method for processing point cloud data at present and is often used in tasks such as object classification, object detection, and scene segmentation based on point cloud. The methods for processing point cloud data using a neural network are generally divided into the following three types: planar mapping, voxelization, and direct processing. The planar mapping method maps 3D point cloud data onto a 2D plane to obtain a bird's-eye view, and then applies a traditional 2D neural network to process it. The voxelization method rasterizes the 3D space, distributes each point into regular spatial grids, and uses a dedicated 3D convolutional neural network to process it. Directly processing the point-to-point cloud data is currently the most efficient solution. Since the preprocessing of the 3D point cloud is avoided, this solution greatly reduces the processing delay and the loss of information volume. The Pointnet network takes the position information and feature values of the point cloud as inputs and directly processes them using a convolutional neural network. Based on Pointnet, Pointnet++ strengthens the ability to extract local features, fuses the features of the neighborhood points and the central point of each sampled point, and becomes the most efficient 3D point cloud neural network at present.

[0003] Due to the applications of 3D point cloud data in reality, such as autonomous driving and virtual reality, extremely high real-time performance is required, so the requirement for the data processing speed is extremely high. Generally speaking, the existing neural networks have a large number of parameters and a long calculation delay, and cannot meet the real-time requirements of point cloud applications. As an efficient parallel computing platform, FPGA is often used to accelerate traditional neural networks, and traditional CNN accelerators based on FPGA emerge in an endless stream. However, there is still relatively little work on accelerating 3D point cloud networks using FPGA, and most of them are designs based on the Pointnet network. Such as the literature "X. Zheng, M. Zhu, Y. Xu, and Y. Li. An fpga based parallel implementation for point cloud neural network. In 2019 IEEE 13 thThe O-Pointnet network adopted in "International Conference on ASIC (ASICON), pages 1–4, 2019" has been improved to be hardware-friendly based on Pointnet and accelerated using FPGA. Lin Bai et al. accelerated the classification and scene segmentation network based on Pointnet in the literature "L. Bai, Y. Lyu, X. Xu, and X. Huang. Pointnet on fpga for real-time lidar point cloud processing. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5, 2020", and designed a PE module containing a multiplier array, an adder tree, and an adder array to support different layers in Pointnet. However, the above hardware architectures are all designed based on the Pointnet model and cannot efficiently support lightweight classification networks. At the same time, due to the still large number of parameters in the model and the insufficient throughput rate, it still cannot meet the requirements of real-time applications.

[0004] The most widely used 3D point cloud processing networks currently are Pointnet and its derivative networks. However, due to the large number of network parameters and long inference latency, they cannot meet the real-time requirements. The current Pointnet network has 1.75M parameters, and the detection accuracy of the classification network is 89.2%. Under the condition of the same number of parameters, the accuracy of the Pointnet++ network can be increased to 91.9%.

[0005] To further reduce the processing latency of point cloud data, FPGA is applied to the work of 3D point cloud processing. Most of the existing FPGA accelerators based on 3D point clouds are designed based on Pointnet and its derivative networks and cannot efficiently support the newly proposed lightweight point cloud classification network. Summary of the Invention

[0006] Object of the Invention: 3D point clouds have extensive applications in fields such as autonomous driving, virtual reality, and intelligent robots. However, due to their sparsity and disorder, the processing speed of neural networks often cannot meet the real-time requirements, and traditional hardware accelerators also result in low acceleration efficiency because they cannot fully adapt to 3D point cloud networks. To address this problem, this solution proposes a method for processing 3D point cloud data based on neural networks and designs an efficient hardware acceleration architecture according to its computational characteristics.

[0007] The method includes: inputting 3D point cloud data into a neural network and outputting the target category represented by the point cloud;

[0008] The neural network includes a feature extraction module LCEM, an LBR block, a max pooling layer, a downsampling layer, and a fully connected layer;

[0009] The feature extraction module LCEM directly processes the input point cloud data, and fuses the features corresponding to the part of the points (neighborhood points) closest to the sampled points through a convolutional layer in the data;

[0010] The LBR block is used for feature extraction. The LBR block includes a linear layer Linear, a batch normalization layer Batchnormalization, and a rectified linear unit layer ReLU;

[0011] The max pooling layer and the downsampling layer reduce the input size of the neural network by inputting multiple values and outputting a single value; when the pooling layer size is 4, the max pooling layer inputs 4 values each time and outputs the maximum value among the 4 values. The downsampling layer reduces the output size to half of the input by setting the stride of the convolutional layer to 2 for each layer;

[0012] The fully connected layer directly outputs the probabilities of the target belonging to each category. The fully connected layer is used to output the probabilities of the target corresponding to each category and outputs according to the category with the highest probability.

[0013] The 3D point cloud data includes a series of scattered points disordered in space, which contains the position features of the objects in space.

[0014] The feature extraction module LCEM includes module A and module B;

[0015] Module A includes two different linear layers, a batch normalization layer Batch normalization, a rectified linear unit layer ReLU, and a softmax layer;

[0016] The softmax layer is used to convert the input features into the probabilities of the target belonging to each category;

[0017] The two different linear layers are respectively denoted as Linear1 and Linear2; the linear layers Linear1 and Linear2 process the input through matrix multiplication. The batch normalization layer Batch normalization normalizes the input, and the rectified linear unit layer ReLU, as an activation function, introduces non-linearity to the neural network;

[0018] The batch normalization layer Batch normalization and the rectified linear unit layer ReLU in module A are the batch normalization layer Batch normalization and the rectified linear unit layer ReLU in the LBR block, and are used more than once in the neural network;

[0019] The B module includes a batch matrix multiplication module BMM and a convolutional layer Conv, which are used to fuse the features of the neighborhood points of the input points.

[0020] The A module uses the position coordinates of the points in the neighborhood of the sampling points as input for training to obtain the weight values of the neighborhood points; the B module multiplies the weight values obtained by the training of the A module by the feature values of the neighborhood points to perform weighted fusion on the neighborhood features.

[0021] The input and weights of the batch matrix multiplication module BMM are segmented according to a certain size (divisible by the parallelism of the reconfigurable computing core RCC. When the parallelism is 256, 4, 8, or 16 can be taken), divided into different batches, and the input of each batch only performs matrix multiplication operations with the weights of the corresponding batch.

[0022] Furthermore, the present invention also provides an accelerator for 3D point cloud data processing, including a reconfigurable computing core RCC. The reconfigurable computing core RCC includes T groups of parallel computing units, and each computing unit has Q multipliers inside, which are used to calculate the product of the input and the weights; the Q multipliers are divided into groups of three each, and the remaining less than three are merged with the previous group. The outputs of each group are added using adders, and the result is used as one of the outputs Psum2 of the reconfigurable computing core RCC; the results of all groups inside each computing unit are accumulated through accumulators to obtain the output of each group, which is used as another output Psum1 of the reconfigurable computing core RCC.

[0023] The reconfigurable computing core RCC completes the calculations in the neural network by adapting to the data streams of different layers, specifically including:

[0024] For two different linear layers Linear1 and Linear2: In the neural network, Linear1 takes the spatial coordinates (x, y, z) as input, obtains the output Psum2 in the reconfigurable computing core RCC as the calculation result and connects to the post-processing module, and performs the next processing according to the connection in the neural network; the input of Linear2 is the feature value of the point cloud. During the calculation, the feature value of the point cloud will be segmented into small blocks with sizes that match the parallelism of the reconfigurable computing core RCC. Each block is separately transferred onto the chip. After being calculated by the reconfigurable computing core RCC, the output Psum1 is obtained as the calculation result and connected to different activation functions of the post-processing module;

[0025] The post-processing module includes a batch processing and rectification module BRM and a normalization exponential function module SFM. The batch processing and rectification module BRM is used to complete the functions of the batch normalization layer Batch normalization and the rectified linear unit layer ReLU, and the normalization exponential function module SFM is used to process the Softmax function;

[0026] For the batch matrix multiplication module BMM: The batch matrix multiplication module BMM calculates N batches of inputs simultaneously, transfers the N batches of inputs and weights from off-chip to the on-chip cache, and processes them in parallel on the reconfigurable computing core RCC. The output Psum2 is used as the calculation result and connected to different activation functions of the post-processing module.

[0027] For the convolutional layer Conv: The input cache fetches data of N channels from off-chip at one time and processes them in parallel on the reconfigurable computing core RCC. After calculation on the reconfigurable computing core RCC, the calculation result Psum2 is obtained, and the results of M output channels are synchronously output. N = 8, 16, 32..., the upper limit of N is the parallelism of the reconfigurable computing core RCC, and the value of M is obtained by dividing the parallelism of the reconfigurable computing core RCC by N.

[0028] The reconfigurable computing core RCC adopts a local parallel computing strategy. The on-chip cache fetches the inputs and weights required by the reconfigurable computing core RCC in one calculation from off-chip each time. After calculation by the reconfigurable computing core RCC, the intermediate result A1 is not output to off-chip, but first stored on-chip, and then the intermediate result A1 is used as the input of the next layer, i.e., the convolutional layer Conv, and directly enters the reconfigurable computing core RCC for the next layer of calculation. Then, the output of the reconfigurable computing core RCC is output as the results of the two layers of the batch matrix multiplication module BMM and the convolutional layer Conv to the output cache and then transported to off-chip.

[0029] The local parallel computing strategy is applied to Linear2 and the batch matrix multiplication module BMM. A part of the inputs of Linear2 are loaded from off-chip to the on-chip cache and calculated using the reconfigurable computing core RCC. The obtained calculation result is used as the intermediate result A2. This part of the intermediate result A2 is sent to a register array of size M×N. After K cycles, the register array is filled with the local results of Linear2. K = 8, 16, 32... (the upper limit of K is the parallelism of the reconfigurable computing core RCC). At this time, the data in the register array is sent to the reconfigurable computing core RCC for the calculation of the batch matrix multiplication module BMM. After the calculation is completed, the results of the two layers of the batch matrix multiplication module BMM and the convolutional layer Conv are sent to the output cache and transferred to off-chip.

[0030] The present invention has the following beneficial effects:

[0031] 1. A brand-new lightweight network for processing 3D point cloud classification problems, which aggregates neighborhood features through learnable weights. Compared with Pointnet, it can still maintain the same accuracy under the condition that the network parameters are compressed by more than 30 times.

[0032] 2. A reconfigurable computing core for hardware acceleration to support LPN.

[0033] 3. A locally parallel computing strategy that greatly reduces the on-chip memory access requirements and off-chip memory access times by splitting and fusing computationally intensive layers.

[0034] 4. The lightweight point cloud classification network (LPN) proposed in the present invention has 44.4K parameters, and the accuracy can still reach 91.8% when the number of parameters is compressed by more than 30 times.

[0035] 5. The present invention can compress the number of parameters of the neural network by more than 30 times, and the hardware inference speed is about 35 times faster than that of the CPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0037] Figure 1a It is a specific structural schematic diagram of the feature extraction module LCEM.

[0038] Figure 1b It is a specific structural schematic diagram of the neural network LPN.

[0039] Figure 2 It is a specific structural schematic diagram of the reconfigurable computing core RCC.

[0040] Figure 3 It is a schematic diagram of the input and weight scheduling when different modules use the same reconfigurable computing core RCC.

[0041] Figure 4 It is a schematic diagram of the locally parallel computing strategy.

[0042] Figure 5 It is a schematic diagram of the overall hardware architecture designed based on the neural network LPN. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to reduce the number of parameters of the point cloud processing network, this solution proposes a lightweight neural network LPN to handle the point cloud classification problem. The neural network LPN mainly consists of a feature extraction module LCEM, an LBR block, a max pooling layer, a downsampling layer, and a fully connected layer FC. The LBR block represents a combined module of a linear layer, a batch normalization layer, and a ReLU layer. The specific structures of the feature extraction module LCEM and the neural network LPN are as Figure 1a 、 Figure 1bAs shown in the figure; the feature extraction module LCEM includes a linear layer Linear1 and Linear2, a normalization layer BN layer, a rectified linear unit layer ReLU layer, a batch matrix multiplication layer BMM layer, and a convolutional layer Conv layer.

[0044] The feature extraction module LCEM fuses the features of the neighborhood of the sampling points, improving the sampling efficiency. Through the max pooling layer and the downsampling layer, the input size of the network decreases layer by layer. At the same time, the use of the feature extraction module LCEM ensures that the model accuracy can still be maintained under the condition of reducing the input size.

[0045] Feature extraction module LCEM:

[0046] The feature extraction module is mainly used to extract local features. As shown in the above figure (a), module A includes two different linear layers (denoted as Linear1 and Linear2 respectively), a batch normalization layer, a ReLU layer, and a softmax layer. Module B contains a batch matrix multiplication module BMM and a convolutional layer Conv. Module A takes the position coordinates of the points in the neighborhood of the sampling points as input for training to obtain the weight values of the neighborhood points. Module B multiplies the weight values obtained by the training of module A by the feature values of the neighborhood points to perform weighted fusion on the neighborhood features.

[0047] Batch matrix multiplication module BMM:

[0048] The batch matrix multiplication module BMM is similar to the linear layer, both based on matrix multiplication operations. The difference is that the input and weights of the batch matrix multiplication module BMM are divided into different batches, and the input of each batch only performs matrix multiplication operations with the corresponding batch of weights.

[0049] Hardware architecture part:

[0050] Reconfigurable computing core (RCC):

[0051] As can be seen in the previous part, the layers in the neural network LPN that involve a large amount of calculations are mainly the following types: linear layer, convolutional layer, batch matrix multiplication module BMM. These layers are all based on multiply-accumulate operations. Therefore, the present invention designs a reconfigurable computing core RCC to parallelly process a large amount of calculations in the neural network LPN. The specific structure of the reconfigurable computing core RCC is as Figure 2 shown.

[0052] There are a total of T groups of parallel computing units in the reconfigurable computing core RCC. Each computing unit has Q multipliers inside, which are used to calculate the product of the input and the weight. Since the input of a part of the linear layer is the three-dimensional coordinates representing the position of points, Q multipliers are divided into groups of 3 each (the remaining less than three are combined with the previous group). The outputs of each group are added using adders, and the result is used as one of the outputs Psum2 of the reconfigurable computing core RCC. And the results of all groups inside each computing unit are accumulated through an accumulator to obtain the output of each group, which is also another output Psum1 of the reconfigurable computing core RCC. In actual design, T is set to 128 and Q is set to 16.

[0053] Adapting to data streams of different layers:

[0054] The main computing layers in the neural network LPN, including the linear layer, the convolutional layer, and the batch matrix multiplication module BMM, can all be uniformly calculated using the reconfigurable computing core RCC, but there are differences in the scheduling of the input and the weight. The present invention designs data streams adapting to different layers to complete the calculations in the network. As Figure 3 shown.

[0055] Linear:

[0056] For the linear layer, RCC mainly completes the matrix multiplication operation in the linear layer. It should be noted that Linear1 in LPN takes the spatial coordinates (x, y, z) as the input, so the output Psum2 in the reconfigurable computing core RCC is obtained as the calculation result and processed in the next step. And the input of Linear2 is the eigenvalue of the point cloud, which is usually an integer multiple of Q. Therefore, during the calculation, it will be divided into small pieces, and after being calculated by the reconfigurable computing core RCC, the output Psum1 is obtained as the calculation result and subsequent processing is carried out.

[0057] Batch matrix multiplication module BMM:

[0058] The batch matrix multiplication module BMM calculates N batches of inputs simultaneously, transfers the N batches of inputs and weights from off-chip to the on-chip cache, and processes them in parallel on the reconfigurable computing core RCC. The output Psum2 is used as the calculation result for the next step.

[0059] Convolutional layer Conv:

[0060] The input cache fetches data of N channels from off-chip at a time and processes them in parallel on the reconfigurable computing core RCC. After being calculated on the reconfigurable computing core RCC, the calculation result Psum2 of the reconfigurable computing core RCC is synchronously output as the result of M output channels.

[0061] Local parallel computing strategy:

[0062] For the main computing layers in the neural network LPN, the present invention counts the on-chip cache and off-chip memory access times required for all layers. Through statistics, it is found that in the way of layer-by-layer calculation, due to the huge amount of data in the output of Linear2 and the input of the BMM layer, the on-chip cache requirement and off-chip memory access times will increase sharply. To solve this problem, the present invention designs a local parallel computing strategy. Each time the cache fetches a part of the input and weights from off-chip, after being calculated by the reconfigurable computing core RCC, the result is not output to off-chip, but first stored on-chip, and then this intermediate result is used as the input of the next layer and directly enters the reconfigurable computing core RCC for the calculation of the next layer. Then, the output of the reconfigurable computing core RCC is output to the output cache as the result of these two layers and then transported to off-chip.

[0063] The local parallel computing strategy is applied to Linear2 and the batch matrix multiplication module BMM. The specific process is as Figure 4 shown.

[0064] A part of the input of Linear2 is loaded from off-chip to the on-chip cache and calculated using the reconfigurable computing core RCC. After obtaining a part of the input, this part of the result is sent to the register array of size M×N. After K cycles, the register array is filled with the local results of Linear2. At this time, the data in the register array is sent to the reconfigurable computing core RCC for the calculation of the BMM layer. After this part of the calculation is completed, the results of the two layers are sent to the output cache and transferred to off-chip. The local parallel computing strategy avoids the off-chip memory access caused by a large amount of output data of the Linear2 layer being transferred to off-chip and then transferred back into the chip, and also reduces the requirement for on-chip cache.

[0065] Post-processing module:

[0066] Batch processing and rectification module BRM:

[0067] It is used to process the batch normalization layer Batch normalization and the rectified linear unit layer ReLU. BRM is implemented by a multiplier, an adder and a comparator. Since the input channels of the batch normalization layer are the same as the output channels of the previous layer, the scheduling strategy can be shared with the output of the previous layer, and the parallelism is the same as that of the reconfigurable computing core RCC, still set to 128.

[0068] Normalized exponential function module SFM:

[0069] A module used to process the Softmax function. The analytical formula of the Softmax function is as follows:

[0070]

[0071] N represents the total number of all categories, x represents the currently selected category, and e is the natural constant.

[0072] Since e x is difficult to implement on hardware, the method in the literature "C.R. Qi, L.Yi, H.Su, and L.J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017" is adopted to approximate e x When the input is restricted within the range of (-1, 0), e x can be approximated as a quadratic function. After simulation, the result of the following function f(x) is extremely close to e x while greatly reducing the computational complexity without affecting the final inference result of the network:

[0073] f(x) = 0.3585×(X + 1.353) 2 + 0.344

[0074] Overall hardware architecture:

[0075] The overall hardware architecture designed based on the neural network LPN is as Figure 5 shown:

[0076] For each layer in the neural network LPN, the input cache and weight cache transfer data from off-chip to on-chip. The reconfigurable computing core RCC calculates the input of each layer in sequence. The output result of the reconfigurable computing core RCC is connected to a selector to select the subsequent connected post-processing module. The output result is temporarily stored in the output cache and finally transmitted to off-chip through the bus. The logic controller sends signals to control the operation of the entire on-chip logic.

[0077] Embodiment

[0078] The lightweight 3D point cloud classification neural network LPN significantly reduces the network parameter quantity while maintaining an accuracy comparable to that of Pointnet (Reference: C.R. Qi, L. Yi, H. Su, and L.J. Guibas. Pointnet++: Deep hierarchical feature learning on pointsets in a metric space. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017), as shown in Table 1 below:

[0079] Table 1

[0080] Parameters / Ratio Latency(ms) Accuracy Pointnet++[3] 1.75M / 1 116.59 91.9％ O-pointnet[1] 1.25M / 1.4 - 88.5％ LPN(ours) 44.43K / 39.39 20.29 91.8％

[0081] The hardware accelerator designed based on LPN is deployed on the Xilinx Kintex UltraScale KCU150 FPGA board. As shown in Table 2 below, compared with GPUs and CPUs, a significant improvement in the same network processing speed is achieved.

[0082] Table 2

[0083] Platform CPU GPU Ours(FPGA) Latency(ms) 20.3 12.1 0.57 Speedup 1× 1.68× 35.6×

[0084] Compared with previous FPGA accelerators for 3D point clouds, the design of the present invention has a higher throughput, as shown in Table 3 below:

[0085] Table 3

[0086]

[0087] The present invention designs a dedicated parallel reconfigurable computing core RCC and an efficient data stream for the neural network LPN. At a frequency of 100 MHz, the processing speed on the FPGA is about 35 times higher than that of the same network on the CPU. Compared with previous 3D point cloud network accelerators implemented on FPGA platforms, the throughput is increased by at least 1.5 times or more. Therefore, it can fully meet the real-time application requirements of point cloud data.

[0088] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit. Among them, the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, it can run the inventive content of a 3D point cloud data processing method based on a neural network and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0089] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of a computer program and its corresponding general hardware platform. Based on such an understanding, the essence of the technical solutions in the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a computer program, that is, a software product. The computer program software product can be stored in the storage medium and includes several instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) including a data processing unit to execute the methods described in each embodiment or some parts of the embodiments of the present invention.

[0090] The present invention provides a 3D point cloud data processing method and an accelerator based on a neural network. There are many methods and ways to specifically implement this technical solution. The above is only the preferred implementation mode of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by the prior art.

Claims

1. A method for processing 3D point cloud data based on a neural network, characterized in that, Including: Input 3D point cloud data into a neural network and output the target category represented by the point cloud; The neural network includes a feature extraction module LCEM, an LBR block, a max pooling layer, a downsampling layer, and a fully connected layer; The feature extraction module LCEM directly processes the input point cloud data, and fuses the features corresponding to the part of the points closest to the sampling points through a convolutional layer in the data; The LBR block is used for feature extraction, and the LBR block includes a linear layer Linear, a batch normalization layer Batchnormalization, and a rectified linear unit layer ReLU; The max pooling layer and the downsampling layer reduce the input size of the neural network by inputting multiple values and outputting a single value; when the pooling layer size is 4, the max pooling layer inputs 4 values each time and outputs the maximum value among the 4 values, and the downsampling layer reduces the output size to half of the input by setting the stride of the convolutional layer to 2 for each layer; The fully connected layer directly outputs the probabilities of each category to which the target belongs. The fully connected layer is used to output the probabilities of each category corresponding to the target and output according to the category with the highest probability; The 3D point cloud data includes a series of scattered points disordered in space, which contains the position features of objects in space; The feature extraction module LCEM includes module A and module B; Module A includes two different linear layers, a batch normalization layer Batch normalization, a rectified linear unit layer ReLU, and a softmax layer; The softmax layer is used to convert the input features into the probabilities of the categories to which the target belongs; The two different linear layers are respectively denoted as Linear1 and Linear2; the linear layers Linear1 and Linear2 process the input through matrix multiplication, the batch normalization layer Batch normalization normalizes the input, and the rectified linear unit layer ReLU introduces non-linearity into the neural network as an activation function; The batch normalization layer Batch normalization and the rectified linear unit layer ReLU in module A are the batch normalization layer Batch normalization and the rectified linear unit layer ReLU in the LBR block, and are used more than once in the neural network; Module B includes a batch matrix multiplication module BMM and a convolutional layer Conv, which are used to fuse the features of the neighboring points of the input points; Module A takes the position coordinates of the points in the neighborhood of the sampling points as input for training to obtain the weight values of the neighboring points; module B multiplies the weight values obtained by the training of module A by the feature values of the neighboring points to perform weighted fusion on the neighboring features; The input and weights of the batch matrix multiplication module BMM are divided according to a certain size into different batches, and the input of each batch only performs matrix multiplication with the weights of the corresponding batch.

2. An accelerator for 3D point cloud data processing implemented by the method according to claim 1, characterized in that It includes a reconfigurable computing core RCC. The reconfigurable computing core RCC includes T groups of parallel computing units. Each computing unit has Q multipliers inside, which are used to calculate the product of the input and the weight. The Q multipliers are divided into groups of three each, and the remaining less than three are combined with the previous group. The outputs of each group are added using an adder, and the result is used as one of the outputs Psum2 of the reconfigurable computing core RCC. The results of all groups inside each computing unit are accumulated through an accumulator to obtain the output of each group, which is used as another output Psum1 of the reconfigurable computing core RCC.

3. The accelerator for point cloud data processing according to claim 2, wherein The reconfigurable computing core RCC completes the calculations in the neural network by adapting to data streams of different layers, specifically including: For two different linear layers Linear1 and Linear2: In the neural network, Linear1 takes the spatial coordinates (x, y, z) as the input. The output Psum2 in the reconfigurable computing core RCC is obtained as the calculation result and connected to the post-processing module for further processing according to the connections in the neural network. The input of Linear2 is the feature values of the point cloud. During the calculation, the feature values of the point cloud are segmented into small blocks with sizes that match the parallelism of the reconfigurable computing core RCC. Each block is separately loaded onto the chip. After being calculated by the reconfigurable computing core RCC, the output Psum1 is obtained as the calculation result and connected to different activation functions of the post-processing module. The post-processing module includes a batch processing and rectification module BRM and a sigmoid function module SFM. The batch processing and rectification module BRM is used to complete the functions of the batch normalization layer Batch normalization and the rectified linear unit layer ReLU. The sigmoid function module SFM is used to process the Softmax function. For the batch matrix multiplication module BMM: The batch matrix multiplication module BMM calculates the inputs of N batches simultaneously. The inputs and weights of N batches are loaded from off-chip into the on-chip cache and processed in parallel on the reconfigurable computing core RCC. The output Psum2 is used as the calculation result and connected to different activation functions of the post-processing module. For the convolutional layer Conv: The input cache fetches data of N channels from off-chip at one time and processes them in parallel on the reconfigurable computing core RCC. After being calculated on the reconfigurable computing core RCC, the calculation result Psum2 is obtained. The results of M output channels are synchronously output. N = 8, 16, 32..., and the upper limit of N is the parallelism of the reconfigurable computing core RCC. The value of M is obtained by dividing the parallelism of the reconfigurable computing core RCC by N.

4. An accelerator for point cloud data processing according to claim 3, characterized in that, The reconfigurable computing core RCC adopts a local parallel computing strategy. Each time the on-chip cache fetches the inputs and weights required by the reconfigurable computing core RCC in one calculation from off-chip. After being calculated by the reconfigurable computing core RCC, the intermediate result A1 is not output to off-chip, but first stored on-chip, and then the intermediate result A1 is used as the input of the next layer, i.e., the convolutional layer Conv, and directly enters the reconfigurable computing core RCC for the next layer of calculation. Then, the output of the reconfigurable computing core RCC is output to the output cache as the results of both the batch matrix multiplication module BMM and the convolutional layer Conv, and then transported to off-chip.

5. An accelerator for point cloud data processing according to claim 4, characterized in that The local parallel computing strategy is applied to Linear2 and the batch matrix multiplication module BMM. A part of the inputs of Linear2 are loaded from off-chip to the on-chip cache and calculated using the reconfigurable computing core RCC. The obtained calculation results are used as the intermediate result A2. This part of the intermediate result A2 is sent to the register array of size M×N. After K cycles, the register array is filled with the local results of Linear2, where K = 8, 16, 32... Then, the data in the register array is sent to the reconfigurable computing core RCC for the calculation of the batch matrix multiplication module BMM. After the calculation is completed, the results of both the batch matrix multiplication module BMM and the convolutional layer Conv are sent to the output cache and transferred to off-chip.

Images