Sparse neural network accelerator for point cloud data

Abstract

The invention discloses a sparse neural network accelerator for point cloud data, and relates to the technical field of accelerators, and the method specifically comprises the steps: a data preprocessing module carries out the preprocessing of point cloud voxel data inputted by a data input module, and obtains sparse feature data, and stores the sparse feature data in an off-chip RAM; the DMA data transmission module is used for carrying weight data and feature data of the off-chip RAM; the weight input module is used for transmitting weight data to the calculation array module; the feature input module is used for compressing and multiplexing sparse feature data and then transmitting the sparse feature data to the calculation array module; the calculation array module comprises a plurality of PE units and is used for carrying out multiply-accumulate calculation on input feature data and weight data, outputting calculated result data, processing the result data through the post-processing module and sending the result data to the output module for caching; and finally, the data is carried to an off-chip RAM through a DMA data transmission module. The accelerator is suitable for a neural network reasoning task of point cloud data, and aims at improving the calculation performance and the processing efficiency in the reasoning task process.

Description

Technical Field

[0001] The present invention relates to the field of accelerator technology, and in particular to a sparse neural network accelerator for point cloud data. Background Art

[0002] In terms of hardware computing architecture, a hardware accelerator is a computing device specifically designed to accelerate specific tasks. It uses a customized chip design to optimize specific types of computing workloads. Hardware accelerators can be optimized for a variety of application areas, including artificial intelligence (AI), machine learning, data analysis, cryptography, and graphics processing.

[0003] Existing general-purpose architectures (e.g., GPUs) have difficulty unleashing the potential of point cloud algorithms for two reasons: (1) GPUs are optimized for computationally intensive workloads with regular execution patterns, such as neural networks. In contrast, point cloud algorithms contain both regular feature computation operations and irregular neighborhood searches and aggregation operations, resulting in low computational utilization and efficiency of GPUs. (2) Bit string-based incremental processing involves bit-to-bit processing, which GPUs cannot support well.

[0004] In terms of algorithms, point cloud target detection algorithms are designed to identify target objects in a given scene and achieve target positioning. However, with the development of artificial intelligence and deep learning, the development speed of 3D point cloud target detection algorithms is gradually accelerating.

[0005] In convolutional neural networks, they can be roughly divided into methods based on raw point clouds and methods based on voxels. Currently, the voxel-based method has become the mainstream in academia. For point cloud accelerators, there are successful solutions for the quantization of point cloud algorithms and the irregular neighbor search and aggregation operations included in the processing of voxels in some accelerators. However, for the nonlinear function ReLU, which is widely used in the 3D sparse convolution calculation process, about half of the elements are zero. The irregularity of sparsity will lead to an unbalanced workload of the data flow after sparse processing, resulting in performance degradation. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a sparse neural network accelerator for point cloud data, aiming to improve the efficiency of the neural network in the reasoning process and reduce unnecessary computing overhead.

[0007] The present invention proposes a sparse neural network accelerator for point cloud data, the accelerator is a heterogeneous computing platform composed of a CPU and an FPGA, the CPU includes a data input and output module, a data preprocessing module and a function instruction module, the FPGA includes a DMA data transmission module, a weight input module and a feature input module, a computing array module, a post-processing module, an output module and a control module;

[0008] The data input module is used for receiving and sending external data required in point cloud neural network reasoning;

[0009] The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation;

[0010] The functional instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module;

[0011] The DMA (direct memory access) data transmission module is used to efficiently transmit data between the external RAM and the FPGA;

[0012] The weight input module is used to transfer the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module;

[0013] The feature input module is used to perform feature compression and data multiplexing on the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module;

[0014] The calculation array module is composed of a plurality of identical PE units, and weight data is stored in these PE units. After the sparse feature data is multiplied and accumulated by the weight data of each PE unit, the result data is cached in the result FIFO, and the result data is taken out from the result FIFO;

[0015] The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the accumulated result data, and the processed data is sent to the output module;

[0016] The output module is used to cache the accumulated result data, and then the data is transferred to the off-chip RAM via the DMA data transmission module;

[0017] The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

[0018] Furthermore, the computing array module is composed of X×Y PE units, which are arranged in a rectangular shape, and adjacent PE units are interconnected for performing parallel multiplication and accumulation calculations on feature data; each PE unit contains a weight input layer, a feature input layer, a result output layer, a weight RAM and a result FIFO; the weight input layer has the functions of receiving, storing and transmitting weight data; the feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, multiplication and accumulation calculations, temporarily storing partial sums, and outputting calculation result data; the result output layer has the functions of receiving result data, accumulating result data and transmitting result data; the weight RAM stores weight data; and the result FIFO caches result data.

[0019] Furthermore, the feature input module removes zero elements from the feature data and caches them in the input FIFO. The processed feature data will be sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO will circulate into the calculation array modules of different input channels, thereby balancing the load of each channel.

[0020] Furthermore, in the calculation array module, the weight data enters the weight input layer of the PE unit, and by judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data not belonging to the PE unit is passed to the next PE unit; when the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and added with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; when the result data enters the result output layer of the PE unit, it is added with the corresponding result data in the result FIFO and passed to the next PE unit.

[0021] The weight data stream is transmitted longitudinally in the computing array module, and the feature data stream is transmitted transversely in the computing array module; the result data stream is transmitted longitudinally in the computing array.

[0022] The advantages of the sparse neural network accelerator for point cloud data provided by the present invention are: the sparse neural network accelerator for point cloud data provided in the structure of the present invention combines the processor (CPU) and the programmable logic (FPGA) part, and significantly improves the efficiency of neural network reasoning by making full use of data sparsity, parallel computing and load balancing optimization strategies, and realizes sparse processing of point cloud data and sparse convolutional neural network accelerated reasoning; in addition, the feature input module dynamically balances the load of each channel according to the sparsity of the feature data, ensuring that all computing array modules can work under the load balancing state, greatly avoiding the idleness of the computing array module, thereby improving the utilization rate of the computing array module; the computing array module is composed of multiple PE units, which can perform computing tasks in parallel. Compared with the serial computing in the general architecture, the accelerator realizes large-scale parallel processing in hardware through the two-dimensional PE array design, can perform multiple computing tasks at the same time, and greatly improves the overall throughput of the system; the DMA data transmission module is used to realize the high-speed transmission of large blocks of data between the CPU and the FPGA, reducing the direct participation of the CPU, so that the CPU resources can be used more for the control and management of high-level tasks, without the need to frequently process large data volume movement tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural schematic diagram of the present invention;

[0024] Figure 2 It is a schematic diagram of the internal structure of the computing array module and PE unit;

[0025] Figure 3 This is a schematic diagram of the internal structure of the feature data input module;

[0026] Figure 4 A schematic diagram of inputting sparse feature data into a feature data input module;

[0027] Figure 5 It is the flow chart of input FIFO data processing in the feature data input module;

[0028] Figure 6 It is a flow chart of the circular FIFO data processing in the characteristic data input module;

[0029] Figure 7 This is a 2D convolution calculation flow chart;

[0030] Figure 8 This is a flowchart for 3D sparse convolution calculation;

[0031] Fig. 9 It is a schematic diagram of each data flow in the computing array module;

[0032] Fig.10Schematic diagram of the weight data flow in the weight input layer;

[0033] Fig.11 Schematic diagram of feature data flow in the feature input layer;

[0034] Fig.12 This is a schematic diagram of the result data flow in the result output layer;

[0035] Fig.13 Schematic diagram of data flow when the accelerator is working. DETAILED DESCRIPTION

[0036] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0037] 1. Accelerator Architecture

[0038] like Figures 1 to 13 As shown, the present invention proposes a sparse neural network accelerator for point cloud data, the accelerator is a heterogeneous computing platform composed of a CPU and an FPGA, the CPU includes a data input and output module, a data preprocessing module and a function instruction module, the FPGA includes a DMA data transmission module, a weight input module, a feature input module, a calculation array module and a post-processing module, an output module and a control module;

[0039] The data input and output module is used for receiving and sending external data required in point cloud neural network reasoning;

[0040] The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation;

[0041] The functional instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module;

[0042] The DMA (direct memory access) data transmission module is used to efficiently transmit data between the external RAM and the FPGA;

[0043] The weight input module is used to transfer the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module;

[0044] The feature input module is used to compress and reuse the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module; the feature input module is as follows: Figure 3 As shown, the module removes zero elements from the feature data and caches them in the input FIFO. The processed feature data will be sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO will circulate into the calculation arrays of different input channels, thereby balancing the load of each channel.

[0045] The calculation array module is composed of a plurality of identical PE units, and weight data is stored in these PE units. After the sparse feature data is multiplied and accumulated by the weight data of each PE unit, the result data is cached in the result FIFO, and the result data is taken out from the result FIFO;

[0046] The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the accumulated result data, and the processed data is sent to the output module;

[0047] The output module is used to cache the accumulated result data, and then the data is transferred to the off-chip RAM via the DMA data transmission module;

[0048] The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

[0049] The calculation array module is composed of X×Y PE units, which are arranged in a rectangular shape. Adjacent PE units are interconnected to perform parallel multiplication and accumulation calculations on feature data. Each PE unit contains a weight input layer, a feature input layer, a result output layer, a weight RAM, and a result FIFO. The specific structure is as follows: Figure 2 As shown. The weight input layer has the functions of receiving, storing and transmitting weight data; the feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, multiplying and accumulating, temporarily storing partial sums, and outputting calculation result data; the result output layer has the functions of receiving result data, accumulating result data and transmitting result data; the weight RAM stores weight data; and the result FIFO caches result data.

[0050] In the calculation array module, the weight data enters the weight input layer of the PE unit. By judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data not belonging to the PE unit is passed to the next PE unit.

[0051] When the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and added with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; when the result data enters the result output layer of the PE unit, it is added to the corresponding result data in the result FIFO and passed to the next PE unit.

[0052] The weight data stream is transmitted longitudinally in the calculation array module, the feature data stream is transmitted transversely in the calculation array module; the result data stream is transmitted longitudinally in the calculation array module.

[0053] This embodiment mainly addresses the problems of reduced efficiency and load imbalance caused by data sparsity in the calculation of 3D sparse convolutions in point cloud networks. Combined with the heterogeneous computing characteristics of CPU and FPGA, efficient sparse convolutional neural network reasoning is achieved. Through a series of hardware optimization measures, this embodiment not only improves the efficiency of sparse neural networks in point cloud data processing, but also provides an excellent solution for high real-time and large data processing scenarios. That is, this embodiment implements and hardware accelerates 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and matrix operations in the calculation process of the point cloud neural network algorithm, and effectively compresses sparse data with a large number of 0 elements generated by the calculation characteristics in 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and matrix operations; a unique input module and computing array module are designed to complete the above operations and achieve hardware acceleration.

[0054] Therefore, the network accelerator of this embodiment combines the processor (CPU) and programmable logic (FPGA) parts, and significantly improves the efficiency of neural network reasoning by making full use of data sparsity, parallel computing and load balancing optimization strategies, and realizes sparse processing of point cloud data and accelerated reasoning of sparse convolutional neural networks. It solves the performance bottleneck problem of traditional general computing architecture when processing sparse data, and provides an efficient and low-energy solution for applications that need to process point cloud data.

[0055] This accelerator is designed for highly sparse point cloud data, aiming to improve the efficiency of neural networks during reasoning and reduce unnecessary computing overhead, especially in scenarios that require high real-time and big data processing.

[0056] 2. Architecture Division

[0057] A heterogeneous computing platform consisting of CPU and FPGA is used. The CPU part is responsible for data flow control and high-level tasks such as data preprocessing and control instructions, while the FPGA part is used to accelerate the core calculation of sparse neural networks.

[0058] 1.CPU part:

[0059] The CPU is responsible for loading the external data required for point cloud neural network reasoning. Then the data is preprocessed. In addition to data management, the CPU is also responsible for high-level decision-making and control of reasoning results. Depending on the application scenario and execution task, the processor can perform corresponding operations according to demand and output control instructions.

[0060] 2.FPGA part:

[0061] The FPGA part is mainly used to accelerate neural network reasoning. Using the parallel computing capabilities of FPGA, sparse neural network operations can be greatly accelerated. Specifically, it includes 3D sparse convolution, 3D submanifold sparse convolution, 2D convolution, and hardware acceleration of activation functions. In order to improve the computing efficiency of neural networks, custom hardware modules will be designed in FPGA to handle sparse data indexing, storage, computing optimization and other functions.

[0062] Through the above settings, hardware acceleration of 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and nonlinear operations in the calculation process of the point cloud neural network algorithm is achieved.

[0063] 3. As an embodiment

[0064] In this embodiment, the specific implementation process of the sparse neural network accelerator for point cloud data combines the heterogeneous computing characteristics of CPU and FPGA, uses CPU for control and data management, and uses FPGA for efficient computing, forming a software and hardware collaborative working mode. The overall architecture is as follows: Figure 1 The following is a description of the specific implementation process of this embodiment:

[0065] 1. Data acquisition and input

[0066] After the system starts, the CPU first obtains weight data and point cloud voxel data from external devices (such as high-speed interfaces and storage devices). Weight data is integer data quantized to 8 bits. Voxel data consists of three-dimensional index coordinates and feature data quantized to 8 bits. After the data is acquired, the CPU stores the data in the off-chip RAM for subsequent data preprocessing and calculation.

[0067] 2. Data Preprocessing

[0068] The three-dimensional index coordinate data is input to the preprocessing module, which aggregates the feature data by performing kernel search, neighborhood search and other operations on the coordinates according to the requirements of neural network reasoning to form a continuous feature data stream. The data stream is transmitted to the FPGA part through the DMA data transmission module.

[0069] 3. Data transmission and control

[0070] In order to speed up data transmission, the DMA data transmission module is responsible for transferring data from the off-chip RAM to the corresponding modules in the FPGA (weight input module, feature input module). The efficient transmission mechanism of the DMA data transmission module can reduce the involvement of the CPU, allowing it to use more resources for management and control tasks without being slowed down by the transmission process of large amounts of data.

[0071] In this process, the CPU configures the registers in the control module through the AXI_Lite bus, thereby achieving the purpose of configuring parameters and managing the working status of each module. The control signal sent by the CPU ensures the synchronous execution of each module and ensures the correctness of the data flow during transmission and calculation.

[0072] 4. Sparse neural network inference acceleration

[0073] The accelerated reasoning of sparse neural networks is mainly completed through the weight input module, feature input module, calculation array module, post-processing module, and output module, which are used to process the computing tasks in 3D sparse convolution, 3D submanifold sparse convolution, and 2D convolution operations.

[0074] Among them, the 2D convolution calculation process is as follows Figure 7 As shown, C i is the number of input channels, C o is the number of output channels, H k is the side length of the convolution kernel, I is the input feature data, W is the weight data, and O is the output result data. The process can be described as formula (1):

[0075]

[0076] Similarly, the 3D sparse convolution calculation process is as follows Figure 8 As shown, the process can be described as formula (2):

[0077]

[0078] Since the convolution kernel of 2D convolution is a planar rectangle, a and b represent the lengths of the two sides of the convolution kernel; and the convolution kernel of 3D sparse convolution is a cube, a, b, and c represent the lengths of the three sides of the convolution kernel.

[0079] 4.1 Weight input and distribution

[0080] In the convolution operation of sparse neural networks, weight is the core parameter. The accelerator transfers weight data from the off-chip RAM to the weight input module through the DMA data transfer module. The weight input module marks the address information of these weight data and sends it to the calculation array module. The weight data enters the weight input layer of each PE unit in the calculation array. The PE unit determines the address information carried by the weight data and stores the weight data belonging to the PE unit in the weight RAM. The weight data that does not belong to the PE unit is passed to the next PE unit. The distribution method is as follows: Fig.10 As shown, this ensures that each PE unit can obtain the correct weight and participate in parallel calculations.

[0081] 4.2 Input, Compression and Recycling of Feature Data

[0082] After the DMA data transfer module has transmitted the weight data and the data preprocessing module has processed the point cloud voxel data, the feature data stream is obtained. At this time, the feature data stream is transferred from the off-chip RAM to the feature input module through the DMA data transfer module. The feature input module workflow is as follows: Figure 4 As shown, first, after the feature data stream passes through the input FIFO, the zero element features in the data stream will be removed to achieve the purpose of compressing the feature data. The dynamic process of processing data is as follows Figure 5 Then, the compressed feature data is sent to the calculation array and the circular FIFO, and the circular FIFO recycles the feature data. The dynamic process of processing data is as follows: Figure 6 As shown, the purpose of balancing the load of each channel is achieved, ensuring that the working efficiency of the entire computing array module is maximized.

[0083] 4.3 Operation process

[0084] The computation array module is designed to support convolution operations, especially sparse convolution operations. Figure 8-9 As shown, it can be seen that the implementation process of sparse convolution operation in the accelerator and the flow of weight data, feature data, and result data. As mentioned earlier, the weight input module sends the weight data and stores it in the weight RAM of each PE in the computing array. The feature input module removes invalid zero elements in the feature data stream and balances the data flow load of each channel. Therefore, when the feature data enters the feature input layer of the PE unit from the feature input module, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and added with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO. While participating in the multiplication and addition calculation, the feature data will also be transmitted to the feature input layer in the next PE. The flow method is as follows: Fig.11When all PEs complete the calculation of a feature data stream, the result data cached in each PE will enter the result output layer of the PE unit, where the result data is added and passed to the result output layer of the next PE unit. The calculation flow is as follows: Fig.12 The operation results will eventually be transmitted to the post-processing module for processing.

[0085] 5. Post-processing and output

[0086] After the calculation is completed, the system enters the post-processing stage. The post-processing module performs a series of operations on the result data output by the calculation array module, including the processing of nonlinear activation functions (such as ReLU) and bit width adjustment. The purpose of bit width adjustment is to map the convolution calculation results back to the appropriate output range to ensure that the subsequent processor can correctly parse the data.

[0087] The post-processed data will be passed to the output module in the FPGA, which caches the data and transfers the processing results back to the off-chip RAM through the DMA data transfer module, and finally handed over to the CPU for further processing or output.

[0088] 6.CPU control and decision making

[0089] After the FPGA completes the sparse convolution calculation and returns the result to the off-chip RAM, the CPU is responsible for high-level decision-making and control. Based on the inference results, the CPU can perform corresponding operations, such as outputting the inference results and issuing further calculation instructions.

[0090] In summary, the entire calculation process is as follows Fig.13 shown.

[0091] 7. Optimize overall system performance

[0092] In order to improve the overall performance of the system, this embodiment is optimized in the following aspects:

[0093] Sparsity Utilization: By designing dedicated hardware modules (CPU, FPGA), the sparsity in point cloud data and sparse neural networks can be fully utilized to reduce unnecessary calculations and save storage and bandwidth resources.

[0094] Parallel computing: The parallel processing capabilities of FPGA are fully utilized, and the collaborative work of a large number of PE unit computing arrays greatly improves computing efficiency.

[0095] Data transmission optimization: Use the DMA module to transmit data, reduce the CPU's involvement in data movement, and improve the overall throughput.

[0096] Dynamic load balancing: The characteristic data input module dynamically balances the load of each channel through circular FIFO processing according to the characteristics of the data flow, ensuring that the computing array can work efficiently under unbalanced data flow.

[0097] Data reuse: Through the weight input module, the weight data is stored in different PE units in a distributed manner. In this way, the weight data can be reused during convolution calculations, which significantly reduces bandwidth overhead. Through the circular FIFO in the feature input module, feature data can be reused multiple times, thereby minimizing the transmission overhead of the data stream and significantly improving the efficiency of convolution calculations.

[0098] The sparse neural network accelerator for point cloud data proposed in this embodiment is optimized for the sparsity characteristics of point cloud algorithm processing, and successfully overcomes the following problems existing in the background technology:

[0099] (1) Efficient sparse processing to reduce computational overhead

[0100] Traditional general hardware architectures such as GPUs cannot fully utilize the characteristics of sparse data, resulting in low computational efficiency when processing sparse point cloud data. This embodiment effectively performs sparse processing on the data by designing dedicated modules (modules in the CPU and FPGA), avoiding invalid calculations of zero-value elements, thereby greatly reducing computational overhead. By eliminating invalid zero elements in advance and skipping zero element calculations in sparse convolutions in the computational array, the computational efficiency can be improved, especially in scenarios with high sparsity. For example, in the processing of point cloud data, the computational efficiency of a typical sparse convolution layer can be increased by more than 2 times.

[0101] (2) Dynamic load balancing to ensure efficient operation

[0102] When processing sparse data, due to irregular data distribution, it is easy to cause unbalanced computing load, which in turn affects performance. The feature input module of this embodiment dynamically balances the load of each channel according to the sparsity of the feature data, ensuring that all computing array modules can work under load balancing, greatly avoiding the idleness of the computing array modules, thereby improving the utilization rate of the computing array modules. This dynamic scheduling mechanism allows the performance of sparse convolution to remain stable under conditions of data input of different densities, and can also achieve higher computing efficiency under load imbalance.

[0103] (3) Low power consumption and high real-time performance

[0104] This embodiment shows significant advantages in scenarios that require high real-time performance and low power consumption. By reducing the ineffective overhead caused by sparse calculations and efficient data flow control, the entire system can significantly reduce power consumption while maintaining high computing performance. In scenarios with high real-time requirements such as unmanned driving and robots, this system performs well and can quickly respond to and process large-scale point cloud data, providing a highly competitive hardware acceleration solution for application scenarios.

[0105] (4) Parallel computing acceleration to improve system throughput

[0106] The parallel processing capability of FPGA is fully utilized in this design. The computation array module consists of multiple PE units, which can perform sparse convolution operations in parallel. Compared with the serial computing in the general architecture, this accelerator realizes large-scale parallel processing in hardware through the two-dimensional PE array design, which can perform multiple convolution computing tasks at the same time, greatly improving the overall throughput of the system.

[0107] (5) Efficient data transmission, reducing CPU burden

[0108] The system uses a DMA data transfer module to achieve high-speed transmission of large blocks of data between the CPU and FPGA, reducing the direct involvement of the CPU, so that CPU resources can be used more for the control and management of high-level tasks without the need to frequently handle large amounts of data movement tasks. This design significantly reduces the occupancy rate of the system bus and improves the overall data transmission efficiency, especially in large-scale point cloud data processing, where data transmission efficiency is significantly improved.

[0109] (6) Software and hardware collaboration to give full play to the advantages of heterogeneous platforms

[0110] This embodiment makes full use of the heterogeneous computing architecture of CPU and FPGA, using the CPU for data management and system control tasks, and the FPGA part for hardware acceleration of sparse neural network reasoning. The CPU is responsible for loading point cloud voxel data, neighborhood search, invalid data removal, and sparse matrix construction, while the FPGA is responsible for large-scale sparse convolution calculations. Such hardware and software collaborative design enables the entire system to perform well when processing point cloud data with high sparsity, and significantly reduces the overall energy consumption of the system.

[0111] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A sparse neural network accelerator for point cloud data, characterized in that: The accelerator is a heterogeneous computing platform composed of a CPU and an FPGA, wherein the CPU includes a data input and output module, a data preprocessing module, and a function instruction module, and the FPGA includes a DMA data transmission module, a weight input module, a feature input module, a computing array module, a post-processing module, an output module, and a control module; The data input and output module is used for receiving and sending external data required in point cloud neural network reasoning; The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation; The functional instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module; The weight input module is used to transfer the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module; The feature input module is used to perform feature compression and data multiplexing on the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module; The calculation array module is composed of a plurality of identical PE units, weight data is stored in the PE units, feature data is multiplied and accumulated by the PE unit weight data, and the result data is cached in the result FIFO, and the result data is taken out from the result FIFO; The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the result data, and the processed data is sent to the output module; The output module is used to cache the result data, and then transfer it to the off-chip RAM through the DMA data transmission module; The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

2. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The computing array module is composed of X×Y PE units, which are arranged in a rectangular shape, and adjacent PE units are connected to each other, and are used to perform parallel multiplication and accumulation calculations on feature data; each PE unit includes a weight input layer, a feature input layer, a result output layer, a weight RAM and a result FIFO; The weight input layer has the functions of receiving, storing and transmitting weight data; The feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, multiplying and accumulating, temporarily storing partial sums, and outputting calculation result data; The result output layer has the functions of caching result data, receiving result data, accumulating result data and transmitting result data; Weight RAM stores weight data; The result FIFO buffers the result data.

3. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The feature input module removes zero elements from the feature data and caches them in the input FIFO. The processed feature data is sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO circulates into the calculation array modules of different input channels, thereby balancing the load of each channel.

4. The sparse neural network accelerator for point cloud data according to claim 2, characterized in that: In the calculation array module, the weight data enters the weight input layer of the PE unit. By judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data not belonging to the PE unit is passed to the next PE unit.

5. The sparse neural network accelerator for point cloud data according to claim 2, characterized in that: When the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and added with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; after the result data enters the result output layer of the PE unit, it is added with the corresponding result data in the result FIFO and passed to the next PE unit.

6. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The weight data stream is transmitted longitudinally in the calculation array module, the feature data stream is transmitted transversely in the calculation array module, and the result data stream is transmitted longitudinally in the calculation array module.

Images