A sparse neural network accelerator for point cloud data

Abstract

The present invention discloses a sparse neural network accelerator for point cloud data, which relates to the technical field of accelerators, and specifically comprises: a data preprocessing module preprocesses the point cloud voxel data input by a data input module to obtain sparse feature data and stores the data in an off-chip RAM; a DMA data transmission module is used to carry weight data and feature data in the off-chip RAM; a weight input module is used to deliver the weight data to a computing array module; a feature input module is used to compress and reuse the sparse feature data and then deliver it to the computing array module; the computing array module includes multiple PE units, which perform multiplication and accumulation calculations on the input feature data and weight data, and output the calculated result data, which is processed by a post-processing module and then delivered to an output module cache; and finally, the result data is transported to the off-chip RAM through the DMA data transmission module; the accelerator is suitable for neural network inference tasks of point cloud data, and aims to improve the computing performance and processing efficiency during the inference task.

Description

Technical Field

[0001] The present invention relates to the field of accelerator technology, and in particular to a sparse neural network accelerator for point cloud data. Background Art

[0002] In terms of hardware computing architecture, a hardware accelerator is a computing device specifically designed to accelerate specific tasks. It uses a customized chip design to optimize specific types of computing workloads. Hardware accelerators can be optimized for a variety of application areas, including artificial intelligence (AI), machine learning, data analysis, cryptography, and graphics processing.

[0003] Existing general-purpose architectures (e.g., GPUs) struggle to unleash the potential of point cloud algorithms for two reasons: (1) GPUs are optimized for computationally intensive workloads with regular execution patterns, such as neural networks. In contrast, point cloud algorithms involve both regular feature computation operations and irregular neighborhood searches and aggregation operations, resulting in low GPU computational utilization and efficiency. (2) Bit-string-based incremental processing involves bit-to-bit processing, which GPUs cannot well support.

[0004] In terms of algorithms, point cloud object detection algorithms aim to identify and locate target objects in a given scene. However, with the development of artificial intelligence and deep learning, the development of 3D point cloud object detection algorithms has gradually accelerated.

[0005] Convolutional neural networks can be broadly categorized into methods based on raw point clouds and voxel-based methods. Currently, voxel-based methods are the mainstream in academia. For point cloud accelerators, some accelerators have successfully implemented solutions for quantizing point cloud algorithms and implementing irregular neighbor searches and aggregation operations when processing voxels. However, for the widely used nonlinear function ReLU in 3D sparse convolution calculations, approximately half of its elements are zero. This irregularity in sparsity can lead to an imbalanced workload in the data stream after sparse processing, resulting in poor performance. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a sparse neural network accelerator for point cloud data, aiming to improve the efficiency of neural networks in the reasoning process and reduce unnecessary computing overhead.

[0007] The present invention proposes a sparse neural network accelerator for point cloud data. The accelerator is a heterogeneous computing platform composed of a CPU and an FPGA. The CPU includes a data input and output module, a data preprocessing module, and a function instruction module. The FPGA includes a DMA data transmission module, a weight input module, a feature input module, a calculation array module, a post-processing module, an output module, and a control module.

[0008] The data input module is used to receive and send external data required for point cloud neural network reasoning;

[0009] The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation;

[0010] The function instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module;

[0011] The DMA (direct memory access) data transmission module is used to efficiently transmit data between the external RAM and the FPGA;

[0012] The weight input module is used to transmit the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module;

[0013] The feature input module is used to compress and reuse the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module;

[0014] The calculation array module is composed of multiple identical PE units, and weight data is stored in these PE units. After the sparse feature data is multiplied and accumulated by the weight data of each PE unit, the result data is cached in the result FIFO, and the result data is taken out from the result FIFO;

[0015] The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the accumulated result data, and the processed data is sent to the output module;

[0016] The output module is used to cache the accumulated result data, and then the data is transferred to the off-chip RAM via the DMA data transmission module;

[0017] The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

[0018] Furthermore, the computing array module is composed of X×Y PE units, which are arranged in a rectangular shape, and adjacent PE units are interconnected for performing parallel multiplication and accumulation calculations on feature data; each PE unit contains a weight input layer, a feature input layer, a result output layer, a weight RAM and a result FIFO; the weight input layer has the functions of receiving, storing and transmitting weight data; the feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, performing multiplication and accumulation calculations, temporarily storing partial sums, and outputting calculation result data; the result output layer has the functions of receiving result data, accumulating result data and transmitting result data; the weight RAM stores weight data; and the result FIFO caches result data.

[0019] Furthermore, the feature input module removes zero elements from the feature data and caches it in the input FIFO. The processed feature data will be sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO will circulate into the calculation array modules of different input channels, thereby balancing the load of each channel.

[0020] Furthermore, in the calculation array module, the weight data enters the weight input layer of the PE unit, and by judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data that does not belong to the PE unit is passed to the next PE unit; when the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and accumulated with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; when the result data enters the result output layer of the PE unit, it is added to the corresponding result data in the result FIFO and passed to the next PE unit.

[0021] The weight data stream is transmitted longitudinally in the calculation array module, and the feature data stream is transmitted transversely in the calculation array module; the result data stream is transmitted longitudinally in the calculation array.

[0022] The advantages of the sparse neural network accelerator for point cloud data provided by the present invention are as follows: the sparse neural network accelerator for point cloud data provided in the structure of the present invention combines a processor (CPU) and a programmable logic (FPGA) part, and by fully utilizing data sparsity, parallel computing and load balancing optimization strategies, significantly improves the efficiency of neural network reasoning, and realizes sparse processing of point cloud data and sparse convolutional neural network accelerated reasoning; in addition, the feature input module dynamically balances the load of each channel according to the sparsity of the feature data, ensuring that all computing array modules can operate in a load-balanced state, greatly avoiding the idleness of the computing array modules, thereby improving the utilization rate of the computing array modules; the computing array modules are composed of multiple PE units and can execute computing tasks in parallel. Compared with serial computing in general architectures, this accelerator realizes large-scale parallel processing in hardware through a two-dimensional PE array design, can perform multiple computing tasks simultaneously, and greatly improves the overall throughput of the system; uses a DMA data transfer module to achieve high-speed transmission of large blocks of data between the CPU and FPGA, reducing the direct involvement of the CPU, so that CPU resources can be more used for the control and management of high-level tasks, without the need to frequently handle large-scale data movement tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural schematic diagram of the present invention;

[0024] Figure 2 It is a schematic diagram of the internal structure of the computing array module and PE unit;

[0025] Figure 3 This is a schematic diagram of the internal structure of the feature data input module;

[0026] Figure 4 Schematic diagram of inputting sparse feature data into the feature data input module;

[0027] Figure 5 This is the flow chart of input FIFO data processing in the feature data input module;

[0028] Figure 6 It is the flow chart of cyclic FIFO data processing in the feature data input module;

[0029] Figure 7 2D convolution calculation flow chart;

[0030] Figure 8 3D sparse convolution calculation flow chart;

[0031] Figure 9 Schematic diagram of each data flow in the calculation array module;

[0032] Figure 10Schematic diagram of the weight data flow in the weight input layer;

[0033] Figure 11 Schematic diagram of feature data flow in the feature input layer;

[0034] Figure 12 This is a schematic diagram of the result data flow in the result output layer;

[0035] Figure 13 Schematic diagram of data flow when working for the accelerator. DETAILED DESCRIPTION

[0036] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] 1. Accelerator Architecture

[0038] like Figures 1 to 13 As shown in the figure, the present invention proposes a sparse neural network accelerator for point cloud data. The accelerator is a heterogeneous computing platform composed of a CPU and an FPGA. The CPU includes a data input and output module, a data preprocessing module, and a function instruction module. The FPGA includes a DMA data transmission module, a weight input module, a feature input module, a calculation array module, a post-processing module, an output module, and a control module.

[0039] The data input and output module is used to receive and send external data required for point cloud neural network reasoning;

[0040] The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation;

[0041] The function instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module;

[0042] The DMA (direct memory access) data transmission module is used to efficiently transmit data between the external RAM and the FPGA;

[0043] The weight input module is used to transmit the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module;

[0044] The feature input module is used to compress and reuse the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module; the feature input module is as follows: Figure 3 As shown in the figure, the module removes zero elements from the feature data and caches it in the input FIFO. The processed feature data will be sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO will circulate into the calculation arrays of different input channels, thereby balancing the load of each channel.

[0045] The calculation array module is composed of multiple identical PE units, and weight data is stored in these PE units. After the sparse feature data is multiplied and accumulated by the weight data of each PE unit, the result data is cached in the result FIFO, and the result data is taken out from the result FIFO;

[0046] The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the accumulated result data, and the processed data is sent to the output module;

[0047] The output module is used to cache the accumulated result data, and then the data is transferred to the off-chip RAM via the DMA data transmission module;

[0048] The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

[0049] The calculation array module is composed of X×Y PE units, which are arranged in a rectangular shape. Adjacent PE units are interconnected and used to perform parallel multiplication and accumulation calculations on feature data. Each PE unit contains a weight input layer, a feature input layer, a result output layer, a weight RAM, and a result FIFO. The specific structure is as follows: Figure 2 As shown in the figure, the weight input layer has the functions of receiving, storing, and transmitting weight data; the feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, performing multiplication and accumulation calculations, temporarily storing partial sums, and outputting calculation result data; the result output layer has the functions of receiving result data, accumulating result data, and transmitting result data; the weight RAM stores weight data; and the result FIFO caches result data.

[0050] In the calculation array module, the weight data enters the weight input layer of the PE unit. By judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data that does not belong to the PE unit is passed to the next PE unit.

[0051] When the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data and the corresponding weight data are multiplied and added. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; when the result data enters the result output layer of the PE unit, it is added to the corresponding result data in the result FIFO and passed to the next PE unit.

[0052] The weight data stream is transmitted longitudinally in the calculation array module, the feature data stream is transmitted transversely in the calculation array module; the result data stream is transmitted longitudinally in the calculation array module.

[0053] This embodiment mainly addresses the problems of reduced efficiency and load imbalance caused by data sparsity in the calculation of 3D sparse convolution in point cloud networks. By combining the heterogeneous computing characteristics of CPU and FPGA, efficient sparse convolutional neural network inference is achieved. Through a series of hardware optimization measures, this embodiment not only improves the efficiency of sparse neural networks in point cloud data processing, but also provides an excellent solution for high real-time and large data processing scenarios. That is, this embodiment implements and hardware accelerates 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and matrix operations in the calculation process of the point cloud neural network algorithm, and effectively compresses sparse data with a large number of 0 elements generated by the calculation characteristics of 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and matrix operations; a unique input module and calculation array module are designed to complete the above operations and achieve hardware acceleration.

[0054] Therefore, the network accelerator in this embodiment combines a processor (CPU) and programmable logic (FPGA) components. By fully leveraging data sparsity, parallel computing, and load balancing optimization strategies, it significantly improves the efficiency of neural network inference, enabling sparse processing of point cloud data and accelerated inference of sparse convolutional neural networks. This overcomes the performance bottleneck of traditional general-purpose computing architectures when processing sparse data, providing an efficient and low-energy solution for applications that require point cloud data processing.

[0055] This accelerator is designed for highly sparse point cloud data, aiming to improve the efficiency of neural networks during inference and reduce unnecessary computing overhead. It performs particularly well in scenarios that require high real-time performance and big data processing.

[0056] 2. Architecture Division of Labor

[0057] A heterogeneous computing platform consisting of a CPU and FPGA is used. The CPU is responsible for data flow control and high-level tasks such as data preprocessing and control instructions, while the FPGA is used to accelerate the core calculations of sparse neural networks.

[0058] 1.CPU part:

[0059] The CPU is responsible for loading the external data required for point cloud neural network inference and then preprocessing the data. In addition to data management, the CPU is also responsible for high-level decision-making and control of inference results. Depending on the application scenario and the task being performed, the processor can perform corresponding operations as needed and output control instructions.

[0060] 2.FPGA part:

[0061] The FPGA is primarily used to accelerate neural network inference. Leveraging the FPGA's parallel computing capabilities, sparse neural network operations can be significantly accelerated. This includes hardware acceleration for 3D sparse convolution, 3D submanifold sparse convolution, 2D convolution, and activation functions. To improve neural network computational efficiency, custom hardware modules are designed within the FPGA to handle sparse data indexing, storage, and computational optimization.

[0062] Through the above settings, hardware acceleration of 3D submanifold sparse convolution, 3D sparse convolution, 2D convolution and nonlinear operations in the calculation process of the point cloud neural network algorithm is achieved.

[0063] 3. As an embodiment

[0064] In this embodiment, the specific implementation process of the sparse neural network accelerator for point cloud data combines the heterogeneous computing characteristics of CPU and FPGA, uses CPU for control and data management, and uses FPGA for efficient computing, forming a software and hardware collaborative working mode. The overall architecture is as follows: Figure 1 The following is a description of the specific implementation process of this embodiment:

[0065] 1. Data acquisition and input

[0066] After the system boots up, the CPU first retrieves weight data and point cloud voxel data from external devices (such as high-speed interfaces and storage devices). Weight data is quantized to 8-bit integers. Voxel data consists of 3D index coordinates and quantized 8-bit feature data. After data acquisition, the CPU stores this data in off-chip RAM for subsequent data preprocessing and calculations.

[0067] 2. Data Preprocessing

[0068] The 3D index coordinate data is input to the preprocessing module. This module aggregates the feature data based on the requirements of the neural network inference by performing operations such as kernel search and neighborhood search on the coordinates, forming a continuous feature data stream. This data stream is then transferred to the FPGA via the DMA data transfer module.

[0069] 3. Data transmission and control

[0070] To accelerate data transfer, the DMA data transfer module is responsible for transferring data from off-chip RAM to the corresponding modules (weight input module and feature input module) in the FPGA. The DMA data transfer module's efficient transmission mechanism reduces CPU involvement, allowing it to devote more resources to management and control tasks without being slowed down by the transfer of large amounts of data.

[0071] During this process, the CPU configures registers in the control modules via the AXI_Lite bus, thereby configuring parameters and managing the operating status of each module. The control signals sent by the CPU ensure the synchronous execution of each module and guarantee the correctness of data flow during transmission and calculation.

[0072] 4. Sparse Neural Network Inference Acceleration

[0073] The accelerated inference of sparse neural networks is mainly completed through the weight input module, feature input module, calculation array module, post-processing module, and output module, which are used to process the computing tasks in 3D sparse convolution, 3D submanifold sparse convolution and 2D convolution operations.

[0074] Among them, the 2D convolution calculation process is as follows Figure 7 As shown, C i is the number of input channels, C o is the number of output channels, H k Is the side length of the convolution kernel, I is the input feature data, W is the weight data, and O is the output result data. The process can be described as formula (1):

[0075]

[0076] Similarly, the 3D sparse convolution calculation process is as follows Figure 8 As shown, the process can be described as formula (2):

[0077]

[0078] Since the convolution kernel of 2D convolution is a planar rectangle, a and b represent the lengths of the two sides of the convolution kernel; and the convolution kernel of 3D sparse convolution is a cube, so a, b, and c represent the lengths of the three sides of the convolution kernel.

[0079] 4.1 Weight Input and Distribution

[0080] In the convolution operation of sparse neural networks, weight is the core parameter. The accelerator transfers weight data from the off-chip RAM to the weight input module through the DMA data transfer module. The weight input module marks the address information of these weight data and sends it to the calculation array module. The weight data enters the weight input layer of each PE unit in the calculation array. The PE unit determines the address information carried by the weight data and stores the weight data belonging to the PE unit in the weight RAM. The weight data that does not belong to the PE unit is passed to the next PE unit. The distribution method is as follows: Figure 10 As shown, this ensures that each PE unit can obtain the correct weight and participate in parallel calculations.

[0081] 4.2 Input, Compression, and Reuse of Feature Data

[0082] After the DMA data transfer module has completed the weight data transmission and the data preprocessing module has also completed the processing of the point cloud voxel data, the feature data stream is obtained. At this time, the feature data stream is transferred from the off-chip RAM to the feature input module through the DMA data transfer module. The feature input module workflow is as follows: Figure 4 As shown, first, after the feature data stream passes through the input FIFO, the zero element features in the data stream will be removed to achieve the purpose of compressing the feature data. The dynamic process of processing data is as follows Figure 5 Then, the compressed feature data is sent to the calculation array and the circular FIFO, which reuses the feature data in a circular manner. The dynamic process of processing data is as follows: Figure 6 As shown, the load of each channel is balanced to ensure the maximum working efficiency of the entire computing array module.

[0083] 4.3 Calculation process

[0084] The computation array module is designed to support convolution operations, especially sparse convolution operations. Figure 8-9 As shown, it can be seen that the implementation process of sparse convolution operation in the accelerator and the flow of weight data, feature data, and result data. As mentioned earlier, after the weight input module sends the weight data, it is stored in the weight RAM of each PE in the calculation array. The feature input module removes invalid zero elements in the feature data stream and balances the data flow load of each channel. Therefore, when the feature data enters the feature input layer of the PE unit from the feature input module, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data is multiplied and accumulated with the corresponding weight data. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO. While participating in the multiplication and accumulation calculation, the feature data will also be transmitted to the feature input layer in the next PE, and the flow method is as follows. Figure 11When all PEs complete the calculation of a feature data stream, the result data cached in each PE will enter the result output layer of the PE unit. The result data is added in the result output layer of the PE and passed to the result output layer of the next PE unit. The calculation flow is as follows Figure 12 The calculation results will eventually be transmitted to the post-processing module for processing.

[0085] 5. Post-processing and output

[0086] After the calculation is complete, the system enters the post-processing phase. This module performs a series of operations on the output data from the computation array module, including nonlinear activation functions (such as ReLU) and bit width adjustment. Bit width adjustment aims to map the convolution results back to the appropriate output range, ensuring that subsequent processors can correctly interpret the data.

[0087] The post-processed data will be passed to the output module in the FPGA, which caches the data and transfers the processing results back to the off-chip RAM through the DMA data transfer module, and finally handed over to the CPU for further processing or output.

[0088] 6.CPU control and decision-making

[0089] After the FPGA completes the sparse convolution calculation and returns the results to off-chip RAM, the CPU is responsible for high-level decision-making and control. Based on the inference results, the CPU can perform corresponding operations, such as outputting the inference results and issuing further calculation instructions.

[0090] In summary, the entire calculation process is as follows Figure 13 shown.

[0091] 7. Overall system performance optimization

[0092] To improve the overall performance of the system, this embodiment has been optimized in the following aspects:

[0093] Sparsity Utilization: By designing dedicated hardware modules (CPU, FPGA), we can fully utilize the sparsity in point cloud data and sparse neural networks to reduce unnecessary calculations and save storage and bandwidth resources.

[0094] Parallel computing: The parallel processing capabilities of FPGA are fully utilized, and the collaborative work of a large number of PE unit computing arrays greatly improves computing efficiency.

[0095] Data transmission optimization: Use the DMA module to transmit data, reduce the CPU's involvement in data movement, and improve the overall throughput.

[0096] Dynamic load balancing: The characteristic data input module dynamically balances the load of each channel through circular FIFO processing based on the characteristics of the data flow, ensuring that the computing array can work efficiently under unbalanced data flow.

[0097] Data reuse: The weight input module stores weight data in a distributed manner across different PE units. This allows weight data to be reused during convolution calculations, significantly reducing bandwidth overhead. A circular FIFO in the feature input module allows feature data to be reused multiple times, minimizing data flow transmission overhead and significantly improving convolution calculation efficiency.

[0098] The sparse neural network accelerator for point cloud data proposed in this embodiment is optimized for the sparsity characteristics of point cloud algorithm processing and successfully overcomes the following problems existing in the background technology:

[0099] (1) Efficient sparse processing to reduce computational overhead

[0100] Traditional general-purpose hardware architectures such as GPUs cannot fully utilize the characteristics of sparse data, resulting in low computational efficiency when processing sparse point cloud data. This embodiment effectively performs sparse processing on the data by designing dedicated modules (various modules in the CPU and FPGA), avoiding invalid calculations of zero-valued elements, thereby significantly reducing computational overhead. By eliminating invalid zero elements in advance and skipping zero element calculations in sparse convolutions in the computational array, computational efficiency can be improved, especially in scenarios with high sparsity. For example, in the processing of point cloud data, the computational efficiency of a typical sparse convolution layer can be increased by more than 2 times.

[0101] (2) Dynamic load balancing to ensure efficient operation

[0102] When processing sparse data, irregular data distribution can easily lead to unbalanced computational loads, which in turn affects performance. The feature input module of this embodiment dynamically balances the load of each channel based on the sparsity of the feature data, ensuring that all computational array modules can operate in a load-balanced state, greatly avoiding idle computational array modules and thus improving computational array module utilization. This dynamic scheduling mechanism ensures that the performance of sparse convolution can remain stable under conditions of data input of varying densities, achieving high computational efficiency even in the case of unbalanced loads.

[0103] (3) Low power consumption and high real-time performance

[0104] This embodiment demonstrates significant advantages in scenarios requiring high real-time performance and low power consumption. By reducing the ineffective overhead associated with sparse computations and effectively controlling data flow, the entire system significantly reduces power consumption while maintaining high computing performance. In scenarios requiring high real-time performance, such as autonomous driving and robotics, this system excels, rapidly responding to and processing large-scale point cloud data, providing a highly competitive hardware acceleration solution for these applications.

[0105] (4) Parallel computing acceleration to improve system throughput

[0106] This design fully exploits the parallel processing capabilities of FPGAs. The computational array module, composed of multiple PE units, can perform sparse convolution operations in parallel. Compared to serial computation in common architectures, this accelerator achieves massive parallel processing in hardware through a two-dimensional PE array design. This allows for simultaneous execution of multiple convolution computations, significantly improving the overall system throughput.

[0107] (5) Efficient data transmission, reducing CPU burden

[0108] The system uses a DMA data transfer module to achieve high-speed transfer of large blocks of data between the CPU and FPGA, reducing direct CPU involvement and freeing up CPU resources for high-level control and management tasks, eliminating the need for frequent large-scale data movement. This design significantly reduces system bus occupancy and improves overall data transfer efficiency, particularly when processing large-scale point cloud data.

[0109] (6) Software and hardware collaboration to fully leverage the advantages of heterogeneous platforms

[0110] This embodiment fully utilizes the heterogeneous computing architecture of the CPU and FPGA, using the CPU for data management and system control tasks, while the FPGA is used for hardware acceleration of sparse neural network inference. The CPU is responsible for loading point cloud voxel data, neighborhood search, invalid data removal, and sparse matrix construction, while the FPGA is responsible for large-scale sparse convolution calculations. This hardware and software collaborative design enables the entire system to perform well when processing highly sparse point cloud data and significantly reduces the overall energy consumption of the system.

[0111] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A sparse neural network accelerator for point cloud data, characterized in that: The accelerator is a heterogeneous computing platform consisting of a CPU and an FPGA. The CPU includes a data input and output module, a data preprocessing module, and a function instruction module. The FPGA includes a DMA data transmission module, a weight input module, a feature input module, a calculation array module, a post-processing module, an output module, and a control module. The data input and output module is used to receive and send external data required for point cloud neural network reasoning; The data preprocessing module is used to process the point cloud voxel data through kernel search, neighborhood search, and feature aggregation to generate a feature data stream that can be used for accelerator calculation; The function instruction module is used to configure the registers in the control module and control the data transmission and reception of the DMA data transmission module; The weight input module is used to transmit the weight data stored in the off-chip RAM to the calculation array module through the DMA data transmission module; The feature input module is used to compress and reuse the feature data stream stored in the off-chip RAM through the DMA data transmission module to obtain sparse feature data and transmit it to the calculation array module; The calculation array module is composed of a plurality of identical PE units, weight data is stored in the PE units, feature data is multiplied and accumulated by the PE unit weight data, and the result data is cached in the result FIFO, and the result data is taken out from the result FIFO; The post-processing module is used to perform nonlinear activation function calculation and bit width adjustment operations on the result data, and the processed data is sent to the output module; The output module is used to cache the result data, and then transfer it to the off-chip RAM via the DMA data transmission module; The control module is used to configure and control the feature input module, weight input module, post-processing module and output module to ensure the correct execution of data transmission and calculation.

2. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The computing array module is composed of X×Y PE units, which are arranged in a rectangular shape. Adjacent PE units are interconnected and used to perform parallel multiplication and accumulation calculations on feature data. Each PE unit contains a weight input layer, a feature input layer, a result output layer, a weight RAM, and a result FIFO. The weight input layer has the function of receiving, storing and transmitting weight data; The feature input layer has the functions of receiving feature data, reading weight data, transmitting feature data, performing multiplication and accumulation calculations, temporarily storing partial sums, and outputting calculation result data; The result output layer has the functions of caching result data, receiving result data, accumulating result data and transmitting result data; Weight RAM stores weight data; The result FIFO buffers the result data.

3. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The feature input module removes zero elements from the feature data and caches it in the input FIFO. The processed feature data is sent to the calculation array module and cached in the circular FIFO for data multiplexing. The feature data in the circular FIFO circulates into the calculation array modules of different input channels, thereby balancing the load of each channel.

4. The sparse neural network accelerator for point cloud data according to claim 2, characterized in that: In the calculation array module, the weight data enters the weight input layer of the PE unit. By judging whether the weight data carries address information, the weight data belonging to the PE unit is stored in the weight RAM, and the weight data that does not belong to the PE unit is passed to the next PE unit.

5. The sparse neural network accelerator for point cloud data according to claim 2, characterized in that: When the feature data enters the feature input layer of the PE unit, the weight data stored in the PE weight RAM is found through the weight address index, and the feature data and the corresponding weight data are multiplied and accumulated. After completing the calculation of a section of feature data stream, the result data is sent to the result FIFO; after entering the result output layer of the PE unit, the result data is added to the corresponding result data in the result FIFO and passed to the next PE unit.

6. The sparse neural network accelerator for point cloud data according to claim 1, characterized in that: The weight data stream is transmitted longitudinally in the calculation array module, the feature data stream is transmitted transversely in the calculation array module, and the result data stream is transmitted longitudinally in the calculation array module.

Images