Neural network compression method for point cloud detection and accelerator

Abstract

The invention discloses a neural network compression method for point cloud detection and an accelerator. The method comprises the following steps: inserting a threshold prediction sub-network in a network model to generate double thresholds, pruning according to a relationship between an activation value and the threshold during network training, performing sparse convolution or standard convolution on a sub-manifold, and applying the threshold obtained by training during network reasoning. According to the accelerator, the point cloud is detected based on the FPGA, a core unit of the accelerator is a calculation unit, a parallel calculation array is formed by 1152 PE units, meanwhile, sub-manifold sparse convolution and standard convolution are supported, a neural network can be compressed in real time through the compression method when reasoning is carried out on the accelerator, the reasoning efficiency is improved, and the calculation efficiency is improved. A hardware-friendly accelerator with low delay and high energy efficiency is realized. According to the compression method disclosed by the invention, the influence of pruning on the detection precision is reduced, the problem that the precision is excessively reduced after pruning is solved, and a scheme for efficiently realizing the compression method on hardware is provided.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence neural network calculation, inference and counting, and specifically relates to a neural network compression method and accelerator for point cloud detection. Background Art

[0002] In recent years, electric vehicles have entered large-scale commercialization, and the accompanying intelligent driving technologies have also begun to be widely used. Leveraging artificial intelligence technologies based on neural networks, intelligent driving can achieve perception and understanding of the vehicle's surrounding environment, thereby improving driving comfort and safety. Many vehicles with intelligent driving functions are currently equipped with lidar. Detection neural networks based on lidar point clouds offer advantages such as more accurate detection of small targets, three-dimensional detection data, and immunity to ambient light. However, the input feature maps of point cloud detection neural networks have a large number of parameters and high sparsity, resulting in significant detection latency when deployed on the vehicle side, making them difficult to meet application requirements in the field of intelligent driving, which requires high real-time performance. The key is to exploit the sparsity of data to compress the computational complexity of point cloud detection neural networks and design a low-latency and energy-efficient deep convolutional neural network accelerator.

[0003] Based on existing research, the main compression methods for point cloud detection neural networks are pruning and convolution optimization. Existing pruning methods primarily employ weight pruning, which reduces network computational overhead and saves storage space by setting redundant parameters in convolutional neural networks to zero. However, the input feature maps of point cloud detection networks have high sparsity. Compared to feature map pruning, weight pruning cannot directly exploit input sparsity, resulting in limited computational reduction. Furthermore, the reduction in weight parameters does not significantly improve memory usage as significantly as feature map pruning. The existing feature map pruning method for point cloud detection networks is to filter activation values ​​according to a threshold, and set activation values ​​less than the threshold to zero or downsample them. In terms of threshold selection, there is a solution that determines the threshold through sparse regularization. This solution can retain high detection accuracy after pruning, but it requires the addition of special pruning units in the accelerator. These pruning units need to match the pruning algorithm and have a relatively complex structure. The patent application with publication number CN116612469A manually sets the threshold and determines the threshold based on the average value of all non-zero voxels. The threshold setting method of this solution is difficult to dynamically adjust according to the data distribution and model structure, and lacks robustness. In the optimization of convolution calculation, the existing scheme combines submanifold sparse convolution with neural network to reduce the amount of calculation and achieve an effect similar to pruning. Among them, the patent application with publication number CN113449612B directly uses the submanifold convolution network to extract features. Although this method can greatly reduce the amount of calculation of the network model and the hardware implementation is relatively simple, the feature extraction ability of the submanifold sparse convolution is weak, so the detection accuracy of the network is greatly sacrificed; the patent application with publication number CN116612469A combines submanifold sparse convolution to design a spatial adaptive convolution algorithm. The convolution kernel size and convolution receptive field range of this scheme are dynamic, which is difficult to implement in hardware. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a neural network compression method and accelerator for point cloud detection. It uses a pruning threshold selection method based on a neural network and combines pruning, standard convolution and submanifold sparse convolution with dual thresholds. It greatly reduces the computational complexity of the point cloud detection network while ensuring detection accuracy. Based on this method, a hardware-friendly, low-latency, and high-efficiency accelerator is implemented.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a neural network compression method for point cloud detection, comprising the following steps:

[0006] S1. Obtain the initial values ​​of the weight parameters when the point cloud detection network model reaches the target accuracy, and insert a threshold prediction subnetwork into each layer of the network to generate the calculation thresholds δ0 and δ1 of each layer;

[0007] S2. Adjust the overall loss function of the network model based on the loss function of the threshold prediction subnetwork, and train the network until the accuracy and sparsity meet the preset requirements. During training, in the forward propagation process, according to the relationship between the activation value and the threshold, three corresponding methods are adopted for calculation, and the weight parameters and threshold parameters of each layer of the trained network are extracted;

[0008] In the network inference phase, according to the threshold parameters of each layer, the three corresponding methods in the forward propagation process during training are adopted to calculate and output the inference results.

[0009] Furthermore, in the aforementioned step S1, the threshold prediction subnetwork is a two-layer cascaded multilayer perceptron, the input includes the feature map mean, standard deviation, 10% quantile of the eigenvalue and 90% quantile of the eigenvalue, and the output includes thresholds δ0 and δ1, and satisfies δ0<δ1.

[0010] Furthermore, in the aforementioned step S2, the loss function of the threshold prediction subnetwork is composed of a linear combination of the sparsity regularization loss and the threshold order constraint loss, and the overall loss function of the network model is composed of a linear combination of the loss function of the threshold prediction subnetwork and the detection loss function of the network model.

[0011] Furthermore, the aforementioned neural network compression method for point cloud detection adopts three corresponding calculation methods during the forward propagation process during network training and the network inference process, specifically: pruning is performed on activation values ​​less than δ0, submanifold sparse convolution calculation is performed on activation values ​​between δ0 and δ1, and standard convolution calculation is performed on activation values ​​greater than δ1.

[0012] Furthermore, in the aforementioned step S3, in the network inference stage, the threshold prediction subnetwork is removed, and the threshold parameters of each layer extracted after the network training is completed are directly used.

[0013] Furthermore, in the aforementioned neural network compression method for point cloud detection, the KITTI dataset is selected as the database for network training.

[0014] Furthermore, the aforementioned neural network compression method for point cloud detection trains the network based on the learning framework Pytorch, uses floating-point calculations to test the data of each layer during the forward propagation process, adjusts the network model and parameters, and manually adjusts the network training parameters. The network weight parameters and the dual threshold parameters of each layer of the network are automatically adjusted in a gradient descent manner during training, and finally obtains the initial values ​​of the weight parameters and threshold parameters when the neural network reaches the target accuracy and sparsity.

[0015] Another aspect of the present invention provides a neural network compression accelerator for point cloud detection, which is used to perform forward derivation of the network based on the activation value of the feature map and the dual threshold of the current layer of the network, including:

[0016] A bus interface unit for receiving data including network model weights and eigenvalues ​​and a control word;

[0017] A configuration register unit for storing a control word;

[0018] The accelerator control unit is configured to generate an initial configuration instruction after reading the control word, send an external data read instruction including an initial address and data amount to the storage scheduling unit, and send a working state conversion instruction to the computing array unit;

[0019] The storage scheduling unit writes the external data obtained through the bus interface unit into the on-chip cache unit after receiving the external data read instruction;

[0020] On-chip cache unit, which caches external data instructions written by the scheduling unit and partial sum results output by the parallel computing unit;

[0021] Point cloud preprocessing unit, which converts point cloud data into a format that meets the processing requirements of the point cloud detection network;

[0022] The feature map block unit divides the feature map into blocks of fixed size and sets the activation values ​​below δ0 in the feature map to zero. If all activation values ​​in a block are zero, the calculation of the block is skipped. For blocks with non-zero activation values, the activation values ​​are sent to the calculation array unit in two batches. The first batch contains all activation values ​​greater than δ0 for standard convolution, and the second batch contains all activation values ​​between δ0 and δ1 for submanifold sparse convolution.

[0023] The computing array unit reads the initial configuration instructions to complete the initial configuration, receives the working state conversion instructions sent by the accelerator control unit to complete the reading of weight and eigenvalue data and the convolution operation, and calculates the feature map in two batches according to the feature map sent by the blocking unit. The first batch performs standard convolution calculation, and the second batch performs sub-manifold sparse convolution calculation. After the calculation is completed, the part and result of the convolution operation are written to the on-chip cache unit.

[0024] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the present invention when executing the computer program.

[0025] The present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any one of the methods described in the present invention when executed by a processor.

[0026] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0027] (1) This paper proposes a neural network compression method and accelerator for point cloud detection. By adopting different calculation methods within different activation value ranges, the scale of feature map parameters during network inference is reduced. This method was designed with the application scenario of vehicle-side lidar point cloud target detection in mind. In view of the detection accuracy, real-time performance, and hardware resource requirements of this application, while ensuring detection accuracy, it reduces redundant activation values ​​and unnecessary convolution calculations in the network, thereby significantly reducing the amount of calculation and memory usage, and improving inference efficiency.

[0028] (2) The present invention reduces the computational complexity of convolution calculations and the inference delay of the network model by precisely controlling the feature map pruning and convolution methods and combining feature map blocking. It also accelerates the point cloud detection network using the FPGA hardware platform. It can support convolution and full connection operations of sparse networks after pruning with different sparsity levels, as well as support hybrid calculations of submanifold sparse convolution and standard convolution, which facilitates the application of point cloud detection models on the vehicle side, improves the real-time performance of the network model, and enhances the practical application value of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the point cloud detection neural network compression method of the present invention.

[0030] Figure 2 This is a schematic diagram of a neural network compression method that combines feature map pruning and submanifold sparse convolution, where (a) is the input feature map and (b) is the output feature map.

[0031] Figure 3 It is a schematic diagram of the overall structure of the accelerator of the present invention.

[0032] Figure 4 Schematic diagram of the operation of the feature map blocking unit of the accelerator of the present invention. DETAILED DESCRIPTION

[0033] In order to better understand the technical content of the present invention, specific embodiments are given and described below with reference to the accompanying drawings.

[0034] Various aspects of the present invention are described herein with reference to the accompanying drawings, which show a number of illustrative embodiments. The embodiments of the present invention are not limited to those described in the accompanying drawings. It should be understood that the present invention can be implemented by any of the various concepts and embodiments described above, as well as the concepts and implementations described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. In addition, some aspects disclosed herein may be used alone or in any appropriate combination with other aspects disclosed herein.

[0035] refer to Figure 1 ,The present invention provides a neural network compression method for point cloud detection.,First, obtain the initialization parameter model and the data set.,The database can select the KITTI data set.

[0036] Then, we load the point cloud detection network model published by the network resource library and insert a threshold prediction subnetwork into each layer of the network model. The threshold prediction subnetwork is a two-layer multi-layer perceptron with 4 input dimensions, 16 hidden dimensions, and 2 output dimensions. The input dimensions include the mean, standard deviation, 10% quantile of the eigenvalue, and 90% quantile of the eigenvalue of the feature map. The output dimensions include thresholds δ0 and δ1, and δ0 < δ1. The loss function of the threshold prediction subnetwork is: Among them, the sparsity regularization loss By the formula Calculation, N represents the number of elements in the feature map, A i Represents the activation value in the feature map, ReLU represents the linear rectification function, and the threshold order constraint loss By the formula Calculate,λ sparse and λ order is a hyperparameter used to balance the effects of sparsity and threshold order. The ratio of these two hyperparameters is 1:1. The loss function of the entire network is adjusted according to the loss function of the threshold prediction sub-network. The total loss function of the network is: is the original loss of the model, λ threshold It is a hyperparameter used to control the impact of threshold prediction loss on the total loss. This hyperparameter controls the sparsity and detection accuracy of the network. The larger the value of the hyperparameter, the higher the sparsity of the network and the lower the detection accuracy. Therefore, it needs to be adjusted according to the sparsity target and accuracy target of the network.

[0037] Next, according to the KITTI dataset publicly available in the network resource library, the parameters are adjusted independently and the convolutional neural network is trained on the dataset. During training, in the process of forward propagation, different calculation methods are adopted according to the relationship between the activation value and the threshold. Pruning is performed on the activation values less than δ0, submanifold sparse convolution calculation is performed on the activation values between δ0 and δ1, and standard convolution calculation is performed on the activation values greater than δ1. The model is trained until the detection accuracy and sparsity meet the goals, and the weight parameters and the thresholds of each layer of the network under high precision are obtained. After the training of the network is completed, the full-precision network model weight parameters and the thresholds of each layer of the network are extracted.

[0038] Finally, when the network model is in inference, the threshold prediction subnetwork is removed, and the threshold parameters of each layer extracted after the network training is completed are directly used for calculation according to the forward propagation process during network training.

[0039] Figure 2 The schematic diagram of neural network compression based on feature map pruning and submanifold sparse convolution during network inference of the present invention is given. As shown in the example, Figure 2 in (a) is the input feature map with a size of 10×10, Figure 2 in (b) is the output feature map with a size of 8×8 and a convolution kernel size of 3×3. According to the output result of the threshold prediction subnetwork in the network training stage, Figure 2 the feature map in (a) has thresholds δ0 and δ1, and satisfies δ0 < δ1. According to these two thresholds, the activation value a of the input feature map is divided into 4 cases: a = 0, represented by a white square in the figure; a < δ0, represented by a light gray square in the figure; δ0 < a < δ1, represented by a dark gray square in the figure; a > δ1, represented by a black square in the figure. For the activation values of a = 0 and a < δ0, no calculation is performed, corresponding to Figure 2 the white squares in (b); for the activation values of δ0 < a < δ1, submanifold sparse convolution is performed, corresponding to the dark gray squares in ②; for the activation values of a > δ1, standard convolution is performed, corresponding to Figure 2 the black squares in (b).

[0040] As Figure 3As shown, another aspect of the present invention provides a neural network compression accelerator for point cloud detection, which is used to perform forward network derivation based on the activation value of the feature map and the dual threshold of the current network layer. The accelerator includes a bus interface unit, a configuration register unit, an accelerator control unit, a storage scheduling unit, an on-chip cache unit, a point cloud preprocessing unit, a computation array unit, and a functional unit. The bus interface unit is configured with master and slave AXI4 bus interfaces, allowing the accelerator to be mounted on any bus using the AXI4 protocol. An external controller (such as a CPU) sends a control word to the configuration register unit via the slave interface of the interface unit to complete the initial configuration of the accelerator. After the initial configuration is completed, the accelerator control unit controls circuit operation based on the configuration information. The accelerator control unit first needs to activate the storage scheduling unit to read external data. It only needs to provide the initial address and data size based on the configuration information, and the storage scheduling unit automatically transfers the data. The storage scheduling unit actively reads data from the off-chip memory through the master interface of the interface unit and stores it in the corresponding area of ​​the on-chip cache unit according to the data type. The data cached by the on-chip cache unit includes point cloud data, feature values, weights, normalization parameters, partial sum results, and final output results. The point cloud preprocessing module divides and expands the point cloud based on the point cloud data cached by the on-chip cache unit, converting discrete points into a regular feature map format. The feature map blocking unit divides the feature map into blocks of fixed size, sets activation values ​​less than δ0 to zero, and feeds blocks with non-zero activation values ​​into the computation array unit.

[0041] Figure 4 This is a schematic diagram of the operation of the feature map block unit of the accelerator of the present invention. During block partitioning, according to the dual threshold of the network layer, activation values ​​less than δ0 are reset to zero, activation values ​​between δ0 and δ1 are grouped into block 1, and activation values ​​greater than δ1 are placed into block 2. Block 1 first enters the computation array unit for standard convolution, and then block 2 enters the computation array for submanifold sparse convolution.

[0042] The computational array unit consists of four groups of 3*3*32 PE arrays, which are used to read feature map data and weight data for convolution operations. When performing convolution calculations, the calculation is divided into two calculation processes based on the threshold δ1: standard convolution calculation and submanifold sparse convolution calculation. When calculating standard convolution, each group of PE arrays shares the convolution kernel in the depth dimension, and the convolution kernel weight data is fixed within the PE unit. The convolution sliding window of the feature map is input into the computational array, and each group of computational arrays obtains 3*3*32 calculation results. These calculation results will be sent to the subsequent addition tree for accumulation to obtain the partial sum result of the channel dimension. The above operation is then repeated until the calculation result of one output channel of the 4*4 input feature map is completed, and finally the calculation of one non-zero block is completed. When calculating the sub-manifold sparse convolution, the pixel points corresponding to the feature map and the surrounding 3*3 area are loaded into a group of PE arrays, and the weights corresponding to the feature map are loaded into the PE array at the same time. Each group of PE arrays at the same depth will complete the calculation of a sub-manifold sparse convolution, and the calculation results are sent to the subsequent addition tree. The convolution kernel scale is first accumulated, and then the channel dimension is summed. When the channel dimension is summed, only the feature map of the input PE array is calculated, because the feature map that is not input into the PE array is zero value and does not need to participate in the summation. The accelerator first calculates the standard convolution and then calculates the sub-manifold sparse convolution. After the calculation is completed, the calculation results of the two are added element by element to obtain the output feature map. The present invention also provides a feasible solution for different compression rate requirements by setting different sparsity regularization loss hyperparameters λ sparse and threshold prediction loss hyperparameter λ threshold , we can find a balance between compression rate and network accuracy degradation. Different compression rates will result in different object detection accuracy after pruning. To a certain extent, the greater the compression rate, the greater the impact of pruning on the original accuracy.

[0043] Based on the preferred solution of the above technical solution, the accelerator control unit consists of six states, corresponding to waiting, point cloud preprocessing, feature map writing, convolution kernel writing, convolution calculation, and calculation result transmission. Each state sends the corresponding control signal to the corresponding submodule to complete the corresponding function.

[0044] In summary, after adopting the technical solution of the present invention, the compression rate of the network can be changed according to different detection accuracy and calculation delay requirements, so that the network model prunes the feature map into a sparse matrix composed of many zero values ​​during the inference phase, and calculates some activation values ​​in the form of submanifold sparse convolution, ensuring that important parameters are not pruned, and less important parameters are retained in the form of submanifold sparse convolution, which can reduce the impact of pruning on the final detection accuracy to a certain extent. The accelerator implemented by the present invention supports the point cloud detection network compression algorithm, and calculates standard convolution and submanifold sparse convolution at the same time. During the calculation process, the network is compressed in real time to reduce the amount of calculation. It has the advantages of low latency and high energy efficiency, and facilitates the deployment of point cloud detection network models on the vehicle side.

[0045] While the present invention has been described above with reference to preferred embodiments, this is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A neural network compression method for point cloud detection, characterized in that: The following steps are involved: S1. Obtain the initial values ​​of the weight parameters when the point cloud detection network model reaches the target accuracy, and insert a threshold prediction subnetwork into each layer of the network to generate the calculation thresholds δ0 and δ1 of each layer; S2. Adjust the overall loss function of the network model based on the loss function of the threshold prediction subnetwork, and train the network until the accuracy and sparsity meet the preset requirements. During training, in the forward propagation process, according to the relationship between the activation value and the threshold, three corresponding methods are adopted for calculation, and the weight parameters and threshold parameters of each layer of the trained network are extracted; In the network inference phase, according to the threshold parameters of each layer, the three corresponding methods in the forward propagation process during training are adopted to calculate and output the inference results.

2. The neural network compression method for point cloud detection according to claim 1, characterized in that: In step S1, the threshold prediction subnetwork is a two-layer cascaded multilayer perceptron, the input includes the feature map mean, standard deviation, 10% quantile of the eigenvalue and the 90% quantile of the eigenvalue, and the output includes thresholds δ0 and δ1, and satisfies δ0<δ1.

3. The neural network compression method for point cloud detection according to claim 1, characterized in that: In step S2, the loss function of the threshold prediction subnetwork is composed of a linear combination of the sparsity regularization loss and the threshold order constraint loss, and the overall loss function of the network model is composed of a linear combination of the loss function of the threshold prediction subnetwork and the detection loss function of the network model.

4. The neural network compression method for point cloud detection according to claim 1, characterized in that: During the forward propagation process of network training and the network inference process, three corresponding methods are adopted for calculation: pruning is performed on activation values ​​less than δ0, submanifold sparse convolution calculation is performed on activation values ​​between δ0 and δ1, and standard convolution calculation is performed on activation values ​​greater than δ1.

5. The neural network compression method for point cloud detection according to claim 1, characterized in that: In step S3, during the network inference phase, the threshold prediction subnetwork is removed, and the threshold parameters of each layer extracted after network training are directly used.

6. The neural network compression method for point cloud detection according to claim 1, characterized in that: The KITTI dataset is selected as the database for network training.

7. The neural network compression method for point cloud detection according to claim 1, characterized in that: The network is trained based on the learning framework Pytorch. Floating-point calculations are used to test the data of each layer during the forward propagation process, and the network model and parameters are adjusted. The training parameters of the network are manually adjusted, and the weight parameters of the network and the dual threshold parameters of each layer of the network are automatically adjusted by gradient descent during training. Finally, the initial values ​​of the weight parameters and threshold parameters are obtained when the neural network reaches the target accuracy and sparsity.

8. A neural network compression accelerator for point cloud detection, used to perform forward network derivation based on a dual threshold of the activation value of the feature map and the current layer of the network, characterized by: include: A bus interface unit for receiving data including network model weights and eigenvalues ​​and a control word; A configuration register unit for storing a control word; The accelerator control unit is configured to generate an initial configuration instruction after reading the control word, send an external data read instruction including an initial address and data amount to the storage scheduling unit, and send a working state conversion instruction to the computing array unit; The storage scheduling unit writes the external data obtained through the bus interface unit into the on-chip cache unit after receiving the external data read instruction; On-chip cache unit, which caches external data instructions written by the scheduling unit and partial sum results output by the parallel computing unit; Point cloud preprocessing unit, which converts point cloud data into a format that meets the processing requirements of the point cloud detection network; The feature map block unit divides the feature map into blocks of fixed size and sets the activation values ​​below δ0 in the feature map to zero. If all activation values ​​in a block are zero, the calculation of the block is skipped. For blocks with non-zero activation values, the activation values ​​are sent to the calculation array unit in two batches. The first batch contains all activation values ​​greater than δ0 for standard convolution, and the second batch contains all activation values ​​between δ0 and δ1 for submanifold sparse convolution. The computing array unit reads the initial configuration instructions to complete the initial configuration, receives the working state conversion instructions sent by the accelerator control unit to complete the reading of weight and eigenvalue data and the convolution operation, and calculates the feature map in two batches according to the feature map sent by the blocking unit. The first batch performs standard convolution calculation, and the second batch performs sub-manifold sparse convolution calculation. After the calculation is completed, the part and result of the convolution operation are written to the on-chip cache unit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images