A precise local densification sparse convolution accelerator
By using a sparse convolution accelerator with precise local densification, the problem of low efficiency in existing sparse convolution processing is solved. Through block-based and local densification processing, the output feature map is directly generated, which improves computational efficiency and reduces memory access time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2024-09-20
- Publication Date
- 2026-07-24
Smart Images
Figure CN119089958B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network acceleration technology, and in particular to a sparse convolution accelerator with precise local density. Background Technology
[0002] Scene perception models play a crucial role in the booming fields of autonomous driving and intelligent robotics in recent years. Point cloud data generated by LiDAR, as a more comprehensive data format, can serve as input data for numerous scene perception models, improving model accuracy and perception performance. However, the large size, high sparsity, low information density, and random and diverse distribution patterns of point cloud data have been hindering the improvement of model inference speed and consuming a large amount of storage resources, resulting in significant computational waste. Therefore, sparse convolution is needed to accelerate it.
[0003] Existing sparse convolution acceleration schemes are mostly based on a gather-scatter architecture. These schemes require traversing the coordinates of the entire input feature map to obtain a rulebook that can guide the convolution operation. Under the guidance of this rulebook, the input feature data is collected to generate the output feature map.
[0004] In this process, the strategy for generating the global rulebook ignores the sparsity of point cloud data. The sparsity of point cloud data refers to the fact that in the vast space perceived by LiDAR, only real objects can reflect the laser beam, generating non-zero point cloud data. This is reflected in the point cloud data as only a few areas containing valid points within the scene represented by the point cloud data, with the rest being blank. Generating the global rulebook requires searching the entire scene, wasting a significant amount of time on areas without valid values. Secondly, the process of retrieving feature values from the rulebook involves a large number of random and repeated reads, resulting in substantial memory access overhead and significantly impacting inference speed. Summary of the Invention
[0005] This application provides a sparse convolution accelerator with precise local density to solve the problem of low efficiency in existing sparse convolution processing techniques.
[0006] The accelerator includes:
[0007] A data loading module is configured to acquire point cloud data and weight data from a neural network; the point cloud data includes image coordinate data and image feature data corresponding to different weights, wherein the image coordinate data and the image feature data correspond one-to-one; the weight data corresponds to different weights.
[0008] A data segmentation module is configured to segment the point cloud data and the weight data into several segments.
[0009] A subgraph generation module is configured to perform offset processing and multiply-accumulate tree processing on the blocks respectively to obtain feature subgraphs corresponding to each weight.
[0010] The kernel accumulation module is configured to divide the feature sub-map and perform convolutional kernel accumulation processing to obtain the target feature map.
[0011] Preferably, each block includes valid data, invalid data, a valid coordinate range, and the data address corresponding to the valid data.
[0012] Preferably, the data segmentation module includes:
[0013] An effective range calculation unit is configured to calculate the effective range based on the coordinates of the point cloud data and the weight data to obtain the effective coordinate range.
[0014] An effective data calculation unit is configured to divide the point cloud data according to the effective coordinate range to obtain the effective data and the invalid data; the effective data includes the effective quantity and the data address;
[0015] The block partitioning unit is configured to divide the valid coordinate range, valid data, invalid data, data address and valid quantity belonging to the same weight into the same block.
[0016] Preferably, the effective range calculation unit is further configured as follows:
[0017] Based on the point cloud data and the weight data, determine the first coordinate that is the first coordinate and the second coordinate that is the last coordinate in the image coordinate data of all weights; the number of the first coordinate and the number of the second coordinate are determined by the convolution kernel of the neural network;
[0018] Filter out the first target coordinates that appear first from all the first coordinates, and filter out the second target coordinates that appear first from all the second coordinates;
[0019] The coordinate range between the first target coordinates and the second target coordinates is marked as the valid coordinate range; coordinate data within the valid coordinate range is the valid data, and coordinate data outside the valid coordinate range is the invalid data.
[0020] Preferably, the subgraph generation module includes:
[0021] A coordinate offset unit is configured to offset the image coordinate data in the corresponding block according to the corresponding weight to obtain a coordinate list.
[0022] The feature multiply-accumulate tree unit is configured to perform multiply-accumulate tree processing on the image feature data in the corresponding block according to the corresponding weights to obtain a feature list; the feature sub-graph includes the coordinate list and the feature list.
[0023] Preferably, the feature multiply-add tree unit is further configured as follows:
[0024] The weights of the corresponding blocks are multiplied by the image feature data to obtain the multiplication result;
[0025] The multiplication results of different convolution channels are accumulated to obtain the feature list;
[0026] The subgraph generation module further includes:
[0027] The block receiving unit is configured to receive blocks with different weights according to the data address and the weight data.
[0028] Preferably, the in-core accumulation module includes:
[0029] A data sorting unit is configured to determine the corresponding feature sub-graph based on the weight data, and sort the coordinate list in the feature sub-graph according to a preset order to obtain sorted data;
[0030] An effective partitioning unit is configured to partition the sorted coordinate data according to the effective coordinate range to obtain a minimum block, wherein the minimum block includes the effective data arranged in a preset order;
[0031] A feature search unit is configured to extract features in the smallest block to obtain a plurality of search feature data corresponding to the effective data.
[0032] A feature accumulation unit is configured to accumulate all the search feature data to obtain target feature data.
[0033] A coordinate extraction unit is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.
[0034] Preferably, the coordinate extraction unit is further configured to:
[0035] Extract the coordinates corresponding to the searched feature data when the feature search unit stops searching, and obtain the extracted coordinate data.
[0036] Preferably, the data loading module includes:
[0037] A data acquisition unit is configured to acquire point cloud data and weight data from the neural network;
[0038] A plurality of data storage units, each of which is configured to store the point cloud data and the weight data with corresponding weights.
[0039] Preferably, the data loading module further includes:
[0040] A data allocation unit is configured to send the point cloud data and the weight data to the corresponding data storage unit according to the corresponding weights;
[0041] The data storage unit is also configured to store the point cloud data, determine the weight data corresponding to the weights based on the point cloud data, and store the weight data.
[0042] As described above, this application provides a precise, locally dense sparse convolution accelerator. The accelerator includes a data loading module configured to acquire point cloud data and weight data from a neural network. The point cloud data includes image coordinate data and image feature data corresponding to different weights, with a one-to-one correspondence between the image coordinate data and the image feature data. The weight data corresponds to different weights. A data segmentation module is configured to segment the point cloud data and the weight data into several segments. A sub-graph generation module is configured to perform offset processing and multiply-accumulate tree processing on each segment to obtain a feature sub-graph corresponding to each weight. An intra-kernel accumulation module is configured to sort the feature sub-graphs according to a preset order to obtain sorted data, divide the sorted data, and perform convolution kernel accumulation processing to obtain a target feature map. This application solves the problem of low efficiency in existing sparse convolution processing techniques through the above accelerator. Attached Figure Description
[0043] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1This is a schematic diagram of a sparse convolution accelerator with precise local density according to this application;
[0045] Figure 2 This is a schematic diagram of the data loading module in a sparse convolution accelerator with precise local density according to this application;
[0046] Figure 3 This is a schematic diagram of the data segmentation module in a sparse convolution accelerator with precise local density according to this application;
[0047] Figure 4 This is a schematic diagram of a subgraph generation module in a sparse convolution accelerator with precise local density, as described in this application.
[0048] Figure 5 This is a schematic diagram of the kernel accumulation module in a sparse convolution accelerator with precise local density according to this application;
[0049] Figure 6 This is a flowchart illustrating the workflow of the data loading module in a precise, locally dense sparse convolution accelerator according to this application.
[0050] Figure 7 This diagram illustrates the relationship between feature sub-maps and target feature maps in a precise, locally dense sparse convolutional accelerator according to this application.
[0051] Figure 8 This is a flowchart illustrating the kernel accumulation module in a sparse convolution accelerator with precise local density, as described in this application. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0053] Figure 1 This is a schematic diagram of a sparse convolution accelerator with precise local density according to this application.
[0054] See Figure 1 As can be seen, this embodiment provides a sparse convolution accelerator with precise local density, the accelerator comprising:
[0055] A data loading module 100 is configured to acquire point cloud data and weight data from a neural network. The point cloud data includes image coordinate data and image feature data corresponding to different weights, with a one-to-one correspondence between the image coordinate data and the image feature data. The weight data corresponds to different weights. Specifically, in this embodiment, the data loading module 100 functions to acquire and store data by acquiring the corresponding point cloud data and the corresponding weight data from the neural network and storing them for later retrieval.
[0056] It should be noted that different weights can obtain the data they need from the same batch of point cloud data according to their own needs. The point cloud data itself does not need to be assigned weights. The point cloud data itself is a batch of points, and the points have coordinate data and feature data. It is also 3D, not a 2D image.
[0057] The accelerator also includes:
[0058] The data segmentation module 200 is configured to segment the point cloud data and the weight data into several blocks. Specifically, in this embodiment, since most point cloud data contains a large number of blank areas, which are meaningless for radar sensing, and existing sparse convolutions traverse all regions including the blank areas, the efficiency of sparse convolutions is very low. In this embodiment, the data segmentation module 200 sets the segmentation method to separate the blocks containing most of the blank areas, thereby avoiding the reading and calculation of blank areas.
[0059] It should be noted that each block includes valid data, invalid data, a valid coordinate range, and the data address corresponding to the valid data.
[0060] The accelerator also includes:
[0061] The sub-image generation module 300 is configured to perform offset processing and multiply-accumulate tree processing on the blocks respectively to obtain feature sub-images corresponding to each weight. Specifically, in this embodiment, the sub-image generation module 300 performs offset processing and multiply-accumulate tree processing on the blocks according to the corresponding weights, wherein the offset processing corresponds to the image coordinate data and the multiply-accumulate tree processing corresponds to the image feature data.
[0062] The accelerator also includes:
[0063] The kernel accumulation module 400 is configured to divide the feature sub-image and perform convolution kernel accumulation processing to obtain the target feature map. Specifically, in this embodiment, the kernel accumulation module 400 completes the precise local densification processing of the feature sub-image to obtain the complete target feature map.
[0064] In the precise local densification processing of the kernel accumulation module 400, "precise" means that the first position of each minimum block must be a non-zero value. Figure 8 For example, if there are 4 minimum blocks, then the non-zero part of the output feature map will be composed of multiple minimum blocks. In this process, there will be a waste phenomenon where the minimum blocks contain zero values, but since the first position is always non-zero, the non-zero position of the output feature map can be located as accurately as possible.
[0065] Local Desaturation: First, let's introduce the difference between the concepts of sparsity and density. In point cloud data, sparsity refers to storing only non-zero values. Figure 8 For example, if there are only 63 non-zero values in an 18x18 range, then the algorithm will only generate 63 data values. If it is densely stored, there will be 18x18=324 data values, but apart from the 63 non-zero data values, the rest are useless zero values.
[0066] Since the distribution of non-zero data cannot be predicted in advance, a dense coordinate range (meaning continuous coordinates) is generated for convenience, and data that falls within this range is processed. This is the meaning of "densification" in "local densification".
[0067] The "local" aspect is reflected in the fact that the minimum blocks in this scheme are generated sequentially, generated on-site by collecting data in real time, and their positions vary depending on the non-zero distribution of the output feature map, rather than being a fixed pattern. Therefore, the minimum blocks will only appear in non-zero positions, which is the embodiment of "local" and also reflects "precision".
[0068] The relationship between the data of the subgraph generation module 300 and the kernel accumulation module 400 can be found in [reference needed]. Figure 7 .
[0069] Figure 7 The shaded area represents the coordinates with values, clearly showing the feature submap. Figure 7 Output sub Figure 1 and 2 The target feature map is generated by shifting the input feature map in different directions. Figure 7 The output feature map is generated by superimposing all the feature submaps.
[0070] Figure 3 This is a schematic diagram of the data segmentation module in a sparse convolution accelerator with precise local density, as described in this application.
[0071] See Figure 3 It can be seen that, further, in some embodiments, the data segmentation module 200 includes:
[0072] An effective range calculation unit 210 is configured to calculate the effective coordinate range based on the coordinates of the point cloud data and the weight data. Specifically, in this embodiment, the effective range calculation unit 210 determines a plurality of first coordinates and a plurality of second coordinates from the image coordinate data of all weights based on the point cloud data and the weight data. The number of first coordinates and the number of second coordinates are determined by the convolution kernel of the neural network. The first target coordinate is selected from all the first coordinates, and the second target coordinate is selected from all the second coordinates. The coordinate range between the first target coordinate and the second target coordinate is marked as the effective coordinate range. The coordinate data within the effective coordinate range is the effective data, and the coordinate data outside the effective coordinate range is the invalid data.
[0073] For example, a 3×3 convolutional kernel has 9 weights, so the data loading module 100 loads 9 sets of data. Each set of data has different content, so the order of the 9 selected first coordinates will also be different. Furthermore, the 9 coordinates have different offset directions; even if the coordinates happen to be the same, the offset will still cause a difference in order. Therefore, from this example, we can deduce the conclusion mentioned above that "the number of first coordinates and the number of second coordinates are determined by the convolutional kernel of the neural network."
[0074] It should be noted that, for the second target coordinates, when all weights obtain the last data of the overall point cloud data in the data loading stage, that is, when the whole work has reached the last block, the selected second target coordinates are the last ones.
[0075] The data segmentation module 200 further includes:
[0076] The effective data calculation unit 220 is configured to divide the point cloud data according to the effective coordinate range to obtain the effective data and the invalid data. The effective data includes the effective quantity and the data address. Specifically, in this embodiment, the effective data calculation unit 220 uses the effective range calculation unit 210 to calculate the effective coordinate range to divide the point cloud data, thereby dividing the effective data and the invalid data. The corresponding effective data also includes the effective quantity and the data address.
[0077] The data segmentation module 200 further includes:
[0078] The block partitioning unit 230 is configured to divide the valid coordinate range, valid data, invalid data, data address and valid quantity belonging to the same weight into the same block.
[0079] Specifically, in this embodiment, the data loading module 100 receives a fixed amount of data, including some data that is not currently used in the segmentation. The data segmentation module 200 reads data from several specific locations of the point cloud data corresponding to each weight. After calculation and processing, it can distinguish between valid and invalid data, valid coordinate ranges, and the data addresses corresponding to the valid data. This information will be used in subsequent processes.
[0080] In the next round of data loading by the data loading module 100, the starting point of the required data for each weight is the first invalid coordinate in the current block. This dynamic block-splitting strategy can gradually increase the similarity of data with different weights in multiple rounds of data block splitting, increase the amount of effective data loaded each time, and reduce redundant data loading. This improves efficiency and speeds up inference.
[0081] It's important to note that the data addresses required for each weight refer to the relative addresses of the coordinate data and feature data. For example, if 128 data points from index 0 to 127 in a DDR are read at once, but the calculated statistics show that only the first 100 are valid (since the coordinates are stored sequentially, the pattern is always that the first part is valid and the latter part is invalid, with no interleaving of valid and invalid data), then the remaining 28 invalid data points must appear in the next block. That is, the next read will start from index 100 in the DDR and read 128 data points in total. Since the amount of valid data calculated for each weight is different, the starting point for the next read will also be different.
[0082] Figure 4 This is a schematic diagram of a subgraph generation module in a sparse convolution accelerator for precise local densification according to this application.
[0083] See Figure 4 It can be seen that, further, in some embodiments, the subgraph generation module 300 includes:
[0084] Coordinate offset unit 310 is configured to offset the image coordinate data in the corresponding block according to the corresponding weight to obtain a coordinate list.
[0085] The feature multiply-accumulate tree unit 320 is configured to perform multiply-accumulate tree processing on the image feature data in the corresponding block according to the corresponding weights to obtain a feature list; the feature sub-graph includes the coordinate list and the feature list.
[0086] Specifically, in this embodiment, the calculation of each block is completed by the coordinate offset unit 310 and the feature multiply-accumulate tree unit 320. The coordinate offset unit 310 performs offset processing on the image coordinate data in the corresponding block according to the corresponding weight, and the feature multiply-accumulate tree unit 320 performs multiply-accumulate tree processing on the image feature data in the corresponding block according to the corresponding weight.
[0087] The point cloud data consists of two sets: coordinates and features. The coordinates are processed by positional offset. For example, the coordinates (x, y) of the top-left weight of a 3x3 convolution kernel become (x+1, y+1), while the coordinates of the bottom-right weight are processed as (x-1, y-1).
[0088] The processing of features involves using a multiply-accumulate tree to multiply the features and their corresponding weights, and then accumulating the feature-weight products from different input channels.
[0089] Furthermore, in some embodiments, the subgraph generation module 300 further includes:
[0090] The block receiving unit 330 is configured to receive blocks with different weights according to the data address and the weight data. Specifically, in this embodiment, the block receiving unit 330 receives blocks with different weights according to the data address and the weight data, and sends the corresponding blocks to the coordinate offset unit 310 and the feature multiply-accumulate tree unit 320.
[0091] Figure 5 This is a schematic diagram of the kernel accumulation module in a sparse convolution accelerator with precise local density according to this application.
[0092] See Figure 5 It can be seen that, further, in some embodiments, the in-core accumulation module 400 includes:
[0093] The data sorting unit 410 is configured to determine the corresponding feature sub-graph based on the weight data, and sort the coordinate list in the feature sub-graph according to a preset order to obtain the sorted data. Specifically, in this embodiment, each weight will provide the first coordinate in its own data, and the coordinates provided by all weights will be sorted.
[0094] It should be noted that the data sorting unit 410 is a data preprocessing unit. Therefore, during the sparse convolution calculation, it is assumed that the work of the data sorting unit 410 has been completed, that is, all the calculated feature sub-images are ordered data.
[0095] The kernel accumulation module 400 also includes:
[0096] An effective partitioning unit 420 is configured to partition the sorted coordinate data according to the effective coordinate range to obtain a minimum block. The minimum block includes the effective data arranged in a preset order. Specifically, in this embodiment, the effective partitioning unit 420 selects the coordinate with the first position in the sorted data, takes this coordinate as the starting point, and takes a fixed number of coordinates after it in sequence as the current effective range to form a minimum block.
[0097] It should be noted that the fixed number of coordinates refers to the number of coordinates within the valid coordinate range.
[0098] The step of collecting coordinates and selecting the highest-ranking coordinate as the starting point of the smallest block can be replaced by generating an index value from the coordinates, then ordering the index values according to the coordinate order, collecting the minimum index value, and using it as the starting point of the smallest block.
[0099] The kernel accumulation module 400 also includes:
[0100] Feature search unit 430 is configured to perform feature extraction in the minimum block to obtain a plurality of search feature data corresponding to the effective data.
[0101] Feature accumulation unit 440 is configured to accumulate all the search feature data to obtain target feature data.
[0102] The coordinate extraction unit 450 is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.
[0103] Specifically, in this embodiment, the workflow of the feature search unit 430, the feature accumulation unit 440, and the coordinate extraction unit 450 will be described by way of example.
[0104] For example, taking coordinates (0, 0) as the starting point and a minimum block size of 16, the effective range is (0, 0), (0, 1)...(0, 15). After confirming the current effective range, each weight searches for points within that range and sends the corresponding features to a set of customized feature accumulation units 440. Since the coordinates are stored in an ordered manner, this search process is unidirectional, and the search stops when the first point that does not meet the range requirements is encountered. This search method can save a significant amount of inference time.
[0105] After confirming that each weight has completed its search, the feature accumulation unit 440 is activated. It processes the data received during the search process into features of the target feature map and outputs them. Simultaneously, the coordinate list of each weight outputs the coordinates of the search termination point, which can then be used to generate the next minimum block. By repeating this process, the input data processing is completed, and the correct target feature map is generated.
[0106] Figure 2 This is a schematic diagram of the data loading module in a sparse convolution accelerator with precise local density according to this application.
[0107] See Figure 2 It is understood that, further, in some embodiments, the data loading module 100 includes:
[0108] Data acquisition unit 110 is configured to acquire point cloud data and weight data from the neural network;
[0109] A plurality of data storage units 120, each of the data storage units 120 being configured to store the point cloud data and the weight data with corresponding weights.
[0110] Specifically, in this embodiment, the data acquisition unit 110 is used to acquire relevant data from the neural network, and each data storage unit 120 is used to store data corresponding to the weights.
[0111] The data loading module 100 further includes:
[0112] Data allocation unit 130 is configured to send the point cloud data and the weight data to the corresponding data storage unit 120 according to the corresponding weights.
[0113] The data storage unit 120 is also configured to store the point cloud data, determine the weight data corresponding to the weights based on the point cloud data, and store the weight data.
[0114] Specifically, in this embodiment, the data allocation unit 130 is used to allocate the data acquired by the data acquisition unit 110 and store the corresponding data in the corresponding data storage unit 120.
[0115] The workflow of the data loading module 100 loading the point cloud data and the weight data is described in [reference needed]. Figure 6 .
[0116] This includes coordinate and feature data. To be precise, 0-17 can be understood as relative addresses, meaning that the coordinates and features are stored in different regions and are not loaded simultaneously. The numbers 0-17 shown in the diagram refer to numbers 0-17 within each region, not absolute addresses in the DDR. During actual reading, the relative addresses will be mapped to absolute addresses.
[0117] Figure 6 The numbers 0-17 in the diagram can be understood as relative addresses, meaning the coordinates and features are stored in different regions and are not loaded simultaneously. The numbers 0-17 shown in the diagram refer to the numbers 0-17 within each region, not absolute addresses in the DDR. During actual reading, the relative addresses will be mapped to absolute addresses.
[0118] Figure 6 The example provides three weights, each with its own data requirements (calculated from the previous block). Since the requirements are different but have similarities, loading data for each weight individually would result in repeated reading behavior. Therefore, a union is calculated, i.e., 2-15. Data is retrieved from the DDR according to this requirement and broadcast to the three weights. Each weight corresponds to a data storage unit of 120. The decision on whether to use the data is made based on the corresponding requirement.
[0119] This embodiment has the following advantages:
[0120] First, a sparse-ordered input data format is adopted. Sparse means that the input data only contains non-zero valid points, which ensures that only valid points are processed. Ordered means that the coordinates of non-zero values are arranged in a certain order, which ensures that data access to memory is sequential reading and writing, saving memory access time.
[0121] Secondly, instead of generating a global rulebook, we directly generate a batch of output sub-maps based on weights using a simple, regular calculation method that does not involve random read / write operations. The number of these sub-maps is the same as the number of weights in the convolutional kernel, and when they are stacked together, they form a complete output feature map.
[0122] Finally, using these output sub-images as input, a batch of minimum blocks are sequentially derived from the output sub-images using a precise local densification method. These minimum blocks accurately locate all non-zero points in the output feature map. Simultaneously with the generation of minimum blocks, the input feature data enters subsequent computation units in a sequential read-write manner, saving significant memory access time. Subsequent computation units can complete the generation of the output feature map through a simple and fixed computation process. In this process, all time and computational resources are focused on the effective points in the output feature map. This avoids wasting computational resources and effectively accelerates the computation process of sparse convolution.
Claims
1. A sparse convolution accelerator with precise local density, characterized in that, The accelerator is applied to a neural network, and the accelerator includes: A data loading module (100) is configured to acquire point cloud data and weight data from a neural network; the point cloud data includes image coordinate data and image feature data corresponding to different weights, and the image coordinate data and the image feature data correspond one-to-one; the weight data corresponds to different weights. A data segmentation module (200) is configured to segment the point cloud data and the weight data into several segments. A subgraph generation module (300) is configured to perform offset processing and multiply-accumulate tree processing on the blocks respectively to obtain feature subgraphs corresponding to each weight; The kernel accumulation module (400) is configured to divide the feature sub-map and perform convolution kernel accumulation processing to obtain the target feature map.
2. The sparse convolution accelerator with precise local density as described in claim 1, characterized in that, Each block includes valid data, invalid data, a valid coordinate range, and the data address corresponding to the valid data.
3. The sparse convolution accelerator with precise local density as described in claim 2, characterized in that, The data segmentation module (200) includes: An effective range calculation unit (210) is configured to calculate the effective range based on the coordinates of the point cloud data and the weight data to obtain the effective coordinate range; An effective data calculation unit (220) is configured to divide the point cloud data according to the effective coordinate range to obtain the effective data and the invalid data; the effective data includes the effective quantity and the data address; The block partitioning unit (230) is configured to divide the valid coordinate range, the valid data, the invalid data, the data address and the valid quantity belonging to the same weight into the same block.
4. A sparse convolution accelerator with precise local density as described in claim 3, characterized in that, The effective range calculation unit (210) is further configured to: Based on the point cloud data and the weight data, determine the first coordinate that is the first coordinate and the second coordinate that is the last coordinate in the image coordinate data of all weights; the number of the first coordinate and the number of the second coordinate are determined by the convolution kernel of the neural network; Filter out the first target coordinates that appear first from all the first coordinates, and filter out the second target coordinates that appear first from all the second coordinates; The coordinate range between the first target coordinates and the second target coordinates is marked as the valid coordinate range; coordinate data within the valid coordinate range is the valid data, and coordinate data outside the valid coordinate range is the invalid data.
5. A sparse convolution accelerator with precise local density as described in claim 4, characterized in that, The subgraph generation module (300) includes: A coordinate offset unit (310) is configured to offset the image coordinate data in the corresponding block according to the corresponding weight to obtain a coordinate list; The feature multiply-accumulate tree unit (320) is configured to perform multiply-accumulate tree processing on the image feature data in the corresponding block according to the corresponding weights to obtain a feature list; the feature sub-graph includes the coordinate list and the feature list.
6. A sparse convolution accelerator with precise local density as described in claim 5, characterized in that, The feature multiply-add tree unit (320) is also configured to: The weights of the corresponding blocks are multiplied by the image feature data to obtain the multiplication result; The multiplication results of different convolution channels are accumulated to obtain the feature list; The subgraph generation module (300) further includes: Block receiving unit (330), the block receiving unit (330) is configured to receive blocks with different weights according to the data address and the weight data.
7. A sparse convolution accelerator with precise local density as described in claim 6, characterized in that, The kernel accumulation module (400) includes: The data sorting unit (410) is configured to determine the corresponding feature sub-graph based on the weight data, and sort the coordinate list in the feature sub-graph according to a preset order to obtain sorted data. An effective partitioning unit (420) is configured to partition the sorted coordinate data according to the effective coordinate range to obtain a minimum block, wherein the minimum block includes the effective data arranged in a preset order; Feature search unit (430) is configured to extract features in the minimum block to obtain a plurality of search feature data corresponding to the effective data; A feature accumulation unit (440) is configured to accumulate all the search feature data to obtain target feature data; The coordinate extraction unit (450) is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.
8. A sparse convolution accelerator with precise local density as described in claim 7, characterized in that, The coordinate extraction unit (450) is further configured to: Extract the corresponding coordinates of the search feature data when the feature search unit (430) stops the feature search, and obtain the extracted coordinate data.
9. A sparse convolution accelerator with precise local density as described in claim 1, characterized in that, The data loading module (100) includes: A data acquisition unit (110) is configured to acquire point cloud data and weight data from the neural network; A plurality of data storage units (120), each of the data storage units (120) being configured to store the point cloud data and the weight data corresponding to the weights.
10. A sparse convolution accelerator with precise local density as described in claim 9, characterized in that, The data loading module (100) also includes: A data allocation unit (130) is configured to send the point cloud data and the weight data to the corresponding data storage unit (120) according to the corresponding weights. The data storage unit (120) is also configured to store the point cloud data, determine the weight data corresponding to the weight based on the point cloud data, and store the weight data.