Precise local densification sparse convolution accelerator

Through the precise locally intensive sparse convolution accelerator, the existing sparse convolution processing problem is solved by using data chunking and dynamic chunking strategies, efficient point cloud data processing is achieved, and the inference speed is significantly accelerated.

CN119089958AActive Publication Date: 2024-12-06NANJING UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411318424.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-12-06
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The existing sparse convolution acceleration scheme is inefficient when processing point cloud data. It is mainly because the process of generating global rulebooks ignores the sparseness of point cloud data, resulting in a lot of time being wasted in invalid areas, and random reads and repeated reads lead to access. The storage overhead is large, which affects the speed of reasoning.

Method used

A sparse convolution accelerator with precise local intensiveness is designed to obtain point cloud data and weight data through the data loading module, the data blocking module performs block processing, and the sub-graph generation module performs offset processing and multiplication and addition tree processing. , the accumulation module in the kernel performs convolution kernel accumulation, generates target feature maps, adopts sparse and ordered input data formats and dynamic blocking strategies to avoid reading and computing of blank areas.

Benefits of technology

It improves the computing efficiency of sparse convolution, reduces memory access time, accurately locates the non-zero position of the output feature map, avoids the waste of computing resources, and significantly speeds up the inference speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089958A_ABST
    Figure CN119089958A_ABST
Patent Text Reader

Abstract

The invention provides a precise local densification sparse convolution accelerator, which comprises a data loading module, and is characterized in that the data loading module is configured to acquire point cloud data and weight data from a neural network; the point cloud data comprises image coordinate data and image feature data corresponding to different weights; the weight data correspond to different weights; the data blocking module is configured to carry out blocking processing on the point cloud data and the weight data to obtain a plurality of blocks; the sub-graph generation module is configured to perform offset processing and multiply-add tree processing on the blocks respectively to obtain feature sub-graphs corresponding to each weight; and the intra-kernel accumulation module is configured to divide the feature sub-graph and perform convolution kernel accumulation processing to obtain a target feature graph. Through the accelerator, the problem of low efficiency of an existing sparse convolution processing technology is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of neural network acceleration technology, and in particular to a precise local densification sparse convolution accelerator. Background Art

[0002] Scene perception models play an important role in the fields of autonomous driving and intelligent robots that have been popular in recent years. The point cloud data generated by LiDAR, as a data form containing more comprehensive information, can be used as input data for many scene perception models, making the models more accurate and the perception effect better. However, the point cloud data itself is large in volume, highly sparse, has low information density, and has random and diverse distribution patterns, which has always hindered the improvement of model reasoning speed, and occupies a large amount of storage resources, resulting in a lot of computing power waste, so sparse convolution acceleration is needed.

[0003] Most existing sparse convolution acceleration solutions are gather-scatter architectures. This type of solution requires traversing the coordinates of the entire input feature map before obtaining a rule book that can guide the convolution work. Under the guidance of this rule book, the input feature data is collected to generate the output feature map.

[0004] In this process, the strategy of generating a global rulebook ignores the sparsity of point cloud data. The so-called sparsity of point cloud data means that in the vast space that the lidar can perceive, only real objects can reflect the laser back and generate non-zero point cloud data. This is reflected in the point cloud data, that is, within the scene range represented by the point cloud data, there are only a few places with valid points, and the rest are blank. The process of generating a global rulebook requires searching the entire scene, wasting a lot of time in places where there are no valid values. Secondly, the process of obtaining feature values ​​according to the rulebook also involves a large amount of random reading and repeated reading, which is a large memory access overhead and will have a great impact on the inference speed. Summary of the invention

[0005] The present application provides a sparse convolution accelerator with precise local densification to solve the problem of low efficiency of existing sparse convolution processing technology.

[0006] The accelerator comprises:

[0007] A data loading module, wherein the data loading module is configured to obtain point cloud data and weight data from a neural network; the point cloud data includes image coordinate data and image feature data corresponding to different weights, the image coordinate data and the image feature data corresponding to one another; the weight data corresponds to different weights;

[0008] A data block module, wherein the data block module is configured to perform block processing on the point cloud data and the weight data to obtain a plurality of blocks;

[0009] A sub-graph generation module, wherein the sub-graph generation module is configured to perform offset processing and multiplication-add tree processing on the blocks respectively to obtain a feature sub-graph corresponding to each weight;

[0010] The in-core accumulation module is configured to divide the feature sub-graph and perform convolution kernel accumulation processing to obtain a target feature graph.

[0011] Preferably, each block includes valid data, invalid data, a valid coordinate range and a data address corresponding to the valid data.

[0012] Preferably, the data segmentation module includes:

[0013] An effective range calculation unit, the effective range calculation unit is configured to perform effective range calculation according to the coordinates of the point cloud data and the weight data to obtain the effective coordinate range;

[0014] A valid data calculation unit, wherein the valid data calculation unit is configured to divide the point cloud data according to the valid coordinate range to obtain the valid data and the invalid data; the valid data includes a valid number and the data address;

[0015] A block division unit, wherein the block delineation unit is configured to divide the valid coordinate range, the valid data, the invalid data, the data address and the valid quantity belonging to the same weight into the same block.

[0016] Preferably, the effective range calculation unit is further configured to:

[0017] Determine, according to the point cloud data and the weight data, a plurality of first coordinates that are closest to the front and a plurality of second coordinates that are closest to the back of the coordinates in the image coordinate data in all the weights; the number of the first coordinates and the number of the second coordinates are determined by the convolution kernel of the neural network;

[0018] Filter out the first target coordinates with the closest coordinates from all the first coordinates, and filter out the second target coordinates with the closest coordinates from all the second coordinates;

[0019] The coordinate range between the first target coordinate and the second target coordinate is marked as the valid coordinate range; the coordinate data within the valid coordinate range is the valid data, and the coordinate data outside the valid coordinate range is the invalid data.

[0020] Preferably, the subgraph generation module comprises:

[0021] A coordinate shifting unit, wherein the coordinate shifting unit is configured to perform shifting processing on the image coordinate data in the corresponding block according to the corresponding weight to obtain a coordinate list;

[0022] A feature multiply-add tree unit is configured to perform a multiply-add tree process on the image feature data in the corresponding block according to corresponding weights to obtain a feature list; the feature subgraph includes the coordinate list and the feature list.

[0023] Preferably, the feature multiply-add tree unit is further configured as:

[0024] Multiplying the weight of the corresponding block and the image feature data to obtain a multiplication result;

[0025] Accumulating the multiplication results of different convolution channels to obtain the feature list;

[0026] The sub-graph generation module further includes:

[0027] A block receiving unit is configured to receive blocks with different weights according to the data address and the weight data.

[0028] Preferably, the in-core accumulation module includes:

[0029] A data sorting unit, wherein the data sorting unit is configured to determine the corresponding characteristic subgraph according to the weight data, and sort the coordinate list in the characteristic subgraph according to a preset order to obtain sorted data;

[0030] an effective division unit, the effective division unit being configured to divide the sorted coordinate data according to the effective coordinate range to obtain a minimum block, wherein the minimum block includes the effective data arranged in a preset order;

[0031] A feature search unit, the feature search unit being configured to perform feature extraction in the minimum block to obtain a plurality of search feature data corresponding to the valid data;

[0032] A feature accumulation unit, wherein the feature accumulation unit is configured to perform accumulation processing on all the search feature data to obtain target feature data;

[0033] A coordinate extraction unit is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.

[0034] Preferably, the coordinate extraction unit is further configured as follows:

[0035] The corresponding coordinates of the search feature data when the feature search unit stops the feature search are extracted to obtain the extracted coordinate data.

[0036] Preferably, the data loading module includes:

[0037] A data acquisition unit, wherein the data acquisition unit is configured to acquire point cloud data and the weight data from the neural network;

[0038] A plurality of data storage units, each of which is configured to store the point cloud data and the weight data corresponding to the weight.

[0039] Preferably, the data loading module further includes:

[0040] A data distribution unit, the data distribution unit being configured to send the point cloud data and the weight data to the corresponding data storage unit according to the corresponding weight;

[0041] The data storage unit is further configured to store the point cloud data, confirm the weight data corresponding to the weight according to the point cloud data, and store the weight data.

[0042] As can be seen from the above content, the present application provides a precise local densification sparse convolution accelerator, the accelerator includes a data loading module, the data loading module is configured to obtain point cloud data and weight data from a neural network; the point cloud data includes image coordinate data and image feature data corresponding to different weights, the image coordinate data and the image feature data correspond one to one; the weight data corresponds to different weights; a data blocking module, the data blocking module is configured to block the point cloud data and the weight data to obtain a number of blocks; a sub-graph generation module, the sub-graph generation module is configured to perform offset processing and multiplication-add tree processing on the blocks respectively, to obtain a feature sub-graph corresponding to each weight; an in-core accumulation module, the in-core accumulation module is configured to sort the feature sub-graphs in a preset order to obtain sorted data, divide the sorted data, and perform convolution kernel accumulation processing to obtain a target feature map. The present application solves the problem of low efficiency of existing sparse convolution processing technology through the above accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1A schematic diagram of a sparse convolution accelerator for accurate local densification for this application;

[0045] Figure 2 A schematic diagram of a data loading module in a sparse convolution accelerator for accurate local densification in this application;

[0046] Figure 3 A schematic diagram of a data partitioning module in a sparse convolution accelerator for accurate local densification in this application;

[0047] Figure 4 A schematic diagram of a subgraph generation module in a sparse convolution accelerator for accurate local densification in this application;

[0048] Figure 5 A schematic diagram of an in-core accumulation module in a sparse convolution accelerator for accurate local densification in this application;

[0049] Figure 6 A workflow diagram of a data loading module in a sparse convolution accelerator for accurate local densification is provided for this application;

[0050] Figure 7 A demonstration diagram of the relationship between feature sub-graphs and target feature graphs in a sparse convolution accelerator for accurate local densification in this application;

[0051] Figure 8 This application provides a workflow diagram of an in-core accumulation module in a sparse convolution accelerator with precise local densification. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Figure 1 Schematic diagram of a sparse convolution accelerator for precise local densification in this application.

[0054] See also Figure 1 It can be seen that this embodiment provides a sparse convolution accelerator for accurate local densification, and the accelerator includes:

[0055] The data loading module 100 is configured to obtain point cloud data and weight data from a neural network; the point cloud data includes image coordinate data and image feature data corresponding to different weights, and the image coordinate data and the image feature data correspond one to one; the weight data corresponds to different weights. Specifically, in this embodiment, the data loading module 100 has the function of acquiring and storing data, by acquiring the corresponding point cloud data and the weight data corresponding to the point cloud data from the neural network, and storing them, waiting for the data to be called at any time.

[0056] It should be noted that different weights are used to obtain part of the data needed from the same batch of point cloud data according to one's own needs. The point cloud data itself does not need to correspond to weights. The point cloud data itself is a batch of points. The points have coordinate data and feature data, and are 3D stereo, not 2D images.

[0057] The accelerator further comprises:

[0058] The data block module 200 is configured to perform block processing on the point cloud data and the weight data to obtain a plurality of blocks. Specifically, in the present embodiment, since there are a large number of blank areas in most of the point cloud data, and these blank areas have no effective meaning in radar perception, and the existing sparse convolution traverses all areas including the blank areas, which results in very low efficiency of the sparse convolution; in the present embodiment, the data block module 200 sets the block mode to separate the blocks where most of the blank areas are located, thereby avoiding reading and calculating the blank areas.

[0059] It should be noted that each of the blocks includes valid data, invalid data, a valid coordinate range and a data address corresponding to the valid data.

[0060] The accelerator further comprises:

[0061] The sub-graph generation module 300 is configured to perform offset processing and multiplication-add tree processing on the blocks respectively to obtain a feature sub-graph corresponding to each weight. Specifically, in this embodiment, the sub-graph generation module 300 performs offset processing and multiplication-add tree processing on the blocks according to the corresponding weights, wherein the offset processing corresponds to the image coordinate data, and the multiplication-add tree processing corresponds to the image feature data.

[0062] The accelerator further comprises:

[0063] The in-core accumulation module 400 is configured to divide the feature sub-graph and perform convolution kernel accumulation processing to obtain a target feature graph. Specifically, in this embodiment, the in-core accumulation module 400 is used to complete the precise local densification processing of the feature sub-graph to obtain the complete target feature graph.

[0064] Among them, in the accurate local densification processing of the in-core accumulation module 400, accurate means that the first position of each minimum block must be a non-zero value. Figure 8 For example, if the minimum block is 4, the non-zero part of the output feature map will be composed of multiple minimum blocks. In this process, there will be a waste of zero values ​​in the minimum block, but because the first position is always non-zero, the non-zero position of the output feature map can be located as accurately as possible.

[0065] Local densification: First, let's introduce the difference between sparse and dense concepts. In point cloud data, sparse means that only non-zero values ​​are stored. Figure 8 For example, there are only 63 non-zero values ​​in the range of 18x18, so this algorithm will generate only 63 data. If it is densely stored, there will be 18x18=324 data, but except for the 63 non-zero data, the rest are useless zero values.

[0066] Since the distribution of non-zero data cannot be predicted in advance, for convenience, a dense coordinate range (meaning that the coordinates are continuous) is generated and the data that fits the range is processed. This is the meaning of "densification" in "local densification".

[0067] "Local" is reflected in the fact that the smallest block in this scheme is generated sequentially and on-site by collecting data in real time. The position varies depending on the non-zero distribution of the output feature map, and it is not a fixed pattern. Therefore, the smallest block will only appear in the non-zero position. This is the embodiment of "local" and also reflects "precision".

[0068] The relationship between the sub-graph generation module 300 and the in-core accumulation module 400 can be seen in Figure 7 .

[0069] Figure 7 The shaded blocks in the middle represent the valued coordinates, and it can be clearly seen that the characteristic subgraph ( Figure 7 Middle output Figure 1 and 2 ) is generated by the input feature map shifting in different directions, and the target feature map ( Figure 7 The output feature map in ( ) is generated by the superposition of all feature sub-maps.

[0070] Figure 3 A schematic diagram of a data partitioning module in a sparse convolution accelerator for precise local densification according to the present application.

[0071] See also Figure 3 It can be seen that, further, in some embodiments, the data segmentation module 200 includes:

[0072] The effective range calculation unit 210 is configured to perform effective range calculation according to the coordinates of the point cloud data and the weight data to obtain the effective coordinate range. Specifically, in the present embodiment, the effective range calculation unit 210 determines the first coordinates of the frontmost coordinates and the second coordinates of the backmost coordinates of the image coordinate data in all weights according to the point cloud data and the weight data; the number of the first coordinates and the number of the second coordinates are determined by the convolution kernel of the neural network; the first target coordinates with the frontmost coordinates are selected from all the first coordinates, and the second target coordinates with the frontmost coordinates are selected from all the second coordinates; the coordinate range between the first target coordinates and the second target coordinates is marked as the effective coordinate range; the coordinate data within the effective coordinate range is the effective data, and the coordinate data outside the effective coordinate range is the invalid data.

[0073] For example, if a 3×3 convolution kernel has 9 weights, then the data loading module 100 loads 9 sets of data, and the content of each set of data is different, so the 9 first coordinates picked out are also in different order. And the 9 coordinates have different offset directions, even if the coordinates happen to be the same, there will be a difference in order due to the offset. Therefore, from this example, we can deduce the conclusion that "the number of the first coordinates and the number of the second coordinates are determined by the convolution kernel of the neural network".

[0074] It should be noted that, for the second target coordinates, when the data obtained by all weights in the data loading link contains the last data of the overall point cloud data, that is, when the entire work comes to the last block, the selected second target coordinates are the last.

[0075] The data segmentation module 200 further includes:

[0076] The valid data calculation unit 220 is configured to divide the point cloud data according to the valid coordinate range to obtain the valid data and the invalid data; the valid data includes a valid number and the data address. Specifically, in the present embodiment, the valid data calculation unit 220 divides the point cloud data using the valid coordinate range calculated by the valid range calculation unit 210 to divide the valid data and the invalid data, and the corresponding valid data also includes a valid number and the data address.

[0077] The data segmentation module 200 further includes:

[0078] The block division unit 230 is configured to divide the valid coordinate range, the valid data, the invalid data, the data address and the valid quantity belonging to the same weight into the same block.

[0079] Specifically, in this embodiment, the data obtained by the data loading module 100 is a fixed amount, among which there will be some data that is not used in the current block. In the data block module 200, the data of several specific positions of the point cloud data corresponding to each weight will be read, and after calculation and processing, valid data and invalid data, valid coordinate range and data address corresponding to the valid data can be distinguished. And this information will be used in the subsequent process.

[0080] In the next round of data loading of the data loading module 100, the starting point of the required data of each weight is the first invalid coordinate in the current block. This dynamic block strategy can gradually improve the similarity of data with different weights in multiple rounds of data blocking, increase the amount of valid data loaded each time, and reduce repeated loading of data, thereby improving efficiency and accelerating reasoning speed.

[0081] It should be noted that the data address required by each weight refers to the relative address of the coordinate data and the feature data. For example, 128 data from 0 to 127 in the ddr are read at one time, but the calculated statistical information shows that only the first 100 are valid (since the coordinates are stored sequentially, the first part must be valid and the second part must be invalid, and there is no interspersed existence of valid and invalid data). The remaining 28 invalid data must appear in the next block, that is, the next time the data is read from ddr number 100, and 128 are read. The amount of valid data calculated by each weight is different, and the starting point of the next reading is different.

[0082] Figure 4 A schematic diagram of a subgraph generation module in a sparse convolution accelerator for precise local densification according to the present application.

[0083] See also Figure 4 It can be seen that, further, in some embodiments, the sub-graph generation module 300 includes:

[0084] A coordinate shifting unit 310, wherein the coordinate shifting unit 310 is configured to perform a shifting process on the image coordinate data in the corresponding block according to the corresponding weight to obtain a coordinate list;

[0085] A feature multiply-add tree unit 320 is configured to perform a multiply-add tree process on the image feature data in the corresponding block according to corresponding weights to obtain a feature list; the feature subgraph includes the coordinate list and the feature list.

[0086] Specifically, in this embodiment, the calculation of each block is completed by the coordinate offset unit 310 and the feature multiplication-addition tree unit 320, wherein the coordinate offset unit 310 performs offset processing on the image coordinate data in the corresponding block according to the corresponding weight, and the feature multiplication-addition tree unit 320 performs multiplication-addition tree processing on the image feature data in the corresponding block according to the corresponding weight.

[0087] There are two groups of point cloud data, one is coordinates and the other is features. The processing of coordinates is position offset. For example, for the upper left corner weight of a 3x3 convolution kernel, the coordinate (x, y) must become (x+1, y+1), while the coordinate processing method corresponding to the lower right corner weight is (x-1, y-1).

[0088] The processing of features is to multiply the features and the corresponding weight values ​​through the multiplication-add tree, and accumulate the feature-weight products of different input channels.

[0089] Furthermore, in some embodiments, the sub-graph generation module 300 further includes:

[0090] The block receiving unit 330 is configured to receive blocks with different weights according to the data address and the weight data. Specifically, in this embodiment, the block receiving unit 330 receives blocks corresponding to different weights according to the data address and the weight data, and sends the corresponding blocks to the coordinate offset unit 310 and the feature multiplication-addition tree unit 320.

[0091] Figure 5 A schematic diagram of an in-core accumulation module in a sparse convolution accelerator for precise local densification according to the present application.

[0092] See also Figure 5 It can be seen that, further, in some embodiments, the in-core accumulation module 400 includes:

[0093] The data sorting unit 410 is configured to determine the corresponding feature subgraph according to the weight data, sort the coordinate list in the feature subgraph according to a preset order, and obtain the sorted data. Specifically, in this embodiment, each weight will provide the first coordinate in its own data, and the coordinates provided by all weights will be sorted.

[0094] It should be noted that the data sorting unit 410 is a data preprocessing unit, so during the sparse convolution calculation process, it is assumed that the work of the data sorting unit 410 has been completed, that is, all calculated feature subgraphs are ordered data.

[0095] The in-core accumulation module 400 further includes:

[0096] The effective division unit 420 is configured to divide the sorted coordinate data according to the effective coordinate range to obtain a minimum block, wherein the minimum block includes the effective data arranged in a preset order. Specifically, in the present embodiment, the effective division unit 420 selects the coordinate with the highest sorting position, takes this coordinate as the starting point, and sequentially takes a fixed number of coordinates thereafter as the current effective range to form a minimum block.

[0097] It should be noted that the fixed number of coordinates is the number of coordinates within the valid coordinate range.

[0098] The step of collecting coordinates and selecting the coordinate with the highest sorting order as the starting point of the smallest block can be replaced by generating an index value through the coordinates, and then the order of the index values ​​is equivalent to the sorting of the coordinates, and the minimum value of the collected index is taken as the starting point of the smallest block.

[0099] The in-core accumulation module 400 further includes:

[0100] A feature search unit 430, wherein the feature search unit 430 is configured to perform feature extraction in the minimum block to obtain a plurality of search feature data corresponding to the valid data;

[0101] A feature accumulation unit 440, wherein the feature accumulation unit 440 is configured to perform accumulation processing on all the search feature data to obtain target feature data;

[0102] The coordinate extraction unit 450 is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.

[0103] Specifically, in this embodiment, the workflow of the feature search unit 430, the feature accumulation unit 440 and the coordinate extraction unit 450 is introduced by way of example.

[0104] For example, the coordinate (0, 0) is taken as the starting point, and the minimum block size is 16, then the valid range is (0, 0), (0, 1) ... (0, 15). After confirming the current valid range, each weight will search for points in the range and send the corresponding features to a set of customized feature accumulation units 440. Since the coordinates are stored in order, this search process is a one-way search, and the search can be stopped when the first point that does not meet the range requirements is encountered. This search method can save a lot of reasoning time.

[0105] After confirming that each weight has completed the search, the feature accumulation unit 440 is started, and the data sent by the search process is processed into the features of the target feature map and output. At the same time, the coordinate list of each weight transmits the coordinates of the search stop point, which can be used to generate the next minimum block. In this cycle, the processing of the input data can be completed and the correct target feature map can be generated.

[0106] Figure 2 A schematic diagram of a data loading module in a sparse convolution accelerator for precise local densification according to the present application.

[0107] See also Figure 2 It can be seen that, further, in some embodiments, the data loading module 100 includes:

[0108] A data acquisition unit 110, wherein the data acquisition unit 110 is configured to acquire point cloud data and the weight data from the neural network;

[0109] A plurality of data storage units 120, each of the data storage units 120 is configured to store the point cloud data and the weight data corresponding to the weight.

[0110] Specifically, in this embodiment, the data acquisition unit 110 is used to acquire relevant data from the neural network, and each of the data storage units 120 is used to store data corresponding to the weight.

[0111] The data loading module 100 further includes:

[0112] A data distribution unit 130, wherein the data distribution unit 130 is configured to send the point cloud data and the weight data to the corresponding data storage unit 120 according to the corresponding weight;

[0113] The data storage unit 120 is further configured to store the point cloud data, determine the weight data corresponding to the weight according to the point cloud data, and store the weight data.

[0114] Specifically, in this embodiment, the data distribution unit 130 is used to distribute the data acquired by the data acquisition unit 110 and store the corresponding data in the corresponding data storage unit 120 .

[0115] The workflow of the data loading module 100 for loading the point cloud data and the weight data is shown in Figure 6 .

[0116] Including coordinates and feature data, to be precise, 0-17 can be understood as relative addresses, that is, coordinates and features are stored in different areas and are not loaded at the same time. The 0-17 shown in the figure is 0-17 in each area, not the absolute address of DDR. When actually reading, the relative address will be mapped to the absolute address.

[0117] Figure 6 The 0-17 in the figure can be understood as relative addresses, that is, the coordinates and features are stored in different areas and are not loaded at the same time. The 0-17 shown in the figure is the 0-17 in each area, not the absolute address of the DDR. When reading, the relative address will be mapped to the absolute address.

[0118] Figure 6 The example in the figure gives three weights, each with its own data requirement (this requirement is calculated by the work of the previous block). Since the requirements are different but have similarities, loading data for each weight individually will repeat the reading behavior, so a union is calculated, that is, 2-15. According to this requirement, data is taken from the DDR and broadcast to the three weights. Each weight corresponds to a data storage unit 120, and whether the data is needed is determined according to the corresponding requirement.

[0119] This embodiment has the following advantages:

[0120] First, a sparse and ordered input data format is used. Sparse means that the input data only contains valid points with non-zero values. This ensures that only valid points are processed. Ordered means that the coordinates of non-zero values ​​are arranged in a certain order. This ensures that data access is sequentially read and written, saving memory access time.

[0121] Secondly, we give up generating a global rulebook and directly generate a batch of output subgraphs based on the weights through a simple, regular calculation method that does not involve random reading and writing. The number of these subgraphs is consistent with the number of weights in the convolution kernel, and the complete output feature map is formed when they are superimposed together.

[0122] Finally, with these output sub-graphs as input, a batch of minimum blocks are deduced from the output sub-graphs in turn by using precise local densification. These minimum blocks will accurately locate all non-zero points in the output feature graph. While the minimum blocks are being generated, the input feature data enters the subsequent computing unit in a sequential read-write manner, saving a lot of memory access time. The subsequent computing unit can complete the generation of the output feature graph through a simple fixed calculation process. In this process, all time and computing resources will be placed at the valid point position of the output feature graph. This avoids the waste of computing resources and can effectively speed up the calculation process of sparse convolution.

Claims

1. A sparse convolution accelerator for accurate local densification, characterized in that: The accelerator is applied to a neural network, and the accelerator comprises: A data loading module (100), the data loading module (100) being configured to obtain point cloud data and weight data from a neural network; the point cloud data comprising image coordinate data and image feature data corresponding to different weights, the image coordinate data and the image feature data corresponding one to one; the weight data corresponding to different weights; A data block module (200), the data block module (200) being configured to perform block processing on the point cloud data and the weight data to obtain a plurality of blocks; A subgraph generation module (300), wherein the subgraph generation module (300) is configured to perform offset processing and multiplication-add tree processing on the blocks respectively to obtain a feature subgraph corresponding to each weight; The in-core accumulation module (400) is configured to divide the feature sub-graph and perform convolution kernel accumulation processing to obtain a target feature graph.

2. The sparse convolution accelerator for accurate local densification according to claim 1, characterized in that: Each block includes valid data, invalid data, a valid coordinate range and a data address corresponding to the valid data.

3. The sparse convolution accelerator for accurate local densification according to claim 2, characterized in that: The data segmentation module (200) comprises: An effective range calculation unit (210), the effective range calculation unit (210) being configured to perform effective range calculation according to the coordinates of the point cloud data and the weight data to obtain the effective coordinate range; A valid data calculation unit (220), the valid data calculation unit (220) being configured to divide the point cloud data according to the valid coordinate range to obtain the valid data and the invalid data; the valid data includes a valid number and the data address; A block division unit (230), wherein the block delineation unit (230) is configured to divide the valid coordinate range, the valid data, the invalid data, the data address and the valid quantity belonging to the same weight into the same block.

4. The sparse convolution accelerator for accurate local densification according to claim 3, characterized in that: The effective range calculation unit (210) is further configured to: Determine, according to the point cloud data and the weight data, a plurality of first coordinates that are closest to the front and a plurality of second coordinates that are closest to the back of the coordinates in the image coordinate data in all the weights; the number of the first coordinates and the number of the second coordinates are determined by the convolution kernel of the neural network; Filter out the first target coordinates with the closest coordinates from all the first coordinates, and filter out the second target coordinates with the closest coordinates from all the second coordinates; The coordinate range between the first target coordinate and the second target coordinate is marked as the valid coordinate range; the coordinate data within the valid coordinate range is the valid data, and the coordinate data outside the valid coordinate range is the invalid data.

5. The sparse convolution accelerator for accurate local densification according to claim 4, characterized in that: The sub-graph generation module (300) comprises: A coordinate shifting unit (310), the coordinate shifting unit (310) being configured to perform a shifting process on the image coordinate data in the corresponding block according to a corresponding weight, so as to obtain a coordinate list; A feature multiplication-addition tree unit (320), wherein the feature multiplication-addition tree unit (320) is configured to perform multiplication-addition tree processing on the image feature data in the corresponding block according to corresponding weights to obtain a feature list; the feature subgraph includes the coordinate list and the feature list.

6. The sparse convolution accelerator for accurate local densification according to claim 5, characterized in that: The feature multiply-add tree unit (320) is further configured to: Multiplying the weight of the corresponding block and the image feature data to obtain a multiplication result; Accumulating the multiplication results of different convolution channels to obtain the feature list; The sub-graph generation module (300) further comprises: The block receiving unit (330) is configured to receive blocks with different weights according to the data address and the weight data.

7. The sparse convolution accelerator for accurate local densification according to claim 6, characterized in that: The in-core accumulation module (400) comprises: A data sorting unit (410), the data sorting unit (410) being configured to determine the corresponding characteristic subgraph according to the weight data, and to sort the coordinate list in the characteristic subgraph according to a preset order to obtain sorted data; An effective division unit (420), the effective division unit (420) being configured to divide the sorted coordinate data according to the effective coordinate range to obtain a minimum block, wherein the minimum block includes the effective data arranged in a preset order; A feature search unit (430), the feature search unit (430) being configured to perform feature extraction in the minimum block to obtain a plurality of search feature data corresponding to the valid data; A feature accumulation unit (440), wherein the feature accumulation unit (440) is configured to perform accumulation processing on all the search feature data to obtain target feature data; A coordinate extraction unit (450), wherein the coordinate extraction unit (450) is configured to extract the corresponding coordinates of the search feature data to obtain extracted coordinate data; the target feature map includes the target feature data and the extracted coordinate data.

8. The sparse convolution accelerator for accurate local densification according to claim 7, characterized in that: The coordinate extraction unit (450) is further configured to: The corresponding coordinates of the search feature data when the feature search unit (430) stops the feature search are extracted to obtain the extracted coordinate data.

9. The sparse convolution accelerator for accurate local densification according to claim 1, characterized in that: The data loading module (100) comprises: A data acquisition unit (110), the data acquisition unit (110) being configured to acquire point cloud data and the weight data from the neural network; A plurality of data storage units (120), each of the data storage units (120) being configured to store the point cloud data and the weight data corresponding to the weight.

10. The sparse convolution accelerator for accurate local densification according to claim 9, characterized in that: The data loading module (100) further comprises: A data distribution unit (130), the data distribution unit (130) being configured to send the point cloud data and the weight data to the corresponding data storage unit (120) according to the corresponding weight; The data storage unit (120) is further configured to store the point cloud data, confirm the weight data corresponding to the weight according to the point cloud data, and store the weight data.

Citation Information

Patent Citations

  • FPGA-oriented sparse convolutional neural network multi-level storage computing system

    CN113780529A

  • Sparse convolution processing method and device in image processing

    CN115311536A

  • Accelerator and acceleration method for sparse convolutional neural network

    CN115640833A