Weight-oriented sparse convolution accelerator hardware device

Abstract

The invention provides a weight-oriented sparse convolution accelerator hardware device, which comprises a part and a generation module configured to obtain features of a preset number of activation points on a feature map and generate a feature matrix; multiplying the feature matrix by the weight matrix to obtain a partial sum corresponding to the activation point; the read-write control module is configured to send the activation point coordinates to the activation point coordinate generation module and send the partial sum to the partial sum summation module; the activation point coordinate generation module is configured to generate an activation point coordinate set according to the ordinate of the activation point coordinates; the partial sum summation module is configured to perform summation on the partial sum to obtain an output feature; and the calculation module is configured to obtain the target features of the feature map based on the activation point coordinate set and the output features, so as to solve the problem that when a conventional sparse convolution accelerator uses a rule table to calculate the feature map, the rule table also needs to be regenerated when the input feature map changes, and the calculation efficiency is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of accelerator architecture, and in particular to a weight-oriented sparse convolution accelerator hardware device. Background Art

[0002] By emitting laser beams and receiving their reflected signals, LiDAR can accurately capture the three-dimensional information of the surrounding environment and generate dense point cloud data. These point cloud data not only contain rich spatial geometric information, but also reflect the structural characteristics and dynamic changes of the environment, providing a solid data foundation for the navigation and obstacle avoidance of autonomous vehicles and the path planning and object recognition of intelligent robots. Therefore, the industry has attached increasing importance to the processing capabilities of point cloud data, and a number of point cloud data processing neural network models based on convolutional neural networks have emerged.

[0003] In order to utilize the sparsity of point cloud data to reduce redundant calculations while ensuring that the convolution operation results are consistent with traditional convolution, the sparse convolution accelerator has gradually developed into a gather-scatter architecture. Under the control of the rule book, it collects the input feature map data within the receptive field (convolution kernel coverage range) in a round of calculation, and assigns the results to the correct position on the output feature map after the operation.

[0004] Since the rule table is dynamically generated based on the input feature map, the rule table should be regenerated when the input feature map changes. The generation of the rule table requires obtaining the coordinates of all activation points (i.e., points whose features are not 0) in the input feature map, which is far more than the receptive field coverage area of ​​the current calculation process, thus reducing the calculation efficiency. Summary of the invention

[0005] The present application provides a weight-oriented sparse convolution accelerator hardware device to solve the technical problem that when the existing sparse convolution accelerator uses a rule table to calculate a feature map, the rule table also needs to be regenerated when the input feature map changes, resulting in a reduction in the computing efficiency of the sparse convolution accelerator.

[0006] The present application provides a weight-oriented sparse convolution accelerator hardware device, which is applied to a convolution accelerator, wherein the convolution accelerator is provided with a convolution kernel, and the convolution kernel is used to generate a weight matrix; the device comprises: a communication-connected partial sum generation module, a read-write control module, an activation point coordinate generation module, a partial sum summation module, and a calculation module;

[0007] The sections and generation modules are configured to:

[0008] Obtain features of a preset number of activation points on the feature map and generate a feature matrix;

[0009] Multiplying the feature matrix and the weight matrix to obtain a partial sum corresponding to the activation point;

[0010] The read-write control module is configured as follows:

[0011] Acquire the activation point coordinates on the feature map and send the activation point coordinates to the activation point coordinate generation module, and send the partial sum to the partial sum summation module;

[0012] The activation point coordinate generation module is configured as follows:

[0013] Generate an activation point coordinate set according to the ordinate of the activation point coordinate; the activation point coordinates in the activation point coordinate set are ordered and non-repetitive;

[0014] The partial summation module is configured as:

[0015] According to the summation rule, the partial sums are summed to obtain output features;

[0016] The computing module is configured as follows:

[0017] Based on the activation point coordinate set and the output features, target features of the feature map are obtained.

[0018] In some embodiments, the partial summation module is further configured to:

[0019] Calculate the coordinate difference between the activation point and the potential neighbor point on the feature map, and use a lookup table to obtain the target potential neighbor point and the first binary bitmap corresponding to the target potential neighbor point; the potential neighbor points are the two activation points on the left, two on the right, and two rows below the activation point;

[0020] Multiplying the first binary bitmap by the partial sum corresponding to the activation point to obtain the partial sum of the target potential neighbor point;

[0021] Rotate the first binary bitmap at a preset angle around the center to obtain a second binary bitmap;

[0022] Fill the partial sum of the target potential neighbor points into the non-zero positions in the second binary bitmap in order to obtain the partial sum of the target potential neighbor points;

[0023] The partial sum of the target potential neighbor point and the partial sum corresponding to the activation point are added to obtain the output feature.

[0024] In some embodiments, the apparatus further comprises:

[0025] A disassembling unit, the disassembling unit being in communication with the part and the generating module, and being configured to:

[0026] Get feature map;

[0027] The feature map is decomposed into activation point features and activation point coordinates.

[0028] In some embodiments, the apparatus further comprises:

[0029] an activation point coordinate buffer, the activation point coordinate buffer being in communication with the disassembly unit and being configured to:

[0030] Storing the activation point coordinates;

[0031] A copying unit, the copying unit is in communication with the activation point coordinate buffer and is configured to:

[0032] Get the coordinates of the activation point and copy it.

[0033] In some embodiments, the apparatus further comprises:

[0034] A splicing unit, the splicing unit is in communication with the part and the generating module and the copying unit, and is configured as follows:

[0035] Receive the activation point coordinates sent by the replication unit, and concatenate the activation point coordinates with the partial sum to obtain a composite partial sum; the composite partial sum includes: the partial sum and the activation point coordinates corresponding to the partial sum.

[0036] In some embodiments, the weight-oriented sparse convolution accelerator hardware device further includes:

[0037] A first FIFO buffer, the first FIFO buffer is communicatively connected with the replication unit and the read / write control module, and is configured as follows:

[0038] receiving and storing the activation point coordinates sent by the replication unit;

[0039] A coordinate line buffer, the coordinate line buffer is communicatively connected with the read-write control module and the activation point coordinate generation module;

[0040] Receiving and storing the activation point coordinates sent by the read-write control module;

[0041] The read-write control module is also configured as:

[0042] After the activation point coordinate generation module generates the activation point coordinate set, the activation point coordinates stored in the coordinate row buffer are cleared.

[0043] In some embodiments, the read-write control module is further configured to:

[0044] Acquire the first ordinate of the activation point coordinates stored in the coordinate line buffer;

[0045] If the absolute value of the difference between the second ordinate of the activation point coordinate sent to the coordinate line buffer and the first ordinate is greater than a preset value, the activation point coordinate corresponding to the first ordinate is stored in the activation point coordinate set and then resent.

[0046] In some embodiments, the apparatus further comprises:

[0047] A second FIFO buffer is communicatively connected with the splicing unit and the read / write control module and is configured as follows:

[0048] receiving and storing the composite part sent by the splicing unit;

[0049] A composite part and a buffer, the composite part and the buffer are communicatively connected with the read-write control module, the part and the summing module;

[0050] receiving and storing the composite part sent by the read-write control module;

[0051] The read-write control module is also configured as:

[0052] After the partial sum summing module finishes summing the composite partial sum, the composite partial sum stored in the composite partial sum buffer is cleared.

[0053] In some embodiments, the read-write control module is further configured to:

[0054] When sending the activation point coordinates to the coordinate line buffer, or sending the composite part sum to the composite part sum buffer, obtaining the ordinate of the activation point coordinates or the activation point coordinates in the composite part sum to be stored;

[0055] Obtaining whether the coordinate line buffer or the composite part and buffer stores the activation point coordinates, and if so, obtaining the vertical coordinates of the stored activation point coordinates;

[0056] Acquire the difference between the stored ordinate of the activation point coordinate and the ordinate of the activation point coordinate to be stored;

[0057] If the difference is less than a preset value, it is stored in the coordinate line buffer or the composite part and buffer;

[0058] If the difference is greater than the preset value, the difference is obtained again after the activation point coordinates or the composite part sum are cleared until the difference is less than the preset value, and the activation point coordinates to be stored are stored in the coordinate row buffer, or the composite part sum to be stored is stored in the composite part sum buffer.

[0059] In some embodiments, the apparatus further comprises:

[0060] An activation point coordinate set buffer, the activation point coordinate set buffer is in communication with the activation point coordinate generation module and the calculation module; and is configured to:

[0061] storing the activation point coordinate set;

[0062] An output feature buffer, the output feature buffer is in communication with the partial summing module and the calculation module; and is configured to:

[0063] The output features are stored.

[0064] The present application provides a weight-oriented sparse convolution accelerator hardware device, which is applied to a convolution accelerator, wherein the convolution accelerator is provided with a convolution kernel, and the convolution kernel is used to generate a weight matrix; the device comprises: a partial sum generation module, a read-write control module, an activation point coordinate generation module, a partial sum summation module, and a calculation module connected in communication; the partial sum generation module is configured to: obtain features of a preset number of activation points on a feature map, and generate a feature matrix; multiply the feature matrix and the weight matrix to obtain the partial sum corresponding to the activation point; the read-write control module is configured to: obtain the activation point coordinates on the feature map and send the activation point coordinates to the activation point coordinate generation module block, sending the partial sum to the partial sum summing module; the activation point coordinate generating module is configured to: generate an activation point coordinate set according to the ordinate of the activation point coordinate; the activation point coordinates in the activation point coordinate set are ordered and non-repetitive; the partial sum summing module is configured to: sum the partial sums according to the summation rule to obtain the output feature; the calculation module is configured to: obtain the target feature of the feature map based on the activation point coordinate set and the output feature, so as to solve the problem that when the current sparse convolution accelerator uses a rule table to calculate the feature map, when the input feature map changes, the rule table also needs to be regenerated, resulting in reduced calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0066] Figure 1 This is a schematic diagram of the structure of the weight-oriented sparse convolution accelerator hardware device in this application;

[0067] Figure 2 It is a schematic diagram of some and generated modules in this application when running;

[0068] Figure 3 This is a schematic diagram of the read-write control module in this application when it is running;

[0069] Figure 4 This is a schematic diagram of the partial summation module in this application during operation.

[0070] Description of reference numerals:

[0071] 1-partial sum generation module; 11-disassembly unit; 12-activation point coordinate buffer; 13-copying unit; 14-splicing unit; 15-second FIFO buffer; 16-composite partial sum buffer; 2-read-write control module; 21-first FIFO buffer; 22-coordinate row buffer; 3-activation point coordinate generation module; 4-partial summation module; 5-calculation module; 6-activation point coordinate set buffer; 7-output feature buffer. DETAILED DESCRIPTION

[0072] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0073] In some technologies, when a sparse convolution accelerator uses a rule table to calculate a feature map, the rule table also needs to be regenerated when the input feature map changes, resulting in reduced computing efficiency. In order to solve this technical problem, the present application provides a weight-oriented sparse convolution accelerator hardware device. The following describes the structure of each part of the weight-oriented sparse convolution accelerator hardware device:

[0074] For example, with the development of new energy vehicles, intelligent robots and other fields, sensors that can perceive 3D scenes, represented by lidar, are being used more and more widely. Among the data types acquired by the above sensors, point cloud data types are very common. Therefore, the industry has attached more and more importance to the processing capabilities of point cloud data, and point cloud data processing neural network models based on convolutional neural networks have emerged. Point cloud data has the characteristics of permutation invariance, sparsity, and unstructuredness. The above characteristics make point cloud data occupy relatively low storage space, but it is difficult to perform sparse convolution operations efficiently and in parallel, which greatly affects the reasoning speed of the point cloud data processing neural network.

[0075] In order to utilize the sparsity of point clouds to reduce redundant calculations while ensuring that the results of convolution operations are consistent with traditional convolutions, sparse convolution accelerators have gradually developed into a gather-scatter architecture. Under the control of a rule book, the input feature map data within the receptive field (i.e., the coverage range of the convolution kernel) in a round of calculation is collected, and the results are assigned to the correct position on the output feature map after the operation.

[0076] Since the rule table is dynamically generated based on the input feature map, the rule table needs to be regenerated when the input feature map changes. The generation of the rule table requires obtaining the coordinates of all activation points (i.e., points whose features are not 0) in the input feature map, which are far more than the receptive field coverage area of ​​the current calculation process, thus reducing the calculation efficiency. At the same time, since in a calculation round, the receptive field coverage area is only determined by the coordinates of the input feature map activation point at its center, and the coordinates of the central activation point of the next calculation round cannot be predicted through the information of the current calculation round, the receptive field coverage area of ​​the next calculation round is unpredictable, resulting in a round of calculation that is difficult to reuse the data required for the previous round of calculation, and the memory access overhead increases, resulting in a reduction in the effect of optimizing the calculation unit on improving the overall hardware running speed.

[0077] Depend on Figure 1It can be seen that the present application provides a weight-oriented sparse convolution accelerator hardware device, which is applied to a convolution accelerator, wherein the convolution accelerator is provided with a convolution kernel, and the convolution kernel is used to generate a weight matrix; it includes: a communication-connected partial sum generation module 1, a read-write control module 2, an activation point coordinate generation module 3, a partial sum summation module 4, and a calculation module 5; the partial sum generation module 1 is configured to: obtain the features of a preset number of activation points on the feature map, and generate a feature matrix; multiply the feature matrix and the weight matrix to obtain the partial sum corresponding to the activation point; the main structure of the partial sum generation module 1 is a pulsation array; the function of the pulsation array is to perform fast matrix multiplication. In the hardware device in the present application, the matrices that the weight-oriented partial sum generation module 1 needs to calculate are the feature matrix F and the weight matrix W, respectively. Among them, each row in the feature matrix F sequentially records the eigenvalues ​​on all channels corresponding to a certain point on the input feature map; each column of the weight matrix W sequentially records the eigenvalues ​​on all channels corresponding to the weight of a certain position of the convolution kernel. The systolic array calculates F×W, and each row of the result matrix represents all partial sums corresponding to a point on the input feature map, and each column of the result matrix represents the partial sum of the weights of all input feature map activation points corresponding to a certain position of the convolution kernel calculated in this round.

[0078] For example, the calculation of the partial sum can be obtained by matrix multiplication. Taking 3×3 convolution as an example, Figure 2 As shown in the figure, the left matrix has n rows and ic_num columns, recording the eigenvalues ​​of all channels of the n activation points of the input feature map; the right matrix has ic_num rows and 9 columns, recording the values ​​of all channels of the 9 weights of the convolution kernel. Multiplying the left and right matrices is equivalent to multiplying each of the n activation points with the 9 weights respectively, and obtaining the partial sum of the 9 weights corresponding to the n points.

[0079] The read-write control module 2 is configured to: obtain the activation point coordinates on the feature map and send the activation point coordinates to the activation point coordinate generation module 3, and send the partial sum to the partial sum summation module 4; the read-write control module 2 is used to parse the obtained activation point coordinates and store them in their respective buffers, and at the same time detect whether the data stored in the buffer has met the conditions for the subsequent module to start calculation, and clear the buffer when the subsequent module calculation is completed; the activation point coordinate generation module 3 is configured to: generate an activation point coordinate set according to the vertical coordinate of the activation point coordinate; the activation point coordinates in the activation point coordinate set The activation point coordinate generation module 3 is used to calculate the coordinates of the activation points on the output feature map, and the activation point coordinates retain the sparse information of the output feature map, which is convenient for subsequent calculations; the activation point coordinates in the activation point coordinate set are used to obtain the output features in the partial summation module 4 in an orderly and non-repetitive manner, ensuring that the output features will not be repeated, thereby improving the running speed; the partial summation module 4 is configured to: sum the partial sums according to the summation rule to obtain the output features; the calculation module 5 is configured to: obtain the target features of the feature map based on the activation point coordinate set and the output features.

[0080] The present application proposes a weight-oriented efficient sparse convolution accelerator hardware device, which changes the collection-scattering operation process compared to the current convolution accelerator device. When acquiring data, the calculation rounds are no longer divided according to the receptive field, but the calculation rounds are divided according to the size of the pulsation array for generating partial sums, and the data read in different calculation rounds will not have repeated parts, reducing the memory access overhead; the convolution kernel weights are directly batch-multiplied with the fixed number of features of the input feature map using the pulsation array to form partial sums, and then subsequent processing is performed to improve the degree of parallel calculation; the generation of the rule table is cancelled, and in the process of summing the partial sums, the addition rule of the partial sums is directly obtained by using the relative position relationship of the activation points in the input feature map, thereby improving the operation efficiency; the partial dense storage method is used to temporarily store the output features with a small number of channels obtained by a round of calculation, and the buffer storage overhead is exchanged for the independence of the two different operation processes of the calculation of the output features and the generation of the coordinates of the activation points of the output feature map, thereby increasing the overlap of different operation times.

[0081] like Figure 4As shown, the partial sum sum module 4 is further configured to: calculate the coordinate difference between the activation point and the potential neighbor point on the feature map, use the lookup table to obtain the target potential neighbor point, and the first binary bitmap corresponding to the target potential neighbor point; the potential neighbor points are the activation points two to the left, two to the right and two rows below the activation point; multiply the first binary bitmap by the partial sum corresponding to the activation point to obtain the partial sum of the target potential neighbor point; rotate the first binary bitmap around the center according to a preset angle to obtain a second binary bitmap; fill the partial sum of the target potential neighbor point into the non-zero position in the second binary bitmap in sequence to obtain the partial sum of the target potential neighbor point; add the partial sum of the target potential neighbor point to the partial sum corresponding to the activation point to obtain the output feature.

[0082] For example, the feature value of a certain activation point of the output feature map is the sum of the partial sums of multiple activation points of the input feature map. Therefore, after obtaining the partial sums of the activation points of the input feature map, it is necessary to sum the partial sums of different activation points under a certain rule. The summation rule consists of two parts: determining which partial sums of potential neighbor points will be used by the difference in coordinates between the potential neighbor point and the reference point; and moving the partial sum of the required potential neighbor point to the correct position and adding it to the partial sum of the reference point. Figure 4 As shown, first calculate the difference between the potential neighbor point and the reference point coordinates, and use LUT (lookup table) to directly query the bitmap (binary bitmap) corresponding to the coordinate difference. Use the bitmap to filter the partial sum of the potential neighbor points and select the partial sum to be included in the sum. Rotate the bitmap about the center to get the position where the partial sum of the potential neighbor points after the move is not 0. Re-fill the filtered partial sum of the potential neighbor points in order to get the partial sum of the potential neighbor points after the move, which can be added to the partial sum of the reference point at the same position.

[0083] For example, Figure 4 As shown, the partial summation module 4 first calculates the coordinate difference between the activation point and the potential neighbor point on the feature map, that is, (Δy, Δx), and uses the lookup table to obtain the target potential neighbor point, that is, the LUT (lookup table) in the following table, and the first binary bitmap corresponding to the target potential neighbor point; multiplies the first binary bitmap with the partial sum corresponding to the activation point to obtain the partial sum of the target potential neighbor point; rotates the first binary bitmap around the center according to a preset angle to obtain a second binary bitmap; fills the partial sum of the target potential neighbor point into the non-zero position in the second binary bitmap in sequence, as shown in FIG. Figure 4 As shown, a partial sum of the target potential neighbor point is obtained; the partial sum of the target potential neighbor point is added to the partial sum corresponding to the activation point to obtain an output feature.

[0084] For example, the LUT storage content of 3×3 convolution is shown in the following table:

[0085]

[0086]

[0087] Depend on Figure 1 It can be seen that the device further includes: a disassembly unit 11, which is in communication with the part and generation module 1 and is configured to: obtain a feature map; disassemble the feature map into activation point features and activation point coordinates. The activation point features are used for subsequent calculation of output features, and the activation point coordinates are used to generate an activation point coordinate set.

[0088] Depend on Figure 1 It can be known that the device further includes: an activation point coordinate buffer 12, which is in communication connection with the disassembly unit 11 and is configured to: store the activation point coordinates; the activation point coordinate buffer 12 is used by the splicing unit 14 to temporarily store the activation point coordinates in the activation point coordinate buffer 12 when splicing the activation point coordinates and the partial sum, and then perform subsequent applications after the splicing is completed; a copying unit 13, which is in communication connection with the activation point coordinate buffer 12 and is configured to: obtain the activation point coordinates and copy them. One of the activation point coordinates copied by the copying unit 13 is used for the generation of the activation point coordinate set, and the other is used for splicing with the partial sum.

[0089] Depend on Figure 1 It can be seen that the device further includes: a splicing unit 14, which is in communication with the partial sum generation module 1 and the copying unit 13, and is configured to: receive the activation point coordinates sent by the copying unit 13, splice the activation point coordinates and the partial sum to obtain a composite partial sum; the composite partial sum includes: the partial sum and the activation point coordinates corresponding to the partial sum. The composite partial sum is used to subsequently generate output features.

[0090] Depend on Figure 1It can be known that the device further includes: a first FIFO buffer 21, the first FIFO buffer 21 is connected in communication with the copying unit 13 and the read-write control module 2, and is configured to: receive and store the activation point coordinates sent by the copying unit 13; a coordinate line buffer 22, the coordinate line buffer 22 is connected in communication with the read-write control module 2 and the activation point coordinate generation module 3; receives and stores the activation point coordinates sent by the read-write control module 2; the read-write control module 2 is also configured to: after the activation point coordinate generation module 3 generates the activation point coordinate set, clear the activation point coordinates stored in the coordinate line buffer 22. The activation point coordinates or the composite part and stored in the coordinate line buffer 22 or the composite part and buffer 16 are deleted in real time by the read-write control module 2, so as to reduce the cache pressure of the coordinate line buffer 22 and the composite part and buffer 16, and improve the operation speed of the device.

[0091] In this embodiment, the read-write control module 2 is further configured to: obtain the first ordinate of the activation point coordinate stored in the coordinate line buffer 22; if the absolute value of the difference between the second ordinate of the activation point coordinate sent to the coordinate line buffer 22 and the first ordinate is greater than a preset value, wait until the activation point coordinate corresponding to the first ordinate is stored in the activation point coordinate set and then resend. While the partial sum is being summed, the coordinates of the activation point of the output feature map can be generated. When the partial sum of the activation point of the input feature map enters the composite partial sum buffer 16, the coordinates of the activation point enter the coordinate line buffer 22. The process of generating the coordinates of the activation point of the output feature map needs to determine the possible influence range of the data recorded in the coordinate line buffer 22 on the output feature map, and determine the real influence range through the new activation point coordinates given by the read-write control module 2. For example, in the scenario of 3×3 convolution, when the coordinate line buffer 22 only stores the horizontal coordinate of the activation point of y=1 on the input feature map, the data in the coordinate line buffer 22 may affect the horizontal coordinates of the three rows of activation points of y=0, y=1, and y=2 on the output feature map. At this time, if the new activation point coordinates given by the read-write control module 2 will not affect the three rows of activation points on the output feature map, then the output feature map activation point coordinate set is the union of the horizontal coordinate of the activation point y=1 on the input feature map, which is decremented by 1, incremented by 1, and itself. If the coordinate line buffer 22 stores the horizontal coordinates of the activation points of y=1 and y=2 on the input feature map, then their possible impact range is the activation points of y=0, y=1, y=2, and y=3 on the output map. At this time, if the vertical coordinate of the new activation point given by the read-write control module 2 is 4, it will also affect the activation point of y=3 on the output feature map. For the new activation point coordinates, it is necessary to wait for the activation point coordinates stored in the coordinate line buffer 22 to complete the calculation and clear, and then store them in the coordinate line buffer 22; then the horizontal coordinates stored in the coordinate line buffer 22, which are decremented by 1, incremented by 1, and themselves, can only be used as the horizontal coordinates of the activation points of y=0, y=1, and y=2 on the output feature map. The actual influence range is determined by subtracting the new activation point coordinates provided by the read-write control module 2 from the possible influence range of the data recorded in the coordinate line buffer 22 on the output feature map, thereby enabling the output feature map activation point coordinates to be determined without duplication or omission.

[0092] Depend on Figure 1It can be seen that it also includes: a second FIFO buffer 15, which is connected in communication with the splicing unit 14 and the read-write control module 2, and is configured to: receive the composite part and store it sent by the splicing unit 14; a composite part and buffer 16, which is connected in communication with the read-write control module 2 and the partial summing module 4; receive the composite part and store it sent by the read-write control module 2; the composite part and buffer 16 is used to separate points belonging to different row positions of the input feature map; the composite part and buffer 16 is provided with a plurality of storage bodies, which respectively store the point coordinates of the input feature map corresponding to their respective row numbers and the data formed by the splicing part and. The read-write control module 2 is also configured to: after the partial summing module 4 completes the summing of the composite part and, then, clear the composite part and stored in the composite part and buffer 16. The activation point coordinates or the composite part and stored in the coordinate line buffer 22 or the composite part and buffer 16 are deleted in real time through the read-write control module 2 to reduce the cache pressure of the coordinate line buffer 22 and the composite part and buffer 16 and improve the operating speed of the device.

[0093] like Figure 3 As shown, the read-write control module 2 is further configured to: when sending the activation point coordinates to the coordinate row buffer 22, or sending the composite part and to the composite part and buffer 16, obtain the vertical coordinate of the activation point coordinates to be stored or the activation point coordinates in the composite part and; obtain whether the activation point coordinates are stored in the coordinate row buffer 22 or the composite part and buffer 16, and if so, obtain the vertical coordinate of the stored activation point coordinates; obtain the difference between the vertical coordinate of the stored activation point coordinates and the vertical coordinate of the activation point coordinates to be stored; if the difference is less than a preset value, store it in the coordinate row buffer 22 or the composite part and buffer 16; if the difference is greater than a preset value, wait until the activation point coordinates or the composite part and are cleared, re-obtain the difference until the difference is less than a preset value, and store the activation point coordinates to be stored in the coordinate row buffer 22, or store the composite part and to be stored in the composite part and buffer 16.

[0094] Exemplarily, taking 3×3 convolution as an example, after obtaining the partial sum of 9 weights corresponding to each activation point, it is necessary to accumulate the partial sums of different activation points to obtain the final output result. When accumulating the partial sums, it is necessary to use certain points as reference centers (called reference points) and compare coordinates with other points (called potential neighbor points) to determine the accumulation rule of the partial sums. At the same time, the reference point also determines the position of the partial sum accumulation result on the output feature map. For example, in the operation of 3×3 convolution, the partial sum accumulation result is located in a 3×3 area centered on the reference point on the output feature map. Figure 3 As shown, in the 3×3 convolution operation, only searching for potential neighbor points in the input feature map containing a row of reference points and the area of ​​two consecutive rows below can ensure that all correct output results appear at least once. Therefore, after obtaining the information of the new activation point, the read-write control module 2 will compare it with the activation point coordinates currently stored in the buffer, i.e., the coordinate row buffer 22 and the composite part and buffer 16. If the new activation point coordinates exceed the two rows below the reference point, they are not written into the buffer temporarily, but the subsequent module operation is started. When the subsequent module operation is completed, a new reference point is selected from the next vertical coordinate activation point stored in the buffer, the buffer storing the old reference point is cleared, and space is freed up to store the new activation point.

[0095] For example, Figure 3 As shown, after the read-write control module 2 obtains the ordinate of the activation point coordinates to be stored or the ordinate of the activation point coordinates in the composite part and, it is determined whether the corresponding coordinate line buffer 22 or the composite part and buffer 16 stores the activation point coordinates or the composite part and. If not, it is directly stored. If so, the activation point coordinates of one row are used as the reference point coordinates, as shown in FIG. Figure 3 As shown, when the stored reference point coordinate is 0, if the activation point coordinate to be stored is 1, and the absolute value of the difference between the two is not greater than 2, it can be directly stored. When the absolute value of the difference between the activation point coordinate to be stored and the reference point coordinate is greater than 2, a new reference point needs to be selected for comparison, and the compared reference point coordinates are used for subsequent calculations. After the calculation is completed, the reference point is cleared until the stored activation point coordinates are stored in the coordinate row buffer 22 or the composite part and buffer 16, completing the read-write control work of the read-write control module 2. In the above manner, the activation point coordinates to be stored can be stored in the coordinate row buffer 22 or the composite part and buffer 16 without omission or duplication.

[0096] Depend on Figure 1It can be seen that the device also includes: an activation point coordinate set buffer 6, which is in communication with the activation point coordinate generation module 3 and the calculation module 5; and is configured to: store the activation point coordinate set; the activation point coordinate set buffer 6 is used to record the coordinates of all activation points of the output feature map, and its function is to store the coordinates of all activation points of the output feature map in an orderly and compact manner, and to make subsequent calculations without searching for the coordinates of the activation points of the output feature map; an output feature buffer 7, which is in communication with the partial summation module 4 and the calculation module 5; and is configured to: store the output feature. The output feature buffer 7 uses the coordinates of the point to which the feature belongs as the address to record the features of the output feature map. Its function is to reconcile the contradiction between the fact that the result generated by the partial summation module 4 does not retain the order of the output coordinates and the fact that subsequent calculations need to rely on ordered coordinates.

[0097] The present application proposes a weight-oriented sparse convolution accelerator hardware device, and the workflow is as follows: the weight-oriented partial sum generation module 1 reads the features of a fixed number of activation points on the input feature map at one time, and performs matrix multiplication with the convolution kernel weight to obtain the partial sum corresponding to the input point. The parts corresponding to these input points are spliced ​​with their coordinates and directly input into the second FIFO buffer 15, and the activation point coordinates are stored separately in the first FIFO buffer 21, and then the device obtains the next calculation data. When there is data in the FIFO buffer, the read-write control module 2 takes out the data in the FIFO buffer and fills them into the coordinate row buffer 22 and the composite partial sum buffer 16 respectively. When the data in the coordinate row buffer 22 and the composite partial sum buffer 16 are prepared, the partial sum summation module 4 and the output and activation point coordinate generation module 3 respectively obtain data from the coordinate row buffer 22 and the composite partial sum buffer 16, calculate the features of the output feature map and the coordinate set of the activation points of the output feature map, and store them in the output feature buffer 7 and the activation point coordinate set buffer 6 respectively, and the partial output channel of the convolution of an input feature map is calculated. The above process is repeated until all channels of the output feature map are calculated. By duplicating and stacking the hardware, the number of output channels that are calculated simultaneously will be increased, and the time spent on calculating all channels of the output feature map will be reduced, thereby increasing the calculation speed.

[0098] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A weight-oriented sparse convolution accelerator hardware device, applied to a convolution accelerator, wherein the convolution accelerator is provided with a convolution kernel, and the convolution kernel is used to generate a weight matrix; characterized in that: include: A partial sum generation module (1) connected in communication, a read-write control module (2), an activation point coordinate generation module (3), a partial sum summation module (4), and a calculation module (5); The part and generation module (1) are configured as follows: Obtain features of a preset number of activation points on the feature map and generate a feature matrix; Multiplying the feature matrix and the weight matrix to obtain a partial sum corresponding to the activation point; The read-write control module (2) is configured as follows: Acquire the activation point coordinates on the feature map and send the activation point coordinates to the activation point coordinate generation module (3), and send the partial sum to the partial sum summation module (4); The activation point coordinate generation module (3) is configured as follows: Generate an activation point coordinate set according to the ordinate of the activation point coordinate; the activation point coordinates in the activation point coordinate set are ordered and non-repetitive; The partial summation module (4) is configured as follows: According to the summation rule, the partial sums are summed to obtain output features; The calculation module (5) is configured as follows: Based on the activation point coordinate set and the output features, target features of the feature map are obtained.

2. The weight-oriented sparse convolution accelerator hardware device according to claim 1, characterized in that: The partial summation module (4) is further configured as follows: Calculating the coordinate difference between the activation point and the potential neighbor point on the feature map, and using a lookup table to obtain the target potential neighbor point and the first binary bitmap corresponding to the target potential neighbor point; The potential neighbor points are the two activation points on the left, two on the right, and two rows below the activation point; Multiplying the first binary bitmap by the partial sum corresponding to the activation point to obtain the partial sum of the target potential neighbor point; Rotate the first binary bitmap at a preset angle around the center to obtain a second binary bitmap; Fill the partial sum of the target potential neighbor points into the non-zero positions in the second binary bitmap in order to obtain the partial sum of the target potential neighbor points; The partial sum of the target potential neighbor point and the partial sum corresponding to the activation point are added to obtain the output feature.

3. The weight-oriented sparse convolution accelerator hardware device according to claim 1, characterized in that: The device also includes: A disassembling unit (11), the disassembling unit (11) being in communication with the part and the generating module (1), and being configured to: Get feature map; The feature map is decomposed into activation point features and activation point coordinates.

4. The weight-oriented sparse convolution accelerator hardware device according to claim 3, characterized in that: The device also includes: An activation point coordinate buffer (12), the activation point coordinate buffer (12) being in communication with the disassembly unit (11) and configured to: Storing the activation point coordinates; A copying unit (13), the copying unit (13) is connected in communication with the activation point coordinate buffer (12), and is configured to: Get the coordinates of the activation point and copy it.

5. The weight-oriented sparse convolution accelerator hardware device according to claim 4, characterized in that: The device also includes: A splicing unit (14), the splicing unit (14) is connected in communication with the part and generation module (1) and the replication unit (13), and is configured to: Receive the activation point coordinates sent by the replication unit (13), and concatenate the activation point coordinates and the partial sum to obtain a composite partial sum; the composite partial sum includes: the partial sum and the activation point coordinates corresponding to the partial sum.

6. The weight-oriented sparse convolution accelerator hardware device according to claim 5, characterized in that: The device also includes: A first FIFO buffer (21), the first FIFO buffer (21) is in communication with the replication unit (13) and the read / write control module (2), and is configured as follows: receiving and storing the activation point coordinates sent by the replication unit (13); A coordinate line buffer (22), the coordinate line buffer (22) being communicatively connected with the read-write control module (2) and the activation point coordinate generation module (3); Receiving and storing the activation point coordinates sent by the read-write control module (2); The read-write control module (2) is also configured as follows: After the activation point coordinate generation module (3) generates the activation point coordinate set, the activation point coordinates stored in the coordinate row buffer (22) are cleared.

7. The weight-oriented sparse convolution accelerator hardware device according to claim 6, characterized in that: The read-write control module (2) is further configured as follows: Obtaining the first ordinate of the activation point coordinates stored in the coordinate line buffer (22); If the absolute value of the difference between the second ordinate of the activation point coordinate sent to the coordinate line buffer (22) and the first ordinate is greater than a preset value, the activation point coordinate corresponding to the first ordinate is stored in the activation point coordinate set and then resent.

8. The weight-oriented sparse convolution accelerator hardware device according to claim 6, characterized in that: The device also includes: A second FIFO buffer (15), the second FIFO buffer (15) is communicatively connected with the splicing unit (14) and the read / write control module (2), and is configured as follows: receiving and storing the composite part sent by the splicing unit (14); A composite part and buffer (16), the composite part and buffer (16) being communicatively connected with the read-write control module (2) and the part and summing module (4); receiving and storing the composite part sent by the read-write control module (2); The read-write control module (2) is also configured as follows: After the partial sum summing module (4) has finished summing the composite partial sum, the composite partial sum stored in the composite partial sum buffer (16) is cleared.

9. The weight-oriented sparse convolution accelerator hardware device according to claim 8, characterized in that: The read-write control module (2) is further configured as follows: When sending the activation point coordinates to the coordinate line buffer (22), or sending the composite part sum to the composite part sum buffer (16), obtaining the ordinate of the activation point coordinates or the activation point coordinates in the composite part sum to be stored; Obtaining whether the coordinate line buffer (22) or the composite part and buffer (16) stores the activation point coordinates, and if so, obtaining the vertical coordinates of the stored activation point coordinates; Acquire the difference between the stored ordinate of the activation point coordinate and the ordinate of the activation point coordinate to be stored; If the difference is smaller than a preset value, it is stored in the coordinate line buffer (22) or the composite part and buffer (16); If the difference is greater than a preset value, the difference is acquired again after the activation point coordinates or the composite part sum are cleared, until the difference is less than the preset value, and the activation point coordinates to be stored are stored in the coordinate row buffer (22), or the composite part sum to be stored is stored in the composite part sum buffer (16).

10. The weight-oriented sparse convolution accelerator hardware device according to claim 1, characterized in that: The device also includes: An activation point coordinate set buffer (6), the activation point coordinate set buffer (6) being in communication connection with the activation point coordinate generation module (3) and the calculation module (5); and being configured to: storing the activation point coordinate set; An output feature buffer (7), the output feature buffer (7) is in communication with the partial summation module (4) and the calculation module (5); and is configured to: The output features are stored.

Images