An accelerator for end-side real-time training

Abstract

The present application discloses an accelerator for end-side real-time training. The accelerator includes a data module, an address decoding module, a computing module, and a control module. The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block. The feature storage block includes a plurality of storage units, and the storage units are used to store each input block in the input group corresponding to each stage. The input block is a data matrix to be multiplied by a corresponding weight value. The weight value sequentially detects the values of all dimensions at the first position in the order from front to back. The weight non-zero value storage block and the weight bitmap storage block are used to store the sparse weight data. The control module is used to control the execution process of each stage. The accelerator provided by the present application can improve the problem of load imbalance and improve the network detection speed.

Description

Technical Field

[0001] The present invention relates to the field of meta-learning, and particularly to an accelerator for end-side real-time training. Background Art

[0002] In an intelligent military system, in order to adapt to a dynamic battlefield, it is often required that combat machines learn real-time data. Usually, deep network training is used to learn data. However, in a dynamic environment, new training samples are continuously generated. However, due to the lightweight requirements of combat machines, the huge amount of data and computing volume of deep network training hinder its efficient deployment on resource-constrained end-sides. And too direct and inefficient deployment has high latency and power consumption, which will seriously affect system performance. Few-shot learning provides a method for effective training on the end-side. Based on the pre-training of deep neural networks, by selecting a suitable few-shot learning algorithm, low-power, low-latency and continuous learning end-side deployment can be achieved. Therefore, it is particularly necessary to design an accelerator for end-side implementation training.

[0003] There are a large number of redundant computations in the existing deep neural network training process. In order to accelerate the computation of neural networks, many existing accelerators have explored methods to skip redundant computations. For example, the article "EIE: Efficient inference engine on compressed deep neural network" in ACM SIGARCH Computer Architecture News designed an accelerator that supports bilateral sparsity of weights and activations. However, since the non-zero values of weights and activations are unstructured, this method faces a huge load imbalance problem, so that the reduction of network computation cannot directly translate into an increase in throughput or a reduction in latency. To address this problem, the article "Cambricon-S: Addressing irregularity in sparse neural networks through a cooperative software / hardware approach" published in the 51st IEEE / ACM International Symposium on Microarchitecture in 2018 designed a hardware-software cooperative accelerator that supports structured sparsity. An algorithmic coarse-grained sparsity method was designed to ensure that the values in a predefined-size box are non-zero. However, on the one hand, this method faces the problem of network accuracy degradation. On the other hand, an accelerator generally only supports one type of predefined box shape, which makes the accelerator less versatile. Summary of the Invention

[0004] The present application proposes an accelerator for end-side real-time training. The accelerator is an unstructured sparse accelerator with a queuing mechanism, which can improve the load imbalance problem existing in existing unstructured accelerators, reduce a large amount of redundant calculations, and avoid the problem of accuracy degradation caused by structured sparsity.

[0005] The present application discloses an accelerator for end-side real-time training. The accelerator includes a data module, an address decoding module, a computing module, and a control module;

[0006] The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block. The feature storage block includes a plurality of storage units;

[0007] The storage unit is used to store each input block in the input group corresponding to each stage. The input group is any input channel set obtained by grouping each input channel, and the input block is a data matrix to be multiplied by the corresponding weight value. The weight value is the weight value to be convolved with the corresponding input block in the corresponding input group;

[0008] The weight bitmap storage block is used to store the weight bitmaps of each stage. Among them, the weight bitmap in the FF stage is obtained by zeroing out and sparsifying the weight matrix, the weight bitmap in the BP stage is obtained by zeroing out and sparsifying the transposed weight matrix, the transposed weight matrix is the transpose of the weight matrix, and the weight bitmap in the WG stage is obtained by lazy sparsifying the weight matrix;

[0009] The weight non-zero value storage block is used to store the unsparsified non-zero weight values of each stage;

[0010] The control module is configured to perform the following steps:

[0011] For any current stage in the FF stage and the BP stage, control the address decoding module to obtain the non-zero weight value positions according to the weight bitmap of the current stage, and obtain the first target input block address according to the non-zero weight value positions;

[0012] Control the computing module to perform the calculation corresponding to the current stage according to the non-zero weight value corresponding to the non-zero weight value position and the first target input block corresponding to the first target input block address, and obtain the output result;

[0013] For the WG stage, control the address decoding module to obtain the error non-zero values, the positions of the error non-zero values, and the weight update positions according to the input error value and the weight bitmap of the WG stage;

[0014] Control the address decoding module to obtain the second target input block address corresponding to the position of the error non-zero value;

[0015] The control calculation module performs the calculations corresponding to the WG stage based on the non-zero error value and the second target input block corresponding to the second target input block address;

[0016] Update each weight value corresponding to the output result of the WG stage according to the weight update position.

[0017] In an implementable manner, the weight bitmaps of the FF stage or the BP stage are obtained through the following method:

[0018] For the weight matrix of any convolutional layer, arrange each weight value in the weight matrix in ascending order of absolute value;

[0019] Determine the weight sparsity threshold according to the preset weight sparsity rate and the total number of weight values in the weight matrix;

[0020] Set the weight values in the weight matrix whose absolute values are less than the weight sparsity threshold to zero, and set the weight values whose absolute values are greater than the weight sparsity threshold to one, to obtain the weight bitmap corresponding to the weight matrix of the current convolutional layer;

[0021] The weight bitmaps corresponding to the weight matrices of all convolutional layers together constitute the weight bitmap of the FF stage or the BP stage.

[0022] In an implementable manner, the weight bitmap of the WG stage is obtained through the following method:

[0023] For the weight matrix of any convolutional layer, arrange each weight value in the weight matrix in ascending order of absolute value;

[0024] Determine the lazy weight sparsity threshold according to the preset lazy weight sparsity rate and the total number of weight values in the weight matrix;

[0025] Set the weight values in the weight matrix whose absolute values are less than the lazy weight sparsity threshold to zero, and set the other weight values to one, to obtain the weight bitmap corresponding to the weight matrix of the network layer;

[0026] The weight bitmaps corresponding to the weight matrices of all convolutional layers together constitute the weight bitmap of the WG stage.

[0027] In an implementable manner, according to the weight bitmap of the current stage, obtain the positions of non-zero weight values, including:

[0028] For any first position in the weight bitmap of the current stage, sequentially detect the values of all dimensions of the first position in the order from front to back, and obtain the positions of non-zero values of all dimensions of the first position;

[0029] Detect the values of all dimensions at other positions in the row where the first position is located in sequence from left to right, and obtain the positions of non-zero values of all dimensions at other positions in the row where the first position is located;

[0030] Detect the values of all dimensions at other positions in the column where the first position is located in sequence from top to bottom, and obtain the positions of non-zero values of all dimensions at other positions in the column where the first position is located;

[0031] Each non-zero value position together constitutes a non-zero weight value position.

[0032] In an implementable manner, the computing module includes a forward and backward computing core, a WG computing core, and a BN computing core, a pooling computing core, and a non-linear function computing core connected in sequence. The forward and backward computing core and the WG computing core are respectively connected to the BN computing core;

[0033] The forward and backward computing core includes a FIFO, an adder tree, and multiple first PEs. The FIFO is used to input the non-zero weight values corresponding to the non-zero weight value positions into the corresponding first PEs after the data volume reaches a preset value, and, according to the first target input block address, obtain the first target input block corresponding to the first target input block address from the feature storage block and input the first target input block into the corresponding first PE to multiply with the non-zero weight value; The adder tree is used to accumulate the calculation results in each first PE;

[0034] The WG computing core includes multiple second PEs, and each second PE is used to calculate the error non-zero values in the WG stage and the second target input block.

[0035] In an implementable manner, the first PE includes a multiply-accumulate unit and a first register. The multiply-accumulate unit is used to multiply and accumulate the non-zero weight value and the corresponding first target input block, and the first register is used to temporarily store the multiplication result and the accumulation result of the multiply-accumulate unit;

[0036] The second PE includes a multiply-accumulate unit and a second register. The multiply-accumulate unit is used to multiply and accumulate the error non-zero value and the corresponding second target input block to obtain a unit weight gradient, and the second register is used to temporarily store the unit weight gradient. The weight update position is used to indicate the unit weight gradient that needs to be updated in the second register.

[0037] In an implementable manner, the data module further includes an output storage block, a weight gradient storage block, and a bus matrix module. The feature storage block, the weight non-zero value storage block, the output storage block, the weight bitmap storage block, and the weight gradient storage block are all connected to the bus matrix module.

[0038] In one implementable manner, the address decoding module includes a non-zero value detection module and an address decoder. The non-zero value detection module is connected to the weight bitmap storage block, and the address decoder is connected to the weight bitmap storage block, the weight non-zero value storage block, the non-zero value detection module, and the bus matrix module respectively;

[0039] The non-zero value detection module is configured to obtain the positions of non-zero weight values according to the weight bitmap of the current stage, and obtain the error non-zero values, the positions of the error non-zero values, and the weight update positions according to the input error value and the weight bitmap of the WG stage;

[0040] The address decoder is configured to obtain a first target input block address corresponding to the position of the non-zero weight value, and obtain a second target input block address corresponding to the position of the error non-zero value.

[0041] In one implementable manner, the control module includes a total control unit and a configuration register.

[0042] The present application discloses an accelerator for end-side real-time training. The accelerator includes a data module, an address decoding module, a computing module, and a control module. The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block. The feature storage block includes a plurality of storage units, and the storage units are configured to store each input block in the input group corresponding to each stage. The input block is a data matrix to be multiplied by the corresponding weight value, and the weight value sequentially detects the values of all dimensions at the first position from front to back. The weight non-zero value storage block and the weight bitmap storage block are configured to store the sparse weight data. The control module is configured to control the execution process of each stage. The accelerator is an unstructured sparse accelerator with a queuing mechanism, which can improve the load imbalance problem existing in the existing unstructured accelerators, reduce a large amount of redundant calculations at the same time, and avoid the problem of accuracy degradation caused by structured sparsity. Description of the Drawings

[0043] Figure 1 It is a schematic top-level architecture diagram of the accelerator for end-side real-time training according to an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of the weight pointing calculation method according to an embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of sparsity according to an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the reading order of the positions of non-zero weight values according to an embodiment of the present application;

[0047] Figure 5 It is a schematic diagram of the calculation process of the FF stage according to an embodiment of the present application;

[0048] Figure 6 Schematic diagram of data reading and calculation process for embodiments of this application;

[0049] Figure 7 Schematic diagram of the calculation process in the WG stage for embodiments of this application. Detailed implementation manners

[0050] For the convenience of explaining the technical solution of the application, some concepts related to this application will be first described below.

[0051] The training of a convolutional neural network generally includes calculations in three stages: FF (feed-forward, forward propagation stage), BP (backward propagation, backward propagation stage), and WG (weight gradient generation, weight update stage). In the FF process, forward propagation is performed on a batch of data to obtain their losses. In the BP process, the error of each intermediate feature map is obtained by backpropagating the loss. In the WG process, the gradient value and update value of the weights are obtained using the error of the intermediate features, and the weights are updated.

[0052] The specific calculations involved in the above three stages are as follows:

[0053] In the FF stage, the intermediate feature map a of the previous convolutional layer l-1 is convolved with the weights W of the current convolutional layer l , and after adding the bias b l , the intermediate feature map of the current convolutional layer is generated through the non-linear function σ. Using z l to represent the convolution result, * represents the convolution operation, and the calculation of FF can be expressed as follows:

[0054] a l = σ(z l ) = σ(W l * a l-1 + b l ) Formula (1)

[0055] The operation of BP is similar to that of FF. The error δ l+1 of the next convolutional layer is convolved with the transposed weights W of this layer l+1 to obtain the error δ l of the current convolutional layer. σ'(z l ) represents the derivative of the non-linear function, and this calculation is expressed by the formula as follows:

[0056] δ l = rot180(W l+1 ) * δ l+1 · σ'(z l ) Formula (2)

[0057] The WG takes the current convolutional layer error δ obtained by BP l and convolves it with the intermediate feature map a of the previous convolutional layer l-1 to obtain the gradient of the current convolutional layer weight W l and updates the weight using the gradient. Let α represent the learning rate, and WG is expressed by the formula as follows:

[0058]

[0059] Based on the above principle, an accelerator hardware implementation is provided. This application discloses an accelerator for end-side real-time training, as Figure 1 shown in the schematic diagram of the top-level architecture of the end-side real-time training accelerator in the embodiments of this application. The accelerator includes a data module, an address decoding module, a computing module, and a control module.

[0060] The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block.

[0061] The weight bitmap storage block is used to store the weight bitmaps of each stage. Among them, the weight bitmap in the FF stage is obtained by zeroing out and sparsifying the weight matrix, the weight bitmap in the BP stage is obtained by zeroing out and sparsifying the weight transposed matrix, the weight transposed matrix is the transpose of the weight matrix, and the weight bitmap in the WG stage is obtained by lazily sparsifying the weight matrix.

[0062] Specifically, the embodiments of this application adopt two types of sparse strategies. One is the input sparsity commonly used in existing work, that is, setting some input data in the convolutional operation to zero and skipping their calculations. In the embodiments of this application, it is the sparsity of the weight matrix or the weight transposed matrix in the FF stage and the BP stage. The other is the special sparsity proposed in the embodiments of this application, which is called lazy sparsity and is used in the WG stage. It can be regarded as a kind of output sparsity, that is, it is considered that some weights are "lazy" and their values do not need to be updated, so there is no need to calculate the gradients of these weights in the WG stage.

[0063] Furthermore, in the embodiments of this application, the weight bitmap in the FF stage or the BP stage is obtained by the following method:

[0064] For the weight matrix of any convolutional layer, arrange the weight values in the weight matrix in ascending order of absolute value;

[0065] According to the preset weight sparsity rate and the total number of weight values in the weight matrix, determine the weight sparsity threshold;

[0066] Set the weight values in the weight matrix whose absolute values are less than the weight sparsity threshold to zero, and set the weight values whose absolute values are greater than the weight sparsity threshold to one, to obtain the weight bitmap corresponding to the weight matrix of the current convolutional layer;

[0067] The weight bitmaps corresponding to the weight matrices of all network layers together constitute the weight bitmap of the FF stage or the BP stage.

[0068] Furthermore, as shown in Figure 3 , the sparse schematic diagram of the embodiment of the present application, the above method is specifically as follows. First, sort the data in the weight matrix W l of the current convolutional layer in ascending order of absolute value, and use the preset sparsity rate to obtain the threshold w th,sparse , and then use formulas (4) and (5) to obtain the sparsity and multiply it by the weight matrix. After such calculation, the new weight matrix is the result obtained by setting the small values in the original weight matrix W l to zero, that is, it has sparsity.

[0069]

[0070]

[0071] Among them, w th,sparse is the weight sparsity threshold, is the sparsity, is the new weight matrix after sparsity.

[0072] Set the non-zero values in the obtained new weight matrix to one, that is, obtain the weight bitmap corresponding to the weight matrix of the current convolutional layer.

[0073] The weight bitmap of the WG stage is obtained through the following method:

[0074] For the weight matrix of any convolutional layer, arrange the weight values in the weight matrix in ascending order of absolute value;

[0075] According to the preset lazy weight sparsity rate and the total number of weight values in the weight matrix, determine the lazy weight sparsity threshold;

[0076] Set the weight values in the weight matrix whose absolute values are less than the lazy weight sparsity threshold to zero, and set the other weight values to one, to obtain the weight bitmap corresponding to the weight matrix of the convolutional layer;

[0077] The weight bitmaps corresponding to the weight matrices of all convolutional layers together constitute the weight bitmap of the WG stage.

[0078] See Figure 3, as shown in the sparse schematic diagram of the embodiments of the present application, specifically, lazy sparsity is applied to the weight W l Introduce the lazy sparsity rate Among them, Utilize Calculate the lazy threshold w th,lazy , arrange the weight values in the weight matrix in ascending order of absolute value, and regard the weights between the absolute value of w th,sparse and w th,lazy as lazy weights. The values of the lazy weights are retained and participate in the calculations of FF and BP to ensure accuracy; however, they are not updated or updated at a very low frequency during the WU process. Optionally, they are updated once every 20 rounds, so as to skip the gradient calculations of these points during the WG process and save the computational consumption of WG.

[0079] The effect of lazy sparsity can be expressed by formulas (6), (7), and (8).

[0080]

[0081] ▽W l =a l-1 *δ l Formula (7)

[0082]

[0083] Among them, w thlazy is the weight lazy threshold, is the lazy sparsity, represents the weight gradient of the current convolutional layer, represents the sparse weight gradient of the current stage. It should be noted that since the part less than the weight sparsity threshold w th,sparse has been set to zero, there is no need to consider the lazy sparsity of this part.

[0084] In addition, the weight non-zero value storage block is used to store the non-zero weight values that are not sparsified in each stage.

[0085] The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block. The feature storage block includes multiple storage units.

[0086] Specifically, the storage unit is used to store each input block in the input group corresponding to each stage. The input group is any set of input channels obtained by grouping each input channel, and the input block is a data matrix to be multiplied by the corresponding weight value. The weight value is the weight value to be convolved with the corresponding input block in the corresponding input group.

[0087] It should be noted that each convolutional layer slides the convolutional kernel on the input data or intermediate feature map to extract the features of the input information in a way of multiplying and accumulating, that is, each element of the convolutional kernel is multiplied by the corresponding elements on each input channel dimension during the sliding process, and the results of each multiplication are accumulated. In the embodiments of the present application, a weight pointing method is adopted for calculation, as Figure 2 As shown in the schematic diagram of the weight pointing calculation method in the embodiments of the present application, each non-zero weight value in each convolutional kernel is multiplied by all the elements it slides over on each input channel dimension. Among them, all the elements that each weight value slides over on a one-dimensional channel form an input block.

[0088] In the embodiments of the present application, the control module is configured to perform the following steps:

[0089] Step 1, for any current stage in the FF stage and the BP stage, control the address decoding module to obtain the non-zero weight value positions according to the weight bit map of the current stage, and obtain the first target input block address according to the non-zero weight value positions.

[0090] In Step 1, the weight bit map of the current stage represents the sparsified weight bit map corresponding to the weight matrix in each convolutional layer in the FF stage. Correspondingly, the weight bit map of the current stage represents the sparsified weight bit map corresponding to the transposed weight matrix in each convolutional layer in the BP stage. Optionally, the weight bit map of the current stage represents the bit map obtained by transposing the weight bit map in the FF stage in the BP stage.

[0091] Furthermore, the non-zero weight value positions mentioned in Step 1 are obtained by the non-zero value monitoring module in the address decoding module. The address decoding module includes a non-zero value detection module and an address decoder. The non-zero value detection module is used to obtain the non-zero weight value positions according to the weight bit map of the current stage. See Figure 4 As shown in the schematic diagram of the non-zero weight value position reading order in the embodiments of the present application, the non-zero weight value positions in Step 1 are obtained through the following order:

[0092] S101, for any first position in the weight bit map of the current stage, sequentially detect the values of all dimensions of the first position in the order from front to back, and obtain the non-zero value positions of all dimensions of the first position.

[0093] S102, sequentially detect the values of all dimensions of other positions in the row where the first position is located in the order from left to right, and obtain the non-zero value positions of all dimensions of other positions in the row where the first position is located.

[0094] S103. Sequentially detect the values of all dimensions at other positions in the column where the first position is located in the order from top to bottom, and obtain the positions of non-zero values of all dimensions at other positions in the column where the first position is located.

[0095] All the positions of non-zero values together constitute the positions of non-zero weight values.

[0096] In addition, each position of non-zero weight value corresponds to a first target input block address.

[0097] Step 2. Control the calculation module to perform the calculation corresponding to the current stage according to the non-zero weight values corresponding to the positions of non-zero weight values and the first target input blocks corresponding to the first target input block addresses, and obtain the output result.

[0098] In Step 2, the calculation module includes a forward and backward calculation core, a WG calculation core, and a BN calculation core, a pooling calculation core, and a non-linear function calculation core connected in sequence. The forward and backward calculation core and the WG calculation core are respectively connected to the BN calculation core. Refer to Figure 1 the schematic diagram of the accelerator top-level architecture for end-side real-time training in the application embodiment.

[0099] Further, the forward and backward calculation core includes a FIFO (First In First Out buffer), an adder tree, and multiple first PEs (processing units). The FIFO is used to input the non-zero weight values corresponding to the positions of non-zero weight values into the corresponding first PEs after the data volume reaches a preset value, and, according to the first target input block address, obtain the first target input block corresponding to the first target input block address from the feature storage block, and input the first target input block into the corresponding first PE to multiply with the non-zero weight values. The adder tree is used to accumulate the calculation results in each first PE.

[0100] Specifically, in combination with Step 1, assuming there are C input channels of dimensions, according to the hardware configuration, there are N first PEs, then the C-dimensional input channels are divided into N groups, C = {C1, C2, …, C N}, each group C n contains 1 or more channels, and each group of channels corresponds to a first PE. Optionally, the C-point grouping method can be customized according to the model characteristics to make the loads of each group relatively balanced. Refer to Figure 5 the schematic diagram of the FF stage calculation process in the application embodiment and Figure 6 the schematic diagram of the data reading and calculation process in the application embodiment as shown, Figure 5In the formula, W1, W2......W9 represent the input non - zero weight values, A1, A2......A9 represent the first target input block addresses corresponding to the non - zero weight values, and BANK1, BANK2……BANK4 represent the storage units in the feature storage block. It should be noted that the number of non - zero weight values, the first target input blocks, and the storage units is not limited by the illustration in the figure; Figure 6 The two pieces of code in it respectively represent the data reading process in the FF stage and the PE parallel computing process in the FF stage. Among them, C, S, R, K, H, W, T H 、T W respectively represent the input channel dimension, the horizontal dimension of the convolution kernel, the vertical dimension of the convolution kernel, the output channel dimension, the vertical dimension of the input data, the horizontal dimension of the input data, the vertical dimension of the input block, and the horizontal dimension of the input block. According to the non - zero weight value position reading order shown in the data reading and computing process schematic diagram of the embodiment of the present application, the non - zero weight values and the first target input block addresses are read, and they are paired and stored in the FIFO to form input pairs. When the data in the FIFO reaches a certain number, the FIFO changes from closed to open, and the input pairs stored first in the FIFO are preferentially output for operation. The non - zero weight value in the input pair first enters the corresponding first PE, and the first target input block address in the input pair reads the first target input block from the corresponding storage unit. The first target input block address enters the corresponding first PE to perform an operation with the non - zero weight value that has entered in advance. Since the non - zero weight value positions are read according to the input dimension, the input pairs are sequentially input into each first PE according to the grouping of the foregoing input channels. After the last first PE is input, then start inputting from the first first PE. According to the non - zero weight value position reading order, each convolution kernel has S×R rounds of input to all first PEs. It should be noted that the first target input block address first obtains the input block from the storage unit of the feature storage block, and then enters the first PE to perform a calculation with the corresponding non - zero weight value. It should be noted that the access strategy of the FIFO in the embodiment of the present application according to channel grouping, and the opening and closing design of the FIFO cooperate to form an accelerator with a queuing mechanism. This implementation method can solve the data reading conflict and load imbalance problems easily brought by sparse convolution parallel computing, and improve the utilization rate of hardware resources.

[0101] Furthermore, the first PE includes a multiply - accumulate unit and a first register. The multiply - accumulate unit is used to multiply and accumulate the non - zero weight value and the corresponding first target input block, and the first register is used to temporarily store the multiplication result and the accumulation result of the multiply - accumulate unit.

[0102] Specifically, the multiplication calculation of the first set of non-zero weight values and the first target input block will be temporarily stored in the first register. The calculation result of the second set of non-zero weight values and the first target input block will be added to the calculation result of the first set to form new temporarily stored data, and so on until all data corresponding to the current convolution kernel and this first PE are calculated. The calculation results of all first PEs will be added in the adder tree to obtain the calculation result of the forward and backward calculation kernels of the current convolution kernel. This result will be sequentially input into the BN calculation kernel and the pooling calculation kernel for relevant calculations. It should be noted that when the control unit determines that all convolution layer calculations in the FF stage are completed, the calculation result at this time will be passed into the non-linear calculation kernel for normalization processing of the forward loss, and finally the output result of the FF stage will be stored in the output storage block.

[0103] In addition, the output result of the FF stage is the loss obtained in the FF stage. This result will be used as the input for backpropagation in the BP stage to calculate the error of each convolution layer. It should be noted that the convolution kernel in the BP stage is the weight transposed matrix. The weight transposed matrix of each convolution layer convolves the error of this stage to obtain the error of the previous stage. The calculations in the BP stage and the FF stage are similar, and the two stages share a calculation kernel, so the calculations in the BP stage will not be elaborated here.

[0104] Step 3: For the WG stage, control the address decoding module to obtain the error non-zero values, the positions of the error non-zero values, and the weight update positions according to the input error value and the weight bitmap of the WG stage.

[0105] Step 4: Control the address decoding module to obtain the second target input block address corresponding to the position of the error non-zero value.

[0106] In Steps 3 and 4, refer to Figure 7 As shown in the schematic diagram of the calculation process of the WG stage in the embodiment of the present application, the non-zero value detection module of the address decoding module obtains the error non-zero values, the positions of the error non-zero values, and the weight update positions by detecting the input error and the weight bitmap. The position of each error non-zero value corresponds to the address of the input block on the intermediate eigenvalue multiplied by it, that is, the address of the input block on the input channel group. The address decoder decodes the input block address to obtain the corresponding input block for corresponding calculations.

[0107] Step 5: Control the calculation module to perform the calculations corresponding to the WG stage according to the error non-zero values and the second target input block corresponding to the second target input block address.

[0108] In step five, relevant operations are performed in the WG stage in the WG calculation core. The WG calculation core includes multiple second PEs, and each second PE is used to calculate the non-zero error values in the WG stage and the second target input block. In the WG stage, the error of each convolutional layer in the BP stage is used to perform convolution on the intermediate feature values. The intermediate feature values are grouped in a similar way to the FF stage as input channels, and each group corresponds to a second PE, and corresponding calculations are performed in the second PE.

[0109] Furthermore, the second PE includes a multiply-accumulate unit and a second register. The multiply-accumulate unit is used to multiply and accumulate the non-zero error value and the corresponding second target input block to obtain the unit weight gradient. The second register is used to temporarily store the unit weight gradient, and the weight update position is used to indicate the unit weight gradient that needs to be updated in the second register.

[0110] Specifically, the data obtained by multiplying the first group of non-zero error values with the input block is stored in the second register. The data obtained by multiplying the second group of non-zero error values with the input block is added to the existing data in the second register, and the resulting result is still stored in the second register. And so on, until all the data corresponding to the current convolution kernel and the second PE are calculated to obtain the unit weight gradient. It should be noted that according to the sparse method in the embodiments of the present application, the non-zero error value and the corresponding second target input block only participate in the operation of the non-zero part of the corresponding unit weight gradient, which greatly improves the operation efficiency.

[0111] Step six, update each weight value corresponding to the output result of the WG stage according to the weight update position.

[0112] In step six, it is judged which of all the second registers need to be updated according to the non-zero value positions on the weight bitmap, and they are updated. It should be noted that according to the weight sparse method described above, the non-zero values on the weight bitmap correspond to the non-zero values in the unit weight gradient obtained by the operation of the non-zero error value and the second target input block in step five, and each non-zero value on the weight bitmap corresponds to the position of a unit weight gradient non-zero value.

[0113] Specifically, referring to Figure 1 , as shown in the schematic diagram of the accelerator top-level architecture for end-side real-time training in the embodiments of the present application, the data module further includes an output storage block, a weight gradient storage block, and a bus matrix module. The feature storage block, the weight non-zero value storage block, the output storage block, the weight bitmap storage block, and the weight gradient storage block are all connected to the bus matrix module.

[0114] Furthermore, the output storage block is used to store the output data of each stage. The weight gradient storage block is used to store the unit weight gradients in each second register during the WG stage, and transfer all the unit weight gradients of the current convolutional layer outside the chip for calculation to obtain the weight update values for updating the weights. The bus matrix module is used for data transmission and scheduling among the data module, the calculation module, and the control module.

[0115] In addition, the control module includes a general control unit and a configuration register.

[0116] Specifically, the general control unit is used to control the hardware logic, and the configuration register stores the configuration of the input data and parameters.

[0117] Thus, in order to achieve real-time training on the edge side, the embodiment of this application adopts a typical few-shot learning method - the Meta Baseline algorithm, as the deployed algorithm. The Meta Baseline algorithm is a type of meta-learning method that can pre-train a strongly generalizable model and fine-tune it on new data to quickly adapt to new scenarios. Therefore, it is a suitable method for fast few-shot training. Also, due to its characteristic of using a relatively small model, it reduces the storage and calculation required for training. Therefore, it is also very suitable for deployment on edge-side hardware. It is worth noting that the non-linear calculation module in the embodiment of this application can support the operation of the loss function in the meta-learning algorithm.

[0118] It is worth noting that since the methods proposed in the embodiment of this application are applicable to the training of general deep learning networks, the application of the embodiment of this application is not limited to the Meta Baseline algorithm, but can be used for other newer and better-performing learning methods.

[0119] In addition, the Meta Baseline algorithm uses similarity measurement to achieve classification. Therefore, its training only involves the learning of the parameters of the convolutional layer and the parameters of the BN (Batch Normalization) layer attached after the convolutional layer, and does not involve the fully connected layer in a general deep learning network. Therefore, the embodiment of this application focuses on the research of the convolutional layer.

[0120] An embodiment of the present application discloses an accelerator for end-side real-time training. The accelerator includes a data module, an address decoding module, a computing module, and a control module; the data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block; the feature storage block includes a plurality of storage units, and the storage units are used to store each input block in the input group corresponding to each stage, and the input block is a data matrix to be multiplied by the corresponding weight value, and the weight value sequentially detects the values of all dimensions at the first position from front to back; the weight non-zero value storage block and the weight bitmap storage block are used to store the sparse weight data; the control module is used to control the execution process of each stage. This accelerator is an unstructured sparse accelerator with a queuing mechanism, which can improve the load imbalance problem existing in the existing unstructured accelerators, reduce a large amount of redundant calculations at the same time, and avoid the problem of accuracy degradation caused by structured sparsity.

[0121] The present application has been described in detail above in conjunction with specific embodiments and exemplary examples, but these descriptions should not be construed as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, various equivalent replacements, modifications, or improvements can be made to the technical solutions and implementation manners of the present application, and these all fall within the scope of the present application. The protection scope of the present application is subject to the appended claims.

Claims

1. An accelerator for end-side real-time training, characterized in that, The accelerator includes a data module, an address decoding module, a computing module, and a control module; The data module includes a feature storage block, a weight non-zero value storage block, and a weight bitmap storage block. The feature storage block includes a plurality of storage units; The storage unit is used to store each input block in the input group corresponding to each stage. The input group is any set of input channels obtained by grouping each input channel, and the input block is a data matrix to be multiplied by the corresponding weight value. The weight value is the weight value to be convolved with the corresponding input block in the corresponding input group; The weight bitmap storage block is used to store the weight bitmaps of each stage. Among them, the weight bitmap in the FF stage is obtained by zeroing and sparsifying the weight matrix, and the weight bitmap in the BP stage is obtained by zeroing and sparsifying the transposed weight matrix. The transposed weight matrix is the transpose of the weight matrix, and the weight bitmap in the WG stage is obtained by lazy sparsifying the weight matrix; The weight non-zero value storage block is used to store the non-zero weight values that are not sparsified in each stage; The control module is configured to perform the following steps: For any current stage in the FF stage and the BP stage, control the address decoding module to obtain the non-zero weight value positions according to the weight bitmap of the current stage, and obtain the first target input block address according to the non-zero weight value positions; Control the computing module to perform the calculation corresponding to the current stage according to the non-zero weight value corresponding to the non-zero weight value position and the first target input block corresponding to the first target input block address, and obtain an output result; For the WG stage, control the address decoding module to obtain the error non-zero value, the position of the error non-zero value, and the weight update position according to the input error value and the weight bitmap of the WG stage; Control the address decoding module to obtain the second target input block address corresponding to the position of the error non-zero value; Control the computing module to perform the calculation corresponding to the WG stage according to the error non-zero value and the second target input block corresponding to the second target input block address; Update each weight value corresponding to the output result of the WG stage according to the weight update position.

2. The accelerator according to claim 1, wherein The weight bitmap in the FF stage or the BP stage is obtained by the following method: For the weight matrix of any convolutional layer, arrange each weight value in the weight matrix in ascending order of absolute value; Determine the weight sparsity threshold according to the preset weight sparsity rate and the total number of weight values in the weight matrix; Zero out the weight values in the weight matrix whose absolute value is less than the weight sparsity threshold, and set the weight values whose absolute value is greater than the weight sparsity threshold to one, to obtain the weight bitmap corresponding to the weight matrix of the current convolutional layer; The weight bitmaps corresponding to the weight matrices of all convolutional layers together constitute the weight bitmap in the FF stage or the BP stage.

3. The accelerator according to claim 2, wherein The weight bitmap in the WG stage is obtained by the following method: For the weight matrix of any convolutional layer, arrange each weight value in the weight matrix in ascending order of absolute value; Determine a lazy weight sparsity threshold according to a preset lazy weight sparsity rate and the total number of weight values in the weight matrix; Set the weight values in the weight matrix whose absolute values are less than the lazy weight sparsity threshold to zero, and set other weight values to one, to obtain a weight bitmap corresponding to the weight matrix of the convolutional layer; The weight bitmaps corresponding to the weight matrices of all convolutional layers together constitute the weight bitmap of the WG stage.

4. The accelerator according to claim 3, characterized in that, The obtaining of the non-zero weight value positions according to the weight bitmap of the current stage includes: For any first position in the weight bitmap of the current stage, sequentially detect the values of all dimensions of the first position in the order from front to back, and obtain the non-zero value positions of all dimensions of the first position; Sequentially detect the values of all dimensions of other positions in the row where the first position is located in the order from left to right, and obtain the non-zero value positions of all dimensions of other positions in the row where the first position is located; Sequentially detect the values of all dimensions of other positions in the column where the first position is located in the order from top to bottom, and obtain the non-zero value positions of all dimensions of other positions in the column where the first position is located; The respective non-zero value positions together constitute the non-zero weight value positions.

5. The accelerator according to claim 1, characterized in that, The calculation module includes a forward and backward calculation core, a WG calculation core, and a BN calculation core, a pooling calculation core, and a non-linear function calculation core connected in sequence. The forward and backward calculation core and the WG calculation core are respectively connected to the BN calculation core; The forward and backward calculation core includes a FIFO, an adder tree, and a plurality of first PEs. The FIFO is used to input the non-zero weight values corresponding to the non-zero weight value positions into the corresponding first PEs after the data volume reaches a preset value, and according to the first target input block address, obtain a first target input block corresponding to the first target input block address from the feature storage block, and input the first target input block into the corresponding first PE to multiply with the non-zero weight value. The adder tree is used to accumulate the calculation results in each first PE; The WG calculation core includes a plurality of second PEs, and each second PE is used to calculate the error non-zero value of the WG stage and a second target input block.

6. The accelerator according to claim 5, wherein, The first PE includes a multiply-accumulate unit and a first register. The multiply-accumulate unit is used to multiply and accumulate the non-zero weight value and the corresponding first target input block, and the first register is used to temporarily store the multiplication result and the accumulation result of the multiply-accumulate unit; The second PE includes the multiply-accumulate unit and a second register. The multiply-accumulate unit is used to multiply and accumulate the error non-zero value and the corresponding second target input block to obtain a unit weight gradient. The second register is used to temporarily store the unit weight gradient, and the weight update position is used to indicate the unit weight gradient that needs to be updated in the second register.

7. The accelerator according to claim 1, wherein The data module further includes an output storage block, a weight gradient storage block, and a bus matrix module. The feature storage block, the weight non-zero value storage block, the output storage block, the weight bitmap storage block, and the weight gradient storage block are all connected to the bus matrix module.

8. The accelerator according to claim 7, wherein The address decoding module includes a non-zero value detection module and an address decoder. The non-zero value detection module is connected to the weight bitmap storage block, and the address decoder is respectively connected to the weight bitmap storage block, the weight non-zero value storage block, the non-zero value detection module, and the bus matrix module; The non-zero value detection module is configured to obtain the positions of non-zero weight values according to the weight bitmap of the current stage, and to obtain the error non-zero values, the positions of the error non-zero values, and the weight update positions according to the input error values and the weight bitmap of the WG stage; The address decoder is configured to obtain a first target input block address corresponding to the position of the non-zero weight value, and to obtain a second target input block address corresponding to the position of the error non-zero value.

9. The accelerator according to claim 1, characterized in that, The control module includes a total control unit and a configuration register.

Images