Hardware Accelerator Architecture Supporting Dynamic Neural Network Sparse Models

Abstract

The present invention provides a hardware accelerator architecture supporting a dynamic neural network sparse model, including: activation data stored in a program system end memory sequentially passes through a CPU and a data moving module, and is input to a first core module; the first core module is used to calculate a saliency map based on its pre-stored first weight parameter and activation data and output it to the second core module; the activation data and the second weight parameter stored in the program system end memory sequentially pass through a CPU and a data moving module, and are input to a second core module; the second core module is used to calculate a model result based on a second weight parameter, activation data, and a saliency map, the first weight parameter is a weight parameter in a saliency map calculation network, the second weight parameter is a weight parameter of a sparse convolutional network combined with saliency, and the second weight parameter is input to the second core module in batches multiple times based on a preset timing. The architecture provided by the present invention reduces the time consumption of calculation in the algorithm application process and improves efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence hardware architecture, and in particular to a hardware accelerator architecture supporting a dynamic neural network sparse model. Background Art

[0002] In order to achieve sparse optimization of deep networks, a commonly used method is to prune the weights of the network. During the training process, unimportant connections are pruned according to the influence of the weights on the results, thereby reducing the computational cost of the network forward reasoning. Although a series of works have verified that the pruning method can significantly reduce the number of network parameters, its static characteristics still have some inherent limitations. On the one hand, when considering different types of image input weight pruning models for reasoning, although the richness of details of various images is different, the computational cost of the static sparse model is consistent. In extreme cases, a blank image requires the same amount of computation as a very detailed image. On the other hand, for different regions of the same image, since pruning only operates at the weight level, the amount of computation required for the key areas of the image and other non-key areas is also consistent in the static pruning model. Therefore, the static sparse model is independent of different inputs and different regions of the same input, and cannot achieve the optimal sparse effect.

[0003] Therefore, researchers have recently paid more and more attention to dynamic sparse models. Compared with static sparse models, dynamic sparse networks can better utilize the feature differences of different inputs, realize feature attention concentration, and allocate more computing resources to pixel areas that have a greater impact on model accuracy. According to the different implementation granularities, dynamic sparse networks can be divided into hierarchical sparsity, channel-level sparsity, and pixel-level sparsity. Pixel-level sparsity can simultaneously achieve dynamic sparse optimization and spatial adaptive sparsity, so the sparse performance is optimal, but its irregularity will also bring additional hardware implementation overhead. Therefore, it is necessary to consider pixel-level dynamic sparse network optimization algorithms and energy-efficient hardware operation architectures to truly achieve efficient reasoning for deep sparse networks.

[0004] Therefore, how to avoid the situation in which there is no matching hardware architecture in the existing dynamic sparse network optimization algorithm to reduce the computational time and improve the efficiency during the algorithm application process is still a problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0005] The present invention provides a hardware accelerator architecture supporting a dynamic neural network sparse model, so as to solve the problem that there is no matching hardware architecture in the existing dynamic sparse network optimization algorithm that can reduce the computational time and improve the efficiency during the algorithm application process.

[0006] The present invention provides a hardware accelerator architecture supporting a dynamic neural network sparse model, including a program system end memory, a central processing unit CPU, a data moving module, a first core module and a second core module.

[0007] The activation data stored in the program system end memory is sequentially input into the first core module through the CPU and the data moving module;

[0008] The first core module is used to calculate a saliency map based on a first weight parameter and the activation data stored in advance, and output the saliency map to the second core module;

[0009] The activation data and the second weight parameter stored in the program system end memory are sequentially input into the second core module through the CPU and the data moving module;

[0010] The second core module is used to calculate the model result based on the second weight parameter, the activation data and the saliency map, and return the model result to the program system end memory through the data moving module and the CPU in sequence;

[0011] Among them, the first weight parameter is the completed learning weight parameter in the saliency map calculation network in the dynamic neural network sparse model, and the second weight parameter is the completed learning weight parameter of the sparse convolutional network combined with saliency in the dynamic neural network sparse model. The second weight parameter is input into the second core module in batches multiple times based on a preset timing.

[0012] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the dynamic neural network sparse model is obtained by training based on sample images and corresponding reference standard labels, and the network structure in the training process of the dynamic neural network sparse model includes a saliency map calculation network to be optimized and a sparse convolutional network combined with saliency to be optimized, wherein:

[0013] The saliency map calculation network to be optimized is used to extract a saliency map from the input sample image, obtain a saliency map of the sample image and output it to the sparse convolution network combined with saliency to be optimized;

[0014] The sparse convolutional network combined with saliency to be optimized is used to perform task prediction based on the input sample image saliency map and the input sample image to obtain a prediction result;

[0015] The prediction result and the reference standard label are used to construct a loss function when training the dynamic neural network sparse model;

[0016] Among them, the task prediction corresponds to the training task target during the training of the dynamic neural network sparse model.

[0017] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the task prediction based on the input sample image saliency map and the input sample image specifically includes:

[0018] Extract features from the input sample image to obtain the original feature map of the C channel;

[0019] The input sample image saliency map is converted into a binary mask with C channels using a uniform quantization rule;

[0020] After obtaining the convolution result feature map of the current layer in any convolution layer of the sparse convolution network combined with significance to be optimized, perform a dot multiplication operation on the convolution result feature map of the current layer and the binary mask to obtain the final result feature map of the layer, and output it to the activation layer, so that the activation layer performs activation processing to obtain the activation result, and then inputs it to the next convolution layer, until the prediction result of the last convolution layer in the sparse convolution network combined with significance to be optimized is obtained, wherein the first convolution layer of the sparse convolution network combined with significance to be optimized is input with the original feature map.

[0021] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the step of converting an input sample image saliency map into a binary mask with a channel number of C according to a uniform quantization rule specifically includes:

[0022] The input sample image saliency map F is quantized into a binary mask M with C channels using the uniform quantization rule by the following formula:

[0023]

[0024]

[0025] Wherein, L is the preset quantization order, S is the quantization step size of the uniform quantization rule, C is the number of channels of the original feature map, F(i,j) is the saliency value of the pixel at position (i,j) on the sample image saliency map F, Q(i,j) is the quantization order value of the pixel at position (i,j) on the uniform quantization mask Q, and h is the width of the original feature map.

[0026] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the construction of the loss function specifically includes:

[0027] The loss function L during training of the dynamic neural network sparse model is constructed by the following formula total :

[0028] L total =Lmain +μL sparse

[0029] L sparse =|l1(F)-γ|

[0030] Among them, L main is the error between the prediction result and the reference standard label, L sparse The sparsity representation of the saliency map calculation network is shown in Figure 2. F is the sample image saliency map, l1() is the 1-norm function, and γ and μ are both preset adjustable hyperparameters.

[0031] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the first core module includes an input storage unit, a first weight storage unit, an on-chip convolution core unit and a saliency map memory, wherein:

[0032] The input storage unit is used to receive the activation data forwarded by the data moving module from the CPU and output it to the on-chip convolution core unit,

[0033] The first weight storage unit is used to store the first weight parameter written in advance and output it to the on-chip convolution core unit;

[0034] The on-chip convolution core unit is used to complete the calculation of all convolutional layers in the saliency map calculation network in a streaming structure based on the first weight parameter and the activation data, obtain a saliency map and output it to the saliency map memory;

[0035] The saliency map memory is used to output the saliency map to the second core module.

[0036] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the first core module also includes a sparse controller, wherein:

[0037] The sparse controller is used to control the on-chip convolution core unit to reduce or enlarge the saliency value of the calculated saliency map according to a preset scaling rule to obtain a scaled saliency map, and control the on-chip convolution core unit to output the scaled saliency map to the saliency map memory;

[0038] The saliency map memory is used to output the scaled saliency map to the second core module.

[0039] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the second core module includes a sparse decoder, an activation value storage unit, a second weight storage unit, a weight loader, an activation layer and a preset calculation array, wherein:

[0040] The sparse decoder is used to receive the saliency map output by the first core module and the activation data input by the data moving module, and decode the activation data based on the binary mask obtained by quantizing the saliency map to obtain a quantized saliency map and output it to the activation value storage unit for storage, so that the activation value storage unit can output it to the preset calculation array;

[0041] The weight loader is used to receive the second weight parameters in batches according to the preset timing, and output them to the second weight storage unit;

[0042] The second weight storage unit is used to output the second weight parameters output by the weight loader to the preset calculation array in batches according to the preset timing;

[0043] The preset calculation array is used to calculate the output result of the convolution layer in the sparse convolution network combined with significance corresponding to the i-th batch based on the second weight parameter of the i-th batch input in the preset time sequence and the activation value corresponding to the i-th batch output by the activation value storage unit, and output it to the activation layer, so that the activation layer can activate the output result and then output it to the activation value storage unit as the activation value of the i+1th batch for storage, until the result of the last convolution layer in the sparse convolution network combined with significance is obtained and returned through the data moving module and the CPU, and output to the data moving module, wherein the activation value of the first batch is the quantized saliency map.

[0044] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the preset computing array is a computing array with a size of 16×16.

[0045] According to a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, the data movement module performs data forwarding between the program system-side memory and the CPU based on the AXI-Stream protocol.

[0046] The hardware accelerator architecture supporting the dynamic neural network sparse model provided by the present invention deploys the operations of the saliency map calculation network of the dynamic neural network sparse model and the sparse convolution network combined with saliency in the first core module and the second core module respectively. The first core module calculates the saliency map of the input activation data based on the first weight parameter of the saliency map calculation network pre-written into its own storage. Since the saliency map calculation network itself is a lightweight network, the number of the first weight parameters is small and convenient to write and store in the first core module in advance. Therefore, only the activation data of the first layer of the network (i.e., the original image to be tested) needs to be input, and the calculations of all subsequent convolutional layers of the network are all in the first core module. The operation is performed on the chip where the computation is performed, which reduces the number of accesses to the system-side memory of the external memory program, thus reducing the computation time. As for the operation of the sparse convolutional network combined with significance on the second core module, since the sparse convolutional network combined with significance is used as the backbone functional network, in order to ensure the accuracy of the corresponding function of the model, there are many weight parameters. Therefore, in order to cooperate with the limited storage capacity on the chip where the second core module is located, the second weight parameters are input to the second core module in batches according to the preset timing for multiple times. The convolutional layers in the sparse convolutional network combined with significance on the second core module can also be iteratively calculated in sequence, that is, the time is short and the use of low storage capacity memory is guaranteed to reduce the hardware cost. Therefore, the sparse convolutional network combined with significance provided by the present invention reduces the computation time and improves the efficiency during the algorithm application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 One of the structural schematic diagrams of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention;

[0049] Figure 2 A process framework diagram for training a dynamic neural network sparse model provided by the present invention;

[0050] Figure 3 A second structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention;

[0051] Figure 4 A third structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention;

[0052] Figure 5 The fourth structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The existing dynamic sparse network optimization algorithms generally have the problem that there is no matching hardware architecture that can reduce the computational time and improve the efficiency during the algorithm application process. Figure 1-Figure 5 A hardware accelerator architecture supporting dynamic neural network sparse models of the present invention is described. Figure 1 One of the structural schematic diagrams of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention, such as Figure 1 As shown, the hardware accelerator architecture includes a program system-side memory 101, a central processing unit CPU 102, a data migration module 103, a first core module 104 and a second core module 105, and is characterized by:

[0055] The activation data stored in the program system end memory 101 is sequentially input into the first core module 104 through the CPU 102 and the data moving module 103;

[0056] The first core module 104 is used to calculate a saliency map based on the first weight parameter and the activation data stored in advance, and output the saliency map to the second core module 105;

[0057] The activation data and the second weight parameter stored in the program system end memory 101 are sequentially input into the second core module 105 through the CPU 102 and the data moving module 103;

[0058] The second core module 105 is used to calculate the model result based on the second weight parameter, the activation data and the saliency map, and return the model result to the program system end memory 101 through the data moving module 103 and the CPU 102 in sequence;

[0059] Among them, the first weight parameter is the completed learning weight parameter in the saliency map calculation network in the dynamic neural network sparse model, and the second weight parameter is the completed learning weight parameter of the sparse convolutional network combined with saliency in the dynamic neural network sparse model. The second weight parameter is input into the second core module in batches multiple times based on a preset timing.

[0060] Specifically, the module division in the hardware accelerator architecture supporting the dynamic neural network sparse model corresponds to the network structure in the dynamic neural network sparse model, and the main modules are the first core module 104 and the second core module 105, which respectively correspond to the saliency map calculation network and the sparse convolution network combined with saliency in the network structure during the training of the dynamic neural network sparse model, wherein the first core module 104 is an on-chip computing core module, because the saliency map calculation network in the dynamic neural network sparse model corresponding to the first core module 104 itself is a lightweight network, wherein the number of weight parameters to be optimized that have been learned is small and the amount of computation is low, therefore, the first weight parameter corresponding to the saliency map calculation network is stored by the first core module 104 itself by pre-writing it into the first core module 104, and there is no need to interact with the external program system-side memory 101 to obtain the weight parameter, thereby saving the time of obtaining the first weight parameter from the external memory. It should be noted here that the dynamic neural network sparse model is used to summarize a series of models that can perform specific tasks based on the training of the dynamic neural network sparse algorithm, that is, the dynamic neural network sparse algorithm can be applied to various deep applications of computer vision, such as obtaining an image super-resolution model based on the dynamic neural network sparse algorithm, a semantic segmentation model based on the dynamic neural network sparse algorithm, and a target detection model based on the dynamic neural network sparse algorithm, etc., which are not specifically limited here. The present invention only designs the hardware architecture adapted to the model trained based on the dynamic neural network sparse algorithm, so that the designed hardware architecture can provide a specific adapted hardware combination architecture for the model trained based on the dynamic neural network sparse algorithm, so as to ensure that the computing efficiency is improved on the basis of lower hardware cost, that is, the computing time is reduced, and the reduction of computing time is mainly achieved by reducing the interactive access to data, and the maintenance of lower hardware cost is mainly achieved by obtaining and using data in a time sequence to avoid the need to use a large capacity of memory, which leads to an increase in hardware cost.

[0061] The second core module 105 corresponds to the sparse convolutional network combined with significance in the network structure during the training of the dynamic neural network sparse model, and the sparse convolutional network combined with significance belongs to the backbone functional network in the dynamic neural network sparse model. As mentioned above, based on the different tasks of the dynamic neural network sparse model, the output results of the backbone functional network are different. For example, if it is an image super-resolution task, the output of the backbone functional network is a super-resolution graphic. If it is a target detection task, the output of the backbone functional network is the size and position information of the detection box, and so on. The second core module 105, as a computing component of the sparse convolutional network combined with saliency, needs to bear a large amount of computing, that is, the network corresponds to a large number of convolutional layers, and each convolutional layer has many weight parameters. At the same time, there is a need to perform sparse operations in combination with the saliency map sent by the first core module 104. Therefore, the second weight parameters are input into the second core module in batches multiple times at a preset timing. Specifically, the second weight parameters to be used in each convolutional layer are input into the second core module 105 in batches according to the timing interval set according to the estimated calculation time of each convolutional layer, so as to ensure that the storage space on the second core module 105 is occupied while not delaying the operation of each convolutional layer on the second core module 105.

[0062] The hardware accelerator architecture supporting the dynamic neural network sparse model provided by the present invention deploys the operations of the saliency map calculation network of the dynamic neural network sparse model and the sparse convolution network combined with saliency in the first core module and the second core module respectively. The first core module calculates the saliency map of the input activation data based on the first weight parameter of the saliency map calculation network pre-written into its own storage. Since the saliency map calculation network itself is a lightweight network, the number of the first weight parameters is small and convenient to write and store in the first core module in advance. Therefore, only the activation data of the first layer of the network (i.e., the original image to be tested) needs to be input, and the calculations of all subsequent convolutional layers of the network are all in the first core module. The operation is performed on the chip where the computation is performed, which reduces the number of accesses to the system-side memory of the external memory program, thus reducing the computation time. As for the operation of the sparse convolutional network combined with significance on the second core module, since the sparse convolutional network combined with significance is used as the backbone functional network, in order to ensure the accuracy of the corresponding function of the model, there are many weight parameters. Therefore, in order to cooperate with the limited storage capacity on the chip where the second core module is located, the second weight parameters are input to the second core module in batches according to the preset timing for multiple times. The convolutional layers in the sparse convolutional network combined with significance on the second core module can also be iteratively calculated in sequence, that is, the time is short and the use of low storage capacity memory is guaranteed to reduce the hardware cost. Therefore, the sparse convolutional network combined with significance provided by the present invention reduces the computation time and improves the efficiency during the algorithm application process.

[0063] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the dynamic neural network sparse model is obtained by training based on sample images and corresponding reference standard labels, and the network structure in the training process of the dynamic neural network sparse model includes a saliency map calculation network to be optimized and a sparse convolutional network combined with saliency to be optimized, wherein,

[0064] The saliency map calculation network to be optimized is used to extract a saliency map from the input sample image, obtain a saliency map of the sample image and output it to the sparse convolution network combined with saliency to be optimized;

[0065] The sparse convolutional network combined with saliency to be optimized is used to perform task prediction based on the input sample image saliency map and the input sample image to obtain a prediction result;

[0066] The prediction result and the reference standard label are used to construct a loss function when training the dynamic neural network sparse model;

[0067] Among them, the task prediction corresponds to the training task target during the training of the dynamic neural network sparse model.

[0068] Specifically, the dynamic neural network sparse model is trained based on sample images and corresponding reference standard labels. As mentioned above, there is no limited task target for the dynamic neural network sparse model here, that is, the algorithm can be applied to a variety of image-based learning tasks. Therefore, the training data set used in the training process of the dynamic neural network sparse model is also composed of sample images and corresponding model task target reference standard labels. For example, when the model task target is target detection, the reference standard label is the size information label and position information label of the annotation box automatically framed by other advantageous algorithms on the sample image. If the model task target is image super-resolution, the reference standard label is the super-resolution image label corresponding to the sample image, and so on.

[0069] Figure 2 The process framework diagram of the dynamic neural network sparse model training provided by the present invention is as follows: Figure 2 As shown in Figure 1, the network structure of the model training consists of two parts: the Importance Map (IMAP) network (i.e., the saliency map calculation network of the adjustable parameters to be optimized) and the main functional network (i.e., the sparse convolutional network of the adjustable parameters combined with saliency to be optimized). The IMAP network is used to indicate whether each pixel in the image / video is located in the focus area. It first generates a single-channel feature map F map, the pixel position with a larger corresponding value represents a greater impact on the training index, so more channels will be allocated to the pixel position during the convolution calculation process to achieve better recovery in this important area. Figure 2 A specific example image processing process in the flowchart is given in the figure. For the input image I in Due to the limitation of resolution and the image must be black and white binary, the input image has limited ability to express the scenery as a landscape image. It can be roughly seen that the left side of the image is the "house" area with high saliency, the middle of the image is the "boat" area with more details, and the end of the image is the background "sky" with less details. The importance indicator map (i.e. saliency map) F is obtained by conversion. map In the "ship" and "house" areas with more details, F map are activated by larger values, while less detailed background areas such as the "sky" are activated by smaller values, allowing the F map To distinguish the sparseness that can be achieved in different regions, and then in F map Based on the IMAP network, a binary 0-1 mask M will eventually be generated. The structure of the mask determines the specific sparse structure of the intermediate features of all convolutional layers. In order not to introduce additional computational costs, the IMAP network is designed as a convolutional network with shallow depth and narrow width. It often contains only a few layers of convolution (such as 3 layers) and reduces the number of channels in the intermediate layers (such as the number of channels of each layer of features does not exceed 3). Therefore, the computational load of the IMAP network in the overall network accounts for a very low proportion and does not constitute the main contradiction of the large amount of computation of the overall network. The main functional network (i.e., the sparse convolutional network with the combined significance of the adjustable parameters to be optimized) completes the functions corresponding to the training task objectives pre-set by the dynamic neural network sparse model, such as the core functions of deep compression / super-resolution restoration. During the convolution process, the feature map of each layer will be dot-multiplied with the binary mask M generated by the IMAP network, and structured sparsity is achieved on both the input activation value and the output activation value, thereby reducing the computational load of the overall network and achieving sparse optimization of the model.

[0070] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the task prediction based on the input sample image saliency map and the input sample image specifically includes:

[0071] Extract features from the input sample image to obtain the original feature map of the C channel;

[0072] The input sample image saliency map is converted into a binary mask with C channels using a uniform quantization rule;

[0073] After obtaining the convolution result feature map of the current layer in any convolution layer of the sparse convolution network combined with significance to be optimized, perform a dot multiplication operation on the convolution result feature map of the current layer and the binary mask to obtain the final result feature map of the layer, and output it to the activation layer, so that the activation layer performs activation processing to obtain the activation result, and then inputs it to the next convolution layer, until the prediction result of the last convolution layer in the sparse convolution network combined with significance to be optimized is obtained, wherein the first convolution layer of the sparse convolution network combined with significance to be optimized is input with the original feature map.

[0074] Specifically, the sparse convolutional network combined with saliency in the training process of the dynamic neural network sparse model realizes the function of performing task prediction based on the input sample image saliency map and the input sample image, which is specifically introduced as follows:

[0075] like Figure 2 As shown in the following block diagram, the flow chart of task prediction based on the input sample image saliency map and the input sample image is shown in the following block diagram. The input of the sparse convolutional network combined with saliency includes both the input image I in It also includes a saliency map output by the saliency map calculation network, and based on the input image I in And the saliency map to get the prediction result I out The output process is as follows: First, perform preliminary feature extraction on the input image to obtain the original feature map, and obtain the target channel number of the original feature map, and then convert the saliency map F with a value range between 0 and 1 into map Quantize to obtain the quantization result corresponding to the target feature number (i.e. the number of channels after sparseness), i.e. F map The pixel with a larger saliency value uses more channels to represent the feature value of the feature map at its corresponding position, but the number of channels will not exceed the number of target channels at most. map The smaller the saliency value of the pixel, the fewer channels are used to represent the eigenvalue of the feature map at the corresponding position. For example, the "calm lake" without details can be excluded from the feature convolution operation. At the very least, the number of channels of the pixel position on the feature map corresponding to the "calm lake" area can be set to 0, that is, the "calm lake" area is completely excluded from the feature operation. For each convolution layer in the sparse convolution network combined with saliency, the feature map output by the previous layer needs to be weighted by the current convolution layer, and then the binary mask obtained after quantization needs to be used for dot product operation to obtain the final feature map of the current layer for output, and then output to the activation layer, so that it can be used as the feature map input to the next convolution layer after the activation operation is completed.

[0076] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the step of converting the input sample image saliency map into a binary mask with a channel number of C according to a uniform quantization rule specifically includes:

[0077] The input sample image saliency map F is quantized into a binary mask M with C channels using the uniform quantization rule by the following formula:

[0078]

[0079]

[0080] Wherein, L is the preset quantization order, S is the quantization step size of the uniform quantization rule, C is the number of channels of the original feature map, F(i,j) is the saliency value of the pixel at position (i,j) on the sample image saliency map F, Q(i,j) is the quantization order value of the pixel at position (i,j) on the uniform quantization mask Q, and h is the width of the original feature map.

[0081] Specifically, since the sparsity of the feature maps calculated by each convolution layer in the sparse convolutional network combined with saliency is completely controlled by the binary mask converted from the saliency map output by the saliency map calculation network, the most suitable binary mask generation method is provided below:

[0082] The sample image saliency map F in the saliency map calculation network is determined by the following formula:

[0083] F(i,j)∈(0,1)

[0084] Among them, I in A sample image is input to the saliency map computation network, It represents the saliency map calculation process function, whose physical structure is composed of a small number of convolutional layers connected in sequence (for example, a three-layer convolutional network). The undetermined parameters in are the parameters to be adjusted that need to be learned in the saliency map calculation network;

[0085] The area with lower saliency value in F is less important. Therefore, the number of channels allocated to the feature map of the less important area should be reduced accordingly. In order to implement the above spatial sparse rule in the feature map, a binary mask M needs to be converted from F during the model training stage, that is:

[0086]

[0087]

[0088] Wherein, L is the preset quantization order, S is the quantization step size of the uniform quantization rule, C is the number of channels of the original feature map, F(i,j) is the saliency value of the pixel at position (i,j) on the sample image saliency map F, Q(i,j) is the quantization order value of the pixel at position (i,j) on the uniform quantization mask Q, and h is the width of the original feature map.

[0089] Since the function tanh can be used to approximate the gradient propagation, end-to-end training of the entire network can be achieved. Therefore, the calculation formulas of the two parameters Q(i,j) and M(k) can be combined into the following formula:

[0090]

[0091] Among them, α is a scaling factor used to control the gradient amplitude, and α is preferably 4. sign() is a sign function used to binarize the input to {0,1} and keep the gradient from output to input unchanged.

[0092] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the construction of the loss function specifically includes:

[0093] The loss function L during training of the dynamic neural network sparse model is constructed by the following formula total :

[0094] L total =L main +μL sparse

[0095] L sparse =|l1(F)-γ|

[0096] Among them, L main is the error between the prediction result and the reference standard label, L sparse The sparsity representation of the saliency map calculation network is shown in Figure 2. F is the sample image saliency map, l1() is the 1-norm function, and γ and μ are both preset adjustable hyperparameters.

[0097] Specifically, the loss function is defined by the above formula. It can be seen that the loss function includes two parts. The first part is the error between the prediction result and the reference standard label, L main , that is, L main The determination principle is based on the error between the prediction result and the reference standard label, but the specific calculation formula changes with the task target of the dynamic neural network sparse model. For example, when the task target is super-resolution image extraction, L main The construction is based on L1 / L2 loss. When the task goal is classification or recognition, L mainThe construction of L is based on the cross entropy formula, etc., and all cases are not described here. The evaluation of the saliency map calculation network is still based on the effect of its output saliency map on the sparse control in the sparse convolutional network combined with saliency. Therefore, the constructed L sparse =|l1(F)-γ|, and take it as the optimization target that also needs to be reduced, that is, the smaller the saliency value in F, the better. That is, the fewer the number of channels that need to participate in the feature map convolution calculation in the sparse convolution network combined with saliency, the better. L sparse This is to make the sparse convolutional network combined with saliency even sparser and optimize L sparse and L main It is an adversarial process, that is, it is necessary to balance the amount of computation and the accuracy of the model, and γ and μ are preset adjustable hyperparameters used to control the balance point.

[0098] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the first core module includes an input storage unit, a first weight storage unit, an on-chip convolution core unit and a saliency map memory, wherein:

[0099] The input storage unit is used to receive the activation data forwarded by the data moving module from the CPU and output it to the on-chip convolution core unit,

[0100] The first weight storage unit is used to store the first weight parameter written in advance and output it to the on-chip convolution core unit;

[0101] The on-chip convolution core unit is used to complete the calculation of all convolutional layers in the saliency map calculation network in a streaming structure based on the first weight parameter and the activation data, obtain a saliency map and output it to the saliency map memory;

[0102] The saliency map memory is used to output the saliency map to the second core module.

[0103] Specifically, Figure 3 The second structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention is as follows: Figure 3 The first core module includes an input storage unit, a first weight storage unit, an on-chip convolution core unit and a saliency map memory, wherein the on-chip convolution core unit is used to complete the calculation of all convolution layers in the saliency map calculation network in a streaming structure based on the first weight parameter and the activation data, obtain the saliency map and output it to the saliency map memory, in order to reduce the number of accesses to the external memory DDR, only the activation data of the first layer needs to be input, and then the calculation of all layers is calculated on the chip using a streaming structure, and the F that determines the sparse characteristics of the overall network is mapInformation is stored on the chip. It should be noted that the system-side memory is preferably a double data rate synchronous dynamic random access memory DDRSDRAM (Double Data Rate SDRAM).

[0104] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the first core module also includes a sparse controller, wherein:

[0105] The sparse controller is used to control the on-chip convolution core unit to reduce or enlarge the saliency value of the calculated saliency map according to a preset scaling rule to obtain a scaled saliency map, and control the on-chip convolution core unit to output the scaled saliency map to the saliency map memory;

[0106] The saliency map memory is used to output the scaled saliency map to the second core module.

[0107] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the second core module includes a sparse decoder, an activation value storage unit, a second weight storage unit, a weight loader, an activation layer and a preset calculation array, wherein:

[0108] The sparse decoder is used to receive the saliency map output by the first core module and the activation data input by the data moving module, and decode the activation data based on the binary mask obtained by quantizing the saliency map to obtain a quantized saliency map and output it to the activation value storage unit for storage, so that the activation value storage unit can output it to the preset calculation array;

[0109] The weight loader is used to receive the second weight parameters in batches according to the preset timing, and output them to the second weight storage unit;

[0110] The second weight storage unit is used to output the second weight parameters output by the weight loader to the preset calculation array in batches according to the preset timing;

[0111] The preset calculation array is used to calculate the output result of the convolution layer in the sparse convolution network combined with significance corresponding to the i-th batch based on the second weight parameter of the i-th batch input in the preset time sequence and the activation value corresponding to the i-th batch output by the activation value storage unit, and output it to the activation layer, so that the activation layer can activate the output result and then output it to the activation value storage unit as the activation value of the i+1th batch for storage, until the result of the last convolution layer in the sparse convolution network combined with significance is obtained and returned through the data moving module and the CPU, and output to the data moving module, wherein the activation value of the first batch is the quantized saliency map.

[0112] Specifically, Figure 4 The third structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention is as follows: Figure 4 As shown, the second core module also includes a sparse controller, which can adjust the sparsity based on the bearing capacity of the current hardware. For example, when it is detected that the computing power of the current hardware is strong, more channel values ​​can be satisfied to participate in the calculation of the feature map. Then the sparse controller will control the on-chip convolution core unit to uniformly "upgrade" the saliency value of the calculated saliency map according to the preset amplification rule, and adjust the saliency value to a larger value. Then, when the second core module performs extraction and fusion calculation of the feature map, the number of channels allocated to each pixel area will increase accordingly. On the contrary, when it is detected that the computing power of the current hardware is weak, the sparse controller will control the on-chip convolution core unit to uniformly "downgrade" the saliency value of the calculated saliency map according to the preset reduction rule, and adjust the saliency value to a smaller value. Then, when the second core module performs extraction and fusion calculation of the feature map, the number of channels allocated to each pixel area will decrease accordingly. It should be noted here that there are many preset scaling rules, which can be simple linear mapping scaling, nonlinear fitting, or even more accurate scaling rules learned based on model training, which are not specifically limited here.

[0113] Based on the above embodiment, in the hardware accelerator architecture supporting the dynamic neural network sparse model, the preset computing array is a computing array of size 16×16.

[0114] Specifically, the computing array used on the second core module is preferably a computing array of size 16×16, balancing computing power and hardware cost.

[0115] Based on the above embodiment, in the hardware accelerator architecture that supports the dynamic neural network sparse model, the data movement module forwards data between the program system-side memory and the CPU based on the AXI-Stream protocol.

[0116] Specifically, Figure 5 The fourth structural diagram of a hardware accelerator architecture supporting a dynamic neural network sparse model provided by the present invention is as follows: Figure 5As shown in the figure, the first core module (IMAP computing core) and the second core module (sparse convolution computing core of the backbone network) on the chip. The data handling module transfers the activation and weight data of the model application calculation from the DDR on the PS (Programming System) side to the on-chip storage via the AXI-Stream (AXIS) protocol via the CPU. For the IMAP computing core, since the network itself is lightweight, with few parameters and low computation, in order to reduce the number of accesses to the external memory DDR, only the activation data of the first layer needs to be input, and then all layers are calculated on the chip using a streaming structure, and the Fmap information that determines the sparse characteristics of the overall network is stored on the chip. For the sparse convolution computing core of the backbone network, the sparse activation layer data will be decoded according to the Fmap and stored in the specified storage location on the chip, and then the computing array will read the input activation layer data from the on-chip memory under the control of the data scheduler and complete the sparse convolution calculation. Finally, the sparse output activation data will be continuously written back to the DDR outside the chip via the buffer and the activation layer.

[0117] The method provided by the present invention relates to a training framework for an activation layer dynamic sparse network in the training process of a dynamic neural network sparse model. The sparse network has the advantages of spatial adaptive sparseness, structuring, hardware friendliness, and controllable sparsity. A hardware accelerator supporting the dynamic sparse architecture performs targeted hardware design on the key IMAP network and sparse convolutional network, and combines sparse data compression, sparse computing scheduling, and adaptive sparse control to achieve high-energy-efficiency and low-latency operation of deep networks.

[0118] The present invention has been verified on the image super-resolution task through algorithm design and hardware deployment process, and the network has been deployed on the ZYNQ XC7Z035 FPGA hardware programmable platform. All multiplication units are instantiated using DSP resources. The array size of sparse convolution is 16×16, requiring 256 DSPs, while the IMAP core contains 3 layers of lightweight dense convolution, using a total of 79 DSPs. The weights and activation values ​​of the network are uniformly quantized using 16-bit fixed-point quantization to achieve forward reasoning. In the actual algorithm operation process, for the SRResNet super-resolution model, the implemented sparse accelerator supports adjustable sparsity control. In the optimal case, it can achieve a frame rate of nearly 40 times and an equivalent energy efficiency improvement compared to the dense case, greatly improving the network's operating reasoning efficiency.

[0119] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hardware accelerator architecture supporting a dynamic neural network sparse model, comprising a program system end memory, a central processing unit CPU, a data moving module, a first core module and a second core module, characterized in that: The activation data stored in the program system end memory is sequentially input into the first core module through the CPU and the data moving module; The first core module is used to calculate a saliency map based on a first weight parameter and the activation data stored in advance, and output the saliency map to the second core module; The activation data and the second weight parameter stored in the program system end memory are sequentially input into the second core module through the CPU and the data moving module; The second core module is used to calculate the model result based on the second weight parameter, the activation data and the saliency map, and return the model result to the program system end memory through the data moving module and the CPU in sequence; The first weight parameter is a completed learning weight parameter in a saliency map calculation network in a dynamic neural network sparse model, the second weight parameter is a completed learning weight parameter of a sparse convolutional network combined with saliency in a dynamic neural network sparse model, and the second weight parameter is input into the second core module in batches multiple times based on a preset time sequence; The dynamic neural network sparse model is obtained by training based on sample images and corresponding reference standard labels. The network structure in the training process of the dynamic neural network sparse model includes a saliency map calculation network to be optimized and a sparse convolutional network combined with saliency to be optimized, wherein: The saliency map calculation network to be optimized is used to extract a saliency map from the input sample image, obtain a saliency map of the sample image and output it to the sparse convolution network combined with saliency to be optimized; The sparse convolutional network combined with saliency to be optimized is used to perform task prediction based on the input sample image saliency map and the input sample image to obtain a prediction result; The prediction result and the reference standard label are used to construct a loss function when training the dynamic neural network sparse model; Among them, the task prediction corresponds to the training task target during the training of the dynamic neural network sparse model.

2. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 1, characterized in that: The task prediction based on the input sample image saliency map and the input sample image specifically includes: Extract features from the input sample image and obtain The original feature map of the channel; The input sample image saliency map is converted into a channel number using a uniform quantization rule. Binarization mask of After obtaining the convolution result feature map of the current layer in any convolution layer of the sparse convolution network combined with significance to be optimized, perform a dot multiplication operation on the convolution result feature map of the current layer and the binary mask to obtain the final result feature map of the layer, and output it to the activation layer, so that the activation layer performs activation processing to obtain the activation result, and then inputs it to the next convolution layer, until the prediction result of the last convolution layer in the sparse convolution network combined with significance to be optimized is obtained, wherein the first convolution layer of the sparse convolution network combined with significance to be optimized is input with the original feature map.

3. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 2, characterized in that: The input sample image saliency map is converted into a channel number of The binary mask includes: The input sample image saliency map is converted into The number of channels is quantized by uniform quantization rule. Binarization mask of : ； ， ； in, is the preset quantization order, is the quantization step size of the uniform quantization rule , is the number of channels of the original feature map, is the sample image saliency map The upper position is The saliency value of the pixel, The position on the uniform quantization mask Q is The quantization level value of the pixel point, is the width of the original feature map.

4. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 3, characterized in that: The construction of the loss function specifically includes: The loss function of the dynamic neural network sparse model training is constructed by the following formula : + ； ； in, is the error representation between the prediction result and the reference standard label, computing a sparse representation of the network for the saliency map, is the sample image saliency map, is a 1-norm function, and All are preset adjustable hyperparameters.

5. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 4, characterized in that: The first core module includes an input storage unit, a first weight storage unit, an on-chip convolution core unit and a saliency map memory, wherein: The input storage unit is used to receive the activation data forwarded by the data moving module from the CPU and output it to the on-chip convolution core unit, The first weight storage unit is used to store the first weight parameter written in advance and output it to the on-chip convolution core unit; The on-chip convolution core unit is used to complete the calculation of all convolutional layers in the saliency map calculation network in a streaming structure based on the first weight parameter and the activation data, obtain a saliency map and output it to the saliency map memory; The saliency map memory is used to output the saliency map to the second core module.

6. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 5, characterized in that: The first core module also includes a sparse controller, wherein: The sparse controller is used to control the on-chip convolution core unit to reduce or enlarge the saliency value of the calculated saliency map according to a preset scaling rule to obtain a scaled saliency map, and control the on-chip convolution core unit to output the scaled saliency map to the saliency map memory; The saliency map memory is used to output the scaled saliency map to the second core module.

7. The hardware accelerator architecture supporting dynamic neural network sparse model according to any one of claims 4 to 6, characterized in that: The second core module includes a sparse decoder, an activation value storage unit, a second weight storage unit, a weight loader, an activation layer and a preset calculation array, wherein: The sparse decoder is used to receive the saliency map output by the first core module and the activation data input by the data moving module, and decode the activation data based on the binary mask obtained by quantizing the saliency map to obtain a quantized saliency map and output it to the activation value storage unit for storage, so that the activation value storage unit can output it to the preset calculation array; The weight loader is used to receive the second weight parameters in batches according to the preset timing, and output them to the second weight storage unit; The second weight storage unit is used to output the second weight parameters output by the weight loader to the preset calculation array in batches according to the preset timing; The preset calculation array is used to calculate the output result of the convolution layer in the sparse convolution network combined with significance corresponding to the i-th batch based on the second weight parameter of the i-th batch input in the preset time sequence and the activation value corresponding to the i-th batch output by the activation value storage unit, and output it to the activation layer, so that the activation layer can activate the output result and then output it to the activation value storage unit as the activation value of the i+1th batch for storage, until the result of the last convolution layer in the sparse convolution network combined with significance is obtained and returned through the data moving module and the CPU, and output to the data moving module, wherein the activation value of the first batch is a quantized saliency map.

8. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 7, characterized in that: The default calculation array is 16 The calculation array.

9. The hardware accelerator architecture supporting dynamic neural network sparse model according to claim 8, characterized in that: The data moving module forwards data between the program system end memory and the CPU based on the AXI-Stream protocol.

Images