Accelerator and acceleration method based on convolutional neural network

Abstract

The application provides a convolutional neural network-based accelerator and an acceleration method. The accelerator comprises a compiler, an acquisition unit, an on-chip storage and a calculation unit. The on-chip storage is divided into a cache mode and a storage mode. In the cache mode, the on-chip storage is divided into multiple cache areas, input feature maps are written into a current cache area row by row, and the obtained intermediate feature maps and the input feature maps are input into a next layer cache area. In the storage mode, the on-chip storage is divided into two storage areas, the input feature maps are written into one storage area, the obtained intermediate feature maps are written into the other storage area, and the calculation is alternately performed in the two storage areas. According to the compiler, the ONNX model is parsed, the input feature maps are classified through an operator group in the ONNX model, different running modes of the on-chip storage are selected for the input feature maps, the repeated data handling in the calculation process is reduced, and the problem of low calculation efficiency of the convolutional neural network hardware accelerator is solved.

Description

Technical Field

[0001] This application relates to the field of neural networks, and more particularly to an accelerator and acceleration method based on convolutional neural networks. Background Technology

[0002] Convolutional neural networks (CNNs) have been widely applied in fields such as computer vision and natural language processing. However, due to the unique computational nature of CNNs, general-purpose processors are not efficient enough to implement them and cannot meet performance requirements. Therefore, various hardware accelerators for CNNs based on Field-Programmable Gate Arrays (FPGAs), Graphics Processing Units (GPUs), and Application-Specific Integrated Circuits (ASICs) have been proposed recently to improve the performance of CNNs.

[0003] Convolutional Neural Network (CNN) hardware accelerators combine low power consumption with high speed for convolution-related operations, making them one of the mainstream hardware options for deploying CNNs. However, due to the wide variety of CNN models, each with different data distributions, weight distributions, computational flows, and varying data sizes across different computational layers, computational efficiency is sacrificed to accommodate these different computational and data flows. Furthermore, within the accelerator, data transmission and computation are sequential, requiring the computational array to wait for data transmission to complete before commencing computation, resulting in wasted computational power during this waiting period and ultimately, low computational efficiency. Summary of the Invention

[0004] This application provides an accelerator and acceleration method based on convolutional neural networks to solve the problem of low computational efficiency of convolutional neural network hardware accelerators.

[0005] The first aspect of this application provides an accelerator based on a convolutional neural network, including: a compiler, an acquisition unit, on-chip memory, and a computing unit;

[0006] The compiler is used to parse the ONNX model, which includes multiple operator groups;

[0007] The acquisition unit is used to acquire the input feature map and weight data;

[0008] The on-chip memory includes a static random access memory (SRAM), which is used to store the input feature map, intermediate feature map, and output feature map. The SRAM has two operating modes: cache mode and storage mode.

[0009] The calculation unit is used to read the weight data and calculate the number of weight data in real time by inputting the number of feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel.

[0010] The compiler is also configured to: perform classification on the input feature map using the set of operators, and obtain the size of the input feature map;

[0011] The compiler is configured to: when the size is greater than or equal to a size threshold and the number of weight data is less than a quantity threshold, set the SRAM's operating mode to cache mode, wherein the SRAM is divided into multiple cache areas in the cache mode;

[0012] The computing unit is used to write the input feature map line by line into the cache area, and perform multiplication and accumulation operations with the weight data to obtain the intermediate feature map; and cache the intermediate feature map and the input feature map to the next layer cache area; and traverse the multi-layer cache area until the output feature map is obtained.

[0013] The compiler is further configured to: set the SRAM's operating mode to storage mode if the size is less than a size threshold or the number of weight data is greater than a quantity threshold, wherein the SRAM is divided into two storage areas in storage mode;

[0014] The computing unit is also used to write the input feature map into one of the storage areas, and perform multiplication and accumulation operations with the weight data to obtain an intermediate feature map, and write the intermediate feature map into another storage area; and to perform calculations alternately in the two storage areas until an output feature map is obtained.

[0015] Optionally, the SRAM includes multiple rows of memory;

[0016] When the SRAM is in cache mode, the cache area has N layers. The number of rows in the first N-1 layers of the cache area is the first number of rows of memory, and the number of rows in the Nth layer of the cache area is the second number of rows of memory.

[0017] When the SRAM is in storage mode, the number of rows in the two storage areas is half the number of rows in the SRAM.

[0018] Optionally, the acquisition module is further configured to acquire a computation graph of a convolutional neural network model, the computation graph including multiple operators;

[0019] The compiler includes a front-end, which is used to perform quantization processing on the convolutional neural network model based on the ONNX model and quantization tools to obtain the operator IR topology graph;

[0020] Based on the operator IR topology graph, the operators in the computation graph are fused to generate a new computation graph.

[0021] Optionally, the compiler includes a middleware;

[0022] The middle section includes an operator grouping module and an input feature map segmentation module;

[0023] The operator grouping module is used to divide the new computation graph into a first operator group and a second operator group. The first operator group is used to calculate the input feature map of the buffer area, and the second operator group is used to calculate the input feature map of the storage area.

[0024] The input feature map segmentation module is used to calculate the alternation information of the input feature map and the intermediate feature map of the second operator group.

[0025] Optionally, the alternation information includes a start row, an end row, a start column, an end column, a start channel, and an end channel;

[0026] The acquisition unit is also used to acquire the image matrix of the input feature map;

[0027] The alternation information is used to provide the second operator group with alternating start rows, end rows or start columns, end columns or start channels and end channels of the image matrix.

[0028] Optionally, the compiler further includes a backend, and the accelerator further includes an instruction register;

[0029] The backend includes an intermediate data storage and scheduling module and an instruction generation module;

[0030] The intermediate data storage scheduling module is used to generate multiple instruction nodes for each of the second operator groups according to the alternating number in the storage area, as well as data node information for generating the instruction nodes;

[0031] The instruction register is used to store the execution instructions and configuration data of the convolutional neural network model;

[0032] The instruction generation module is used to convert the execution instructions and configuration data into binary data, and is also used to assign values ​​to the instruction register according to the binary data.

[0033] Optionally, the backend may further include a node mapping module;

[0034] The node mapping module is used to map different computing nodes through the front-end, middle-end, intermediate data storage and scheduling module and the instruction generation module;

[0035] The different computing nodes are sorted to generate a top-level computing graph, which is used to provide computing nodes for the acquisition unit, on-chip memory, and computing unit.

[0036] Optionally, the accelerator further includes a parameter configuration unit, which is used to pre-store the configuration parameters for each layer of the convolutional neural network model, including the channels of the input feature map, the channels of the intermediate feature map, and the size of the convolutional kernel.

[0037] Optionally, the on-chip memory further includes a weighted cache, and the accelerator further includes a dual-port buffer;

[0038] When the SRAM is in cache mode, the weight cache is used to store weight data.

[0039] When the SRAM is in storage mode, the weight data is stored using the dual-port buffer.

[0040] A second aspect of this application provides an acceleration method based on a convolutional neural network, applied to the convolutional neural network-based accelerator described in the first aspect, comprising:

[0041] The ONNX model is analyzed, and the ONNX model includes multiple operator groups;

[0042] Obtain the input feature map and weight data;

[0043] Read the weight data and calculate the number of weight data in real time by inputting the number of feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel;

[0044] The input feature map is classified using the set of operators, and the size of the input feature map is obtained.

[0045] If the size is greater than or equal to a size threshold and the number of weight data is less than a quantity threshold, the operating mode of the Static Random Access Memory (SRAM) is set to cache mode. In cache mode, the SRAM is divided into multiple cache areas. The input feature map is written to the cache area line by line and multiplied and accumulated with the weight data to obtain an intermediate feature map. The intermediate feature map and the input feature map are then cached to the next layer of cache area. The multiple cache areas are traversed until the output feature map is obtained.

[0046] If the size is less than the size threshold or the number of weight data is greater than the number threshold, the SRAM is set to storage mode, wherein the SRAM is divided into two storage areas in storage mode; the input feature map is written into one of the storage areas and multiplied and accumulated with the weight data to obtain an intermediate feature map, and the intermediate feature map is written into the other storage area; the calculation is performed alternately in the two storage areas until the output feature map is obtained.

[0047] As can be seen from the above technical solutions, this application provides an accelerator and acceleration method based on convolutional neural networks. The accelerator includes a compiler, an acquisition unit, on-chip memory, and a computing unit. By dividing the on-chip memory into a cache mode and a storage mode, in cache mode, the on-chip memory is divided into multiple cache areas. The input feature map can be written line by line into the current cache area, and the obtained intermediate feature map and the input feature map are input into the next cache area. This process is repeated across multiple cache areas until the output feature map is obtained. In storage mode, the on-chip memory is divided into two storage areas. The input feature map is written into one storage area, and the obtained intermediate feature map is written into the other storage area. Computation is performed alternately in the two storage areas until the output feature map is obtained. Furthermore, based on the compiler's parsing of the ONNX model, the input feature map can be classified using the operator groups in the ONNX model. Different operating modes of the on-chip memory can be selected for the input feature map to reduce redundant data transfer during computation, thereby solving the problem of low computational efficiency in convolutional neural network hardware accelerators. Attached Figure Description

[0048] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of an accelerator structure based on a convolutional neural network, as shown in one embodiment.

[0050] Figure 2 This is a diagram illustrating the convolution operation process;

[0051] Figure 3 This is a schematic diagram of the compiler result shown in one embodiment;

[0052] Figure 4 This is a schematic diagram of the pipeline mode structure;

[0053] Figure 5 This is a schematic diagram of the pipelined computation process.

[0054] Figure 6 This is a schematic diagram of the ping-pong mode structure;

[0055] Figure 7 This is a schematic diagram of the calculation process for the Ping Pong mode;

[0056] Figure 8 This is a chart comparing the input and output of data. Detailed Implementation

[0057] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0058] Convolution operations have two inputs: the input feature map and the convolution kernel weights. The convolution operation is a multiplication and addition operation between the convolution kernel and the input feature map, and the calculation formula is as follows:

[0059] A(i,j)=(I*K)(i,j)=∑ m ∑ n I(m,n)K(im,jn);

[0060] Where A is the result of the convolution operation, I is the input feature map, and K is the convolution kernel. See also Figure 2 , Figure 2 This is a diagram illustrating the convolution operation process. For example, let's take a 2D convolution operation with a kernel size of 3×3 as an example. Assume the size of the input feature map is (5, 5) and the stride is 1. The convolution operation can be viewed as the sliding of the convolution kernel window across the input feature map. A stride of 1 means that the convolution kernel slides one grid at a time.

[0061] Figure 2 The intermediate result, or intermediate feature map, is obtained by multiplying and adding the numbers within the black boxes of the input feature map and the convolution kernel. This process is repeated, with the convolution kernel sliding across the input feature map to obtain the output feature map. For 3D convolution operations, the number of channels in the input feature map and the convolution kernel remains the same. During computation, the results from different channels are accumulated to obtain the final output. If multiple convolution kernels perform convolution operations on the same input feature map, the results are accumulated along the output channel direction to obtain the 3D output.

[0062] Calculations show that performing a convolution operation with kernel size (k, k, z, f) on an input feature map of size (x, y, z) (where x and y are the length and width of the input feature map, respectively, and are greater than the length and width of the convolution kernel k; z is the number of channels in the input feature map; and f is the number of convolution kernels) results in an output size of (x-k+1, y-k+1, f). It can be concluded that the more channels there are in the input and output, the larger the amount of weight data required for the corresponding computational layer.

[0063] Convolutional neural networks (CNNs) typically involve multiple layers of convolution operations with varying feature map sizes. In the input portion of a CNN, the feature maps have larger length and width dimensions, smaller channel count dimensions, and smaller weight data. Conversely, in the middle and final output portions of the CNN, the feature maps have smaller length and width dimensions, larger channel counts, and larger weight data.

[0064] When performing convolution operations, hardware accelerators need to write the input feature map data to on-chip memory (IMM) for faster read / write speeds. Since convolution operations are related to the position of data within the input feature map, the writing process must be planned according to the map's layout. For feature maps with large length and width dimensions, if the limited on-chip memory space is to be accommodated, the input feature map needs to be sliced ​​and loaded into on-chip memory in batches for processing and storage. However, because data near the center of the input feature map undergoes multiple convolution operations in a single operation, excessive slicing leads to these data being repeatedly written to on-chip memory for processing, resulting in wasted memory space. If the memory can only hold the first three rows of data, then that row needs to be moved to on-chip memory three times. This space waste increases with the number of convolutional layers and slows down the processing speed.

[0065] See Figure 1 This application provides an accelerator based on a convolutional neural network, including: a compiler, an acquisition unit, on-chip memory, and a computing unit.

[0066] The compiler is used to parse the ONNX model, which includes multiple operator groups; the acquisition unit is used to acquire the input feature map and weight data; the on-chip memory includes static random access memory (SRAM), which is used to store the input feature map, and the SRAM can operate in cache mode and storage mode; the computation unit is used to read the weight data and calculate the amount of weight data in real time by using the number of input feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel.

[0067] The compiler is also used to: perform classification on the input feature map using a set of operators, and to obtain the size of the input feature map.

[0068] The compiler is used to set the SRAM's operating mode to cache mode when the size is greater than or equal to the size threshold and the number of weight data is less than the number threshold. In cache mode, the SRAM is divided into multiple layers of cache.

[0069] The computation unit is used to write the input feature map line by line into the buffer, and perform multiplication and accumulation operations with the weight data to obtain the intermediate feature map. The intermediate feature map and the input feature map are then cached in the next layer of the buffer. The process is repeated for multiple layers of buffers until the output feature map is obtained.

[0070] The compiler is also used to: set the SRAM mode to storage mode if the size is smaller than the size threshold or the number of weight data is greater than the number threshold. In storage mode, the SRAM is divided into two storage areas.

[0071] The computation unit is also used to write the input feature map into one storage area, perform multiplication and accumulation operations with the weight data to obtain an intermediate feature map, and write the intermediate feature map into another storage area; the computation is performed alternately in the two storage areas until the output feature map is obtained.

[0072] The compiler's input model is the ONNX model, an open file format designed for machine learning. It supports the conversion of models from many popular deep learning frameworks, such as PyTorch, TensorFlow, and Caffe2, into ONNX models to facilitate network model parsing and deployment. This allows different deep learning frameworks to store model data in the same format. In this embodiment, the computation unit utilizes a convolutional neural network for computation, and the compiler parses the ONNX model into intermediate expressions for reconstructing the convolutional neural network.

[0073] See Figure 3 In some embodiments, the compiler comprises three parts: a front-end, a middleware, and a back-end.

[0074] The front-end is used to quantize the convolutional neural network model based on the ONNX model and quantization tools to obtain the operator IR topology graph. The front-end includes a quantization information loading and parsing module, an operator fusion optimization module, and an operator computation order scheduling module. According to the operator IR topology graph, the operators in the computation graph are fused to generate a new computation graph.

[0075] Given an ONNX model, it is first put into a quantization tool for quantization with a given width and number of fixed points. The quantization tool will output the quantization result as a quantization file. After quantization, the compiler front end receives the ONNX model and its corresponding quantization file, and the quantization information is loaded into the parsing module into the front end's intermediate representation (IR), which is then provided to the operator fusion optimization module for parsing and optimization.

[0076] The operator fusion optimization module further processes the data to obtain an operator IR topology that is more closely aligned with the convolutional neural network model used by the original computation unit than the model itself. This module can be used to perform fusion processing on consecutive operators in a convolutional neural network. For example, convolution, batch normalization, and activation functions are three consecutive operators. The module includes a data path, allowing computational data to flow sequentially through these three operators before being output to on-chip cache or off-chip storage, rather than computing each operator individually.

[0077] After operator fusion is completed, the operator computation order scheduling module reconnects and schedules the computation of all operators to form a new computation graph.

[0078] After forming a new computation graph, the IR can be further transformed according to different processing methods of the input feature graph. That is, operators are grouped according to the computation mode of the cache area and storage area to increase the number of continuous computing nodes on the chip and reduce data interaction with the off-chip main memory.

[0079] For example, for a computation graph to be grouped, that is, the new computation graph formed above, it can be divided into two segments according to the weight data volume of the convolutional neural network and the size of the input feature map in topological order. The upper segment is used for computation in the buffer area, and the lower segment is used for computation in the storage area; that is, the operator is divided.

[0080] In some embodiments, the middle section includes an operator grouping module; the operator grouping module is used to divide the new computation graph into a first operator group and a second operator group, the first operator group is used to compute the input feature map of the buffer area, and the second operator group is used to compute the input feature map of the storage area.

[0081] In some embodiments, the SRAM includes multiple rows of memory; when the SRAM operates in cache mode, the cache has N layers, the first N-1 layers of cache have a first number of rows of memory, and the Nth layer of cache has a second number of rows of memory; when the SRAM operates in storage mode, the number of rows in the two storage areas is half the number of rows in the SRAM.

[0082] For example, SRAM selects 32 rows of memory. When the SRAM is in cache mode, the cache has 11 layers. The first 10 layers of the cache have 3 rows each, and the 11th layer of the cache has 2 rows each. During operation, the input feature map is written into the instruction cache line by line for the computing unit to use. The intermediate feature map obtained is stored in another cache.

[0083] For example again, when the SRAM is in storage mode, the SRAM is divided into two storage areas, Abank and Bbank, with 16 rows of memory in both Abank and Bbank.

[0084] The computing unit includes a computing array. During operation, the computing array reads weight data and performs multiplication and accumulation operations with the input feature map until the output feature map is obtained.

[0085] During computation, the computing unit also calculates the amount of weight data in real time and sets the SRAM operating mode based on the size of the intermediate feature maps and the amount of weight data. In some embodiments, the accelerator also includes a parameter configuration unit, which is used to pre-store the configuration parameters for each convolutional layer of the convolutional neural network model. The configuration parameters include the number of channels in the input feature map, the number of channels in the intermediate feature map, and the size of the convolutional kernel. During computation, the computing unit calculates the amount of weight data in real time by considering the number of channels in the input feature map, the number of channels in the intermediate feature map, and the size of the convolutional kernel. The compiler weighs the amount of weight data and the size of the input feature map against the SRAM operating mode to determine the minimum data throughput.

[0086] As the neural network extends further, the size of the input feature map gradually decreases, but the number of channels in the input feature map and intermediate feature maps increases, leading to a larger amount of weight data. In the early stages of the convolutional neural network, when the input feature map size is large and the weight data is small, the SRAM is set to cache mode. In cache mode, the SRAM is divided into multiple layers of cache. In this embodiment, the calculation process in the cache is defined as pipeline mode.

[0087] See Figure 4 , Figure 5Taking a 3×3 multi-layer convolution with 11 layers in the buffer as an example, the pipeline mode is introduced. When calculating the multi-layer convolutional neural network, the first three rows of the input of the first convolution are first stored in the buffer [0]. The calculation array calculates the result (the result is a row, which is the intermediate feature map of the first convolution and the input feature map of the second convolution), and the result is stored in the buffer [1]. At this time, the data of the first row in the buffer [0] no longer needs to participate in the convolution operation, so the fourth row of the input of the first convolution is stored to cover the position of the first row. Then, according to the second, third and fourth rows of input data, the second row of the result is calculated in the calculation array and stored in the buffer [1]. Similarly, the fifth row of the input of the first convolution is stored to cover the position of the second row in the buffer [0] and the third row of the result is calculated. At this time, there are already three rows of result data in the buffer [1]. These three rows of result data can be calculated using the calculation array to output the result of the second convolution and store the result of the second convolution in the buffer [2]. Similarly, 10 three-row buffers can perform up to 10 layers of 3×3 convolution operations simultaneously without requiring image slicing of the input feature map. Additionally, this mode involves rapid data switching, requiring quick and repeated reading of weight data.

[0088] Since the weight data is relatively small, it can be read from on-chip memory. In some embodiments, the on-chip memory also includes a weight cache, which stores the weight data when the SRAM is in cache mode.

[0089] As the depth of a convolutional neural network increases, the length and width of the input feature map gradually decrease, while the number of channels gradually increases, resulting in a larger amount of weight data. At this point, the SRAM is set to storage mode, where it is divided into two storage areas. In this embodiment, the calculation process in the storage area is defined as ping-pong mode.

[0090] In some embodiments, the middle section includes an input feature map segmentation module; the input feature map segmentation module is used to calculate the alternation information of the input feature map and the intermediate feature map of the second operator group. The alternation information includes a start row, an end row, a start column, an end column, a start channel, and an end channel; the acquisition unit is also used to acquire the image matrix of the input feature map; the alternation information is used to provide the second operator group with the start row, end row, or start column, end column, or start channel and end channel of the alternating image matrix.

[0091] See Figure 6 , Figure 7For example, during computation, the input feature map is stored in Abank, read in by the computation array for computation, and the result is placed in Bbank (as shown by arrow 1 in the figure); when computing the next convolution layer, the computation array reads the result of the previous computation from Bbank, and the result is placed in Abank (as shown by arrow 2 in the figure), thus completing the computation of two convolution layers.

[0092] For example, in ping-pong computing mode, the alternating computation process is to segment the input feature map. When segmenting, the segmentation position needs to be preset, i.e., the alternating information. For example, if the input feature map is 30 rows, 30 columns, and 64 channels, and the map needs to be segmented in one row direction, if it is segmented into 1 / 3 and put into the storage area, the alternating information is that rows 0-10, columns 0-33, and channels 0-63 are one group, rows 11-20, columns 0-33, and channels 0-63 are another group, and so on. The start and end information of this row, column, and channel can be provided to the storage area so that the storage area has the segmentation conditions.

[0093] For the number of cut rows and additional data transport rate when using only the ping-pong calculation mode, please refer to Tables 1-4.

[0094] Table 1 ResNet34

[0095]

[0096] Table 2 Mobile Net

[0097]

[0098] Table 3 YOLOv3-tiny

[0099]

[0100] Table 4 YOLOv3

[0101]

[0102]

[0103] The number of image slicing rows and the additional data transport rate when the above neural network models use a combination of data flow mode and ping-pong computing mode are shown in Tables 5-8.

[0104] Table 5 ResNet34

[0105]

[0106] Table 6 Mobile Net

[0107]

[0108] Table 7 YOLOv3-tiny

[0109]

[0110] Table 8 YOLOv3

[0111]

[0112]

[0113] As shown in Tables 1-8, using a combination of two data streams significantly reduces the number of image slices and the rate of additional data transport.

[0114] When the feature map size is small, the input feature map can be directly placed into the buffer without slicing. However, in this case, for pipelined mode, the amount of weight data required for computation is large, making it difficult to fit into on-chip memory and hindering fast read / switch functionality. In some embodiments, the accelerator also includes a dual-port buffer, which stores weight data when the SRAM is in storage mode.

[0115] In some embodiments, the compiler backend includes an intermediate data storage scheduling module and an instruction generation module. The intermediate data storage scheduling module is used to generate multiple instruction nodes for each second operator group according to the alternating number in the storage area, as well as data node information for generating instruction nodes. That is, after the splitting conditions are set, the operator group is split into multiple instruction nodes according to the number of splits.

[0116] For different computational models in the cache and storage areas, the compiler generates different instructions. In some embodiments, the accelerator also includes an instruction register, which stores the execution instructions and configuration data of the convolutional neural network model; the instruction generation module is used to convert the execution instructions and configuration data into binary data, and also to assign values ​​to the instruction register based on the binary data. During initialization, binary data can be placed in main memory, and during operation, it can be read and the corresponding instruction register can be assigned a value.

[0117] In some embodiments, the backend further includes a node mapping module; the node mapping module is used to map different computing nodes to the frontend, middle, intermediate data storage scheduling module and instruction generation module; the different computing nodes are sorted to generate a top-level computing graph, which provides computing nodes for the acquisition unit, on-chip memory, and computing unit. Here, a computing node is a group of operators computed during the computation process. The execution process of the acquisition unit, on-chip memory, and computing unit is to traverse this top-level computing graph and execute each computing node until the output feature map is obtained.

[0118] The compiler can evaluate the convolutional neural network model based on the data flow in the buffer and storage areas. Depending on the operator input / output size and the amount of weight data, different computation flows (pipeline mode and ping-pong computation mode) can be specified to reduce redundant data handling and improve handling efficiency.

[0119] Based on the aforementioned accelerator based on a convolutional neural network, some embodiments of this application also provide an acceleration method based on a convolutional neural network, including:

[0120] Analyze the ONNX model, which consists of multiple operator groups;

[0121] Obtain the input feature map and weight data;

[0122] Read the weight data and calculate the amount of weight data in real time by inputting the number of feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel.

[0123] The input feature map is classified using a set of operators, and the size of the input feature map is obtained for judgment.

[0124] If the size is greater than or equal to the size threshold and the number of weight data is less than the number threshold, the operating mode of the static random access memory (SRAM) is set to cache mode. In cache mode, the SRAM will be divided into multiple cache areas. The input feature map is written to the current cache area line by line and multiplied and accumulated with the weight data to obtain the intermediate feature map. The intermediate feature map and the input feature map are then cached in the next cache area. The multiple cache areas are traversed until the output feature map is obtained.

[0125] If the size is smaller than the size threshold or the number of weight data is greater than the number threshold, the SRAM is set to storage mode. In storage mode, the SRAM is divided into two storage areas. The input feature map is written into one storage area and multiplied and accumulated with the weight data to obtain an intermediate feature map. The intermediate feature map is then written into the other storage area. The calculation is performed alternately in the two storage areas until the output feature map is obtained.

[0126] When accelerating computation in a convolutional neural network, the acceleration method provided in this embodiment configures the computation flow (pipeline computation mode, ping-pong computation mode) according to the size of the input feature map and the weight data size of the convolution computation, and stores the data in the cache or storage area according to the computation flow type for the computation array to read and write.

[0127] See Figure 8As can be seen, when using a hybrid pipeline and ping-pong computing model to guide the computation flow, the amount of data input and output is significantly reduced compared to using only the ping-pong computing model. This demonstrates that configuring the two data flow models based on the input feature map size and the number of weights can reduce the amount of data transport, increase the data transport speed, and allocate more time to the computation array, thereby improving the computational efficiency of the hardware accelerator.

[0128] As can be seen from the above technical solutions, this application provides an accelerator and acceleration method based on convolutional neural networks. The accelerator includes a compiler, an acquisition unit, on-chip memory, and a computation unit. By dividing the on-chip memory into a cache mode and a storage mode, in cache mode, the on-chip memory is divided into multiple cache areas. The input feature map can be written line by line into the current cache area, and the obtained intermediate feature map and the input feature map are input into the next cache area. This process is repeated across multiple cache areas until the output feature map is obtained. In storage mode, the on-chip memory is divided into two storage areas. The input feature map is written into one storage area, and the obtained intermediate feature map is written into the other storage area. Computation is performed alternately in the two storage areas until the output feature map is obtained. Furthermore, based on the compiler's parsing of the ONNX model, the input feature map can be classified using the operator groups in the ONNX model. Different operating modes of the on-chip memory can be selected for the input feature map to reduce redundant data transfer during computation, thereby solving the problem of low computational efficiency in convolutional neural network hardware accelerators.

[0129] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. An accelerator based on a convolutional neural network, characterized in that, include: A compiler for parsing an ONNX model, the ONNX model comprising multiple operator groups; The acquisition unit is used to acquire the input feature map and weight data; On-chip memory, including static random access memory (SRAM), the SRAM is used to store the input feature map, intermediate feature map and output feature map, and the SRAM has two operating modes: cache mode and storage mode. The calculation unit is used to read the weight data and calculate the amount of weight data in real time by inputting the number of feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel; The compiler is also configured to: perform classification on the input feature map using the set of operators, and obtain the size of the input feature map; The compiler is configured to: when the size is greater than or equal to a size threshold and the number of weight data is less than a quantity threshold, set the SRAM's operating mode to cache mode, wherein the SRAM is divided into multiple cache areas in the cache mode; The computing unit is used to write the input feature map line by line into the cache area, and perform multiplication and accumulation operations with the weight data to obtain an intermediate feature map; and cache the intermediate feature map and the input feature map to the next layer cache area; and traverse the multi-layer cache area until the output feature map is obtained. The compiler is further configured to: set the SRAM's operating mode to storage mode if the size is less than a size threshold or the number of weight data is greater than a quantity threshold, wherein the SRAM is divided into two storage areas in storage mode; The computing unit is also used to write the input feature map into one of the storage areas, and perform multiplication and accumulation operations with the weight data to obtain an intermediate feature map, and write the intermediate feature map into another storage area; and to perform calculations alternately in the two storage areas until an output feature map is obtained.

2. The accelerator based on a convolutional neural network according to claim 1, characterized in that, The SRAM includes multiple rows of memory; When the SRAM operates in cache mode, the cache area has N layers. The number of rows in the first N-1 layers of the cache area is the first number of rows of memory, and the number of rows in the Nth layer of the cache area is the second number of rows of memory. When the SRAM is in storage mode, the number of rows in the two storage areas is half the number of rows in the SRAM.

3. The accelerator based on a convolutional neural network according to claim 1, characterized in that, The acquisition module is also used to acquire the computation graph of the convolutional neural network model, the computation graph including multiple operators; The compiler includes a front-end, which is used to perform quantization processing on the convolutional neural network model based on the ONNX model and quantization tools to obtain the operator IR topology graph; Based on the operator IR topology graph, the operators in the computation graph are fused to generate a new computation graph.

4. The accelerator based on a convolutional neural network according to claim 3, characterized in that, The compiler includes a middleware; The middle section includes an operator grouping module and an input feature map segmentation module; The operator grouping module is used to divide the new computation graph into a first operator group and a second operator group. The first operator group is used to calculate the input feature map of the buffer area, and the second operator group is used to calculate the input feature map of the storage area. The input feature map segmentation module is used to calculate the alternation information of the input feature map and the intermediate feature map of the second operator group.

5. The accelerator based on a convolutional neural network according to claim 4, characterized in that, The alternation information includes the starting row, ending row, starting column, ending column, starting channel, and ending channel; The acquisition unit is also used to acquire the image matrix of the input feature map; The alternation information is used to provide the second operator group with alternating start rows, end rows or start columns, end columns or start channels and end channels of the image matrix.

6. The accelerator based on a convolutional neural network according to claim 4, characterized in that, The compiler also includes a backend, and the accelerator also includes an instruction register; The backend includes an intermediate data storage and scheduling module and an instruction generation module; The intermediate data storage scheduling module is used to generate multiple instruction nodes for each of the second operator groups according to the alternating number in the storage area, as well as data node information for generating the instruction nodes; The instruction register is used to store the execution instructions and configuration data of the convolutional neural network model; The instruction generation module is used to convert the execution instructions and configuration data into binary data, and to assign values ​​to the instruction register based on the binary data.

7. The accelerator based on a convolutional neural network according to claim 6, characterized in that, The backend also includes a node mapping module; The node mapping module is used to map different computing nodes through the front-end, middle-end, intermediate data storage and scheduling module and the instruction generation module; The different computing nodes are sorted to generate a top-level computing graph, which is used to provide computing nodes for the acquisition unit, on-chip memory, and computing unit.

8. The accelerator based on a convolutional neural network according to claim 7, characterized in that, The accelerator also includes a parameter configuration unit, which is used to pre-store the configuration parameters for each layer of the convolutional neural network model. The configuration parameters include the channels of the input feature map, the channels of the intermediate feature map, and the size of the convolutional kernel.

9. The accelerator based on a convolutional neural network according to claim 1, characterized in that, The on-chip memory also includes a weighted cache, and the accelerator also includes a dual-port buffer; When the SRAM is in cache mode, the weight cache is used to store weight data. When the SRAM is in storage mode, the weight data is stored using the dual-port buffer.

10. An acceleration method based on convolutional neural networks, characterized in that, An accelerator based on a convolutional neural network as described in any one of claims 1-9, comprising: The ONNX model is analyzed, and the ONNX model includes multiple operator groups; Obtain the input feature map and weight data; Read the weight data and calculate the number of weight data in real time by inputting the number of feature map channels, the number of intermediate feature map channels, and the size of the convolution kernel; The input feature map is classified using the set of operators, and the size of the input feature map is obtained. If the size is greater than or equal to a size threshold and the number of weight data is less than a quantity threshold, the operating mode of the Static Random Access Memory (SRAM) is set to cache mode. In cache mode, the SRAM is divided into multiple cache areas. The input feature map is written to the cache area line by line and multiplied and accumulated with the weight data to obtain an intermediate feature map. The intermediate feature map and the input feature map are then cached to the next layer of cache area. The multiple cache areas are traversed until the output feature map is obtained. If the size is less than the size threshold or the number of weight data is greater than the number threshold, the SRAM is set to storage mode, wherein the SRAM is divided into two storage areas in storage mode; the input feature map is written into one of the storage areas and multiplied and accumulated with the weight data to obtain an intermediate feature map, and the intermediate feature map is written into the other storage area; the calculation is performed alternately in the two storage areas until the output feature map is obtained.

Images