A hardware accelerator and acceleration method for convolutional neural networks

Abstract

This invention discloses a convolutional neural network hardware accelerator, comprising: a memory that stores input feature map data in an input channel-first order; a feature map input module that reads input feature map data in an input channel-first reading order; a feature map caching module that caches the input feature map data in an input channel-first storage order; a convolution kernel module that stores convolution kernel data in an input channel-first storage order; a computation module that sequentially and in parallel reads M input feature map data and M convolution kernel data in each clock cycle, performs point-to-point multiplication on the input feature map data and the M convolution kernel data, accumulates all the point-to-point multiplication results to obtain the convolution result, and outputs the convolution result in parallel to N output channels as output feature map data; and a feature map output module that writes the output feature map data into the memory in an output channel-first storage order. This invention improves the computational efficiency of convolution operations in the hardware accelerator.

Description

Technical Field

[0001] This invention relates to the field of deep learning hardware acceleration technology, and in particular to a convolutional neural network hardware accelerator and acceleration method. Background Technology

[0002] Convolutional Neural Networks (CNNs) are among the most important algorithms in deep learning, widely used in fields such as object recognition, autonomous driving, and artificial intelligence. Convolution operations constitute the most computationally intensive part of CNNs. As CNNs become increasingly deeper and computationally intensive, with varying data sizes in each layer, improving the computational efficiency and optimizing convolution operations has become a critical issue for hardware accelerators.

[0003] In existing technical solutions, convolution calculation is not flexible enough, cannot efficiently support convolution operations of different sizes, and cannot make full use of the computing resources on hardware accelerators, resulting in low computational efficiency. Furthermore, before the convolution calculation of the next layer, the feature map results of the previous layer need to be converted into an additional format, which requires additional processing time and hardware resources. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a convolutional neural network hardware accelerator and acceleration method, which can improve the overall computational efficiency of convolutional neural networks, implement convolutional neural networks without complex logic, and reduce the consumption of hardware resources.

[0005] To achieve the above objectives, the present invention provides a convolutional neural network hardware accelerator, the accelerator comprising:

[0006] The memory is used to store input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns.

[0007] The feature map input module is used to read input feature map data from the memory in an input channel-first read mode;

[0008] The feature map caching module is used to cache the input feature map data output by the feature map input module in a manner that prioritizes the input channel, and the number of input feature map data written to the feature map caching module in parallel within a single clock cycle is M, where M is an integer greater than or equal to 1;

[0009] The convolution kernel module includes N convolution kernel cache modules, which correspond to N output channels. Each convolution kernel cache module stores convolution kernel data (IC, KH, KW) in a way that prioritizes the input channels. M convolution kernel data are written to the convolution kernel cache module in parallel within a single clock cycle, where N is an integer greater than or equal to 1, KH is the number of convolution kernel rows, and KW is the number of convolution kernel columns.

[0010] The computation module includes N computation units, which are used to sequentially read M input feature map data in parallel from the feature map cache module, take these M input feature map data as a set, copy them into N sets of data, and output them in parallel to the N computation units. It also reads M convolution kernel data in parallel from each convolution kernel cache module and outputs them in parallel to the corresponding computation unit. Each computation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. Each computation unit performs M multiplication operations simultaneously. It takes P clock cycles to read all the data required for a single convolution operation. All multiplication results are summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation, finally outputting output feature map data for N output channels, where P = KH * KW * L. Indicates rounding up;

[0011] The feature map output module is used to write the output feature map data of the N output channels into the memory in a way that prioritizes the output channels, and use the output feature map data of the current layer as the output feature map data after all the output feature map data of the output channels are stored, and use the current layer output feature map data as the input feature map data of the next layer convolution operation.

[0012] Preferably, the feature map input module reads input feature map data from the memory in a channel-priority reading manner. Specifically, it first reads the input feature map data of all input channels in the same row and column, then reads the input feature map data of all input channels in the next row and column, until the input feature data of all input channels in all rows and columns has been read.

[0013] Preferably, the feature map caching module caches the input feature map data according to the input channel priority order, specifically by grouping the data of all input channels in the same row and column into groups of M data points, with the number of groups being... The total number of data groups across all rows and columns is IH*IW*L. Each group of input feature map data is written to the feature map cache module in parallel after grouping.

[0014] Preferably, the storage address of the input feature map data in the feature map caching module includes the following features:

[0015] The storage addresses of the M input feature map data in the same group are the same; the storage addresses of the input feature map data in different rows and columns are different; the storage addresses of the input feature map data in adjacent columns in the same row are continuous; the storage address interval of the input feature map data in the same column in adjacent rows is L.

[0016] Preferably, the convolution kernel caching module caches the convolution kernel data in order of input channel priority, specifically including: grouping the convolution kernel data of all input channels in the same row and column of the same output channel into groups of M data points, with the number of groups being... The total number of data groups in all rows and columns of the same output channel is KH*KW*L. Each group of convolution kernel data is written to the convolution kernel cache module in parallel after grouping.

[0017] Preferably, the storage address of the convolution kernel data in the convolution kernel cache module has the following characteristics: the storage address of the same group of convolution kernel data is the same; the storage addresses of convolution kernel data in adjacent columns of the same row are continuous; and the storage address interval of convolution kernel data in the same column of adjacent rows is L.

[0018] Preferably, when the actual cache depth of the feature map cache module is less than the maximum cache depth required by the input feature map data, the input feature map data cache is read and written in a circular storage manner. When the data is written to the maximum address of the cache, it is rewritten to address 0, and the data written later overwrites the data written earlier and that has already been read.

[0019] Preferably, the features of the calculation module reading the read address of the input feature map data from the feature map cache module include:

[0020] Input feature map data located at P read addresses are continuously read within P clock cycles, where P = KH * KW * L, and the P addresses satisfy the following conditions: each KW * L read address is consecutive; the offset between adjacent KW * L read addresses is IW * L.

[0021] When calculating the next column of the same row in the output feature map data, the values ​​of P read addresses are all increased by sride*L, where sride is the span.

[0022] When calculating the next row of the same column in the output feature map data, the values ​​of all P read addresses are increased by IW*L;

[0023] The values ​​obtained by modulo the P read addresses with the cache depth of the feature map cache module are taken as the actual cache read addresses.

[0024] Preferably, the feature map output module is specifically used for:

[0025] The output feature map data of N output channels are grouped into a set of output feature map data in an adjacent manner. The output feature map data in different rows and columns are output sequentially according to the result output order. All N output channel output feature map data are output in parallel within the same clock cycle. The output feature map data of the N output channels are written into the memory once or multiple times according to the data parallelism of the memory read and write interface.

[0026] To achieve the above objectives, the present invention provides a hardware acceleration method for convolutional neural networks, the method comprising:

[0027] The memory stores the input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns.

[0028] The feature map input module reads input feature map data from the memory in an input channel-first read mode;

[0029] The input feature map data output by the feature map input module is cached in the feature map cache module in a way that prioritizes the input channel, and M input feature map data are written to the feature map cache module in parallel within a single clock cycle, where M is an integer greater than or equal to 1;

[0030] The convolutional kernel data (IC, KH, KW) are stored in N convolutional kernel cache modules in a way that prioritizes the input channels. The N convolutional kernel cache modules correspond to N output channels. In a single clock cycle, M convolutional kernel data are written to the convolutional kernel cache modules in parallel, where N is an integer greater than or equal to 1, KH is the number of convolutional kernel rows, and KW is the number of convolutional kernel columns.

[0031] M input feature map data are sequentially read in parallel from the feature map cache module. These M input feature map data are treated as a set, copied into N sets, and output in parallel to N computation units. Similarly, M convolution kernel data are sequentially read in parallel from each convolution kernel cache module and output in parallel to the corresponding computation unit. Each computation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. A total of P clock cycles are required to read all the data needed for a single convolution operation. All multiplication results are summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation. Finally, N output channel feature map data are output, where P = KH * KW * L. Indicates rounding up;

[0032] The feature map output module writes the output feature map data of the N output channels into the memory in a channel-priority storage manner. After the output feature map data of all output channels is stored, it is used as the current layer output feature map data, and the current layer output feature map data is used as the input feature map data for the next layer convolution operation.

[0033] Compared with existing technologies, the convolutional neural network hardware accelerator and acceleration method of this invention have the following advantages: They improve computational efficiency and speed while maintaining the versatility of convolutional computation, and allow for flexible adjustment of parallelism to adapt to different scales of computing resources; based on this invention, the format and order of output and input feature map data are unified during convolutional operations in front and back layers, eliminating the need for additional conversion of input and output data order between front and back layers; the conversion processing speed between convolutional layers is fast, and the conversion logic is reduced, saving a significant amount of hardware resources and improving computational performance; the storage and retrieval of feature map data can be achieved through simple processing logic, and it is compatible with the overall computational architecture of the convolutional neural network accelerator, saving a significant amount of hardware resources while meeting computational flexibility and parallelism requirements. Attached Figure Description

[0034] Figure 1 This is a system schematic diagram of a convolutional neural network hardware accelerator according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a convolution operation in the prior art.

[0036] Figure 3 This is a schematic diagram illustrating the data dimension transformation process of the input feature map data.

[0037] Figure 4 This is a schematic diagram illustrating the data dimension transformation process of the convolution kernel data.

[0038] Figure 5 This is a schematic diagram of convolutional layers and function layers.

[0039] Figure 6 This is a schematic diagram of the storage of input feature map data in memory according to an embodiment of the present invention.

[0040] Figure 7 This is a schematic diagram of the storage of the input feature map in the input feature map caching module according to an embodiment of the present invention.

[0041] Figure 8 A schematic diagram of the storage of the input feature map in the input feature map caching module according to an embodiment of the present invention.

[0042] Figure 9This is a schematic diagram of the storage of convolution kernel data in the convolution kernel cache module according to an embodiment of the present invention.

[0043] Figure 10 This is a schematic diagram illustrating the output order of data read from the feature map cache module and the convolution kernel cache module according to an embodiment of the present invention.

[0044] Figure 11 This is a schematic diagram showing the output feature map in row 0 and column 0 according to an embodiment of the present invention, displayed in three dimensions.

[0045] Figure 12 This is a schematic diagram of an input feature map caching module with circular dynamic storage according to an embodiment of the present invention.

[0046] Figure 13 This is a schematic diagram of the output sequence of the output feature map results according to an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0048] like Figure 1 As shown in one embodiment of the present invention, the present invention provides a convolutional neural network hardware accelerator, the accelerator comprising:

[0049] The memory 10 is used to store input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns;

[0050] Feature map input module 11 is used to read input feature map data from the memory in an input channel priority reading mode;

[0051] The feature map caching module 12 is used to cache the input feature map data output by the feature map input module in a way that prioritizes the input channel, where M is an integer greater than or equal to 1;

[0052] The convolution kernel module 13 includes N convolution kernel cache modules, which correspond to N output channels. Each convolution kernel cache module is used to store convolution kernel data (IC, KH, KW) in a way that prioritizes the input channel. In a single clock cycle, M convolution kernel data are written to the convolution kernel cache module in parallel, where N is an integer greater than or equal to 1, KH is the number of convolution kernel rows, and KW is the number of convolution kernel columns.

[0053] The calculation module 14 includes N calculation units, which are used to sequentially read M input feature map data in parallel from the feature map cache module, take these M input feature map data as a set of data, copy them into N sets of data, and output them in parallel to the N calculation units. It also reads M convolution kernel data in parallel from each convolution kernel cache module and outputs them in parallel to the corresponding calculation unit. Each calculation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. Each calculation unit performs M multiplication operations simultaneously. It takes P clock cycles to read all the data required for a single convolution operation. All multiplication results are summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation, finally outputting output feature map data for N output channels, where P = KH * KW * L. Indicates rounding up;

[0054] The feature map output module 15 is used to write the output feature map data of the N output channels into the memory in a way that prioritizes the output channels, and use the output feature map data of the current layer as the current layer output feature map data after all the output feature map data of the output channels are stored, and use the current layer output feature map data as the input feature map data of the next layer convolution operation.

[0055] Before performing convolution operations, the input feature map is sometimes padded to accommodate data size adjustments. The input feature map data size is padded from (IC, IH, IW) to (IC, IH + 2*pad, IW + 2*pad), where IC is the number of input channels, IH is the input height (number of rows), IW is the input width (number of columns), and Pad is the padding value. The convolution operation process is as follows: Figure 2 As shown, the convolutional kernel data (IC, KH, KW) represents the number of input channels, KH the kernel height (i.e., the number of kernel rows), and KW the kernel width (i.e., the number of kernel columns). Data corresponding to the convolutional kernel data of a single output channel is selected from the padded input feature map data (IC, IH+2*pad, IW+2*pad). Figure 2 The data within the gray area of ​​the filled feature map is multiplied point-to-point with the corresponding points of the convolution kernel data for different output channels and then summed to obtain a result data for each channel in the output feature map. Figure 2The gray part in the output feature map is used to obtain a data point in row 0 and column 0 of the output channel of the output feature map through convolution operation. Then, the gray part in the filled feature map is offset by stride in the row direction. The above calculation process is repeated. After the calculation of a row is completed, the stride is offset in the column direction and the calculation of a row is performed. By repeating the above process, the output feature map result data of all row and column positions can be obtained.

[0056] The data dimension transformation process of the input feature map data and the data dimension transformation process of the convolution kernel will now be explained to facilitate a better understanding of this invention. For example... Figure 3 The data dimension transformation process shown in the diagram transforms the input feature map data from three dimensions (F0(IC,IH,IW)) to two dimensions (F1(IH*IW,IC)) participating in the convolution operation. Then, the two-dimensional input feature map F1(IH*IW,IC) is divided into L two-dimensional input feature maps F2(IH*IW,M). M represents the parallelism of a single output channel, where M is an integer greater than or equal to 1. When IC is not divisible by M, the size of the effective data for the last (L-1)th two-dimensional feature map is (IH*IW, S), where S = IC % M. Each row of this two-dimensional feature map needs to be padded with zeros to expand its width from S to M. The L two-dimensional input feature maps F2(IH*IW, M) are then recombine into a single two-dimensional feature map F3(IH*IW*L, M). For example... Figure 4 The data dimension transformation process of the convolutional kernel shown is similar to that of the input feature map. Taking one output channel as an example, one of the OC output channels participating in the convolution operation—a 3D convolutional kernel K0(IC,KH,KW)—is completely transformed from 3D to 2D convolutional kernel K1(KH*KW,IC), where KH and KW represent the height and width of the kernel. Then, the 2D convolutional kernel K1(KH*KW,IC) is divided into L 2D convolutional kernels K2(KH*KW,M). The effective data size of the last, L-1th, 2D feature map is (KH*KW,S), where S = IC % M. Each row of this 2D feature map needs to be padded with 0 values ​​to expand its width from S to M. Finally, the L 2D convolutional kernels K2(KH*KW,M) are recombine into a single 2D convolutional kernel K3(P,M), where P = KH*KW*L. This invention is based on such a data dimension transformation. It only requires sequentially and in parallel extracting M convolution kernel data from the two-dimensional convolution kernel K3(P,M) and simultaneously sequentially and in parallel extracting the corresponding M input feature map data from the two-dimensional input feature map F3(IH*IW*L,M). The M convolution kernel data and the M input feature map data are then multiplied point-to-point, and this parallel extraction and point-to-point multiplication is performed P times. All multiplication results are then accumulated to obtain the convolution operation result for a single output channel. Similarly, this can be extended to the convolution operation of all OC output channels to complete the entire convolution operation. The above method allows convolution operations to be easily adapted to hardware circuits and can fully leverage the advantages of parallel computing in hardware circuits. However, actual hardware resources are limited, and it is generally impossible to calculate all output channels simultaneously. Therefore, N output channels can be calculated simultaneously in each round, and the convolution operation of all output channels can be completed by calculating Q rounds, where 1≤N≤OC. The principles of the present invention have been explained above.

[0057] Specifically, this invention provides a convolutional neural network hardware accelerator, which includes a memory, a feature map input module, a feature map cache module, a convolutional kernel module, a computation module, and a feature map output module. The memory stores input feature map data (IC, IH, IW) in an input channel-first manner, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns. The memory can be any type of random access memory, either on-chip or off-chip, such as SRAM or DDR SDRAM. The input feature map data is three-dimensional data with three dimensions: input channels, rows, and columns. It is stored in the memory in an input channel-first manner; for example, the data of all input channels at a certain row and column position is stored first, then the data of all input channels at the next row and column position, and so on. The input feature map data is stored in the memory in this order according to input channel priority, so that the input feature map data of different input channels in the same row and column are located at adjacent addresses in the memory.

[0058] The feature map input module reads input feature map data from the memory into the feature map cache module in an input channel-first read mode. The parallelism of input feature map data output within the same clock cycle is M, meaning M data points are read from the memory in parallel within the same clock cycle, where M is an integer greater than or equal to 1. Specifically, the feature map input module reads input feature map data from the memory in an input channel-first read mode: first, it reads the input feature map data of all input channels in the same row and column, then it reads the input feature map data of all input channels in the next row and column, and so on, until the input feature map data of all input channels in all rows and columns has been read. This includes two reading methods: one is reading in the order of input channel, input column, and input row; the other is reading in the order of input channel, input row, and input column. The reading order, specifically according to input channel, input column, and input row, involves: first reading the input feature map data of all input channels in the same row and column; then reading the input feature map data of all input channels in the next column of the same row; and so on, until the input feature map data of all input channels in all rows and columns has been read. During the data reading process, M input feature data are read in parallel within the same clock cycle, meaning the data of M input channels in the current row and column are read in parallel at one time.

[0059] The feature map caching module caches the input feature map data output by the feature map input module in a priority manner, prioritizing the input channel. Within a single clock cycle, M pieces of input feature map data are written to the feature map caching module in parallel. The cache depth of the feature map caching module is at least P, where P = KH * KW * L. This indicates rounding up, supporting parallel input and output of M input feature map data. Similar to the above embodiment, the input feature map data is stored in the feature map cache module with input channels as the priority, specifically in two ways: one is to store the data sequentially in the order of input channel, input column, and input row; the other is to store it sequentially in the order of input channel, input row, and input column. The order of storing in the order of input channel, input column, and input row specifically includes: first storing the input feature map data containing all input channels in the same row and column, then storing the input feature map data containing all input channels in the next column of the same row. Once the input feature map data for all input channels in all columns of the same row is stored, the same order is followed to store the input feature map data for all input channels in all columns of the next row, until the input feature map data for all input channels in all rows and all columns is stored. The storage order, following the reading sequence of input channels, input rows, and input columns, specifically includes: first storing the input feature map data for all input channels in the same column and row; then storing the input feature map data for all input channels in the next row of the same column; once the input feature map data for all input channels in all rows of the same column is stored, the same order is repeated for the input feature map data for all input channels in all rows of the next column, until the input feature map data for all input channels in all columns and rows is stored. During data storage, M input feature data are written in parallel within the same clock cycle, meaning M input channels in the current row and column are written in parallel at once. Generally, the data stored first is stored at the lower address of the feature map cache module, and the data stored later is stored at the higher address of the feature map cache module.

[0060] Input feature map data is written to the feature map cache module in order of input channel priority. If the number of data points (IC) of all input channels in the same row and column of the input feature map data is less than or equal to the number of input feature map data points (M) written in parallel within a single clock cycle (i.e., less than or equal to the maximum number of data points (M) that the feature map cache module can support for parallel writing), then the data points of all input channels in the same row and column of the input feature map data are grouped together and written to the feature map cache module in parallel, with the write address corresponding to each group of input feature map data incrementing sequentially. If the number of data points (M) of all input channels in the same row and column of the input feature map data is greater than the number of input feature map data points (M) written in parallel within a single clock cycle, then the data points of all input channels in the same row and column are grouped into groups of M, with the number of groups being [missing information]. in This indicates rounding up. The total number of data groups across all rows and columns is IH*IW*L. Each group of input feature map data is written to the feature map cache module in parallel. Generally, the write address corresponding to each group of input feature map data is incremented sequentially.

[0061] According to a specific embodiment of the present invention, the storage address of the input feature map data in the feature map caching module has the following characteristics: the storage addresses of the M input feature map data in the same group are the same; the storage addresses of the input feature map data in different rows and columns are different; the storage addresses of the input feature map data in adjacent columns in the same row are continuous; and the storage address interval of the input feature map data in the same column of adjacent rows is L. If IC is not divisible by M, then the number S of the last group of data in the same row and column must be less than M, and S = IC % M. When the cache depth of the feature map caching module is less than the cache depth required by the input feature map data, the input feature map data cache is read and written in a circular storage manner, that is, when the data is written to the maximum address of the cache, it is rewritten to address 0, and the data written later overwrites the data written earlier. The writing of data is controlled by a dedicated cache controller to ensure that the overwritten data has been used and the corresponding cache space can be released.

[0062] The convolutional kernel module includes N convolutional kernel cache modules, each corresponding to one of the N output channels. N is an integer greater than or equal to 1. Each kernel cache module stores convolutional kernel data (IC, KH, KW) in a channel-first manner, where IC is the number of input channels, KH is the number of kernel rows, and KW is the number of kernel columns. It supports parallel input and output of M convolutional kernel data. The N kernel cache modules store the N convolutional kernel data required for the current convolution operation. N can be set to the same number as the number of output channels in the output feature map data, or it can be set less than the number of output channels. When set less, Q rounds of computation are required to obtain the output feature map data for all output channels. During each round of computation, each kernel cache module stores the convolutional kernel data for the corresponding output channel and outputs the data to the corresponding computation unit. Each convolutional kernel buffer module stores data in input channel priority order, writing convolutional kernel data from different input channels sequentially into the convolutional kernel buffer module. It supports parallel writing of M convolutional kernel data within a single clock cycle. If the number of convolutional kernel data (IC) for all input channels in the same row and column is less than or equal to the maximum number of data items (M) that the convolutional kernel buffer module can support for parallel writing, then the convolutional kernel data for all input channels in the same row and column are written as a group into the convolutional kernel buffer module, with the write address for each group of convolutional kernel data incrementing sequentially. If the number of convolutional kernel data for all input channels in the same row and column is greater than M, the number of groups is... in The method involves rounding up and sequentially writing each group of convolutional kernel data into the convolutional kernel cache module in parallel, with the write address corresponding to each group of convolutional kernel data being written sequentially increasing. The number of addresses occupied by the convolutional kernel data on a single output channel within the convolutional kernel cache module is P. This scheme ensures that the convolutional kernel data output from the convolutional kernel cache module corresponds one-to-one with the input feature map data used in the computation. The characteristics of the storage addresses of the convolutional kernel data in the convolutional kernel cache module include: the storage addresses of the same M convolutional kernel data in the same group are the same; the storage addresses of convolutional kernel data in adjacent columns of the same row are continuous; and the storage address interval of convolutional kernel data in the same column of adjacent rows is L.

[0063] The computation module comprises N computation units. During convolution operations, M input feature map data are read in parallel from the feature map cache module within a single clock cycle and output to the N computation units respectively. Similarly, M convolution kernel data are read in parallel from each convolution kernel cache module and output to the corresponding computation unit. Each computation unit can perform M multiplications simultaneously, typically consisting of M multipliers, and supports the accumulation of the M multiplication results. Within a single clock cycle, the M input feature map data and M convolution kernel data are multiplied point-to-point in the corresponding multipliers. After P clock cycles, the results of all point-to-point multiplications are accumulated to obtain the N convolution calculation results, corresponding to the convolution operation results of the N output channels. If N is less than the number of output channels, Q rounds of computation are required to obtain the output feature map data for all output channels. The accelerator also includes a function module 16, which comprises N output channel function operation modules, corresponding to the function operation modules for the N output channels. The results of N convolution operations are processed by corresponding function operation modules to obtain output feature map data for N output channels. A convolutional neural network typically contains multiple convolutional layers and function layers. The final output feature map result needs to be obtained after sequentially performing calculations on all layers for subsequent processing. Function layers generally contain sub-layers such as batchnorm, scaling, ReLU, and pooling. Figure 5 As shown. In hardware implementation, specific modules implement the corresponding functions. Each module can perform corresponding function operations on the data from N output channels in parallel. The convolutional layer and function layer are considered as a large layer in the convolutional neural network, with the convolutional layer as the starting point of this large layer. The output feature map of the previous large layer becomes the input feature map of the next large layer. Since the function module follows the convolutional module in hardware implementation, the results of the convolutional operations can be directly piped, thus the processing time of the function operations is almost negligible.

[0064] According to a specific embodiment of the present invention, the feature of the calculation module reading the read address of the input feature map data from the feature map cache module includes: continuously reading P read addresses of input feature map data within the same clock cycle, where P = KH * KW * L, and the P addresses satisfying the following conditions: each KW * L read address is consecutive; the offset between adjacent KW * L read addresses is IW * L; when calculating the next column of the same row in the output feature map data, the values ​​of the P read addresses all increase by sride * L, where sride is the span; when calculating the same column of the next row in the output feature map data, the values ​​of the P read addresses all increase by IW * L, and the value of the P read addresses modulo the cache depth of the feature map cache module is taken as the actual cache read address. The calculation module reading convolution kernel data from the convolution kernel cache module specifically includes: the read addresses of the convolution kernel cache modules of different output channels are the same within the same clock cycle; the read addresses increment during one convolution calculation; and the convolution kernel data required for the calculation is sequentially output to the corresponding calculation unit. When calculating the output feature map results on different rows and columns of the same output channel, the convolution kernel buffer traverses the same P addresses each time. When it is necessary to read the data of the input feature map padding, the output data of the input feature map buffer is replaced with padding data (usually 0) and output to the computation unit. Alternatively, the reading of the input feature map padding data and the corresponding convolution kernel data can be skipped during computation, thus directly skipping the multiplication operation between the two. However, this does not affect the result of the current convolution operation because the values ​​of all channels in the input feature map padding are 0, and the result after multiplying with convolution kernel data of any value is also 0. Skipping the reading and computation of the input feature map padding data during computation will reduce the number of clock cycles used for the current convolution operation, thereby improving the performance of the convolution operation.

[0065] A specific embodiment of the present invention, such as Figure 6 The diagram shows the storage of input feature map data in memory. The transformed input feature map F1 (IC=16, IH=IW=5) is arranged in the storage order of input channel priority, input column, and input row (HWC). X_Y_Z within the box represents the data at position Y row and Z column of channel X. The actual storage method of input feature map F1 in the input feature map cache module according to input channel priority (M=8, IC=16, IH=IW=5, KH=KW=3) is as follows. Figure 7 As shown in the figure, the dark gray area represents the input feature map data required for the current calculation (the output feature map result calculated at row 0, column 0), and the white area represents the input feature map data not currently involved in the calculation. The numbers within the boxes represent the data for the corresponding channels. The coordinate positions HxWy within the corresponding boxes represent the x-row, y-column positions of all channels' data within the corresponding cache address. Figure 8This demonstrates the input feature map data that participates in the computation when calculating the next output feature map result (0 rows and 1 column). In other words, in this storage method, the sliding of the selection window for the input feature map data corresponding to the convolutional kernel data participating in the computation is converted into the offset of the input feature map cache address. Figure 9 This diagram illustrates the channel-priority storage of convolutional kernel data in the kernel cache module (M=8, IC=16, KH=KW=3). The numbers within the boxes represent the data for the corresponding channels. The coordinates within the boxes (HxWy) represent the x-row and y-column positions of all channels' data within the corresponding cache address. When calculating the results of different rows and columns of the output feature map within the same output channel, the kernel cache outputs the kernel data sequentially according to their address order. The output order of data read from the feature map cache module and the kernel cache module is as follows: Figure 10 As shown, the input feature map data and convolution kernel data under the same clock cycle are multiplied correspondingly in the multiplier of the computing unit. The multiplication results between multipliers can be added in parallel or in cascades according to the actual implementation of the computing unit. The multiplication and addition results generated in the preceding and following clock cycles are accumulated. Finally, after all clock cycles of one convolution operation, all data points are multiplied point by point and accumulated to obtain the convolution operation result. After function operation, the result at one row and column position of the output feature map is obtained. Figure 11 This is a 3D diagram showing the result of calculating the output feature map (row 0, column 0). The gray areas of the input feature map and convolution kernel represent the data participating in the convolution operation. Figure 10 The data in the gray area corresponds to this.

[0066] Figure 12 This demonstrates an implementation of an input feature map caching module that uses a circular storage method to dynamically update data. Theoretically, the input feature map cache only needs to store the number of rows required for the current computation to begin convolution operations. For example, when pad is 0 and KH is 3, only 3 rows need to be stored. Generally, the number of rows that the input feature map cache can store is greater than the number of rows required for computation. The cache depth shown in the diagram can store 4 rows of input feature map data, thus allowing simultaneous data reading for computation and data writing for updates. When input feature map data has already been used for computation, the cache space for that row is released, and subsequently written feature map data overwrites the previously written data. This method avoids storing all input feature map data in the on-chip cache, saving significant storage resources.

[0067] The feature map output module writes the output feature map data of the N output channels into the memory in a channel-first storage manner. Once the output feature map data of all output channels is stored, it becomes the current layer's output feature map data, which is then used as the input feature map data for the next layer's convolution operation. N computation units output N convolution operation results, which, after being processed by N function operation modules, output the output feature map data of the N output channels in parallel. The output feature map data of the N output channels are grouped according to their adjacent channels, and the output feature map data in different rows and columns are output sequentially according to the result output order. The output feature map data of the N output channels is output within the same clock cycle, and the output feature map data of the N output channels is written into the memory once or multiple times according to the memory's parallelism. Figure 13 The diagram illustrates the output order and merging method of the output feature map results when N=8, OH=OW=3. When the number of computation units N is greater than or equal to the number of output channels OC, all computations in this layer are completed after all current computation units output all feature map data sequentially. The merged output feature map data is then stored in memory according to the output channel priority storage method. When the number of computation units N is less than the number of output channels OC, after N computation units complete a single convolution operation, only a portion of the convolution operation results are completed, outputting N output channels' output feature map data. There are still OH*OW*(OC-N) results that have not been calculated. Therefore, it is necessary to update the convolution kernel data of the remaining output channels to the convolution kernel cache module, reread the input feature map data, and sequentially calculate the output feature map data of the remaining output channels. The output parallelism of the output feature map data is N, similar to the input of the input feature map data. Depending on the data parallelism of the memory read / write interface, the N output feature map data can be written to memory in one or multiple batches. Generally, M and N are powers of 2, and the bit width of a single output feature map is also typically a power of 2. The memory read / write interface bit width is also typically a power of 2. Therefore, this type of data interface conversion can be implemented with fewer logic resources. Furthermore, since both reading input feature map data from memory and writing output feature map data follow a channel-first approach, the format and order of output and input feature map data are unified during convolutional operations in both layers. Feature map data writing and reading can be achieved through simple processing logic, and the system is compatible with the entire convolutional neural network accelerator architecture, saving significant hardware resources while maintaining computational flexibility and parallelism.

[0068] Without considering the impact of data input / output bandwidth, when the operating clock frequency is the same, the computing power of a convolutional neural network hardware accelerator depends on the value of M*N; the larger M and the larger N are, the stronger the performance. In practical applications, the maximum values ​​of M and N are limited by the quantity of hardware resources such as multipliers, caches, and storage resources. Furthermore, in some application areas, performance is not the only metric pursued. Therefore, the values ​​of M and N can be flexibly adjusted according to actual performance requirements and data input / output bandwidth to adapt the hardware accelerator to different sizes of hardware resources, optimize computational efficiency as much as possible, and minimize the proportion of idle resources and idle time of computing units during convolutional neural network operations.

[0069] According to an embodiment of the present invention, the present invention provides a hardware acceleration method for convolutional neural networks, the method comprising:

[0070] The memory stores the input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns.

[0071] The feature map input module reads input feature map data from the memory in an input channel-first read mode;

[0072] The input feature map data output by the feature map input module is cached in the feature map cache module in a way that prioritizes the input channel, and M input feature map data are written to the feature map cache module in parallel within a single clock cycle, where M is an integer greater than or equal to 1;

[0073] The convolutional kernel data (IC, KH, KW) are stored in N convolutional kernel cache modules in a way that prioritizes the input channels. The N convolutional kernel cache modules correspond to N output channels. In a single clock cycle, M convolutional kernel data are written to the convolutional kernel cache modules in parallel, where N is an integer greater than or equal to 1, KH is the number of convolutional kernel rows, and KW is the number of convolutional kernel columns.

[0074] M input feature map data are sequentially read in parallel from the feature map cache module. These M input feature map data are treated as a set, copied into N sets, and output in parallel to N computation units. Similarly, M convolution kernel data are sequentially read in parallel from each convolution kernel cache module and output in parallel to the corresponding computation unit. Each computation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. A total of P clock cycles are required to read all the data needed for a single convolution operation. The multiplication results are then summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation. Finally, N output channel feature map data are output, where P = KH * KW * L. Indicates rounding up;

[0075] The feature map output module writes the output feature map data of the N output channels into the memory in a channel-priority storage manner. After the output feature map data of all output channels is stored, it is used as the current layer output feature map data, and the current layer output feature map data is used as the input feature map data for the next layer convolution operation.

[0076] Although preferred embodiments of the invention have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of the invention as disclosed in the appended claims.

Claims

1. A convolutional neural network hardware accelerator, characterized in that, The hardware accelerator includes: The memory is used to store input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns. The feature map input module is used to read input feature map data from the memory in an input channel-first read mode; The feature map caching module is used to cache the input feature map data output by the feature map input module in a manner that prioritizes the input channel, and the number of input feature map data written to the feature map caching module in parallel within a single clock cycle is M, where M is an integer greater than or equal to 1; The convolution kernel module includes N convolution kernel cache modules, which correspond to N output channels. Each convolution kernel cache module stores convolution kernel data (IC, KH, KW) in a way that prioritizes the input channels. M convolution kernel data are written to the convolution kernel cache module in parallel within a single clock cycle, where N is an integer greater than or equal to 1, KH is the number of convolution kernel rows, and KW is the number of convolution kernel columns. The computation module includes N computation units, which are used to sequentially read M input feature map data in parallel from the feature map cache module, copy the M input feature map data into N sets of input feature map data, and output them in parallel to the N computation units. It also reads M convolution kernel data in parallel from each convolution kernel cache module and outputs them in parallel to the corresponding computation unit. Each computation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. Each computation unit performs M multiplication operations simultaneously. It takes P clock cycles to read all the data required for a single convolution operation. All multiplication results are summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation, ultimately outputting output feature map data for N output channels, where P = KH * KW * L. Indicates rounding up; The feature map output module is used to write the output feature map data of the N output channels into the memory in a way that prioritizes the output channels, and use the output feature map data of the current layer as the output feature map data after all the output feature map data of the output channels are stored, and use the current layer output feature map data as the input feature map data of the next layer convolution operation.

2. The convolutional neural network hardware accelerator as described in claim 1, characterized in that, The feature map input module reads input feature map data from the memory in a channel-priority reading manner. Specifically, it first reads the input feature map data of all input channels in the same row and column, then reads the input feature map data of all input channels in the next row and column, until the input feature data of all input channels in all rows and columns has been read.

3. The convolutional neural network hardware accelerator as described in claim 2, characterized in that, The feature map caching module caches the input feature map data according to the order of input channel priority, specifically including: Group the data from all input channels in the same row and column into groups of M data points each, with the number of groups being M. The total number of data groups across all rows and columns is IH*IW*L. Each group of input feature map data is written to the feature map cache module in parallel after grouping.

4. The convolutional neural network hardware accelerator as described in claim 3, characterized in that, The features of the storage address of the input feature map data in the feature map caching module include: The storage addresses of the M input feature map data in the same group are the same; the storage addresses of the input feature map data in different rows and columns are different; the storage addresses of the input feature map data in adjacent columns in the same row are continuous; the storage address interval of the input feature map data in the same column in adjacent rows is L.

5. The convolutional neural network hardware accelerator as described in claim 4, characterized in that, The convolution kernel caching module caches the convolution kernel data in order of input channel priority. Specifically, it includes grouping the convolution kernel data of all input channels in the same row and column of the same output channel into groups of M data points each, with the number of groups being... The total number of data groups in all rows and columns of the same output channel is KH*KW*L. Each group of convolution kernel data is written to the convolution kernel cache module in parallel after grouping.

6. The convolutional neural network hardware accelerator as described in claim 5, characterized in that, The characteristics of the storage address of the convolution kernel data in the convolution kernel cache module include: The storage addresses of the M convolutional kernel data in the same group are the same; the storage addresses of the convolutional kernel data in adjacent columns of the same row are continuous; the storage address interval of the convolutional kernel data in the same column of adjacent rows is L.

7. The convolutional neural network hardware accelerator as described in claim 6, characterized in that, When the actual cache depth of the feature map caching module is less than the maximum cache depth required by the input feature map data, the input feature map data cache is read and written in a circular storage manner. When the data is written to the maximum address of the cache, it is rewritten to address 0. The data written later overwrites the data written earlier and that has already been read.

8. The convolutional neural network hardware accelerator as described in claim 6, characterized in that, The features of the read address from which the calculation module reads the input feature map data from the feature map caching module include: Input feature map data located at P read addresses are continuously read within P clock cycles, where P = KH * KW * L, and the P read addresses satisfy the following conditions: each KW * L read address is consecutive; the offset between adjacent KW * L read addresses is IW * L. When calculating the next column of the same row in the output feature map data, the values ​​of P read addresses are all increased by sride*L, where sride is the span. When calculating the next row and the same column in the output feature map data, the values ​​of all P read addresses are increased by IW*L; the values ​​of the P read addresses modulo the feature map cache module cache depth are taken as the actual cache read addresses.

9. The convolutional neural network hardware accelerator as described in claim 8, characterized in that, The feature map output module is specifically used for: The output feature map data of N output channels are grouped into a set of output feature map data in an adjacent manner. The output feature map data in different rows and columns are output sequentially according to the result output order. All N output channel output feature map data are output in parallel within the same clock cycle. The output feature map data of the N output channels are written into the memory once or multiple times according to the data parallelism of the memory read and write interface.

10. A method for accelerating a convolutional neural network hardware accelerator as described in any one of claims 1-9, characterized in that, The method includes: The memory stores the input feature map data (IC, IH, IW) in the order of input channel priority, where IC is the number of input channels, IH is the number of input rows, and IW is the number of input columns. The feature map input module reads input feature map data from the memory in an input channel-first read mode; The input feature map data output by the feature map input module is cached in the feature map cache module in a way that prioritizes the input channel, and M input feature map data are written to the feature map cache module in parallel within a single clock cycle, where M is an integer greater than or equal to 1; The convolutional kernel data (IC, KH, KW) are stored in N convolutional kernel buffer modules in a way that prioritizes the input channels. The N convolutional kernel buffer modules correspond to N output channels. In a single clock cycle, M convolutional kernel data are written to the convolutional kernel buffer modules in parallel, where N is an integer greater than or equal to 1, KH is the number of convolutional kernel rows, and KW is the number of convolutional kernel columns. M input feature map data are sequentially read in parallel from the feature map cache module. These M input feature map data are treated as a set, copied into N sets, and output in parallel to N computation units. Similarly, M convolution kernel data are sequentially read in parallel from each convolution kernel cache module and output in parallel to the corresponding computation unit. Each computation unit performs point-to-point multiplication of the M input feature map data and the M convolution kernel data. Every clock cycle, a new set of M input feature map data and the M convolution kernel data are used for point-to-point multiplication. A total of P clock cycles are required to read all the data needed for a single convolution operation. All multiplication results are summed to obtain the convolution result. The N convolution results are output in parallel to the function module based on output channel processing for computation. Finally, N output channel feature map data are output, where P = KH * KW * L. Indicates rounding up; The feature map output module writes the output feature map data of the N output channels into the memory in a channel-priority storage manner. After the output feature map data of all output channels is stored, it is used as the current layer output feature map data, and the current layer output feature map data is used as the input feature map data for the next layer convolution operation.

Images