A CNN accelerator data access method and system

Abstract

The present invention relates to a CNN accelerator data access method and system, belonging to the technical field of calculation and inference, comprising: presetting training parameters, obtaining input feature maps and storing them in a buffer area; judging and processing the input feature maps in the buffer area and obtaining pre-fetched data through register array processing, and looping the pre-fetched data to obtain post-order data; performing convolution calculation processing on the post-order data to obtain convolution data; performing batch normalization, RELU activation function activation, quantization, pooling and FIFO cache processing on the convolution data in sequence to obtain output results; judging and processing the number of processing layers of the output results, thereby realizing full on-chip storage and obtaining target coordinates; and effectively reducing memory bandwidth and power consumption.

Description

Technical Field

[0001] The present invention relates to a CNN accelerator data access method and system, belonging to the technical field of calculation and inference. Background Art

[0002] In recent years, deep learning has greatly promoted the development of machine learning. Convolutional Neural Network (CNN) is a deep feedforward neural network with local connection, weight sharing and convergence characteristics, which is an excellent learning model in deep learning. It is widely used in many application fields such as image information and voice information due to its extremely advanced performance. Compared with traditional methods, it requires more computing and memory resources while achieving high-precision and real-time target recognition capabilities, which must rely on large servers. General-purpose processors can no longer meet its needs. The development of the CNN algorithm application industry can be promoted by accelerating the computing process through underlying applications combined with the optimization of deep learning algorithms.

[0003] At present, embedded mobile terminals are facing the challenges of limited on-chip resources, high performance and low power consumption. CNN networks have a large number of parameters, most of which are stored in external memory. Memory bandwidth has become a serious problem in accelerating CNN networks. For large multiplication and addition operations and complex control signals, in-depth research on data flow and computing mode is needed to maximize the application of on-chip resources. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a CNN accelerator data access method and system that can effectively reduce memory bandwidth and power consumption.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a CNN accelerator data access method, comprising:

[0007] Preset training parameters, obtain input feature maps and store them in the buffer;

[0008] The input feature map in the buffer area is judged and processed to obtain pre-fetched data through the register array, and the pre-fetched data is cyclically processed to obtain post-sequence data;

[0009] Perform convolution calculation on the sequenced data to obtain convolution data;

[0010] The convolution data is processed in turn by batch normalization, RELU activation function activation, quantization, pooling and FIFO buffering to obtain the output result;

[0011] The number of processing layers of the output result is judged and processed, so as to achieve full on-chip storage and obtain the target coordinates.

[0012] Furthermore, the training parameters include the size and weight of the input feature map.

[0013] Furthermore, the buffer area stores input feature maps of different channels.

[0014] Furthermore, the method for determining and processing the input feature map in the buffer area includes:

[0015] Calculate the input feature map address according to the input feature map in the buffer area, and determine whether the input feature map address is an edge of the input feature map;

[0016] If the judgment result is yes, the input feature map in the buffer area is padded with zeros, and the input feature map after the zero padding is processed by the register array;

[0017] If the judgment result is no, the input feature map in the buffer area is directly processed by the register array.

[0018] Furthermore, the method for calculating the input feature map address according to the input feature map in the buffer area includes:

[0019]

[0020] Among them, address is the input feature map address, channel is the channel offset address, is the cache partition of the feature map, [w] is the feature Figure 1 Row pixel value And take the value of the nearest power of 2 upwards, group is the group offset address, and row is the row offset address.

[0021] Further, the method of obtaining pre-fetched data by register array processing includes:

[0022] Starting from the zeroth address of the input feature map of all channels in the buffer area, take the first M rows, the M+1th row and the M+2th row for three cycles to complete the pre-fetching of the input feature map of (M+2) rows × 3 columns;

[0023] The third address of the input feature map after pre-fetching all channels in the cache area is taken (M+2) rows×3 columns and then three cycles are performed to obtain the pre-fetched data.

[0024] Furthermore, the method of looping the pre-fetched data to obtain the post-sequence data includes:

[0025] The last row of pre-fetched data output by the register array is used as the first row of data, and the remaining rows of data output by the register array are used as the second to M rows of data in sequence;

[0026] Directly output the pre-fetched data output from the register array;

[0027] The first row of pre-fetched data output by the register array is used as the last row of data, and the remaining rows of data output by the register array are used as the first to (M-1) rows of data in sequence.

[0028] The method for determining the number of processing layers of the output result includes:

[0029] The number of processing layers of the output result obtained by the first calculation is recorded as 1, and the number of processing layers is judged;

[0030] If the number of processing layers of the output result is less than the number of weight layers, the output result is used as the input feature map of the next layer for secondary calculation. The number of processing layers of the output result obtained after the secondary calculation is the number of processing layers obtained by the previous calculation plus 1 through the layer counter;

[0031] If the number of processing layers of the output result is equal to the number of weight layers, the output result is output as the target coordinates.

[0032] In a second aspect, the present invention provides a CNN accelerator data access system, comprising:

[0033] Data cache module: used to preset training parameters, obtain input feature maps and store them in the cache area;

[0034] Data preprocessing module: used to judge and process the input feature map in the buffer area and obtain pre-fetched data through the register array, and cyclically process the pre-fetched data to obtain post-sequence data;

[0035] Convolution calculation module: used to perform convolution calculation on the sequenced data to obtain convolution data;

[0036] Data post-processing module: used to perform batch normalization, RELU activation function activation, quantization, pooling and FIFO buffering on the convolution data in sequence to obtain the output result;

[0037] Output judgment module: judge the number of processing layers of the output result to achieve full on-chip storage and obtain the target coordinates.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. By judging and processing the input feature map in the cache area, the memory bandwidth and power consumption can be effectively reduced, thereby improving the running speed of the CNN accelerator; the input feature map after judgment and processing is processed through the register array to obtain pre-fetched data, and then the pre-fetched data is cyclically processed to obtain post-sequence data. Each cycle realizes the reuse of M-1 rows of input feature maps of the previous cycle, further improving the running speed of the CNN accelerator.

[0040] 2. By judging the number of processing layers of the output results, full on-chip storage is achieved, which effectively reduces the high transmission bandwidth pressure between off-chip data interactions and reduces local storage overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a CNN accelerator data access method provided by an embodiment of the present invention.

[0042] Figure 2 A schematic diagram of a cache area in a CNN accelerator data storage system provided in an embodiment of the present invention.

[0043] Figure 3 A flowchart of register array processing in a CNN accelerator data access method provided by an embodiment of the present invention.

[0044] Figure 4 A flowchart of convolution calculation processing in a CNN accelerator data access method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0046] Embodiment 1:

[0047] FPGA has powerful parallel processing capabilities and rich logic resources. For example, DSP can perform multiplication and accumulation operations with higher calculation accuracy, making it very suitable as a CNN implementation platform.

[0048] like Figure 1 As shown, a CNN accelerator data access method provided by an embodiment of the present invention includes steps S1 to S8 as described below.

[0049] S1. Preset training parameters, obtain input feature maps and store them in the buffer area.

[0050] It should be noted that the training parameters include the size and weight of the input feature map. The weight can be obtained by neural network training. The input feature maps of different channels are stored in the buffer area.

[0051] S2: The input feature map in the buffer is judged and processed, and the pre-fetched data is obtained through the register array processing, and the pre-fetched data is cyclically processed to obtain the post-sequence data.

[0052] The process of judging and processing the input feature map in the buffer is as follows:

[0053] The input feature map address is calculated based on the input feature map in the buffer. The calculation process is as follows:

[0054]

[0055] Among them, address is the input feature map address, channel is the channel offset address, is the cache partition of the feature map, [w] is the feature Figure 1 Row pixel value And take the value of the nearest power of 2 upwards, group is the group offset address, and row is the row offset address.

[0056] like Figure 2 The figure shows a schematic diagram of a buffer area in a CNN accelerator data storage system, calculating an input feature map address according to an input feature map in the buffer area, and determining whether the input feature map address is an edge of the input feature map;

[0057] If the judgment result is yes, the input feature map in the cache area is padded with zeros, and the input feature map after the zero padded is processed by the register array; for example: the zero padded (0) at the upper left corner of the input feature map, the row and group in the address are represented by virtual addresses, and the address is invalid at this moment. The input feature map is not taken out from the cache area, but the zero padded (0) data is used as the obtained input feature map, and the pre-fetched data is obtained through the register array processing.

[0058] If the judgment result is no, the input feature map in the buffer area is directly processed by the register array to obtain pre-fetched data.

[0059] like Figure 3 As shown, the register array processes the pre-fetched data in the channel direction, row direction, and column direction, and the process is as follows:

[0060] Starting from the zeroth address of the input feature map of all channels in the cache, take the first M rows, the M+1th row and the M+2th row and perform three cycles to complete the prefetching of the input feature map of (M+2) rows × 3 columns. Specifically, first start from the zeroth address of the first channel in the cache, and need three cycles to complete the prefetching of the feature map data of (M+2) rows × 3 columns and store them into the register array, and take the first M rows, the M+1th row, and the M+2th row respectively. Each cycle requires 3 clock cycles to complete the prefetching of 3 columns respectively, and then switch to the next channel of the cache, and perform the above three cycles until the prefetching of the (M+2) rows × 3 columns of the zeroth address of all channels in the cache is completed.

[0061] After pre-fetching the input feature map of all channels in the cache, the third address is fetched (M+2) rows × 3 columns, and then three cycles are performed to obtain the pre-fetched data. Specifically, after the (M+2) rows × 3 columns of the zeroth address of all channels in the cache are pre-fetched, switch back to the first channel, and pre-fetch (M+2) rows × 3 columns for the third address of the feature map cache, and continue the above three cycles until all input feature maps are fetched to obtain the pre-fetched data.

[0062] The process of looping the pre-fetched data to obtain the post-sequence data is as follows:

[0063] The last row of pre-fetched data output by the register array is used as the first row of data, and the remaining rows of data output by the register array are used as the second to M rows of data in sequence; specifically, for the pre-fetched data obtained in the first cycle, the sorting processing logic uses the last row of pre-fetched data output by the register array as the first row of data, and the remaining rows of data output by the register array are used as the second to M rows of data in sequence, to obtain M rows of sequenced data and perform convolution calculation;

[0064] Directly output the pre-fetched data output by the register array; specifically, for the pre-fetched data obtained in the second cycle, the sorting processing logic directly performs convolution calculation on the pre-fetched data output by the register array;

[0065] The first row of prefetched data output by the register array is used as the last row of data, and the remaining rows of data output by the register array are used as the first to (M-1) rows of data in sequence; specifically, for the prefetched data obtained in the third cycle, the sorting processing logic uses the first row of prefetched data output by the register array as the last row of data, and the remaining rows of data output by the register array are used as the first to (M-1) rows of data in sequence, to obtain M rows of sequenced data and perform convolution calculations.

[0066] It should be noted that, for the pre-fetched data of the fourth and subsequent cycles, the above-mentioned loop processing is performed in sequence to obtain the post-sequence data, and the post-sequence data is subjected to convolution calculation.

[0067] This embodiment can effectively reduce memory bandwidth and power consumption by performing judgment processing on the input feature map in the cache area, thereby improving the operating speed of the CNN accelerator; the input feature map after judgment processing is processed through the register array to obtain pre-fetched data, and then the pre-fetched data is cyclically processed to obtain post-sequence data. Each cycle realizes the reuse of M-1 rows of input feature maps of the previous cycle, further improving the operating speed of the CNN accelerator.

[0068] S3. Perform convolution calculation on the sequenced data to obtain convolution data.

[0069] Each PE unit works independently, performs multiplication and accumulation on a post-order data and a weight, and uses a register to save the multiplication and accumulation results. The data selector is used to determine whether the PE unit mode is the multiplication and accumulation calculation mode or the partial sum transfer mode. After the post-order data of all channels corresponding to the convolution window are input into the PE unit, it indicates the multiplication and accumulation calculation mode, and then starts the transfer mode of the convolution result.

[0070] like Figure 4 As shown in FIG. 1 , the PE units of M rows × N columns represent that M sequential data are convolved at the same time, and are multiplied and accumulated with N columns of different weights to obtain the convolution calculation of N output channels.

[0071] In the row direction, there are M rows of convolution computing units (PE units). The PE units in each row are connected from left to right. The sequence data is transmitted from left to right to the adjacent PE units in a pulsating manner, so that the sequence data can be reused between different PE units in the same row. The multiplication and accumulation is from right to left. This horizontal data transmission method only requires a small driver to achieve;

[0072] There are N columns of PE units in the column direction. The upper and lower PE units are not connected to each other. The weight data is transmitted in the form of broadcast and simultaneously transmitted to different PE units in the same column and retained in each PE unit. When the input sequence data is updated, the weight data remains unchanged, thereby realizing the reuse of weights within the PE unit.

[0073] S4. The convolution data is subjected to batch normalization, RELU activation function activation, quantization, pooling, and FIFO buffering in sequence to obtain the output result.

[0074] The convolution data is batch normalized to obtain normalized data, the normalized data is activated by the RELU activation function to obtain activated data, the activated data is quantized to obtain quantized data, and the quantized data is pooled and FIFO cached to obtain the output result. It should be noted that the RELU activation function only retains results greater than zero; the quantization process is to extract valid 4-bit data by truncation; pooling and FIFO are to output the pooling result of a calculation The quantized data is stored in the FIFO buffer and is output with the secondary calculation pooling result. The quantized data is concatenated up and down into M×1 quantized data as the output result.

[0075] S5. Determine the number of processing layers of the output result, thereby achieving full on-chip storage and obtaining target coordinates.

[0076] The process of determining the number of processing layers of the output result is as follows:

[0077] The number of processing layers of the output result obtained by the first calculation is recorded as 1, and the number of processing layers is judged;

[0078] If the number of processing layers of the output result is less than the number of weight layers, the output result is used as the input feature map for secondary calculation. The number of processing layers of the output result obtained after the secondary calculation is the number of processing layers obtained by the previous calculation plus 1 through the layer counter;

[0079] If the number of processing layers of the output result is equal to the number of weight layers, the output result is output as the target coordinates.

[0080] This embodiment implements full on-chip storage by determining the number of processing layers of the output result, effectively reduces the high transmission bandwidth pressure between off-chip data interactions, and reduces local storage overhead.

[0081] Embodiment 2:

[0082] A CNN accelerator data access system provided by an embodiment of the present invention includes:

[0083] Data cache module: used to preset training parameters, obtain input feature maps and store them in the cache area;

[0084] Data preprocessing module: used to judge and process the input feature map in the buffer area and obtain pre-fetched data through the register array, and cyclically process the pre-fetched data to obtain post-sequence data;

[0085] Convolution calculation module: used to perform convolution calculation on the sequenced data to obtain convolution data;

[0086] Data post-processing module: used to perform batch normalization, RELU activation function activation, quantization, pooling and FIFO buffering on the convolution data in sequence to obtain the output result;

[0087] Output judgment module: used to judge the number of processing layers of the output results to obtain the target coordinates.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A CNN accelerator data access method, characterized in that: include: Preset training parameters, obtain input feature maps and store them in the buffer; The input feature map in the buffer area is judged and processed to obtain pre-fetched data through the register array, and the pre-fetched data is cyclically processed to obtain post-sequence data; Perform convolution calculation on the sequenced data to obtain convolution data; The convolution data is processed in turn by batch normalization, RELU activation function activation, quantization, pooling and FIFO buffering to obtain the output result; The number of processing layers of the output result is judged and processed, so as to realize full on-chip storage and obtain the target coordinates; The process of obtaining pre-fetched data by processing the register array includes: Starting from the zeroth address of the input feature map of all channels in the cache, take the first M rows, the M+1th row, and the M+2th row for three cycles to complete the pre-fetching of the input feature map of (M+2) rows × 3 columns; The third address of the input feature map after pre-fetching all channels in the cache is fetched by (M+2) rows × 3 columns and then looped three times to obtain the pre-fetched data; The step of looping the pre-fetched data to obtain the post-sequence data includes: The last row of pre-fetched data output by the register array is used as the first row of data, and the remaining rows of data output by the register array are used as the second to M rows of data in sequence; Directly output the pre-fetched data output from the register array; The first row of pre-fetched data output by the register array is used as the last row of data, and the remaining rows of data output by the register array are used as the first to (M-1) rows of data in sequence; The determining and processing of the number of processing layers of the output result includes: The number of processing layers of the output result obtained by the first calculation is recorded as 1, and the number of processing layers is judged; If the number of processing layers of the output result is less than the number of weight layers, the output result is used as the input feature map of the next layer for secondary calculation. The number of processing layers of the output result obtained after the secondary calculation is the number of processing layers obtained by the previous calculation plus 1 through the layer counter; If the number of processing layers of the output result is equal to the number of weight layers, the output result is output as the target coordinates.

2. The CNN accelerator data access method according to claim 1, characterized in that: The training parameters include the size and weight of the input feature map.

3. The CNN accelerator data access method according to claim 1, characterized in that: The buffer area stores input feature maps of different channels.

4. The CNN accelerator data access method according to claim 1, characterized in that: The method for judging and processing the input feature map in the buffer area includes: Calculate the input feature map address according to the input feature map in the buffer area, and determine whether the input feature map address is an edge of the input feature map; If the judgment result is yes, the input feature map in the buffer area is padded with zeros, and the input feature map after the zero padding is processed by the register array; If the judgment result is no, the input feature map in the buffer area is directly processed by the register array.

5. The CNN accelerator data access method according to claim 4, characterized in that: The method for calculating the input feature map address according to the input feature map in the buffer area includes: ； in, is the input feature map address, channel is the channel offset address, is the cache partition of the feature map, is the pixel value of a row in the feature map And take the value of the nearest power of 2 upwards, is the group offset address, The row offset address.

6. A CNN accelerator data access system, characterized in that: include: Data cache module: used to preset training parameters, obtain input feature maps and store them in the cache area; Data preprocessing module: used to judge and process the input feature map in the buffer area and obtain pre-fetched data through the register array, and cyclically process the pre-fetched data to obtain post-sequence data; Convolution calculation module: used to perform convolution calculation on the sequenced data to obtain convolution data; Data post-processing module: used to perform batch normalization, RELU activation function activation, quantization, pooling and FIFO buffering on the convolution data in sequence to obtain the output result; Output judgment module: judge the number of processing layers of the output result, so as to achieve full on-chip storage and obtain the target coordinates; The process of obtaining pre-fetched data by processing the register array includes: Starting from the zeroth address of the input feature map of all channels in the cache, take the first M rows, the M+1th row, and the M+2th row for three cycles to complete the pre-fetching of the input feature map of (M+2) rows × 3 columns; The third address of the input feature map after pre-fetching all channels in the cache is fetched by (M+2) rows × 3 columns and then looped three times to obtain the pre-fetched data; The step of looping the pre-fetched data to obtain the post-sequence data includes: The last row of pre-fetched data output by the register array is used as the first row of data, and the remaining rows of data output by the register array are used as the second to M rows of data in sequence; Directly output the pre-fetched data output from the register array; The first row of pre-fetched data output by the register array is used as the last row of data, and the remaining rows of data output by the register array are used as the first to (M-1) rows of data in sequence; The determining and processing of the number of processing layers of the output result includes: The number of processing layers of the output result obtained by the first calculation is recorded as 1, and the number of processing layers is judged; If the number of processing layers of the output result is less than the number of weight layers, the output result is used as the input feature map of the next layer for secondary calculation. The number of processing layers of the output result obtained after the secondary calculation is the number of processing layers obtained by the previous calculation plus 1 through the layer counter; If the number of processing layers of the output result is equal to the number of weight layers, the output result is output as the target coordinates.

Images