Architecture auto-optimization method for data flow oriented neural network accelerator design

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a kind of architecture automatic optimization method for data stream neural network accelerator design. A feedforward data stream architecture FPGA accelerator network interlayer FIFO depth simulation calculation method and a calculation storage resource allocation method are proposed. The data stream neural network accelerator architecture optimization parameter search speed is improved, and the time consumed by manual optimization and parameter adjustment and comprehensive wiring in the accelerator design process is reduced. And through the automatic search of parallelism and FIFO parameters, the efficient flow of each calculation unit of the network is guaranteed, and a more reasonable and efficient design scheme can be obtained. The method simplifies the process of neural network algorithm to FPGA accelerator implementation, reduces the dependence of users on neural network accelerator designers, enables algorithm designers to easily use without hardware design experience, and improves the ease of use of the tool.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a method for automatically optimizing the architecture of a neural network accelerator designed for data flow. BACKGROUND

[0002] In recent years, convolutional neural networks have been widely used and have achieved great success in image processing, speech recognition, data analysis and other fields. However, the mainstream convolutional neural network algorithm currently contains millions of floating-point parameters and billions of floating-point operations, and the traditional CPU cannot meet the real-time requirements. The current mainstream hardware acceleration platforms are GPU, ASIC and FPGA. GPU is composed of a large number of parallel computing units, uses data-level parallelism to accelerate the execution of applications, has mature tools such as CUDA and OpenGL, and has low algorithm development difficulty, but its power consumption is usually high. ASIC uses improved hardware architecture to accelerate algorithms, and has high computing efficiency for specific neural network algorithms. However, its specificity also leads to poor flexibility. FPGA is composed of a large number of reconfigurable logic circuits, has low development cost, low power consumption and high flexibility, and has become one of the research hotspots of convolutional neural network accelerators in recent years.

[0003] FPGA neural network accelerators can be divided into two mainstream architectures, namely processing engine architecture and data flow architecture. The processing engine architecture usually adopts the form of systolic array, is suitable for multiple neural network layer types, and has universality. The accelerator calculates each layer in sequence according to the network topology structure, and the calculation input of each layer is read from DDR, and the calculation result is stored in DDR. A large amount of resources are consumed during data transmission between different layers. In comparison, the data flow architecture maps each layer of the neural network to a calculation module, and converts the neural network computation graph into a pipeline for FPGA acceleration. The calculation result of the previous layer is directly output to the next layer after being generated, and does not need to use DDR for caching, so that the data transmission between layers is simplified, and the data transmission time is shortened. At present, there are many data flow architecture accelerators including BNN-PYNQ of Xilinx Company, Ultra_net of Beijing University of Technology and ReBNet of University of California, San Diego.

[0004] Since the existing FPGA accelerator development tools are far from being as mature as GPU development tools, it is still a very tedious work to implement a data flow architecture accelerator on an FPGA platform, i.e., to map a certain convolutional neural network model to an FPGA accelerator. The development tool of a feedforward data flow architecture accelerator realizes the conversion of a network model to an FPGA IP core, which reduces the development difficulty to a certain extent, but the parallelism of each layer, the resource type of a multiplier and a data cache, the FIFO size between layers and other parameters still need to be manually configured. The parameter configuration of different models on an FPGA with different total resources often differs greatly, and special complex adjustment is needed to obtain the near-optimal running speed of the model on a specific FPGA. Therefore, an architecture automatic optimization method for data flow neural network accelerator design is urgently needed to solve the above problems. SUMMARY

[0005] The application provides an architecture automatic optimization method for data flow neural network accelerator design to solve the problem that the parallelism of each layer, the resource type of a multiplier and a data cache, the FIFO size between layers and other parameters still need to be manually configured in the prior art, and the parameter configuration of different models on an FPGA with different total resources often differs greatly, and special complex adjustment is needed to obtain the near-optimal running speed of the model on a specific FPGA.

[0006] To achieve the above object, the application provides the following technical solutions:

[0007] An architecture automatic optimization method for data flow neural network accelerator design, comprising:

[0008] Step 1: calculating the parallelism of each layer of a neural network and the expected running time According to the parallelism of each layer , a set of parallelism parameters of each layer is calculated; each layer selects a set of parallelism parameters that makes the running time of each layer less than the expected running time and the parallelism is the minimum;

[0009] Step 2: using a high-level synthesis template to generate an IP core of each layer of the neural network according to the set of parallelism parameters that makes the running time of each layer less than the expected running time and the parallelism is the minimum obtained in step 1;

[0010] Step 3: connecting the IP cores of each layer of the neural network according to the topology of the neural network, inserting a custom FIFO module between each IP core to construct an accelerator module, and adding a simulation control module to combine with the accelerator module to form a FIFO size simulation calculation model;

[0011] The size setting value of each custom FIFO module is calculated using a FIFO size simulation calculation model;

[0012] The custom FIFO modules of the accelerator module are replaced with general FIFOs, and the depth of each general FIFO is modified to correspond to the size setting value of the corresponding custom FIFO module obtained by the calculation of the FIFO size simulation calculation model;

[0013] Step four: the accelerator module is synthesized using the VIVADO tool to obtain a resource usage synthesis report;

[0014] If the synthesis fails, steps one to three are repeated until the synthesis succeeds or the number of retries exceeds a threshold value;

[0015] If the number of repetitions exceeds the threshold value, it indicates that the resource allocation under the current throughput rate on the FPGA cannot be completed, and the process is ended;

[0016] If the synthesis succeeds and the program termination condition is met, step five is entered;

[0017] If the synthesis succeeds and the program termination condition is not met, the expected throughput rate is adjusted according to the resource allocation, and step one is entered;

[0018] Step five: output the result.

[0019] Preferably, the step one neural network parallelism calculation method is:

[0020] Given a quantized trained network model, a given number of resources available on the FPGA, and an expected throughput rate iteration initial value

[0021] According to the type and parameters of the neural network layer, the number of multiplication operations required for inference of each layer i of an N-layer neural network is calculated, where 1≤i≤N;

[0022] For a convolution layer with a convolution kernel size K i , an output feature map width OFMW i height OFMH i , a feature map input channel number IFM_CH i , and a feature map output channel number OFM_CH i , the number of multiplication operations required for inference is The calculation formula is:

[0023]

[0024] For a fully connected layer with a width W i and a height H i , the number of multiplication operations required for inference is The calculation formula is:

[0025]

[0026] Parallelism of the i-th layer of the neural network If the i-th layer is a convolutional layer, the parallelism is:

[0027]

[0028] If the i-th layer is a fully connected layer, the parallelism is:

[0029]

[0030] where p i is the output channel parallelism, s i is the input channel parallelism, and q i is the input feature map pixel parallelism.

[0031] Convolutional layer running time

[0032] Fully connected layer running time

[0033] where T(clk) is the clock cycle of the FPGA.

[0034] Preferably, the expected running time calculation method in step one is:

[0035] Expected running time where is the expected throughput of the m-th iteration.

[0036] Preferably, in step two, the IP core includes multipliers and data cache areas.

[0037] where multipliers with bit numbers greater than or equal to 8 bits are constructed by DSPs, and multipliers with bit numbers less than 8 bits are constructed by LUTs.

[0038] The data cache area is divided into a weight cache area and a calculation intermediate variable cache area.

[0039] The weight cache area is constructed by BRAM, and the calculation intermediate variable cache area is constructed by LUTRAM.

[0040] Preferably, in step three, the simulation control module is composed of a clock unit, a data generation unit, a data measurement unit, and a data receiving unit; wherein:

[0041] The clock unit is used to provide clock driving for the data generation unit, the data measurement unit, and the data receiving unit.

[0042] A data generating unit is configured to input an input image into the accelerator module;

[0043] A data measuring unit is configured to continuously observe the data cache amount of the custom FIFO module during operation and record the maximum value;

[0044] A data receiving unit is configured to receive simulation results.

[0045] Preferably, in step three, the FIFO size setting value calculation process is as follows:

[0046] The data generating unit inputs the input image into the accelerator module at a time interval t(pix), and the data amount each time is CHin; CHin is the number of input image channels;

[0047] The time interval t(pix) is as follows:

[0048]

[0049] wherein Win is the input image width, Hin is the input image height, is the expected operation time;

[0050] The data measuring unit continuously observes the data cache amount of the custom FIFO module during operation and records the maximum value;

[0051] The data receiving unit continuously receives simulation results, and the simulation ends when two image results are received.

[0052] The data measuring unit outputs the maximum value of the data cache amount of each custom FIFO module, and after rounding up the output data cache amount maximum value to the power of 2, the rounded value is used as the size setting value of the corresponding custom FIFO module.

[0053] Preferably, in step four, the specific process is as follows:

[0054] The accelerator module is synthesized and implemented using the VIVADO tool, and a resource usage report is obtained;

[0055] It is checked whether the DSP resource usage exceeds the total amount of DSP resources in the FPGA, and if so, the multiplier construction method of the IP core that exceeds is changed from DSP resource construction to LUT resource construction;

[0056] It is checked whether the BRAM resource usage exceeds the threshold of the total amount of BRAM resources in the FPGA; if so, the BRAM resource utilization rate Low IP core conversion to LUTRAM resource construction method, until the BRAM resource usage is lower than the threshold of the total amount of BRAM resources in the FPGA; if the DDR storage is enabled, in the order of full connection layer priority high, OFM small convolution layer priority medium, OFM large convolution layer priority low, the excess IP core data storage construction method is constructed from BRAM resource to DDR resource; and at this time should meet:

[0057]

[0058] In the formula, F clk Clock frequency, Width DMA DMA data width, The expected throughput of the mth iteration;

[0059] Check if the LUTRAM resource usage exceeds the threshold of the total amount of LUTRAM resources in the FPGA; if it exceeds, the BRAM resource utilization High IP core conversion to BRAM resource construction method or DDR resource construction method, until the LUTRAM resource usage is lower than the threshold of the total amount of LUTRAM resources in the FPGA;

[0060] Again, use VIVADO tool to synthesize the accelerator module, get the resource usage report;

[0061] According to the synthesis report, get the resource usage rate of DSP, LUT, BRAM and LUTRAM, and make conversion adjustment;

[0062] Again, according to the report, get the adjusted BRAM and LUTRAM resource usage rate of IP core in all modules of FPGA;

[0063] Configure FIFO resource as LUTRAM, use VIVADO synthesis, get synthesis report;

[0064] Configure FIFO resource as BRAM, use VIVADO synthesis, get synthesis report;

[0065] According to the synthesis report, get the resource amount required by each FIFO in LUTRAM and BRAM construction;

[0066] According to the BRAM and LUTRAM resource usage rate of IP core in all modules of FPGA and the resource amount required by each FIFO in LUTRAM and BRAM construction, configure the FIFO resource;

[0067] The specific process of FIFO resource configuration is as follows:

[0068] The selected FIFO selects the side with lower resource utilization in the BRAM resource and the LUTRAM resource at this time to build; the BRAM and LUTRAM resource utilization of the IP core in all modules of the FPGA at this time is calculated; the next FIFO selects the side with lower resource utilization in the BRAM resource and the LUTRAM resource at this time to build; the above steps are repeated until the configuration of the last FIFO resource is completed;

[0069] The convolutional layer configuration, the fully connected layer configuration and the FIFO configuration obtained according to the above steps are used to regenerate the IP core.

[0070] Preferably, the BRAM resource utilization of the convolutional layer core and the fully connected layer is The calculation formula is:

[0071]

[0072] bitw i is the weight data bit width, is the number of BRAMs used to store weights, H i is the height of the fully connected layer, W i is the width of the fully connected layer, K i is the convolution kernel size, OFMW i is the output feature map width, OFMH i is the output feature map height; wherein The calculation formula is:

[0073]

[0074] The calculation formula of ω is:

[0075]

[0076] In the formula, the parallelism degree is

[0077] memw i is the memory bit width, and the calculation formula is:

[0078]

[0079] Preferably, in step four, the program termination condition is:

[0080] The program running time exceeds a threshold value, or the FPGA resource utilization is higher than a threshold value, or the expected throughput iteration number exceeds a threshold value.

[0081] Preferably, in step five, the output result includes:

[0082] ​Each layer parallelism parameter configuration; network optimization running speed; accelerator IP core; each IP core calculation and storage resource type, FIFO size.

[0083] The application has the advantages that:

[0084] The FPGA accelerator network layer FIFO depth simulation calculation method and the calculation and storage resource allocation method using the feedforward data flow architecture FPGA accelerator realize automatic search of the FPGA neural network accelerator parameters, reduce the workload of manual parameter setting, and reduce a large amount of time consumed due to manual optimization, parameter adjustment and comprehensive wiring in the accelerator design process.

[0085] The parallelism and FIFO parameters searched automatically guarantee efficient flow of each calculation unit of the network, and improve the usability of the tool. BRIEF DESCRIPTION OF DRAWINGS

[0086] The drawings constituting a part of the specification of the application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application. In the drawings:

[0087] Figure 1 It is an automatic optimization method flowchart for a data flow neural network accelerator design architecture.

[0088] Figure 2 It is a matrix vector threshold unit IP core structure schematic diagram.

[0089] Figure 3 It is a processing unit structure schematic diagram.

[0090] Figure 4 It is a FIFO size simulation calculation model schematic diagram. DETAILED DESCRIPTION

[0091] The application will be described in detail below with reference to the drawings and in combination with the embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0092] The following detailed description is exemplary description, which aims to provide further detailed description of the application. Unless otherwise specified, all technical terms used in the application have the same meaning as that generally understood by the general technical personnel in the field to which the application belongs. The terms used in the application are only used to describe the specific embodiments, and are not intended to limit the exemplary embodiments according to the application.

[0093] Embodiment 1:

[0094] An automatic optimization method for a data flow neural network accelerator design architecture, specifically comprising:

[0095] Step one, calculate the network layer operation number, according to the expected throughput rate of the mth iteration and the operation number, calculate the parallelism parameter of each layer; Specifically:

[0096] Given the quantized trained network model, such as ONNX format; Given the number of resources available to FPGA, including LUT resource Res LUT , LUTRAM resource Res LUTRAM , Block RAM (BRAM) resource Res BRAM , DSP resource Res DSP ; Given the expected throughput rate iteration initial value

[0097] According to the type and parameters of neural network layer, the multiplication operation number required for each layer i of N-layer neural network inference is calculated, where 1≤i≤N. For the convolution layer with convolution kernel size K i , output feature map width OFMW i height OFMH i , input channel number of feature map IFM_CH i , output channel number of feature map OFM_CH i , the multiplication operation number required for inference is The calculation method is shown in (1):

[0098]

[0099] For the fully connected layer with width W i , height H i , the multiplication operation number required for inference is The calculation method is shown in (2):

[0100]

[0101] The parallelism of each layer of network model is represented by , and the parallel parameter set of each layer of network model is constructed. For the convolution layer, there are three parallel parameters, where p i is the output channel parallelism, s i is the input channel parallelism, and q i is the input feature map pixel parallelism. The parallel parameters need to satisfy s i ≤IFM_CH i , p i ≤OFM_CH i , q i ≤OFMW i , IFM_CH i %s i =0, OFM_CH i %p=0, OFMWi q = 0. For the fully connected layer, there are two parallel parameters, The parallel parameters need to satisfy s i ≤ W i , p i ≤ H i , W i s i = 0, H i p i = 0.

[0102] The running time of each layer is calculated after the above parallel parameters are applied. The convolution layer running time The fully connected layer running time Where T(clk) is the clock cycle of the FPGA. The expected running time Where is the expected throughput of the mth iteration.

[0103] Select the set of parameters that make the running time of each layer is less than the expected running time, and the parallelism is the smallest.

[0104] Step two, select the parallelism parameter and resource configuration method, and generate the IP core of each layer according to the high-level synthesis template. Specifically:

[0105] Using the high-level synthesis template, generate the IP core of each layer of the neural network according to the parallelism parameter.

[0106] Where the multiplier greater than or equal to 8 bits in each layer is constructed by DSP, and the multiplier less than 8 bits is constructed by LUT;

[0107] The data buffer is divided into a weight buffer and a calculation intermediate variable buffer; the weight buffer is constructed by BRAM, and the intermediate variable buffer is constructed by LUTRAM.

[0108] The matrix vector threshold unit IP core for performing matrix multiplication is as shown in Figure 2 It is composed of input and output buffers, weight buffers, and processing units, and its input and output interfaces are AXI-Stream (AXIS).

[0109] The processing unit structure is as shown in Figure 3 It is composed of multipliers, accumulators, and threshold comparators.

[0110] Step three, connect the IP cores of each layer of the neural network according to the neural network topology to construct the accelerator, insert FIFO between each IP core to construct a simulation calculation model, and simulate the FIFO size. Specifically:

[0111] A custom FIFO module of observable data buffer size is added between each IP core of the accelerator to construct an accelerator module; a simulation control module is additionally added, and the simulation control module and the accelerator module form a FIFO size simulation calculation module, as shown in Figure 4

[0112] The simulation control module is composed of a clock unit, a data generation unit, a data measurement unit and a data receiving unit.

[0113] The accelerator module is composed of a neural network layer IP core and a FIFO.

[0114] The clock unit is used to generate a simulation clock to drive other modules; the simulation control module is used to generate and measure data. The simulation process is controlled by using python.

[0115] The simulation method is as follows: start simulation, the data generation unit inputs the input image into the accelerator module at a time interval t(pix), and the data amount is CHin each time, wherein CHin is the number of input image channels.

[0116] The time interval t(pix) calculation formula is: Wherein Win is the input image width, and Hin is the input image height.

[0117] The data measurement unit continuously observes the data buffer size of the custom FIFO module during operation and records the maximum value. The data receiving unit continuously receives the simulation results, and the simulation is ended when 2 images are received. The data measurement unit outputs the maximum value of each custom FIFO data buffer size (depth), and the values are rounded up to 2n after being taken to the corresponding FIFO size setting value.

[0118] The custom FIFO of the accelerator module is replaced with a general FIFO, and the depth of each FIFO is modified to correspond to the result value of the previous step.

[0119] Step four, the resource usage of each IP core of the accelerator is obtained by comprehensive implementation, and the IP core resource allocation is adjusted when the total amount of FPGA resources is exceeded. Specifically:

[0120] The FIFO resource is configured as LUTRAM and BRAM, and VIVADO is used for synthesis. According to the synthesis report, the resource amount required for each FIFO to be constructed in LUTRAM and BRAM modes is obtained.

[0121] ​Each convolutional layer and fully connected layer IP core can store weights in BRAM, LUTRAM or DDR. When using BRAM to store weights, the amount of weight data is usually less than the total amount of data that the BRAM used to build the IP core can accommodate. The ratio of the amount of weight data to the total amount of data that the BRAM can accommodate is referred to as BRAM resource utilization. BRAM resource utilization The calculation formula is as formula (3):

[0122]

[0123] bitw i is the weight data bit width, is the number of BRAMs used to store weights, and the calculation formula is as formula (4):

[0124]

[0125] The calculation formula of ω is as formula (5):

[0126]

[0127] memw i is the memory bit width, and the calculation formula is as formula (6):

[0128]

[0129] When using DDR to store weights, the clock frequency F clk , DMA data width Width DMA and throughput limit the weight storage amount of DDR, and should satisfy the inequality (7):

[0130]

[0131] Res i is the amount of weight data required between the IP core and the DDR for the i-th layer of convolutional or fully connected layer to complete the calculation of one image, and the calculation method is as formula (8):

[0132]

[0133] The accelerator module is synthesized using the VIVADO tool to obtain a resource usage report and obtain the resource usage of each IP core. If the actual usage of a certain resource of the accelerator module is higher than the total amount of the resource of the FPGA, the calculation and storage resource allocation of the IP core needs to be adjusted.

[0134] The method in this paper only adjusts the IP core with large calculation and storage resource amount, i.e. convolutional and fully connected IP core. The IP core with small calculation and storage resource amount, such as down-sampling and addition IP core, is not adjusted.

[0135] The convolution and fully connected IP core calculation and storage resource type adjustment method is as follows:

[0136] (1) Check whether the DSP resource usage exceeds the total amount of DSP resources in the FPGA. Adjust the multiplier construction method of the IP core that exceeds the total amount of DSP resources in the FPGA from DSP resource construction to LUT resource construction.

[0137] (2) Check whether the BRAM resource usage exceeds 90% (or other threshold value) of the total amount of BRAM resources in the FPGA. If it exceeds, the BRAM resource utilization of the IP core with low priority is converted to LUTRAM resource construction method first until the requirement of “BRAM resource usage is lower than 90% (or other threshold value) of the total amount of BRAM resources in the FPGA” is met. If DDR storage is enabled, adjust the data storage construction method of the IP core that exceeds the total amount of BRAM resources in the FPGA from BRAM resource construction to DDR resource construction in the order of fully connected layer priority high, OFM small convolution layer priority medium, and OFM large convolution layer priority low. The formula (7) should be met.

[0138] (3) Check whether the LUTRAM resource usage exceeds 90% (or other threshold value) of the total amount of LUTRAM resources in the FPGA. If it exceeds, the BRAM resource utilization of the IP core with high priority is converted to BRAM or DDR resource construction method until the requirement of “LUTRAM resource usage is lower than 90% (or other threshold value) of the total amount of LUTRAM resources in the FPGA” is met.

[0139] (4) Balance the configuration of FIFO resources to make the BRAM and LUTRAM resource usage of all module IP cores in the FPGA close to each other.

[0140] (5) Re-generate the IP core according to the above configuration, build the accelerator according to the neural network topology structure connection, and use the VIVADO tool for synthesis implementation.

[0141] (6) If the synthesis implementation fails, further allocate the IP core calculation and storage resources by repeating the above steps (1)-(5) until the VIVADO tool synthesis implementation is passed;

[0142] If the number of retries exceeds the threshold value, it indicates that the resource allocation under the current throughput rate in the FPGA cannot be completed, and the program ends.

[0143] (7) If the synthesis experiment is successful, determine whether the neural network accelerator parameter automatic search reaches the termination condition;

[0144] The termination condition of the neural network accelerator parameter automatic search is that the program running time exceeds the threshold value, or the FPGA resource usage rate is higher than the threshold value, or the expected throughput rate iteration number exceeds the threshold value.

[0145] At this time, the neural network accelerator parameter automatic search reaches the termination condition, the neural network accelerator parameter automatic search process is ended, and step five is entered;

[0146] If the termination condition is not reached, the expected throughput is adjusted according to the resource allocation, and step one is returned to continue iteration.

[0147] Step five, output each layer parallelism parameter configuration; output the running speed of the optimized network; output the accelerator IP core for deployment; output the IP core calculation and storage resource type, FIFO size.

[0148] Embodiment 2:

[0149] The VGG model after 2bit quantization contains 6 convolutional layers and 3 fully connected layers, the input is a 32*32*3 image, and the output is 10 classification results. The selected FPGA model is xc7z020clg400, the available resource quantity Res LUT is 53200, Res LUTRAM is 17400, Res BRAM is 140, and Res DSP is 220. The FPGA resource utilization rate of 75% is selected as the automatic search termination condition. The expected throughput is 100fps as the initial value of iteration, and after 4 iterations, the values are 250fps, 500fps, 750fps, and 1000fps in turn.

[0150] The method takes as an example for introduction.

[0151] According to formula (1), the multiplication operation number of each of the 5 convolutional layers is calculated According to formula (2), the multiplication operation number of each of the 3 fully connected layers is calculated The calculation results of IOP are shown in Table 1.

[0152] The parallel parameters of each layer are calculated The convolutional layer has p i , s i , q i three parallel parameters, for Conv1, because IFM_CH1 is 3, s1 takes the value range {1, 3}; because OFM_CH1 is 64, p1 takes the value range {1, 2, 4, 8, 16, 32, 64}; because OFMW1 is 30, q i takes the value range {1, 2, 3, 5, 6, 10, 15, 30}. The expected running time i.e. 1000us. Calculate network layer Conv1 parallelism satisfy running time less than expected running time condition, take 972us, take the minimum value 16, at this time s = 1, p = 16, q = 1. Calculate the running time of all layers in this way and parallelism The results are shown in Table 1. The actual throughput rate is obtained 868fps.

[0153] Table 1 Network layer operation, parallelism and running time

[0154]

[0155] The resource configuration method for the neural network layer is as follows: according to the number of multiplier bits, the multiplier construction method is determined, and the multiplier is constructed by DSP when it is greater than or equal to 8bit, and the multiplier is constructed by LUT when it is less than 8bit. In this example, 2bit multipliers are used, so LUT is used to construct them. In this example, data cache is divided into weight cache and intermediate variable cache. The weight cache is constructed by BRAM (BRAM), and the intermediate variable cache is constructed by LUTRAM. A high-level synthesis template is used, and the resource configuration and parallelism parameters are applied to generate the IP core of each layer of the neural network.

[0156] A custom FIFO module is added between each IP core of the accelerator to construct a simulation model to measure the FIFO value. The amount of data input each time is the number of image channels, and the data amount is 3 bytes. According to the formula The time interval t(pix) is 0.976us. The data measurement unit continuously observes the usage of the custom FIFO module during operation and records the maximum value. The data receiving unit continuously receives the simulation results, and the simulation is ended when 2 images are received. The data measurement unit outputs the maximum value of the usage (depth) of each custom FIFO, and these values are rounded up to the power of 2 to set the size of the corresponding FIFO. The calculated FIFO size results are shown in Table 2.

[0157] The FIFO resource is configured as LUTRAM and BRAM respectively, and VIVADO is used for synthesis. According to the synthesis report, the resource amount required for each FIFO to be constructed in LUTRAM and BRAM is obtained, and the results are shown in Table 2.

[0158] Table 2 FIFO depth simulation results and resource usage under two construction methods

[0159] FIFO 1 2 3 4 5 6 7 8 9 10 11 12 Depth 32 1024 32 1024 32 32 512 32 1024 512 64 64 LUTRAM 8 256 32 1024 128 32 512 32 1024 256 8 8 BRAM 0.5 0.5 1 1 2 1 1 1 1 0.5 0.5 0.5

[0160] The BRAM resource usage is calculated according to formula (3). The BRAM resource utilization rate is calculated according to formula (4). The calculation results are shown in Table 3.

[0161] Table 3 BRAM Resource Usage and Utilization Rate

[0162]

[0163] The accelerator module was synthesized using Python and the VIVADO TCL scripting tool. A resource usage report was obtained to show the resource usage of each IP core. Analysis of the resource report revealed that the actual BRAM resource usage was 148, exceeding the total FPGA BRAM resource limit of 140. This is based on the methodology described in this paper. The size will sort the network layers. The weights of the smaller Conv1, Conv2, Conv3, and Conv4 layers were converted from BRAM construction to LUTRAM resource construction. After the conversion, the BRAM resource usage was 118, lower than the total FPGA BRAM resource of 140. After resource adjustment, the BRAM resource usage of Conv1, Conv2, Conv3, and Conv4 layers decreased to 0, while the corresponding LUTRAM resource usage increased to 88, 2052, 3078, and 6156, respectively. Further checking the network layer LUTRAM resource usage showed that the LUTRAM resource usage was 13370, and the total resource was 17400, which did not exceed 80% of the total LUTRAM resource in the FPGA, so no reallocation was needed. A balanced configuration of FIFO resources was implemented to make the BRAM and LUTRAM resource usage of all FPGA modules similar. The FIFO resource configuration results are shown in Table 4.

[0164] Table 4 FIFO Resource Configuration Results

[0165]

[0166] The IP cores were regenerated according to the above configuration. An accelerator was constructed based on the neural network topology and synthesized using VIVADO. The results are as follows: LUT resource usage was 38518, accounting for 72% of the total; DSP resource usage was 0, accounting for 0% of the total; LUTRAM resource usage was 13602, accounting for 78% of the total; and BRAM resource usage was 112, accounting for 80% of the total. Based on the automatic search termination condition, since the LUTRAM and BRAM resource utilization rates in the FPGA were both higher than 75%, the automatic parameter search process for the neural network accelerator ended. Output parallelism parameters. Actual throughput after network optimization Output accelerator IP core, compute and memory resource types, FIFO sizes, etc.

[0167] From a conceptual point of view, the present application can be implemented by other embodiments which do not depart from the spirit or essential characteristics of the application. Therefore, the above disclosed embodiments are merely illustrative in all aspects and are not the only ones. All modifications within the scope of the present application or within the equivalent scope of the present application are intended to be embraced by the present application.

[0168] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon.

[0169] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for functionally implementing the

[0170] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for functionally implementing the

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for functionally implementing the

[0172] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered within the protection scope of the claims of the present application.

Claims

1. An architecture auto-optimization method for dataflow oriented neural network accelerator design, characterized in that, Comprise: Step one: calculate the parallelism of each layer of the neural network and the expected running time , according to the parallelism of each layer calculate a set of parallelism parameters of each layer; select a set of parallelism parameters of each layer that makes the running time of each layer less than the expected running time and the parallelism is the smallest Step two: using a high-level synthesis template, according to the running time of each layer obtained in step one, the running time of each layer is less than the expected running time and parallelism a minimum set of parallelism parameter sets, generating IP cores for each layer of the neural network; Step three: connecting IP cores of each layer of the neural network according to the neural network topology, inserting a custom FIFO module between each IP core, and constructing an accelerator module; add a simulation control module to combine with the accelerator module to form a FIFO size simulation calculation model; Use the FIFO size simulation calculation model to calculate the size setting value of each custom FIFO module; Change the custom FIFO module of the accelerator module to a general FIFO, and modify the depth of each general FIFO to correspond to the size setting value of the corresponding custom FIFO module obtained by the FIFO size simulation calculation model; Step four: use the VIVADO tool to synthesize the accelerator module to obtain a resource usage synthesis report; If the synthesis fails, repeat steps one to three until the synthesis is successful or the number of retries exceeds the threshold; If the number of repetitions exceeds the threshold, it indicates that the resource allocation under the current throughput rate on the FPGA cannot be completed, and the process ends; If the synthesis is successful and meets the program termination condition, proceed to step five; If the synthesis is successful and does not meet the program termination condition, adjust the expected throughput rate based on the resource allocation, and proceed to step one; Step five: output the results; The parallelism calculation method for each layer of the neural network in step one is: Given a quantized trained network model, given the number of resources available to the FPGA, given the desired throughput rate iteration initial value ; According to the neural network layer type and parameters, calculate the number of multiplication operations required for each layer i of an N-layer neural network, where 1≤𝑖≤𝑁; For convolution kernel size Output feature map width high Number of input channels for feature map Number of feature map output channels The number of multiplication operations required for inference in convolutional layers. The calculation formula is: For a fully connected layer with a width of and a height of , the number of multiplication operations required for inference is calculated by the formula: computing the parallelism of the i-th layer of the neural network , if the i-th layer is a convolutional layer, the parallelism is: If the i-th layer is a fully connected layer, the parallelism is: wherein is the output channel parallelism, is the input channel parallelism, is the input feature map pixel parallelism; Convolutional layer run time ​ Fully connected layer run time ; wherein is the clock period of the FPGA; The expected running time calculation method in step one is: Expected run time wherein is the expected throughput rate for the mth iteration.

2. The method of claim 1, wherein the method is performed by a computer system. In step two, the IP core includes a multiplier and a data buffer area; Multipliers with bit numbers greater than or equal to 8 bits are constructed by DSP, and multipliers with bit numbers less than 8 bits are constructed by LUT; The data buffer area is divided into a weight buffer area and a calculation intermediate variable buffer area; The weight buffer area is constructed by BRAM, and the calculation intermediate variable buffer area is constructed by LUTRAM.

3. The automatic architecture optimization method for dataflow neural network accelerator design as described in claim 1, characterized in that, In step three, the simulation control module is composed of a clock unit, a data generation unit, a data measurement unit, and a data receiving unit; wherein: The clock unit provides clock driving for the data generation unit, the data measurement unit, and the data receiving unit; The data generation unit inputs the input image into the accelerator module; The data measurement unit continuously observes the data buffer amount of the custom FIFO module during operation and records the maximum value; The data receiving unit receives the simulation results.

4. The method of claim 3, wherein the method further comprises: In step three, the FIFO size setting value calculation process is: The data generation unit inputs the input image at a time interval into the accelerator module, the amount of data input at a time being is the number of input image channels;​ Time interval is: ； where Win is the input image width, Hin is the input image height, T is the desired run time; The data measurement unit continuously observes the data buffer amount of the custom FIFO module during operation and records the maximum value; The data receiving unit continuously receives the simulation results, and the simulation ends when 2 image results are received; The data measurement unit outputs the maximum data buffer amount of each custom FIFO module, and after rounding up the output data buffer amount maximum value to 2 raised to the power of n, it is used as the size setting value of the corresponding custom FIFO module.

5. The method of claim 1, wherein, In step four, the specific process is: Use the VIVADO tool to synthesize the accelerator module to obtain a resource usage report; Check whether the DSP resource usage exceeds the total amount of DSP resources in the FPGA, and adjust the multiplier construction method of the IP core that exceeds the part from the DSP resource construction to the LUT resource construction; whether the BRAM resource usage exceeds the proportion threshold of the total amount of BRAM resources in the FPGA; if it exceeds, the BRAM resource utilization low IP core conversion to LUTRAM resource construction mode, until the BRAM resource usage is lower than the proportion threshold of the total amount of BRAM resources in the FPGA; if the DDR storage is enabled, in the order of full connection layer priority high, convolution layer priority medium OFM small, and convolution layer priority low OFM large, the excess IP core data storage construction mode is constructed from BRAM resource to DDR resource; and at this time, the following should be met: wherein is the clock frequency, is the DMA data width, is the expected throughput for the mth iteration; checking whether the LUTRAM resource usage exceeds a proportion threshold of the total amount of LUTRAM resources in the FPGA; if it exceeds, the BRAM resource utilization a high IP core is converted into a BRAM resource construction method or a DDR resource construction method until the LUTRAM resource usage is lower than the proportion threshold of the total amount of LUTRAM resources in the FPGA; Again, use the VIVADO tool to synthesize the accelerator module to obtain the resource usage report; According to the synthesis report, get the DSP, LUT, BRAM and LUTRAM resource usage rate, and perform conversion adjustment; Again, synthesize and implement according to the report to get the adjusted BRAM and LUTRAM resource usage rate of the IP core in all modules of the FPGA; Configure the FIFO resource as LUTRAM, use VIVADO to synthesize, and obtain the synthesis report; Configure the FIFO resource as BRAM, use VIVADO to synthesize, and obtain the synthesis report; According to the synthesis report, obtain the resource amount required for each FIFO to be constructed in the LUTRAM mode and the BRAM mode; According to the BRAM and LUTRAM resource usage rate of the IP core in all modules of the FPGA and the resource amount required for each FIFO to be constructed in the LUTRAM mode and the BRAM mode, configure the FIFO resource; The specific process of FIFO resource configuration is as follows: The selected FIFO selects the side with lower resource usage rate in the BRAM resource and the LUTRAM resource for construction at this time; calculate the BRAM and LUTRAM resource usage rate of the IP core in all modules of the FPGA at this time; the next FIFO selects the side with lower resource usage rate in the BRAM resource and the LUTRAM resource for construction at this time; repeat the above steps until the last FIFO resource configuration is completed; According to the convolution layer configuration, the fully connected layer configuration and the FIFO configuration obtained in the above steps, regenerate the IP core.

6. The method of claim 5, wherein the method further comprises: Convolutional layer kernel fully connected layer BRAM resource utilization The calculation formula is: wherein is the weight data bit width, is the number of BRAMs used to store weights, H i is the fully connected layer height, W i is the fully connected layer width, K i is the convolution kernel size, is the output feature map width, is the output feature map height; wherein The calculation formula is: In the formula, The calculation formula is: In the formula, is the parallelism degree; The memory bit width is calculated as follows: 。 7. The automatic architecture optimization method for dataflow neural network accelerator design as described in claim 1, characterized in that, In step four, the program termination conditions are: The program running time exceeds the threshold value, or the FPGA resource usage rate is higher than the threshold value, or the expected throughput rate exceeds the threshold value.

8. The method of claim 5, wherein the method further comprises: In step five, the output results include: Each layer parallelism parameter configuration; network optimized running speed; accelerator IP core; each IP core calculation and storage resource type, FIFO size.

Images