A method for optimizing cross-layer memory access bandwidth to support deep neural network acceleration

Abstract

The present invention relates to the field of neural network optimization technology, and in particular to a method for optimizing cross-layer memory access bandwidth to support deep neural networks. The method comprises the following steps: 1. designing a bandwidth model for single-layer block optimization and loop sequence optimization; 2. performing single-layer independent block optimization and loop sequence optimization; 3. performing cross-layer joint block optimization and loop sequence optimization; and 4. performing composite block optimization and loop sequence optimization. The method can implement a data block strategy for minimum DRAM access, a loop unrolling control strategy, and a data storage mapping update strategy, achieving a fine-grained data access and data flow solution that maximizes data reuse, and effectively supporting automatic code generation and computational mapping optimization for optimizing computational and memory access efficiency.

Description

Technical Field

[0001] The present invention relates to the field of neural network optimization technology, and in particular to a method for optimizing bandwidth for accelerating cross-layer memory access in deep neural networks. Background Art

[0002] When various deep neural network models are deployed and applied in practice, a large number of input feature map coefficients and weight coefficients are stored in large-capacity DRAM memories. The computing core of a CNN accelerator has limited computing resources, and the on-chip SRAM cache capacity is also limited. When performing actual multi-layer convolution calculations, the input feature map coefficients and weight coefficients are loaded from off-chip DRAM into the on-chip SRAM cache near the computing core. The computing core's MAC array is typically only capable of performing a portion of the convolution calculation. The resulting output feature map coefficients need to be stored in DRAM, and the computing core then continues to perform convolution calculations on other input feature coefficients. The large amount and diversity of data in CNN network layers results in large and complex data transfers between the computing core, on-chip cache, and off-chip DRAM. DRAM-based off-chip memory access is the most energy-intensive operation in deep neural network (DNN) accelerators, representing the intelligent computing memory wall problem.

[0003] Optimizing network model memory access on a specific CNN acceleration engine platform requires addressing the following: CNN convolution weight and feature coefficient block partitioning, core compute unit computation mapping, external pipeline scheduling (i.e., loop ordering strategy), and data storage mapping update strategy. This requires code generation and implementation mapping for optimized computation and memory access efficiency. This optimization solution space is extremely large, involving variables such as batch size concurrency (Tz), feature map width and height (Tn*Tm) of the compute core involved in the computation, on-chip cache input feature dimensions (Ti), on-chip cache output feature dimensions (Tj), the number of cross-layer optimization layers, and loop priority strategy types. These loop priority strategies are also characterized by the following data reuse patterns: input coefficient reuse (iro), output coefficient reuse (oro), and weight coefficient reuse (wro). These control variables are each discretely assigned a certain number of candidate values, constituting a very large possible solution space. Without an analytic functional expression for the SRAM cache consumption and external memory access bandwidth models for different parameter combinations, finding an optimal solution is extremely difficult. The on-chip SRAM cache is inherent to the compute core and constitutes a constraint that must be considered during the solution. If an analyzable mathematical expression model of on-chip SRAM consumption and external memory access bandwidth under different parameter combinations can be constructed, then it is possible to solve the memory access bandwidth minimization solution that satisfies the cache SRAM constraint offline.

[0004] Currently, existing work generally focuses only on single-layer bandwidth access optimization, without considering that the output feature data of the previous layer is already in the cache and can be reused by the input feature coefficients of subsequent layers. This cross-layer data reuse is very effective for the efficient deployment of network layers with a high proportion of feature coefficients. Some work does not consider input batch concurrency. Input batch concurrency can effectively reuse weight coefficients and is very effective for the WRO mode with weight coefficient priority. Some work also organizes data according to the data access paradigm required for core calculations when building bandwidth access function models. This process will cause some overlapping data in the input and output feature map coefficients. The repeated loading of this overlapping data reduces the accuracy of the model and the efficiency of coefficient optimization. In addition, some work considers the bandwidth consumption in an ideal logical sense when building bandwidth models, without considering the actual DRAM structure, resulting in inconsistencies between the actual DRAM access bandwidth consumption and the logical access bandwidth. This problem has not been considered by most existing work.

[0005] At the same time, the optimal data flow strategy that considers batch concurrency strength optimization, block strategy optimization, loop order strategy optimization, and cross-layer data reuse optimization, and achieving globally optimized on-chip DRAM access and on-chip data reuse under the constraints of on-chip cache and computing core computing resources, is a problem that needs further research and solution. It is of great significance for the efficient deployment and computational mapping of network models on specific convolutional accelerator hardware platforms. Summary of the Invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a method for optimizing cross-layer memory access bandwidth to support deep neural network acceleration. The specific technical solution is as follows:

[0007] A method for optimizing cross-layer memory access bandwidth to support deep neural network acceleration, comprising:

[0008] Step 1: Design a single-layer block optimization and loop sequence optimization bandwidth model;

[0009] Step 2: Single-layer independent block optimization and loop sequence optimization;

[0010] Step 3: Cross-layer joint block optimization and loop order optimization;

[0011] Step 4: Composite block optimization and loop sequence optimization.

[0012] Furthermore, the model design of step 1 specifically includes:

[0013] According to the characteristics of deep convolution calculation data flow, three data access modes are set, including input feature map coefficient reuse IRO mode, output feature map coefficient reuse ORO mode and weight coefficient reuse WRO mode;

[0014] According to the data flow characteristics of the three data access modes, the single access data volume and the number of cyclic accesses of the input feature map, output feature map and weight in each mode are set.

[0015] Furthermore, the single access data volume of the input feature map, output feature map and weight in each mode specifically includes: input feature map coefficients: Bif1, Bif2, Bif3, output feature map coefficients: Bof1, Bof2, Bof3, and weight coefficients: Bwgt1, Bwgt2, Bwgt3; the number of cyclic accesses of the input feature map, output feature map and weight in each mode specifically includes: input feature map coefficient access times: Cif1, Cif2, Cif3, output feature map coefficient access times: Cof1, Cof2, Cof3, and weight coefficient access times: Cwgt1, Cwgt2, Cwgt3.

[0016] Furthermore, the step 2 specifically includes:

[0017] Step 2.1: Divide the feature data calculated by the single-layer convolution into blocks and obtain the block parameters;

[0018] Step 2.2: Traverse all possible block combinations, calculate the memory bandwidth and storage occupancy under the three data access modes respectively, and compare the storage threshold and minimum memory bandwidth to determine the optimal block combination and data access mode.

[0019] Furthermore, the step 2.1 specifically includes: using the step sizes of 1, stepM, stepN, stepI, and stepJ to divide Z, M, N, I, and J into blocks respectively to obtain block parameters Tz, Th, Tw, Tm, Tn, Ti, and Tj; wherein Z is the maximum batch number, I is the number of input feature map channels, J*M*N is the feature map coefficient dimension, Th=S*(Tm-1)+P, Tw=S*(Tn-1)+Q, S is the convolution step size, I*J*P*Q is the convolution kernel size, and Tp=P and Tq=Q are taken; Tz is the batch cycle block size, Tm*Tn and Th*Tw are the horizontal and vertical block sizes of the data feature map coefficients, Ti is the input feature map channel block size, and Tj is the output feature map channel block size.

[0020] Furthermore, in step 2.2, the calculation of the memory bandwidth and storage occupancy is specifically as follows:

[0021] Bandwidth of input feature map ifmaps: BW if = B if *C if *size(ifmaps);

[0022] Bandwidth of output feature maps ofmaps: BW of = B of *C of *size(ofmaps);

[0023] Bandwidth of convolution kernel weight: BW wgt = B wgt *C wgt *size(weight);

[0024] Storage usage: SRAM tot = B if *size(ifmaps)+B wgt *size(weight)+B of *size(ofmaps);

[0025] Total bandwidth: BW tot = BW if +BW of +BWwgt;

[0026] Among them, B if 、B of 、B wgt Represents the data size of input feature map, output feature map and convolution kernel respectively, C if 、C of 、C wgt Respectively represent the number of cyclic visits to the input feature map, output feature map and convolution kernel, and size represents the size;

[0027] The comparison of the storage threshold and the minimum bandwidth to determine the optimal block combination and data access mode is as follows:

[0028] Set the storage threshold SRAM_TH, the minimum bandwidth minBW, in SRAM tot <=SRAM_TH and BW tot2 <=minBW, that is, if the storage occupancy of the current mode does not exceed the storage threshold SRAM_TH and the total bandwidth is smaller, the minimum bandwidth and data access mode are updated, that is, the optimal block combination and mode are finally obtained to minimize the total bandwidth and meet the storage limit.

[0029] Furthermore, the step 3 specifically includes:

[0030] Step 3.1, traverse the block combination of two adjacent layers of convolution calculation data;

[0031] Step 3.2: Calculate the bandwidth and storage usage of two adjacent layers of different block combinations in a loop according to the ORO mode, set storage constraints, and check whether the calculated storage usage meets the storage constraints in a loop according to the IRO mode. Select the block combination with the smallest total bandwidth.

[0032] Furthermore, the step 3.1 specifically includes:

[0033] For the u-th layer and the u+1-th layer, that is, the two adjacent layers, the feature data of the u-th layer convolution calculation is first divided into blocks to obtain the u-th layer block combination; then the u-th layer block parameters are reused to obtain the u+1-th layer block combination, where Tw(u+1)=Tn(u), Th(u+1)=Tm(u), that is: the width and height of the feature map calculated by the u-th layer convolution are reused as the weight parameters of the feature map calculated by the u+1-th layer convolution.

[0034] Furthermore, the step 4 specifically includes:

[0035] Step 4.1: Optimize each layer individually first, then jointly optimize adjacent layers.

[0036] Step 4.2: Dynamically select the optimal block partitioning strategy by comparing the results of individual optimization and joint optimization.

[0037] Furthermore, the step 4.1 is specifically as follows: each layer is optimized separately by using the single-layer independent block optimization and loop sequence optimization, and adjacent layers are jointly optimized by using the cross-layer joint block optimization and loop sequence optimization.

[0038] The present invention can implement data blocking strategies, loop expansion control strategies, and data storage mapping update strategies for minimum DRAM access, and realize data access and data flow solutions with fine-grained maximum data reuse, and can effectively support automatic code generation and computational mapping optimization for optimizing computation and memory access efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the code design of the three data access modes IRO / ORO / WRO in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the block relationship of an embodiment of the present invention;

[0041] Figure 3 This is a code implementation flow chart of single-layer independent block optimization and loop sequence optimization according to an embodiment of the present invention;

[0042] Figure 4 This is a code implementation flow chart of cross-layer joint block optimization and loop order optimization according to an embodiment of the present invention;

[0043] Figure 5 This is a code implementation flow chart of composite block optimization and loop sequence optimization according to an embodiment of the present invention;

[0044] Figure 6 2 is a diagram comparing the results of separate optimization and joint optimization of the VGG network layer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0046] This embodiment provides a method for optimizing cross-layer memory access bandwidth to support accelerated deep neural networks, specifically including the following:

[0047] Step 1: Design a single-layer block optimization and loop sequence optimization bandwidth model.

[0048] According to the characteristics of deep convolution calculation data flow, three data access modes are determined: input feature coefficient reuse IRO, output feature coefficient reuse ORO and weight coefficient reuse WRO, such as Figure 1 shown.

[0049] According to the characteristics of IRO, ORO and WRO data streams, the single access data volume of the input feature map, output feature map and weight in the three data access modes is determined, specifically including the input feature map coefficients: Bif1, Bif2, Bif3, the output feature map coefficients: Bof1, Bof2, Bof3, and the weight coefficients: Bwgt1, Bwgt2, Bwgt3; the number of cyclic accesses of the input feature map, output feature map and weight in the three data access modes, specifically including the number of input feature map coefficient accesses: Cif1, Cif2, Cif3, the number of output feature map coefficient accesses: Cof1, Cof2, Cof3, and the number of weight coefficient accesses: Cwgt1, Cwgt2, Cwgt3. Among them, the batch cycle block size is set to Tz, the horizontal and vertical block sizes of the data feature map coefficients are Tm*Tn and Th*Tw, the input feature map channel block size is Ti, and the output feature map channel block size is Tj, as shown in Table 1 below.

[0050]

[0051] Table 1: Single access data volume and number of cyclic accesses in the three modes of IRO / ORO / WRO.

[0052] Step 2: Single-layer independent block optimization and loop sequence optimization.

[0053] The goal of optimizing the block and loop order of single-layer convolution calculation is to determine the block parameters Tz, Tm, Tn, Th, Tw, Ti, and Tj. Assume that the maximum batch size is Z, the number of input feature map channels is I, the input feature map coefficient dimension is J*M*N, and the convolution kernel size is I*J*P*Q. Because the current mainstream convolution kernel P*Q is relatively small, generally 3*3, then Tp=P and Tq=Q can be taken. The relationship between the block parameters Tz, Th, Tw, Tm, Tn, Ti, Tj and Z, M, N, I, J is as follows: Figure 2 As shown in the figure, Th=S*(Tm-1)+P, Tw=S*(Tn-1)+Q, and S is the convolution step size.

[0054] In the different modes listed in Table 1, Z, M, N, I, and J are divided into blocks using step sizes of 1, stepM, stepN, stepI, and stepJ to obtain the block parameters Tz, Th, Tw, Tm, Tn, Ti, and Tj. Under different values ​​of Tz, Th, Tw, Tm, Tn, Ti, and Tj, the data size Bif / Bof / Bwgt stored in the computing core and on-chip cache, as well as the number of DRAM cycle reads Cif / Cof / Cwgt are calculated. Assuming that the total on-chip SRAM cache size is SRAM_TH, the required SRAM size and the amount of data read from the DRAM external memory are calculated based on the characteristics of each combined data flow. Then compare the bandwidth consumption of all Tz, Th, Tw, Tm, Tn, Ti, and Tj values ​​that meet the requirement that the overall SRAM consumption does not exceed SRAM_TH, determine the minimum bandwidth case minBW as the optimal block combination, and input different cycle sequence modes mode. The above specific implementation process can be referred to as follows Figure 3 The algorithm function 1 is shown.

[0055] Step 3: Cross-layer joint block optimization and loop order optimization.

[0056] There is a possibility of reuse of coefficients between the output and input features of two adjacent layers. For example, the output feature map coefficient of the u-th layer is the input feature map coefficient of the u+1-th layer. If the adjacent layers are optimized separately in blocks, then after the feature map coefficient of the u-th layer is calculated, a DRAM write operation will be initiated to store these output feature map coefficients. Then, when the next layer of convolution is calculated, a DRAM read operation will be initiated again to read these feature coefficients. This results in redundant feature map coefficient write and read operations.

[0057] If the u-th layer is optimized to ORO mode alone, then when the output feature map coefficients are stored in the on-chip SRAM, the convolution calculation of the u+1 layer is continued to be initiated. Assuming that the u+1 layer loop adopts IRO mode, the feature map coefficients of the previous layer cached in the on-chip SRAM can be reused to the maximum extent. This ORO+IRO combination of two adjacent layers can maximize the reuse between the output and input feature map coefficients.

[0058] Theoretically, input and output feature map coefficients can be reused across multiple layers. However, this requires considering the need to reuse weight coefficients and the trade-offs of batch concurrency. Because on-chip SRAM cache is limited, it's generally difficult to reuse data across more than two layers simultaneously. Therefore, the method in this embodiment only considers data reuse between two adjacent layers.

[0059] The ORO+IRO combination of two adjacent layers means that the weight coefficients of the two adjacent layers need to be cached. The parameters of different network models are different, and not all adjacent layers have better memory bandwidth consumption. First, the overall SRAM consumption must not exceed the available on-chip SRAM budget, and the data reuse maximization constraint must also be met. This optimization selection process can be referred to as follows: Figure 4 In the algorithm function 2 shown, cross_check is used to identify whether cross-layer combination optimization has better data reuse performance than individual optimization.

[0060] Composite block optimization and loop order optimization.

[0061] A deep neural network usually contains many layers, each with different parameters M, N, I, and J. Different parameter combinations of different adjacent layers will lead to different block and loop optimization strategies. Some adjacent layers may be optimized separately, and the results obtained by optimizing the loop sequence IRO / ORO / WRO may be better; some adjacent layers may achieve better performance by optimizing the combination of ORO and IRO. It is necessary to input the parameters of different layers and perform combined optimization / individual optimization comparison on each two adjacent layers to determine the optimization choice for the entire convolutional layer sequence. The implementation strategy can be referred to as follows: Figure 5 The flow of Algorithm 3 is shown.

[0062] This technical solution can make full use of the characteristics of different layers to achieve the best cross-layer block and loop order optimization. Here, taking the VGG network as an example, the single layer optimization results and the composite optimization results are compared. Figure 6 As shown in the figure, comparing the VGG network layer optimization results between individual optimization (Bandwidth no_cross) and joint optimization (BANDWIDTH) in the case of 512K SRAM, approximately 21.19% bandwidth savings are achieved. Each layer first displays the M, N, I, and J parameters, followed by the optimized (Tz, Tm, Tn, Ti, Tj) and loop order mode (1, 2, and 3 correspond to IRO, ORO, and WRO, respectively). As can be seen from the figure, using the optimized method, in some ORI+IRO loop order optimization modes, the overall memory access cost is significantly lower than when optimizing each layer individually, even though each layer also uses its own optimal loop order optimization mode and optimal (Tz, Tm, Tn, Ti, Tj) combination.

[0063] In summary, the content of the present invention is aimed at the structural differences of different neural network models, which are composed of different numbers of network layers. The convolution kernel size configuration of each layer may be different, and the number of input channels and output channels of each layer are also different. According to the different parameters of each layer, the mathematical model of cache consumption and external memory access bandwidth consumption under different combination modes is determined, and it is analyzed that bandwidth and memory consumption are important supports for the selection of optimal blocking and loop sequence strategies. It is also considered that there is a possibility of reuse of output and input feature coefficients between two adjacent layers. Is it better to optimize the ORO+IRO combination of two adjacent layers or to optimize the two layers using separate blocking and loop sequence to minimize the overall bandwidth consumption? Different network model parameters are different, and the results vary greatly. Under the condition that the overall SRAM consumption does not exceed the available SRAM budget on the chip, and at the same time, it is also necessary to meet the data reuse maximization constraint, a cross-layer combination optimization or a separate single-layer optimization strategy is determined to achieve the optimal selection of blocking and loop sequence between multiple layers.

[0064] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the implementation process of the present invention is described in detail above, it is still possible for those familiar with the art to modify the technical solutions described in the above examples or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for optimizing cross-layer memory access bandwidth to support deep neural network acceleration, characterized in that: include: Step 1: Design a single-layer block optimization and loop sequence optimization bandwidth model, specifically including: According to the characteristics of deep convolution calculation data flow, three data access modes are set, including input feature map coefficient reuse IRO mode, output feature map coefficient reuse ORO mode and weight coefficient reuse WRO mode; According to the data flow characteristics of the three data access modes, the input feature map, output feature map and weight are set in each mode for the single access data volume and the number of loop accesses; Step 2: Single-layer independent block optimization and loop sequence optimization, specifically including: Step 2.1: Divide the feature data calculated by the single-layer convolution into blocks and obtain the block parameters; Step 2.2: Traverse all possible block combinations, calculate the memory bandwidth and storage usage under the three data access modes, and compare the storage threshold and minimum memory bandwidth to determine the optimal block combination and data access mode. Step 3: Cross-layer joint block optimization and loop order optimization, specifically including: Step 3.1: Traverse the block combination of two adjacent layers of convolution calculation data, specifically including: For the u-th layer and the u+1-th layer, that is, the two adjacent layers, the feature data of the u-th layer convolution calculation is first divided into blocks to obtain the u-th layer block combination; then the u-th layer block parameters are reused to obtain the u+1 layer block combination, where Tw(u+1)=Tn(u), Th(u+1)=Tm(u), that is: the width Tn and height Tm of the u-th layer convolution calculation feature map are reused as the weight parameters of the u+1 layer convolution calculation feature map: Tw(u+1) and Th(u+1); Step 3.2: Calculate the bandwidth and storage usage of two adjacent layers of different block combinations in a loop according to the ORO mode, set storage constraints, and then traverse in a loop according to the IRO mode to check whether the calculated storage usage meets the storage constraints. Select the block combination with the smallest total bandwidth. Step 4: Composite block optimization and loop sequence optimization, specifically including: Step 4.1: First, optimize each layer individually using the single-layer independent block optimization and loop sequence optimization, and then jointly optimize adjacent layers using the cross-layer joint block optimization and loop sequence optimization; Step 4.2: Dynamically select the optimal block partitioning strategy by comparing the results of individual optimization and joint optimization.

2. The optimization method according to claim 1, wherein: The single access data volume of the input feature map, output feature map and weight in each mode specifically includes: input feature map coefficients: Bif1, Bif2, Bif3, output feature map coefficients: Bof1, Bof2, Bof3, and weight coefficients: Bwgt1, Bwgt2, Bwgt3; the number of cyclic accesses of the input feature map, output feature map and weight in each mode specifically includes: input feature map coefficient access times: Cif1, Cif2, Cif3, output feature map coefficient access times: Cof1, Cof2, Cof3, and weight coefficient access times: Cwgt1, Cwgt2, Cwgt3.

3. The optimization method according to claim 1, wherein: The step 2.1 specifically includes: using the step sizes of 1, stepM, stepN, stepI, and stepJ to divide Z, M, N, I, and J into blocks respectively to obtain block parameters Tz, Th, Tw, Tm, Tn, Ti, and Tj; wherein Z is the maximum batch number, I is the number of input feature map channels, J*M*N is the feature map coefficient dimension, Th=S*(Tm-1)+P, Tw=S*(Tn-1)+Q, S is the convolution step size, I*J*P*Q is the convolution kernel size, and Tp=P and Tq=Q are taken; Tz is the batch cycle block size, Tm*Tn and Th*Tw are the horizontal and vertical block sizes of the data feature map coefficients, Ti is the input feature map channel block size, and Tj is the output feature map channel block size.

4. The optimization method according to claim 3, wherein: In step 2.2, the calculation of the memory bandwidth and storage occupancy is specifically as follows: Bandwidth of input feature map ifmaps: BW if = B if *C if *size(ifmaps); Bandwidth of output feature maps ofmaps: BW of = B of *C of *size(ofmaps); Bandwidth of convolution kernel weight: BW wgt = B wgt *C wgt *size(weight); Storage usage: SRAM tot = B if *size(ifmaps)+B wgt *size(weight)+B of *size(ofmaps); Total bandwidth: BW tot = BW if +BW of +BWwgt; Among them, B if 、B of 、B wgt Represents the data size of input feature map, output feature map and convolution kernel respectively, C if 、C of 、C wgt Respectively represent the number of cyclic visits to the input feature map, output feature map and convolution kernel, and size represents the size; The comparison of the storage threshold and the minimum bandwidth to determine the optimal block combination and data access mode is as follows: Set the storage threshold SRAM_TH, the minimum bandwidth minBW, in SRAM tot <=SRAM_TH and BW tot2 <=minBW, that is, if the storage occupancy of the current mode does not exceed the storage threshold SRAM_TH and the total bandwidth is smaller, the minimum bandwidth and data access mode are updated, that is, the optimal block combination and mode are finally obtained to minimize the total bandwidth and meet the storage limit.

Images