A mapping method and accelerator architecture for implementing deep fusion of complex operators
By dividing the PE array into a first array and a second array and adding hard connections, parallel computation of convolution operator data blocks was achieved, solving the problem of excessive bandwidth pressure in the accelerator and improving execution efficiency and performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2023-02-02
- Publication Date
- 2026-07-03
Smart Images
Figure CN116070678B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of convolutional neural networks, and in particular to a mapping method and accelerator architecture for achieving deep fusion of complex operators. Background Technology
[0002] Today, convolutional neural networks (CNNs) outperform humans in both speed and accuracy for a wide range of complex tasks. While graphics processing units (GPUs) and tensor processing units (TPUs) can effectively accelerate deep learning networks (DNNs), acceleration on embedded devices becomes crucial when DNNs are deployed in fields such as medical computing or vehicle automation. Many dedicated DNN accelerator designs have been proposed, and different accelerators have varying processing times for neural networks, adapting to different application scenarios. The faster the inference speed of an accelerator, the more applications it can be suitable for.
[0003] Existing accelerators cannot effectively address the problem of excessive on-chip bandwidth pressure. Compared to operator-separate computation, fusion techniques can eliminate access to off-chip memory and reduce off-chip bandwidth pressure, but intermediate data still accesses buffers within the accelerator. While simple operator fusion techniques eliminate buffer access energy consumption, their applicability is limited and they cannot achieve fusion between complex operators. Cross-convolution operator fusion techniques can achieve the fusion of complex operators, but their fusion granularity is insufficient, and a large number of buffer accesses still exist within the accelerator. Summary of the Invention
[0004] To address the aforementioned issues, a mapping method and accelerator architecture for deep fusion of complex operators are proposed. By dividing the PE array of the architecture into two parts, a first array and a second array, parallel computation of data blocks of two convolution operators in the subgraph can be achieved, completely eliminating the storage access of intermediate data. Furthermore, the convolution operator data blocks in the subgraph can be computed in parallel with balanced time, improving the execution efficiency of this type of subgraph. Moreover, by adding new hard connections to the PE array, the overall execution time of the subgraph is further reduced, improving the performance of the subgraph.
[0005] The first aspect of this application proposes a mapping method for achieving deep fusion of complex operators, including:
[0006] The input feature map information is obtained, and the PE array is divided to determine the optimal division result of the PE array.
[0007] Based on the execution flow of the PE array and the optimal partitioning result, the configuration information is determined;
[0008] Based on the configuration information, the target execution of the PE array is achieved.
[0009] Optionally, the step of acquiring input feature map information, dividing the PE array, and determining the optimal division result of the PE array includes:
[0010] Based on the input feature map information, determine the output data of the second convolution operator. The formula is as follows:
[0011] ,
[0012] ,
[0013] in, and These are the height and width of the output data of the second convolution operator, respectively. These are the height and width of the input feature map, respectively. The size of the convolution kernel of the first convolution operator. The size of the convolution kernel of the second convolution operator;
[0014] Traverse the output data Determine the output data blocks of the second convolution operator. ,in, And in When the time comes, the traversal ends;
[0015] Divide the output data of the second convolution operator into blocks. Determine the shape of the first array With the first execution time The formula is as follows:
[0016] ,
[0017] ,
[0018] ,
[0019] in, Let represent the height and width of the first array, respectively, and satisfy . Otherwise, re-traverse the output data. ,in, The size of the PE array;
[0020] According to the shape of the first array Determine the shape of the second array With the second execution time ;
[0021] According to the first execution time With the second execution time Determine the total execution time The formula is as follows:
[0022] ;
[0023] Traversing parameters All possible values up to the total execution time Minimum, wherein, in the total execution time When it is at its minimum, the first array With the second array This is the optimal partitioning result.
[0024] Optionally, the shape of the first array Determine the shape of the second array With the second execution time ,include:
[0025] Initialize the height of the second array And iterate through them until... ;
[0026] Determine the number of PEs in the second array. The formula is as follows:
[0027] ,
[0028] in, , The width of the second array is given by [value]. satisfy ;
[0029] According to the shape of the second array Determine the second execution time The formula is as follows:
[0030] ,
[0031] in, is the number of iterations for the second array.
[0032] Optionally, the execution flow based on the PE array, according to the optimal partitioning result, determines the configuration information, including:
[0033] Based on the optimal partitioning result, allocate to the first array One bank is allocated for the second array. One bank;
[0034] from The input feature map is read from a bank to determine the execution data stream of the first array, and the output of the first array is determined therefrom.
[0035] The output of the first array is input into the second array to determine the optimal execution flow of the second array and to determine the output of the second array;
[0036] Write back the output of the second array in blocks. Each bank, and calculates each bank in the second array. After outputting one data point, the horizontal leftward pulse is stored in [the specified location]. In each bank.
[0037] Optionally, the from The process involves reading blocks of the input feature map from each bank, determining the execution data stream of the first array, and thereby determining the output of the first array, including:
[0038] The data is filled with horizontal pulses, so that each PE in the first array can acquire input data;
[0039] After the input data is filled, the first weight value in the first array is broadcast, and all PEs calculate the corresponding partial sum;
[0040] In the following During the frame, each PE obtains new input data from the PE to its left and multiplies and accumulates it with the next weight value broadcast.
[0041] Repeat the above 3 steps Next, to process the remaining Row weights are used to determine the first array. Output data.
[0042] Optionally, the method further includes:
[0043] When filling data in a horizontal pulsating manner, each PE gets data from the leftmost PE, the leftmost PE column gets data from banks, and each time a new row weight is calculated, each leftmost PE will access the next bank, while the last PE returns to access the first bank.
[0044] Optionally, the step of inputting the output of the first array into the second array, determining the optimal execution flow of the second array, and thereby determining the output of the second array includes:
[0045] The output of the first array is pulsed to the second array, and at the last beat of the pulsed operation, the first weight of the last row of the second convolution operator kernel is broadcast, and PE performs a multiply-accumulate operation.
[0046] In the following During the pulse, the data in the second array pulses to the right, and the remaining data in that row of the second convolution operator kernel is calculated. The partial sums are used to complete the calculation of the partial sums corresponding to the last row of the second convolution operator's convolution kernel;
[0047] For the remaining The second array uses a row convolution kernel. The PE (Precision Output) of the second array obtains input data from the previous row and the weight data of the next-to-last row broadcast. This completes one iteration of the second array, and the output is... One data point;
[0048] pass The second iteration determines the second array. Output data.
[0049] The second aspect of this application proposes an accelerator architecture for deep fusion of complex operators, comprising a main controller, a configuration memory, a data memory, and a PE array, wherein...
[0050] The main controller is used to control the execution of the architecture;
[0051] The configuration memory is used to store the configuration information of the PE array for each cycle;
[0052] The data storage device is used to store the input data and output data of the PE array;
[0053] The PE array consists of multiple PEs arranged in a two-dimensional array. The connection methods between the PEs include mesh topology, secondary diagonal and lower PE connections, and the PEs are configured to perform different operations according to the configuration information.
[0054] Optionally, the data storage includes:
[0055] A first weight buffer is used to store the weight data of the first convolution operator and to broadcast the weights of the first convolution operator.
[0056] The second weight buffer is used to store the weight data of the second convolution operator and to broadcast the weights of the second convolution operator.
[0057] A feature map buffer is used for communication with the PE array via a fully interconnected crossbar.
[0058] Optionally, the feature map buffer includes an input buffer and an output buffer, wherein,
[0059] The input buffer is fully interconnected, wherein the front Each bank is divided into the input buffer, and All PEs are fully interconnected;
[0060] The output buffer is a direct connection, wherein, after Each bank is divided into the output buffers, and each is connected to... One PE is directly connected.
[0061] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:
[0062] By dividing the PE array of the architecture into two parts, the parallel computation of the data blocks of the two convolution operators in the subgraph can be realized, completely eliminating the storage access of intermediate data, and realizing the parallel computation of the data blocks of each convolution operator in the subgraph in a balanced time, thereby improving the execution efficiency of this type of subgraph. Furthermore, by adding new hard connections to the PE array, the overall execution time of the subgraph can be further reduced, thereby improving the performance of the subgraph.
[0063] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0064] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0065] Figure 1 This is a flowchart illustrating a mapping method for implementing deep fusion of complex operators according to an exemplary embodiment of this application;
[0066] Figure 2 This is a general flowchart illustrating a mapping method for implementing deep fusion of complex operators according to an exemplary embodiment of this application;
[0067] Figure 3 This is a schematic diagram illustrating the partitioning results of the first and second arrays according to an exemplary embodiment of this application;
[0068] Figure 4 It is a database that stores input data according to the exemplary embodiments of this application;
[0069] Figure 5 This is a schematic diagram illustrating the result of filling the first array with data according to an exemplary embodiment of this application;
[0070] Figure 6 This is a schematic diagram illustrating the calculation results of the first array according to an exemplary embodiment of this application;
[0071] Figure 7 This is a schematic diagram illustrating the result of the first iteration of pulsating filling data for the second array according to an exemplary embodiment of this application;
[0072] Figure 8 This is an architectural diagram illustrating an accelerator architecture for implementing deep fusion of complex operators according to exemplary embodiments of this application;
[0073] Figure 9 This is a schematic diagram of a PE structure shown according to an exemplary embodiment of this application. Detailed Implementation
[0074] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0075] Figure 1 This is a flowchart illustrating a mapping method for implementing deep fusion of complex operators according to an exemplary embodiment of this application, such as... Figure 1 As shown, the method includes:
[0076] Step 101: Obtain the input feature map information, divide the PE array, and determine the optimal division result of the PE array.
[0077] First, this method only applies to convolution operators with both input and output channels of 1, and a stride of 1. The data stream in this invention is a fixed output data stream, meaning each PE calculates one output value, i.e.:
[0078] .
[0079] like Figure 2 As shown below, the process of determining the optimal partitioning result of the PE array will be described in detail.
[0080] The first step is to calculate the subgraph output shape (X,Y), that is, to determine the output data of the second convolution operator based on the input feature map information. The formula is as follows:
[0081] ,
[0082] ,
[0083] in, and These represent the height and width of the output data from the second convolution operator, respectively. These represent the height and width of the input feature map, respectively. The size of the convolution kernel of the first convolution operator. This represents the size of the convolution kernel of the second convolution operator.
[0084] The second step is to iterate through the output data. Determine the output data blocks of the second convolution operator. ,in, And in When the time is right, the traversal ends.
[0085] The traversal process is as follows: initialization In the subsequent traversal calculations, First, increment the step size by 1. hour, Then increment by 1 step until... The traversal is complete.
[0086] The third step is to divide the data into blocks based on the output data of the second convolution operator. Determine the shape of the first array With the first execution time The formula is as follows:
[0087] ,
[0088] ,
[0089] ,
[0090] in, Let represent the height and width of the first array, respectively, and satisfy . Otherwise, re-examine the output data. ,in, The size of the PE array;
[0091] Fourth step, based on the shape of the first array Determine the shape of the second array With the second execution time .
[0092] The specific process is as follows:
[0093] Initialize the height of the second array In subsequent traversals, increment by a step size of 1 until... ;
[0094] Determine the number of PEs in the second array. The formula is as follows:
[0095] ,
[0096] in, , This is the width of the second array, since the output data width of the second array is also the array width. It cannot be greater than the output data width of the first array. ,but satisfy ;
[0097] According to the shape of the second array Determine the second execution time The formula is as follows:
[0098] ,
[0099] in, is the number of iterations for the second array.
[0100] In this embodiment of the application, since the output data that the second array needs to calculate is Therefore, the second array requires multiple iterations, with the number of iterations being [number missing]. .
[0101] Fifth step, based on the first execution time With the second execution time Determine the total execution time The formula is as follows:
[0102] .
[0103] In this embodiment of the application, since the first array and the second array are executed in a pipelined manner, the maximum value is selected for the final execution time of the array.
[0104] Step 6: Iterate through the parameters All possible values up to the total execution time Minimum, where, in total execution time When it is at its minimum, the first array With the second array This is the optimal partitioning result.
[0105] In this embodiment of the application, the optimal partitioning result of the first array and the second array is achieved through the above steps.
[0106] Step 102: Determine the configuration information based on the execution flow and optimal partitioning results of the PE array.
[0107] like Figure 2 As shown in the embodiments of this application, the process of determining the configuration information is as follows.
[0108] The first step is to allocate resources to the first array based on the optimal partitioning result. One bank is allocated for the second array. A bank.
[0109] in, Each bank and the first array The PE (Programmable Array) is fully interconnected and used to input feature map blocks. Banks and the second array PE direct connection, transmit directly row by row Output data in rows.
[0110] The second step, from The input feature map is read from blocks in each bank to determine the execution data stream of the first array, and thus the output of the first array is determined.
[0111] Specifically, the input data for the first array comes from... To prevent memory access conflicts, each PE in the first array accesses one bank at a time. Since each PE needs to complete [the necessary steps] with [other banks]... Row convolution kernels are used, and each bank stores only one row of input data, with the remaining rows... Rows of data are stored sequentially in subsequent banks. Specifically, when the last bank is reached, the next row of data is stored starting from the first bank, ensuring that consecutive rows of input data are stored in adjacent banks. Throughout this process, to complete the convolution calculation, each row of the first array's PE needs to be accessed sequentially. A bank.
[0112] In addition, for both the first and second arrays, the execution process is divided into two parts: filling data and calculating multiplication and accumulation.
[0113] For the first array, the process of determining the execution data flow for the first array is as follows:
[0114] The data is filled with horizontal pulses, so that each PE in the first array can acquire input data;
[0115] After the input data is filled, the first weight value in the first array is broadcast, and all PEs calculate the corresponding partial sum;
[0116] In the following During the frame, each PE obtains new input data from the PE to its left and multiplies and accumulates it with the next weight value broadcast.
[0117] Repeat the above 3 steps Next, to process the remaining Row weights are used to determine the first array. Output data.
[0118] In the horizontal pulsating data filling process, each PE obtains data from the leftmost PE, the leftmost PE column obtains data from banks, and each time a new row weight is calculated, each leftmost PE will access the next bank, while the last PE returns to access the first bank.
[0119] The third step is to input the output of the first array into the second array, determine the optimal execution flow of the second array, and determine the output of the second array.
[0120] In this embodiment of the application, the size of the second array is It requires multiple iterations to calculate Only one output data. In each iteration, only the PE column is used in the calculation. The rest Columns are used to store the remaining values in each row of the convolution kernel. Input data multiplied by weights.
[0121] The iteration process of the second array is as follows: the output of the first array is pulsed to the second array, and at the last beat of the pulsed operation, the first weight of the last row of the second convolution operator kernel is broadcast, and PE performs multiply-accumulate operation;
[0122] In the following During the pulse, the data in the second array pulses to the right, and the remaining data in that row of the second convolution operator kernel is calculated. The partial sums are used to complete the calculation of the partial sums corresponding to the last row of the second convolution operator's convolution kernel;
[0123] For the remaining The second array uses a convolutional kernel. The PE (Peer Equation) of the second array receives input data from the previous row and then receives the weight data from the next row after the previous row. This completes one iteration of the second array, and the output is... Data points.
[0124] After determining the data for the first iteration of the second array, the same steps are followed to determine the second array. Output data.
[0125] The fourth step is to write back the output of the second array in blocks. Each bank, and in the second array, calculates... After outputting one data point, the horizontal leftward pulse is stored in [the specified location]. In each bank, this movement delay will be hidden in the next iteration calculation.
[0126] Therefore, the entire process of generating configuration information has been explained.
[0127] Step 103: Based on the configuration information, execute the target of the PE array.
[0128] like Figure 2 As shown in the embodiments of this application, according to the above-mentioned array partitioning and configuration information process, configuration information has been generated in the configuration memory of the target architecture. Each PE obtains its own configuration information from the configuration memory, thereby realizing the standard execution of the array.
[0129] To illustrate the specific implementation method of the present invention in detail, a specific embodiment is presented.
[0130] The input feature map information for this specific embodiment is: The hardware parameters are ,in, The number of target architecture banks.
[0131] according to Figure 2 The flowchart shown shows that the output shape of the subgraph is as follows: The output data block size is The input data block size is The size of the first array is The size of the second array is ,like Figure 3 As shown, the array width is only 8, so when At that time, the width is extended from the adjacent PE.
[0132] Among them, such as Figure 3 As shown, the first array is the 5×9 yellow line section, and the second array is the 1×9 orange line section.
[0133] The following details the storage method of the input feature map, the calculation process of the first convolution operator block and the second convolution operator block, and finally the calculation time of the sub-map.
[0134] First, we will explain in detail how the input feature map is stored.
[0135] The input data for the subgraph is divided into blocks ( For storage, if every 11 data items are stored in the same bank for easy row-by-row access by the PE, then 7 banks are needed. But... The corresponding input buffer has only 5 banks. Data is stored using a modulo operation, meaning the first 5 rows of data are stored sequentially in the 5 banks, and the remaining two rows are stored sequentially in the first two banks, such as... Figure 4 As shown. Because there is overlap between adjacent blocks, in order to reuse the overlapping data of the previous block, the new block ( It also starts storing from the first bank, instead of continuing from the third bank.
[0136] Next, the calculation process of the first convolution operator block division will be explained in detail.
[0137] Step 1: First, horizontally fill the data. In the first array, rows 1, 2, 3, 4, and 5, PE access banks 1, 2, 3, 4, and 5 respectively. After 9 cycles, the data filling is complete, and the filling effect is as follows. Figure 5As shown in yellow, while filling the last column of data, the first convolution operator broadcasts the first row and first column weight value of 1, and each PE calculates the first partial sum.
[0138] Step 2: In the next two cycles, each PE obtains the next input data from the PE to its left. The leftmost PE retrieves data from the bank, multiplies the new input data by the newly broadcast weight values (2 and 3 in sequence), and accumulates them to the partial sum. At this point, the weights (1, 2, 3) of the first row have been calculated.
[0139] Step 3: Next, calculate the weights of the second row (4, 5, 6). First, fill the PE array: PEs in rows 1, 2, 3, and 4 access banks 2, 3, 4, and 5 respectively, and PE in row 5 accesses bank 1. At the end of the filling, broadcast weight 4. Each PE updates its partial sum. The PE array shifts right once per pulse, broadcasting the new weight. In this example, there are 2 pulses, broadcasting weights 5 and 6 respectively. The partial sum is updated by multiplying and accumulating.
[0140] Step 4: Next, calculate the weights of the last row (7, 8, 9). The calculation process is the same as in Step 3: fill first and then pulsate, but the banks visited also need to be changed accordingly: PEs in rows 1, 2, and 3 visit banks 3, 4, and 5 respectively, while PEs in rows 4 and 5 visit banks 1 and 2 respectively.
[0141] After executing the above four steps, each PE in the first array now contains output data, totaling 5×9 data points, as follows: Figure 6 The orange numbers in the middle indicate the output data. This output data is the input data for the second convolution operator, and it is stored in the register file in the first array PE for use by the second convolution operator.
[0142] Therefore, the first array calculates the block size as follows: The time for outputting data is .
[0143] Then, the calculation process of the second convolution operator block is explained in detail.
[0144] The second convolution operator is assigned 1×9 PEs using a mapping algorithm. However, the output block size of the input data block (5×9) convolved by a 2×2 convolution kernel is (4×8). Therefore, the second array needs to iterate 4 times, calculating 1×8 output results each time. To minimize the data systolic period, the output data is calculated starting from the last row of the convolution kernel. The specific calculation steps are shown below.
[0145] Step 1: Vertically fill the second array with input data. Pulse the data in the second row of PE to the sixth row. Simultaneously, during this pulsation, all data in the first array pulsates downwards. At this point, the fifth row of PE has obtained the data from the first row of PE register file, such as... Figure 7 As shown. When the second array PE receives the input data, it is multiplied by the weight 3 broadcast by the second convolution kernel to obtain a partial sum.
[0146] Step 2: The data in the second array is shifted one clock cycle to the right, and weight 4 is broadcast, updating the partial sum. At this point, the weight (3,4) of the last row has been calculated.
[0147] Step 3: Obtain the input data calculated with the weights (1, 2) of the first row. From Figure 7 As can be seen from the diagram, the data needed for each row of the second array's PE is found in the previous row's PE. Therefore, a single pulse is sufficient to obtain the input data, which is then multiplied by the broadcast weight 1. Next, the second array pulses to the right once to obtain new input data, which is then multiplied by weight 2. At this point, the array has completed one iteration of calculation, producing 1×8 data points. These data points are then pulsed left and transmitted to the directly connected output bank.
[0148] Step 4: Next, the second array begins its second iteration. This is consistent with the previous three steps, except that the initial data filling involves pulsating the data from row 3 to row 6 (PE). First, calculate the convolution kernels (3, 4) in the last row, then obtain new input data from the previous row's PE and calculate it with the previous row's convolution kernels (1, 2).
[0149] Step 5: The second array undergoes a third iteration calculation. This is consistent with steps one through three, except that the ultimate goal of filling the data is to pulsate the data from row 4 to row PE in row 6.
[0150] Step Six: The second array undergoes its fourth iteration. This is consistent with steps one through three, except that the ultimate goal of filling the data is to pulsate the data from row 5 to row 6 (PE).
[0151] Through the above six steps, the second array can directly process the output data of the first array, and only write the data block results of the subgraph back to the bank.
[0152] Therefore, the second array calculates the block size as follows: The time for outputting data is .
[0153] Finally, the computation time for the subgraph is calculated.
[0154] By processing the remaining data blocks of the subgraph using the steps described above, a complete output feature map can be obtained. The total execution time for the subgraph is [time missing]. .
[0155] In this embodiment of the application, by dividing the PE array of the architecture into two parts, the parallel computation of the data blocks of the two convolution operators in the subgraph can be realized, the storage access of intermediate data can be completely eliminated, and the data blocks of each convolution operator in the subgraph can be computed in parallel with balanced time, thereby improving the execution efficiency of this type of subgraph.
[0156] Figure 8 This is an architectural diagram illustrating an accelerator architecture for implementing deep fusion of complex operators according to exemplary embodiments of this application, such as... Figure 8 As shown, the architecture includes a main controller, configuration memory, data memory, and PE array, wherein...
[0157] The main controller is used to control the execution of the architecture;
[0158] The configuration memory is used to store the configuration information of the PE array for each cycle;
[0159] The data storage device is used to store the input and output data of the PE array;
[0160] The PE array consists of multiple PEs arranged in a two-dimensional array. The connection methods between PEs include mesh topology, secondary diagonal and lower PE connections, and PEs are configured to perform different operations according to the configuration information.
[0161] Optional, the data storage includes:
[0162] The first weight buffer is used to store the weight data of the first convolution operator and to broadcast the weights of the first convolution operator.
[0163] The second weight buffer is used to store the weight data of the second convolution operator and to broadcast the weights of the second convolution operator.
[0164] Feature map buffer for communication with the PE array via a fully interconnected crossbar.
[0165] Optionally, the feature map buffer includes an input buffer and an output buffer, wherein,
[0166] The input buffer is fully interconnected, where the first Each bank is divided into input buffers, and All PEs are fully interconnected;
[0167] The output buffer is directly connected, where, Each bank is divided into output buffers, and each is connected to... One PE is directly connected.
[0168] In this embodiment, the PE array consists of A×A PEs, and each PE comprises the following parts: a multiply-accumulate calculation unit, a partial sum register, a register file for storing output results, a register for storing transmitted data, and a configuration buffer, etc. Figure 9 As shown. The PE's function can be configured to perform different operations using configuration information. The multiply-accumulate calculation unit can perform multiply-accumulate operations in one cycle. It multiplies the input data from MUX A with the weights from MUX B, adds the partial sum from the partial sum register, updates the partial sum, and puts it back into the partial sum register. When an output data is calculated, it is saved to the register file. In addition, the PE has a small register for storing data transferred from the first array to the second array.
[0169] Secondly, the feature map buffer is partitioned based on the PE array. Each subarray requires a number of banks equal to its row height; therefore, the first... Each bank is divided into input buffers, and All PEs are fully interconnected; next... Each bank is divided into output buffers. One PE is directly connected.
[0170] In this embodiment, by dividing the PE array of the architecture into two parts, a first array and a second array, parallel computation of the data blocks of the two convolution operators in the subgraph is achieved. Furthermore, by adding new hard connections to the PE array, the overall execution time of the subgraph is further reduced, and the performance of the subgraph is improved.
[0171] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A mapping method for achieving deep fusion of complex operators, characterized in that, include: The input feature map data of the convolutional neural network is acquired, and the processing unit array (PE array) in the accelerator is divided according to the input feature map data to determine the optimal division result of the PE array. The PE array is divided into a first array for performing a first convolution operator and a second array for performing a second convolution operator. The division process is as follows: the output data size of the second convolution operator is determined according to the size of the input feature map and the kernel size of the two-level convolution operators; the output data of the second convolution operator is traversed and divided to determine different output data block methods. Based on the output data block method, determine the shape of the first array and the corresponding first execution time, and determine whether the first array meets the resource constraints of the PE array. When the resource constraints are not met, the output data is re-traversed and divided; based on the shape of the first array, the shape of the second array and the corresponding second execution time are further determined. Based on the first execution time and the second execution time, the corresponding total execution time is determined; different output data block methods and second array partitioning methods are traversed, and the total execution time corresponding to each group of partitioning results is compared. When the total execution time is the minimum, the corresponding first array and second array are determined as the best partitioning result of the PE array. Based on the execution data flow of the PE array and the optimal partitioning result, the configuration information of the PE array is generated, including: allocating resources to the first array according to the optimal partitioning result. One bank is allocated for the second array. A bank; from The input feature map is read in blocks from each bank to determine the execution data flow of the first array, and thereby determine the output of the first array; the output of the first array is input into the second array to determine the optimal execution flow of the second array, and thus determine the output of the second array; the output blocks of the second array are written back. Each bank, and calculates each bank in the second array. After outputting one data point, the horizontal leftward pulse is stored in [the specified location]. Among the banks; of which, The height of the first array, For the height of the second array, This is the width of the second array; According to the configuration information, the PE array in the convolutional neural network accelerator is controlled to perform convolution calculations, so that the output data of the first convolution operator is directly input into the second convolution operator for calculation, thereby completing the target execution of the PE array.
2. The method according to claim 1, characterized in that, The process of acquiring the input feature map data of the convolutional neural network, dividing the PE array, and determining the optimal division result of the PE array includes: Based on the input feature map data, determine the output data of the second convolution operator. The formula is as follows: ， ， in, and These are the height and width of the output data of the second convolution operator, respectively. These are the height and width of the input feature map, respectively. The size of the convolution kernel of the first convolution operator. The size of the convolution kernel of the second convolution operator; Traverse the output data Determine the output data blocks of the second convolution operator. ,in, And in When the time comes, the traversal ends; Divide the output data of the second convolution operator into blocks. Determine the shape of the first array With the first execution time The formula is as follows: ， ， ， in, Let represent the height and width of the first array, respectively, and satisfy . Otherwise, re-traverse the output data. ,in, The size of the PE array; According to the shape of the first array Determine the shape of the second array With the second execution time ; According to the first execution time With the second execution time Determine the total execution time The formula is as follows: ； Traversing parameters All possible values up to the total execution time Minimum, wherein, in the total execution time When it is at its minimum, the first array With the second array This is the optimal partitioning result.
3. The method according to claim 2, characterized in that, According to the shape of the first array Determine the shape of the second array With the second execution time ,include: Initialize the height of the second array And iterate through them until... ; Determine the number of PEs in the second array. The formula is as follows: ， in, , The width of the second array is given by [value]. satisfy ; According to the shape of the second array Determine the second execution time The formula is as follows: ， in, is the number of iterations for the second array.
4. The method according to claim 3, characterized in that, The from The process involves reading blocks of the input feature map from each bank, determining the execution data stream of the first array, and thereby determining the output of the first array, including: The data is filled with horizontal pulses, so that each PE in the first array can acquire input data; After the input data is filled, the first weight value in the first array is broadcast, and all PEs calculate the corresponding partial sum; In the aftermath During the frame, each PE obtains new input data from the PE to its left and multiplies and accumulates it with the next weight value broadcast. Repeat the above 3 steps Next, to process the remaining Row weights are used to determine the first array. Output data.
5. The method according to claim 4, characterized in that, The method further includes: When filling data in a horizontal pulsating manner, each PE gets data from the leftmost PE, the leftmost PE column gets data from banks, and each time a new row weight is calculated, each leftmost PE will access the next bank, while the last PE returns to access the first bank.
6. The method according to claim 5, characterized in that, The step of inputting the output of the first array into the second array, determining the optimal execution flow of the second array, and thereby determining the output of the second array includes: The output of the first array is pulsed to the second array, and at the last beat of the pulsed operation, the first weight of the last row of the second convolution operator kernel is broadcast, and PE performs a multiply-accumulate operation. In the aftermath During the pulse, the data in the second array pulses to the right, and the remaining data in that row of the second convolution operator kernel is calculated. The partial sums are used to complete the calculation of the partial sums corresponding to the last row of the second convolution operator's convolution kernel; For the remaining The second array uses a row convolution kernel. The PE (Precision Output) of the second array obtains input data from the previous row and the weight data of the next-to-last row broadcast. This completes one iteration of the second array, and the output is... One data point; pass The second iteration determines the second array. Output data.
7. An accelerator architecture applied to the method of any one of claims 1-6, characterized in that, It includes a main controller, configuration memory, data memory, and PE array, among which, The main controller is used to control the execution of the architecture; The configuration memory is used to store the configuration information of the PE array for each cycle; The data storage device is used to store the input data and output data of the PE array; The PE array consists of multiple PEs arranged in a two-dimensional array. The connection methods between the PEs include mesh topology, secondary diagonal and lower PE connections, and the PEs are configured to perform different operations according to the configuration information.
8. The architecture according to claim 7, characterized in that, The data storage device includes: A first weight buffer is used to store the weight data of the first convolution operator and to broadcast the weights of the first convolution operator. The second weight buffer is used to store the weight data of the second convolution operator and to broadcast the weights of the second convolution operator. A feature map buffer is used for communication with the PE array via a fully interconnected crossbar.
9. The architecture according to claim 8, characterized in that, The feature map buffer includes an input buffer and an output buffer, wherein... The input buffer is fully interconnected, wherein the front Each bank is divided into the input buffer, and All PEs are fully interconnected; The output buffer is a direct connection, wherein, after Each bank is divided into the output buffers, and each is connected to... One PE is directly connected.
Citation Information
Patent Citations
Deep learning convolution operation implementation method based on systolic array hardware architecture
CN111506343A
Configurable universal convolutional neural network accelerator
CN113592068A