A generalized architecture design method for deep learning processors

Abstract

A generalized architecture design method for a deep learning processor, the present invention relates to a generalized architecture design method for a deep learning processor. The purpose of the present invention is to solve the problem that the existing deep learning processor is not compatible with other computationally intensive tasks other than deep learning in the intelligent IoT scenario, resulting in an increase in chip area, an increase in cost, a deviation in utilization, and low computational energy efficiency. The process is: the architecture includes a general-purpose processor for task parsing and a deep learning accelerator for accelerated computing. When performing a deep learning convolution computing task, the general-purpose processor parses the organization form of the data outside the chip, loads the data into the on-chip cache Buffer, and the sequence controller loads the data into the MAC array in the convolution order for calculation. When performing intensive computing tasks, a reordering module is added to reorder the Buffer data into the on-chip cache Buffer2 based on the optimal operator calculation form, and transmit the Buffer2 data to the MAC array for calculation. The present invention is used in the field of deep learning processors.

Description

Technical Field

[0001] The present invention relates to a generalized architecture design method for a deep learning processor. Background Art

[0002] In order to meet the performance requirements of deep learning in application fields such as unmanned driving and face recognition, dedicated deep learning processors provide significantly more energy-efficient deep learning processing than general-purpose processors, becoming one of the key computing engines of modern computing systems and being widely integrated into end, edge, and cloud computing systems. However, in many actual application scenarios, especially smart IoT scenarios, applications include not only deep learning computing tasks, but also many other computing-intensive tasks, such as signal processing and data analysis. Customizing dedicated accelerators for each different type of task, although theoretically can achieve higher computing energy efficiency, will also bring many side effects. On the one hand, more heterogeneous accelerators will directly lead to an increase in chip area, while increasing the demand and complexity of system interconnection design, thereby increasing chip costs. On the other hand, under different application configurations, differences in computing requirements will inevitably lead to deviations in the utilization of dedicated accelerators. However, each dedicated accelerator is often independent, and many of the accelerator's hardware resources, such as on-chip cache and computing arrays, occupy a large chip area and power consumption, but cannot be reused across accelerators, which will also damage the chip's computing energy efficiency and cost. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that existing deep learning processors are not compatible with other computationally intensive tasks other than deep learning in smart IoT scenarios, resulting in increased chip area, increased cost, utilization deviation, and low computing energy efficiency, and to propose a generalized architecture design method for deep learning processors.

[0004] A generalized architecture design method for a deep learning processor includes the following steps:

[0005] Step 1: The system is divided into software part and hardware part;

[0006] The software part is the general-purpose processor, which is responsible for programming the hardware;

[0007] The hardware part is the deep learning accelerator, which is responsible for hardware calculations;

[0008] General-purpose processors include deep learning convolutional computing tasks and intensive computing tasks;

[0009] The general-purpose processor reads the data and analyzes the organization of the data in the off-chip memory;

[0010] When performing intensive computing tasks, go to step 2;

[0011] When performing deep learning convolution calculation tasks, go to step 3;

[0012] Step 2: Analyze the intensive computing task to obtain the optimal operator computing form, load the organization form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, reorder the data in the on-chip cache Buffer based on the optimal operator computing form, and load the reordered data into the on-chip cache Buffer2;

[0013] The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory;

[0014] Step 3: Load the data in step 1 in the off-chip memory into the on-chip buffer, transfer the data in the buffer to the MAC array, and transfer the output result of the MAC array to the off-chip memory.

[0015] The beneficial effects of the present invention are:

[0016] The present invention improves the deep learning processor and proposes a generalized architecture design method for the deep learning processor, so that the data on the cache is rearranged on the chip through instructions and sent to the multiplication and addition array for calculation according to the basic calculation operator definition.

[0017] The present invention takes the widely used deep learning processor as the core and attempts to generalize its computing power so that it can add more similar common computing tasks such as signal processing and data analysis on the basis of supporting deep learning processing. On the premise of ensuring the computing performance of deep learning processing, it can be shared by a variety of different computing tasks, improve the hardware resource utilization of dedicated processors, and reduce the area and cost of smart IoT chips.

[0018] The present invention generalizes the memory access function of the deep learning processor under the premise of minimizing the increase in hardware overhead, and realizes more types of intensive calculations. While supporting the fixed-bit-width deep learning convolution calculations supported by the original hardware, the present invention also supports variable-bit-width deep learning convolution calculations, supports variable-bit-width calculations that can be converted to vector-level and matrix-level multiplication and addition types, and supports variable-bit-width calculations that can splice and fill data between layers when multiple layers are nested. Compared with general-purpose processors, its computing speed is improved, and compared with dedicated processors, the corresponding hardware overhead is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Design architecture diagram for generalization;

[0020] Figure 2 It is a block diagram of variable bit width multiplication and addition concatenation calculation;

[0021] Figure 3It is a control block diagram of a variable bit width multiplication-addition concatenation sequence, Wt is the abbreviation of weight weight of convolution, da is the abbreviation of data data of convolution, p0 is the multiplication result of mul0 multiplier, p1 is the multiplication result of mul1 multiplier, p2 is the multiplication result of mul2 multiplier, and P3 is the multiplication result of mul3 multiplier;

[0022] Figure 4 The hardware structure diagram of programmable memory access, prog_addr_start is the starting address of programmable memory access; prog_burst_lenth is the burst transmission length, that is, the amount of data when sending consecutive addresses. For example, if the starting address is 10 and the burst length is 3, the data of the three addresses 101112 will be transmitted; DQclk is a D flip-flop, which is the basic unit structure of the hardware circuit; rearranged_en is the enable of memory rearrangement;

[0023] Figure 5 The rearranged_code is the decoded coordinate of the rearrangement, code is the bit segment, code is 7 bits, the upper three bits select which data, and the lower four bits select a part of the data; rearranged_code_en is the decoding enable, only when it is valid, the data at this position will be rearranged; zero is 0;

[0024] Figure 6 The hardware structure diagram for filling in the unit, padding value is the value to be filled, such as the 1 in the fft just now; reg_bit is the variable bit width selection, such as 4bit 8bit or 16bit; padding en is the padding enable; Dat_in is the input data; Dat_out is the output data;

[0025] Figure 7a Loading timing diagram of cache data for deep learning accelerator;

[0026] Figure 7b This is the timing diagram for deep learning accelerator MAC array data calculation;

[0027] Figure 8 The vector transformation diagram for FFT intensive calculation, W r N represents the rotation factor, x m (p) represents one of the input data of a set of butterfly operations, x m (q) represents a set of butterfly operations on another input data, x m+1 (p) represents one of the output data of a set of butterfly operations, x m+1 (q) represents another output data of a set of butterfly operations;

[0028] Fig. 9 is a 16-point FFT butterfly operation diagram, W N n represents the rotation factor, Xm represents the input signal data, () represents the corresponding coordinates, and Dout represents the frequency domain data of the final output;

[0029] Fig.10 is the vector-vector multiplication mapping diagram of FFT, wr represents the real part of the rotation factor, wi represents the imaginary part of the rotation factor, pr represents the real part of one input data of a group of butterfly operations, pi represents the imaginary part of one input data of a group of butterfly operations, qr represents the real part of another input data of a group of butterfly operations, and qi represents the imaginary part of another input data of a group of butterfly operations;

[0030] Fig.11 This is the programmable memory access programming logic diagram of FFT. Padding-1 means filling the 0 in the upper left corner with 1. DETAILED DESCRIPTION

[0031] Specific implementation method 1: The specific process of a generalized architecture design method for a deep learning processor in this implementation method is as follows:

[0032] When a general-purpose processor reads in deep learning calculations or other intensive calculations, it reads in the corresponding data to be calculated, and then sends this data to the off-chip DRAM in the compiled order. The general-purpose processor parses this data;

[0033] Step 1: The system is divided into software part and hardware part;

[0034] The software part is a general-purpose processor (such as Cortex-M3 and PicoRV32), which is responsible for programming the hardware.

[0035] The hardware part is the deep learning accelerator (deep learning accelerators such as NVDLA and DianNao), which is responsible for hardware calculations;

[0036] General-purpose processors include deep learning convolutional computing tasks and intensive computing tasks;

[0037] The general-purpose processor reads the data and analyzes the organization of the data in the off-chip memory (DRAM, which is an off-chip memory with a long access time);

[0038] The general-purpose processor arranges the read data in order. For example, for two-dimensional, three-dimensional or higher-dimensional data, it will be flattened into one-dimensional data and stored in the memory. The software side needs to let the hardware know where the data to be arranged is based on the location of each data in the memory. It is this data that is parsed.

[0039] When performing intensive computing tasks, go to step 2;

[0040] When performing deep learning convolution calculation tasks, go to step 3;

[0041] Step 2: Analyze the intensive computing task to obtain the optimal operator computing form, load the organizational form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, reorder the data in the on-chip cache Buffer based on the optimal operator computing form, and load the reordered data into the on-chip cache Buffer2 (if the reordered data are the same, only one of the same data is taken for reordering, and the reordered data is loaded into the same position in the on-chip cache Buffer2;);

[0042] The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory;

[0043] Step 3: Load the data in step 1 in the off-chip memory into the on-chip buffer, transfer the data in the buffer to the MAC array, and transfer the output result of the MAC array to the off-chip memory.

[0044] For example: the output dimension is 3*3*2 three-dimensional data, but the data in the memory is one-dimensional. The three-dimensional data needs to be arranged in one dimension. It is necessary to define a specific complete w and the distance of w*h in the memory, which is the process of flattening the three-dimensional data into one-dimensional data.

[0045] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that step 2: parses the intensive computing task to obtain the optimal operator computing form, loads the organizational form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, rearranges the data in the on-chip cache Buffer based on the optimal operator computing form, and loads the rearranged data into the on-chip cache Buffer2 (if the rearranged data are the same, only one of the same data is taken for rearrangement, and the rearranged data is loaded into the same position in the on-chip cache Buffer2;);

[0046] The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory;

[0047] The specific process is:

[0048] Operators for intensive computing tasks include vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, and vector-vector expansion multiplication;

[0049] The vector-vector multiplication a1×b1 completes the vector-vector product calculation, the dimension of the vector a1 or b1 is programmable, and the dimension of the vector a1 or b1 refers to the amount of data in the vector a1 or b1;

[0050] Matrix-vector multiplication completes the product calculation of a matrix and a vector. The number of vectors in a matrix (a matrix has as many vectors as its columns) is programmable and the dimension of the vector is programmable. The dimension of the vector refers to the number of data in the quantity.

[0051] Matrix-matrix multiplication completes the matrix-matrix product calculation. The number of vectors in the matrix and the dimension of the vectors in the matrix can be programmably controlled.

[0052] Vector-to-vector multiplication completes the diagonal addition of the results of matrix-matrix multiplication. The number of vectors in the matrix and the dimension of the vectors in the matrix can be programmably controlled.

[0053] If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n Not the same and vectors b1,b2,b n Not the same, vector-vector multiplication a1×b1, a2×b2,…, a n ×b n Or vector-vector multiplication a1×b1, a2×b2,…,a n ×b n ;

[0054] If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n Same or vector b1,b2,b n If the same vectors (vectors a1, a2, a n ) is extracted, and another set of different vectors (vectors b1, b2, b n ) concatenation is converted into a matrix, and vector-vector multiplication is converted into matrix-vector multiplication;

[0055] If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n The same vectors b1, b2, b n If the same, then a set of identical vectors (vectors a1, a2, a n) is extracted, and another set of identical vectors (vectors b1, b2, b n ) concatenation is converted into a matrix, and vector-vector multiplication is converted into matrix-vector multiplication;

[0056] If a set of matrix vectors are multiplied by A1×b1,A2×b2,…,A n ×b n In the matrix A1, A2, A n The same, matrices A1, A2, A n Extract and transform vectors b1, b2, b n (vector b1, b2, b n The same or different ones are all spliced ​​into a matrix) (for example, A1, A2, A n are the same, it becomes A1×b1,A1×b2,…,A1×b n , and then extract A1, it becomes A1×(b1,b2,…,b n ), (b1,b2,…,b n ) is composed of b1, b2, …, b n The concatenated matrices are converted into matrices, and matrix-vector multiplication becomes matrix-matrix multiplication;

[0057] A1, A2, A n represents the matrix, b1, b2, b n represents a vector;

[0058] If the diagonal sum of the results of a matrix-matrix multiplication A×B is obtained, a vector-vector expansion multiplication is obtained;

[0059] B1, B2, B n represents a matrix;

[0060] That is, the optimal operator calculation form is obtained when determining the intensive computing task in hardware calculation mapping;

[0061] Load the organization form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, reorder the data in the on-chip cache Buffer based on the optimal operator calculation form, and load the reordered data into the on-chip cache Buffer2 (if the reordered data are the same, only one of the same data is taken for reordering, and the reordered data is loaded into the same position in the on-chip cache Buffer2;);

[0062] The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory.

[0063] Generate operator, memory access, and output instructions;

[0064] The other steps and parameters are the same as those in the first embodiment.

[0065] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the data of the on-chip cache Buffer is rearranged based on the optimal operator calculation form, and the rearranged data is loaded into the on-chip cache Buffer2. The specific process is:

[0066] Split the data in the on-chip buffer into the optimal operator calculation form based on the dense formula corresponding to the dense calculation;

[0067] The optimal operator calculation form is loaded into the on-chip cache Buffer2.

[0068] The other steps and parameters are the same as those in the first or second embodiment.

[0069] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that, in step 3: the organization form of the data in step 1 in the off-chip memory is loaded into the on-chip cache Buffer, the data in the cache Buffer is transmitted to the MAC array, and the output result of the MAC array is transmitted to the off-chip memory;

[0070] The data in the cache Buffer2 is transferred to the MAC array through sequence control;

[0071] The specific process is:

[0072] Step 31: Convert the operator of the intensive computing task into a sequence control form in the deep learning convolution computing task;

[0073] Step 32: The sequence control module loads the data in the cache Buffer according to the sequence control form in the deep learning convolution calculation task and transmits it to the MAC array.

[0074] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0075] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that the step 31 converts the operator of the intensive computing task into a sequence control form in the deep learning convolution computing task; the specific process is:

[0076] Operators for intensive computing tasks include vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, and vector-vector expansion multiplication;

[0077] The general deep learning accelerator contains a MAC array, where one MAC unit loads one convolution kernel of one data in one cycle, and converts the MAC array into a vector-vector multiplication;

[0078] A MAC array includes multiple MAC units;

[0079] The general deep learning accelerator includes a MAC array. Each MAC array in multiple MAC units in one cycle loads a different convolution kernel of the same data, converting the MAC array into a matrix-vector multiplication;

[0080] The general deep learning accelerator contains a MAC array, multiple MAC units in multiple cycles load different convolution kernels with different data, and convert the MAC array into matrix-matrix multiplication;

[0081] A general deep learning accelerator includes a MAC array, multiple MAC units in multiple cycles load different convolution kernels with different data, a set of matrix-matrix multiplication calculation results are diagonally added, and the MAC array is converted into a vector-vector expansion multiplication.

[0082] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0083] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that the deep learning convolution calculation task includes a control instruction with a variable bit width;

[0084] The intensive computing tasks include control instructions with variable bit widths.

[0085] For deep learning convolutional calculations and intensive calculations, different bit width requirements will be imposed due to the importance of data and computational efficiency. For the more important data, a higher precision bit width calculation may be adopted, and for the data of general importance but requiring computational efficiency, a lower precision bit width calculation may be adopted. For deep learning convolutional calculations and intensive calculations, there will be similar requirements and application environments.

[0086] For example: based on the original 8-bit data support, three bit width forms of 4bit, 8bit and 16bit are introduced, and the mixed bit width combination is selected through variable bit width control instructions.

[0087] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0088] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the MAC is a multiplication-accumulation operation (Multiply Accumulate, MAC);

[0089] The MAC array is an array of multiple multiplication and accumulation units. Matrix-matrix multiplication requires multiple MAC units, i.e., MAC arrays, to complete;

[0090] The MAC array is a 4-bit MAC array, and different bit width combinations are obtained by splicing 4 bits.

[0091] For example, 4bit*8bit can be split into two 4bit*4bit multipliers for calculation.

[0092] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0093] The bit width of the data to be calculated is different. For example, if we want to calculate a convolution, which is the multiplication and accumulation of the input feature data and weight data, it may be specified that data is 4 bits and weight is 8 bits. Then in terms of memory, different bit widths are closely arranged in the memory. For example, if the memory has a 64-bit bandwidth, 4-bit data will be stored 16 times, and 8-bit weight will be stored 8 times. However, the software side does not need to parse this arrangement. It only needs to know the bit width required by the data. The hardware will read the data in the corresponding memory according to the parsed bit width requirements.

[0094] The data is organized in the off-chip memory as follows:

[0095] Continuous organization, linked organization, index organization;

[0096] The intensive computing tasks are: signal processing, data analysis, data compression, etc.;

[0097] Signal processing refers to FFT fast Fourier transform and FIR digital filter; data analysis refers to KNN classification algorithm, data compression refers to DCT discrete cosine transform and RNN recurrent neural network;

[0098] The deep learning convolution calculation task is a typical calculation form in AI deep learning. It mainly inputs feature data and weight data weight for convolution calculation, that is, multiplying the corresponding position data and finally accumulating them together. Therefore, the convolution part of the deep learning processor is equivalent to a multiplication accumulator for the convolution data stream, which is an array for vector multiplication calculation;

[0099] The continuous organization method is also called the continuous allocation method, which requires each file to be allocated an adjacent disk block;

[0100] Advantages: Easy sequential access: It is very easy to access sequential files and the access speed is very fast;

[0101] Disadvantages: It requires continuous space to be allocated for the file, and the length of the file must be known in advance; it is not flexible to delete or insert records; for files that grow dynamically, it is difficult to allocate space for them because it is difficult to know the final size of the file in advance;

[0102] Link organization method (divided into implicit link and explicit link), using link organization method can allocate multiple non-contiguous disk blocks for files;

[0103] Advantages: Eliminate external disk fragmentation and improve memory utilization; very easy to insert, delete and modify; can adapt to the dynamic growth of files;

[0104] Index organization methods are divided into single index and multi-index organization methods.

[0105] Off-chip memory refers to storage other than computer memory and CPU cache. This type of storage can generally save data after power failure. Common off-chip memories include hard disks, floppy disks, optical disks, USB flash drives, etc.

[0106] Define basic calculation operators according to the MAC array. The basic calculation operators include vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, and vector-vector expansion multiplication. The calculation dimension of each type of operator can be configured according to the programming situation (that is, which basic calculation operator is selected through programming, which is equivalent to setting a calculation mode);

[0107] Vector-vector multiplication completes the product calculation of vector and vector, and the dimension of vector can be controlled programmably;

[0108] Matrix-vector multiplication completes the product calculation of matrix and vector. The number of vectors in the matrix (a matrix has as many vectors as its columns) and the dimension of the vector can be controlled programmably.

[0109] Matrix-matrix multiplication completes the matrix-matrix product calculation. The number of vectors in the matrix and the dimension of the vectors in the matrix can be programmably controlled.

[0110] Vector-vector expansion multiplication completes the product calculation of high-dimensional vectors and vectors, and the dimension of the vector is programmable.

[0111] Analysis: According to the calculation form of convolution in the neural network, the convolution calculation process can be abstracted into basic calculation operators based on the multiplication and accumulation unit;

[0112] For different computing algorithms, that is, intensive computing tasks, based on parallelism, for example, the rotation factors between the signal and the butterfly calculation of the rotation factor in FFT may be the same. At this time, we can extract the rotation factor part of the FFT, extract the entire calculation as matrix-vector multiplication, and then map it to the corresponding calculation form.

[0113] The principle of analysis is to maximize data reuse as much as possible, and then convert basic calculations into hardware-supported calculations to ensure maximum hardware utilization.

[0114] According to the combination of data precision (this part will be determined according to the requirements of the algorithm, for example, when a certain layer has a small impact on the result during convolution or other intensive calculations, the bit width of the layer will be smaller, such as 4 bits;), variable bit width control instructions are generated (that is, the specific precision of the data part and the weight part is selected, and a bit segment is defined, such as [1:0], when this segment is equal to 00, it is 4 bits, when it is 01, it is 8 bits, and when it is 10, it is 16 bits. After the instruction is issued, the specific execution is done by the hardware;) and the data reuse sequence (this part is related to the data organization, for example, the data is organized in 64 bits. If the calculation of data 4 bits × weight 8 bits is performed, 64 bits contain 8 weights, but 8 weights only need 8 4-bit data calculations, that is, 32 bits, so only 32 bits of the 64-bit data passed in at a time can be calculated, and the remaining 32 bits need to be read repeatedly next time to calculate, so there is a need for a repeated reading reuse part;);

[0115] For an intensive calculation that requires multiple layers of calculation, the output data of the previous layer will be output to the off-chip DRAM according to the output form of the originally designed deep learning convolution calculation, but the output data of the previous layer will also be used as the input of the next layer, and the data read by the next layer is arranged in the form of deep learning convolution calculation. For intensive calculations, it may not be possible to directly load and use it in the MAC array. At this time, it is necessary to rearrange the data according to the data arrangement method required by the computationally intensive task, so that this piece of data is input in the form required by the computationally intensive task when it is sent to the MAC array.

[0116] Determine the optimal operator calculation form for intensive computing tasks when mapping them to hardware calculations:

[0117] Determine the programmable memory access instructions:

[0118] Figure 8 , Fig. 9 , Fig.10 This is a mapping case using FFT as an example. Normally, wr and wi represent the real and imaginary parts of the rotation factors, and pr, qr, pi, and qi represent the real and imaginary parts of the signal data. We first abstract a butterfly operation into vector multiplication. Then we extract the same rotation factors, abstract multiple sets of vector-vector multiplications into matrix-vector multiplications, and finally rearrange the densely arranged data in the abstract vector form according to the order of the figure.

[0119] Figure 8 Densely compute vector transform graphs for FFT;

[0120] x m+1 (p) = xm (p)+x m (q)×W r N

[0121] x m+1 (q) = x m (p)-x m (q)×W r N

[0122] Where q = p + 2 m

[0123] W r N represents the rotation factor, x m (p) represents one of the input data of a set of butterfly operations, x m (q) represents a set of butterfly operations on another input data, x m+1 (p) represents one of the output data of a set of butterfly operations, x m+1 (q) represents another output data of a set of butterfly operations, m is the level of FFT, for example, 16 points are divided into 4 levels and m takes 0, 1, 2, 3;

[0124] Fig. 9 is a 16-point FFT butterfly operation diagram, W N n represents the rotation factor, Xm represents the input signal data, () represents the corresponding coordinates, and Dout represents the frequency domain data of the final output;

[0125] Fig.10 This is the vector-vector multiplication mapping diagram of FFT, see Figure 8 , wr represents the real part of the rotation factor, wi represents the imaginary part of the rotation factor, pr represents the real part of one of the input data of a set of butterfly operations, pi represents the imaginary part of one of the input data of a set of butterfly operations, qr represents the real part of one of the input data of a set of butterfly operations, qi represents the imaginary part of one of the input data of a set of butterfly operations;

[0126] Fig.11 Program the logic diagram of the programmable memory access of FFT, and rearrange the data into the vector form we need according to the various parts of the programmable memory access.

[0127] x m (p)_real->pr;x m (p)_imag->pi;x m (q)_real->qr;x m (q)_imag->qi

[0128] W N_real->wr;W r N _imag->wi

[0129] x m+1 (p)_real=x m (p)_real×1+x m (q)_real×W r N _real-x m (q)_imag×W r N _imag=(x m (p)_real,x m (q)_real,x m (q)_imag)×(1,WrN_real,-WrN_imag) -1 =(pr,qr,qi)×(1,wr,-wi) -1

[0130] x m+1 (p)_imag=x m (p)_imag×1+x m (q)_real×W r N _imag+x m (q)_imag×WrN_real=(x m (p)_imag,x m (q)_real,x m (q)_imag)×(1,W r N _imag,W r N _real) -1 =(pi,qr,qi)×(1,wi,wr) -1

[0131] x m+1 (q)_real=x m (p)_real×1-x m (q)_real×W r N _real+x m (q)_imag×W r N _imag=(x m (p)_real,x m (q)_real,x m (q)_imag)×(1,-W rN _real,W r N _imag) -1 =(pr,qr,qi)×(1,-wr,wi) -1

[0132] x m+1 (q)_imag=x m (p)_imag×1-x m (q)_real×W r N _imag-x m (q)_imag×W r N _real=(x m (p)_imag,x m (q)_real,x m (q)_imag)×(1,-W r N _imag,-W r N _real) -1 =(pi,qr,qi)×(1,-wi,-wr) -1

[0133] real is the real part, imag is the imaginary part, -> is an arrow, which is the abbreviation used to represent the former. In abbreviated form, xm(p)_real is represented by the abbreviation pr.

[0134] According to the data flow structure of the hardware, determine the data (if it is a deep learning convolution calculation, it is the data of BUFFER; if it is a computationally intensive task, it is the data of BUFFER2.) at different hardware operation levels (the operation level is the calculation performed by different MAC arrays in one cycle; the operation levels of multiple cycles are combined into a higher-level block operation level; multiple block operation levels are combined to form a complete convolution calculation as the top operation level.) Data access form (equivalent to the input data at different operation levels. The data is first arranged in the prescribed order and then sent to the MAC array. It is a sequence control part. Data is loaded directly from the off-chip DRAM to the on-chip cache in batches, but the data required by MAC is in the data arrangement form at each level of the data flow. It is necessary to load the data from the on-chip cache to the MAC array through a sequence control part. It belongs to the structure of a general deep learning accelerator.), and generate the corresponding sequence (the sequence formed by loading the sequentially arranged data in the order of convolution calculation) control instructions;

[0135] Hardware Structure

[0136] This part of the function is mainly controlled by the sequence control module (sequence control) in the accelerator inputting the data structure of the MAC array.

[0137] Vector-vector multiplication corresponds to the multiplication and addition of two sets of vectors in a MAC unit in a single cycle (vector product, the data at corresponding positions in the vector are multiplied and then added together); the sequence controller serially loads two sets of vectors in each cycle (for example, there is a set of data, and one of them is loaded in each cycle until all the data is loaded. The corresponding is parallel loading, that is, all the data is loaded in one cycle.) One set of calculation cycles (a set of calculation cycles refers to a set of MAC arrays in a single cycle) is one clock cycle;

[0138] Each MAC unit operation in one cycle is equivalent to performing a group of vector product operations, that is, the multiplication and accumulation of two vectors. In the same cycle, multiple MAC units have one vector in the vector product of each group that is the same. Extracting the same vectors from each group, we can get the matrix-vector product operation of different MACs in a single cycle. Based on the matrix-vector product, another different vector is loaded in multiple cycles, thus forming a matrix-matrix multiplication calculation.

[0139] like Figure 7a , 7b As shown in the figure, data is loaded according to the time axis and sent to each MAC. Each dat and wt is a vector. The upper part is the process of serial data loading, and the lower part is the process of parallel data sending to each MAC array. Each MAC is a matrix-vector multiplication, and multiple MACs are combined to form a matrix-matrix multiplication.

[0140] Matrix-vector multiplication corresponds to single-cycle multiple groups of MAC units (single-cycle multiple groups of MAC are abstracted as matrix-vector multiplication). Each group of vectors in the matrix is ​​sent to different MAC arrays, and the vectors are sent to each MAC array in parallel (multiple vectors in the matrix, and multiple vectors form a matrix. Each vector in the matrix is ​​sent to each MAC unit in parallel); the sequence controller serially loads each part of the vector data within the number of cycles of the matrix data vector (according to the serial loading process, one vector is loaded in each cycle, so the number of cycles required for a complete matrix load is the same as the number of matrix vectors), and the vector is loaded in the first cycle (here it is equivalent to being divided into two processes, One is the process of loading data, and the other is the process of calculation. The loading process is serial, and the number of loading cycles is the same as the number of matrix vectors. The single cycle mentioned above refers to the calculation process, which is completed in a single cycle. Here we are talking about the loading process, which is multiple cycles, but there is only one vector in the matrix-vector multiplication, so the vector only needs one cycle to load). A set of calculation cycles (calculation only needs one cycle, a single cycle, but loading requires multiple cycles, and then calculation and loading are equivalent to parallel execution. While calculating this set of data, the next set of data is loaded, so the calculation cycle is based on the larger loading process) is the number of vectors in the matrix;

[0141] Matrix-matrix multiplication corresponds to multi-cycle multi-group MAC arrays. Each group of vectors in the A matrix is ​​sent to a different MAC array. In each cycle in the B matrix (copy the data to MAC, and then each MAC sends one), a group of vectors is selected and sent to each MAC array in parallel. Each group of vectors in the B matrix is ​​calculated serially in multiple cycles (one vector of the B matrix is ​​calculated in each cycle, that is, serially, and multiple cycles calculate multiple vectors). The sequence controller loads each part of the vector data serially within the number of cycles of the vectors of the two groups of matrices (the number of cycles is the same as the number of vectors in the matrix, and the vectors of the two matrices are loaded serially during the loading process). A group of calculation cycles is the number of cycles of the vectors of the larger matrix (the number of cycles is the same as the number of vectors in the larger matrix).

[0142] The basic multiplication unit of the MAC array is modified to a 4-bit multiplier. For combinations of different bit widths, we complete the calculation by extending the sign bit and shifting and splicing. Figure 2 This is a block diagram of 8-bit and 8-bit variable-width multiplication and addition.

[0143] Figure 2The four muls in the third part are multipliers. The multiplication part of MAC is designed to be 4bit in the form of the smallest bit width among 4bit, 8bit, and 16bit, and then the multipliers of other bit width combinations are completed by splicing 4bit multipliers. Here, take 8bit multiplied by 8bit as an example, split it into two 4bit data, and then extend the sign bit of the upper four bits and the lower four bits, fill the sign bit of the upper bit and fill the 0 of the lower bit, and then send them to the four multipliers respectively, and finally shift and add. The problem of sign bit is involved in shifting. For the result of the lower 4 bits multiplied by the lower 4 bits, the sign bit is extended to 0, and the others are extended to the high bits.

[0144] For the sequence control part, the data read from the cache is arranged in the data order required for shift splicing under the programmed bit width combination, and then sent to the MAC array. Figure 3 This is the control block diagram of the 8-bit and 8-bit variable-width multiplication-addition splicing sequence.

[0145] Let's take 8-bit by 8-bit as an example. There are 16 4-bit multipliers in a MAC. The data is tightly arranged in the memory, but according to the principle of multiplication splicing mentioned above, when the data is sent to the MAC array, the data needs to be sent according to the data required for splicing. As shown in the figure, 16 multipliers can load 32-bit data at a time, which results in the input 64-bit data cannot be calculated at once, and the remaining 32-bit data needs to be calculated the next time it is loaded.

[0146] Programmable memory access class

[0147] Function

[0148] This part controls the data rearrangement process required when data is loaded from the on-chip cache BUFFER to BUFFER2. This part mainly introduces three sets of instructions: programmable memory access, in-memory re-arrangement, and in-unit filling.

[0149] Programmable memory access - as the master switch of this type of instruction, it is mainly responsible for data loading and the use of other control instructions (memory reordering and unit filling), introducing the starting address of memory access and burst transfer length to achieve data access to any segment in the memory;

[0150] In-memory reordering - when a cache data needs to be composed of different bit segments from different data, the required data is read through programmable memory access, and the different bit segments of the read data are rearranged and combined to form new data through in-memory reordering;

[0151] In-memory re-arrangement is due to the re-combination of a single entry of intermediate data when the system is running. The data of this entry may be the concatenation of several groups of data from other groups of entries. Therefore, we define the existence of re-arrangement here to ensure that the system does not need to process the intermediate data generated between different hardware layers through the software side during operation. The in-memory data can be rearranged only through programming to complete the calculation of the entire system.

[0152] Rearrange_code_en: When a group of data is rearranged according to 16 4-bit data, the instruction controls the rearrangement of every 4 data as a group. When this bit is set, it means that the data corresponding to this bit needs to be rearranged, and the 4-bit data at this position is selected by coordinate decoding. When this bit is cleared, it means that the data at this position does not need to be rearranged, and the data at this position is set to zero.

[0153] Rearrange_code: Rearrange the specific decoding part in memory, and perform corresponding decoding logic on a group of data to be rearranged that are temporarily stored in the register according to the coordinate unit.

[0154] Command reference Figure 5 The figure shows the specific logic of one set of data in the 16 decoding units. The decoding of other parts can be obtained in the same way. Each decoding logic generates a 4-bit data, and finally the 16 sets of decoding result data are spliced ​​into 16-bit data and output to the next module.

[0155] Fill in the unit - when certain bit segments of the read data need to be filled with fixed values, fill in the data of the specified bit segments through filling in the unit;

[0156] When the data width is 4 bits, a group of 64 bits will contain 16 data. At this time, a 16-bit instruction is used to control the processing of the corresponding position. When the corresponding instruction bit is set, it means that the data corresponding to the position is processed accordingly. When the bit is cleared, it means that the data of the bit remains unchanged.

[0157] Cyclic output - When the data sent to the MAC array is cyclic (cyclic data means the same data exists after rearrangement), the cyclic output in the width direction of the cubic data is introduced, and the data to be cyclic is arranged to the first layer position of the cubic data to achieve cyclic output (circulation in the w direction).

[0158] For three-dimensional data such as images, we usually use width, height, and channel to represent the length, width, and height. The width direction loop means that when the hardware reads a three-dimensional data, it reads the data in the order of w, h, and c. When a w direction reading is completed, it will continue to read the w direction data at the next height of h. When the loop mode is turned on, the w direction data at the previous height of h will be read repeatedly.

[0159] Hardware Structure

[0160] Figure 4 It is a programmable memory access structure. This part of the hardware structure mainly generates the address of the cache read data according to the programming address received by the instruction and the burst transfer length, and generates the corresponding read data signal according to the read channel timing. After the data in the cache is loaded, it is read out in sequence.

[0161] Figure 5 It is a memory re-arrangement structure, receives data sent to the register from the programmable memory, selects the data temporarily stored in the register and the specific bit segment part of the data according to the decoding instruction, completes the splicing of a complete set of 64-bit data according to the selection, and outputs it to the filling module in the unit.

[0162] Figure 6 It is a filling structure within the unit. This part mainly selects to fill the specified bit segment of this part of the data with the fixed value specified by the instruction according to the decoding instruction. At the same time, when the data is in different bit widths, the combination of a piece of data is inconsistent, so the filling bit segment will be processed separately according to the different bit widths.

[0163] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A generalized architecture design method for a deep learning processor, characterized by: The specific process of the method is: Step 1: The system is divided into software part and hardware part; The software part is the general-purpose processor, which is responsible for programming the hardware; The hardware part is the deep learning accelerator, which is responsible for hardware calculations; General-purpose processors include deep learning convolutional computing tasks and intensive computing tasks; The general-purpose processor reads the data and analyzes the organization of the data in the off-chip memory; When performing intensive computing tasks, go to step 2; When performing deep learning convolution calculation tasks, go to step 3; Step 2: Analyze the intensive computing task to obtain the optimal operator computing form, load the organization form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, reorder the data in the on-chip cache Buffer based on the optimal operator computing form, and load the reordered data into the on-chip cache Buffer2; The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory; Step 3: Load the data in step 1 in the off-chip memory into the on-chip buffer, transfer the data in the buffer to the MAC array, and transfer the output result of the MAC array to the off-chip memory; The step 2: parsing the intensive computing task to obtain the optimal operator computing form, loading the organizational form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, rearranging the data in the on-chip cache Buffer based on the optimal operator computing form, and loading the rearranged data into the on-chip cache Buffer2; The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory; The specific process is: Operators for intensive computing tasks include vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, and vector-vector expansion multiplication; The vector-vector multiplication a1×b1 completes the vector-vector product calculation, the dimension of the vector a1 or b1 is programmable, and the dimension of the vector a1 or b1 refers to the amount of data in the vector a1 or b1; Matrix-vector multiplication completes the product calculation of a matrix and a vector, the number of vectors in the matrix is ​​programmable, the dimension of the vector is programmable, and the dimension of the vector refers to the number of data in the quantity; Matrix-matrix multiplication completes the matrix-matrix product calculation. The number of vectors in the matrix and the dimension of the vectors in the matrix can be programmably controlled. Vector-to-vector multiplication completes the diagonal addition of the results of matrix-matrix multiplication. The number of vectors in the matrix and the dimension of the vectors in the matrix can be programmably controlled. If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n Not the same and vectors b1,b2,b n Not the same, vector-vector multiplication a1×b1, a2×b2,…, a n ×b n Or vector-vector multiplication a1×b1, a2×b2,…,a n ×b n ; If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n Same or vector b1,b2,b n If they are the same, the same vectors are extracted, and another set of different vectors are concatenated and converted into a matrix, and the vector-vector multiplication becomes a matrix-vector multiplication; If a set of vector vectors is multiplied by a1×b1, a2×b2,…,a n ×b n The vector a1, a2, a n The same vectors b1, b2, b n If they are the same, extract one set of identical vectors, concatenate another set of identical vectors and convert them into matrices, and convert vector-vector multiplication into matrix-vector multiplication. If a set of matrix vectors are multiplied by A1×b1,A2×b2,…,A n ×b n In the matrix A1, A2, A n The same, matrices A1, A2, A n Extract and transform vectors b1, b2, b n The concatenation is converted into a matrix, and the matrix-vector multiplication becomes a matrix-matrix multiplication; A1, A2, A n represents the matrix, b1, b2, b n represents a vector; If the diagonal sum of the results of a matrix-matrix multiplication A×B is obtained, a vector-vector expansion multiplication is obtained; B represents a matrix; That is, the optimal operator calculation form is obtained; Load the organization form of the data in step 1 in the off-chip memory into the on-chip cache Buffer, rearrange the data in the on-chip cache Buffer based on the optimal operator calculation form, and load the rearranged data into the on-chip cache Buffer2; The data in the cache Buffer2 is transferred to the MAC array, and the output result of the MAC array is transferred to the off-chip memory; The data of the on-chip cache Buffer is rearranged based on the optimal operator calculation form, and the rearranged data is loaded into the on-chip cache Buffer2. The specific process is as follows: Split the data in the on-chip buffer into the optimal operator calculation form based on the dense formula corresponding to the dense calculation; Load the optimal operator calculation form into the on-chip cache Buffer2; The step 3: loading the organization form of the data in step 1 in the off-chip memory into the on-chip buffer, transferring the data in the buffer to the MAC array, and transferring the output result of the MAC array to the off-chip memory; The specific process is: Step 31: Convert the operator of the intensive computing task into a sequence control form in the deep learning convolution computing task; Step 32: The sequence control module loads the data in the cache Buffer according to the sequence control form in the deep learning convolution calculation task and transmits it to the MAC array; The step 31, converting the operator of the intensive computing task into a sequence control form in the deep learning convolution computing task; the specific process is: Operators for intensive computing tasks include vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, and vector-vector expansion multiplication; The general deep learning accelerator contains a MAC array, where one MAC unit loads one convolution kernel of one data in one cycle, and converts the MAC array into a vector-vector multiplication; The general deep learning accelerator includes a MAC array. Each MAC array in multiple MAC units in one cycle loads a different convolution kernel of the same data, converting the MAC array into a matrix-vector multiplication; The general deep learning accelerator contains a MAC array, multiple MAC units in multiple cycles load different convolution kernels with different data, and convert the MAC array into matrix-matrix multiplication; The general deep learning accelerator includes a MAC array, multiple MAC units in multiple cycles load different convolution kernels with different data, and a set of matrix-matrix multiplication calculation results are added diagonally to convert the MAC array into a vector-vector expansion multiplication; The deep learning convolution calculation task includes control instructions with variable bit width; The intensive computing tasks include control instructions with variable bit widths; The MAC is a multiplication-accumulation operation; The MAC array is an array composed of multiple multiplication and accumulation units; The MAC array is a 4-bit MAC array, and different bit width combinations are obtained by splicing 4 bits.

Images