In-memory accelerator for accelerating neural network model reasoning

Abstract

The invention provides an in-memory accelerator for accelerating reasoning of a neural network model, the accelerator comprises an in-memory calculation processing unit, a local storage unit and a post-processing unit, the local storage unit is used for storing input data of the neural network model; the in-memory calculation processing unit comprises a plurality of in-memory calculation arrays, each in-memory calculation array is used for executing calculation operation, obtaining input data corresponding to the calculation operation from the local storage unit in sequence according to the size of the sliding window, and performing matrix-vector multiplication calculation on the input data and a corresponding tensor kernel according to a preset iteration process to obtain a tensor kernel; wherein the input data and the tensor kernel are rearranged according to the dimensions of the input data and the tensor kernel and then are subjected to matrix-vector multiplication calculation; the tensor kernel is obtained by performing tensor decomposition on the original weight of the neural network model and storing the original weight into the in-memory calculation array; and the post-processing unit is used for performing post-processing according to a result of the matrix-vector multiplication calculation to obtain a reasoning result of the neural network model.

Description

Technical Field

[0001] The present invention relates to the field of in-memory computing, specifically to the field of neural network accelerator technology, and more specifically to an in-memory accelerator for accelerating neural network model reasoning. Background Art

[0002] Deep neural networks (DNNs), with their powerful feature extraction capabilities, have achieved remarkable results in various practical applications, including image classification, video analysis, and speech recognition. The growing capabilities of DNNs rely heavily on the expansion of model parameters. According to the neural scaling law, DNN models are still being gradually expanded to improve performance and enhance functionality.

[0003] The processing-in-memory (PIM) architecture is a new computing model that breaks away from the traditional von Neumann computing system. It integrates computing logic into memory cells, allowing computations to be performed on data in memory, thus avoiding the performance bottlenecks caused by the frequent movement of data between the processor and memory in traditional computing models. This architecture is particularly well-suited for the inference process of deep neural networks, as DNN inference requires a large number of matrix multiplications and data access operations. These operations can be performed in parallel and in situ in the PIM architecture, greatly improving computational efficiency.

[0004] However, the growth rate of in-memory computing accelerators lags far behind that of deep neural network models. Even medium-sized deep neural network models require dozens or even hundreds of in-memory computing accelerators to work together. The data transmission and synchronization overhead during parallel processing of multiple accelerators may offset the gains brought by in-memory computing devices. The mismatch between the size of deep neural network models and the capacity of currently implemented in-memory computing accelerators has hindered the widespread application of in-memory computing. Research has found that Tensor Train Decomposition (TTD) can achieve a very high compression rate with a small loss of compression accuracy, but after tensor train decomposition, the original element-by-element calculation method is still used, which has many redundant calculations and reduces the inference speed of the accelerator.

[0005] It should be noted that this background information is provided solely to introduce relevant information of the present invention to facilitate understanding of the technical solution of the present invention. It does not necessarily constitute prior art. In the absence of evidence demonstrating that the relevant information was disclosed prior to the filing date of the present invention, the relevant information should not be considered prior art. Summary of the Invention

[0006] Therefore, the purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide an in-memory accelerator for accelerating neural network model reasoning.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, an in-memory accelerator for accelerating the inference of a neural network model is provided, the accelerator comprising: an in-memory computing processing unit, a local storage unit and a post-processing unit, wherein: the local storage unit is used to store the input data of the neural network model; the in-memory computing processing unit comprises a plurality of in-memory computing arrays, each in-memory computing array is used to perform computing operations, and obtain input data corresponding to the computing operations from the local storage unit in sequence according to the size of the sliding window, and perform matrix-vector multiplication calculations on the input data and the corresponding tensor cores according to a preset iterative process, wherein the input data and the tensor cores are rearranged according to their dimensions and then subjected to matrix-vector multiplication calculations; the tensor cores are obtained by performing tensor decomposition of the original weights of the neural network model and stored in the in-memory computing array; the post-processing unit is used to perform post-processing according to the results of the matrix-vector multiplication calculations to obtain the inference results of the neural network model.

[0009] In some embodiments of the present invention, the preset iterative process includes: obtaining the calculation results of the previous processing layer as the input data of the current layer, and rearranging the calculation results; performing multiple matrix-vector multiplication calculations on the rearranged calculation results and all tensor cores of the current processing layer to obtain the calculation results of the current processing layer; wherein, before each matrix-vector multiplication calculation, it is necessary to obtain the output of the previous tensor core and rearrange it; wherein, the input data of the first processing layer is the input data obtained from the local storage unit in sequence according to the size of the sliding window, and the calculation result of the last processing layer is the final calculation result.

[0010] In some embodiments of the present invention, the in-memory computing array is configured to rearrange the tensor cores according to their dimensions and a preset decomposition rank to obtain tensor cores for performing matrix-vector multiplication calculations.

[0011] In some embodiments of the present invention, each matrix-vector multiplication step includes: obtaining the output of the previous tensor core As the input data of the current tensor core, it is rearranged into n-sized ones according to the dimension. k r d-2 ×m1…m k-1 n k+1 …n q The matrix of r0 Among them, n k express The kth row, m k-1 express The k-1th column, r d-2 Indicates the decomposition rank of the d-2th tensor core; obtains the dth tensor core matrix G hd , and rearrange it into m-sized k r d ×n k r d-1 Matrix Among them, r d represents the decomposition rank of the d-th tensor core, r d-1 Represents the decomposition rank of the d-1th tensor core; calculation The results are rearranged into Get the output result of the current tensor core.

[0012] In some embodiments of the present invention, the calculation result of each processing layer is calculated as follows:

[0013]

[0014] in, Represents the calculation result of the hth processing layer, represents the calculation result of the h-1th processing layer, G hd represents the d-th tensor core of the h-th processing layer, × d The operator representing the d-th multiplication operation, d∈[1,p].

[0015] In some embodiments of the present invention, the in-memory computing processing unit also includes a digital-to-analog conversion unit, a sampling and holding unit, an analog-to-digital conversion unit, a shift-and-accumulate unit, and an output cache unit, wherein: the digital-to-analog conversion unit is used to convert the input data of the neural network model from a digital signal to an analog signal, and input it into the corresponding in-memory computing array, so that the in-memory computing array performs matrix-vector multiplication calculation; the sampling and holding unit is used to sample the calculation results of the in-memory computing array, and input the sampling results into the analog-to-digital conversion unit; the analog-to-digital conversion unit is used to convert the sampling results into digital signals, and input them into the shift-and-accumulate unit; the shift-and-accumulate unit is used to shift-and-accumulate the conversion results in the analog-to-digital conversion unit, and store the shift-and-accumulate results into the output cache unit.

[0016] In some embodiments of the present invention, the local storage unit is further used to store the shift accumulation result stored in the output buffer unit, wherein the local storage unit performs data transmission with the in-memory computing processing unit via a bus.

[0017] In some embodiments of the present invention, the post-processing unit is configured to perform post-processing according to the result of the matrix-vector multiplication calculation, and the post-processing includes accumulation processing, activation processing and / or pooling processing.

[0018] In some embodiments of the present invention, the tensor core is obtained in the following manner: obtain a neural network model to be compressed, the neural network model including multiple processing layers; obtain a training set, the training set including multiple image samples and labels indicating the categories to which the image samples belong; determine the tensor decomposition form of the original weights of each processing layer based on the dimensionality of the original weights corresponding to each processing layer; perform vector decomposition on the original weights of each processing layer according to the tensor decomposition form and a preset decomposition rank to obtain an initial tensor core corresponding to each processing layer, and all initial tensor cores constitute an initial compression model corresponding to the model; use the training set to fine-tune the initial compression model to obtain a trained compression model, which includes the tensor cores of each processing layer after fine-tuning.

[0019] According to a second aspect of the present invention, there is provided a method for accelerating the inference of a neural network model using an in-memory accelerator as described in the first aspect of the present invention, the method comprising: obtaining input data of the neural network model and tensor cores of each processing layer; obtaining corresponding input data in sequence according to the size of the sliding window, and rearranging the input data and the corresponding tensor cores according to their dimensions to obtain rearranged input data and tensor cores; performing matrix-vector multiplication calculation on the rearranged input data and tensor cores according to a preset iterative calculation process; and post-processing the result of the matrix-vector multiplication calculation to obtain an inference result of the neural network model.

[0020] Compared with the prior art, the advantages of the present invention are:

[0021] (1) It is possible to use tensor decomposition to decompose the weights of a deep neural network model into several tensor cores, reducing the number of model parameters and storage overhead, thereby compressing the neural network model to a size suitable for in-memory accelerator deployment.

[0022] (2) The data after tensor decomposition is rearranged, which reduces the data redundancy in the calculation process to a certain extent. Combined with the calculation according to the preset iterative process, compared with the original element-by-element calculation method, it further reduces the data redundancy in the calculation process, improves the calculation performance, and speeds up the inference speed of the accelerator.

[0023] (3) All in-memory computing arrays in each in-memory computing processing unit in the accelerator share one analog-to-digital conversion unit, which not only saves area but also reduces energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0025] Figure 1 Schematic diagram of an in-memory accelerator structure for accelerating neural network model reasoning according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a flow chart of iterative calculation according to an embodiment of the present invention;

[0027] Figure 3 The figure is a flowchart of performing a matrix-vector multiplication operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] As mentioned in the background technology section, the growth rate of the scale of in-memory computing accelerators lags far behind that of deep neural network models. Even medium-sized deep neural network models require dozens or even hundreds of in-memory computing accelerators to work together. The data transmission and synchronization overhead during parallel processing of multiple accelerators may offset the gains brought by in-memory computing devices. The mismatch between the size of deep neural network models and the capacity of currently implemented in-memory computing accelerators hinders the widespread application of in-memory computing. After research, it was found that tensor train decomposition can achieve a very high compression rate and a small loss of compression accuracy, but the original element-by-element calculation method has a lot of redundant calculations, which reduces the inference speed of the accelerator.

[0030] In order to solve the above problems, the present invention proposes an in-memory accelerator for accelerating the inference of a neural network model, wherein the accelerator includes an in-memory computing processing unit, a local storage unit and a post-processing unit, wherein: the local storage unit is used to store the input data of the neural network model; the in-memory computing processing unit includes a plurality of in-memory computing arrays, each of which is used to perform computing operations, and obtains the input data corresponding to the computing operation from the local storage unit in sequence according to the size of the sliding window, and performs matrix-vector multiplication calculation on the input data and the corresponding tensor core according to a preset iterative process, wherein the input data and the tensor core are The cores are rearranged according to their dimensions and then matrix-vector multiplication calculations are performed. The tensor cores are obtained by tensor decomposition of the original weights of the neural network model and stored in the in-memory computing array. First, the input data and the tensor cores are rearranged according to their dimensions, which reduces the data redundancy in the calculation process to a certain extent. Combined with the calculation according to the preset iterative process, compared with the original element-by-element calculation method, the data redundancy in the calculation process is further reduced, the computing performance is improved, and the inference speed of the accelerator is accelerated; the post-processing unit is used to perform post-processing according to the results of the matrix-vector multiplication calculation to obtain the inference result of the neural network model.

[0031] In order to better understand the present invention, the technical solution of the present invention is described in detail below with reference to specific embodiments and drawings.

[0032] The present invention proposes an in-memory accelerator for accelerating neural network model reasoning, such as Figure 1 As shown, the accelerator includes a local memory unit for storing input data, an in-memory processing element (PE) for performing computational operations, and a post-processing unit for post-processing the computational results to obtain inference results for the neural network model. The following details the inference process of the in-memory accelerator.

[0033] 1. Get data

[0034] First, read the neural network model. The neural network model can use an existing model, such as AlexNet, VGG, ResNet, MLP, etc., or a suitable model can be constructed based on actual research. In order to improve the availability and applicability of the in-memory accelerator, the present invention uses a compression method based on tensor train decomposition to efficiently compress the neural network model to obtain the tensor core required for the computing operation. According to one embodiment of the present invention, the tensor core is obtained in the following manner: obtain a neural network model to be compressed, the neural network model includes multiple processing layers; obtain a training set, the training set includes multiple image samples and labels indicating the categories to which the image samples belong; determine the tensor decomposition form of the original weights of each processing layer according to the dimensional size of the original weights corresponding to each processing layer; perform vector decomposition on the original weights of each processing layer according to the tensor decomposition form and the preset decomposition rank to obtain the initial tensor core corresponding to each processing layer, and all the initial tensor cores constitute the initial compression model corresponding to the model; use the training set to fine-tune the initial compression model to obtain a trained compression model, which includes the tensor cores of each processing layer after fine-tuning. It is worth noting that tensor train decomposition represents high-dimensional tensors as the product of a series of low-rank matrices shaped like a train. Algorithms for implementing TTD include TT-SVD, DMRG, and TT-Cross. The technical solution of this embodiment can achieve at least the following beneficial technical effects: through tensor train decomposition, the weights of the deep neural network model are decomposed into several tensor cores, reducing the number of model parameters, thereby compressing the deep neural network model to a size suitable for deployment on in-memory computing accelerators.

[0035] To demonstrate the beneficial technical effects achieved by the above embodiments, the present invention conducted an experimental comparison, selecting three datasets: MNIST, Cifar-10, and Imagenet-1K. The MLP model was selected for the MNIST dataset, the AlexNet, VGG16, and ResNet18 models were selected for the Cifar-10 dataset, and the VGG16 and ResNet18 models were selected for the Imagenet-1K dataset. The experimental results are shown in Table 1:

[0036] Table 1 Compression effect

[0037]

[0038] As can be seen from Table 1, this method can effectively compress deep neural network models, with the highest compression rate reaching 75.9% and the average compression rate being 15.9%. The minimum accuracy loss after compression is 0.3%. Overall, the accuracy loss is within an acceptable range and exceeds the compression rate achieved by the quantization algorithm adopted by the existing technical solutions.

[0039] According to one embodiment of the present invention, the tensor cores obtained by tensor train decomposition are stored in an in-memory computing array, and the in-memory computing array rearranges them according to the dimension of the tensor core and a preset decomposition rank (for example, the decomposition rank is set to 1, 2, 3, etc.) to obtain tensor cores for matrix-vector multiplication calculations.

[0040] According to one embodiment of the present invention, the training set may be an existing one, such as MNIST, Cifar-10, and Imagenet-1K, or may be acquired through acquisition equipment.

[0041] According to an example of the present invention, for a q-dimensional tensor Each element in this tensor can be calculated using the following formula (1):

[0042]

[0043] in, represents the kth tensor core, and i k is the loop variable, n k is a tensor In the kth dimension, r k is the decomposition rank of the kth dimension, It means A slice of Represents a size r k-1 ×r k A two-dimensional matrix. Each The r contained in k The set (r0, r1, ..., r q ) is called the decomposition rank, where r0=r q =1.

[0044] Through the above formula, we can get q tensor cores Therefore, the q-dimensional tensor Can be passed through a series of tensor cores To indicate that, at this time, the q-dimensional tensor is stored The space cost is from Changes to When a smaller set of decomposition ranks is selected, the compressed storage overhead will be much smaller than the original overhead.

[0045] According to an example of the present invention, for a matrix Assume that the two dimensions have the following factorization and To perform a tensor decomposition on this matrix, first rearrange the matrix into a tensor According to the formula of tensor train decomposition, each element of this tensor can be calculated by formula (2):

[0046]

[0047] in, i k and j k is the loop variable, n k and m k It represents the size of the tensor w in the kth dimension, It means A slice of For a size r k-1 ×r k Therefore, each element of W can be represented by a two-dimensional tuple ((j1,i1),(j2,i2),…,(j q ,i q )) to index, where (j1,j2,…,j q ) can traverse each column of W, (i1,i2,…,i q ) can traverse each row of W.

[0048] For matrix-vector multiplication Y=XW, where Assume that the matrix W is decomposed into tensor trains according to formula (2) to obtain a series of tensor cores First, rearrange Y and X into tensors and According to the following formula, the result of matrix-vector multiplication is Each element of can be calculated by formula (3):

[0049]

[0050] in, Indicates indexing the column elements of W, there are q columns in total, Indicates indexing the row elements of W, with a total of q rows.

[0051] 2. Perform calculations

[0052] Based on the tensor core obtained in the above manner, the input data corresponding to the calculation operation is obtained from the local storage unit in sequence according to the size of the sliding window, and the input data is subjected to matrix-vector multiplication calculation with the tensor core.

[0053] The existing calculation method can use a series of tensor cores generated after decomposition to directly calculate the results, thereby avoiding the extra time and storage overhead of restoring the matrix first. However, directly calculating element by element through formula (3) will lead to many repeated calculations (for example, calculating and When, for each group Requires repeated calculations Such redundant calculations can significantly worsen inference latency, leading to longer calculation times and energy waste. Therefore, the present invention designs an iterative process to transform the above-mentioned element-by-element calculation process into a multi-stage inference calculation, eliminating repeated calculation processes and reducing data redundancy during the inference process. According to one embodiment of the present invention, the preset iterative process includes: obtaining the calculation results of the previous processing layer as the input data of the current layer, and rearranging the calculation results; performing multiple matrix-vector multiplication calculations on the rearranged calculation results and all tensor cores of the current processing layer to obtain the calculation results of the current processing layer; wherein, before each matrix-vector multiplication calculation, the output of the previous tensor core needs to be obtained and rearranged; wherein, the input data of the first processing layer is the input data obtained from the local storage unit in sequence according to the size of the sliding window, and the calculation result of the last processing layer is the final calculation result. The technical solution of this embodiment can at least achieve the following beneficial technical effects: after the data is rearranged, the data redundancy in the calculation process is appropriately reduced; the data calculation is performed through iterative calculation, which reduces the redundancy in the original element-by-element calculation and speeds up the inference speed.

[0054] According to an example of the present invention, Figure 2 As shown, the specific iterative process is as follows: S1, obtain input data, and execute the following steps from the first tensor core of the first processing layer according to the input data: S2, convert the input data P (h-1) Rearrange; S3, the rearranged data and the first tensor core G of the first processing layer 11 Perform calculations to obtain the output result P of the first tensor core 01 ; S4, determine whether d is equal to p (that is, determine whether the output result is the calculation result of the last tensor core). If so, execute step S5. If not, use the calculation result of the tensor core as the input data of the next tensor core and return to execute step S2. S5, determine whether h is equal to s (that is, determine whether the output result of this layer is the calculation result of the last processing layer). If it is the output result, if not, use the output result of this layer as the input data of the next layer and return to execute step S2.

[0055] According to an example of the present invention, Figure 3 As shown in the figure, the process of obtaining the calculation results of each processing layer is shown. Steps T1-T4 show the process of a matrix-vector multiplication calculation. For the h-th processing layer, the specific process of obtaining its calculation results is as follows: T1, obtain the output of the previous tensor core As the input data of the current tensor core; T2, rearrange it into a size of n according to the dimension k r d-2×m1…m k-1 n k+1 …n q The matrix of r0 Among them, n k express The kth row, m k-1 express The k-1th column, r d-2 Indicates the decomposition rank of the d-2th tensor core; T3, obtain the dth tensor core matrix G hd , and rearrange it into m-sized k r d ×n k r d-1 Matrix Among them, r d represents the decomposition rank of the d-th tensor core, r d-1 Indicates the decomposition rank of the d-1th tensor core; T4, calculation The results are rearranged into Get the output result of the current tensor core; T5, determine whether d is equal to p, that is, determine Is it equal to If yes, execute step T6 and output is the calculation result of the hth processing layer. Otherwise, the calculation result of the tensor core is used as the input data of the next tensor core and the process returns to step T2; where p is the total number of tensor cores in each processing layer.

[0056] According to one embodiment of the present invention, the calculation result of each processing layer is calculated as follows:

[0057]

[0058] in, Represents the calculation result of the hth processing layer, represents the calculation result of the h-1th processing layer, G hd represents the d-th tensor core of the h-th processing layer, × d The operator representing the d-th multiplication operation, d∈[1,n], h∈[1,s]. When h=1, That is, for the first processing layer, To obtain the input data corresponding to the calculation operation from the local storage unit in sequence according to the size of the sliding window

[0059] The schematic process of deriving formula (4) is as follows: For an N-dimensional tensor and a two-dimensional matrix The tensor Perform tensor-matrix contraction operation with matrix B to obtain an N-dimensional tensor Its tensor Each element in can be obtained by formula (5):

[0060]

[0061] According to formula (5), we can first transform the tensor Rearrange to size I n ×I1…I n-1 I n+1 …I N The matrix A is then computed by matrix-matrix multiplication BA to obtain a matrix of size J×I1…I n-1 I n+1 ...the matrix P of IN, and then rearrange P into a tensor It can be seen that the tensor-matrix contraction operation can be completed through reshaping and matrix multiplication operations.

[0062] In order to simplify the element-by-element calculation formula (3) by using the tensor-matrix contraction operation, the tensor core in formula (3) needs to be Convert to tensor core matrix in Therefore, tensor cores Converted into a nested two-dimensional matrix G k , G k The size is m k ×n k A two-dimensional matrix where each element is also of size r d-1 ×r d Based on this, formula (3) can be equivalent to formula (6):

[0063]

[0064] According to formula (5), we can use tensor-matrix contraction operation to simplify formula (6) and obtain formula (4).

[0065] According to an example of the present invention, suppose the result of the k-1th matrix-vector multiplication operation in any processing layer is You can get: Each element is of size r0×r k-1 A tensor core G k Each element of is of size r k-1 ×r k Matrix. Calculate using formula (4) First, Rearrange into a size of nx×m1…m k-1 n k+1 …n d The matrix P k-1 , then calculate G k P k-1 The product P k , and finally the matrix P k Rearrange into a tensor Since P k-1 , G k 、P k The elements of are all matrices. To further represent the matrix operations between elements, Rearrange all elements in into a n-th order k r k-1 ×m1…m k-1 n k+1 …n d The matrix of r0 G k All elements in are rearranged into a size of m k r k ×n k r k-1 Matrix Then calculate Then reshape the result into in Each element is of size r0×r k Matrix. Schematically, it can be rearranged as shown in Table 2.

[0066] Table 2 Rearrangement

[0067]

[0068]

[0069] According to Table 2, from k=1 to k=2, first Reshape The shape is n1r0×n2n3n4, calculated The shape is m1r1×n2n3n4, which is reshaped into The shape is m1×n2×n3×n4, each element is r0×r1; then Reshape The shape is n2r1×m1n3n4, calculated The shape is m2r2×m1n3n4, which is reshaped into The shape is m1×m2×n3×n4, each element is r0×r2, and so on. Assume that the input data That is to say n1=2, n2=2, n3=3, n4=3, Will Rearrange to a matrix of n1×n2n3n4 The first tensor core of the first processing layer G k Each element of is a matrix of size r0×r1, where m1=2, n1=2, r0=1, r1=2, so The shape is m1×n2×n3×n4, each element is r0×r1, assuming The data is: The corresponding for: The technical solution of this embodiment can at least achieve the following beneficial technical effects: after rearranging the data in a certain format, data redundancy in the calculation process is appropriately reduced, and the reasoning speed is effectively improved.

[0070] According to one embodiment of the present invention, the in-memory computing processing unit includes a plurality of in-memory computing arrays ( Figure 1 Only two in-memory computing arrays are shown), a digital-to-analog conversion unit, a sampling and holding unit, an analog-to-digital conversion unit, a shift-accumulation unit, and an output buffer unit, wherein: the digital-to-analog conversion unit is used to convert the input data of the neural network model from a digital signal to an analog signal, and input it to the corresponding in-memory computing array, so that the in-memory computing array executes the above-mentioned iterative process to perform matrix-vector multiplication calculation; the sampling and holding unit is used to sample the calculation results of the in-memory computing array and input the sampling results to the analog-to-digital conversion unit; the analog-to-digital conversion unit is used to convert the sampling results into digital signals and input them to the shift-accumulation unit; the shift-accumulation unit is used to shift-accumulate the conversion results in the analog-to-digital conversion unit, and store the shift-accumulation results in the output buffer unit. Each in-memory computing array is connected to a digital-to-analog conversion unit and a sampling and holding unit, and all in-memory computing arrays share one analog-to-digital conversion unit. The technical solution of this embodiment can at least achieve the following beneficial technical effects: since the energy consumption and area of ​​the analog-to-digital conversion unit are relatively high, all in-memory computing arrays in each in-memory computing processing unit share one analog-to-digital conversion unit, saving area and reducing energy consumption overhead.

[0071] 3. Output results

[0072] The result of the matrix-vector multiplication calculation is post-processed to obtain the inference result of the neural network model.

[0073] According to one embodiment of the present invention, the post-processing unit includes an accumulation unit, an activation unit and a pooling unit, wherein the accumulation unit is used to accumulate and sum the calculation results; the activation unit is used to activate the accumulation results according to the activation function (for example: ReLU function, Sigmoid function and Tanh function, etc.); the pooling unit is used to pool the activated results to obtain the inference results of the neural network model.

[0074] According to one embodiment of the present invention, the local storage unit is also used to store the shift accumulation results, the intermediate calculation results of each layer and the output results stored in the output cache unit, wherein the local storage unit transmits data with the in-memory computing processing unit via a bus.

[0075] Correspondingly, the present invention also proposes a method for accelerating the reasoning of a neural network model using the in-memory accelerator as described above, the method comprising: obtaining the input data of the neural network model and the tensor cores of each processing layer; obtaining the corresponding input data in sequence according to the size of the sliding window, and rearranging the input data and the corresponding tensor cores according to their dimensions to obtain the rearranged input data and tensor cores; performing matrix-vector multiplication calculation on the rearranged input data and tensor cores according to a preset iterative calculation process; and post-processing the result of the matrix-vector multiplication calculation to obtain the inference result of the neural network model.

[0076] In summary, the in-memory accelerator for accelerating neural network model inference proposed in this invention has the following advantages:

[0077] (1) It is possible to use tensor decomposition to decompose the weights of a deep neural network model into several tensor cores, reducing the number of model parameters and storage overhead, thereby compressing the neural network model to a size suitable for in-memory accelerator deployment.

[0078] (2) The data after tensor decomposition is rearranged, which reduces the data redundancy in the calculation process to a certain extent. Combined with the calculation according to the preset iterative process, compared with the original element-by-element calculation method, it further reduces the data redundancy in the calculation process, improves the calculation performance, and speeds up the inference speed of the accelerator.

[0079] (3) All in-memory computing arrays in each in-memory computing processing unit in the accelerator share one analog-to-digital conversion unit, which not only saves area but also reduces energy consumption.

[0080] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0081] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0082] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0083] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An in-memory accelerator for accelerating neural network model inference, characterized in that: The accelerator includes an in-memory computing processing unit, a local storage unit, and a post-processing unit, wherein: The local storage unit is used to store input data of the neural network model; The in-memory computing processing unit includes a plurality of in-memory computing arrays, each of which is used to perform a computing operation, sequentially obtain input data corresponding to the computing operation from the local storage unit according to the size of the sliding window, and perform a matrix-vector multiplication calculation on the input data and the corresponding tensor core according to a preset iterative process, wherein the input data and the tensor core are rearranged according to their dimensions before performing the matrix-vector multiplication calculation; the tensor core is obtained by performing a tensor decomposition of the original weights of the neural network model and stored in the in-memory computing array; The post-processing unit is used to perform post-processing based on the result of the matrix-vector multiplication calculation to obtain the inference result of the neural network model.

2. The accelerator according to claim 1, characterized in that The preset iterative process includes: Obtaining the calculation results of the previous processing layer as input data for the current layer, and rearranging the calculation results; Perform multiple matrix-vector multiplications on the rearranged calculation results and all tensor cores of the current processing layer to obtain the calculation results of the current processing layer. Before each matrix-vector multiplication calculation, the output of the previous tensor core needs to be obtained and rearranged. The input data of the first processing layer is the input data obtained from the local storage unit in sequence according to the size of the sliding window, and the calculation result of the last processing layer is the final calculation result.

3. The accelerator according to claim 2, characterized in that The in-memory computing array is configured to rearrange the tensor cores according to their dimensions and a preset decomposition rank to obtain tensor cores for performing matrix-vector multiplication calculations.

4. The accelerator according to claim 3, characterized in that The steps for each matrix-vector multiplication calculation include: Get the output of the previous Tensor Core As the input data of the current tensor core, it is rearranged into n-sized ones according to the dimension. k r d-2 ×m1…m k-1 n k+1 …n q The matrix of r0 Among them, n k express The kth row, m k-1 express The k-1th column, r d-2 represents the decomposition rank of the d-2th tensor core; Get the d-th tensor core matrix G hd , and rearrange it into m-sized k r d ×n k r d-1 Matrix Among them, r d represents the decomposition rank of the d-th tensor core, r d-1 represents the decomposition rank of the d-1th tensor core; calculate The results are rearranged into Get the output result of the current tensor core.

5. The accelerator according to claim 4, characterized in that The calculation method of the calculation results of each processing layer is: in, Represents the calculation result of the hth processing layer, represents the calculation result of the h-1th processing layer, G hd represents the d-th tensor core of the h-th processing layer, × d The operator representing the d-th multiplication operation, d∈[1,p].

6. The accelerator according to any one of claims 1 to 5, characterized in that: The in-memory computing processing unit further includes a digital-to-analog conversion unit, a sampling and holding unit, an analog-to-digital conversion unit, a shift-and-accumulate unit, and an output buffer unit, wherein: The digital-to-analog conversion unit is used to convert the input data of the neural network model from a digital signal to an analog signal, and input the analog signal to the corresponding in-memory computing array, so that the in-memory computing array performs a matrix-vector multiplication calculation; The sampling and holding unit is used to sample the calculation results of the in-memory calculation array and input the sampling results into the analog-to-digital conversion unit; The analog-to-digital conversion unit is used to convert the sampling result into a digital signal and input it into the shift accumulation unit; The shift accumulation unit is used to perform shift accumulation on the conversion result in the analog-to-digital conversion unit, and store the shift accumulation result in the output buffer unit.

7. The accelerator according to any one of claims 1 to 5, characterized in that: The local storage unit is further used to store the shift accumulation result stored in the output buffer unit, wherein the local storage unit performs data transmission with the in-memory computing processing unit via a bus.

8. The accelerator according to any one of claims 1 to 5, characterized in that: The post-processing unit is configured to perform post-processing according to the result of the matrix-vector multiplication calculation, and the post-processing includes accumulation processing, activation processing and / or pooling processing.

9. The accelerator according to any one of claims 1 to 5, characterized in that: The tensor core is obtained as follows: Obtaining a neural network model to be compressed, wherein the neural network model includes multiple processing layers; Obtaining a training set, wherein the training set includes a plurality of image samples and labels indicating categories to which the image samples belong; Determine the tensor decomposition form of the original weights of each processing layer according to the dimension of the original weights corresponding to each processing layer; Performing vector decomposition on the original weights of each processing layer according to the tensor decomposition form and the preset decomposition rank to obtain an initial tensor core corresponding to each processing layer, wherein all the initial tensor cores constitute an initial compression model corresponding to the model; The initial compression model is fine-tuned using the training set to obtain a trained compression model, which includes the fine-tuned tensor cores of each processing layer.

10. A method for accelerating neural network model reasoning using the in-memory accelerator according to any one of claims 1 to 9, characterized in that: The method comprises: Obtain the input data of the neural network model and the tensor cores of each processing layer; Obtaining corresponding input data in sequence according to the size of the sliding window, and rearranging the input data and the corresponding tensor cores according to their dimensions to obtain rearranged input data and tensor cores; Performing matrix-vector multiplication on the rearranged input data and the tensor core according to a preset iterative calculation process; The result of the matrix-vector multiplication calculation is post-processed to obtain the inference result of the neural network model.

Images