A heterogeneous storage and computing device and method for accelerating ViT neural network
By combining the photoelectric memory and computing integrated device and the DRAM memory and computing integrated device, the matrix vector multiplication operation of the ViT neural network is optimized, which solves the problems of power consumption and area overhead in hardware implementation, and achieves more efficient computing speed and lower work Consumption.
Patent Information
- Application Number
- CN202411104910.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-13
AI Technical Summary
When implementing the hardware, the ViT neural network has large parameters and dense matrix vector multiplication operations, resulting in power consumption walls and storage wall problems. The traditional von Neumann architecture is difficult to achieve efficient acceleration. Optoelectronic computing arrays require a large amount of hardware area when dealing with frequent weight transformations. and high time and power consumption overhead.
A heterogeneous computing device was designed, combining the integrated optical and electronic memory and DRAM memory and integrated DRAM memory and computing devices. The optical and electrical matrix vector multiplication unit performs fixed weight operations, and the DRAM matrix vector multiplication unit performs frequent update operations, reducing the hardware area. and the overhead of accessing off-chip storage of the computing unit.
It realizes smaller hardware area, faster computing speed and lower power consumption, significantly accelerating the inference process of ViT neural networks.
Smart Images

Figure CN118627566B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a heterogeneous storage and computing device and method for accelerating a ViT neural network, and belongs to the field of hardware accelerated neural networks. Background Art
[0002] Traditional Transformer neural networks have shown excellent performance in the field of natural language processing and have been widely used, but they have not shown obvious adaptability to the field of computer vision. The ViT (Vision Transformer) neural network utilizes the multi-head attention calculation mechanism in the traditional Transformer network, which can capture richer feature information and aggregate features at different levels. By building a suitable network structure, it is cleverly applied to the field of computer vision, and therefore shows obvious performance advantages over traditional convolutional neural networks.
[0003] However, the ViT neural network has a large number of parameters and involves a large number of matrix-vector multiplication operations. When implementing it in hardware, if a processor based on the traditional von Neumann architecture is used, then the neural network acceleration task will be limited by the power consumption wall and the storage wall.
[0004] The storage-computing integrated device is a new type of computing device that integrates storage and computing functions. It can effectively solve the bottleneck problem caused by data transfer in the traditional von Neumann architecture and show better performance when facing intensive computing. The Chinese patent with publication number CN 110276440 A - A convolution operation accelerator based on an optoelectronic computing array and its method discloses a scheme for implementing a matrix-vector multiplication calculation unit by replacing the traditional hardware with an optoelectronic computing array. This scheme can achieve good results for deep neural networks whose weights do not need to change frequently. However, the multi-head attention layer of the ViT neural network involves a calculation process in which the weights change frequently. If only the optoelectronic computing array is used to implement this scheme, a large number of optoelectronic storage-computing integrated devices will undoubtedly be needed, resulting in a large amount of hardware area overhead. At the same time, the computing unit will repeatedly access the off-chip storage, resulting in higher time and power consumption overhead. Summary of the invention
[0005] In order to achieve smaller area overhead, faster computing speed and lower power consumption, the present invention provides a heterogeneous storage and computing device for accelerating ViT neural networks, the device comprising: multiple cascaded encoding modules and a classification prediction module, each encoding module comprising two normalization sub-modules, a multi-head attention calculation sub-module, two addition units and a fully connected layer sub-module; the normalization sub-module is used to perform layer normalization operations on the input matrix; the multi-head attention calculation sub-module is used to perform multi-head attention calculations on the input matrix, and pass the output matrix into the addition unit; the addition unit is used to add the input matrix to the output matrix; the fully connected layer sub-module is used to perform two full connection operations and two linear rectification operations on the input data, and pass the output matrix into the addition unit.
[0006] The classification prediction module includes a data storage unit, an optoelectronic matrix vector multiplication unit, and a softmax function submodule, which are used to perform a full connection operation on the output matrix of the upper encoding module and realize the classification output function.
[0007] Furthermore, the multi-head attention calculation submodule includes four optoelectronic matrix-vector multiplication units, a matrix splitting unit, a DRAM matrix-vector multiplication unit, a softmax function submodule, and a matrix splicing unit; the optoelectronic matrix-vector multiplication unit is used to perform matrix-vector multiplication operations with fixed weights; the matrix splitting unit is used to split a single input matrix into multiple matrices by column; the DRAM matrix-vector multiplication unit is used to perform matrix-vector multiplication operations with frequently updated weights; the softmax function submodule is used to perform softmax function transformation on the input matrix; the matrix splicing unit is used to splice multiple input matrices into a single matrix by column.
[0008] Furthermore, the fully connected layer submodule includes two optoelectronic matrix-vector multiplication units and two linear rectification units connected in sequence.
[0009] Furthermore, the classification prediction module includes a data storage unit, an optoelectronic matrix vector multiplication unit and a softmax function submodule which are connected in sequence.
[0010] The present invention also provides a method for using the above-mentioned heterogeneous storage and computing device to accelerate the ViT neural network, comprising the following steps:
[0011] (1) Encoding: Input the encoding module input data to the normalization submodule and perform layer normalization operation. Input the layer normalization result to the multi-head attention calculation submodule and perform multi-head attention calculation. Use the addition unit to add the input matrix and output matrix in the multi-head attention calculation submodule. Use the normalization submodule to perform layer normalization operation on the addition result. Then input the layer normalization result to the fully connected layer submodule and perform full connection operation and linear rectification operation. Then use the addition unit to add the input matrix and output matrix of the fully connected layer submodule to obtain the output matrix of the first-level encoding module. Then use it as the input data of the second-level encoding module to continue encoding until the final output matrix of the encoding module is obtained after passing through all encoding modules.
[0012] (2) Classification prediction: used to perform a full connection operation on the matrix finally output by the encoding module and classify the output, thereby obtaining the final classification prediction result.
[0013] The beneficial effects of the present invention are:
[0014] The present invention provides a heterogeneous storage and computing device and method for accelerating the ViT neural network, which utilizes an array of optoelectronic storage and computing devices to realize the storage and calculation of fixed values in the ViT neural network, and utilizes a DRAM storage and computing device to realize the storage and calculation of constantly changing weight values in the ViT neural network. The advantages of the two are combined to perform intensive matrix-vector multiplication operations, which reduces the overall area of the hardware architecture while greatly reducing the time and power consumption overhead caused by the computing unit repeatedly accessing off-chip storage, thereby accelerating the reasoning process of the ViT neural network. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0016] Figure 1 It is an overall block diagram of the heterogeneous storage and computing device for accelerating the ViT neural network provided by the present invention;
[0017] Figure 2 It is a schematic diagram of the internal structure of the multi-head attention calculation submodule in the heterogeneous storage and computing device for accelerating the ViT neural network provided by the present invention;
[0018] Figure 3 It is a schematic diagram of the internal structure of the fully connected layer submodule in the heterogeneous storage and computing device for accelerating the ViT neural network provided by the present invention;
[0019] Figure 4It is a schematic diagram of the calculation array of the optoelectronic matrix-vector multiplication unit in the heterogeneous storage and computing device for accelerating the ViT neural network provided by the present invention;
[0020] Figure 5 It is a schematic diagram of the computing array of the DRAM matrix-vector multiplication unit in the heterogeneous storage and computing device for accelerating the ViT neural network provided by the present invention. DETAILED DESCRIPTION
[0021] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0022] Embodiment 1:
[0023] This embodiment provides a heterogeneous storage and computing device and method for accelerating the ViT neural network. Figure 1 The device includes: multiple cascaded encoding modules and a classification prediction module, each encoding module includes two normalization submodules, a multi-head attention calculation submodule, two addition units and a full connection layer submodule; wherein the normalization submodule is used to perform layer normalization operation on the input matrix; the multi-head attention calculation submodule is used to perform multi-head attention calculation on the input matrix, the addition unit is used to add the input matrix to the output matrix, and the full connection layer submodule is used to perform two full connection operations and two linear rectification operations on the input data. The classification prediction module is used to perform a full connection operation on the output matrix of the upper encoding module and realize the classification output function, including a data storage unit, a first photoelectric matrix vector multiplication unit and a first softmax function submodule.
[0024] For ease of description, the two normalization submodules in each encoding module are respectively recorded as the first normalization submodule and the second normalization submodule, and the two addition units are respectively recorded as the first addition unit and the second addition unit; Figure 1 As shown, the first normalization submodule, the multi-head attention calculation submodule, the first addition unit, the second normalization submodule, the fully connected layer submodule and the second addition unit are connected in sequence; the first normalization submodule performs layer normalization operation on the input matrix and then inputs it into the multi-head attention calculation submodule, the multi-head attention calculation submodule performs multi-head attention calculation on the input matrix, and passes the output matrix into the first addition unit; the first addition unit adds the input matrix and the output matrix and then inputs it into the second normalization submodule for normalization operation, the fully connected layer submodule performs two full connection operations and two linear rectification operations on the data after the normalization operation, and passes the output matrix into the second addition unit.
[0025] In the classification prediction module, the data storage unit, the first photoelectric matrix-vector multiplication unit and the first softmax function submodule are connected in this order; the data storage unit is used to store the output result of the last-level encoding module and is input to the first photoelectric matrix-vector multiplication unit as an excitation, the first photoelectric matrix-vector multiplication unit is used for matrix-vector multiplication operations with fixed weights, and the first softmax function submodule is used to perform softmax function operations on the output results of the first photoelectric matrix-vector multiplication unit.
[0026] like Figure 2 As shown, the multi-head attention calculation submodule in this implementation scheme includes four optoelectronic matrix-vector multiplication units, a matrix splitting unit, a DRAM matrix-vector multiplication unit, a second softmax function submodule, and a matrix splicing unit; wherein the optoelectronic matrix-vector multiplication unit is used to perform matrix-vector multiplication operations with fixed weights; the matrix splitting unit is used to split a single input matrix into multiple matrices by column; the DRAM matrix-vector multiplication unit is used to perform matrix-vector multiplication operations with frequently updated weights; the second softmax function submodule is used to perform softmax function transformation on the input matrix; and the matrix splicing unit is used to splice multiple input matrices into a single matrix by column. For ease of description, the four optoelectronic matrix-vector multiplication units of the multi-head attention calculation submodule are respectively recorded as the second optoelectronic matrix-vector multiplication unit, the third optoelectronic matrix-vector multiplication unit, the fourth optoelectronic matrix-vector multiplication unit, and the fifth optoelectronic matrix-vector multiplication unit, as shown in Figure 2 As shown, the data output from the previous level are respectively input into the second photoelectric matrix-vector multiplication unit, the third photoelectric matrix-vector multiplication unit and the fourth photoelectric matrix-vector multiplication unit to perform matrix-vector multiplication operations with fixed weights, and the results of each matrix-vector multiplication operation are then input into the matrix splitting unit to be split into multiple small matrices, and multiplication-accumulation operations are performed with the weights in the DRAM calculation array, and then the multiplication-accumulation output matrix is input into the second softmax function submodule, and after being changed by the softmax function, it is input into the matrix splicing unit, and a single matrix is obtained by splicing, and finally the matrix-vector multiplication operation with fixed weights is performed again in the fifth photoelectric matrix-vector multiplication unit; the results output by the second photoelectric matrix-vector multiplication unit, the third photoelectric matrix-vector multiplication unit and the fourth photoelectric matrix-vector multiplication unit are respectively split into multiple small matrices by the matrix splitting unit. When the matrix-vector multiplication operation is performed in the DRAM matrix-vector multiplication unit, the weights stored in the DRAM matrix-vector multiplication unit are repeatedly erased and written according to the calculation requirements, thereby saving a large number of devices required to store weights using the photoelectric matrix-vector multiplication unit.
[0027] like Figure 3As shown, the fully connected layer submodule in this implementation scheme includes two photoelectric matrix-vector multiplication units and two linear rectification units connected in sequence. The linear rectification unit is used to perform a linear rectification (RELU function) operation on the input matrix. For the convenience of description, the two photoelectric matrix-vector multiplication units in the fully connected layer submodule are respectively recorded as the sixth photoelectric matrix-vector multiplication unit and the seventh photoelectric matrix-vector multiplication unit, and the two linear rectification units are respectively recorded as the first linear rectification unit and the second linear rectification unit; as shown Figure 3 As shown, the sixth photoelectric matrix-vector multiplication unit, the first linear rectification unit, the seventh photoelectric matrix-vector multiplication unit and the second linear rectification unit are connected in sequence.
[0028] like Figure 4 and Figure 5 As shown in FIG. 1 , they are respectively schematic diagrams of the calculation arrays of the optoelectronic matrix-vector multiplication unit and the DRAM matrix-vector multiplication unit in the above device; Figure 4 As shown, the optoelectronic matrix vector multiplication unit is a 768×768 cross array composed of optoelectronic storage and computing integrated devices (i.e., the optoelectronic storage and computing integrated devices disclosed in the Chinese patent publication number CN 110276440 A - A convolution operation accelerator based on an optoelectronic computing array and its method), each row of the cross array is called a word line, each column is called a bit line, and the optoelectronic storage and computing integrated devices are located at the intersection of rows and columns. Figure 5 As shown, the DRAM matrix-vector multiplication unit is a 197×197 cross array composed of a DRAM storage-computing integrated computing array. Each row of the cross array is called a word line, and each column is called a bit line. Each column in the array contains a DAC for writing weights to the device. The DRAM storage-computing integrated device is located at the intersection of rows and columns.
[0029] If an optoelectronic storage and computing integrated device is used, it will be difficult to achieve the same function as the DRAM matrix-vector multiplication unit in the present application. This is because each input stimulus in the ViT neural network is different, and the weights stored in the DRAM matrix-vector multiplication unit will change with the stimulus. Therefore, it is necessary to combine a DRAM storage and computing integrated device that can be frequently erased to realize this calculation process. If it is implemented through an optoelectronic storage and computing integrated device, each input stimulus will take at least twice the area of the DRAM matrix-vector multiplication unit. Since the stimulus is constantly changing, the corresponding area will increase exponentially. Therefore, the ViT neural network cannot be realized using an optoelectronic storage and computing integrated device.
[0030] according to Figures 1 to 5 The specific steps of using the device to accelerate the ViT neural network in this embodiment are as follows, taking a twelve-level encoding module as an example for explanation:
[0031] (1) In this embodiment, the input matrix I of the first-level encoding module has a dimension of 197×768, which is input to the first normalization submodule to perform layer normalization operation to obtain a 197×768-dimensional matrix A, which is then input to the multi-head attention submodule. The 768×768-dimensional weight matrix Stored in the second photoelectric matrix vector multiplication unit, using matrix A and weight matrix Multiply them together to get a 197×768 dimensional matrix Q; similarly, pre-compute a 768×768 dimensional weight matrix , After being stored in the third photoelectric matrix-vector multiplication unit and the fourth photoelectric matrix-vector multiplication unit respectively, the matrix A and the weight matrix can be obtained based on the photoelectric matrix-vector multiplication unit. , The 197×768-dimensional matrices K and V are multiplied and output respectively, and the calculation formula is:
[0032] (1)
[0033] (2) Input the 197×768 dimensional matrices Q, K, and V into the matrix splitting unit, and split Q, K, and V into n matrices of 197×768 dimensions by column. A small matrix, that is .
[0034] (3) Then, 197× Dimensional Matrix After transposition, it is input into the DRAM matrix-vector multiplication unit and stored in the DRAM matrix-vector multiplication unit as the weight. Dimensional Matrix As the matrix input stimulus, the multiplication and accumulation operation is performed with the weight of the DRAM matrix-vector multiplication unit, and then the multiplication and accumulation output matrix is input into the second softmax function submodule, and the 197×197 dimensional matrix is obtained after the softmax function is changed. , then erase the weights stored in the DRAM matrix-vector multiplication unit, and then store 197× Dimensional Matrix Input to the DRAM matrix-vector multiplication unit and stored in the DRAM matrix-vector multiplication unit as the new weight, converting the 197×197 dimensional matrix As the matrix input stimulus, the multiplication and accumulation operation is performed with the weights in the DRAM matrix-vector multiplication unit, which means that an attention calculation is completed, so that 197× The output matrix of Similarly, for the remaining matrices The attention calculation is also performed in sequence to obtain the remaining 197× Dimensional output matrix , and its calculation formula is:
[0035] (2)
[0036] (3) n 197× Dimensional Matrix After the matrix splicing unit is input, a matrix splicing operation is performed to splice the n input matrices into a matrix O with a dimension of 197×768 by column. After that, the 768×768-dimensional weight matrix is input into the fifth photoelectric matrix vector multiplication unit and stored as weights in the fifth photoelectric calculation array. The 197×768-dimensional matrix O is used as the input excitation and the weights are multiplied and accumulated to obtain the calculation result of the multi-head attention calculation submodule. The input matrix and the output matrix in the multi-head attention calculation submodule are added by the addition unit, and the layer normalization operation is performed on the addition result by the normalization submodule; the layer normalization result is then input into the fully connected layer submodule, and the fully connected operation is performed. After that, the input matrix and the output matrix of the fully connected layer submodule are added by the addition unit, and the 197×768-dimensional output matrix of the first-level encoding module is obtained. Then the output matrix of the current level encoding module is used as the input matrix of the next level encoding module, and the encoding operation is continued until the twelfth level encoding module is passed, thereby obtaining the final output matrix of the encoding module.
[0037] (4) Finally, the final output vector of the twelve-level encoding module is input into the data storage unit in the classification prediction module as the input excitation matrix of the classification prediction module. The photoelectric matrix-vector multiplication unit in the classification prediction module performs the matrix-vector multiplication operation of the input excitation matrix and the weight matrix to obtain a 197×10 dimensional output vector. The softmax function submodule in the classification prediction module performs the softmax function transformation on the matrix-vector multiplication unit output vector, thereby obtaining the final classification prediction result of the ViT neural network.
[0038] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A heterogeneous storage and computing device for accelerating ViT neural network, characterized in that: The heterogeneous storage and computing device includes a plurality of cascaded encoding modules and a classification prediction module; wherein each encoding module includes two normalization submodules, a multi-head attention calculation submodule, two addition units and a fully connected layer submodule; the multi-head attention calculation submodule uses an optoelectronic matrix vector multiplication unit to perform a matrix vector multiplication operation with fixed weights, and uses a DRAM matrix vector multiplication unit to perform a matrix vector multiplication operation with frequently updated weights; The multi-head attention calculation submodule includes four photoelectric matrix-vector multiplication units, a matrix splitting unit, a DRAM matrix-vector multiplication unit, a second softmax function submodule and a matrix splicing unit; wherein the four photoelectric matrix-vector multiplication units are respectively recorded as the second photoelectric matrix-vector multiplication unit, the third photoelectric matrix-vector multiplication unit, the fourth photoelectric matrix-vector multiplication unit and the fifth photoelectric matrix-vector multiplication unit, the second photoelectric matrix-vector multiplication unit, the third photoelectric matrix-vector multiplication unit and the fourth photoelectric matrix-vector multiplication unit are respectively used to perform matrix-vector multiplication operations with fixed weights on the data output by the previous level, and input the operation results into the matrix splitting unit to split them into multiple small matrices and then input them into the DRAM matrix-vector multiplication unit, the multiple small matrices are respectively multiplied and added with the weights of the DRAM matrix-vector multiplication unit and then input into the matrix splicing unit for splicing, and the spliced matrix is input into the fifth photoelectric matrix-vector multiplication unit to perform multiplication and addition operations and then output the results.
2. The heterogeneous storage and computing device according to claim 1, characterized in that: The classification prediction module is used to perform a full connection operation on the output matrix of the upper encoding module and realize the classification output function; the classification prediction module includes a data storage unit, a first photoelectric matrix vector multiplication unit and a first softmax function submodule connected in sequence.
3. The heterogeneous storage and computing device according to claim 2, characterized in that: The fully connected layer submodule includes two photoelectric matrix-vector multiplication units and two linear rectification units connected in sequence, and the linear rectification unit is used to perform a linear rectification operation on an input matrix.
4. The heterogeneous storage and computing device according to claim 3, characterized in that: The linear rectification unit adopts a RELU function.
5. A heterogeneous storage and computing method for accelerating ViT neural network, characterized in that: The method is implemented based on the heterogeneous storage and computing device described in any one of claims 1 to 4. The method uses an optoelectronic matrix-vector multiplication unit to accelerate operations of fixed weights in the ViT neural network, and uses a DRAM matrix-vector multiplication unit to implement operations with frequently updated weights.
6. The heterogeneous storage and computing method according to claim 5, characterized in that: The method comprises: Step 1: The input of the first-level encoding module is matrix I, which is layer-normalized to obtain matrix A; Step 2: Pre-set the fixed weight matrix , , Stored in the corresponding photoelectric matrix-vector multiplication unit in the multi-head attention calculation submodule, and the matrix Q, K and V are obtained after performing the matrix-vector multiplication operation; Step 3: Split the matrices Q, K, and V into n small matrices. ; Step 4: Transform the matrix After transposition, the matrix vector multiplication unit is input into DRAM and stored as weight. As matrix input stimulus, perform multiplication and accumulation operation with the weights stored in the DRAM matrix vector multiplication unit and then perform softmax function transformation to obtain the matrix ; Step 5: Erase the weights stored in the DRAM matrix-vector multiplication unit and store the matrix Input to DRAM matrix-vector multiplication unit for storage as new weights; Step 6: Transform the matrix As matrix input stimulus, with the new weights in the DRAM matrix-vector multiplication unit After performing the multiplication and accumulation operation, the output matrix is obtained ; Step 7: For the remaining matrices Execute steps 4 to 6 to get the output matrix ; Step 8: n matrices Perform matrix concatenation to obtain matrix O; Step 9, after performing a matrix-vector multiplication operation with fixed weights on the matrix O, the output matrix of the multi-head attention calculation submodule is obtained; Step 10, add the input matrix and output matrix in the multi-head attention calculation submodule, perform layer normalization operation and full connection operation, and then add the input matrix and output matrix of the full connection operation to obtain the output matrix of the first-level encoding module; Step 11, using the output of the first-stage coding module as the input matrix of the next-stage coding module, and continuing to perform the coding operation until the final output matrix of the coding module is obtained after passing through all coding modules; Step 12: Input the final output matrix of the encoding module into the classification prediction module to obtain the final classification prediction result.
