Transformer neural network acceleration device and method based on optoelectronic storage and computing integrated device

Abstract

The application provides a transformer neural network acceleration device based on a photoelectric storage and calculation integrated device and a method thereof.The device comprises: (1) an encoding module comprising: a multi-head attention calculation submodule for performing a multi-head attention calculation operation on an input vector; a feedforward fully connected layer submodule for performing a fully connected operation and a linear rectification operation; an addition unit for adding the input vector and the output vector in the upper module; a normalization submodule for performing a normalization operation on the input vector from the addition unit; (2) a decoding module comprising: a multi-head attention calculation submodule, a feedforward fully connected layer submodule, an addition unit and a normalization submodule; (3) a classification prediction module for performing a fully connected operation on a one-dimensional vector output by the upper module and performing a classification output operation.The device of the application can accelerate the implementation of the transformer neural network and greatly reduce the power consumption, area and calculation delay of the hardware architecture.

Description

Technical Field

[0001] This invention relates to a transformer neural network acceleration device and method based on opto-in-memory computing devices, belonging to the technical field of hardware-accelerated neural networks. Background Technology

[0002] The development of neural networks is inseparable from the continuous improvement of computing power and the continuous optimization of computing architecture. In the traditional von Neumann architecture, data processing units and storage units are separate. In practical applications, data needs to be repeatedly read from off-chip storage units and transferred to on-chip computing units, resulting in a large amount of time and energy consumption. General-purpose processors under the traditional von Neumann architecture face difficulties when performing neural network acceleration tasks. Therefore, scientists have proposed a new type of processor based on in-memory computing devices, which integrates memory access and computation into one, thereby achieving superior performance in accelerating neural networks.

[0003] Transformer neural network models have wide applications in many fields such as natural language processing and computer vision. However, due to the large number of matrix-vector multiplication operations in neural networks, the extensive data movement in traditional von Neumann architectures leads to high computational latency and high energy consumption. Computation arrays based on optoelectronic in-memory computing devices (such as CN110276440A) can effectively solve this problem by integrating memory access and computation into a single operation, thereby significantly reducing the energy and time consumed during neural network acceleration. Summary of the Invention

[0004] This invention aims to provide a device for accelerating transformer neural networks based on opto-in-memory computing devices, achieving lower latency, lower power consumption, and smaller area. Another objective of this invention is to provide a method for accelerating transformer neural networks using the aforementioned device.

[0005] The technical solution adopted by the device of the present invention is as follows:

[0006] A transformer neural network acceleration device based on opto-in-memory computing devices, the device comprising:

[0007] The encoding module includes a first multi-head attention calculation submodule, a first feedforward fully connected layer submodule, two first addition units, and two first normalization submodules. The first multi-head attention calculation submodule performs multi-head attention calculation on the input vector and passes the output vector to a first addition unit. The first feedforward fully connected layer submodule performs fully connected operations and linear rectification operations on the input data and passes the output vector to another first addition unit. The first addition unit adds the input vector and the output vector and passes the result to the normalization submodule. The first normalization submodule performs a normalization operation on the input vector from the first addition unit.

[0008] The decoding module includes two second multi-head attention calculation submodules, a second feedforward fully connected layer submodule, three second addition units, and three second normalization submodules. The second multi-head attention calculation submodules perform multi-head attention calculation operations on the input vector and pass the output vector to the second addition units. The second feedforward fully connected layer submodules perform fully connected operations and linear rectification operations on the input data and pass the output vector to the second addition units. The second addition units add the input vector and the output vector and pass the result to the second normalization submodules. The second normalization submodules perform normalization operations on the input vector from the second addition units.

[0009] The classification prediction module performs a fully connected operation on the one-dimensional vector output by the decoding module, and then performs a classification output operation to obtain the final classification prediction result.

[0010] Furthermore, both the first and second multi-head attention calculation submodules include a data storage unit, a first matrix-vector multiplication unit, a multiplication unit, a mask selection submodule, and a first softmax function submodule connected in sequence. The data storage unit is used to buffer input data and output data. The first matrix-vector multiplication unit is used to perform multi-head attention operations within the multi-head attention calculation submodule. The multiplication unit is used to perform multiplication operations on the input data. The mask selection submodule is enabled when the enable signal is high, sets some elements of the input matrix to zero, and outputs the matrix directly when the enable signal is low. The first softmax function submodule is used to perform normalization operations on the one-dimensional input vector.

[0011] Furthermore, both the first feedforward fully connected layer submodule and the second feedforward fully connected layer submodule include a data register unit, a second matrix-vector multiplication unit, and a linear rectification unit connected in sequence; the second matrix-vector multiplication unit is used to perform the fully connected operation within the feedforward fully connected layer submodule.

[0012] Furthermore, the classification prediction module includes a data storage unit, a third matrix-vector multiplication unit, and a second softmax function submodule connected in sequence; the second softmax function submodule outputs the classification probability.

[0013] Furthermore, the matrix-vector multiplication unit includes a word line driving circuit and a bit line driving circuit, a computing array based on opto-in-memory computing devices, a source line driving circuit, a current-to-voltage conversion circuit, an analog-to-digital conversion circuit, and a shift-to-accumulate circuit connected in sequence; the output terminal of the word line driving circuit is connected to the computing array based on opto-in-memory computing devices.

[0014] This invention also provides a method for using the above-mentioned transformer neural network acceleration device based on an opto-in-memory computing device, comprising the following steps:

[0015] (1) Encoding: Input data from the encoding module is input to the first multi-head attention calculation submodule to perform multi-head attention operation. The input vector and output vector of the first multi-head attention calculation submodule are added by the first addition unit. Then, the result is normalized by the first normalization submodule. The normalized result is input to the first feedforward fully connected layer submodule to perform fully connected operation and linear rectification operation. Then, the input vector and output vector of the first feedforward fully connected layer submodule are added and the result is normalized to obtain the output vector of the encoding module.

[0016] (2) Decoding: Input data from the decoding module is input to the second multi-head attention calculation submodule to perform multi-head attention operation. The input vector and output vector of the second multi-head attention calculation submodule are added by the second addition unit, and the result is normalized by the second normalization submodule. The normalized result and the output vector of the encoding module are input to the next second multi-head attention calculation submodule, and the input vector and output vector of the second multi-head attention calculation submodule are added and normalized. The normalized result is then input to the second feedforward fully connected layer submodule to perform fully connected operation and linear rectification operation. The input vector and output vector of the second feedforward fully connected layer submodule are then added and the result is normalized to obtain the output vector of the decoding module.

[0017] (3) Classification prediction: This is used to perform a fully connected operation on the one-dimensional vector output by the decoding module, and then perform a classification output operation to obtain the final classification prediction result.

[0018] This invention provides a device and method for accelerating transformer neural networks. By utilizing the advantages of opto-in-memory computing arrays in performing matrix-vector multiplication operations, the power consumption caused by repeated access to off-chip memory by the computing unit is significantly reduced, thereby accelerating the inference process of the transformer neural network model. Attached Figure Description

[0019] Figure 1 This is an overall block diagram of the device of the present invention;

[0020] Figure 2 This is a schematic diagram of the internal structure of (a) the multi-head attention computation submodule and (b) the feedforward fully connected layer submodule;

[0021] Figure 3 This is a schematic diagram of the internal structure of a matrix-vector multiplication unit;

[0022] Figure 4 This is a schematic diagram of an array of optoelectronic in-memory computing devices. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is an overall block diagram of the Transformer neural network acceleration device based on an opto-in-memory computing device in this embodiment. It includes: six encoding modules, each comprising a multi-head attention calculation submodule, a feedforward fully connected layer submodule, two addition units, and a normalization submodule. The multi-head attention calculation submodule performs multi-head attention calculation on the input vector and passes the output vector to an addition unit. The feedforward fully connected layer submodule performs fully connected operations and linear rectification operations on the input data and passes the output vector to another addition unit. The addition unit adds the input vector and output vector from the previous module and passes the result to the normalization module. The normalization submodule performs normalization operations on the input vector from the addition unit. There are also six decoding modules, each comprising a multi-head attention calculation submodule, a feedforward fully connected layer submodule, an addition unit, and a normalization submodule. Finally, a classification prediction module performs a fully connected operation on the one-dimensional vector output by the previous decoding module, and then performs a classification output operation to obtain the final classification prediction result.

[0025] like Figure 2As shown, the multi-head attention computation submodule in this embodiment includes a data storage unit, a matrix-vector multiplication unit, a multiplication unit, a mask selection submodule, and a softmax function submodule. The data storage unit is used to buffer input and output data. The matrix-vector multiplication unit is used to perform multi-head attention computation and the mapped matrix-vector multiplication operation in the fully connected layer computation. The multiplication unit is used to perform multiplication operations on the input data. The mask selection submodule is enabled when the enable signal is high, setting some elements of the input matrix to zero before outputting; it is disabled when the enable signal is low, directly outputting the input matrix. The softmax function submodule is used to perform normalization operations on the one-dimensional input vector. The feedforward fully connected layer submodule includes a data storage unit, a matrix-vector multiplication unit, and a linear rectification unit; the linear rectification unit is used to perform ReLU (linearly rectified) activation function operations on the input data. The classification prediction module includes a data storage unit, a matrix-vector multiplication unit, and a softmax function submodule.

[0026] Figure 3 This is a schematic diagram of the matrix-vector multiplication unit in this implementation scheme. The matrix-vector multiplication unit includes a bit line driving circuit, a word line driving circuit, a source line driving circuit, a computing array based on opto-in-memory computing devices, a current-to-voltage conversion circuit, an analog-to-digital conversion circuit, and a shift-accumulator circuit, used to perform the mapped matrix-vector multiplication operation in multi-head attention operations and fully connected layer operations.

[0027] Figure 4 This is a schematic diagram of the computing array in this embodiment. The computing array is an M×N cross array composed of opto-in-memory computing devices (such as CN110276440A). Each row of the computing array is called a word line, and each column is called a bit line. The opto-in-memory computing devices are located at the intersection of rows and columns.

[0028] The opto-in-memory computing devices in the aforementioned computing array can only store non-negative numbers. However, when performing matrix-vector multiplication in the Transformer neural network, the elements within the matrix may be negative. We solved this problem by using a double array method. The first array is used to store the positive values ​​in the weights, and the second matrix is ​​used to store the absolute values ​​of the negative values ​​in the weights. The remaining positions in both arrays are set to zero, thus achieving the purpose of storing negative weights. Then, during the matrix-vector multiplication operation, the result of the matrix-vector multiplication operation of the first matrix is ​​subtracted from the result of the operation of the second matrix to obtain the final operation result containing negative weights.

[0029] according to Figures 1 to 4 The specific steps for accelerating the transformer neural network using this device in this embodiment are as follows:

[0030] (1) In this embodiment, the input vector I of the first encoding module has a dimension of 1×N. In this embodiment, N is 512. It is stored in the data register unit of the multi-head attention calculation submodule in the first encoding module. The expression of the one-dimensional vector I is I=[i1,i2,……,i 512 After selecting the opto-in-memory computing device to be operated on in the matrix-vector multiplication unit of the multi-head attention calculation submodule through the bit line driving circuit, word line driving circuit, and source line driving circuit, a working voltage is applied to the source, drain, and gate of the opto-in-memory computing device. Subsequently, exposure operations of different durations are performed on the device unit to be operated on, thereby increasing the 512×512 dimensional weight matrix W. q The input excitation vector I is stored in a computing array. It is input sequentially from least significant bit to most significant bit into the opto-in-memory computing (OIM) array. The input vector is then multiplied element-wise with the weights stored in the OIM array cells. The sources of the OIM devices in the same column are connected, and the source lines of the OIM array output accumulated photocurrent. The analog current is then converted to a digital value through a current-to-voltage converter and an analog-to-digital converter. Finally, the digital output is multiplied by a shift-accumulator circuit, achieving matrix-vector multiplication. After these steps, the input vector I and the weight matrix W are obtained. q The output after multiplication is a 1×512 dimensional vector Q. Similarly, the 512×512 dimensional weight matrix W... k W v After being stored in the opto-in-memory computing device array, the input vector I and the weight matrix W can be obtained based on the matrix-vector multiplication unit. k W v The formulas for calculating the 1×512 dimensional vectors K and V output after multiplication are as follows:

[0031]

[0032] (2) Input the first element of the 1×512 vector K into the multiplication unit, and perform multiplication with all elements of the 1×512 vector Q in sequence. Similarly, input the remaining elements of vector K into the multiplication unit and perform the same operation to obtain the 512×512 output matrix A, the calculation formula of which is:

[0033] A = K T Q (2)

[0034] (3) Input matrix A into the mask selection submodule. The enable signal determines whether the mask selection submodule is enabled: when the enable signal is high, the mask selection submodule is enabled and multiplies the input matrix element-wise with a lower triangular matrix where all elements are 1, then outputs the result to the next submodule. When the enable signal is low, the mask selection submodule directly outputs the input matrix. In the multi-head attention calculation submodule of the encoding module, the mask selection submodule is disabled, and matrix A is directly output to the softmax function submodule. The softmax function submodule performs a normalization operation on each row of matrix A, making the sum of the elements in each row of matrix A equal to 1. The normalized matrix A is denoted as... And The data is stored in an opto-in-memory computing (OIM) array. Then, the one-dimensional input excitation vector V is sequentially input into the OIM array from least significant bit to most significant bit. The matrix-vector multiplication unit then multiplies the 1×512 dimensional input vector V with a 512×512 dimensional matrix. The matrix-vector multiplication operation is used to obtain the final 1×512 dimensional vector O output by the multi-head attention calculation submodule, and its calculation formula is as follows:

[0035]

[0036] (4) The 1×512-dimensional output vector O of the multi-head attention calculation submodule and the 1×512-dimensional input vector I of the multi-head attention calculation submodule are jointly fed into the addition unit in the encoding module to perform an addition operation. The addition result is output to the next-level normalization submodule to perform a normalization operation. The output vector of the normalization submodule is fed into the data register unit of the feedforward fully connected layer submodule as the input activation vector of the feedforward fully connected layer submodule. The one-dimensional input activation vector is multiplied by the matrix-vector multiplication unit of the feedforward fully connected layer submodule and the fully connected layer weight matrix W is then multiplied. f1 For the matrix-vector multiplication operation, the linear rectified unit in the feedforward fully connected layer submodule adds a ReLU (linear rectified) activation function to the output data of the matrix-vector multiplication unit, and then inputs the output vector of the linear rectified unit into the matrix-vector multiplication unit and the fully connected layer weight matrix W. f2 Perform a matrix-vector multiplication operation to obtain the final output vector of the feedforward fully connected layer submodule. Then, pass the input excitation vector and output vector of the feedforward fully connected layer submodule to the addition unit for addition. Then, output the addition result to the next-level normalization submodule for normalization. The normalization submodule will obtain the final 1×512-dimensional output vector of this encoding module. Then, use the output vector of this encoding module as the input vector of the next encoding module and repeat the internal operation of the encoding module until the final output results of 6 encoding modules are obtained.

[0037] (5) Store the 1×512-dimensional decoding module input vector J into the data register unit of the multi-head attention calculation submodule in the decoding module. The expression of the one-dimensional vector J is J=[j1,j2,……,j 512 After inputting the one-dimensional vector J into the multi-head attention calculation submodule, the masking module inside the multi-head attention calculation submodule is activated and a masking operation is performed. The remaining steps are completely consistent with the operation of the multi-head attention calculation submodule in the encoding module, so they are omitted. This will result in a 1×512-dimensional output vector. The input and output vectors of the multi-head attention calculation submodule will be jointly input into the next-level addition unit to perform a one-dimensional vector addition operation. The addition result will be output to the next-level normalization submodule to perform a normalization operation. The output data of the normalization submodule and the final output data of the six encoding modules will be jointly input into the next-level multi-head attention calculation submodule to perform multi-head attention calculation. At this time, the masking module inside the multi-head attention calculation submodule is closed, and no masking operation is performed. The steps are completely consistent with the operation of the multi-head attention calculation submodule in the encoding module.

[0038] (6) Then, the 1×512-dimensional output vector of the multi-head attention calculation submodule and the 1×512-dimensional output vector of the normalization submodule are input into the addition unit to perform the addition operation. The addition result is output to the next-level normalization submodule to perform the normalization operation. The output data of the normalization submodule is input into the data register unit of the feedforward fully connected layer submodule as the input excitation vector of the feedforward fully connected layer submodule. Here, the operation process of the feedforward fully connected layer submodule is completely consistent with the operation process of the feedforward fully connected layer submodule in the encoding module. Therefore, the expression is omitted, resulting in a 1×512 dimensional output vector. Then, the input excitation vector and output vector of the feedforward fully connected layer submodule are fed into the addition unit for addition. The addition result is then output to the next-level normalization submodule for normalization. The normalization submodule will obtain the final 1×512 dimensional output vector of this decoding module. The output vector of this decoding module is then used as the input vector of the next decoding module, and the internal operations of the decoding module are repeated to obtain the final 1×512 dimensional output vectors of the six decoding modules.

[0039] (7) Finally, the final output vectors of the 6 decoding modules are input into the data storage unit in the classification prediction module as the input activation vector of the classification prediction module. The matrix-vector multiplication unit in the classification prediction module performs a matrix-vector multiplication operation between the one-dimensional input activation vector and the weight matrix of the fully connected layer to obtain a 1×10-dimensional output vector. The softmax function submodule in the classification prediction module will perform a normalization operation on the output vector of the matrix-vector multiplication unit to obtain the classification prediction result of the transformer neural network.

Claims

1. A transformer neural network acceleration device based on optoelectronic in-memory computing device, characterized in that, The device comprises: The encoding module comprises a first multi-head attention calculation submodule, a first feedforward fully connected layer submodule, two first addition units and two first normalization submodules; the first multi-head attention calculation submodule is configured to perform a multi-head attention calculation operation on an input vector and transmit an output vector to one of the first addition units; the first feedforward fully connected layer submodule is configured to perform a fully connected operation and a linear rectification operation on input data and transmit an output vector to the other of the first addition units; the first addition unit is configured to add an input vector and an output vector and transmit a result to a normalization submodule; and the first normalization submodule is configured to perform a normalization operation on an input vector from the first addition unit; The decoding module comprises two second multi-head attention calculation submodules, a second feedforward fully connected layer submodule, three second addition units and three second normalization submodules; the second multi-head attention calculation submodule is configured to perform a multi-head attention calculation operation on an input vector and transmit an output vector to a second addition unit; the second feedforward fully connected layer submodule is configured to perform a fully connected operation and a linear rectification operation on input data and transmit an output vector to the second addition unit; the second addition unit is configured to add an input vector and an output vector and transmit a result to a second normalization submodule; and the second normalization submodule is configured to perform a normalization operation on an input vector from the second addition unit; The classification prediction module is configured to perform a fully connected operation on a one-dimensional vector output by the decoding module and then perform a classification output operation, thereby obtaining a final classification prediction result; the first multi-head attention calculation submodule and the second multi-head attention calculation submodule each comprise, in sequence, a data register unit, a first matrix-vector multiplication unit, a multiplication unit, a mask selection submodule and a first softmax function submodule; the data register unit is configured to buffer input data and output data; the first matrix-vector multiplication unit is configured to perform a multi-head attention operation in the multi-head attention calculation submodule; the multiplication unit is configured to perform a multiplication operation on input data; the mask selection submodule is configured to, when an enable signal is at a high level, output a part of elements of an input matrix after zeroing, and when the enable signal is at a low level, directly output the input matrix; and the first softmax function submodule is configured to perform a normalization operation on a one-dimensional input vector. The first matrix vector multiplication unit comprises a word line driving circuit and a bit line driving circuit, a computing array based on a photoelectric storage and computing integrated device, a source line driving circuit, a current-voltage conversion circuit, an analog-digital conversion circuit and a shift accumulation circuit connected in sequence; the output end of the word line driving circuit is connected with the computing array based on the photoelectric storage and computing integrated device; the computing array based on the photoelectric storage and computing integrated device adopts two arrays, the first array is used for storing positive values in weights, the second array is used for storing absolute values of negative values in weights, and the remaining positions in the two arrays are set to zero; then, in the process of performing matrix vector multiplication operation, the operation result of the first matrix is subtracted from the operation result of the second array, so that the final operation result containing negative weights can be obtained.

2. The transformer neural network acceleration device based on optoelectronic storage and computing integrated device according to claim 1, wherein, The first and second feedforward full connection layer submodules each comprise a data register unit, a second matrix vector multiplication unit and a linear rectifier unit connected in sequence; the second matrix vector multiplication unit is used for performing full connection operation in the feedforward full connection layer submodule. 3.The transformer neural network acceleration device based on optoelectronic storage and computing integrated device of claim 1, wherein, The classification prediction module comprises a data register unit, a third matrix vector multiplication unit and a second softmax function submodule connected in sequence; the second softmax function submodule outputs classification probability.

Images