Memory computing integrated chip and operation method thereof

Abstract

This application discloses an in-memory computing chip, comprising multiple blocks connected by an on-chip network. Each block includes an in-memory computing module and a vector processing module. The in-memory computing chip is used to obtain the number of attention heads according to the configuration parameters of the network model, and to store the attention weights of the neural network in the in-memory computing modules of multiple blocks according to the number of attention heads. Each in-memory computing module stores all types of attention weights. The input data is convolved with the attention weights to obtain multiple feature vectors, each feature vector including multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block for multi-head attention operation to obtain multiple attention head matrices. This application also provides an operation method for the in-memory computing chip, which can realize multi-head parallel operation of the attention mechanism, improve computational efficiency, and reduce the latency of the inference process.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and more specifically, to a memory computing chip and its operating method. Background Technology

[0002] The rapid development of artificial intelligence in recent years is largely attributed to the advancements in deep learning technology. Deep learning has enabled significant progress in many AI (Artificial Intelligence) fields, prompting the industry to rethink AI chip architecture and computing models. Currently, CPUs (Central Processing Units), GPUs (Graphics Processing Units), and ASICs (Application Specific Integrated Circuits) have all undergone architectural optimization for AI applications, achieving better real-time system response speeds and lower power consumption. However, these chip architectures inevitably involve data exchange between storage and computing units. Due to the "memory wall" of the von Neumann architecture, problems arise such as large data transfer volumes, slow data transfer speeds, and high energy consumption. To overcome this bottleneck, a new computing architecture, Computing In Memory (CIM), has been proposed. CIM embeds computing power within memory to perform multiplication and addition operations on two-dimensional and three-dimensional matrices. Convolution operations are a core algorithm in the training and inference of deep learning models; therefore, CIM is well-suited for deep learning, capable of handling large computational workloads and improving computational efficiency.

[0003] When training or inferring using Transformer-based models such as BERT, the input data In is convolved with attention weights to obtain multiple feature vectors, and these feature vectors are then convolved to obtain the attention head matrix. However, because the input data In is constantly changing, the feature vectors are dynamically changing. Subsequent convolution operations on these feature vectors use dynamic data, making it impossible to further accelerate the convolution calculation on the CIM unit. Furthermore, directly using a Vector Processing Unit (VPU) for convolution calculations would be computationally inefficient and time-consuming. Summary of the Invention

[0004] In view of the above problems, the purpose of this invention is to provide an in-memory computing chip that can realize multi-head parallel operation of attention mechanism, improve computing efficiency, and reduce the latency of inference process.

[0005] According to a first aspect of the present invention, an in-memory computing chip is provided, the in-memory computing chip comprising a plurality of blocks connected by an on-chip network, each block comprising an in-memory computing module and a vector processing module, wherein the in-memory computing chip is configured to obtain the number of attention heads according to the configuration parameters of the network model, and store the attention weights of the neural network in the in-memory computing modules of the plurality of blocks according to the number of attention heads, wherein the in-memory computing modules of the plurality of blocks store all types of attention weights; perform convolution operation on input data and attention weights to obtain a plurality of feature vectors, each feature vector comprising a plurality of feature vector matrices; and send one of the plurality of feature vector matrices corresponding to each feature vector to the vector processing module of each block for multi-head attention operation to obtain a plurality of attention head matrices; wherein the number of blocks is the same as the number of attention heads.

[0006] Preferably, the attention weights include a first weight, a second weight, and a third weight; the in-memory computing chip stores the first weight, the second weight, and the third weight in the in-memory computing module of each block.

[0007] Preferably, the storage and computation module of each block performs convolution operations on the input data and attention weights to obtain multiple feature vectors, and splits each of the multiple feature vectors into multiple feature vector matrices according to the configuration parameters of the network model; each block sends one of the multiple feature vector matrices corresponding to each feature vector to the vector processing module of each block.

[0008] Preferably, the plurality of feature vectors includes a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices.

[0009] Preferably, the attention weights include a first weight, a second weight, and a third weight, wherein the first weights include a plurality of first weight matrices, the second weights include a plurality of second weight matrices, and the third weights include a plurality of third weight matrices.

[0010] Preferably, the in-memory computing chip stores one of the multiple first weight matrices, one of the multiple second weight matrices, and one of the multiple third weight matrices in the in-memory computing module of each block.

[0011] Preferably, the storage module of each block performs convolution calculation on one of the multiple first weight matrices, one of the multiple second weight matrices, and one of the multiple third weight matrices to obtain a first eigenvector matrix, a second eigenvector matrix, and a third eigenvector matrix.

[0012] Preferably, the vector processing module of each block performs similarity calculation on the corresponding first feature vector matrix and the corresponding second feature vector matrix to obtain the corresponding association matrix, and performs convolution operation on the association matrix and the corresponding third feature vector matrix to obtain the corresponding attention head matrix.

[0013] Preferably, each block further includes: a memory module for storing data; and a control module connected to the storage and computing module and the memory module, for reading input data from the memory module and providing the input data to the storage and computing module, and for receiving multiple feature vectors from the storage and computing module and splitting each feature vector into multiple corresponding feature vector matrices and providing them to the memory module.

[0014] Preferably, the storage-computing module includes a storage-computing array and an addition tree unit; wherein, the storage-computing array is used to store attention weights and perform matrix multiplication operations on the input data and attention weights; the addition tree unit is used to perform addition calculations on the data output by the storage-computing array to obtain the corresponding feature vector.

[0015] Preferably, the in-memory computing chip includes multiple blocks connected by an on-chip network. Each block includes an in-memory computing module and a vector processing module. The operation method includes: obtaining the number of attention heads according to the configuration parameters of the network model, and storing the attention weights of the neural network in the in-memory computing modules of multiple blocks according to the number of attention heads. The in-memory computing modules of multiple blocks store all types of attention weights. Multiple feature vectors are obtained by convolving the input data with the attention weights. Each feature vector includes multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block for multi-head attention operation to obtain multiple attention head matrices. The number of blocks is the same as the number of attention heads.

[0016] Preferably, the attention weights include a first weight, a second weight, and a third weight.

[0017] Preferably, the operation method includes: storing the first weight, the second weight, and the third weight in the storage module of each block; the storage module of each block performs a convolution operation between the input data and the attention weights to obtain multiple feature vectors, and splits each of the multiple feature vectors into multiple feature vector matrices; each block sends one of the multiple feature vector matrices corresponding to each feature vector to the vector processing module of each block; the vector processing module of each block performs multi-head attention operation based on the feature vector matrix corresponding to each feature vector to obtain the corresponding attention head matrix.

[0018] Preferably, the plurality of feature vectors includes a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices.

[0019] Preferably, the first weight includes a plurality of first weight matrices, the second weight includes a plurality of second weight matrices, and the third weight includes a plurality of third weight matrices.

[0020] Preferably, the operation method includes: splitting the first weight, second weight, and third weight into multiple first weight matrices, multiple second weight matrices, and multiple third weight matrices, respectively; storing the multiple first weight matrices, the multiple second weight matrices, and the multiple third weight matrices in multiple block-level storage modules, respectively; each block-level storage module performing convolution operations with the input data and the corresponding first weight matrix, second weight matrix, and third weight matrix to obtain the corresponding first feature vector matrix, second feature vector matrix, and third feature vector matrix; and each block sending the corresponding first feature vector matrix, second feature vector matrix, and third feature vector matrix to its own vector processing module for multi-head attention operations to obtain the corresponding attention head matrix.

[0021] The in-memory computing chip and its operation method provided by this invention obtain the number of attention heads according to the configuration parameters of the network model, and store the attention weights of the neural network in multiple in-memory computing modules according to the number of attention heads. All types of attention weights are stored in the in-memory computing modules of multiple blocks. The input data is convolved with the attention weights to obtain multiple feature vectors, each feature vector including multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block for multi-head attention operation to obtain multiple attention head matrices. This can realize multi-head parallel operation of the attention mechanism. Data transmission is carried out within the block, reducing data transmission and communication overhead, improving computing efficiency, and reducing the latency of the inference process.

[0022] Furthermore, since no weight splitting is performed during the training phase, and all attention weights are stored in the memory module of each block during the inference phase, the DMA transfer time can be reduced, thereby further improving computational efficiency.

[0023] Furthermore, since the weights have already been split according to the network model configuration parameters during the training phase, the multiple weight matrices of each weight split are stored in the storage and computation module of each block during the inference phase. The input data is convolved with different weight matrices to obtain different feature vector matrices. The vector processing module of each block performs attention operations on different feature vector matrices to obtain the corresponding attention head matrix. No additional DMA transfer operations are required, which further improves the computational efficiency. Attached Figure Description

[0024] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0025] Figure 1 This diagram illustrates the structure of a conventional in-memory computing chip.

[0026] Figure 2 A schematic diagram of the structure of an in-memory computing chip provided according to an embodiment of the present invention is shown.

[0027] Figure 3 This diagram illustrates the data flow of the in-memory computing chip provided in the first embodiment of the present invention.

[0028] Figure 4 This diagram illustrates the data flow of the in-memory computing chip provided in the second embodiment of the present invention.

[0029] Figure 5 A flowchart illustrating the operation method of the in-memory computing chip provided according to the first embodiment of the present invention is shown.

[0030] Figure 6 A flowchart illustrating the operation method of the in-memory computing chip provided according to the second embodiment of the present invention is shown. Detailed Implementation

[0031] Various embodiments of the invention will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by the same or similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale.

[0032] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. Figure 1 This diagram illustrates the structure of a conventional in-memory computing chip. Figure 1 As shown, the in-memory computing chip includes multiple tiles 110 connected via an on-chip network. Each tile 110 includes an in-memory computing module 111, a memory module 112, and a control module 113, etc.

[0033] After the control module 113 issues a calculation start signal, the control module 113 reads the input data from the memory module 112 and sends it to the in-memory computing module 111 for calculation. The in-memory computing module 111 includes an in-memory array, such as a memristor (Resistive Random Access Memory, RRAM) array.

[0034] When using a Transformer-based model such as BERT for inference, the input data In and the attention weights (e.g., the query matrix W) are... Q Key weight W K and value weight W V The process involves convolution to obtain multiple feature vectors (e.g., query vector Q, key vector K, and value vector V), followed by multi-head attention operations to obtain the attention head matrix Z. Since the attention weights are static data, in-memory processing can be accelerated. However, the constantly changing input data In results in dynamic feature vectors, making in-memory convolution impossible to accelerate on a dedicated computing module. Directly using a Vector Processing Unit (VPU) for convolution would be computationally inefficient and time-consuming.

[0035] Figure 2 This diagram illustrates the structure of the in-memory computing chip provided in an embodiment of the present invention. Figure 3 This diagram illustrates the data flow of the in-memory computing chip provided in the first embodiment of the present invention. Figure 2 As shown, the in-memory computing chip includes multiple tiles 210, each tile 210 including an in-memory computing module 211, a memory module 212, a control module 213, and a vector processing module 214.

[0036] The in-memory computing chip is used to obtain the number of attention heads according to the configuration parameters of the network model, and to store the attention weights of the neural network in multiple in-memory computing modules 211 according to the number of attention heads. All types of attention weights are stored in the multiple in-memory computing modules 211. The input data is convolved with the attention weights to obtain multiple feature vectors, each feature vector including multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module 214 of each block to perform multi-head attention operation to obtain multiple attention head matrices Z.

[0037] In this embodiment, the configuration parameters of the network model are, for example, the embedding layer dimension. When the embedding layer dimension is 512 and the row vector dimension of the matrix is ​​64, the number of attention heads is 8. When the embedding layer dimension is 768 and the row vector dimension of the matrix is ​​64, the number of attention heads is 12.

[0038] The storage module 211 is used to store attention weights and perform convolution operations on the input data and attention weights to obtain multiple feature vectors.

[0039] In this embodiment, the attention weights include a first weight, a second weight, and a third weight; the in-memory computing chip stores the first weight, the second weight, and the third weight in the in-memory computing modules of the first Tile1 to the Nth TileN, where N is a positive integer and is the same as the number of attention heads.

[0040] Specifically, the first Tile1 stores the first weight, the second weight, and the third weight; the second Tile2 stores the first weight, the second weight, and the third weight; the third Tile3 stores the first weight, the second weight, and the third weight; ..., the (N-1)th TileN-1 stores the first weight, the second weight, and the third weight; and the Nth TileN stores the first weight, the second weight, and the third weight.

[0041] In this embodiment, the in-memory computing module 211 includes an in-memory array 2111 and an addition tree unit 2112; wherein, the in-memory array 2111 is used to store attention weights and perform matrix multiplication operations on the input data and attention weights; the addition tree unit 2112 is used to perform addition calculations on the data output by the in-memory array to obtain the corresponding feature vector. The in-memory array 2111 is, for example, a memristor array, but is not limited to this.

[0042] Specifically, the storage array multiplies each element of the input data In in the i-th row with each element of the attention weight in the j-th column to obtain a row of data; the addition tree unit adds the data in the row one by one to obtain the value corresponding to the i-th row and j-th column of the feature vector.

[0043] Memory module 212 is used to store data. This data may be, for example, input data In, multiple feature vectors, multiple feature vector matrices, attention head matrices, etc., but is not limited to these.

[0044] In this embodiment, the plurality of feature vectors includes a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices. Each block sends one of the first feature vector matrices from the first feature vector, one of the second feature vector matrices from the second feature vector, and one of the third feature vector matrices from the third feature vector to its own vector processing module 214.

[0045] The control module 213 is connected to the storage and computing module 211 and the memory module 212, and is used to read input data In from the memory module 212 and provide the input data In to the storage and computing module 211, and to receive multiple feature vectors from the storage and computing module 211 and split each feature vector into multiple corresponding feature vector matrices and provide them to the memory module 212.

[0046] In this embodiment, the control module 213 sends a feature vector matrix corresponding to the different feature vectors after splitting to its own vector processing module 214.

[0047] The vector processing module 214 is used to perform convolution operations on the corresponding first feature vector matrix and the second feature vector matrix to obtain the corresponding correlation matrix S, and to perform multi-head attention operations on the correlation matrix S and the corresponding third feature vector matrix to obtain multiple attention head matrices Z.

[0048] Specifically, each tile's storage module stores a first weight (e.g., the query weight W). Q ), second weight (key weight W) K ) and the third weight (value weight W) V Accordingly, each tile generates a first feature vector (e.g., a query vector Q), a second feature vector (e.g., a key vector K), and a third feature vector (e.g., a value vector V). Each tile splits the first feature vector into multiple first feature vector matrices (e.g., multiple query vector matrices [q1,q2,q3,……,qN]), the second feature vector into multiple second feature vector matrices (e.g., multiple key vector matrices [k1,k2,k3,……,kN]), and the third feature vector into multiple third feature vector matrices (e.g., multiple value vector matrices [v1,v2,v3,……,vN]). That is, Q = In * WQ = [q1,q2,q3,……,qN]; K = In * WK = [k1,k2,k3,……,kN]; V = In * WV = [v1,v2,v3,……,vN].

[0049] Each tile stores the query weight W.Q Key weight W K Sum weight W V The input data In is stored with the query weight W for each tile. Q The query vector Q is obtained by performing a convolution operation, and the input data In is combined with the key weight W stored in each tile. K The convolution operation yields the key vector K, and the input data In is combined with the weights W stored in each tile. V Convolution is performed to obtain the value vector V. Each tile splits the query vector Q into N query vector matrices [q1,q2,q3,……,qN-1,qN]; splits the key vector K into N key vector matrices [k1,k2,k3,……,kN-1,kN]; and splits the value vector V into N value vector matrices [v1,v2,v3,……,vN-1,vN].

[0050] The first Tile 1 sends the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1 to the vector processing module of the first Tile 1; the second Tile 2 sends the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2 to the vector processing module of the second Tile 2; the third Tile 3 sends the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3 to the vector processing module of the third Tile 3; ..., the Nth Tile N sends the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN to the vector processing module of the Nth Tile N.

[0051] The vector processing module of the first tile 1 performs a convolution operation on the first query vector matrix q1 and the first key vector matrix k1 to obtain the first association matrix S1. The first association matrix S1 is then convolved with the first value vector matrix v1 to obtain the first attention head matrix Z1. The vector processing module of the second tile 2 performs a convolution operation on the second query vector matrix q2 and the second key vector matrix k2 to obtain the second association matrix S2. The second association matrix S2 is then convolved with the second value vector matrix v2 to obtain the second attention head matrix Z2. The vector processing module of the third tile 3 performs a convolution operation on the third query vector matrix q3 and the third key vector matrix k3 to obtain the third association matrix S3. 3. Convolve the third value vector matrix v3 to obtain the third attention head matrix Z3; ..., the vector processing module of the (N-1)th Tile N-1 convolves the (N-1)th query vector matrix qN-1 and the (N-1)th key vector matrix kN-1 to obtain the (N-1)th association matrix SN-1, and convolves the (N-1)th association matrix SN-1 with the (N-1)th value vector matrix vN-1 to obtain the (N-1)th attention head matrix ZN-1; the vector processing module of the Nth Tile N convolves the Nth query vector matrix qN and the Nth key vector matrix kN to obtain the Nth association matrix SN, and convolves the Nth association matrix SN with the Nth value vector matrix vN to obtain the Nth attention head matrix ZN. Association matrix Used to characterize the similarity between the query vector and the key vector; Attention head matrix Z = S*V

[0052] Here, Softmax() represents the normalization operation of the attention head matrix on the row vectors, K T K is the transpose of the key vector, and d is the row vector dimension of the matrix.

[0053] The in-memory computing chip provided by this invention obtains the number of attention heads according to the configuration parameters of the network model, and stores the attention weights of the neural network in multiple in-memory computing modules according to the number of attention heads. All types of attention weights are stored in the in-memory computing modules of multiple blocks. The input data is convolved with the attention weights to obtain multiple feature vectors, each feature vector including multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block for multi-head attention operation to obtain multiple attention head matrices. This can realize multi-head parallel operation of the attention mechanism. Data transmission is carried out within the block, reducing data transmission and communication overhead, improving computing efficiency, and reducing the latency of the inference process.

[0054] Furthermore, since no weight splitting is performed during the training phase, and all attention weights are stored in the memory module of each block during the inference phase, the DMA transfer time can be reduced, thereby further improving computational efficiency.

[0055] Figure 4 This diagram illustrates the data flow of an in-memory computing chip according to a second embodiment of the present invention. Compared to the first embodiment, the in-memory computing chip in this embodiment splits the attention weights into multiple weight matrices according to the network model configuration parameters, and stores the multiple weight matrices in the in-memory computing module of each block.

[0056] In this embodiment, the attention weights include a first weight, a second weight, and a third weight; the first weights include multiple first weight matrices, the second weights include multiple second weight matrices, and the third weights include multiple third weight matrices.

[0057] The in-memory computing chip stores one of the multiple first weight matrices, one of the multiple first weight matrices, and one of the multiple third weight matrices in the in-memory computing modules of the first Tile1 to the Nth TileN, where N is a positive integer and the same as the number of attention heads. That is, the first Tile1 stores the first weight matrix, the second weight matrix, and the third weight matrix; the second Tile2 stores the first weight matrix, the second weight matrix, and the third weight matrix; the third Tile3 stores the first weight matrix, the second weight matrix, and the third weight matrix; ..., the (N-1)th TileN-1 stores the first weight matrix, the second weight matrix, and the third weight matrix; and the Nth TileN stores the first weight matrix, the second weight matrix, and the third weight matrix.

[0058] Specifically, the first weight (e.g., the query weight W) Q ) is split into multiple first weight matrices (e.g., multiple query weight matrices [W]). Q1 W Q2 W Q3 ,……,W QN ]), and the second weight (e.g., the query weight W) K ) is split into multiple second weight matrices (e.g., multiple query weight matrices [W]). K1 W K2 W K3 ,……,W KN The third weight (e.g., query weight W) will be used. V ) is split into multiple third weight matrices (e.g., multiple query weight matrices [W]). V1 W V2 W V3 ,……,W VN ]).

[0059] The in-memory computing chip will query the first weight matrix W Q1 First bond weight matrix W K1 First-value weight matrix W V1 The data is stored in the memory module of the first Tile1. The memory module of the first Tile1 will compare the input data In with the first query weight matrix W. Q1 First bond weight matrix W K1 First-value weight matrix W V1 Convolution operations are performed to obtain the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1. The in-memory computing chip then applies the second query weight matrix W. Q2 The second bond weight matrix W K2 Second-value weight matrix W V2 The data is stored in the memory module of the second Tile2. The memory module of the second Tile2 will compare the input data In with the second query weight matrix W. Q2 The second bond weight matrix W K2 Second-value weight matrix W V2 Convolution operations are performed to obtain the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2. The in-memory computing chip then applies the third query weight matrix W. Q3 Third bond weight matrix W K3 Third-value weight matrix W V3 The data is stored in the memory module of the third Tile 3. The memory module of the third Tile 3 will compare the input data In with the third query weight matrix W. Q3 Third bond weight matrix W K3 Third-value weight matrix W V3 Convolution operations are performed to obtain the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3. ... The in-memory computing chip then applies the Nth query weight matrix W... QN The Nth bond weight matrix W KN The Nth value weight matrix W VN The stored data is processed in the memory module of the Nth Tile N. The memory module of the Nth Tile N will compare the input data In with the Nth query weight matrix W. QN The Nth bond weight matrix W KN The Nth value weight matrix W VN Perform convolution operations to obtain the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN.

[0060] The first Tile 1 sends the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1 to the vector processing module of the first Tile 1; the second Tile 2 sends the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2 to the vector processing module of the second Tile 2; the third Tile 3 sends the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3 to the vector processing module of the third Tile 3; ..., the Nth Tile N sends the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN to the vector processing module of the Nth Tile N.

[0061] The remainder of this embodiment is the same as the remainder of the first embodiment, and will not be repeated here.

[0062] The in-memory computing chip provided by this invention has already performed weight splitting according to the network model configuration parameters during the training phase. During the inference phase, the multiple weight matrices of each weight split are stored in the in-memory computing module of each block. The input data is convolved with different weight matrices to obtain different feature vector matrices. The vector processing module of each block performs attention operations on different feature vector matrices to obtain the corresponding attention head matrix. No additional DMA transfer operations are required, which further improves the computational efficiency.

[0063] Figure 5 A flowchart illustrating the operation method of the in-memory computing chip provided according to a first embodiment of the present invention is shown. Figure 5 As shown, the operation method includes the following steps.

[0064] In step S110, the number of attention heads is obtained according to the configuration parameters of the network model, and the attention weights of the neural network are stored in the storage modules of multiple blocks according to the number of attention heads. All types of attention weights are stored in the storage modules of multiple blocks.

[0065] In this embodiment, the configuration parameters of the network model are, for example, the embedding layer dimension. When the embedding layer dimension is 512 and the row vector dimension of the matrix is ​​64, the number of attention heads is 8. When the embedding layer dimension is 768 and the row vector dimension of the matrix is ​​64, the number of attention heads is 12.

[0066] The attention weights include a first weight, a second weight, and a third weight; the in-memory computing chip stores the first weight, the second weight, and the third weight in the in-memory computing modules of the first Tile1 to the Nth TileN, where N is a positive integer and is the same as the number of attention heads.

[0067] Specifically, the first Tile1 stores the first weight, the second weight, and the third weight; the second Tile2 stores the first weight, the second weight, and the third weight; the third Tile3 stores the first weight, the second weight, and the third weight; ..., the (N-1)th TileN-1 stores the first weight, the second weight, and the third weight; and the Nth TileN stores the first weight, the second weight, and the third weight.

[0068] In step S120, the multiple feature vectors obtained by convolving the input data with the attention weights are all split into multiple corresponding feature vector matrices according to the number of attention heads.

[0069] In this embodiment, the plurality of feature vectors includes a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices. Each block sends one of the first feature vector matrices from the first feature vector, one of the second feature vector matrices from the second feature vector, and one of the third feature vector matrices from the third feature vector to its own vector processing module.

[0070] In step S130, one of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block to perform multi-head attention operation to obtain multiple attention head matrices.

[0071] In this embodiment, the vector processing module of each block performs convolution operation on the corresponding first feature vector matrix and the second feature vector matrix to obtain the corresponding correlation matrix S, and performs multi-head attention operation on the correlation matrix S and the corresponding third feature vector matrix to obtain the attention head matrix Z.

[0072] Specifically, each tile's storage module stores a first weight (e.g., the query weight W). Q ), second weight (key weight W) K ) and the third weight (value weight W) VAccordingly, each tile generates a first feature vector (e.g., a query vector Q), a second feature vector (e.g., a key vector K), and a third feature vector (e.g., a value vector V). Each tile splits the first feature vector into multiple first feature vector matrices (e.g., multiple query vector matrices [q1,q2,q3,……,qN]), the second feature vector into multiple second feature vector matrices (e.g., multiple key vector matrices [k1,k2,k3,……,kN]), and the third feature vector into multiple third feature vector matrices (e.g., multiple value vector matrices [v1,v2,v3,……,vN]). That is, Q = In * WQ = [q1,q2,q3,……,qN]; K = In * WK = [k1,k2,k3,……,kN]; V = In * WV = [v1,v2,v3,……,vN].

[0073] Each tile stores the query weight W. Q Key weight W K Sum weight W V The input data In is stored with the query weight W for each tile. Q The query vector Q is obtained by performing a convolution operation, and the input data In is combined with the key weight W stored in each tile. K The convolution operation yields the key vector K, and the input data In is combined with the weights W stored in each tile. V Convolution is performed to obtain the value vector V. Each tile splits the query vector Q into N query vector matrices [q1,q2,q3,……,qN-1,qN]; splits the key vector K into N key vector matrices [k1,k2,k3,……,kN-1,kN]; and splits the value vector V into N value vector matrices [v1,v2,v3,……,vN-1,vN].

[0074] The first Tile 1 sends the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1 to the vector processing module of the first Tile 1; the second Tile 2 sends the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2 to the vector processing module of the second Tile 2; the third Tile 3 sends the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3 to the vector processing module of the third Tile 3; ..., the Nth Tile N sends the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN to the vector processing module of the Nth Tile N.

[0075] The vector processing module of the first tile 1 performs a convolution operation on the first query vector matrix q1 and the first key vector matrix k1 to obtain the first association matrix S1. The first association matrix S1 is then convolved with the first value vector matrix v1 to obtain the first attention head matrix Z1. The vector processing module of the second tile 2 performs a convolution operation on the second query vector matrix q2 and the second key vector matrix k2 to obtain the second association matrix S2. The second association matrix S2 is then convolved with the second value vector matrix v2 to obtain the second attention head matrix Z2. The vector processing module of the third tile 3 performs a convolution operation on the third query vector matrix q3 and the third key vector matrix k3 to obtain the third association matrix S3. 3. Convolve the third value vector matrix v3 to obtain the third attention head matrix Z3; ..., the vector processing module of the (N-1)th Tile N-1 convolves the (N-1)th query vector matrix qN-1 and the (N-1)th key vector matrix kN-1 to obtain the (N-1)th association matrix SN-1, and convolves the (N-1)th association matrix SN-1 with the (N-1)th value vector matrix vN-1 to obtain the (N-1)th attention head matrix ZN-1; the vector processing module of the Nth Tile N convolves the Nth query vector matrix qN and the Nth key vector matrix kN to obtain the Nth association matrix SN, and convolves the Nth association matrix SN with the Nth value vector matrix vN to obtain the Nth attention head matrix ZN. Association matrix Used to characterize the similarity between the query vector and the key vector; Attention head matrix Z = S*V

[0076] Here, Softmax() represents the normalization operation of the attention head matrix on the row vectors, K T K is the transpose of the key vector, and d is the row vector dimension of the matrix.

[0077] The in-memory computing chip operation method provided by this invention obtains the number of attention heads according to the configuration parameters of the network model, and stores the attention weights of the neural network in multiple in-memory computing modules according to the number of attention heads. All types of attention weights are stored in the in-memory computing modules of multiple blocks. The input data is convolved with the attention weights to obtain multiple feature vectors, each feature vector including multiple feature vector matrices. One of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block for multi-head attention operation to obtain multiple attention head matrices. This can realize multi-head parallel operation of the attention mechanism. Data transmission is carried out within the block, reducing data transmission and communication overhead, improving computing efficiency, and reducing the latency of the inference process.

[0078] Furthermore, since no weight splitting is performed during the training phase, and all attention weights are stored in the memory module of each block during the inference phase, the DMA transfer time can be reduced, thereby further improving computational efficiency.

[0079] Figure 6 A flowchart illustrating the operation method of the in-memory computing chip provided according to a first embodiment of the present invention is shown. Figure 6 As shown, the operation method includes the following steps.

[0080] In step S210, the number of attention heads is obtained according to the configuration parameters of the network model, and the attention weights of the neural network are split into multiple first weight matrices, multiple second weight matrices, and multiple third weight matrices according to the number of attention heads.

[0081] In this embodiment, the configuration parameters of the network model are, for example, the embedding layer dimension. When the embedding layer dimension is 512 and the row vector dimension of the matrix is ​​64, the number of attention heads is 8. When the embedding layer dimension is 768 and the row vector dimension of the matrix is ​​64, the number of attention heads is 12.

[0082] The attention weights include a first weight, a second weight, and a third weight. The first weight includes multiple first weight matrices, the second weight includes multiple second weight matrices, and the third weight includes multiple third weight matrices.

[0083] The in-memory computing chip stores one of the multiple first weight matrices, one of the multiple first weight matrices, and one of the multiple third weight matrices in the in-memory computing modules of the first Tile1 to the Nth TileN, where N is a positive integer and is the same as the number of attention heads.

[0084] Specifically, the first weight (e.g., the query weight W) Q ) is split into multiple first weight matrices (e.g., multiple query weight matrices [W]). Q1 W Q2 W Q3 ,……,W QN ]), and the second weight (e.g., the query weight W) K ) is split into multiple second weight matrices (e.g., multiple query weight matrices [W]). K1 W K2 W K3 ,……,W KN The third weight (e.g., query weight W) will be used. V ) is split into multiple third weight matrices (e.g., multiple query weight matrices [W]). V1 W V2 W V3 ,……,WVN ]).

[0085] In step S220, multiple first weight matrices are stored in multiple block storage modules, multiple second weight matrices are stored in multiple block storage modules, and multiple third weight matrices are stored in multiple block storage modules.

[0086] In this embodiment, the in-memory computing chip stores one of the multiple first weight matrices, one of the multiple first weight matrices, and one of the multiple third weight matrices in the in-memory computing modules of the first Tile1 to the Nth TileN, where N is a positive integer and the same as the number of attention heads. That is, the first Tile1 stores the first weight matrix, the second weight matrix, and the third weight matrix; the second Tile2 stores the first weight matrix, the second weight matrix, and the third weight matrix; the third Tile3 stores the first weight matrix, the second weight matrix, and the third weight matrix; ..., the (N-1)th TileN-1 stores the first weight matrix, the second weight matrix, and the third weight matrix; and the Nth TileN stores the first weight matrix, the second weight matrix, and the third weight matrix.

[0087] Specifically, the in-memory computing chip will use the first query weight matrix W Q1 First bond weight matrix W K1 First-value weight matrix W V1 The data is stored in the memory module of the first Tile1. The memory module of the first Tile1 will compare the input data In with the first query weight matrix W. Q1 First bond weight matrix W K1 First-value weight matrix W V1 Convolution operations are performed to obtain the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1. The in-memory computing chip then applies the second query weight matrix W. Q2 The second bond weight matrix W K2 Second-value weight matrix W V2 The data is stored in the memory module of the second Tile2. The memory module of the second Tile2 will compare the input data In with the second query weight matrix W. Q2 The second bond weight matrix W K2 Second-value weight matrix W V2 Convolution operations are performed to obtain the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2. The in-memory computing chip then applies the third query weight matrix W. Q3 Third bond weight matrix W K3 Third-value weight matrix W V3The data is stored in the memory module of the third Tile 3. The memory module of the third Tile 3 will compare the input data In with the third query weight matrix W. Q3 Third bond weight matrix W K3 Third-value weight matrix W V3 Convolution operations are performed to obtain the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3. ... The in-memory computing chip then applies the Nth query weight matrix W... QN The Nth bond weight matrix W KN The Nth value weight matrix W VN The stored data is processed in the memory module of the Nth Tile N. The memory module of the Nth Tile N will compare the input data In with the Nth query weight matrix W. QN The Nth bond weight matrix W KN The Nth value weight matrix W VN Perform convolution operations to obtain the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN.

[0088] In step S230, the storage module of each block performs convolution operations on the input data with the corresponding first weight matrix, second weight matrix, and third weight matrix to obtain the corresponding first eigenvector matrix, second eigenvector matrix, and third eigenvector matrix.

[0089] In this embodiment, the first Tile1 sends the first query vector matrix q1, the first key vector matrix k1, and the first value vector matrix v1 to the vector processing module of the first Tile1; the second Tile2 sends the second query vector matrix q2, the second key vector matrix k2, and the second value vector matrix v2 to the vector processing module of the second Tile2; the third Tile3 sends the third query vector matrix q3, the third key vector matrix k3, and the third value vector matrix v3 to the vector processing module of the third Tile3; ..., the Nth TileN sends the Nth query vector matrix qN, the Nth key vector matrix kN, and the Nth value vector matrix vN to the vector processing module of the Nth TileN.

[0090] In step S240, one of the multiple feature vector matrices corresponding to each feature vector is sent to the vector processing module of each block to perform multi-head attention operation to obtain multiple attention head matrices.

[0091] In this embodiment, the vector processing module of the first Tile 1 performs a convolution operation on the first query vector matrix q1 and the first key vector matrix k1 to obtain the first association matrix S1. The first association matrix S1 is then convolved with the first value vector matrix v1 to obtain the first attention head matrix Z1. The vector processing module of the second Tile 2 performs a convolution operation on the second query vector matrix q2 and the second key vector matrix k2 to obtain the second association matrix S2. The second association matrix S2 is then convolved with the second value vector matrix v2 to obtain the second attention head matrix Z2. The vector processing module of the third Tile 3 performs a convolution operation on the third query vector matrix q3 and the third key vector matrix k3 to obtain the third association matrix S3. Matrix S3 is convolved with the third value vector matrix v3 to obtain the third attention head matrix Z3; ..., the vector processing module of the (N-1)th Tile N-1 convolves the (N-1)th query vector matrix qN-1 and the (N-1)th key vector matrix kN-1 to obtain the (N-1)th association matrix SN-1, and convolves the (N-1)th association matrix SN-1 with the (N-1)th value vector matrix vN-1 to obtain the (N-1)th attention head matrix ZN-1; the vector processing module of the Nth Tile N convolves the Nth query vector matrix qN and the Nth key vector matrix kN to obtain the Nth association matrix SN, and convolves the Nth association matrix SN with the Nth value vector matrix vN to obtain the Nth attention head matrix ZN. Association matrix Attention head matrix Z = S*V,

[0092] Here, Softmax() represents the normalization operation of the attention head matrix on the row vectors, K T K is the transpose of the key vector, and d is the row vector dimension of the matrix.

[0093] The in-memory computing chip provided by this invention has already performed weight splitting according to the network model configuration parameters during the training phase. During the inference phase, the multiple weight matrices of each weight split are stored in the in-memory computing module of each block. The input data is convolved with different weight matrices to obtain different feature vector matrices. The vector processing module of each block performs attention operations on different feature vector matrices to obtain the corresponding attention head matrix. No additional DMA transfer operations are required, which further improves the computational efficiency.

[0094] As described above, these embodiments of the present invention do not exhaustively cover all details, nor do they limit the invention to the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to effectively utilize the invention and its modifications. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A memory computing chip, characterized in that, The in-memory computing chip includes multiple blocks connected by an on-chip network. Each block includes an in-memory computing module and a vector processing module. The number of blocks is the same as the number of attention heads in the network model. The in-memory computing chip is used to obtain the number of attention heads according to the configuration parameters of the network model, and to store all types of attention weights of the neural network in the in-memory computing module of each block according to the number of attention heads, so that multiple blocks can perform parallel computing on multi-head attention. The storage module of each block is used to perform convolution operations on the input data with all types of attention weights stored locally to obtain multiple feature vectors, and to split each feature vector into multiple feature vector matrices according to the number of attention heads; Each block is used to send one of the multiple feature vector matrices corresponding to each feature vector to its own vector processing module; The vector processing module of each block is used to perform multi-head attention operations on the received multiple feature vector matrices to obtain an attention head matrix corresponding to each block.

2. The in-memory computing chip according to claim 1, characterized in that, The attention weights include a first weight, a second weight, and a third weight; the in-memory computing chip stores the first weight, the second weight, and the third weight in the in-memory computing module of each block.

3. The in-memory computing chip according to claim 2, characterized in that, The plurality of feature vectors include a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices.

4. The in-memory computing chip according to claim 1, characterized in that, The attention weights include a first weight, a second weight, and a third weight. The first weight includes multiple first weight matrices, the second weight includes multiple second weight matrices, and the third weight includes multiple third weight matrices.

5. The in-memory computing chip according to claim 4, characterized in that, The in-memory computing chip stores one of the multiple first weight matrices, one of the multiple second weight matrices, and one of the multiple third weight matrices in the in-memory computing module of each block.

6. The in-memory computing chip according to claim 5, characterized in that, Each block's storage module performs convolution calculations on one of the multiple first weight matrices, one of the multiple second weight matrices, and one of the multiple third weight matrices to obtain the first eigenvector matrix, the second eigenvector matrix, and the third eigenvector matrix.

7. The in-memory computing chip according to claim 1, 3, or 6, characterized in that, The vector processing module of each block performs similarity calculation on the corresponding first feature vector matrix and the corresponding second feature vector matrix to obtain the corresponding association matrix, and performs convolution operation on the association matrix and the corresponding third feature vector matrix to obtain the corresponding attention head matrix.

8. The in-memory computing chip according to claim 1, characterized in that, Each block also includes: Memory modules are used to store data; A control module, connected to the in-memory module and the memory module, is used to read input data from the memory module and provide the input data to the in-memory module, and to receive multiple feature vectors from the in-memory module and split each feature vector into multiple corresponding feature vector matrices and provide them to the memory module.

9. The in-memory computing chip according to claim 1, characterized in that, The storage-based computing module includes a storage-based computing array and an addition tree unit; The storage array is used to store attention weights and perform matrix multiplication on the input data and attention weights. The addition tree unit is used to perform addition calculations on the data output by the storage array to obtain the corresponding feature vector.

10. A method for operating a memory computing chip, characterized in that, The in-memory computing chip includes multiple blocks connected by an on-chip network. Each block includes an in-memory computing module and a vector processing module. The operation method includes: The number of attention heads is obtained according to the configuration parameters of the network model, and all types of attention weights of the neural network are stored in the storage module of each block according to the number of attention heads, wherein the number of blocks is the same as the number of attention heads; The storage module of each block performs convolution operations on the input data with all types of attention weights stored locally to obtain multiple feature vectors, and splits each feature vector into multiple feature vector matrices according to the number of attention heads; Each block sends one of the multiple feature vector matrices corresponding to each feature vector to its own vector processing module; The vector processing module of each block performs multi-head attention operations on the received multiple feature vector matrices to obtain an attention head matrix corresponding to each block.

11. The operating method according to claim 10, characterized in that, The attention weights include a first weight, a second weight, and a third weight.

12. The operating method according to claim 11, characterized in that, The plurality of feature vectors include a first feature vector, a second feature vector, and a third feature vector; wherein, the first feature vector includes a plurality of first feature vector matrices; the second feature vector includes a plurality of feature vector matrices; and the third feature vector includes a plurality of third feature vector matrices.

13. The operating method according to claim 11, characterized in that, The first weight includes multiple first weight matrices, the second weight includes multiple second weight matrices, and the third weight includes multiple third weight matrices.

14. The operating method according to claim 11, characterized in that, The operation method includes: The first weight, second weight, and third weight are each split into multiple first weight matrices, multiple second weight matrices, and multiple third weight matrices, respectively. Multiple first weight matrices are stored in multiple storage modules, multiple second weight matrices are stored in multiple storage modules, and multiple third weight matrices are stored in multiple storage modules. The storage and computation module of each block performs convolution operations on the input data with the corresponding first weight matrix, second weight matrix, and third weight matrix to obtain the corresponding first eigenvector matrix, second eigenvector matrix, and third eigenvector matrix; Each block sends its corresponding first feature vector matrix, second feature vector matrix, and third feature vector matrix to its own vector processing module for multi-head attention operation to obtain the corresponding attention head matrix.

Images