Accelerator chips for large model algorithms

Abstract

The present invention provides an accelerator chip for a large model algorithm, and relates to the technical field of artificial intelligence chips. The accelerator chip includes: a matrix-vector multiplication unit and a vector unit; the matrix-vector multiplication unit is used for matrix-vector multiplication operations, data storage and transmission of the large model algorithm; the vector unit is used for vector operations, data storage and control of the large model algorithm; wherein the matrix-vector multiplication unit includes a read-only memory and a static random access memory; the read-only memory is used to store base model parameters, and the base model parameters are used to calculate base model matrix vectors; the static random access memory is used to store fine-tuning parameters, and the fine-tuning parameters are used to calculate the fine-tuning matrix. The present invention can break the storage bandwidth limitation at a relatively low cost.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence chip technology, and in particular to an accelerator chip for large model algorithms. Background Art

[0002] The computational intensity of large model algorithms is mainly attributed to their architecture and the scale of data they process, which usually contain hundreds of millions to tens of billions of parameters. Training these parameters requires multiple passes through large data sets, a process that requires a lot of computing resources. In order to accelerate these computational processes, chips such as graphics processing units (GPUs) are currently used in large model algorithms due to their intensive computing power.

[0003] However, when GPU chips perform computationally intensive tasks, the computing power of GPU chips and the storage bandwidth of on-chip static random-access memory (SRAM) do not match. For example, the computing power of GPU chips can support 500 tokens / s, but the storage bandwidth of on-chip SRAM can only support 10 tokens / s, which cannot fully utilize the computing power of GPU chips. This is the limitation brought by the "storage wall". How to break the storage bandwidth limitation at a lower cost is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides an accelerator chip for a large model algorithm, so as to achieve the purpose of breaking the limitation of storage bandwidth at a relatively low cost.

[0005] The present invention provides an accelerator chip for a large model algorithm, comprising: a matrix-vector multiplication unit and a vector unit; the matrix-vector multiplication unit is used for matrix-vector multiplication operations, data storage and transmission of the large model algorithm; the vector unit is used for vector operations, data storage and control of the large model algorithm; Among them, the matrix-vector multiplication unit includes a read-only memory and a static random access memory; the read-only memory is used to store base model parameters, and the base model parameters are used to calculate the base model matrix vector; the static random access memory is used to store fine-tuning parameters, and the fine-tuning parameters are used to calculate the fine-tuning matrix.

[0006] According to an accelerator chip for a large model algorithm provided by the present invention, a first preset number of matrix-vector multiplication units are arranged in a matrix to form a matrix-vector multiplication unit area; a second preset number of matrix-vector multiplication unit areas are arranged in a matrix.

[0007] According to an accelerator chip for a large model algorithm provided by the present invention, a vector unit is arranged in each row between every two horizontally adjacent matrix-vector multiplication unit areas; the vector unit is used to transmit the data pulses to be calculated to each of the matrix-vector multiplication units in sequence along the horizontal direction.

[0008] According to an accelerator chip for a large model algorithm provided by the present invention, an accumulation unit is further provided in each column between every two vertically adjacent matrix-vector multiplication unit areas; the matrix-vector multiplication unit is used to transfer the data to be calculated or the intermediate calculation results to the matrix-vector multiplication units adjacent in the horizontal and vertical directions; the accumulation unit is used to accumulate the data to be calculated or the intermediate calculation results transferred by each of the matrix-vector multiplication units in the vertical direction, obtain the accumulated calculation results, and transfer the accumulated calculation results to the corresponding vector units along the set data transfer path.

[0009] According to an accelerator chip for a large model algorithm provided by the present invention, the vector unit is also used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along the set data transmission path, the data transmitted by the matrix-vector multiplication unit and the calculation result of the vector unit according to the calculation scenario.

[0010] According to an accelerator chip for large model algorithms provided by the present invention, the vector units between every two horizontally adjacent matrix-vector multiplication unit areas form a vector unit group; an operation unit is also arranged between two vertically adjacent vector unit groups, and the operation unit is used to operate on the data transmitted by the corresponding vector units, and broadcast the operation results to the corresponding vector units for storage.

[0011] According to an accelerator chip for a large model algorithm provided by the present invention, each of the matrix-vector multiplication units comprises: A base model matrix vector multiplication unit, comprising the read-only memory, for performing base model matrix vector calculation according to input data to be calculated, control information in the working signal and the base model parameters stored in the read-only memory; A fine-tuning matrix-vector multiplication unit, comprising the static random access memory, for storing the fine-tuning parameters in the working signal into the static random access memory, and performing fine-tuning matrix-vector calculation according to input data to be calculated and the fine-tuning parameters stored in the static random access memory; A first selector is used to select a data path for output from the output result of the base model matrix-vector multiplication unit, the output result of the fine-tuning matrix-vector multiplication unit, the input data to be calculated and 0; The second selector is used to select one path of data from the operands transmitted in the horizontal direction, the operands transmitted in the vertical direction and 0 for output; an accumulator, used for accumulating the output data of the first selector and the output data of the second selector; A first register, used to optimize the data timing on the output path of the accumulator in the horizontal direction and the vertical direction; a third selector, used to select two paths of data from the intermediate calculation result transmitted in the horizontal direction, the intermediate calculation result transmitted in the vertical direction, and the output result of the accumulator temporarily stored in the first register; A second register, used for optimizing the data timing on the output path in the horizontal direction of the third selector; The third register is used to optimize the data timing on the vertical output path of the third selector.

[0012] According to an accelerator chip for a large model algorithm provided by the present invention, each of the vector units comprises: a storage module, a calculation module, a first transmission module, a second transmission module and a control module; The control modules in the vector units at the upper and lower ends of two vector unit groups adjacent to each other in the vertical direction are used to transmit corresponding control information to each vector unit; The control module is used to transmit corresponding control information to each of the matrix-vector multiplication units, the corresponding accumulation unit and the operation unit in the horizontal direction according to the received control information; The operation module is used to read the corresponding data in the storage module to perform vector operation; The first transfer module is used to transfer the vector operation result of the operation module to the adjacent vector module; The second transfer module is used to transfer the vector operation result of the operation module across the vector module; The storage module is used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along the set data transmission path, the data transmitted by the matrix-vector multiplication unit and the calculation result of the vector unit according to the calculation scenario.

[0013] According to an accelerator chip for a large model algorithm provided by the present invention, the operation module includes at least one basic unit selected from the group consisting of a multiplication basic unit, an addition basic unit, a division basic unit, a delay basic unit, a trigonometric function basic unit, a square root basic unit, a comparison basic unit, an exponential basic unit and an accumulation basic unit.

[0014] According to an accelerator chip for a large model algorithm provided by the present invention, the storage unit of the storage module is a static random access memory, and the port width and depth of the storage module match the computing power of the accelerator chip.

[0015] The accelerator chip for a large model algorithm provided by the present invention mainly includes a matrix-vector multiplication unit and a vector unit. The matrix-vector multiplication unit is used for matrix-vector multiplication operations, data storage and transmission of the large model algorithm. The vector unit is used for vector operations, data storage and control of the large model algorithm. Among them, the matrix-vector multiplication unit includes a read-only memory and a static random access memory; the read-only memory is used to store base model parameters, and the base model parameters are used to calculate the base model matrix vector; that is, the read-only memory is used as the main memory; the static random access memory is used to store fine-tuning parameters, and the fine-tuning parameters are used to calculate the fine-tuning matrix; that is, the static random access memory is used as an additional memory. Since the read-only memory in the matrix-vector multiplication unit is used as the main memory and the static random access memory is used as the additional memory, compared with the traditional static random access memory as the main memory, the storage bandwidth of the read-only memory is larger, so that the computing power of the accelerator chip can match the storage bandwidth of the read-only memory, and the computing power of the accelerator chip can be fully utilized, so that the storage bandwidth limitation can be broken at a relatively low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the structural schematic diagrams of the accelerator chip for large model algorithms provided in an embodiment of the present invention.

[0018] Figure 2 This is the second structural schematic diagram of the accelerator chip for large model algorithms provided in an embodiment of the present invention.

[0019] Figure 3 This is the third structural schematic diagram of the accelerator chip for large model algorithms provided in an embodiment of the present invention.

[0020] Figure 4 This is the fourth structural schematic diagram of the accelerator chip for large model algorithms provided in an embodiment of the present invention.

[0021] Figure 5 It is a structural diagram of a matrix-vector multiplication unit provided in an embodiment of the present invention.

[0022] Figure 6 It is a schematic diagram of control information transmission of a vector unit provided by an embodiment of the present invention.

[0023] Figure 7 It is a schematic diagram of the process of implementing the Norm operator using an application accelerator chip provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] First, the technical background of the present invention is introduced.

[0026] 1) Large language models and storage walls Large Language Model (LLM), or LLM for short, is a class of artificial intelligence systems designed to understand and process human language. LLMs are trained on large datasets covering a wide range of human discourse, enabling them to perform a variety of language-related tasks, from translation and summarization to conversation and content creation. Over the past few years, the field of LLMs has witnessed significant development, characterized by major advances in model architecture, training methods, and data processing capabilities.

[0027] At the same time, this type of application is also one of the most computationally intensive applications in contemporary artificial intelligence research. The computational intensity of LLM is mainly attributed to its architecture and the scale of data processed. LLM usually contains hundreds of millions to tens of billions of parameters, which are learning weights that determine how LLM processes and generates language. Training these parameters requires multiple passes through a large data set, a process that requires a lot of computing resources. In order to accelerate these computing processes, chip products such as GPUs are currently used in LLMs due to their intensive computing power. However, the architectural evolution of GPU chips has not solved the challenges of large computing power and LLMs. Under the traditional von Neumann architecture, GPU chips face the "storage wall" problem when performing computationally intensive tasks, which results in the power consumption and performance of computing chips being limited by the data transfer between the processor and the memory, which seriously limits the improvement of artificial intelligence chips in terms of computing scale, density, efficiency, etc.

[0028] The following is a quantitative analysis of the memory access limitations of LLM.

[0029] A typical off-chip storage bandwidth is 100-200GByte / s. Taking the Llama2 7B parameter model as an example, the common parameter number system is fp16 number system, so the parameter size of this model is 14GByte. When batch=1, each token is calculated and the parameters are traversed once, so the memory access bandwidth supports a maximum of 200 / 14=10token / s.

[0030] A typical computing power chip is about 100TOPS. Most of the calculations in LLM are matrix-vector multiplications, which are equal to the number of parameters. That is, it takes 14GOPs to calculate one token. Therefore, from the perspective of computing power, it can support 100000 / 14=10000token / s. The computing power of common general computing power chips refers to the computing power of matrix*matrix. Matrix*vector will have parallelism loss. Assuming the loss is 1 / 16, the existing computing power chips can support about 500token / s from the perspective of computing power.

[0031] The chip computing power can support 500 tokens / s, but the memory access bandwidth can only support 10 tokens / s. The chip computing power cannot be fully utilized. This is the limitation brought by the "storage wall".

[0032] 2) Chip area and cost The area of ​​on-chip SRAM is relatively large. Taking TSMC's 7nm process as an example, 1MByte requires an area of ​​0.5 square millimeters. Assuming that all models of Llama2 7B need to be stored, 14GByte / 0.5=7000 square millimeters are required. However, the upper limit of the area of ​​a conventional single chip is about 600 square millimeters. If it is larger, it will bring serious problems with interconnection and yield. An area of ​​7000 square millimeters is completely unacceptable and unachievable.

[0033] 3) Low-precision training and LoRA fine-tuning technology In the rapidly developing field of artificial intelligence, it is becoming increasingly important to use large language models in an efficient and effective way. Fine-tuning a large model is to adapt it to a specific target task or domain. Although pre-trained large language models perform well on a variety of tasks, they are not designed for a specific task. Fine-tuning can adjust the model according to the needs of the target task, so that its performance on the task can be further improved. Low-Rank Adaptation is a parameter-efficient fine-tuning technique. The core idea is to perform an implicit low-rank transformation on the weight matrix of a large model. The overall idea and concept are related to principal component analysis (PCA) and singular value decomposition (SVD), which approximates a high-dimensional matrix or data set through a lower-dimensional representation. That is, trying to find a (linear) combination of a few dimensions in the original feature space (or matrix) that can capture most of the information in the data set. In summary, LoRA fine-tuning technology can adapt most of the large model application scenarios to the needs of new scenarios by adding a small number of parameters based on the lama basic parameter model.

[0034] Classic LLM uses 16-bit floating point values ​​(i.e. FP16 or BF16), and the main computational cost of LLM comes from floating point addition and multiplication operations. One way to address these challenges is to use post-training quantization to create low-bit models for inference. Model quantization is an efficient compression neural network technique that can significantly reduce the model size of LLM, thereby significantly saving GPU memory consumption.

[0035] In view of the limitations brought by the storage wall and the low-precision training and LoRA fine-tuning technology, the present invention provides an accelerator chip for large model algorithms, which can break the storage bandwidth limitation at a relatively low cost.

[0036] Combine the following Figure 1-Figure 7 The accelerator chip of the present invention for large model algorithms is described.

[0037] Please refer to Figure 1 , Figure 1 FIG. 1 is one of the structural diagrams of an accelerator chip for a large model algorithm provided by an embodiment of the present invention. Figure 1 As shown, the accelerator chip mainly includes: a matrix-vector multiplication unit 1 and a vector unit 2. The matrix-vector multiplication unit 1 is used for matrix-vector multiplication operations, data storage and transmission of large model algorithms. The vector unit 2 is used for vector operations, data storage and control of large model algorithms.

[0038] The matrix-vector multiplication unit 1 includes a read-only memory 11 and a static random access memory 12. The read-only memory 11 is used to store base model parameters, which are used to calculate base model matrix vectors. The static random access memory 12 is used to store fine-tuning parameters, which are used to calculate fine-tuning matrices.

[0039] Since the read-only memory 11 in the matrix-vector multiplication unit 1 serves as the main memory and the static random access memory 12 serves as the additional memory, compared with the traditional static random access memory as the main memory, the memory access bandwidth of the traditional static random access memory must match the chip computing power, and the required area far exceeds the upper limit of the area of ​​a single chip, while the memory access bandwidth of the read-only memory 11 must match the chip computing power, and the required area is lower than the upper limit of the area of ​​a single chip, that is, for the read-only memory and static random access memory of the same area on a single chip, the read-only memory 11 has a larger storage bandwidth, so that the computing power of the accelerator chip can match the storage bandwidth of the read-only memory.

[0040] The accelerator chip for a large model algorithm provided by an embodiment of the present invention mainly includes a matrix-vector multiplication unit and a vector unit. The matrix-vector multiplication unit is used for matrix-vector multiplication operations, data storage and transmission of the large model algorithm. The vector unit is used for vector operations, data storage and control of the large model algorithm. Among them, the matrix-vector multiplication unit includes a read-only memory and a static random access memory; the read-only memory is used to store base model parameters, and the base model parameters are used to calculate the base model matrix vector; that is, the read-only memory is used as the main memory; the static random access memory is used to store fine-tuning parameters, and the fine-tuning parameters are used to calculate the fine-tuning matrix; that is, the static random access memory is used as an additional memory. Since the read-only memory in the matrix-vector multiplication unit is used as the main memory and the static random access memory is used as the additional memory, compared with the traditional static random access memory as the main memory, the storage bandwidth of the read-only memory is larger, so that the computing power of the accelerator chip can match the storage bandwidth of the read-only memory, and the computing power of the accelerator chip can be fully utilized, so that the storage bandwidth limitation can be broken at a relatively low cost.

[0041] In one embodiment, a first preset number of matrix-vector multiplication units are arranged in a matrix to form a matrix-vector multiplication unit region. A second preset number of matrix-vector multiplication unit regions are arranged in a matrix.

[0042] For example, Figure 2 As shown in FIG. 1 , the matrix-vector multiplication unit area is represented by the MU area, and the four MU areas are arranged in a 2×2 matrix. Figure 3 As shown, each MU region includes 64 matrix-vector multiplication units arranged in an 8×8 matrix.

[0043] It should be noted that the number of MU regions and the number of matrix-vector multiplication units in each MU region are not limited thereto.

[0044] In one embodiment, a vector unit is disposed in each row between every two horizontally adjacent matrix-vector multiplication unit regions, and the vector unit is used to sequentially transmit the data pulses to be calculated to each matrix-vector multiplication unit along the horizontal direction.

[0045] For example, Figure 2 As shown, for the upper left and upper right MU areas adjacent in the horizontal direction, vector units Vu0-Vu7 are respectively set in the 8 rows between the two MU areas; for the lower left and lower right MU areas adjacent in the horizontal direction, vector units Vu8-Vu15 are respectively set in the 8 rows between the two MU areas.

[0046] like Figure 3 As shown, the vector unit Vu0 can transmit the data pulses to be calculated to the matrix-vector multiplication units MU0_0-MU0_7 in sequence along the horizontal direction.

[0047] In one embodiment, each column between every two vertically adjacent matrix-vector multiplication unit regions is further provided with an accumulation unit. The matrix-vector multiplication unit is used to transfer data to be calculated or intermediate calculation results to matrix-vector multiplication units adjacent in the horizontal and vertical directions. The accumulation unit is used to perform accumulation calculation on the data to be calculated or the intermediate calculation results transferred by each matrix-vector multiplication unit in the vertical direction, obtain the accumulation calculation result, and transfer the accumulation calculation result to the corresponding vector unit along the set data transfer path.

[0048] For example, Figure 2 As shown, for the MU areas adjacent to the upper left and lower left in the vertical direction, an accumulation unit ADD is set in each column position between the two MU areas; for the MU areas adjacent to the upper right and lower right in the vertical direction, an accumulation unit ADD is set in each column position between the two MU areas.

[0049] like Figure 3 As shown, for the MU areas adjacent to the upper right and lower right in the vertical direction, the 8 columns between the two MU areas are respectively provided with accumulation units ADD0-ADD7; for the MU areas adjacent to the upper left and lower left in the vertical direction, the 8 columns between the two MU areas are respectively provided with accumulation units ADD8-ADD15.

[0050] like Figure 3 As shown, the matrix-vector multiplication unit MU0_0 can transfer the data to be calculated or the intermediate calculation results to the matrix-vector multiplication unit MU0_1 adjacent to the horizontal direction and the matrix-vector multiplication unit MU1_0 adjacent to the vertical direction.

[0051] like Figure 4As shown, the accumulation unit ADD0 can accumulate the data to be calculated or the intermediate calculation results transmitted by the matrix vector multiplication units MU0_0-MU7_0 in the vertical direction, obtain the accumulated calculation results, and transmit the accumulated calculation results along the set data transmission path (ADD0-MU7_0-MU6_0-MU5_0-MU4_0-MU3_0-MU2_0-MU1_0-MU0_0-Vu0) to the vector unit Vu0. The accumulation unit ADD1 can accumulate the data to be calculated or the intermediate calculation results transmitted by the matrix vector multiplication units MU0_1-MU7_1 in the vertical direction, obtain the accumulated calculation results, and transmit the accumulated calculation results along the set data transmission path (ADD1-MU7_1-MU6_1-MU5_1-MU4_1-MU3_1-MU2_1-MU1_1-MU1_0-Vu1) to the vector unit Vu1. By analogy, the accumulation unit ADD6 can accumulate the data to be calculated or the intermediate calculation results transmitted by the matrix vector multiplication units MU0_6-MU7_6 in the vertical direction, obtain the accumulated calculation results, and transmit the accumulated calculation results along the set data transmission path (ADD6-MU7_6-MU6_6-MU6_5-MU6_4-MU6_3-MU6_2-MU6_1-MU6_0-Vu6) to the vector unit Vu6. The accumulation unit ADD7 can accumulate the data to be calculated or the intermediate calculation results transmitted by the matrix vector multiplication units MU0_7-MU7_7 in the vertical direction, obtain the accumulated calculation results, and transmit the accumulated calculation results along the set data transmission path (ADD7-MU7_7-MU7_6-MU7_5-MU7_4-MU7_3-MU7_2-MU7_1-MU7_0-Vu7) to the vector unit Vu7.

[0052] In one embodiment, the vector unit is also used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along a set data transmission path, the data transmitted by the matrix-vector multiplication unit, and the calculation result of the vector unit according to the calculation scenario.

[0053] For example, assuming that A×B+C×D is calculated, the vector unit needs to first calculate A×B and then store the result of A×B, then calculate C×D and then store the result of C×D, then read the results of A×B and C×D, calculate the cumulative sum of A×B+C×D, and finally store the cumulative sum of A×B+C×D.

[0054] It should be noted that the above computing scenarios are only used as examples. In specific implementations, different calculations and storage are performed according to different computing scenarios.

[0055] In one embodiment, each vector unit between two horizontally adjacent matrix-vector multiplication unit regions is a vector unit group. An operation unit is also provided between two vertically adjacent vector unit groups, and the operation unit is used to operate on the data transmitted by the corresponding vector units and broadcast the operation results to the corresponding vector units for storage.

[0056] For example, Figure 2 As shown, the vector units Vu0-Vu7 between the upper left and upper right MU regions adjacent in the horizontal direction are vector unit group 1, and the vector units Vu8-Vu15 between the lower left and lower right MU regions adjacent in the horizontal direction are vector unit group 2. A calculation unit Reduce is also provided between the vector unit group 1 and the vector unit group 2 adjacent in the vertical direction. Reduce is used to calculate the data transmitted by the vector units Vu0-Vu15, and broadcast the calculation results to the vector units Vu0-Vu15 for storage.

[0057] Combine the following Figure 4 This paper gives an example of how to implement matrix multiplication operation.

[0058] Suppose that the matrix-matrix multiplication operation of [2048, 2048]*[2048, 2048] is to be calculated, [2048, 2048] represents a matrix of 2048 rows and 2048 columns. The matrix-matrix multiplication operation of [2048, 2048]*[2048, 2048] is split into a matrix-vector multiplication operation of multiplying 2048 row vectors [1, 2048] by the matrix [2048, 2048].

[0059] Taking the matrix-vector multiplication of [1, 2048]*[2048, 2048]@fp16*int2 as an example, taking the MU area in the upper right corner as an example, the vector unit passes the 128 fp16 data read to the MU area, that is, it is passed to each matrix-vector multiplication unit in turn along the horizontal direction. Each matrix-vector multiplication unit outputs the operation result to the matrix-vector multiplication unit on the lower side along the vertical direction as the accumulated value, and the operation and accumulation are performed according to each column of the matrix-vector multiplication unit. The final accumulated value is completed in the accumulation unit ADD and returned to the corresponding vector unit along the path pointed by the dotted line. The MU area in the upper left corner is the same, which will not be repeated here. The matrix-vector multiplication operation results are evenly distributed to each vector unit in order. After multiple matrix-vector multiplication operations, the matrix-matrix multiplication operation of [2048, 2048]*[2048, 2048] can be completed.

[0060] In one embodiment, if Figure 5As shown, each matrix-vector multiplication unit includes: a base model matrix-vector multiplication unit MMS, a fine-tuning matrix-vector multiplication unit MMW, a first selector mmode, a second selector amode, an accumulator ADD, a first register REG1, a third selector isel, a second register REG2 and a third register REG3.

[0061] A base model matrix-vector multiplication unit MMW includes a read-only memory, which is used to perform base model matrix-vector calculations based on input data to be calculated, control information in a working signal iwork, and base model parameters stored in the read-only memory, such as implementing an @fp16*int2 matrix-vector multiplication operation of [1,128]*[128,128]. This embodiment is not limited to this.

[0062] The fine-tuning matrix-vector multiplication unit MMW includes a static random access memory, which is used to store the fine-tuning parameters in the working signal iwork in the static random access memory, and perform fine-tuning matrix-vector calculations based on the input data to be calculated and the fine-tuning parameters stored in the static random access memory, for example, to implement the @fp16*fp8 matrix-vector multiplication operation of [1,16]*[16,16]. This embodiment is not limited to this.

[0063] The first selector mmode is used to select a data path for output from the output result of the base model matrix-vector multiplication unit MMS, the output result of the fine-tuning matrix-vector multiplication unit MMW, the input data to be calculated and 0.

[0064] Specifically, the first selector mmode has 4 data paths: ① the output data path of the base model matrix-vector multiplication unit MMS. After selecting this data path, the output data of the first selector mmode is the operation result of the base model matrix-vector multiplication unit MMS; ② the output data path of the fine-tuning matrix-vector multiplication unit MMW. After selecting this data path, the output data of the first selector mmode is the operation result of the fine-tuning matrix-vector multiplication unit MMW; ③ the same data path. After selecting this data path, the input data and output data of the first selector mmode are consistent, and both are data to be calculated; ④ the 0 data path. After selecting this data path, the output data of the first selector mmode is 0, and it is only used as a data transmission path.

[0065] The second selector amode is used to select a data path from the operand icas_x transmitted in the horizontal direction, the operand icas_y transmitted in the vertical direction, and 0 for output.

[0066] Specifically, the second selector amode can select the source of the accumulated data. The direction of the accumulated data source in different computing scenarios is different, and it may not be accumulated. Select the direction of the accumulated data source as the horizontal direction, that is, select the operand icas_x transmitted in the horizontal direction as the accumulated data. Select the direction of the accumulated data source as the vertical direction, that is, select the operand icas_y transmitted in the numerical direction as the accumulated data. Select 0 to indicate no accumulation.

[0067] The accumulator ADD is used to accumulate the output data of the first selector mmode and the output data of the second selector amode.

[0068] It should be noted that when only data transfer is performed, one operand of the accumulator ADD is 0, so it can also play the role of data transfer.

[0069] The first register REG1 is used to optimize the data timing on the output paths of the accumulator ADD in the horizontal direction and the vertical direction, that is, the timing of ocas_x and ocas_y.

[0070] The third selector isel is used to select two data from the intermediate calculation result ires_x transmitted in the horizontal direction, the intermediate calculation result ires_y transmitted in the vertical direction, and the output result (ocas_x and ocas_y) of the accumulator ADD temporarily stored in the first register REG1.

[0071] The second register REG2 is used to optimize the data timing on the horizontal output path of the third selector isel, that is, the timing of ores_x.

[0072] The third register REG3 is used to optimize the data timing on the vertical output path of the third selector isel, that is, the timing of ores_y.

[0073] In one embodiment, each vector unit includes: a storage module VuMem, a calculation module VuElement, a first transfer module LocReduce, a second transfer module LocBcast and a control module VuCtrl.

[0074] The control modules in the vector units at the upper and lower ends of two vector unit groups adjacent to each other in the vertical direction are used to transmit corresponding control information to each vector unit.

[0075] For example, Figure 6As shown, vector units Vu0-Vu7 are vector unit group 1, vector units Vu8-Vu15 are vector unit group 2, and the vector units at the upper and lower ends of vector unit group 1 and vector unit group 2 adjacent in the vertical direction are vector units Vu0 and Vu15. After the control module VuCtrl in the vector unit Vu0 receives the instruction input in the 0th cycle, it transmits corresponding control information to the vector units Vu1-Vu7 in the vertical downward direction in the 1st to 7th cycles. After the control module VuCtrl in the vector unit Vu15 receives the instruction input in the 0th cycle, it also transmits corresponding control information to the vector units Vu14-Vu8 in the vertical upward direction in the 1st to 7th cycles. Then, the vector unit Vu0 pulsates along the horizontal direction in the nth cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region, and the vector unit Vu1 pulsates along the horizontal direction in the n+1th cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region, and so on, the vector unit Vu7 pulsates along the horizontal direction in the n+7th cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region. Similarly, the vector unit Vu15 pulsates along the horizontal direction in the nth cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region, and the vector unit Vu14 pulsates along the horizontal direction in the n+1th cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region, and so on, the vector unit Vu8 pulsates along the horizontal direction in the n+7th cycle to sequentially transmit information to each matrix-vector multiplication unit of the MU region.

[0076] The control module VuCtrl is used to transmit corresponding control information, such as connection relationship, transmission direction, etc., to each matrix-vector multiplication unit, corresponding accumulation unit ADD and operation unit Reduce in the horizontal direction according to the received control information.

[0077] The operation module VuElement is used to read the corresponding data in the storage module VuMem to perform vector operations.

[0078] Optionally, the operation module VuElement includes at least one basic unit selected from the group consisting of a multiplication basic unit, an addition basic unit, a division basic unit, a delay basic unit, a trigonometric function basic unit, a square root basic unit, a comparison basic unit, an exponential basic unit and an accumulation basic unit.

[0079] The first transfer module LocReduce is used to transfer the vector operation result of the operation module VuElement to the adjacent vector module.

[0080] The second transmission module LocBcast is used to transmit the vector operation result of the operation module VuElement across the vector modules.

[0081] The storage module VuMem is used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along the set data transmission path, the data transmitted by the matrix-vector multiplication unit and the calculation result of the vector unit according to the calculation scenario.

[0082] Optionally, the storage unit of the storage module VuMem is a static random access memory, and the port width and depth of the storage module VuMem match the computing capability of the accelerator chip.

[0083] Combine the following Figure 7 The specific process of implementing the Norm operator using the accelerator chip of this embodiment is described.

[0084] The Norm operator is one of the typical operators in the lama network, which is used to improve the numerical stability and robustness of large models and improve gradient flow.

[0085] like Figure 7 As shown, the specific process of implementing the Norm operator using the accelerator chip of this embodiment includes the following steps.

[0086] 1) Operator splitting: Split the Norm operator into multiple basic instructions, such as the pow instruction, the reducemean data dimension reduction instruction, the addsqrt square root instruction, and the div division instruction.

[0087] 2) Mode configuration: Configure each basic instruction in a format that can be parsed by the hardware. Taking addsqrt as an example, the configuration table is shown in Table 1.

[0088] Table 1

[0089] Among them, dly represents the working time of the target node, src0 represents the source of operand 0, and src1 represents the source of operand 1; each node has unique parameters: sram_ren0 represents whether read port 0 reads data, sram_ren1 represents whether read port 1 reads data, mtftype represents the specific working type of multifunction, and multifunction represents a set of calculations, such as sin, cos, e to the power of x, log2x, etc.

[0090] As shown in Table 1, the SRAM of the vector unit is read in cycle 0, and the read address is divided into param0 and param1; addition is performed in cycle 1, and the source of the addition data is the read data of the SRAM; square root operation is performed in cycle 2, and the data source is the addition basic unit; output is performed in cycle 3, and the output data is the result of the square root operation.

[0091] The configuration information of multiple basic instructions is configured in the control module VuCtrl.

[0092] 3) Workflow incentives: Arrange the Norm operator according to the basic instruction execution and inject incentives into the circuit of the accelerator chip.

[0093] According to the bandwidth design, each thread (one row of matrix-vector multiplication units is one thread) processes 64 vectors at a time, 4096 / 16 threads / 64=4, that is, the pow instruction is repeated 4 times, then the Norm operator can be injected into the circuit according to the following basic instruction sequence, and the control module VuCtrl will parse the information and send the data and information of each computing node to the corresponding computing unit at a specific delay. Norm=[powx4+reduce0+reduce1+addsqrt+div+mulx8], that is, the Norm operator is split into: pow instruction repeated 4 times, reduce instruction repeated 0 times, reduce instruction repeated 1 time, addsqrt instruction, div instruction, mulx instruction repeated 8 times.

[0094] 4) Pulsation transmission inside the hardware: Workflow excitation is input from both ends of vector units Vu0 and Vu15, and a vector unit Vu is transmitted to the middle every cycle, and each vector unit Vu is pulsed and started. The reason for the pulse start is that in physical implementation, the signal cannot travel an infinite distance. Finally, each vector unit Vu completes the operator calculation and writes the result back to the storage module VuMem.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An accelerator chip for large model algorithms, characterized in that: include: Matrix-vector multiplication unit and vector unit; the matrix-vector multiplication unit is used for matrix-vector multiplication operation, data storage and transmission of the large model algorithm; The vector unit is used for vector operations, data storage and control of the large model algorithm; Among them, the matrix-vector multiplication unit includes a read-only memory and a static random access memory; the read-only memory is used to store base model parameters, and the base model parameters are used to calculate the base model matrix vector; the static random access memory is used to store fine-tuning parameters, and the fine-tuning parameters are used to calculate the fine-tuning matrix.

2. The accelerator chip for large model algorithms according to claim 1, characterized in that: The first preset number of matrix-vector multiplication units are arranged in a matrix to form a matrix-vector multiplication unit area; the second preset number of matrix-vector multiplication unit areas are arranged in a matrix.

3. The accelerator chip for large model algorithms according to claim 2, characterized in that: A vector unit is arranged in each row between every two horizontally adjacent matrix-vector multiplication unit regions; the vector unit is used to transmit the data pulses to be calculated to each matrix-vector multiplication unit in sequence along the horizontal direction.

4. The accelerator chip for large model algorithms according to claim 3, characterized in that: An accumulation unit is also provided in each column between every two vertically adjacent matrix-vector multiplication unit areas; the matrix-vector multiplication unit is used to transfer the data to be calculated or the intermediate calculation results to the matrix-vector multiplication units adjacent in the horizontal and vertical directions; the accumulation unit is used to accumulate the data to be calculated or the intermediate calculation results transferred by each of the matrix-vector multiplication units in the vertical direction, obtain the accumulated calculation results, and transfer the accumulated calculation results to the corresponding vector units along the set data transfer path.

5. The accelerator chip for large model algorithms according to claim 4, characterized in that: The vector unit is also used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along the set data transmission path, the data transmitted by the matrix-vector multiplication unit and the calculation result of the vector unit according to the calculation scenario.

6. The accelerator chip for large model algorithms according to claim 5, characterized in that: The vector units between every two horizontally adjacent matrix-vector multiplication unit areas form a vector unit group; an operation unit is also arranged between two vertically adjacent vector unit groups, and the operation unit is used to perform operations on the data transmitted by the corresponding vector units, and broadcast the operation results to the corresponding vector units for storage.

7. The accelerator chip for large model algorithms according to claim 6, characterized in that: Each of the matrix-vector multiplication units comprises: A base model matrix vector multiplication unit, comprising the read-only memory, for performing base model matrix vector calculation according to input data to be calculated, control information in the working signal and the base model parameters stored in the read-only memory; A fine-tuning matrix-vector multiplication unit, comprising the static random access memory, for storing the fine-tuning parameters in the working signal into the static random access memory, and performing fine-tuning matrix-vector calculation according to input data to be calculated and the fine-tuning parameters stored in the static random access memory; A first selector is used to select a data path for output from the output result of the base model matrix-vector multiplication unit, the output result of the fine-tuning matrix-vector multiplication unit, the input data to be calculated and 0; The second selector is used to select one path of data from the operands transmitted in the horizontal direction, the operands transmitted in the vertical direction and 0 for output; an accumulator, used for accumulating the output data of the first selector and the output data of the second selector; A first register, used to optimize the data timing on the output path of the accumulator in the horizontal direction and the vertical direction; a third selector, used to select two paths of data from the intermediate calculation result transmitted in the horizontal direction, the intermediate calculation result transmitted in the vertical direction, and the output result of the accumulator temporarily stored in the first register; A second register, used for optimizing the data timing on the output path in the horizontal direction of the third selector; The third register is used to optimize the data timing on the vertical output path of the third selector.

8. The accelerator chip for large model algorithms according to claim 6, characterized in that: Each of the vector units comprises: a storage module, a calculation module, a first transmission module, a second transmission module and a control module; The control modules in the vector units at the upper and lower ends of two vector unit groups adjacent to each other in the vertical direction are used to transmit corresponding control information to each vector unit; The control module is used to transmit corresponding control information to each of the matrix-vector multiplication units, the corresponding accumulation unit and the operation unit in the horizontal direction according to the received control information; The operation module is used to read the corresponding data in the storage module to perform vector operation; The first transfer module is used to transfer the vector operation result of the operation module to the adjacent vector module; The second transfer module is used to transfer the vector operation result of the operation module across the vector module; The storage module is used to selectively store at least one of the data transmitted by other vector units, the data transmitted by the accumulation unit along the set data transmission path, the data transmitted by the matrix-vector multiplication unit and the calculation result of the vector unit according to the calculation scenario.

9. The accelerator chip for large model algorithms according to claim 8, characterized in that: The operation module includes at least one basic unit selected from the group consisting of a multiplication basic unit, an addition basic unit, a division basic unit, a delay basic unit, a trigonometric function basic unit, a square root basic unit, a comparison basic unit, an exponential basic unit and an accumulation basic unit.

10. The accelerator chip for large model algorithms according to claim 8, characterized in that: The storage unit of the storage module is a static random access memory, and the port width and depth of the storage module match the computing capability of the accelerator chip.

Images