Accelerator for accelerating reasoning process of unstructured sparse large language model

Abstract

The invention discloses an accelerator for accelerating the reasoning process of an unstructured sparse large language model and an operation method of the accelerator. The accelerator comprises a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnection bus module, a calculation core module and a router module. The interconnection bus module is connected with the direct memory access module, the memory controller module and the computing core module; the processor module is connected with the direct memory access module; the DDR memory module is connected with the memory controller module; the router module is connected with the computing core module and the adjacent router module; and the calculation core module obtains to-be-calculated data from the off-chip DDR memory module through the interconnection bus module. According to the method, the limit that data reuse is carried out only along a single row or a single column on a traditional systolic array is broken through, cross-row and cross-column data reuse is achieved, the pressure of hardware equipment is relieved, the positions of non-zero elements are coded through bit masks, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to an accelerator for accelerating the reasoning process of an unstructured sparse large language model and an operation method thereof. Background Art

[0002] With the development of deep learning, large language models have achieved remarkable success in many applications. For example, in popular applications such as chatbots, interactive conversations with human users are carried out. During the conversation, the large oracle model first understands the user's questions with context (called the pre-filling phase), and then generates replies token by token (called the decoding phase). However, the power of large language models comes from huge parameters and calculations. For example, ChatGPT, based on the GPT4 model developed by OpenAI, is currently the most popular chatbot. It has 1.8 trillion weight parameters, and a single token generation requires at least 540TFLOP computing power, which poses a huge challenge to the limited storage, memory bandwidth and computing power of hardware devices.

[0003] Parameter sparsification of large oracle models is an effective method to further speed up large oracle models by reducing computational complexity and memory usage. Some algorithmic research works have also proposed other structured sparsities of different granularities. However, if the performance of large oracle models with different sparsity patterns on the WikiText2 dataset is focused on the two metrics of parameter reduction (higher the better) and perplexity (lower the better), it is found that unstructured sparsity obtains 7.22 perplexity and 60% parameter reduction. This observation suggests that unstructured sparsity outperforms structured sparsity in terms of parameter reduction while maintaining model performance. However, it is challenging to fully exploit the significant parameter reduction brought by unstructured sparsity because it is difficult to exploit data reuse from its irregular sparsity pattern.

[0004] The reasoning process of the Big Prophet model consists only of the Transformer decoder module. First, the shape of 𝐿×𝐷 (hidden dimension size) is generated through three linear operations to generate 𝑄 (query), 𝐾 (key), 𝑉 (value) tensors. Secondly, the multi-head attention (MHA) is calculated and layer normalization is applied on the hidden dimension. Finally, the feed-forward network (FFN) expands the hidden dimension size to 4 times 𝐷 and shrinks it back to the 𝐷 dimension. The Big Prophet model will stack many decoder layers, and the text generation is divided into a pre-filling stage and multiple consecutive generation stages. The pre-filling stage outputs the first output token from the context in the decoder order, and then in the generation stage, each step takes the output token of the previous step as input to generate the next token. This process will continue until certain conditions are met, such as the generation of an EOS token. The difference between these two stages is that on the one hand, in the generation stage its hidden state input is a 1×𝐷 vector, which is different from the pre-filling stage which tends to be compute-bound and has a matrix-matrix multiplication constraint with a large 𝐿. The generation stage will tend to be memory-bound and have a vector-matrix multiplication constraint. On the other hand, the generation stage performs cross-attention, which connects the K and V tensors generated by the current decoding step with the tensors of the pre-filling stage and the previous generation stage steps. The reuse of the KV cache on the chip ensures the autoregressive property and avoids recomputation. However, it can be seen that the KV cache has a long usage period until the end of the conversation, which brings challenges to memory management.

[0005] Existing work does not support large unstructured sparse language models well. Google proposed FlexTPU for unstructured sparse computing, which forms a Z-shaped pattern by arranging non-zero values ​​in rows on the systolic array and then storing them continuously from the upper left PE to the lower right PE, called Z-shaped mapping. Although this method can reduce the number of iterations, it only considers fully utilizing the processing elements (PE) in space, but requires more waiting in time. For example, for a row of the array, it may come from 3 different rows of the original unstructured sparse matrix, which will cause its input feeding time to be extended by three times. In addition, during the mapping process, a separator needs to be inserted between different rows as an interval, which also causes PE to be idle and not fully filled with non-zero values. In order to implement these operations, the design inside each PE becomes complicated, and a four-state state machine is embedded for control, which brings the overhead of data path selection. Summary of the invention

[0006] The existing unstructured sparse large language models in the prior art have a large number of parameters and computing requirements, which require high storage, memory bandwidth and computing power of hardware devices. At the same time, due to their irregular sparse patterns, data reuse becomes difficult, and the existing hardware and algorithms do not support unstructured sparse large language models well, resulting in problems of low memory management and computing efficiency. The present invention provides an accelerator for accelerating the reasoning process of unstructured sparse large language models, breaking the limitation of data reuse along a single row or a single column on a traditional systolic array, realizing cross-row and cross-column data reuse, alleviating the pressure on hardware devices, and using bit masks to encode the positions of non-zero elements to improve computing efficiency.

[0007] In order to achieve the above object, the present invention provides the following technical solutions.

[0008] In a first aspect, the present invention provides an accelerator for accelerating the reasoning process of an unstructured sparse large language model, comprising: a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnect bus module, a computing core module and a router module; The interconnect bus module is connected to the direct memory access module, the memory controller module and the computing core module; the processor module is connected to the direct memory access module; the DDR memory module is connected to the memory controller module; the router module is connected to the computing core module and the adjacent router module; The computing core module obtains the data to be calculated from the off-chip DDR memory module through the interconnection bus module; The processor runs the compiler to generate instructions and weight parameters for the large language model and sends them to the DDR memory module through the direct memory access module. After that, the computing core module obtains the instructions and executes them one by one. The computing core modules transfer data to each other through the router module to realize multi-core computing of nonlinear functions; the computing core modules obtain the data to be calculated in the DDR memory module through the interconnection bus module; The memory controller module processes the memory read and write requests from the interconnect bus module and sends the requests to the DDR memory module.

[0009] As a further improvement of the present invention, the computing core module includes: an instruction control unit module, a bit mask decoder module, a diagonal data stream systolic array module, a post-processing module, an instruction cache module, a mask cache module, a weight cache module, a feature cache module, a result cache module, a hyperparameter cache module and a KV cache module; The processor module runs the software compiler to generate instructions for the sparse large oracle model and sends them to the DDR memory module through the direct memory access module; the instruction controller unit module obtains instructions from the instruction cache module and prepares for execution; the computing core module takes data from the weight cache module and the feature cache module, and feeds the data to the diagonal data flow systolic array module for dense or sparse matrix calculations; the calculation results are saved to the result cache module and processed by the post-processing module; the processed results are then saved to the DDR memory module again, ready for the next round of calculations or for use by other system components; Instruction control unit module: used to control the behavior of each module in the computing core module during operation; Bit mask decoder module: used to decode the bit mask matrix of the unstructured sparse large language model, translate the position information before sparseness, so as to obtain the position of the position information before sparseness in the original matrix, and feed the translation result to the diagonal data flow systolic array module; Diagonal Data Flow Systolic Array Module: used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations for unstructured sparsified large language models; Post-processing module: used to perform nonlinear operations in large language models and special data conversion processes; Instruction cache module: used to cache instructions in the model reasoning process for the instruction control unit module to obtain; Mask cache module: used to cache the bit mask matrix of the unstructured sparse large language model before it is compressed, so that the bit mask decoder module can obtain it. It reads data from the DDR memory module to update it in each round of calculation. Weight cache module: used to cache the weight parameters of the unstructured sparse large language model after compression, as the left-to-be-multiplied matrix of the diagonal data stream systolic array; Feature cache module: used to cache the feature map data input by the user of the large language model, which is fed into the right-to-be-multiplied matrix of the diagonal data flow systolic array module; KV cache module: used to cache the K and V matrices of the generation phase in the large language model inference process and used when the next word is generated; Result cache module: used to cache the partial sum results after the systolic array multiplication of the diagonal data stream, and perform cyclic accumulation of multiple rounds of matrix multiplication. After completion, it is saved to the off-chip DDR memory module or the on-chip KV cache module.

[0010] As a further improvement of the present invention, the diagonal data stream systolic array module includes an input feed module, a non-zero loader module, a processing element module, a crossbar switch module, a result merging module and a result accumulation module; The input feed module is connected to the first-level cross switch module; the non-zero loader module is connected to each column of data lines of the processing element (PE) module from the top; the cross switch of each level is connected to the processing element (PE) module on the diagonal, and the cross switch modules connecting the adjacent previous and next levels; each processing element module is connected to the cross switch on the diagonal, and is connected to the upper and lower adjacent processing element modules in the same column; the result merging module is connected to the processing element module in the last row, and is connected to the result accumulation module.

[0011] As a further improvement of the present invention, data is obtained from the input feeding module and arranged into the staggered form required by the array through streaming; the non-zero loader module cooperates with the bit mask decoder module to feed the matching weights and masks into the array together; wherein, the input feature data enters from the upper left corner and is fed downward each time; and the weight flows in from the top and is fixedly stored inside the processing element module; the processing element module obtains the required feature data from the cross switch module according to the ID number of the weight, performs multiplication and accumulation operations and decides whether to accumulate the output result of a processing element module; each processing element module column result is output from the bottom to the result merging module, and is accumulated across columns in the result accumulation module, and then the partial sum is accumulated for multiple rounds to obtain the final result.

[0012] As a further improvement of the present invention, the diagonal data stream systolic array module uses diagonal data streams to perform dense matrix multiplication, which is specifically expressed as follows: Assume that the systolic array performs the matrix multiplication 𝑌=𝑋𝑊𝑇; Among them: 𝑌∈R𝑀×𝑁 is the output matrix of each round; 𝑋∈R𝑀×𝑁 is the input matrix of each round; 𝑊∈R𝑁×𝑁 is the weight matrix of each round; 𝑁 is the array size; 𝑀 is the size of the input row block; The input feed module obtains an input vector of length 𝑀𝑁 and flows it into the array from the upper left corner along the 𝑘 direction; the weight tensor simultaneously flows vertically from top to bottom along the 𝑗 direction; in addition, the weight input to the 𝑖+1 column has a cycle delay with the 𝑖th column; the 𝑁 parallel crossbar switch modules are placed orthogonal to the 𝑘 direction, and each processing element module is connected to its local crossbar switch module to obtain the input tensor; When the input vector arrives at the 𝑘th crossbar module, each connected processing element module performs a multiplication between the input along the 𝑘 direction and the weight along the 𝑗 direction; after that, the processing element module forwards the partial sum to the next processing element module along the 𝑗 direction for accumulation; the bottom processing element module stores the PSUM to the output buffer; wherein the multiplication is pipelined along the diagonal 𝑘 direction; The diagonal data flow systolic array module performs sparse matrix multiplication using diagonal data flow; since 𝑊 is sparse, the bit mask uses 0 and 1 to represent zero elements and non-zero elements respectively; the bit mask decoder module converts the bit mask into (ID, CAS, MEG); Among them, ID is the l𝑜𝑔2𝑁-bit index in the original sparse matrix, which helps the processing element module select the corresponding input for calculation; CAS is a 1-bit cascade indicator, indicating whether the current processing element module should cascade the PSUM with the PSUM of the previous processing element module for accumulation; MEG is a 1-bit merge indicator used to merge cross-column PSUMs; The non-zero loader module feeds non-zero weights into an array with corresponding (ID, CAS, MEG); When the input flows along the 𝑘 direction, the processing element module will perform multiplication according to the ID of the weight; the processing element module selects an arbitrary value from the local crossbar; after that, the PSUM is forwarded to the next processing element module along the 𝑗 direction, where CAS indicates whether accumulation should be performed; check whether the PSUM output by MEG from the bottom processing element module needs to be merged.

[0013] As a further improvement of the present invention, the bit mask decoder module comprises: a bit mask buffer module, a collection module, a decoder FIFO module and a splicing module; The bit mask buffer module is connected to the collection module; the collection module is connected to the decoder FIFO module, and the decoder FIFO module is connected to the splicing module; The collection module is responsible for reading the original bit mask from the bit mask buffer module, and through the aggregation operation, it concentrates the scattered bit information to the low-order part, and sets the high-order part to 0. The processed data is temporarily stored in the decoder FIFO buffer, waiting for further processing; the splicing module decides when to read new data from the FIFO based on the weighted valid signal and splices it with the existing data.

[0014] As a further improvement of the present invention, the bit mask decoder module translates the bit mask into an indicator to guide the correct sparse calculation on the diagonal data stream systolic array; the input is the first row to the Nth row, which is a sparse bit mask; and then it is aggregated into a compact new row; The collection module obtains the original bit mask from the bit mask buffer. After multiple rounds of continuous aggregation, the bits are finally summed to the low bits, while the high bits are 0 and saved in a decoder FIFO buffer. The splicing module takes out a new row from the IFFO according to the weighted valid signal. If the length is not enough for the new row output, it reads new data from the FIFO. After obtaining the data, it determines whether the data meets the new row length requirement. If it does, it outputs it. The remaining data is continuously and seamlessly spliced ​​and output in sequence.

[0015] As a further improvement of the present invention, the post-processing module includes: an activation function module, a SoftMax module, a residual addition module, a layer normalization module, a transposition module and a rearrangement module; The activation function module is used to execute the activation function in the large language model reasoning process; The SoftMax module is used to map the value range of each node data to [0, 1] and constrain the sum of the output values ​​of each output node to 1; The residual addition module is used to add the feature data of the previous step to the feature data of the current step during the inference process; The layer normalization module performs normalization operations on the input data to constrain the data value to be within a certain range; The transpose module is used to rotate a 32x32 matrix in the direction of rows and columns, and then save the rotated matrix to the DDR memory module; The reordering module is used to reorder the jumping data into a sequential form and then save the whole block to the DDR memory module.

[0016] As a further improvement of the present invention, the KV cache module includes: an overflow module and a reload module; The overflow module is used to manage KV cache overflow and implement the last-in-first-out strategy; The reload module is used to split the on-chip memory into multiple blocks, each of which holds a fixed number of labeled KV tensors.

[0017] As a further improvement of the present invention, the overflow module is used to manage KV cache overflow and implement a last-in-first-out strategy, including: When the KV cache continues to grow and exceeds the capacity of the on-chip memory, the overflow module is responsible for transferring these excess KV tensors to the off-chip DDR memory module; During the cross-attention period of the decoding stage, due to the need to frequently access the KV cache data, the overflow module prioritizes retaining the KV tensors of the early steps in the on-chip memory, and if there is insufficient space, it overflows the KV tensors of the later steps.

[0018] As a further improvement of the present invention, the reload module is used to split the on-chip memory into multiple blocks, each block storing a fixed number of marked KV tensors, including: The on-chip memory is split into 𝑁 blocks, each of which stores 𝑀 labeled KV tensors; the first 𝑀×(𝑁−1) labeled KV tensors are pinned to the first 𝑁−1 memory blocks; they will always remain on-chip to reduce off-chip memory traffic; The most recently generated KV tensor will be written to the last memory block, and after 𝑀 labeled KV tensors are accumulated, the memory block will overflow to the off-chip memory; Where 𝑀 is chosen to be equal to the size of the systolic array.

[0019] In a second aspect, the present invention provides an operation method of an accelerator for accelerating an unstructured sparse large language model reasoning process, comprising: The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transmit the required data to the computing core module; The processor module converts the large language model described in high-level language into specific instructions and weight parameters, and stores this information into the DDR memory module through the direct memory access module; Start the computing core module, which reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module to jointly complete complex nonlinear function calculations. The memory controller module processes the memory read and write requests from the interconnect bus module and sends the requests to the DDR memory module to ensure that the memory access requests of each module are processed.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention breaks the limit of data reuse along a single row or a single column on the traditional systolic array through the coordinated work of the interconnect bus module, the computing core module and the router module, and realizes data reuse across rows and columns. In addition, the diagonal data flow mode adopted in the present invention significantly improves the data reuse rate, reduces unnecessary memory access, and thus reduces the memory bandwidth requirement. By encoding the non-zero element position using a bit mask, the present invention can efficiently process sparse matrices, reduce invalid calculations of zero elements, and improve calculation efficiency. At the same time, this sparse coding scheme also reduces off-chip memory traffic and reduces memory access latency. The computing core module in the present invention obtains the data to be calculated from the DDR memory module through the interconnect bus module and executes instructions one by one. The computing core modules transmit data to each other through the router module, realize multi-core calculation of nonlinear functions, and further improve calculation efficiency. The cooperation of the memory controller module and the DDR memory module enables memory read and write requests to be efficiently processed. At the same time, the present invention fixes the early KV cache tensor in the on-chip memory, reduces the reloading memory traffic, and improves the memory bandwidth utilization. This strategy also reduces the number of accesses to the off-chip memory and reduces system power consumption. Therefore, the present invention can more effectively utilize hardware resources, including processors, memory, and computing cores. This reduces the requirements for hardware devices, so that the reasoning process of unstructured sparse large language models can be efficiently run on a wider range of hardware platforms. At the same time, the accelerator design of the present invention has good scalability and flexibility. By adjusting the number and configuration of computing core modules, it can adapt to unstructured sparse large language models of different scales and complexities. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In the drawings: Figure 1 The overall structural diagram of an accelerator for accelerating the reasoning process of an unstructured sparse large language model according to the present invention; Figure 2 A schematic diagram of the systolic array module structure of an angular data stream of an accelerator for accelerating the reasoning process of an unstructured sparse large language model according to the present invention; Figure 3 A schematic diagram of the circuit structure of a processing element (PE) module and a crossbar switch module of an accelerator for accelerating the reasoning process of an unstructured sparse large language model according to the present invention; Figure 4 A schematic diagram of the structure of a result merging module and a result accumulating module of an accelerator for accelerating the reasoning process of an unstructured sparse large language model according to the present invention; Figure 5A schematic diagram of the structure of a bit mask decoder module of an accelerator for accelerating the reasoning process of an unstructured sparse large language model according to the present invention; Figure 6 The present invention is a schematic diagram of the overflow module and the overload module structure of an accelerator for accelerating the reasoning process of an unstructured sparse large language model. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0024] The unstructured sparse large language models in the existing technology have a huge number of parameters and computing requirements, and have high requirements on the storage, memory bandwidth and computing power of hardware devices. At the same time, their irregular sparse patterns make data reuse difficult, and the existing hardware and algorithms do not support unstructured sparse large language models well, resulting in problems with memory management and low computing efficiency.

[0025] The present invention provides an accelerator for accelerating the reasoning process of unstructured sparse large language models, such as Figure 1 As shown, the accelerator includes a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnect bus module, a computing core module and a router module.

[0026] The processor module is interconnected with the direct memory access module; the memory controller module is interconnected with the DDR memory module; the computing core module is interconnected with the router module; the interconnection bus module is simultaneously interconnected with the direct memory access module, the memory controller module and the computing core module, specifically: the interconnection bus module is connected with the direct memory access module; the interconnection bus module is connected with the memory controller module; the interconnection bus module is interconnected with the computing core module.

[0027] The number of memory controller modules, DDR memory modules, computing core modules and router modules is 4.

[0028] During the reasoning process of the sparse large oracle model, the computing core module establishes a communication link with the external DDR memory module through the interconnect bus module to ensure that the data to be calculated can be transmitted to the computing core module. During this process, the processor is responsible for running the compiler, generating corresponding instructions and weight parameters for the large language model, and sending this information seamlessly to the DDR memory module through the direct memory access module. Once the preparations are ready, the computing core module is started, reads the DDR memory module instructions, and executes these instructions one by one to advance the reasoning process of the model. In order to improve computing efficiency and handle complex operations, the computing core modules can transfer data to each other through the router module. This design not only supports linear calculations, but also realizes multi-core parallel calculations of nonlinear functions, thereby greatly improving the speed and performance of model reasoning.

[0029] The processor module is used to run the software compiler, decompose the unstructured sparse large language model into basic operators and processes, and generate hardware executable machine code according to the instruction set definition; when processing weight parameters, the processor module will arrange the weight parameters in order according to the node address. This arrangement helps to improve memory access efficiency and reduce the time overhead of data transfer, thereby improving the overall computing performance. For each computing core module, the processor will generate a corresponding hexadecimal txt file. These files contain all the instructions and operands that the core needs to execute, and are the basis for the computing core to execute model reasoning.

[0030] The direct memory access module is used to implement data loading and saving requests issued by the processor module, and directly transfer data to the DDR memory module through the interconnect bus module. This design allows data to move quickly between the memory and the processor without the intervention of the central processor, thereby greatly freeing up processor resources and improving the overall performance of the accelerator.

[0031] The memory controller module is used to process memory read and write requests from the interconnect bus module and convert these requests from the Advanced eXtensible Interface (AXI) protocol to the DDR protocol (Double Data Rate Synchronous Dynamic Random Access Memory). This conversion process is the key to ensure that data can be transmitted correctly and efficiently between other modules and the DDR memory.

[0032] The memory controller module first receives the memory read and write requests in the AIX protocol format from the interconnect bus module, and then parses these requests to extract the request type, address information, and data content. The parsed AIX protocol request is converted into a DDR protocol request. The converted DDR protocol request is sent to the DDR memory module for actual read and write operations. After completing the memory access operation, the memory controller module receives the response from the DDR memory module and converts it from the DDR protocol back to the AIX protocol, and finally returns the processing result to the interconnect bus module that initiated the request.

[0033] The interconnect bus module is used to provide interconnection for the direct memory access module, the memory controller module and the computing core module, and realize data transmission between these modules and arbitration of multi-port access.

[0034] The interconnect bus module can accurately and quickly route data from the source to the destination according to the requests of each module. When multiple modules try to access the same resource at the same time, the interconnect bus module is responsible for access arbitration to ensure orderly and fair allocation of access rights to avoid conflicts and deadlocks. The interconnect bus module also has bandwidth management functions, which can dynamically adjust the bandwidth allocation of each data channel according to the real-time load of the system to achieve the best overall performance, and can detect and trigger error recovery operations during data transmission to ensure the reliability of data transmission and the stability of the system.

[0035] The router module is used for data synchronization between different computing core modules. Each computing core module is equipped with a router module. A router module is not only connected to its own computing core module but also to the connected router. Each computing core module is equipped with a dedicated router module, which can promote efficient data communication and synchronization between different computing core modules, thereby enhancing the overall computing power and data processing efficiency of the system.

[0036] The router module can receive data from its corresponding computing core module and forward this data to other computing core modules that need this information. In a multi-core parallel computing environment, synchronization operations are sometimes required between different computing core modules to ensure the correctness and consistency of the calculation. The router module supports various synchronization mechanisms, including but not limited to barrier synchronization, event synchronization, etc.

[0037] Each router module is not only connected to its corresponding computing core module, but also to other adjacent router modules to form a network. This network design can be a variety of topologies such as ring, tree, and mesh. Secondly, the router module has flow control and congestion avoidance mechanisms, which can monitor the network load during data transmission, take measures to prevent data congestion and packet loss, and ensure the stable operation of the network. At the same time, fault tolerance is also taken into account, that is, in the event of a failure of some routers or computing core modules, the accelerator can continue to operate or resume normal operation after the failure is recovered.

[0038] The computing core module is used to execute the reasoning process of the unstructured sparse large language model. It receives the start control signal from the processor module and then reads the instructions and weight parameters from the DDR memory module. During the reasoning process, some intermediate results are also written to the DDR memory module until the entire reasoning is completed and the final result is stored back to the DDR memory module. At this time, the processor module can read the reasoning result through the direct memory access module.

[0039] The computing core module is responsible for executing the instruction sequence compiled and issued by the processor module during the reasoning process of the sparse large oracle model. Each computing core module has powerful parallel computing capabilities and a highly optimized algorithm library to cope with the complex computing requirements during the reasoning process of the large language model.

[0040] Before the inference process begins, the computing core module is in standby state. Once it receives the start control signal from the processor module, the computing core module begins to read instructions and corresponding weight parameters from the DDR memory module. After that, the computing core module gradually performs various arithmetic and logical operations according to the read instruction sequence. During the execution process, the computing core module also shares intermediate results with other modules to support more complex operations and algorithms. At different stages of the inference process, the computing core module writes some intermediate results back to the DDR memory module. This helps to disperse the pressure of data processing and ensure the consistency and availability of data during long-cycle inference processes. When all instructions are executed, the computing core module stores the final inference results in the specified area of ​​the DDR memory module. At this time, the computing core module sends a completion signal to the processor module to indicate that the inference process has ended. After receiving the completion signal, the processor module can read the inference results from the DDR memory module through the direct memory access module. These results can then be used for subsequent processing tasks, such as result analysis, decision making, etc.

[0041] The computing core modules include: instruction control unit module, bit mask decoder module, diagonal data flow systolic array module, post-processing module, instruction cache module, mask cache module, weight cache module, feature cache module, result cache module, hyperparameter cache module and KV cache module.

[0042] like Figure 1 As shown, the instruction controller unit module is interconnected with the instruction cache module, the diagonal data stream systolic array module and the post-processing module; the bit mask decoder module is connected with the mask cache module and the diagonal data stream systolic array module; the diagonal data stream systolic array module is connected with the bit mask decoder module, the weight cache module, the feature cache module, the KV cache module, and the result cache module; the post-processing module is connected with the result cache module and the hyperparameter cache module.

[0043] The processor module runs the software compiler to generate instructions for the sparse large oracle model and sends them to the DDR memory module through the direct memory access module. The instruction controller unit module obtains instructions from the instruction cache module and prepares them for execution. The computation core module takes data from the weight cache module and the feature cache module, which are fed to the diagonal data flow systolic array module for dense or sparse matrix calculations. The bit mask decoder module performs necessary translation work for unstructured sparsity in this process. The calculation results are first saved to the result cache module. Then, the calculation results are further processed by the post-processing module, such as performing nonlinear operations. The processed results are then saved to the DDR memory module again, ready for the next round of calculations or for use by other system components. The above process will be iterated for multiple rounds according to the specific needs of the model. During the entire inference process, the system reduces data access latency and improves computing efficiency through optimized data paths and efficient cache management strategies, thereby realizing the acceleration process of large oracle model inference. The hyperparameter cache module is used to store and manage various hyperparameters required in the model inference process. When performing its functions, the post-processing module needs to refer to certain hyperparameters in the hyperparameter cache module to adjust its processing strategy.

[0044] The processor module runs the software compiler to generate instructions for the sparse large oracle model, which are sent to the DDR memory module through the direct memory access module. The computing core module performs the reasoning process of the sparse large oracle model, each time by taking data from the weight cache module and the feature cache module, feeding it to the diagonal data flow systolic array module for dense or sparse matrix calculations, where the bit mask decoder module translates for unstructured sparsity. The output is saved to the result cache module, and then it can be sent to the post-processing module for non-linear operations. Then it is sent to the DDR memory module, and multiple rounds of operation are performed to complete the acceleration process of large oracle model reasoning.

[0045] Instruction control unit module: used to control the behavior of each module in the computing core module during operation. In each round of computing cycle, multiple instructions are obtained from the instruction cache module to form an instruction block, which is decoded into the value of each register. At the beginning of each round of computing cycle, the instruction control unit module obtains an instruction block from the instruction cache module. This instruction block contains several instructions, which together constitute the operation guide for the next computing cycle. After obtaining the instruction block, the instruction control unit module will decode it and convert the instructions into operations for each sub-module in the computing core module, thereby controlling the specific behavior of each sub-module in the computing core module.

[0046] like Figure 2 As shown, the diagonal data flow systolic array module is used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations of the unstructured sparsified large language model.

[0047] The diagonal data flow systolic array module includes an input feeding module, a non-zero loader module, a processing element (PE) module, a crossbar switch module, a result merging module and a result accumulation module.

[0048] The input feed module is connected to the first-level cross switch module in the upper left corner; the non-zero loader module is connected to each column of data lines of the processing element (PE) module from the top; the cross switch of each level is connected to the processing element (PE) module on the diagonal, and the cross switch modules connecting the adjacent previous and next levels; each processing element (PE) module is connected to the cross switch where the diagonal line is located, and is connected to the upper and lower adjacent processing element (PE) modules in the same column; the result merging module is connected to the processing element (PE) module in the last row, and is connected to the result accumulation module.

[0049] The systolic array module of diagonal data flow realizes effective data reuse for dense and sparse computing modes. It obtains data from the input feed module and then arranges it into the staggered form required by the array through streaming. The non-zero loader module cooperates with the bit mask decoder module to feed the matching weights and masks into the array together. Among them, the input feature data enters from the upper left corner and is fed down each time. The weight flows in from the top and is fixedly stored inside the processing element (PE) module. The processing element (PE) module obtains the required feature data from the cross switch module according to the ID number of the weight, performs multiplication and accumulation operations, and decides whether to accumulate the output results of a processing element (PE) module. The column results of each processing element (PE) module are output from the bottom to the result merging module, and cross-column accumulation is performed in the result accumulation module, and then the partial sums are accumulated for multiple rounds to obtain the final result.

[0050] When processing sparse matrices, the non-zero loader module works with the bit mask decoder module to ensure that the weights and masks are correctly fed into the array. The input feature data enters from the upper left corner and is passed down in sequence, while the weights flow in from the top and are fixedly stored inside the processing element (PE) module. The processing element (PE) module is the core component of the systolic array. It obtains the required feature data from the crossbar switch module according to the ID number of the weight and performs multiplication and accumulation operations. At the same time, the processing element (PE) module is also responsible for determining whether the output result of the previous processing element (PE) module needs to be accumulated. The result of each processing element (PE) module will be output downward to the result merging module, and then cross-column accumulation will be performed through the result accumulation module, and then the final result will be obtained through multiple rounds of accumulation. The systolic array module of diagonal data flow realizes the effective processing of dense and sparse computing modes through efficient input and output mechanisms and collaboration between processing element (PE) modules, improving computing efficiency and performance.

[0051] For the diagonal data flow to perform dense matrix (DMM, Dense Matrix Multiplication) multiplication, the specific expression is as follows: Assume that the systolic array performs the matrix multiplication 𝑌=𝑋𝑊𝑇; Among them: 𝑌∈R𝑀×𝑁 is the output matrix of each round; 𝑋∈R𝑀×𝑁 is the input matrix of each round; 𝑊∈R𝑁×𝑁 is the weight matrix of each round; 𝑁 is the array size; 𝑀 is the size of the input row block.

[0052] like Figure 2 As shown, at the beginning of each round of calculation, the input feed module obtains input vectors of 𝑀𝑁 length and simultaneously flows them into the array from the upper left corner along the 𝑘 direction. The weight tensor simultaneously flows vertically from top to bottom along the 𝑗 direction. In addition, the weight input to the 𝑖+1 column has a cycle delay with the 𝑖th column. 𝑁 parallel crossbar modules are placed orthogonally to the 𝑘 direction, and each processing element (PE) module is connected to its local crossbar module to obtain the input tensor (if necessary). When the input vector arrives at the 𝑘th crossbar module, each connected processing element (PE) module performs a multiplication between the input along the 𝑘 direction and the weight along the 𝑗 direction. The processing element (PE) module then forwards the PSUM (Partial Sum) to the next processing element (PE) module along the 𝑗 direction for accumulation. Finally, the bottom processing element (PE) module stores the PSUM to the output buffer. In addition, the multiplication is pipelined along the diagonal 𝑘 direction.

[0053] For sparse matrix multiplication (SpMM, Sparse Matrix-MatrixMultiplication) using diagonal data flow. Since 𝑊 is sparse, the design function of the systolic array that supports SpMM ensures that the corresponding elements are multiplied and accumulated, and keeps the processing element (PE) module as busy as possible. Before each round of calculation, the bit mask uses 0 and 1 to represent zero elements and non-zero elements respectively. The bit mask decoder module converts the bit mask into (ID, CAS, MEG).

[0054] Among them, ID is the l𝑜𝑔2𝑁-bit index in the original sparse matrix, which helps the processing element (PE) module select the corresponding input for calculation; CAS is a 1-bit cascade indicator, indicating whether the current processing element (PE) module should accumulate the PSUM by cascading with the PSUM of the previous processing element (PE) module; MEG is a 1-bit merge indicator used to merge cross-column PSUMs.

[0055] The non-zero loader feeds non-zero weights to an array with corresponding (ID, CAS, MEG). As the input flows in the 𝑘 direction, each processing element (PE) module will perform multiplication based on the ID of the weight. Each processing element (PE) module can select an arbitrary value from its local crossbar to provide sufficient bandwidth for the input and keep the processing element (PE) module busy. After that, the PSUM is forwarded to the next processing element (PE) module in the 𝑗 direction, where CAS indicates whether accumulation should be performed. This allows multiple compressed weight columns to be compressed into the same processing element (PE) module column and share inputs. Finally, check whether the PSUM output by MEG from the bottom processing element (PE) module needs to be merged.

[0056] like Figure 3 As shown, the processing element (PE) module accepts the branch update weight, ID number and cascade enable on the left, and saves it in the weight cache module, where the cascade is used to control the connection between the input and the MAC (Multiplier-Accumulator) multiplication accumulation to achieve whether to accumulate a processing element (PE) module. The ID number is sent to the cross switch to select a required data from N data. In the cross switch module, it is fed to the next round of calculation through the register.

[0057] The cross switch module includes: a register group module and a selector group module.

[0058] The input data is connected to the register group module and the selector group module for subsequent processing.

[0059] For sparse computing, the crossbar module needs to broadcast the input vector and select the corresponding data according to the given ID number from the bit mask decoder module. Since each crossbar module is a selector group module with a large number of inputs, registers are inserted to make it a multi-stage pipeline to optimize timing convergence. For dense computing, since data reuse is fixed, the crossbar module can be bypassed and connected to a fixed port to save power consumption caused by input selection.

[0060] The processing element (PE) module includes: a multiplication-accumulation module, a PE weight cache module, an input cascade control module and an output register module.

[0061] The PE weight cache module is connected to the multiplication accumulation module and the input cascade control module. The input cascade control module and the multiplication accumulation module are connected to the output register module.

[0062] To adapt to unstructured sparse computation with variable length accumulation size, a processing element (PE) module with reconfigurable input and partial sum (PSUM) data flow control is adopted. Each processing element (PE) module is connected to its local crossbar switch module through the line of IDW (input data width) + ID + CAS bits to receive input, index and cascade indicator. In order to update the weight corresponding to each processing element (PE) module, a counter is used to scan from 1 to 𝑁, and when the counter is equal to the index in the processing element (PE) module, the PE weight buffer module will be updated. After calculation, the processing element (PE) module reads the index [ID] number from the buffer and sends it to the local crossbar switch module to select the required value from the 𝑁 elements of the input vector. The data is then fed to the multiplication and accumulation module based on the hard MAC-IP of DSP48E2 to perform multiplication and accumulation operations. After that, the [CAS] flag is used to decide whether to adopt the PSUM value from the upper adjacent processing element (PE) module and switch the connection path of the cascade chain. When [CAS] is valid, it retains a cascade chain, otherwise it is connected to the bypass, where input P[j] is registered to MEG[j+1] and connected to output P[j+1]. Because the PCIN of the MAC is not used, the result is connected to output P[0]. The PSUM flows vertically from top to bottom, where the accumulation is controlled by the cascade indicator. Multiple PSUMs can be output on a single PE column, in which case they are shifted to the 𝑃 port and output together at the bottom PE. Therefore, flexible sparsity rates can be supported by configuring the P[0:M-1] bit width and the number of registers. For example, 𝑀=1, 2, 3, and 4 can support sparsity rates of 0%, 50%, 33%, and 25%, respectively.

[0063] like Figure 4As shown, the result merging module is the different results from the processing element (PE) module. It is necessary to merge and add the current highest and the next lowest, and then collect the data to the low bit, and then align the rows according to the valid bit information of each data. Finally, a whole row of data is saved in the result buffer module, where it is necessary to read out the result of the previous step and the new value for co-location accumulation.

[0064] The result merge module is based on sparse coding. Non-zero values ​​originally from the same column can be arranged on two PEcols. Each PEcol in the array can output 𝑀×PSUM, for example, M=4, whose index is the original column. When PEcol[0] outputs "1", "2", and PEcol[1] outputs "2", "3", and "4", this means that P[2] and P[0] originally came from the same column "2" and need to be merged. Since PEcol[i] and PEcol[i+1] differ by one cycle, the previous data can be retained in the register. MEG indicates which of the 𝑀 entries is merged. PEcol[i] sends the low-order P[0] to the high P[MEG] of PEcol[i+1], matching the width of the PSUM buffer. MEG is collected and compared hierarchically from local to global. The valid signal enables the write address of each port, and elements of different cycles are temporarily cached in the distributed RAM to align the entire row.

[0065] The result accumulation module is used to align the result data, send it to the multi-round accumulator module, and write it to the specified address in the PSUM buffer. To avoid conflicts, the buffer is designed as a double ping-pong structure. For intensive calculations, merging is not required, so only the P[0] port of each PEcol is used.

[0066] like Figure 5 As shown in the figure, the bit mask decoder module is used to decode the bit mask matrix of the unstructured sparse large language model, translate the position information before sparseness, and obtain their positions in the original matrix. And add three registers of index value, cascade value, and fusion value to each weight, which are fed together to the diagonal data flow systolic array module to control the behavior of each PE calculation in the processing element PE module.

[0067] The bit mask decoder module receives a sparse bit mask matrix, and its task is to interpret these bit masks and recover the position information of non-zero elements in the original data. This is especially important for processing large sparse matrices. The decoded bit mask information is used to control when each PE in the processing element (PE) module performs calculations and how to process its input data. This control mechanism ensures efficient processing of sparse matrices.

[0068] The bit mask decoder module includes: a bit mask buffer module, a collection module, a decoder FIFO module and a splicing module.

[0069] The bit mask buffer module is connected to the collection module. The collection module is connected to the decoder FIFO module, and the decoder FIFO module is connected to the splicing module.

[0070] The position of each non-zero element in the sparse matrix is ​​encoded with a 1-bit mask. The bit mask decoder module is used to convert the binary mask into (ID, CAS, MEG) to control the behavior of PE. Since each bit mask has the information of 𝑛 rows.

[0071] First, collect the number of the first non-zero elements in each row of the position to get the ID.

[0072] The aligned non-zero elements are evenly distributed to different processing element (PE) blocks, so MEG is set to 1 when the non-zero elements are distributed to different processing element (PE) block groups.

[0073] There are two criteria for CAS to be set to 0: the first non-zero element in each row; or MEG is set to 1; otherwise CAS is set to 1.

[0074] The collection module is used to read the bit mask with a width of [0:N-1] from the bit mask buffer module in each calculation cycle. The tree structure decoder will go through several levels to find the position of 1, that is, in-4→out-4, in-8→out-8 until in-N→out-N, such as Figure 5 All non-zero elements are arranged in big-endian format and stored into the decoder FIFO module.

[0075] The splicing module uses a counter and a shift register to get aligned non-zero elements from the decoder FIFO module. When the counter indicates that there are enough non-zero elements, it will be directly output. Otherwise, it will continue to read new items from the decoder FIFO module and connect them to the shift register.

[0076] Therefore, the collection module is responsible for reading the original bit mask from the bit mask buffer, and through a series of aggregation operations, the scattered bit information is concentrated into the low-order part, and the high-order part is set to 0. The processed data is temporarily stored in the decoder FIFO buffer, awaiting further processing. The splicing module decides when to read new data from the FIFO based on the weighted valid signal and splices it with the existing data. This process involves determining the length of the new data and how to temporarily store the insufficient length of the data in the shift register for subsequent seamless splicing.

[0077] The bit mask decoder module translates the bit mask into ID (index), CAS (cascade), and MEG (merge) indicators to guide the correct sparse calculation on the diagonal data stream systolic array. The input is the first row to the Nth row, which is a sparse bit mask. Next, it is aggregated into a compact new row. The collection module obtains the original bit mask from the bit mask buffer. After multiple rounds of continuous aggregation, it is finally aggregated to the leftmost low bit, and the high bit is 0 and saved in a decoder FIFO buffer. The splicing module on the right takes out the new row from the IFFO according to the weight valid signal. If it is not enough for the new row output, it reads new data from the FIFO. After getting the data, it determines whether it is enough for the new row length according to the number of counters. If it is enough, it is output, and then the rest is saved in the shift register, and the seamless splicing output is continuous and continuous.

[0078] Post-processing module: used to perform nonlinear operations in large language models and special data conversion processes.

[0079] The post-processing modules include: activation function module, SoftMax module, residual addition module, layer normalization module, transposition module and rearrangement module.

[0080] The residual addition module is connected to the layer normalization module. The remaining modules are connected in parallel to the input and output of the port.

[0081] The activation function module is used to execute the activation function in the large language model reasoning process, including the implementation of EXP and GELU functions; the SoftMax module is used to map the value range of each node data to [0, 1], and constrain the sum of the output values ​​of each output node to be 1; the residual addition module is used to add the feature data of the previous step to the feature data of the current step during the reasoning process; the layer normalization module performs normalization operations on the input data and constrains the data value to be within a certain range; the transpose module is used to rotate a 32x32 matrix in the direction of rows and columns, and then save the rotated matrix to the DDR memory module; the rearrangement module is used to rearrange the jumping data into a sequential form, and then save the entire block to the DDR memory module.

[0082] The post-processing module is responsible for performing non-linear operations and special data transformation processes. In addition to non-linear operations, the post-processing module can also perform special data transformation processes such as matrix transposition, matrix decomposition, feature normalization, etc. These operations are very important to improve the accuracy and performance of the model.

[0083] The instruction cache module is used to cache instructions in the model inference process for the instruction control unit module to obtain. Whenever the instruction is about to be read, the corresponding data is read from the DDR memory module to update the cache.

[0084] The instruction cache module can cache some instructions in the model reasoning process so that they can be quickly provided to the instruction control unit module. When the instructions in the instruction cache module are about to be read, it will automatically read the corresponding data from the DDR memory module to update the cache to ensure the continuity and correctness of the instructions. The main function of the instruction cache module is to reduce the number of accesses to the DDR memory, thereby reducing the power consumption and latency of the system. In addition, the instruction cache module can also increase the access speed of instructions, thereby improving the execution efficiency and performance of the processor.

[0085] Mask cache module: used to cache the bit mask matrix of the unstructured sparse large language model before it is compressed for the bit mask decoder module to obtain, and read data from the DDR memory module to update it in each round of calculation.

[0086] The mask cache module can cache a portion of the uncompressed bit mask matrix so that it can be quickly provided to the bit mask decoder module. In each round of calculation, the mask cache module reads the corresponding data from the DDR memory module to update the cache to ensure the continuity and correctness of the bit mask matrix. The main function of the mask cache module is to reduce the number of accesses to the DDR memory module, thereby reducing the power consumption and latency of the system. In addition, the mask cache module can also increase the access speed of the bit mask matrix, thereby improving the execution efficiency and performance of the bit mask decoder module.

[0087] The weight cache module is used to cache the weight parameters of the unstructured sparse large language model after compression as the left-to-be-multiplied matrix of the diagonal data stream systolic array.

[0088] The feature cache module is used to cache the feature map data of the large language model user input, which is fed into the right-to-be-multiplied matrix of the diagonal data flow systolic array module.

[0089] Result cache module: used to cache the partial sum results after the diagonal data stream systolic array multiplication, and perform cyclic accumulation of multiple rounds of matrix multiplication. After completion, it is saved to the off-chip DDR memory module or the on-chip KV cache module.

[0090] like Figure 6 As shown, the KV cache module is used to cache the K and V matrices of the generation phase in the large language model inference process and is used when the next word is generated.

[0091] The KV cache module includes: an overflow module and a reload module.

[0092] The overflow module and the reload module are connected to the DDR memory module.

[0093] The role of the overflow module is that since the KV cache will continue to grow, the limited on-chip memory cannot always accommodate all the KV caches. The KV cache must be overflowed to the off-chip memory and reloaded in the next decoding step for calculation, which will generate huge off-chip memory traffic during the cross-attention period of the decoding stage. Considering that each KV tensor remains active until the end of the conversation, the KV tensors of the early steps should always be kept in the on-chip memory, and if there is insufficient space, the later KV tensors are overflowed. Therefore, a simple but effective KV cache on-chip allocation scheme is used, that is, the early KV tensors are fixed in the on-chip memory and the late KV tensors are overflowed using a last-in-first-out (LIFO) strategy. Therefore, the overflow overflows the KV tensor in a block size containing the 𝑀-labeled KV tensors.

[0094] The reload module is constructed by splitting the on-chip memory into 𝑁 blocks, each of which holds 𝑀 labeled KV tensors. The first 𝑀×(𝑁−1) labeled KV tensors are pinned to the first 𝑁−1 memory blocks. They will always remain on-chip to reduce off-chip memory traffic. In addition, the latest generated KV tensor will be written to the last memory block, which will overflow to the off-chip memory once it accumulates 𝑀 labeled KV tensors. 𝑀 is chosen to be equal to the size of the contraction array so that block reloads can overlap computations well. Merged KV cache memory accesses. Due to CAS latency, off-chip memory access requests for small vectors will incur huge latency overhead. Therefore, reloading loads KV tensors in blocks to avoid performance degradation.

[0095] Overflow module and reload module, in order to overflow the whole block and reload in batches, use LIFO for last in first out. On the left is the overflow rule, cache the K vector, and let it accumulate 32 before the whole block overflows to the off-chip memory. On the right is the next time you need to use the earlier block, reload multiple blocks at one time.

[0096] In summary, in a traditional systolic array, data can only be reused along a single row or a single column, while in the present invention, a diagonal data flow systolic array is used, and data can be reused along the diagonal direction, realizing data reuse across rows and columns, thereby improving computing efficiency and hardware utilization. In addition, the area overhead of the diagonal data flow systolic array is also very small, because it only needs to add a small amount of logic and storage resources to the original systolic array to achieve it.

[0097] For the pruned LLM parameters, the present invention uses a bit mask to encode the non-zero element position, thereby minimizing the off-chip memory traffic. And through the bit mask decoder module for decoding, the detailed value index and cumulative index can be quickly obtained at runtime, thereby improving the computational efficiency and hardware utilization.

[0098] Finally, the present invention fixes the early KV cache tensor method in the on-chip memory to reduce the reload memory traffic. In addition, the method of merging memory accesses is also adopted to improve the memory bandwidth utilization. The advantage is that it can effectively reduce the number of off-chip memory accesses, thereby reducing memory latency and power consumption.

[0099] It should be clear that the number of memory controller modules, DDR memory modules, computing core modules and router modules in the present invention is 4, but it can be increased or decreased according to actual needs.

[0100] A second object of the present invention is to provide an operation method of an accelerator for accelerating an unstructured sparse large language model reasoning process, comprising: The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transmit the required data to the computing core module; The processor module converts the large language model described in high-level language into specific instructions and weight parameters, and stores the information into the DDR memory module through the direct memory access module; Start the computing core module, which reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module to jointly complete complex nonlinear function calculations. The memory controller module processes the memory read and write requests from the interconnect bus module and sends the requests to the DDR memory module to ensure that the memory access requests of each module are processed.

[0101] Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but rather with reference to the preceding claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a disclaimer of such subject matter, nor should it be considered that the applicants did not consider such subject matter to be part of the disclosed inventive subject matter.

[0102] The above content is a further detailed description of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, they can also make several simple deductions or substitutions, which should be regarded as belonging to the protection scope of the present invention determined by the submitted claims.

Claims

1. An accelerator for accelerating the inference process of unstructured sparse large language models, characterized in that: include: Processor module, direct memory access module, memory controller module, DDR memory module, interconnect bus module, computing core module and router module; The interconnect bus module is connected to the direct memory access module, the memory controller module and the computing core module; the processor module is connected to the direct memory access module; the DDR memory module is connected to the memory controller module; the router module is connected to the computing core module and the adjacent router module; The computing core module obtains the data to be calculated from the off-chip DDR memory module through the interconnection bus module; The processor runs the compiler to generate instructions and weight parameters for the large language model and sends them to the DDR memory module through the direct memory access module; After that, the computing core module obtains the instructions and executes them one by one; The computing core modules transfer data to each other through the router module to realize multi-core computing of nonlinear functions; the computing core modules obtain the data to be calculated in the DDR memory module through the interconnection bus module; The memory controller module processes the memory read and write requests from the interconnect bus module and sends the requests to the DDR memory module.

2. The accelerator for accelerating the inference process of unstructured sparse large language models according to claim 1, characterized in that: The computing core module includes: an instruction control unit module, a bit mask decoder module, a diagonal data stream systolic array module, a post-processing module, an instruction cache module, a mask cache module, a weight cache module, a feature cache module, a result cache module, a hyperparameter cache module and a KV cache module; The processor module runs the software compiler to generate instructions for the sparse large oracle model and sends them to the DDR memory module through the direct memory access module; the instruction controller unit module obtains instructions from the instruction cache module and prepares for execution; the computing core module takes data from the weight cache module and the feature cache module, and feeds the data to the diagonal data flow systolic array module for dense or sparse matrix calculations; the calculation results are saved to the result cache module and processed by the post-processing module; the processed results are then saved to the DDR memory module again, ready for the next round of calculations or for use by other system components; Instruction control unit module: used to control the behavior of each module in the computing core module during operation; Bit mask decoder module: used to decode the bit mask matrix of the unstructured sparse large language model, translate the position information before sparseness, so as to obtain the position of the position information before sparseness in the original matrix, and feed the translation result to the diagonal data flow systolic array module; Diagonal Data Flow Systolic Array Module: used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations for unstructured sparsified large language models; Post-processing module: used to perform nonlinear operations in large language models and special data conversion processes; Instruction cache module: used to cache instructions in the model reasoning process for the instruction control unit module to obtain; Mask cache module: used to cache the bit mask matrix of the unstructured sparse large language model before it is compressed, so that the bit mask decoder module can obtain it. It reads data from the DDR memory module to update it in each round of calculation. Weight cache module: used to cache the weight parameters of the unstructured sparse large language model after compression, as the left-to-be-multiplied matrix of the diagonal data stream systolic array; Feature cache module: used to cache the feature map data input by the user of the large language model, which is fed into the right-to-be-multiplied matrix of the diagonal data flow systolic array module; KV cache module: used to cache the K and V matrices of the generation phase in the large language model inference process and used when the next word is generated; Result cache module: used to cache the partial sum results after the systolic array multiplication of the diagonal data stream, and perform cyclic accumulation of multiple rounds of matrix multiplication. After completion, it is saved to the off-chip DDR memory module or the on-chip KV cache module.

3. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 2, characterized in that: The diagonal data stream systolic array module includes an input feed module, a non-zero loader module, a processing element module, a crossbar switch module, a result merging module and a result accumulation module; The input feed module is connected to the first-level cross switch module; the non-zero loader module is connected to each column of data lines of the processing element (PE) module from the top; the cross switch of each level is connected to the processing element (PE) module on the diagonal, and the cross switch modules connecting the adjacent previous and next levels; each processing element module is connected to the cross switch on the diagonal, and is connected to the upper and lower adjacent processing element modules in the same column; the result merging module is connected to the processing element module in the last row, and is connected to the result accumulation module.

4. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 3, characterized in that: The data is obtained from the input feeding module and arranged into the staggered form required by the array through streaming; the non-zero loader module cooperates with the bit mask decoder module to feed the matching weights and masks into the array; wherein, the input feature data enters from the upper left corner and is fed downward each time; and the weight flows in from the top and is fixedly stored inside the processing element module; the processing element module obtains the required feature data from the cross switch module according to the ID number of the weight, performs multiplication and accumulation operations and decides whether to accumulate the output result of a processing element module; each processing element module column result is output from the bottom to the result merging module, and is accumulated across columns in the result accumulation module, and then the partial sum is accumulated for multiple rounds to obtain the final result.

5. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 3, characterized in that: The diagonal data flow systolic array module uses diagonal data flow to perform dense matrix multiplication, which is specifically expressed as follows: Assume that the systolic array performs the matrix multiplication 𝑌=𝑋𝑊𝑇; Among them: 𝑌∈R𝑀×𝑁 is the output matrix of each round; 𝑋∈R𝑀×𝑁 is the input matrix of each round; 𝑊∈R𝑁×𝑁 is the weight matrix of each round; 𝑁 is the array size; 𝑀 is the size of the input row block; The input feed module obtains an input vector of length 𝑀𝑁 and flows it into the array from the upper left corner along the 𝑘 direction; the weight tensor simultaneously flows vertically from top to bottom along the 𝑗 direction; in addition, the weight input to the 𝑖+1 column has a cycle delay with the 𝑖th column; the 𝑁 parallel crossbar switch modules are placed orthogonal to the 𝑘 direction, and each processing element module is connected to its local crossbar switch module to obtain the input tensor; When the input vector arrives at the 𝑘th crossbar module, each connected processing element module performs a multiplication between the input along the 𝑘 direction and the weight along the 𝑗 direction; after that, the processing element module forwards the partial sum to the next processing element module along the 𝑗 direction for accumulation; the bottom processing element module stores the PSUM to the output buffer; wherein the multiplication is pipelined along the diagonal 𝑘 direction; The diagonal data flow systolic array module performs sparse matrix multiplication using diagonal data flow; since 𝑊 is sparse, the bit mask uses 0 and 1 to represent zero elements and non-zero elements respectively; the bit mask decoder module converts the bit mask into (ID, CAS, MEG); Among them, ID is the l𝑜𝑔2𝑁-bit index in the original sparse matrix, which helps the processing element module select the corresponding input for calculation; CAS is a 1-bit cascade indicator, indicating whether the current processing element module should cascade the PSUM with the PSUM of the previous processing element module for accumulation; MEG is a 1-bit merge indicator used to merge cross-column PSUMs; The non-zero loader module feeds non-zero weights into an array with corresponding (ID, CAS, MEG); When the input flows along the 𝑘 direction, the processing element module will perform multiplication according to the ID of the weight; the processing element module selects an arbitrary value from the local crossbar; after that, the PSUM is forwarded to the next processing element module along the 𝑗 direction, where CAS indicates whether accumulation should be performed; check whether the PSUM output by MEG from the bottom processing element module needs to be merged.

6. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 2, characterized in that: The bit mask decoder module includes: a bit mask buffer module, a collection module, a decoder FIFO module and a splicing module; The bit mask buffer module is connected to the collection module; the collection module is connected to the decoder FIFO module, and the decoder FIFO module is connected to the splicing module; The collection module is responsible for reading the original bit mask from the bit mask buffer module, and through the aggregation operation, it concentrates the scattered bit information to the low-order part, and sets the high-order part to 0. The processed data is temporarily stored in the decoder FIFO buffer, waiting for further processing; the splicing module decides when to read new data from the FIFO based on the weighted valid signal and splices it with the existing data.

7. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 6, characterized in that: The bit mask decoder module translates the bit mask into indicators to guide the correct sparse calculation on the diagonal data stream systolic array; the input is the first row to the Nth row, which is a sparse bit mask; then it is aggregated into a compact new row; The collection module obtains the original bit mask from the bit mask buffer. After multiple rounds of continuous aggregation, the bits are finally summed to the low bits, while the high bits are 0 and saved in a decoder FIFO buffer. The splicing module takes out a new row from the IFFO according to the weighted valid signal. If the length is not enough for the new row output, it reads new data from the FIFO. After obtaining the data, it determines whether the data meets the new row length requirement. If it does, it outputs it. The remaining data is continuously and seamlessly spliced ​​and output in sequence.

8. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 2, characterized in that: The post-processing module includes: an activation function module, a SoftMax module, a residual addition module, a layer normalization module, a transposition module and a rearrangement module; The activation function module is used to execute the activation function in the large language model reasoning process; The SoftMax module is used to map the value range of each node data to [0, 1] and constrain the sum of the output values ​​of each output node to 1; The residual addition module is used to add the feature data of the previous step to the feature data of the current step during the inference process; The layer normalization module performs normalization operations on the input data to constrain the data value to be within a certain range; The transpose module is used to rotate a 32x32 matrix in the direction of rows and columns, and then save the rotated matrix to the DDR memory module; The reordering module is used to reorder the jumping data into a sequential form and then save the whole block to the DDR memory module.

9. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 2, characterized in that: The KV cache module includes: an overflow module and a reload module; The overflow module is used to manage KV cache overflow and implement the last-in-first-out strategy; The reload module is used to split the on-chip memory into multiple blocks, each of which holds a fixed number of labeled KV tensors.

10. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 9, characterized in that: The overflow module is used to manage KV cache overflow and implement a last-in-first-out strategy, including: When the KV cache continues to grow and exceeds the capacity of the on-chip memory, the overflow module is responsible for transferring these excess KV tensors to the off-chip DDR memory module; During the cross-attention period of the decoding stage, due to the need to frequently access the KV cache data, the overflow module prioritizes retaining the KV tensors of the early steps in the on-chip memory, and if there is insufficient space, it overflows the KV tensors of the later steps.

11. The accelerator for accelerating the reasoning process of unstructured sparse large language models according to claim 9, characterized in that: The reload module is used to split the on-chip memory into multiple blocks, each block storing a fixed number of labeled KV tensors, including: The on-chip memory is split into 𝑁 blocks, each of which stores 𝑀 labeled KV tensors; the first 𝑀×(𝑁−1) labeled KV tensors are pinned to the first 𝑁−1 memory blocks; they will always remain on-chip to reduce off-chip memory traffic; The most recently generated KV tensor will be written to the last memory block, and after 𝑀 labeled KV tensors are accumulated, the memory block will overflow to the off-chip memory; Where 𝑀 is chosen to be equal to the size of the systolic array.

12. A method for operating an accelerator for accelerating the reasoning process of an unstructured sparse large language model, characterized in that: An accelerator for accelerating the reasoning process of an unstructured sparse large language model according to any one of claims 1 to 11, comprising: The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transmit the required data to the computing core module; The processor module converts the large language model described in high-level language into specific instructions and weight parameters, and stores this information into the DDR memory module through the direct memory access module; Start the computing core module, which reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module to jointly complete complex nonlinear function calculations. The memory controller module processes the memory read and write requests from the interconnect bus module and sends the requests to the DDR memory module to ensure that the memory access requests of each module are processed.

Images