Accelerator for accelerating inference process of unstructured sparse large language model and operating method therefor

Abstract

Disclosed are an accelerator for accelerating an inference process of an unstructured sparse large language model and an operating method therefor. The accelerator comprises: a processor module, a direct memory access module, memory controller modules, DDR memory modules, an interconnection bus module, computing core modules, and router modules. The interconnection bus module is connected to the direct memory access module, the memory controller modules, and the computing core modules. The processor module is connected to the direct memory access module. The DDR memory modules are connected to the memory controller modules. Each router module is connected to a computing core module and an adjacent router module. The computing core modules acquire, by means of the interconnect bus module, data to be computed from the off-chip DDR memory modules. According to the present invention, the limitation of data reuse only along a single row or a single column on a traditional systolic array is broken, achieving data reuse across rows and columns, and alleviating pressure on a hardware device. Furthermore, a bit mask is used to encode a non-zero element position, thereby improving computing efficiency.

Description

Accelerators and their operation methods for speeding up the inference process of unstructured sparse large language models Technical Field

[0001] This invention relates to the field of information technology, and specifically to an accelerator and its operating method for accelerating the reasoning process of unstructured sparse large language models. Background Technology

[0002] With the development of deep learning, large language models have achieved remarkable success in many applications. For example, they are used in popular applications like chatbots for interactive dialogue with human users. During the dialogue, the large language model first understands the user's question with context (called the pre-filling stage), and then generates a response token by token (called the decoding stage). However, the power of large language models comes from their massive number of parameters and computational demands. For example, ChatGPT, developed by OpenAI and based on the GPT4 model, is currently one of the most popular chatbots. It has 1.8 trillion weight parameters, and generating a single token requires at least 540 TFLOPs of computational power, posing a significant challenge to the limited storage, memory bandwidth, and computing power of hardware devices.

[0003] Parameter sparsity for large oracle models is an effective method that can further accelerate them by reducing computational complexity and memory usage. Some algorithmic studies have also proposed other structured sparsities at different granularities. However, when examining the performance of large oracle models with different sparsity patterns on the WikiText2 dataset, focusing on two metrics—parameter reduction (higher is better) and perplexity (lower is better)—unstructured sparsity achieves a perplexity of 7.22 and a 60% parameter reduction. This observation demonstrates that unstructured sparsity outperforms structured sparsity in parameter reduction while maintaining model performance. However, fully leveraging the significant parameter reduction offered by unstructured sparsity is challenging because data reuse from its irregular sparsity patterns is difficult.

[0004] The reasoning process of the Big Prophecy model consists only of the Transformer's decoder module. First, × The (hidden dimension size) shape is generated through three linear operations. (Query) (key), The hidden dimension is first processed using a (value) tensor, followed by multi-head attention (MHA) computation, and layer normalization is applied to the hidden dimension. Finally, a feedforward network (FFN) expands the hidden dimension size to four times. and shrink it back Dimensions. The Big Prophet model will stack many decoder layers, and text generation is divided into a pre-filling stage and multiple consecutive generation stages. In the pre-filling stage, the first output token is output from the context according to the decoder order. Then, in the generation stage, each step uses the output token of the previous step as input to generate the next token. This process continues until certain conditions are met, such as generating an EOS token. The difference between these two stages is that, in the generation stage, its hidden state input is 1× Vectors, which tend to be computationally limited and have a large Unlike the pre-filling stage with matrix-matrix multiplication constraints, the generation stage tends to be memory-constrained and has vector-matrix multiplication constraints. On the other hand, the generation stage performs cross-attention, concatenating the K and V tensors generated in the current decoding step with the tensors from the pre-filling stage and previous generation stage steps. On-chip KV cache reuse ensures autoregressive properties and avoids recalculation. However, the KV cache has a long lifespan, lasting until the end of the session, which poses challenges for memory management.

[0005] Existing work does not perform well in supporting unstructured sparse large language models. Google proposed FlexTPU for unstructured sparse computation, which forms a Z-shaped pattern, called Z-mapping, by arranging non-zero values ​​consecutively in rows on a systolic array and then storing them consecutively from the top-left PE to the bottom-right PE. While this approach reduces the number of iterations, it only considers fully utilizing the processing elements (PEs) in space, while requiring more time for waiting. For example, a row in the array might come from three different rows of the original unstructured sparse matrix, which triples its input feed time. Furthermore, a separator needs to be inserted between different rows during the mapping process, which also results in PEs being idle and not fully filled with non-zero values. To implement these operations, the internal design of each PE becomes complex, embedding a four-state state machine for control, which introduces data path selection overhead. Technical issues

[0006] Existing technologies for unstructured sparse large language models place high demands on hardware storage, memory bandwidth, and computing power due to their massive number of parameters and computational requirements. Furthermore, their irregular sparsity patterns make data reuse difficult, and existing hardware and algorithms do not adequately support unstructured sparse large language models, resulting in inefficient memory management and computational performance. Technical solutions

[0007] Existing technologies address the challenges of unstructured sparse large language models. These models, with their massive number of parameters and computational demands, require significant hardware resources in terms of storage, memory bandwidth, and computing power. Furthermore, their irregular sparsity patterns hinder data reuse, and existing hardware and algorithms offer limited support for these models, resulting in poor memory management and computational efficiency. This invention provides an accelerator to accelerate the inference process of unstructured sparse large language models. It breaks away from the limitations of traditional systolic arrays that only reuse data along a single row or column, enabling cross-row and cross-column data reuse, thus alleviating hardware pressure. Moreover, it utilizes bitmasks to encode non-zero element positions, improving computational efficiency.

[0008] To achieve the above objectives, the present invention provides the following technical solution.

[0009] In a first aspect, the present invention provides an accelerator for accelerating the inference process of unstructured sparse large language models, comprising: a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnect bus module, a computing core module, and a router module;

[0010] The interconnect bus module is connected to the direct memory access module, the memory controller module, and the computing core module; the processor module is connected to the direct memory access module; the DDR memory module is connected to the memory controller module; and the router module is connected to the computing core module and adjacent router modules.

[0011] The computing core module obtains the data to be computed from the off-chip DDR memory module through the interconnect bus module;

[0012] The processor runs the compiler to generate instructions and weight parameters for the large language model and sends them to the DDR memory module through the direct memory access module; then, the computation core module retrieves the instructions and executes them one by one.

[0013] The computing core modules exchange data with each other through the router module to enable multi-core computation of nonlinear functions; the computing core modules obtain the data to be computed from the DDR memory module through the interconnect bus module.

[0014] The memory controller module processes memory read / write requests from the interconnect bus module and sends these requests to the DDR memory module.

[0015] As a further improvement of the present invention, the computing core module includes: an instruction control unit module, a bitmask decoder module, a diagonal data stream systolic array module, a post-processing module, an instruction cache module, a mask cache module, a weight cache module, a feature cache module, a result cache module, a hyperparameter cache module, and a KV cache module.

[0016] The processor module runs a software compiler to generate instructions for the sparse big oracle model and sends them to the DDR memory module via the direct memory access module; the instruction controller unit module retrieves instructions from the instruction cache module and prepares them for execution; the computation core module retrieves data from the weight cache module and the feature cache module and feeds the data to the diagonal data stream systolic array module for dense or sparse matrix computation; the computation results are saved to the result cache module and processed by the post-processing module; the processed results are then saved back to the DDR memory module, ready for the next round of computation or for use by other system components;

[0017] Instruction and control unit module: Used to control the behavior of various modules within the computing core module during operation;

[0018] Bitmask decoder module: used to decode the bitmask matrix of unstructured sparse large language model, translate the position information before sparsification, thereby obtaining the position of the position information before sparsification in the original matrix, and feed the translation result to the diagonal data stream systolic array module;

[0019] Diagonal data stream systolic array module: used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations on unstructured sparse large language models;

[0020] Post-processing module: Used to perform non-linear operations in large language models and special data transformation processes;

[0021] Instruction cache module: used to cache instructions during the model inference process for the instruction control unit module to access;

[0022] Mask caching module: used to cache the bitmask matrix of unstructured sparse large language models before compression, so that the bitmask decoder module can obtain it. It is updated by reading data from the DDR memory module in each round of calculation.

[0023] Weight caching module: used to cache the weight parameters after compression of unstructured sparse large language models, as the left matrix to be multiplied in the diagonal data stream systolic array;

[0024] Feature caching module: used to cache feature map data of user input for large language models, which is fed into the right matrix to be multiplied of the diagonal data stream systolic array module;

[0025] KV caching module: used to cache the K and V matrices during the generation phase of the large language model inference process, and used when the next word is generated;

[0026] Result caching module: used to cache part of the result after the diagonal data stream pulsating array multiplication, and to perform multiple rounds of matrix multiplication in a loop accumulation. After completion, it is saved to the off-chip DDR memory module or the on-chip KV cache module.

[0027] As a further improvement of the present invention, the diagonal data stream pulsating array module includes an input feed module, a non-zero loader module, a processing element module, a cross switch module, a result merging module, and a result accumulation module;

[0028] The input feed module is connected to the first-level crossbar switch module; the non-zero loader module is connected from the top to each column of data lines of the processing element (PE) module; the crossbar switch of each level is connected to the processing element (PE) module on its diagonal, and to the crossbar switch modules of adjacent preceding and following levels; each processing element module is connected to the crossbar switch on its diagonal, and to the processing element modules above and below it in the same column; the result merging module is connected to the processing element module in the last row, and to the result accumulation module.

[0029] As a further improvement of the present invention, the input feed module acquires data and arranges it into the staggered form required by the array through streaming; the non-zero loader module, in conjunction with the bitmask decoder module, matches the weights and the mask and feeds them into the array together; wherein, the input feature data enters from the upper left corner and is fed down each time; while the weights flow in from the top and are fixedly stored inside the processing element module; the processing element module obtains the required feature data from the cross switch module according to the ID number of the weight, performs multiplication and accumulation operations and decides whether to accumulate the output result of the processing element module; the column result of each processing element module is output to the result merging module below, and cross-column accumulation is performed in the result accumulation module, and then the partial sum is accumulated in multiple rounds to obtain the final result.

[0030] As a further improvement of the present invention, the diagonal data stream pulsating array module performs dense matrix multiplication using a diagonal data stream, as specifically shown below:

[0031] Assuming the systolic array performs matrix multiplication = ;

[0032] in: ∈R × This is the output matrix for each round; ∈R × This is the input matrix for each round; ∈R × This is the weight matrix for each round; It is the array size; It is the size of the input line block;

[0033] Input feed module acquisition The length of the input vector is calculated by moving it from the top left corner along... Direction flows into the array; weight tensors simultaneously along The flow direction is vertical, from top to bottom; in addition, the input to The weight of column +1 and the first The column has a periodic delay; Parallel cross switch module and The modules are orthogonally placed, and each processing element module is connected to its local cross switch module to obtain the input tensor.

[0034] When the input vector reaches the th When there are multiple crossbar switch modules, each connected processing element module executes along... Direction input and along The weights of the directions are multiplied; then, the element processing module forwards the partial sum to the elements along the direction. The next processing element module in the direction performs the accumulation; the bottom processing element module stores PSUM into the output buffer; where multiplication occurs along the diagonal. Directional assembly line;

[0035] The diagonal data stream pulsating array module performs sparse matrix multiplication using a diagonal data stream; because It is sparse, and the bitmask uses 0 and 1 to represent zero elements and non-zero elements respectively; the bitmask decoder module converts the bitmask into (ID, CAS, MEG).

[0036] Where ID is a value in the original sparse matrix. 2 The bit index helps the processing element module select the appropriate input for calculation; CAS is a 1-bit concatenation indicator that indicates whether the current processing element module should concatenate the PSUM with the PSUM of the previous processing element module for accumulation; MEG is a 1-bit merge indicator used to merge PSUM across columns.

[0037] The non-zero loader module feeds non-zero weights into arrays with corresponding (ID, CAS, MEG);

[0038] When the input is along During directional flow, the processing element module performs multiplication based on the weight's ID; the processing element module selects arbitrary values ​​from the local cross switches; subsequently, PSUM flows along... The direction is forwarded to the next processing element module, where CAS indicates whether accumulation should be performed; it is checked whether the PSUM output from the bottom processing element module of MEG needs to be merged.

[0039] As a further improvement of the present invention, the bitmask decoder module includes: a bitmask buffer module, a collection module, a decoder FIFO module, and a splicing module;

[0040] The bitmask buffer module is connected to the collection module; the collection module is connected to the decoder FIFO module; and the decoder FIFO module is connected to the splicing module.

[0041] The collection module is responsible for reading the original bitmask from the bitmask buffer module, and through the aggregation operation, it concentrates the scattered bit information into the low-order part, while setting the high-order part to 0. The processed data is then temporarily stored in the decoder FIFO buffer, awaiting further processing. The splicing module determines when to read new data from the FIFO based on the weighted valid signal and splices it with the existing data.

[0042] As a further improvement of the present invention, the bitmask decoder module translates the bitmask into indicators to guide the correct sparse computation on the diagonal data stream systolic array; the input is the first row to the Nth row, which is a sparse bitmask; then it is aggregated into a compact new row;

[0043] The collection module obtains the original bitmask from the bitmask buffer. After multiple rounds of continuous aggregation, the bitmasks are finally aggregated into the low bits, while the high bits are 0, and are stored in a decoder FIFO buffer.

[0044] The splicing module retrieves a new row from the IFFO based on the weighted valid signal. If the new row is not long enough for output, it reads new data from the FIFO. After acquiring the data, it determines whether the data meets the new row length requirement. If it does, the data is output. The remaining data is then continuously and seamlessly spliced ​​together and output.

[0045] As a further improvement of the present invention, the post-processing module includes: an activation function module, a SoftMax module, a residual addition module, a layer normalization module, a transpose module, and a rearrangement module;

[0046] The activation function module is used to execute activation functions during the inference process of large language models;

[0047] The SoftMax module is used to map the value range of each node's data to [0, 1], and constrain the sum of the output values ​​of each output node to 1;

[0048] The residual addition module is used to add the feature data from the previous step to the feature data of the current step during the inference process;

[0049] The layer normalization module performs a normalization operation on the input data, constraining the data values ​​to a certain range;

[0050] The transpose module is used to rotate the row and column orientations of a 32x32 matrix and then save the rotated matrix to the DDR memory module;

[0051] The rearrangement module is used to rearrange the skipping data into an ordered form, and then save the entire block to the DDR memory module.

[0052] As a further improvement of the present invention, the KV cache module includes: an overflow module and an overload module;

[0053] The overflow module is used to manage key-value cache overflows and implement a last-in-first-out (LIFO) strategy.

[0054] The overload module is used to split the on-chip memory into multiple blocks, each block storing a fixed number of labeled KV tensors.

[0055] As a further improvement of the present invention, the overflow module is used to manage KV cache overflow and implement a last-in-first-out (LIFO) strategy, including:

[0056] When the KV cache grows and exceeds the capacity of the on-chip memory, the overflow module is responsible for transferring these excess KV tensors to the off-chip DDR memory module.

[0057] During the cross-attention phase of the decoding stage, since the KV cache data needs to be accessed frequently, the overflow module prioritizes retaining the KV tensors of the earlier steps in on-chip memory. If there is insufficient space, the KV tensors of the later steps will overflow.

[0058] As a further improvement of the present invention, the overload module is used to divide the on-chip memory into multiple blocks, each block storing a fixed number of labeled KV tensors, including:

[0059] On-chip memory split into Each block stores [number of blocks]. A labeled KV tensor; the previous ×( 1) The KV tensor of each label is fixed to the previous one. One memory block; will always remain on-chip to reduce off-chip memory traffic;

[0060] The newly generated KV tensor will be written to the last memory block, accumulating the collected data. A KV tensor with a label, the memory block will overflow to off-chip memory;

[0061] Among them, the selection It is equal to the size of the shrinking array.

[0062] Secondly, the present invention provides a method for operating an accelerator to accelerate the inference process of unstructured sparse large-scale language models, comprising:

[0063] The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transfer the required data to the computing core module;

[0064] The processor module converts the large language model described by the high-level language into specific instructions and weight parameters, and stores this information in the DDR memory module through the direct memory access module;

[0065] The computing core module is started, and it reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module and work together to complete complex nonlinear function calculations.

[0066] The memory controller module processes memory read / write requests from the interconnect bus module and sends the requests to the DDR memory module, ensuring that memory access requests from each module are processed. Beneficial effects

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] This invention breaks through the limitations of traditional systolic arrays that only reuse data along a single row or column by working collaboratively with the interconnect bus module, computing core module, and router module, achieving cross-row and cross-column data reuse. Furthermore, the diagonal data flow pattern employed in this invention significantly improves data reuse rate, reduces unnecessary memory accesses, and thus lowers memory bandwidth requirements. By using bitmasks to encode non-zero element positions, this invention can efficiently process sparse matrices, reducing invalid calculations on zero elements and improving computational efficiency. Simultaneously, this sparse encoding scheme also reduces off-chip memory traffic and lowers memory access latency. The computing core module in this invention obtains data to be computed from the DDR memory module through the interconnect bus module and executes instructions one by one. The computing core modules exchange data through the router module, enabling multi-core computation of nonlinear functions and further improving computational efficiency. The coordinated operation of the memory controller module and the DDR memory module allows for efficient processing of memory read and write requests. Additionally, this invention fixes the early KV cache tensor in on-chip memory, reducing memory reloading traffic and improving memory bandwidth utilization. This strategy also reduces the number of off-chip memory accesses, lowering system power consumption. Therefore, this invention can more effectively utilize hardware resources, including processors, memory, and computing cores. This reduces the requirements for hardware devices, enabling the inference process of unstructured sparse large language models to run efficiently on a wider range of hardware platforms. Simultaneously, the accelerator design of this invention has good scalability and flexibility. By adjusting the number and configuration of computing core modules, it can adapt to unstructured sparse large language models of different sizes and complexities. Attached Figure Description

[0069] Figure 1 is a schematic diagram of the overall structure of an accelerator for accelerating the reasoning process of unstructured sparse large language models according to the present invention.

[0070] Figure 2 is a schematic diagram of the pulsating array module structure of the angular data stream of an accelerator for accelerating the inference process of unstructured sparse large language models according to the present invention.

[0071] Figure 3 is a schematic diagram of the circuit structure of the processing element (PE) module and the cross switch module of an accelerator for accelerating the reasoning process of unstructured sparse large language models according to the present invention.

[0072] Figure 4 is a schematic diagram of the result merging module and result accumulation module of an accelerator for accelerating the reasoning process of unstructured sparse large language models according to the present invention.

[0073] Figure 5 is a schematic diagram of the bitmask decoder module structure of an accelerator for accelerating the inference process of unstructured sparse large language models according to the present invention.

[0074] Figure 6 is a schematic diagram of the overflow module and overload module structure of an accelerator for accelerating the reasoning process of unstructured sparse large language models according to the present invention. Embodiments of the present invention

[0075] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0077] Existing technologies address the challenges of unstructured sparse large language models, which have a large number of parameters and computational requirements, demanding high levels of hardware storage, memory bandwidth, and computing power. Furthermore, their irregular sparse patterns make data reuse difficult, and existing hardware and algorithms do not provide adequate support for unstructured sparse large language models, resulting in poor memory management and computational efficiency.

[0078] This invention provides an accelerator for accelerating the inference process of unstructured sparse large language models, as shown in Figure 1. The accelerator includes a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnect bus module, a computing core module, and a router module.

[0079] The processor module is interconnected with the direct memory access module; the memory controller module is interconnected with the DDR memory module; the computing core module is interconnected with the router module; the interconnect bus module is interconnected with the direct memory access module, the memory controller module, and the computing core module. Specifically: the interconnect bus module is connected to the direct memory access module; the interconnect bus module is connected to the memory controller module; and the interconnect bus module is connected to the computing core module.

[0080] The number of memory controller modules, DDR memory modules, computing core modules, and router modules is 4.

[0081] During the inference process of the sparse big oracle model, the computation core module establishes a communication link with the external DDR memory module through the interconnect bus module, ensuring that the data to be computed can be transferred to the computation core module. The processor is responsible for running the compiler during this process, generating corresponding instructions and weight parameters for the large language model, and seamlessly sending this information to the DDR memory module through the direct memory access module. Once ready, the computation core module is started, reads the instructions from the DDR memory module, and executes these instructions one by one to advance the model's inference process. To improve computational efficiency and handle complex operations, the computation core modules can exchange data through the router module. This design not only supports linear computation but also enables multi-core parallel computation of nonlinear functions, thereby significantly improving the speed and performance of model inference.

[0082] The processor module runs a software compiler that decomposes unstructured, sparse, large language models into basic operators and procedures, and generates hardware-executable machine code based on the instruction set definition. When processing weight parameters, the processor module arranges the weight parameters in order according to node addresses. This arrangement helps improve memory access efficiency and reduce data transfer time, thereby improving overall computational performance. For each computing core module, the processor generates a corresponding hexadecimal txt file. These files contain all the instructions and operands required for execution by that core, forming the basis for the computing core's model inference.

[0083] The Direct Memory Access (DMI) module handles data load and save requests from the processor module, enabling direct data transfer between the DDR memory module and the interconnect bus module. This design allows data to move rapidly between memory and the processor without central processing unit intervention, significantly freeing up processor resources and improving the overall performance of the accelerator.

[0084] The memory controller module is used to process memory read and write requests from the interconnect bus module and convert these requests from the AIX protocol (Advanced eXtensible Interface, AXI bus protocol) to the DDR protocol (Double Data Rate Synchronous Dynamic Random Access Memory). This conversion process is a key step in ensuring that data can be transferred correctly and efficiently between other modules and DDR memory.

[0085] The memory controller module first receives memory read / write requests in AIX protocol format from the interconnect bus module. It then parses these requests, extracting the request type, address information, and data content. The parsed AIX protocol requests are converted to DDR protocol requests. The converted DDR protocol requests are sent to the DDR memory module for the actual read / write operation. After completing the memory access operation, the memory controller module receives the response from the DDR memory module, converts it back from DDR protocol to AIX protocol, and finally returns the processing result to the interconnect bus module that initiated the request.

[0086] The interconnect bus module is used to provide interconnection between the direct memory access module, the memory controller module, and the computing core module, enabling data transmission and arbitration of multi-port access between these modules.

[0087] The interconnect bus module accurately and quickly routes data from source to destination based on requests from various modules. When multiple modules attempt to access the same resource simultaneously, the interconnect bus module handles access arbitration, ensuring orderly and fair allocation of access rights and preventing conflicts and deadlocks. The interconnect bus module also features bandwidth management capabilities, dynamically adjusting bandwidth allocation across data channels based on real-time system load to achieve optimal overall performance. Furthermore, it can detect and trigger error recovery operations during data transmission, ensuring data transmission reliability and system stability.

[0088] Router modules are used for data synchronization between different computing core modules. Each computing core module is equipped with one router module. A router module connects not only to its own computing core module but also to connected routers. Each computing core module has a dedicated router module, which facilitates efficient data communication and synchronization between different computing core modules, thereby enhancing the overall computing power and data processing efficiency of the system.

[0089] The router module receives data from its corresponding computing core module and forwards this data to other computing core modules that require this information. In a multi-core parallel computing environment, synchronization operations are sometimes necessary between different computing core modules to ensure the correctness and consistency of computations. The router module supports various synchronization mechanisms, including but not limited to barrier synchronization and event synchronization.

[0090] Each router module is connected not only to its corresponding computing core module but also to other adjacent router modules, forming a network. This network design can adopt various topologies such as ring, tree, and mesh. Secondly, the router modules possess flow control and congestion avoidance mechanisms, enabling them to monitor network load during data transmission and take measures to prevent data congestion and packet loss, ensuring stable network operation. Furthermore, fault tolerance is also considered, meaning that even if some routers or computing core modules fail, the accelerator can continue operating or resume normal operation after the fault is recovered.

[0091] The computational core module is used to execute the inference process of the unstructured sparse large language model. It receives the start control signal from the processor module, then reads the instructions and weight parameters from the DDR memory module. During the inference process, some intermediate results are also written to the DDR memory module until the entire inference is completed and the final result is stored back to the DDR memory module. At this point, the processor module can read the inference result through the direct memory access module.

[0092] The computational core modules are responsible for executing the instruction sequences compiled and issued by the processor modules during the inference process of the sparse large oracle model. Each computational core module has powerful parallel computing capabilities and a highly optimized algorithm library to handle the complex computational requirements during the inference process of large language models.

[0093] Before the inference process begins, the computation core module is in a standby state. Upon receiving a start control signal from the processor module, the computation core module begins reading instructions and corresponding weight parameters from the DDR memory module. Then, the computation core module executes various arithmetic and logical operations step-by-step according to the read instruction sequence. During execution, the computation core module also shares intermediate results with other modules to support more complex calculations and algorithms. At different stages of the inference process, the computation core module writes some intermediate results back to the DDR memory module. This helps distribute the pressure of data processing and ensures data consistency and availability during long-cycle inference. When all instructions have been executed, the computation core module stores the final inference result in a designated area of ​​the DDR memory module. At this point, the computation core module sends a completion signal to the processor module, indicating that the inference process has ended. After receiving the completion signal, the processor module can read the inference result from the DDR memory module through the direct memory access module. These results can then be used for subsequent processing tasks, such as result analysis and decision-making.

[0094] The core computational modules include: instruction control unit module, bitmask decoder module, diagonal data stream systolic array module, post-processing module, instruction cache module, mask cache module, weight cache module, feature cache module, result cache module, hyperparameter cache module, and KV cache module.

[0095] As shown in Figure 1, the instruction controller unit module is interconnected with the instruction cache module, the diagonal data stream systolic array module, and the post-processing module; the bitmask decoder module is interconnected with the mask cache module and the diagonal data stream systolic array module; the diagonal data stream systolic array module is interconnected with the bitmask decoder module, the weight cache module, the feature cache module, the KV cache module, and the result cache module; and the post-processing module is interconnected with the result cache module and the hyperparameter cache module.

[0096] The processor module runs a software compiler to generate instructions for the sparse big oracle model and sends them to the DDR memory module via the direct memory access module. The instruction controller unit module fetches instructions from the instruction cache module and prepares them for execution. The computation core module retrieves data from the weight cache module and the feature cache module, and this data is fed to the diagonal data stream systolic array module for dense or sparse matrix computation. The bitmask decoder module performs necessary translation work for unstructured sparseness during this process. The computation results are first saved to the result cache module. Then, the computation results are further processed by the post-processing module, such as performing nonlinear operations. The processed results are then saved back to the DDR memory module, ready for the next round of computation or for use by other system components. The above process is executed iteratively in multiple rounds according to the specific needs of the model. Throughout the inference process, the system reduces data access latency and improves computational efficiency by optimizing data paths and using efficient cache management strategies, thereby accelerating the inference process of the big oracle model. The hyperparameter cache module is used to store and manage various hyperparameters used in the model inference process. When performing its functions, the post-processing module needs to refer to certain hyperparameters in the hyperparameter cache module to adjust its processing strategy.

[0097] The processor module runs a software compiler to generate instructions for the sparse big oracle model, which are then sent to the DDR memory module via the direct memory access module. The computation core module executes the inference process of the sparse big oracle model, retrieving data from the weight cache and feature cache modules each time and feeding it to the diagonal data stream systolic array module for dense or sparse matrix computation. The bitmask decoder module translates the unstructured sparse data. The output is saved to the result cache module, and then proceeds to the post-processing module for nonlinear operations. Finally, the output is sent to the DDR memory module. This process is repeated multiple times to accelerate the inference process of the big oracle model.

[0098] Instruction Control Unit Module: This module controls the behavior of each module within the computing core module during operation. In each computation cycle, it fetches multiple instructions from the instruction cache module to form an instruction block, which is then decoded into the values ​​of various registers. At the beginning of each computation cycle, the instruction control unit module fetches an instruction block from the instruction cache module. This instruction block contains several instructions that together form the operation guide for the following computation cycle. After fetching the instruction block, the instruction control unit module decodes it, converting the instructions into operations for each sub-module within the computing core module, thereby controlling the specific behavior of each sub-module within the computing core module.

[0099] As shown in Figure 2, the diagonal data stream pulsating array module is used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations on unstructured sparse large language models.

[0100] The diagonal data stream pulsating array module includes an input feed module, a non-zero loader module, a processing element (PE) module, a cross switch module, a result merging module, and a result accumulation module.

[0101] The input feed module is connected to the first-level crossbar switch module in the upper left corner; the non-zero loader module is connected from the top to each column of data lines of the processing element (PE) module; the crossbar switch of each level is connected to the processing element (PE) module on its opposite diagonal, and to the crossbar switch modules of adjacent levels; each processing element (PE) module is connected to the crossbar switch on its diagonal, and to the adjacent processing element (PE) modules above and below in the same column; the result merging module is connected to the processing element (PE) module in the last row, and to the result accumulation module.

[0102] The diagonal data stream pulsating array module enables efficient data reuse for both dense and sparse computational modes. Data is acquired from the input feed module and then stream-sorted into the required staggered array format. A non-zero loader module, in conjunction with a bitmask decoder module, feeds weights and masks into the array. Input feature data enters from the top left corner and is fed downwards each time. Weights enter from the top and are permanently stored within the processing element (PE) modules. Each PE module, based on the weight's ID, retrieves the required feature data from the cross-switching module, performs multiplication and accumulation operations, and determines whether to add the output of the previous PE module. The column results from each PE module are output to the result merging module, where cross-column accumulation is performed in the result accumulation module. Multiple rounds of partial sum accumulation are then performed to obtain the final result.

[0103] When processing sparse matrices, the non-zero loader module and the bitmask decoder module work together to ensure that weights and masks are correctly fed into the array. Input feature data enters from the top left corner and is passed down sequentially, while weights flow in from the top and are permanently stored within the processing element (PE) modules. The PE modules are the core component of the systolic array; they retrieve the required feature data from the cross-switch module based on the weight IDs and perform multiplication and accumulation operations. Simultaneously, the PE modules are responsible for determining whether to accumulate the output of another PE module. The results from each PE module are output down to the result merging module, then accumulated across columns by the result accumulation module, and finally, through multiple rounds of accumulation, the final result is obtained. The diagonal data stream systolic array module, through its efficient input / output mechanism and collaboration between the PE modules, effectively handles both dense and sparse computational modes, improving computational efficiency and performance.

[0104] The specific representation of performing dense matrix multiplication (DMM) using a diagonal data stream is as follows:

[0105] Assuming the systolic array performs matrix multiplication = ;

[0106] in: ∈R × This is the output matrix for each round; ∈R × This is the input matrix for each round; ∈R × This is the weight matrix for each round; It is the array size; It is the size of the input line block.

[0107] As shown in Figure 2, at the beginning of each round of calculation, the input feed module obtains... The input vector of length, and simultaneously they are transferred from the top left corner along... The direction flows into the array. The weight tensor simultaneously moves along... The flow direction is vertical, from top to bottom. Additionally, the input... The weight of column +1 and the first The column has a periodic delay. Parallel cross switch module and Orthogonally placed, each processing element (PE) module is connected to its local cross-switch module to obtain the input tensor (if needed). When the input vector reaches the... When there are multiple cross switch modules, each connected processing element (PE) module executes along... Direction input and along The weights are multiplied along the direction. Then, the Element Processing (PE) module forwards the PSUM (Partial Sum) to the elements along the direction. The next processing element (PE) module in the direction performs the accumulation. Finally, the bottom processing element (PE) module stores PSUM into the output buffer. Additionally, the multiplication occurs diagonally. Directional assembly line.

[0108] For performing sparse matrix multiplication (SpMM) using a diagonal data stream. Because... It is sparse, supporting the design features of SpMM's systolic array, ensuring that corresponding elements are multiplied and accumulated, and keeping the Processing Element (PE) module as busy as possible. Before each round of computation, the bitmask uses 0s and 1s to represent zero and non-zero elements respectively. The bitmask decoder module converts the bitmask into (ID, CAS, MEG).

[0109] Where ID is a value in the original sparse matrix. 2 The bit index helps the processing element (PE) module select the appropriate input for computation; CAS is a 1-bit cascade indicator that indicates whether the current processing element (PE) module should cascade the PSUM with the PSUM of the previous processing element (PE) module for accumulation; MEG is a 1-bit merge indicator used to merge PSUM across columns.

[0110] The non-zero loader feeds non-zero weights into an array with corresponding (ID, CAS, MEG). As the input... During directional flow, each processing element (PE) module performs multiplication based on the weight's ID. Each PE module can select arbitrary values ​​from its local crossbar switch to provide sufficient bandwidth for the input and keep the PE module busy. Afterwards, PSUM flows along... The direction is forwarded to the next processing element (PE) module, where CAS indicates whether accumulation should be performed. This allows multiple compressed weight columns to be compressed into the same processing element (PE) module column and share the input. Finally, it is checked whether the PSUM output from the bottom processing element (PE) module needs to be merged into MEG.

[0111] As shown in Figure 3, the Processing Element (PE) module receives updated weights, ID numbers, and cascading enable information from the left branch and saves them to the weight cache module. The cascading control is used to manage the connection between the input and the MAC (Multiplier-Accumulator) multiplication accumulation, determining whether to accumulate a processing element (PE) module. The ID number is sent to the crossbar switch to select a desired data point from N data points. The crossbar switch module then feeds this data into the next round of computation via registers.

[0112] The cross switch module includes: a register group module and a selector group module.

[0113] Input data is connected to the register group module and the selector group module for subsequent processing.

[0114] For sparse computation, the crossbar switch module needs to broadcast the input vector and select the appropriate data based on a given ID number from the bitmask decoder module. Since each crossbar switch module is a selector group module with a large number of inputs, inserting registers makes it a multi-stage pipeline to optimize timing convergence. For intensive computation, since data reuse is fixed, the crossbar switch module can be bypassed and connected to a fixed port to save power consumption from input selection.

[0115] The Element Processing (PE) module includes: a multiplication and accumulation module, a PE weight buffer module, an input cascade control module, and an output register module.

[0116] The PE weight cache module is connected to the multiplication-accumulation module and the input cascade control module. The input cascade control module and the multiplication-accumulation module are connected to the output register module.

[0117] To accommodate unstructured sparse computation with variable-length cumulative sizes, a processing element (PE) module with reconfigurable input and partial sum (PSUM) dataflow control is employed. Each PE module is connected to its local crossbar switch module via a line of IDW (input data width) + ID + CAS bits to receive inputs, indices, and cascading indicators. To update the weights corresponding to each PE module, a counter is used to scan from 1 to... When the counter equals the index in the Processing Element (PE) module, the PE weight buffer module updates. The PE module reads the index [ID] number from the buffer and sends it to the local cross-switch module to process the input vector. The desired value is selected from the elements. The data is then fed to the multiply-accumulate module of the DSP48E2-based hard MAC-IP to perform the multiply-accumulate operation. Afterwards, the [CAS] flag is used to determine whether to take the PSUM value from the upper-level adjacent processing element (PE) module and toggle the connection path of the cascade chain. When [CAS] is valid, it retains a cascade chain; otherwise, it connects to a bypass, in which case the input P[j] is registered to MEG[j+1] and connected to the output P[j+1]. Because the MAC's PCIN is not used, the result is connected to the output P[0]. PSUM flows vertically from top to bottom, with the accumulation controlled by the cascade indicator. Multiple PSUMs can be output on a single PE column, in which case they are shifted to... The ports are output together at the bottom PE. Therefore, flexible sparsity rates can be supported by configuring the P[0:M-1] bit width and the number of registers. For example, =1, 2, 3, and 4 can support sparsity rates of 0%, 50%, 33%, and 25%, respectively.

[0118] As shown in Figure 4, the result merging module receives different results from the processing element (PE) module. It needs to merge the current highest value and the next lowest value by adding them together, then collect the data down to the lower bits, and finally align the rows according to the valid bit information of each data. The entire row of data is then saved in the result buffer module, which requires reading the result from the previous step and the new value to perform the same-bit accumulation.

[0119] The merging module, based on sparse encoding, allows non-zero values ​​originally from the same column to be arranged in two PEcols. Each PEcol in the array can be output. ×PSUM, for example, M=4, its index is the original column. When PEcol[0] outputs "1" and "2", and PEcol[1] outputs "2", "3", and "4", it means that P[2] and P[0] originally came from the same column "2" and need to be merged. Since PEcol[i] and PEcol[i+1] are one cycle apart, the previous data can be kept in the register. MEG indicates merging. Which of the entries. PEcol[i] sends the lower-order P[0] to the higher-order P[MEG] of PEcol[i+1], matching the width of the PSUM buffer. MEG is collected and compared hierarchically from local to global. A valid signal enables the write address of each port, and elements from different cycles are temporarily cached in distributed RAM to align the entire row.

[0120] The result accumulation module is used to align the result data, send it to the multi-round accumulator module, and write it to a specified address in the PSUM buffer. To avoid collisions, the buffer is designed as a double ping-pong structure. For intensive computations, merging is not required, so only the P[0] port of each PEcol is used.

[0121] As shown in Figure 5, the bitmask decoder module is used to decode the bitmask matrix of the unstructured sparse large language model, translating the positional information before sparsity to obtain their positions in the original matrix. It also adds three registers—index value, concatenation value, and fusion value—to each weight, which are then fed together into the diagonal data stream systolic array module to control the behavior of each PE calculation in the element processing (PE) module.

[0122] The bitmask decoder module receives a sparse bitmask matrix, and its task is to decode these bitmasks to recover the position information of the non-zero elements in the original data. This is especially important for processing large sparse matrices. Using the decoded bitmask information, the module controls when and how each PE in the processing element (PE) module performs calculations and processes its input data. This control mechanism ensures efficient processing of sparse matrices.

[0123] The bitmask decoder module includes: a bitmask buffer module, a collection module, a decoder FIFO module, and a splicing module.

[0124] The bitmask buffer module is connected to the collection module. The collection module is connected to the decoder FIFO module, and the decoder FIFO module is connected to the splicing module.

[0125] In the sparse matrix, the position of each non-zero element is encoded using a 1-bit mask. The bitmask decoder module is used to convert the binary mask into (ID, CAS, MEG) to control the behavior of the PE. Since each bitmask has... Information about the line.

[0126] First, collect the number of the first non-zero elements in each row to obtain the ID.

[0127] Aligned non-zero elements are evenly distributed among different processing element (PE) modules, so MEG is set to 1 when non-zero elements are distributed among different groups of processing element (PE) modules.

[0128] There are two criteria for setting CAS to 0: the first non-zero element in each row; or MEG is set to 1; otherwise, CAS is set to 1.

[0129] The collection module reads a bitmask of width [0:N-1] from the bitmask buffer module in each computation cycle. The tree-structured decoder finds the position of 1 through several levels, i.e., in-4→out-4, in-8→out-8, up to in-N→out-N, as shown in Figure 5. All non-zero elements are arranged in big-endian format and stored in the decoder FIFO module.

[0130] The splicing module uses a counter and a shift register to retrieve aligned non-zero elements from the decoder FIFO module. When the counter indicates that there are enough non-zero elements, they are output directly. Otherwise, new items are read from the decoder FIFO module and concatenated into the shift register.

[0131] Therefore, the collection module is responsible for reading the original bitmask from the bitmask buffer and, through a series of aggregation operations, concentrating the scattered bit information into the low-order bits while setting the high-order bits to 0. This processed data is then temporarily stored in the decoder's FIFO buffer, awaiting further processing. The concatenation module determines when to read new data from the FIFO based on the weighted valid signal and concatenates it with existing data. This process involves determining the length of the new data and how to temporarily store any insufficient data in a shift register for seamless concatenation later.

[0132] The bitmask decoder module translates the bitmask into ID (index), CAS (cascade), and MEG (merge) indicators to guide correct sparse computation on the diagonal data stream systolic array. The input consists of rows 1 to NT, which is a sparse bitmask. These are then aggregated into compact new rows. The collection module retrieves the original bitmask from the bitmask buffer, and after multiple rounds of aggregation, finally gathers it to the leftmost least significant bit, while the most significant bits are set to 0, and saves it to a decoder FIFO buffer. The concatenation module on the right retrieves new rows from the FIFO based on the weighted valid signal. If the new row is not long enough for output, it reads new data from the FIFO, checks the count against a counter to determine if it's long enough, outputs it, and saves the remaining data to a shift register. This continuous, seamless concatenation process is then performed.

[0133] Post-processing module: Used to perform non-linear operations in large language models and special data transformation processes.

[0134] The post-processing module includes: activation function module, SoftMax module, residual addition module, layer normalization module, transpose module, and rearrangement module.

[0135] The residual summing module and the layer normalization module are connected. The remaining modules are connected in parallel to the input and output ports.

[0136] The activation function module executes activation functions during the inference process of the large language model, including the implementation of EXP and GELU functions; the SoftMax module maps the value range of each node's data to [0, 1] and constrains the sum of the output values ​​of each output node to 1; the residual addition module adds the feature data from the previous step to the feature data of the current step during the inference process; the layer normalization module performs normalization operations on the input data, constraining the data values ​​to a certain range; the transpose module rotates a 32x32 matrix in the direction of its rows and columns, and then saves the rotated matrix to the DDR memory module; the rearrangement module rearranges the skipped data into an ordered form, and then saves the entire block to the DDR memory module.

[0137] The post-processing module is responsible for performing nonlinear operations and special data transformation processes. In addition to nonlinear operations, the post-processing module can also perform special data transformation processes, such as matrix transpose, matrix factorization, and feature normalization. These operations are crucial for improving the accuracy and performance of the model.

[0138] The instruction cache module caches instructions during model inference for the instruction control unit module to access. Whenever an instruction is about to be fully read, the corresponding data is retrieved from the DDR memory module to update the cache.

[0139] The instruction cache module caches a portion of the instructions used during model inference, enabling them to be quickly provided to the instruction control unit module. When the instructions in the cache are about to be read out, it automatically reads the corresponding data from the DDR memory module to update the cache, ensuring instruction continuity and correctness. The main function of the instruction cache module is to reduce the number of DDR memory accesses, thereby reducing system power consumption and latency. Furthermore, the instruction cache module can improve instruction access speed, thus enhancing processor execution efficiency and performance.

[0140] The bitmask caching module is used to cache the bitmask matrix of an unstructured sparse large language model before it is compressed, so that the bitmask decoder module can access it. It is updated by reading data from the DDR memory module in each round of calculation.

[0141] The mask caching module caches a portion of the uncompressed bitmask matrix for rapid delivery to the bitmask decoder module. In each round of calculation, the mask caching module reads the corresponding data from the DDR memory module to update the cache, ensuring the continuity and correctness of the bitmask matrix. The primary function of the mask caching module is to reduce the number of accesses to the DDR memory module, thereby reducing system power consumption and latency. Furthermore, the mask caching module can improve the access speed of the bitmask matrix, thus enhancing the execution efficiency and performance of the bitmask decoder module.

[0142] The weight caching module is used to cache the weight parameters after compression of an unstructured sparse large language model, serving as the left matrix to be multiplied in the diagonal data stream systolic array.

[0143] The feature caching module is used to cache the feature map data of the large language model user input, which is fed into the right matrix to be multiplied of the diagonal data stream systolic array module.

[0144] The result caching module is used to cache the partial sums and results after multiplication of the diagonal data stream systolic array and to perform cyclic accumulation of multiple rounds of matrix multiplication. After completion, the results are saved to an off-chip DDR memory module or an on-chip KV cache module.

[0145] As shown in Figure 6, the KV caching module is used to cache the K and V matrices during the generation phase of the large language model inference process, and is used when the next word is generated.

[0146] The KV cache module includes an overflow module and an overload module.

[0147] The overflow module and overload module are connected to the DDR memory module.

[0148] The overflow module addresses the issue that, as the key-value (KV) cache grows continuously, limited on-chip memory cannot always accommodate all KV caches. KV caches must be overflowed into off-chip memory and reloaded for computation in the next decoding step, generating significant off-chip memory traffic during cross-attention in the decoding phase. Considering that each KV tensor remains active until the end of the session, KV tensors from earlier steps should always be retained in on-chip memory, overflowing later KV tensors if space is insufficient. Therefore, a simple yet efficient on-chip KV cache allocation scheme is used: early KV tensors are pinned to on-chip memory, and later KV tensors are overflowed using a Last-In-First-Out (LIFO) strategy. This overflow process ensures that the KV cache is fully utilized. The block size of the marked KV tensor overflows the KV tensor.

[0149] The reload module is achieved by splitting the on-chip memory into Each block stores [number of blocks]. A KV tensor with labels. The previous ×( 1) The KV tensors of each label are fixed to the previous one. One memory block. These will always remain on-chip to reduce off-chip memory traffic. In addition, the most recently generated KV tensor will be written to the last memory block once it has been accumulated. A tagged KV tensor will cause this memory block to overflow into off-chip memory. (Select) The size of the shrinking array is equal to that of the array so that block reloading can be well overlapped in computation. Merged KV cache memory accesses. Due to CAS latency, off-chip memory access requests for small vectors would incur significant latency overhead. Therefore, reloading is performed, loading KV tensors in blocks to avoid performance degradation.

[0150] The overflow and reload modules use a Last-In-First-Out (LIFO) approach to handle block overflows and batch reloads. The left side shows the overflow rules: the K vector is cached, and only when 32 vectors are accumulated will the entire block overflow into off-chip memory. The right side shows the next time an earlier block needs to be used, allowing multiple blocks to be reloaded at once.

[0151] In summary, in traditional systolic arrays, data can only be reused along a single row or column. However, this invention employs a diagonal dataflow systolic array, allowing data to be reused along the diagonal direction, achieving cross-row and cross-column data reuse, thereby improving computational efficiency and hardware utilization. Furthermore, the area overhead of the diagonal dataflow systolic array is very small, as it only requires adding a small amount of logic and storage resources to the existing systolic array.

[0152] For the pruned LLM parameters, this invention employs bitmasks to encode the positions of non-zero elements, thereby minimizing off-chip memory traffic. Furthermore, decoding via a bitmask decoder module allows for rapid acquisition of detailed value indices and cumulative metrics at runtime, thus improving computational efficiency and hardware utilization.

[0153] Finally, this invention employs a method that fixes early KV cache tensors in on-chip memory to reduce memory reload traffic. Furthermore, it utilizes a method of merging memory accesses to improve memory bandwidth utilization. Its advantage lies in its ability to effectively reduce the number of off-chip memory accesses, thereby lowering memory latency and power consumption.

[0154] It should be noted that the number of memory controller module, DDR memory module, computing core module and router module in this invention is 4, but the number can be increased or decreased according to actual needs.

[0155] The second objective of this invention is to provide a method for operating an accelerator to accelerate the inference process of unstructured sparse large-scale language models, comprising:

[0156] The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transfer the required data to the computing core module;

[0157] The processor module converts the large language model described by the high-level language into specific instructions and weight parameters, and stores the information in the DDR memory module through the direct memory access module;

[0158] The computing core module is started, and it reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module and work together to complete complex nonlinear function calculations.

[0159] The memory controller module processes memory read / write requests from the interconnect bus module and sends the requests to the DDR memory module, ensuring that memory access requests from each module are processed.

[0160] Many embodiments and applications beyond the examples provided will be apparent to those skilled in the art upon reading the foregoing description. Therefore, the scope of this teaching should not be determined by reference to the foregoing description, but rather by reference to the foregoing claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the applicant has not considered that subject matter as part of the disclosed inventive subject matter.

[0161] The above content provides a further detailed description of the present invention. It should not be construed that the specific embodiments of the present invention are limited to this. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention as defined by the submitted claims.

Claims

1. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models, characterized in that, include: Processor module, direct memory access module, memory controller module, DDR memory module, interconnect bus module, computing core module and router module; The interconnect bus module is connected to the direct memory access module, the memory controller module, and the computing core module; the processor module is connected to the direct memory access module; the DDR memory module is connected to the memory controller module; and the router module is connected to the computing core module and adjacent router modules. The computing core module obtains the data to be computed from the off-chip DDR memory module through the interconnect bus module; The processor runs the compiler to generate instructions and weight parameters for the large language model and sends them to the DDR memory module through the direct memory access module. Then, the core computing module receives instructions and executes them one by one; The computing core modules exchange data with each other through the router module to enable multi-core computation of nonlinear functions; the computing core modules obtain the data to be computed from the DDR memory module through the interconnect bus module. The memory controller module processes memory read / write requests from the interconnect bus module and sends these requests to the DDR memory module.

2. An accelerator for accelerating the inference process of unstructured sparse large-scale language models according to claim 1, characterized in that, The core computing module includes: an instruction control unit module, a bitmask decoder module, a diagonal data stream systolic array module, a post-processing module, an instruction cache module, a mask cache module, a weight cache module, a feature cache module, a result cache module, a hyperparameter cache module, and a KV cache module. The processor module runs a software compiler to generate instructions for the sparse big oracle model and sends them to the DDR memory module via the direct memory access module; the instruction controller unit module retrieves instructions from the instruction cache module and prepares them for execution; the computation core module retrieves data from the weight cache module and the feature cache module and feeds the data to the diagonal data stream systolic array module for dense or sparse matrix computation; the computation results are saved to the result cache module and processed by the post-processing module; the processed results are then saved back to the DDR memory module, ready for the next round of computation or for use by other system components; Instruction and control unit module: Used to control the behavior of various modules within the computing core module during operation; Bitmask decoder module: used to decode the bitmask matrix of unstructured sparse large language model, translate the position information before sparsification, thereby obtaining the position of the position information before sparsification in the original matrix, and feed the translation result to the diagonal data stream systolic array module; Diagonal data stream systolic array module: used to perform sparse-dense matrix multiplication and dense-dense matrix multiplication operations on unstructured sparse large language models; Post-processing module: Used to perform non-linear operations in large language models and special data transformation processes; Instruction cache module: used to cache instructions during the model inference process for the instruction control unit module to access; Mask caching module: used to cache the bitmask matrix of unstructured sparse large language models before compression, so that the bitmask decoder module can obtain it. It is updated by reading data from the DDR memory module in each round of calculation. Weight caching module: used to cache the weight parameters after compression of unstructured sparse large language models, as the left matrix to be multiplied in the diagonal data stream systolic array; Feature caching module: used to cache feature map data of user input for large language models, which is fed into the right matrix to be multiplied of the diagonal data stream systolic array module; KV caching module: used to cache the K and V matrices during the generation phase of the large language model inference process, and used when the next word is generated; Result caching module: used to cache part of the result after the diagonal data stream pulsating array multiplication, and to perform multiple rounds of matrix multiplication in a loop accumulation. After completion, it is saved to the off-chip DDR memory module or the on-chip KV cache module.

3. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 2, characterized in that, The diagonal data stream pulsating array module includes an input feed module, a non-zero loader module, a processing element module, a cross switch module, a result merging module, and a result accumulation module. The input feed module is connected to the first-level cross switch module; the non-zero loader module is connected from the top to each column of data lines of the processing element module; the cross switch of each level is connected to the processing element module on its diagonal, and to the cross switch modules of adjacent preceding and following levels; each processing element module is connected to the cross switch on its diagonal, and to the adjacent processing element modules above and below in the same column; the result merging module is connected to the processing element module in the last row, and to the result accumulation module.

4. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 3, characterized in that, The input feed module acquires data and arranges it into the staggered form required by the array through streaming. The non-zero loader module, in conjunction with the bitmask decoder module, matches the weights and masks and feeds them into the array. The input feature data enters from the top left corner and is fed downwards each time. The weights flow in from the top and are fixedly stored inside the processing element module. The processing element module obtains the required feature data from the cross switch module according to the ID number of the weights, performs multiplication and accumulation operations, and decides whether to accumulate the output result of the processing element module. The column result of each processing element module is output to the result merging module below, and cross-column accumulation is performed in the result accumulation module. Then, the partial sums are accumulated in multiple rounds to obtain the final result.

5. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 3, characterized in that, The diagonal data stream pulsating array module performs dense matrix multiplication using a diagonal data stream, as specifically shown below: Assume the pulsating array performs matrix multiplication Y = XWT; Where: Y∈RM×N is the output matrix for each round; X∈RM×N is the input matrix for each round; W∈RN×N is the weight matrix for each round; N is the array size; M is the size of the input row block; The input feed module obtains an input vector of length MN and flows it into the array from the top left corner along the k direction; the weight tensor simultaneously flows vertically from top to bottom along the j direction; in addition, the weights input to column i+1 have a periodic delay with column i; N parallel cross switch modules are placed orthogonally to the k direction, and each processing element module is connected to its local cross switch module to obtain the input tensor. When the input vector reaches the k-th cross-switch module, each connected processing element module performs a multiplication between the input along the k-direction and the weights along the j-direction; then, the processing element module forwards the partial sum to the next processing element module along the j-direction for accumulation; the bottom processing element module stores the PSUM into the output buffer; where the multiplication is pipelined along the diagonal k-direction. The diagonal data stream systolic array module performs sparse matrix multiplication using a diagonal data stream; since W is sparse, the bit mask uses 0 and 1 to represent zero elements and non-zero elements respectively; the bit mask decoder module converts the bit mask into (ID, CAS, MEG). Where ID is a log2 N-bit index in the original sparse matrix, which helps the processing element module select the corresponding input for calculation; CAS is a 1-bit concatenation indicator, indicating whether the current processing element module should concatenate PSUM with the PSUM of the previous processing element module for accumulation; MEG is a 1-bit merge indicator, used to merge PSUM across columns. The non-zero loader module feeds non-zero weights into arrays with corresponding (ID, CAS, MEG); As the input flows along the k direction, the processing element module performs multiplication based on the weight's ID; the processing element module selects an arbitrary value from the local cross switch; then, PSUM is forwarded along the j direction to the next processing element module, where CAS indicates whether accumulation should be performed; it is checked whether the PSUM output from the bottom processing element module needs to be merged from MEG.

6. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 2, characterized in that, The bitmask decoder module includes: a bitmask buffer module, a collection module, a decoder FIFO module, and a splicing module; The bitmask buffer module is connected to the collection module; the collection module is connected to the decoder FIFO module; and the decoder FIFO module is connected to the splicing module. The collection module is responsible for reading the original bitmask from the bitmask buffer module, and through the aggregation operation, it concentrates the scattered bit information into the low-order part, while setting the high-order part to 0. The processed data is then temporarily stored in the decoder FIFO buffer, awaiting further processing. The splicing module determines when to read new data from the FIFO based on the weighted valid signal and splices it with the existing data.

7. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 6, characterized in that, The bitmask decoder module translates the bitmask into indicators to guide the correct sparse computation on the diagonal data stream systolic array; the input is the first to the Nth row, which is a sparse bitmask; then it is aggregated into compact new rows; The collection module obtains the original bitmask from the bitmask buffer. After multiple rounds of continuous aggregation, the bitmasks are finally aggregated into the low bits, while the high bits are 0, and are stored in a decoder FIFO buffer. The splicing module retrieves a new row from the IFFO based on the weighted valid signal. If the new row is not long enough for output, it reads new data from the FIFO. After acquiring the data, it determines whether the data meets the new row length requirement. If it does, the data is output. The remaining data is then continuously and seamlessly spliced ​​together and output.

8. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 2, characterized in that, The post-processing module includes: an activation function module, a SoftMax module, a residual addition module, a layer normalization module, a transpose module, and a rearrangement module; The activation function module is used to execute activation functions during the inference process of large language models; The SoftMax module is used to map the value range of each node's data to [0, 1], and constrain the sum of the output values ​​of each output node to 1; The residual addition module is used to add the feature data from the previous step to the feature data of the current step during the inference process; The layer normalization module performs a normalization operation on the input data, constraining the data values ​​to a certain range; The transpose module is used to rotate the row and column orientations of a 32x32 matrix and then save the rotated matrix to the DDR memory module; The rearrangement module is used to rearrange the skipping data into an ordered form, and then save the entire block to the DDR memory module.

9. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 2, characterized in that, The KV cache module includes: an overflow module and an overload module; The overflow module is used to manage key-value cache overflows and implement a last-in-first-out (LIFO) strategy. The overload module is used to split the on-chip memory into multiple blocks, each block storing a fixed number of labeled KV tensors.

10. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 9, characterized in that, The overflow module is used to manage KV cache overflows and implement a last-in-first-out (LIFO) strategy, including: When the KV cache grows and exceeds the capacity of the on-chip memory, the overflow module is responsible for transferring these excess KV tensors to the off-chip DDR memory module. During the cross-attention phase of the decoding stage, since the KV cache data needs to be accessed frequently, the overflow module prioritizes retaining the KV tensors of the earlier steps in on-chip memory. If there is insufficient space, the KV tensors of the later steps will overflow.

11. An accelerator for accelerating the reasoning process of unstructured sparse large-scale language models according to claim 9, characterized in that, The overload module is used to divide the on-chip memory into multiple blocks, each block storing a fixed number of labeled KV tensors, including: The on-chip memory is divided into N blocks, each block storing M labeled KV tensors; the first M×(N-1) labeled KV tensors are fixed to the first N-1 memory blocks; they will always remain on-chip to reduce off-chip memory traffic; The newly generated KV tensor will be written to the last memory block. Once M marked KV tensors have been collected, the memory block will overflow to off-chip memory. Here, M is chosen to be equal to the size of the shrinking array.

12. A method for operating an accelerator to accelerate the reasoning process of unstructured sparse large-scale language models, characterized in that, An accelerator for accelerating the inference process of unstructured sparse large language models according to any one of claims 1 to 11, comprising: The computing core module sends a data request to the interconnect bus module, and the interconnect bus module communicates with the DDR memory module to transfer the required data to the computing core module; The processor module converts the large language model described by the high-level language into specific instructions and weight parameters, and stores this information in the DDR memory module through the direct memory access module; The computing core module is started, and it reads instructions and weight parameters from the DDR memory module and performs calculations according to the instructions. Different computing core modules exchange data through the router module and work together to complete complex nonlinear function calculations. The memory controller module processes memory read / write requests from the interconnect bus module and sends the requests to the DDR memory module, ensuring that memory access requests from each module are processed.

Images