Model inference task processing system and model inference task processing method

Abstract

The application discloses a model inference task processing system and method, relates to the technical field of model inference, and comprises a plurality of pre-filling stage hardware accelerators and a plurality of decoding stage hardware accelerators; the pre-filling stage hardware accelerator comprises a systolic array, and the systolic array processes a matrix multiplication matrix calculation task in a model inference process; the decoding stage hardware accelerator comprises a plurality of multiply-add trees, and the multiply-add trees process a matrix multiplication vector calculation task in the model inference process; the scale of the systolic array is the same as the total scale of the plurality of multiply-add trees; the computing instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible; and the computing node is used for determining a target pre-filling stage hardware accelerator and a target decoding stage hardware accelerator based on a model inference task, so as to process the model inference task. The technical effect of balancing TTFT and TPOT and improving the model inference efficiency is achieved.

Description

Technical Field

[0001] This application relates to the field of model reasoning technology, and in particular to a model reasoning task processing system and a model reasoning task processing method. Background Technology

[0002] Large Language Models (LLMs) based on Transformers divide the inference process into a prefill phase and a decoding phase. The input to the prefill phase consists of multiple tokens, and its computational operation is GEMM (matrix multiplication). The input to the decoding phase is a single token, and its computational operation is GEMV (matrix multiplication by a vector). In related technologies, hardware accelerators are optimized only for one type of computational operation, either GEMM or GEMV.

[0003] Under the optimized design of hardware accelerators in related technologies, if the hardware accelerator is optimized for GEMM computation operation types, the performance of the decoding stage is reduced. If the hardware accelerator is optimized for GEMV computation operation types, the waiting time in the prefill stage is increased, making it impossible to balance the first token generation time (Time To First Token, abbreviated as TTFT) and the single token generation time (Time Per Output Token, abbreviated as TPOT). Summary of the Invention

[0004] This application provides a model inference task processing system and a model inference task processing method to at least solve the problem in related technologies where hardware accelerators are only optimized for one type of computational operation in GEMM or GEMV, and cannot balance the first token generation time and the single token generation time.

[0005] This application provides a model inference task processing system, the system including at least one computing node; each computing node includes multiple pre-filling stage hardware accelerators and multiple decoding stage hardware accelerators; The pre-filling stage hardware accelerator includes a systolic array, which is used to process the matrix multiplication task in the pre-filling stage of the model inference process. The hardware accelerator for the decoding stage includes multiple multiply-accumulate trees, each of which is used to process the matrix-vector multiplication task in the decoding stage of the model inference process. The size of the systolic array in the pre-filling stage hardware accelerator is the same as the total size of the multiple multiply-accumulate trees in the decoding stage hardware accelerator; the computation instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible. The computing node is used to acquire the model inference task, determine the target pre-fill stage hardware accelerator from multiple pre-fill stage hardware accelerators based on the model inference task, and determine the target decoding stage hardware accelerator from multiple decoding stage hardware accelerators, so as to process the model inference task based on the target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner.

[0006] This application also provides a model inference task processing method, applied to any computing node in any of the above-mentioned model inference task processing systems, the method comprising: The model inference task is obtained, and a target pre-filling stage hardware accelerator is determined from multiple pre-filling stage hardware accelerators based on the model inference task. A target decoding stage hardware accelerator is also determined from multiple decoding stage hardware accelerators. The model inference task is processed based on the target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model inference task processing methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model inference task processing methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model inference task processing methods.

[0010] Through this application, each computing node includes multiple pre-filling stage hardware accelerators and multiple decoding stage hardware accelerators. The pre-filling stage hardware accelerator includes a systolic array, which is used to process the matrix multiplication task in the pre-filling stage of the model inference process. The decoding stage hardware accelerator includes multiple multiply-accumulate trees, each of which is used to process the matrix multiplication task in the decoding stage of the model inference process. The size of the systolic array in the pre-filling stage hardware accelerator is the same as the total size of the multiple multiply-accumulate trees in the decoding stage hardware accelerator. The computing instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible. The computing node is used to obtain the model inference task, determine the target pre-filling stage hardware accelerator from the multiple pre-filling stage hardware accelerators based on the model inference task, and determine the target decoding stage hardware accelerator from the multiple decoding stage hardware accelerators, so as to process the model inference task based on the target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner. By designing different hardware accelerators based on the computational characteristics of different stages, model inference can be performed using the corresponding hardware accelerators at different stages. Therefore, this solves the technical problem in related technologies where hardware accelerators are only optimized for one type of computational operation in GEMM or GEMV, and cannot balance the first token generation time and the single token generation time. This achieves the technical effect of balancing the first token generation time and the single token generation time, thereby improving the efficiency of model inference. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a large language model based on Transformer in related technologies. Figure 2 This is a schematic diagram illustrating the use of pulsating arrays to process GEMM computing tasks in related technologies. Figure 3 This is a schematic diagram illustrating the use of pulsating arrays to process GEMV computation tasks in related technologies. Figure 4 This application provides a schematic diagram of the structure of a model reasoning task processing system. Figure 5 A schematic diagram of the pre-filling stage hardware accelerator provided in the embodiments of this application; Figure 6This is a schematic diagram of the hardware accelerator for the decoding stage provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Figure 1 This is a structural diagram of a large language model based on Transformer in related technologies, such as... Figure 1 As shown, the reasoning process of a large language model is essentially a series of loops of operations, which are called a block. The output of the i-th block is the input of the (i+1)-th block. Each block includes a first normalization layer, a multi-head self-attention layer, a first residual connection layer, a second normalization layer, a feedforward neural network layer, and a second residual connection layer.

[0017] Feedforward neural network layers and multi-head self-attention layers are the main sources of computational power requirements. The number of block loops in a complete inference operation of a large language model is approximately ten to several dozen times.

[0018] In a Transformer-based Large Language Model (LLM) inference, both input and output are represented by tokens. Each token represents the mathematical representation of a single character or word, i.e., a vector. LLM inference is mainly divided into a pre-filling phase and a decoding phase. The pre-filling phase feeds all tokens corresponding to all words input by the user into the LLM at once. Since the total number of tokens corresponding to all words input by the user is generally greater than one, multiple vectors need to be calculated and processed, requiring parallel computation. After the pre-filling phase completes its computation, it outputs a new token representing the newly generated word, which serves as the input for the decoding phase. In other words, the decode phase always uses only one token as input. The decode phase employs an autoregressive approach, generating only one new token each time, which is immediately used as the input for the next round of decoding.

[0019] Mathematically, the prefill and decode phases exhibit distinct computational behaviors within the feedforward neural network layer and the multi-head self-attention layer. Because the prefill phase takes multiple tokens as input, its computational operations in these two layers are GEMM (Generative Execution Model), which are computationally intensive. Conversely, the decode phase takes a single token as input, so its computational operations in these layers are GEMV (Generative Execution Model), which are storage-intensive.

[0020] Current LLM inference needs to meet two typical metrics: TTFT and TPOT. TTFT measures the time required in the prefill phase to generate the first token from the user-input token. TPOT measures the time required in the decode phase to generate each token. Neither of these times should be too long, otherwise it will negatively impact the user experience.

[0021] Current hardware accelerators are typically optimized for only one type of computational operation in either GEMM or GEMV. When the prefill and decode phases run on the same hardware accelerator architecture, resource contention and architectural trade-offs can lead to a conflict between TTFT and TPOT. Specifically, if the hardware accelerator is optimized for GEMM operations (prioritizing the prefill phase to reduce TTFT), it may decrease the performance of the decode phase, thus increasing TPOT. Conversely, if the hardware accelerator is optimized for GEMV operations (prioritizing the decode phase to reduce TPOT), it may increase the waiting time of the prefill phase, leading to an increase in TTFT and an inability to balance TTFT and TPOT.

[0022] Furthermore, using hardware accelerators developed specifically for GEMM to perform GEMV calculations would result in significant resource waste. Hardware accelerators for GEMM are typically based on systolic array structures. Specifically, Figure 2 This diagram illustrates the use of pulsating arrays to process GEMM computation tasks in related technologies. The GEMM computation task can be A1×B1+C1, where A1 and B1 are matrices, and C1 can be either a matrix or a vector. When A1, B1, and C1 are all matrices, as shown... Figure 2 As shown, matrices A1, B1, and C1 are input into a pulsating array, which performs the operation A1×B1+C1 and outputs the calculation result. The pulsating array consists of multiple processing elements (PEs), also known as computational units, which are arranged and connected in a matrix-like manner. Figure 3 This diagram illustrates the use of pulsating arrays to process GEMV computation tasks in related technologies. A GEMV computation task can be A2×B2+C2, where A2 is a vector, B2 is a matrix, and C2 is generally a vector, but can also be a matrix. When A2 is a vector and B2 and C2 are matrices, as shown... Figure 3 As shown, A2, B2, and C2 are input into the systolic array, which performs the operation A2×B2+C2 and outputs the calculation result. When using the systolic array to process GEMV computation tasks, most processing units are in an idle state.

[0023] To address the aforementioned technical problems, this application provides a model inference task processing system and a model inference task processing method. The model inference task processing system includes: at least one computing node; each computing node includes multiple pre-filling stage hardware accelerators and multiple decoding stage hardware accelerators; the pre-filling stage hardware accelerators include systolic arrays, which are used to process the matrix multiplication task in the pre-filling stage of the model inference process; the decoding stage hardware accelerators include multiple multiply-accumulate trees, each multiply-accumulate tree being used to process the matrix multiplication task in the decoding stage of the model inference process; the pre-filling stage hardware accelerators... The size of the systolic array in the pre-filling stage is the same as the total size of the multiple multiply-accumulate trees in the decoding stage hardware accelerator; the computation instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible; the computing node is used to acquire the model inference task, determine the target pre-filling stage hardware accelerator from multiple pre-filling stage hardware accelerators based on the model inference task, and determine the target decoding stage hardware accelerator from multiple decoding stage hardware accelerators, so as to process the model inference task based on the target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner. The system provided by the above scheme designs different hardware accelerators according to the computational characteristics of different stages, so as to use the corresponding hardware accelerators for model inference at different stages. Therefore, it can solve the technical problem in related technologies where hardware accelerators are only optimized for one type of computational operation in GEMM or GEMV, and cannot balance the first token generation time and the single token generation time. It achieves the technical effect of balancing the first token generation time and the single token generation time, and improving the model inference efficiency. Furthermore, the computing instructions between hardware accelerators at different stages are fully compatible, achieving seamless integration with upper-layer software development and facilitating the distributed cluster deployment of different hardware accelerators.

[0024] Embodiments of this application provide a model inference task processing system. Figure 4 This is a schematic diagram of the structure of the model inference task processing system provided in the embodiments of this application, as shown below. Figure 4 As shown, the model inference task processing system includes at least one computing node. Each computing node includes multiple pre-filling stage hardware accelerators and multiple decoding stage hardware accelerators.

[0025] The pre-filling stage hardware accelerator includes a systolic array, which processes matrix multiplication tasks during the pre-filling stage of model inference. The systolic array comprises multiple processing units arranged and connected in a matrix-like pattern. A matrix multiplication task can be A3×B3+C3, where A3 and B3 are matrices, and C3 is typically a vector, but can also be a matrix. For example, with A3 and B3 having dimensions of 32×32, the matrix multiplication task requires 32×32×32=32k multiplications. With A3 and B3 having dimensions of 64×64, the matrix multiplication task requires 64×64×64=256k multiplications.

[0026] The size of the systolic array can be set according to the resources of the chip housing the hardware accelerator in the pre-filling stage. Specifically, when the chip housing the hardware accelerator in the pre-filling stage is a Field Programmable Gate Array (FPGA) chip, the size of the systolic array can be configured according to the FPGA chip's resources, such as 16×16, 32×32, 64×64, 128×128, 256×256, etc. The size of the systolic array is defined by the number of rows and columns of the processing units it contains. For example, a 16×16 systolic array means that the processing units in the array are arranged in a two-dimensional grid of 16 rows and 16 columns, containing a total of 16×16=256 processing units.

[0027] Due to limited FPGA resources, some systolic arrays can be set to a size of 32×32, operating at a clock frequency of 250MHz, achieving a computing power of 500 GFLOPS, or 500 billion floating-point operations per second. Other systolic arrays can be set to a size of 64×64, operating at a clock frequency of 250MHz, achieving a computing power of 2 TFLOPS, or two trillion floating-point operations per second.

[0028] The hardware accelerator in the decoding stage comprises multiple multiply-accumulate trees, each processing the matrix-vector multiplication task during the decoding phase of the model inference process. Each multiply-accumulate tree includes multiple processing units. The matrix-vector multiplication task can be A4×B4+C4, where A4 is a vector, B4 is a matrix, and C4 is generally a vector. For example, with B4 having a dimension of 32×32, the task requires 32×32=1k multiplications. With B4 having a dimension of 64×64, the task requires 64×64=4k multiplications.

[0029] As can be seen, the computational complexity of matrix-vector multiplication differs greatly from that of matrix-matrix multiplication. Therefore, in this embodiment, the hardware accelerator for the decoding stage uses a multiply-accumulate tree structure instead of a systolic array.

[0030] The size of the multiply-accumulate tree can be set according to the resources of the chip where the hardware accelerator in the decoding stage is located. Specifically, when the chip where the hardware accelerator in the decoding stage is located is a Field Programmable Gate Array (FPGA) chip, the size of the multiply-accumulate tree can be configured according to the resources of the FPGA chip.

[0031] Due to limited FPGA resources, the length of the multiply-accumulate tree can be 256, meaning it includes 256 processing units. The hardware accelerator in the decoding stage can accommodate four multiply-accumulate trees (256 × 4 = 1024 = 32 × 32), processing four data streams in parallel. If a systolic array is used, the same computing power would require 256 / 32 × 4 = 32 FPGAs.

[0032] The length of the multiply-accumulate tree can be set to 1024, meaning it includes 1024 processing units. The hardware accelerator in the decoding stage can accommodate four multiply-accumulate trees (1024 × 4 = 4096 = 64 × 64), processing four data streams in parallel. If a systolic array is used, the same computing power would require 1024 / 64 × 4 = 64 FPGAs.

[0033] The size of the systolic array in the pre-filling stage hardware accelerator is the same as the total size of the multiple multiply-accumulate trees in the decoding stage hardware accelerator; the computation instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible.

[0034] In this context, if the size of the systolic array is 16×16, then the scale of the systolic array is 16; if the size of the systolic array is 32×32, then the scale of the systolic array is 32, and so on. Similarly, if the length of the multiply-accumulate tree is 16, then the scale of the multiply-accumulate tree is 16, and so on. That is to say, when the scale of the systolic array is 32 and the length of the multiply-accumulate tree is 16, to ensure that the total scale of the multiple multiply-accumulate trees in the decoding stage hardware accelerator is the same as the scale of the systolic array, the decoding stage hardware accelerator includes two multiply-accumulate trees.

[0035] The computing node is used to acquire the model inference task, determine the target pre-fill stage hardware accelerator from multiple pre-fill stage hardware accelerators based on the model inference task, and determine the target decoding stage hardware accelerator from multiple decoding stage hardware accelerators, so as to process the model inference task based on the target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner.

[0036] The model inference task processing system provided in this application embodiment designs different hardware accelerators according to the computational characteristics of different stages, so as to use the corresponding hardware accelerators for model inference at different stages. Therefore, it can solve the technical problem in the related technology that the hardware accelerator is only optimized for one type of computational operation in GEMM or GEMV, and cannot balance the first token generation time and the single token generation time. It achieves the technical effect of balancing the first token generation time and the single token generation time and improving the efficiency of model inference.

[0037] In some alternative implementations, the pre-filling stage hardware accelerator further includes a first storage module, a second storage module, and a first storage read control module.

[0038] The first and second storage modules are used to ping-pong store the input data of the matrix multiplication calculation task, provided that the size of the matrix multiplication calculation task is no larger than the size of the systolic array.

[0039] It should be noted that the size of the systolic array is generally no larger than the size of the first storage module. The size of the first storage module is the same as the size of the second storage module. The size of the matrix multiplication task is determined based on the size of the matrices being multiplied. For example, if the size of the matrices being multiplied in the matrix multiplication task is 32×32, then the size of the matrix multiplication task is 32. The first and second storage modules can be used to store matrix multiplication tasks with a size of 512.

[0040] It is understandable that the first and second storage modules ping-pong store the input data for the matrix multiplication calculation task, allowing data input and calculation to be executed in parallel.

[0041] The first storage read control module is used to read the input data of the matrix multiplication calculation task from the first storage module or the second storage module, and input the input data of the matrix multiplication calculation task into the pulsating array.

[0042] The pulsating array is specifically used to process the input data of the matrix multiplication task to obtain the output result of the matrix multiplication task.

[0043] The model inference task processing system provided in this application embodiment achieves seamless overlap between data loading and data calculation by using the first storage module and the second storage module to store input data in a ping-pong manner, thereby maximizing the utilization rate of the pulsating array and significantly improving the processing throughput of the prefill stage.

[0044] In some alternative implementations, the pre-filling stage hardware accelerator also includes a first accumulation module.

[0045] The pre-filling stage hardware accelerator is used to divide the matrix multiplication task into multiple first matrix multiplication subtasks when the size of the matrix multiplication task is no larger than the size of the first storage module but larger than the size of the systolic array.

[0046] The first and second storage modules are also used for ping-pong storage of the input data for the first matrix multiplication subtask.

[0047] The first storage read control module is also used to read the input data of the first matrix multiplication subtask from the first storage module or the second storage module, and input the input data of the first matrix multiplication subtask into the pulsating array.

[0048] The pulsating array is specifically used to process the first matrix multiplication subtask based on the input data of the first matrix multiplication subtask, obtain the output result of the first matrix multiplication subtask, and send the output result of the first matrix multiplication subtask to the first accumulation module.

[0049] The first accumulation module is used to accumulate the output results of multiple first matrix multiplication subtasks corresponding to the matrix multiplication task to obtain the output result of the matrix multiplication task.

[0050] The model inference task processing system provided in this application embodiment enables a fixed-size systolic array to efficiently process large-scale matrix multiplications through intelligent block strategy and on-chip accumulation mechanism.

[0051] In some alternative implementations, the pre-filling stage hardware accelerator also includes a second accumulation module.

[0052] The pre-filling stage hardware accelerator is also used to divide the matrix multiplication task into multiple second matrix multiplication subtasks when the size of the matrix multiplication task is larger than the size of the first storage module; and to divide the second matrix multiplication subtask into multiple third matrix multiplication subtasks when the size of the second matrix multiplication subtask is larger than the size of the systolic array.

[0053] The first and second storage modules are also used for ping-pong storage of the input data for the third matrix multiplication subtask.

[0054] The first storage read control module is also used to read the input data of the third matrix multiplication subtask from the first storage module or the second storage module, and input the input data of the third matrix multiplication subtask into the pulsating array.

[0055] The pulsating array is specifically used to process the input data of the third matrix multiplication subtask based on the input data of the third matrix multiplication subtask, obtain the output result of the third matrix multiplication subtask, and send the output result of the third matrix multiplication subtask to the first accumulation module.

[0056] The first accumulation module is also used to accumulate the output results of multiple third matrix multiplication subtasks corresponding to the second matrix multiplication subtask to obtain the output result of the second matrix multiplication subtask, and send the output result of the second matrix multiplication subtask to the second accumulation module.

[0057] The second accumulation module is used to accumulate the output results of multiple second matrix multiplication subtasks corresponding to the matrix multiplication task to obtain the output result of the matrix multiplication task.

[0058] Figure 5 A schematic diagram of the pre-filling stage hardware accelerator provided in the embodiments of this application is shown below. Figure 5 As shown, the pre-filling stage hardware accelerator has a multi-layer architecture. The bottom layer is the first matrix operation module, which is a systolic array. The middle layer includes a first storage module, a second storage module, a first storage read control module, a first storage write control module, a first data alignment module, a first transpose module, a second data alignment module, a first output module, and a first accumulation module.

[0059] The descriptions of the first storage module, the second storage module, the first storage read control module, and the first accumulation module are as described above and will not be repeated here.

[0060] The first storage write control module is used to ping-pong store the input data of the matrix multiplication calculation task into the first storage module and the second storage module.

[0061] The first storage write control module is also used to ping-pong store the input data of the first matrix multiplication subtask into the first storage module and the second storage module.

[0062] The first storage write control module is also used to ping-pong store the input data of the third matrix multiplication subtask into the first storage module and the second storage module.

[0063] The first data alignment module is used to perform data alignment processing on the input data of the matrix multiplication calculation task before the first storage read control module inputs the input data of the matrix multiplication calculation task into the systolic array, so as to meet the input requirements of the systolic array.

[0064] The first data alignment module is also used to perform data alignment processing on the input data of the first matrix multiplication subtask before the first storage read control module inputs the input data of the first matrix multiplication subtask into the systolic array, so as to meet the input requirements of the systolic array.

[0065] The first data alignment module is also used to perform data alignment processing on the input data of the third matrix multiplication subtask before the first storage read control module inputs the input data of the third matrix multiplication subtask into the systolic array, so as to meet the input requirements of the systolic array.

[0066] The first transpose module is used to transpose the output of the pulsating array when transpose operation is required.

[0067] The second data alignment module is used to perform data alignment processing on the output results of the systolic array to meet the output requirements of the systolic array.

[0068] The first output module is used to output the output result after the transpose operation of the systolic array or the output result after the data alignment processing of the systolic array.

[0069] The first accumulation module includes a first accumulation write control module, a first accumulation read control module, a first accumulation storage area, and a first accumulation operation module. The first accumulation write control module writes the received output of a first matrix-matrix multiplication subtask or a third matrix-matrix multiplication subtask into the first accumulation storage area. The first accumulation read control module reads the output of the first matrix-matrix multiplication subtask or the third matrix-matrix multiplication subtask from the first accumulation storage area and sends it to the first accumulation operation module. The first accumulation operation module accumulates the outputs of multiple first matrix-matrix multiplication subtasks corresponding to the matrix-matrix multiplication subtask to obtain the output of the matrix-matrix multiplication subtask. The first accumulation operation module also accumulates the outputs of multiple third matrix-matrix multiplication subtasks corresponding to the second matrix-matrix multiplication subtask to obtain the output of the second matrix-matrix multiplication subtask.

[0070] The outermost layer includes a second accumulation module, which comprises a second accumulation write control module, a second accumulation read control module, a second accumulation storage area, and a second accumulation operation module. The second accumulation write control module writes the received output of the second matrix multiplication subtask to the second accumulation storage area. The second accumulation read control module reads the output of the second matrix multiplication subtask from the second accumulation storage area and sends it to the second accumulation operation module. The second accumulation operation module accumulates the outputs of multiple second matrix multiplication subtasks corresponding to the matrix multiplication task to obtain the final output of the matrix multiplication task.

[0071] It is understandable that the first and second storage modules can be used to store matrix multiplication tasks with a storage size of 512. After block computation, the first accumulation module can accumulate to obtain a maximum matrix multiplication result of 512×512. Adding another accumulation operation at the outermost layer allows for matrix multiplication results up to 2097152×2097152. That is, matrix multiplications up to 2097152×2097152 can be automatically calculated in blocks by the hardware accelerator during the pre-filling stage. Larger matrix multiplications can be obtained directly through external instructions.

[0072] The model inference task processing system provided in this application embodiment adopts a layered design for the hardware accelerator in the pre-filling stage. The size of the pulsation array is configurable and can support a maximum of 2097152×2097152 matrix multiplication, thus maximizing the performance of the hardware accelerator.

[0073] In some alternative implementations, the decoding stage hardware accelerator also includes a third storage module, a fourth storage module, and a second storage read control module.

[0074] The third and fourth storage modules are used to ping-pong store the input data for the matrix-vector multiplication task, provided that the size of the matrix-vector multiplication task is no larger than the size of the multiply-add tree.

[0075] The second storage read control module is used to read the input data of the matrix multiplication vector calculation task from the third or fourth storage module and input the input data of the matrix multiplication vector calculation task into the target multiply-add tree.

[0076] The target multiply-add tree is used to process the input data of the matrix-vector multiplication task to obtain the output result of the matrix-vector multiplication task.

[0077] The model inference task processing system provided in this application embodiment achieves seamless overlap between data loading and data computation by using the third and fourth storage modules to store input data in a ping-pong manner, maximizing the utilization of the systolic array and significantly improving the processing throughput of the decoding stage.

[0078] In some alternative implementations, the hardware accelerator for the decoding stage also includes a third accumulation module.

[0079] The hardware accelerator in the decoding stage is used to divide the matrix multiplication vector calculation task into multiple first matrix multiplication vector calculation subtasks when the size of the matrix multiplication vector calculation task is no larger than the size of the third storage module but larger than the size of the multiply-add tree.

[0080] The third and fourth storage modules are also used for ping-pong storage of the input data for the first matrix-vector multiplication subtask.

[0081] The second storage read control module is also used to read the corresponding number of input data for the first matrix multiplication vector calculation subtask from the third or fourth storage module based on the number of multiply-accumulate trees in the hardware accelerator of the decoding stage, and input the corresponding number of input data for the first matrix multiplication vector calculation subtask to the corresponding multiply-accumulate trees respectively.

[0082] The multiply-accumulate tree is specifically used to process the input data of the first matrix-vector multiplication subtask based on the input data of the first matrix-vector multiplication subtask, obtain the output result of the first matrix-vector multiplication subtask, and send the output result of the first matrix-vector multiplication subtask to the third accumulation module.

[0083] The third accumulation module is used to accumulate the output results of multiple first matrix-vector calculation subtasks corresponding to the matrix-vector calculation task to obtain the output result of the matrix-vector calculation task.

[0084] The model inference task processing system provided in this application embodiment enables a fixed-size systolic array to efficiently process large-scale matrix multiplication vector calculation tasks through intelligent block strategy and on-chip accumulation mechanism.

[0085] In some alternative implementations, the hardware accelerator for the decoding stage also includes a fourth accumulation module.

[0086] The hardware accelerator in the decoding stage is also used to divide the matrix-vector multiplication task into multiple second matrix-vector multiplication subtasks when the size of the matrix-vector multiplication task is larger than the size of the third storage module; and to divide the second matrix-vector multiplication subtask into multiple third matrix-vector multiplication subtasks when the size of the second matrix-vector multiplication subtask is larger than the size of the multiply-add tree.

[0087] The third and fourth storage modules are also used for ping-pong storage of the input data for the third matrix-vector multiplication subtask.

[0088] The second storage read control module is also used to read the corresponding number of input data for the third matrix multiplication vector calculation subtask from the third storage module or the fourth storage module based on the number of multiply-accumulate trees in the hardware accelerator of the decoding stage, and input the corresponding number of input data for the third matrix multiplication vector calculation subtask to the corresponding multiply-accumulate trees respectively.

[0089] The multiply-accumulate tree is specifically used to process the input data of the third matrix multiply-vector calculation subtask based on the input data of the input third matrix multiply-vector calculation subtask, obtain the output result of the third matrix multiply-vector calculation subtask, and send the output result of the third matrix multiply-vector calculation subtask to the third accumulation module.

[0090] The third accumulation module is also used to accumulate the output results of multiple third matrix multiplication vector calculation subtasks corresponding to the second matrix multiplication vector calculation subtask, obtain the output result of the second matrix multiplication vector calculation subtask, and send the output result of the second matrix multiplication vector calculation subtask to the fourth accumulation module.

[0091] The fourth accumulation module is used to accumulate the output results of multiple second matrix-vector calculation subtasks corresponding to the matrix-vector calculation task to obtain the output result of the matrix-matrix calculation task.

[0092] Figure 6 This is a schematic diagram of the hardware accelerator for the decoding stage provided in an embodiment of this application, as shown below. Figure 6 As shown, the hardware accelerator in the decoding stage adopts the same layered design as the hardware accelerator in the pre-filling stage. That is, the hardware accelerator in the decoding stage has a multi-layer architecture, in which the bottom layer consists of multiple second matrix operation modules (matrix), which are multiply-accumulate trees. Each multiply-accumulate tree can complete a GEMV operation independently. Figure 6 The following example illustrates the decoding stage hardware accelerator, which includes four multiply-accumulate trees. The number of multiply-accumulate trees in the decoding stage hardware accelerator can be determined based on the size of the systolic array included in the pre-filling stage hardware accelerator. This variable number of multiply-accumulate trees allows the architecture to adapt to accelerator cards with more diverse hardware resources. The size of each multiply-accumulate tree can be 16, 32, 64, 128, etc., corresponding to the size of the systolic array.

[0093] The intermediate layer includes a third storage module, a fourth storage module, a second storage read control module, a second storage write control module, a third data alignment module, a second transpose module, a fourth data alignment module, a second output module, and a third accumulation module. The descriptions of the third storage module, the fourth storage module, the second storage read control module, and the third accumulation module are as described above and will not be repeated here.

[0094] The second storage write control module is used to ping-pong store the input data of the matrix-vector multiplication calculation task to the third and fourth storage modules. It is also used to ping-pong store the input data of the first matrix-vector multiplication calculation subtask to the third and fourth storage modules. Furthermore, it is used to ping-pong store the input data of the third matrix-vector multiplication calculation subtask to the third and fourth storage modules.

[0095] The third data alignment module is used to perform data alignment processing on the input data of the matrix multiplication-vector calculation task before the second storage read control module inputs it into the target multiply-accumulate tree, to meet the input requirements of the multiply-accumulate tree. It is also used to perform data alignment processing on the corresponding number of input data of the first matrix multiplication-vector calculation subtasks read by the second storage read control module before inputting them into the corresponding multiply-accumulate trees, to meet the input requirements of the multiply-accumulate tree. Furthermore, it is used to perform data alignment processing on the corresponding number of input data of the third matrix multiplication-vector calculation subtasks read by the second storage read control module before inputting them into the corresponding multiply-accumulate trees, to meet the input requirements of the multiply-accumulate tree.

[0096] The second transpose module is used to transpose the output of the multiply-accumulate tree when transpose is required.

[0097] The fourth data alignment module is used to perform data alignment processing on the output results of the multiply-accumulate tree to meet the output requirements of the multiply-accumulate tree.

[0098] The second output module is used to output the result of the transpose operation of the multiply-accumulate tree or the result of the data alignment processing of the multiply-accumulate tree.

[0099] The third accumulation module includes a third accumulation write control module, a third accumulation read control module, a third accumulation storage area, and a third accumulation operation module. The third accumulation write control module writes the received output of either the first or third matrix-vector multiplication subtask to the third accumulation storage area. The third accumulation read control module reads the output of either the first or third matrix-vector multiplication subtask from the third accumulation storage area and sends it to the third accumulation operation module. The third accumulation operation module accumulates the outputs of multiple first matrix-vector multiplication subtasks corresponding to the matrix-vector multiplication subtask to obtain the output of the matrix-vector multiplication subtask. The third accumulation operation module also accumulates the outputs of multiple third matrix-vector multiplication subtasks corresponding to the second matrix-vector multiplication subtask to obtain the output of the second matrix-vector multiplication subtask.

[0100] The outermost layer includes a fourth accumulation module, which comprises a fourth accumulation write control module, a fourth accumulation read control module, a fourth accumulation storage area, and a fourth accumulation operation module. The fourth accumulation write control module writes the received output of the second matrix-vector multiplication subtask to the fourth accumulation storage area. The fourth accumulation read control module reads the output of the second matrix-vector multiplication subtask from the fourth accumulation storage area and sends it to the fourth accumulation operation module. The fourth accumulation operation module accumulates the outputs of multiple second matrix-vector multiplication subtasks corresponding to the matrix-vector multiplication task to obtain the final output of the matrix-vector multiplication task.

[0101] In some alternative implementations, the compute node is specifically used for: Based on the length of the user input tokens and the size of the systolic array in the model inference task, the actual length of the input tokens processed by the systolic array in the pre-filling stage is determined.

[0102] Obtain the number of multiply-accumulate trees in the hardware accelerator during the decoding stage.

[0103] Based on the actual length of the input tokens processed by the pre-filling stage systolic array and the number of multiply-accumulate trees in the decoding stage hardware accelerator, the ratio of the number of target pre-filling stage hardware accelerators to the number of target decoding stage hardware accelerators is determined.

[0104] Based on the ratio of the number of hardware accelerators in the target pre-filling stage to the number of hardware accelerators in the target decoding stage, the target pre-filling stage hardware accelerator is determined from multiple pre-filling stage hardware accelerators, and the target decoding stage hardware accelerator is determined from multiple decoding stage hardware accelerators.

[0105] The model inference task processing system provided in this application maximizes the performance of hardware accelerators and improves the overall throughput of the system by deploying different numbers of hardware accelerators for the decoding stage and the computing power ratio of hardware accelerators at different stages, based on the actual length of the input tokens processed by the systolic array in the pre-filling stage.

[0106] In some alternative implementations, the compute node is specifically used for: The length of the input tokens actually processed by the pre-filling stage systolic array is determined by the following formula: N3 = Ceil(N1 / N2) × N2 Where N3 is the length of the input tokens actually processed by the systolic array in the pre-filling stage, Ceil represents rounding up, N1 is the length of the user input tokens in the model inference task, and N2 is the size of the systolic array.

[0107] In some alternative implementations, the compute node is specifically used for: Based on the actual input term length processed by the pre-filling stage systolic array and the number of multiply-accumulate trees in the decoding stage hardware accelerator, the ratio of the target pre-filling stage hardware accelerator to the target decoding stage hardware accelerator is determined as follows: Ceil(N3 / N2)×N:1 Where N3 is the actual length of the input tokens processed by the systolic array in the pre-padding stage, Ceil represents rounding up, N2 is the size of the systolic array, and N is the number of multiply-accumulate trees in the hardware accelerator of the decoding stage.

[0108] To maximize the overall system throughput, the number of hardware accelerators in the prefill and decode stages needs to be appropriately matched. Assuming the user input token length is N1 in the prefill stage, and the size of both the systolic array and the multiply-accumulate tree is N2 (N2 = 16, 32, 64, 128, etc.), due to architectural limitations, the computing power of a single systolic array is approximately N2 times that of a single multiply-accumulate tree of the same size. That is, the computing power of the systolic array is N2 × N2, while the computing power of a single multiply-accumulate tree is N2. Therefore, the computing power of a single hardware accelerator in the prefill stage is N2 times that of a single hardware accelerator in the decode stage. Let the actual input token length N3 processed by the systolic array in the prefill stage be Ceil(N1 / N2) × N2, where Ceil represents rounding up. When the decode stage hardware accelerator contains N multiply-accumulate trees, the reasonable ratio of the number of hardware accelerators in the prefill stage to the number of hardware accelerators in the decode stage is Ceil(N3 / N2) × N: 1.

[0109] In some alternative implementations, the compute node is specifically used for: If the model inference task fails to be processed using the hardware accelerators for the target pre-filling stage and the target decoding stage, the steps for processing the model inference task using the hardware accelerators for the target pre-filling stage and the target decoding stage should be re-executed.

[0110] An alarm is triggered if the number of times the steps for processing the model inference task based on the target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator are re-executed exceeds a preset threshold, and the model inference task still fails to be processed.

[0111] The model inference task processing system provided in this application embodiment, through a hierarchical fault tolerance mechanism, ensures system reliability while avoiding unnecessary service interruptions, thereby achieving highly available model inference services.

[0112] This application provides a model inference task processing method, applicable to any computation stage of any of the above-mentioned model inference task processing systems, the method comprising: The model inference task is obtained, and a target pre-filling stage hardware accelerator is determined from multiple pre-filling stage hardware accelerators based on the model inference task. A target decoding stage hardware accelerator is also determined from multiple decoding stage hardware accelerators. The model inference task is processed based on the target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-filling stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner.

[0113] For a description of the features in the embodiment corresponding to the model reasoning task processing method, please refer to the relevant description of the embodiment corresponding to the model reasoning task processing system, which will not be repeated here.

[0114] The model inference task processing method provided in this application's embodiments designs different hardware accelerators according to the computational characteristics of different stages. The prefill stage employs a hierarchical systolic array, the size of which is configurable, supporting automatic block-based computation and accumulation of matrix multiplication up to 2097152×2097152. The decode stage uses a variable-number multiply-accumulate tree instead of the systolic array, making it possible for the architecture to adapt to accelerator cards with more diverse hardware resources. The overall accelerator implementation employs various optimization techniques, applicable to any LLM, maximizing hardware performance and improving AI computational energy efficiency. The computational instructions are compatible between different accelerators, with no software awareness, facilitating distributed cluster deployment of different accelerators.

[0115] It should be noted that the pre-filling stage hardware accelerator and the decoding stage hardware accelerator can be installed on any of the following devices: FPGA, Application Specific Integrated Circuit (ASIC), Tensor Processing Unit (TPU), Deep Learning Processing Unit (DPU), Neural Network Processing Unit (NPU), and General-Purpose Graphics Processing Unit (GPGPU). They can also be directly applied to embedded edge AI accelerator deployments.

[0116] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a processor 701 and a memory 702, in which a computer program is stored. The processor 701 is configured to run the computer program to perform the steps in any of the above-described embodiments of the model inference task processing method.

[0117] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model inference task processing method when it is run.

[0118] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0119] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the model inference task processing method.

[0120] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the model inference task processing method.

[0121] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] The above provides a detailed description of the model reasoning task processing system and method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model reasoning task processing system, characterized in that, The system includes at least one computing node; each computing node includes multiple pre-filling stage hardware accelerators and multiple decoding stage hardware accelerators; The pre-filling stage hardware accelerator includes a systolic array, which is used to process the matrix multiplication task in the pre-filling stage of the model inference process. The hardware accelerator for the decoding stage includes multiple multiply-accumulate trees, each of which is used to process the matrix-vector multiplication task in the decoding stage of the model inference process. The size of the systolic array in the pre-filling stage hardware accelerator is the same as the total size of the multiple multiply-accumulate trees in the decoding stage hardware accelerator; the computation instructions of the pre-filling stage hardware accelerator and the decoding stage hardware accelerator are compatible. The computing node is used to acquire the model inference task, determine the target pre-fill stage hardware accelerator from multiple pre-fill stage hardware accelerators based on the model inference task, and determine the target decoding stage hardware accelerator from multiple decoding stage hardware accelerators, so as to process the model inference task based on the target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator. The target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator work together in a cascaded manner.

2. The system according to claim 1, characterized in that, The pre-filling stage hardware accelerator also includes a first storage module, a second storage module, and a first storage read control module; The first storage module and the second storage module are used to ping-pong store the input data of the matrix multiplication calculation task when the size of the matrix multiplication calculation task is not greater than the size of the systolic array. The first storage read control module is used to read the input data of the matrix multiplication calculation task from the first storage module or the second storage module, and input the input data of the matrix multiplication calculation task into the pulsating array; The pulsating array is specifically used to process the matrix multiplication task based on the input data of the matrix multiplication task, and obtain the output result of the matrix multiplication task.

3. The system according to claim 2, characterized in that, The pre-filling stage hardware accelerator also includes a first accumulation module; The pre-filling stage hardware accelerator is used to divide the matrix multiplication calculation task into multiple first matrix multiplication calculation subtasks when the size of the matrix multiplication calculation task is no larger than the size of the first storage module but larger than the size of the systolic array. The first storage module and the second storage module are also used to ping-pong store the input data of the first matrix multiplication subtask; The first storage read control module is further configured to read the input data of the first matrix multiplication subtask from the first storage module or the second storage module, and input the input data of the first matrix multiplication subtask into the pulsating array; The pulse array is specifically used to process the first matrix multiplication subtask based on the input data of the first matrix multiplication subtask, obtain the output result of the first matrix multiplication subtask, and send the output result of the first matrix multiplication subtask to the first accumulation module. The first accumulation module is used to accumulate the output results of multiple first matrix multiplication subtasks corresponding to the matrix multiplication calculation task to obtain the output result of the matrix multiplication calculation task.

4. The system according to claim 3, characterized in that, The pre-filling stage hardware accelerator also includes a second accumulation module; The pre-filling stage hardware accelerator is further configured to divide the matrix multiplication calculation task into multiple second matrix multiplication calculation subtasks when the size of the matrix multiplication calculation task is greater than the size of the first storage module; and to divide the second matrix multiplication calculation subtask into multiple third matrix multiplication calculation subtasks when the size of the second matrix multiplication calculation subtask is greater than the size of the systolic array. The first storage module and the second storage module are also used to ping-pong store the input data of the third matrix multiplication matrix calculation subtask; The first storage read control module is further configured to read the input data of the third matrix multiplication subtask from the first storage module or the second storage module, and input the input data of the third matrix multiplication subtask into the pulsating array; The pulsating array is specifically used to process the third matrix multiplication subtask based on the input data of the third matrix multiplication subtask, obtain the output result of the third matrix multiplication subtask, and send the output result of the third matrix multiplication subtask to the first accumulation module. The first accumulation module is further configured to accumulate the output results of multiple third matrix multiplication subtasks corresponding to the second matrix multiplication subtask to obtain the output result of the second matrix multiplication subtask, and send the output result of the second matrix multiplication subtask to the second accumulation module; The second accumulation module is used to accumulate the output results of multiple second matrix multiplication subtasks corresponding to the matrix multiplication calculation task to obtain the output result of the matrix multiplication calculation task.

5. The system according to claim 1, characterized in that, The hardware accelerator for the decoding stage also includes a third storage module, a fourth storage module, and a second storage read control module; The third storage module and the fourth storage module are used to ping-pong store the input data of the matrix multiplication vector calculation task when the size of the matrix multiplication vector calculation task is not greater than the size of the multiply-add tree. The second storage read control module is used to read the input data of the matrix multiplication vector calculation task from the third storage module or the fourth storage module, and input the input data of the matrix multiplication vector calculation task into the target multiply-add tree; The target multiply-add tree is used to process the matrix-vector multiplication task based on the input data of the matrix-vector multiplication task, and obtain the output result of the matrix-vector multiplication task.

6. The system according to claim 5, characterized in that, The hardware accelerator for the decoding stage also includes a third accumulation module; The hardware accelerator in the decoding stage is used to divide the matrix multiplication vector calculation task into multiple first matrix multiplication vector calculation subtasks when the size of the matrix multiplication vector calculation task is no larger than the size of the third storage module but larger than the size of the multiply-add tree. The third and fourth storage modules are also used for ping-pong storage of the input data of the first matrix-vector multiplication subtask; The second storage read control module is also used to read the corresponding number of input data of the first matrix multiplication vector calculation subtask from the third storage module or the fourth storage module based on the number of multiply-accumulate trees in the decoding stage hardware accelerator, and input the corresponding number of input data of the first matrix multiplication vector calculation subtask to the corresponding multiply-accumulate trees respectively. The multiply-accumulate tree is specifically used to process the first matrix multiplication vector calculation subtask based on the input data of the input first matrix multiplication vector calculation subtask, obtain the output result of the first matrix multiplication vector calculation subtask, and send the output result of the first matrix multiplication vector calculation subtask to the third accumulation module. The third accumulation module is used to accumulate the output results of multiple first matrix-vector calculation subtasks corresponding to the matrix-vector calculation task to obtain the output result of the matrix-vector calculation task.

7. The system according to claim 1, characterized in that, The computing node is specifically used for: Based on the length of the user-input lexical units in the model inference task and the size of the systolic array, the actual length of the input lexical units processed by the systolic array in the pre-filling stage is determined. Obtain the number of multiply-accumulate trees in the hardware accelerator of the decoding stage; Based on the actual length of the input tokens processed by the systolic array in the pre-filling stage and the number of multiply-accumulate trees in the hardware accelerator of the decoding stage, the ratio of the number of hardware accelerators in the target pre-filling stage to the number of hardware accelerators in the target decoding stage is determined. Based on the ratio of the number of target pre-filling stage hardware accelerators to the number of target decoding stage hardware accelerators, a target pre-filling stage hardware accelerator is determined from multiple pre-filling stage hardware accelerators, and a target decoding stage hardware accelerator is determined from multiple decoding stage hardware accelerators.

8. The system according to claim 7, characterized in that, The computing node is specifically used for: The length of the input tokens actually processed by the systolic array during the pre-filling stage is determined by the following formula: N3 = Ceil(N1 / N2) × N2 Where N3 is the length of the input tokens actually processed by the systolic array in the pre-filling stage, Ceil represents rounding up, N1 is the length of the user-input tokens in the model inference task, and N2 is the size of the systolic array.

9. The system according to claim 7, characterized in that, The computing node is specifically used for: Based on the actual input term length processed by the systolic array in the pre-filling stage and the number of multiply-accumulate trees in the hardware accelerator of the decoding stage, the ratio of the number of hardware accelerators in the target pre-filling stage to the number of hardware accelerators in the target decoding stage is determined as follows: Ceil(N3 / N2)×N:1 Where N3 is the actual length of the input tokens processed by the systolic array in the pre-filling stage, Ceil represents rounding up, N2 is the size of the systolic array, and N is the number of multiply-accumulate trees in the hardware accelerator of the decoding stage.

10. A method for processing model reasoning tasks, characterized in that, Applied to any computing node in the model inference task processing system according to any one of claims 1 to 9, the method includes: A model inference task is obtained, and a target pre-fill stage hardware accelerator is determined from multiple pre-fill stage hardware accelerators based on the model inference task, and a target decoding stage hardware accelerator is determined from multiple decoding stage hardware accelerators. The model inference task is then processed based on the target pre-fill stage hardware accelerator and the target decoding stage hardware accelerator, which work together in a cascaded manner.

Images