Large model reasoning-oriented hardware acceleration system and hardware acceleration method

Abstract

The invention belongs to the related technical field of hardware acceleration, and particularly relates to a hardware acceleration system and method oriented to large model reasoning, and the system comprises an FPGA reasoning acceleration module; according to calculation dimensions of different stages in the reasoning process, a linear calculation kernel, a multi-head attention calculation kernel, a feedforward neural network layer calculation kernel and a quantization normalization and residual connection kernel are divided; the four cores are distributed in different SLR regions of the FPGA, and the whole of each core is located in the same SLR region; hardware-level FIFO queues are integrated at data transmission interfaces among the cores; blocking type data flow control logic is adopted, each kernel autonomously starts or pauses calculation operation of the kernel only according to data availability of an input queue and space availability of an output queue of the kernel, a calculation result is directly pushed to an input queue of a downstream kernel, and a self-triggering execution chain based on data flow is formed. According to the method, the problems of relatively low reasoning efficiency and high resource consumption of the edge scene under the word element sequence can be solved.

Description

Technical Field

[0001] This invention belongs to the field of hardware acceleration technology, and more specifically, relates to a hardware acceleration system and method for large model inference. Background Technology

[0002] The Transformer architecture has driven breakthroughs in large-scale language models, but as its parameter size continues to increase, the inference process faces challenges such as high computational overhead and high latency. In recent years, edge scenarios have sought to achieve localized inference, requiring low latency, privacy protection, and high energy efficiency. Field-Programmable Gate Arrays (FPGAs), with their high parallelism, low power consumption, and high customizability, have shown broad application prospects in edge inference.

[0003] High-Level Synthesis (HLS) represents an automated design process that compiles the logical behavior of an algorithm from a high-level description language into a corresponding FPGA hardware implementation at the register-transfer level (RTL). Automated hardware development using HLS not only improves productivity but also increases design flexibility, making it widely used in large-scale scenarios requiring new functionality, such as machine learning, graphics processing, and domain-specific accelerators. Currently, major HLS tools on the market, such as Vivado HLS and Intel HLS, support C / C++ as the development language and describe hardware behavior through additional compiler directives or instructions.

[0004] As the computational demands of large-scale language models continue to grow, deploying compact and efficient inference architectures on edge devices is becoming increasingly important. However, existing accelerators are typically designed for batch inference and high throughput, thus being deployed on high-end platforms with abundant hardware resources and dedicated hardware components. They are mostly designed for data center scenarios with high request-level parallelism and sufficient budgets in terms of hardware, power consumption, and physical size; however, they may be inefficient for edge scenarios with word-based sequences and limited hardware resources. Furthermore, due to the nature of multi-head attention computation, significant time overhead still exists even without sparse attention computation or with inappropriate sparse attention computation. Based on the above analysis, the inference efficiency of current FPGA-based large-model hardware accelerators in edge scenarios still needs improvement. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a hardware acceleration system and hardware acceleration method for large model inference, which aims to solve the technical problems of low efficiency and serious resource consumption in edge scene inference under existing word meta-sequence.

[0006] To achieve the above objectives, according to one aspect of the present invention, a hardware acceleration system for large model inference is provided, comprising: an FPGA inference acceleration module for executing the entire inference process from input data to output inference results; The system is divided into four kernels based on the computational dimensions at different stages of the inference process: linear computation kernel, multi-head attention computation kernel, feedforward neural network layer computation kernel, and quantization normalization and residual connection kernel. These four kernels are distributed in different SLR regions of the FPGA, and each kernel is located in the same SLR region. Each kernel has multiple hardware units with the same computational dimension in its SLR region, and each hardware unit is reused through the time architecture. Data transfer and execution scheduling among the four kernels are achieved collaboratively through the following mechanisms: Hardware queue mechanism: A FIFO queue is integrated at the data transmission interface between kernels to transmit data to be processed between kernels with low latency; HLS blocking mechanism: It adopts blocking data flow control logic, so that when each kernel reads data, if its input queue is empty, it will automatically block and wait; when writing data, if its output queue is full, it will automatically block and wait. Each kernel can start or pause its calculation operation independently only according to the data availability of its input queue and the space availability of its output queue. The calculation result is directly pushed to the input queue of the downstream kernel, forming a self-triggering execution chain based on data flow.

[0007] Furthermore, the complete inference process architecture executed by the FPGA inference acceleration module is the Transformer architecture or a variant thereof.

[0008] Furthermore, the linear computation kernel is a computational unit adapted to n * m matrix projection computation, used to compute a set of Q, K, V vectors; where n is the dimension of the activation vector that serves as the input to the linear computation kernel, and n and m are also the dimensions of the two dimensions of the weight matrix in the matrix projection computation, respectively. The computing unit includes k matrix projection calculation parts and one summation part; when each vector calculation is performed, the calculation results of the k matrix projection calculation parts are summed by the summation part to realize parallel computing in the computing unit, where k represents the number of data transmission channels between the HBM located on the FPGA and the linear computing core.

[0009] Furthermore, when the linear computation kernel calculates a set of Q, K, V vectors, it first performs matrix projection calculation on the K and V vectors, and while inputting the calculation results into the multi-head attention computation kernel for position encoding, it then calculates the Q vector.

[0010] Furthermore, the feedforward neural network layer computation kernel is used for dimensionality-up projection calculation and dimensionality-down projection calculation of the feedforward neural network layer, as well as the calculation of nonlinear activation functions specific to the inference architecture to be executed; In the feedforward neural network layer computation kernel, each projection calculation is implemented by a dedicated computation unit that adapts to an n * m matrix projection calculation; where n is the dimension of the activation vector that serves as the input to the computation unit, and n and m are also the dimensions of the two dimensions of the weight matrix in the corresponding projection calculation. The computing unit contains k projection calculation parts and one summation part; when each corresponding projection calculation is performed, the calculation results of the k projection calculation parts are summed by the summation part to realize parallel computing in the computing unit, where k represents the number of data transmission channels between the HBM located on the FPGA and the computing kernel of the feedforward neural network layer.

[0011] Furthermore, the quantization, normalization, and residual connection kernels are used to quantize the activation vectors to be input into the linear computation kernel and the feedforward neural network layer computation kernel for projection computation; they are also used to perform residual connection and normalization processing based on the normalization function on the output results calculated by the multi-head attention computation kernel and the feedforward neural network layer computation kernel, respectively.

[0012] Furthermore, the multi-head attention computation kernel is used to perform position encoding on a set of K and V vectors output by the linear computation kernel, and to store the position-encoded K and V vectors in the K cache and V cache located in HBM; it is also used to divide the Q vector and the position-encoded K and V vectors into multiple heads, and to perform attention computation on one head corresponding to a set of Q, K, and V vectors of each head after division, thereby realizing multi-head attention computation as a whole.

[0013] Furthermore, the multi-head attention computation kernel includes: a word-level sparse attention prediction unit and a position encoding unit; The word-level sparse attention prediction unit adopts Top-K sparse attention calculation. In the multi-layer logical reasoning of a word, during the multi-head attention calculation of the first layer, the Q vector corresponding to the word, which is input from the linear calculation kernel to the multi-head attention calculation kernel, is combined with the K and V vectors of each word stored in the cache to perform multi-head attention calculation. The K words with the greatest influence of the K and V vectors in the first layer multi-head attention calculation result are marked. In the multi-head attention calculation of subsequent layers, only the marked K words are multi-head attention calculated. The position encoding unit is used to execute the various position encoding algorithms contained in the complete inference process architecture.

[0014] Furthermore, the part of the FPGA inference acceleration module that involves reading data from the high-bandwidth memory (HBM) all adopt burst transmission. Each time data is transmitted, the entire transmission bandwidth of the specific channel corresponding to the data to be transmitted between the kernel of this part and the high-bandwidth memory (HBM) is used.

[0015] According to another aspect of the present invention, a hardware acceleration method for large model inference is provided, comprising: Based on the large model inference architecture to be executed, a hardware acceleration system as described in any one of claims 1 to 9 is configured accordingly, and the large model inference is executed using the configured hardware acceleration system.

[0016] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention proposes a hardware acceleration system for large model inference. When a single SLR region cannot fully accommodate the entire inference process, the FPGA inference acceleration module divides the computation into several core functional modules based on different computational dimensions. These modules are distributed across different SLR regions. Data transmission and execution control between the functional cores are achieved through the following mechanisms: Hardware queue direct connection mechanism: A hardware-level FIFO queue direct connection interface is established between cores spanning SLR regions, enabling lock-free real-time transmission of data streams across SLR boundaries; HLS blocking control mechanism: Based on high-level synthesis tools... The automatically generated blocking flow control logic causes the kernel to block reading when the input queue is empty and block writing when the output queue is full. Through collaborative action, it achieves self-synchronization without dedicated control logic: each kernel autonomously starts and stops computation based solely on its local queue state (data readable / space writable), completing global execution synchronization across SLRs without a central scheduler or additional control circuitry. It constructs a data-driven execution chain: kernel computation is strictly triggered by the arrival of data in the input queue, and the output result is pushed to the downstream kernel queue in real time, forming a streaming execution pipeline across SLR regions. This mechanism allows all intermediate data to be directly transmitted between SLRs via hardware queues, completely avoiding access to off-chip memory. Therefore, this invention effectively solves the technical problems of low inference efficiency and severe resource consumption in edge scenarios under existing word meta-sequences.

[0017] 2. The hardware acceleration system provided by this invention adopts a Top-K sparse attention prediction mechanism in its multi-head attention calculation kernel. When calculating multi-head attention, it only calculates the K most important attention scores, which greatly reduces the amount of computation in multi-head attention calculation and improves the speed of inference.

[0018] 3. The hardware acceleration system provided by this invention designs a dedicated dimension hardware unit when matrix projection calculation is involved, and uses multiple HBM channels to transmit data. The parallel transmission of multiple channels improves transmission and calculation efficiency. When calculating matrix multiplication, the matrix multiplication is divided according to the number of HBM channels, calculated separately, and then summed. When transmitting data, burst transmission is used, and the amount of data transmitted each time reaches the maximum bandwidth that the HBM channel can transmit, thereby improving the efficiency of data transmission.

[0019] 4. This invention also proposes that the host functional module and the FPGA inference acceleration module transmit data to the high-bandwidth memory on the FPGA side via the PCIe bus, and read data from the high-bandwidth memory to realize data transmission between the two modules; this invention implements the inference process on the FPGA, reduces the delay of the first word output through design optimization, and greatly improves the throughput, which can reduce the use of hardware resources while ensuring fast inference output speed. Attached Figure Description

[0020] Figure 1 A schematic diagram of the overall structure of a hardware acceleration system for large model inference provided in an embodiment of the present invention; Figure 2 The host functional module structure and flowchart of the hardware acceleration system provided in the embodiments of the present invention; Figure 3 A schematic diagram of the structure of each kernel unit of the FPGA inference acceleration module of the hardware acceleration system provided in the embodiments of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 A hardware acceleration system for large model inference includes: an FPGA inference acceleration module for executing the entire inference process from input data to output inference results; The system is divided into four kernels based on the computational dimensions at different stages of the inference process: linear computation kernel, multi-head attention computation kernel, feedforward neural network layer computation kernel, and quantization normalization and residual connection kernel. These four kernels are distributed in different SLR regions of the FPGA, and each kernel is located in the same SLR region. Each kernel has multiple hardware units with the same computational dimension in its SLR region, and each hardware unit is reused through the time architecture. Data transfer and execution scheduling among the four kernels are achieved collaboratively through the following mechanisms: Hardware queue mechanism: A FIFO queue is integrated at the data transmission interface between kernels to transmit data to be processed between kernels with low latency; HLS blocking mechanism: It adopts blocking data flow control logic, so that when each kernel reads data, if its input queue is empty, it will automatically block and wait; when writing data, if its output queue is full, it will automatically block and wait. Each kernel can start or pause its calculation operation independently only according to the data availability of its input queue and the space availability of its output queue. The calculation result is directly pushed to the input queue of the downstream kernel, forming a self-triggering execution chain based on data flow.

[0023] In this embodiment, the FPGA inference acceleration module uses a lot of hardware resources, so a complete inference process cannot be placed in a single SLR region. Therefore, based on the computational dimensions of different stages in the inference process, the entire inference architecture is divided into four main cores, and these four cores are distributed in different SLR regions of the FPGA. Each of the four cores is located in one SLR region (logic block region) and will not be placed across SLR regions. After the cores are divided, one core corresponds to multiple hardware units with the same computational dimensions in one SLR region. Each hardware unit is reused through a time architecture, reducing the use of hardware resources.

[0024] The FPGA inference acceleration module contains multiple parallel processing cores; data transfer and execution scheduling between these cores are achieved collaboratively through the following mechanisms: Hardware queue mechanism: A (hardware-implemented) FIFO queue is directly integrated at the data transmission interface between kernels for low-latency transmission of data to be processed; HLS blocking mechanism: Utilizing High-Level Synthesis (HLS) tools, automatically generated blocking data flow control logic ensures that each kernel automatically blocks and waits when reading data if its input queue is empty, and automatically blocks and waits when writing data if its output queue is full. Through the synergy of the hardware queue mechanism and the HLS blocking mechanism, self-synchronization without dedicated control logic (no dedicated control program driving the inference process) is achieved. Each kernel autonomously starts or pauses its computational operations based solely on the data availability of its input queue and the space availability of its output queue, achieving kernel execution synchronization without relying on a central scheduler or additional dedicated state control logic. Data-driven execution: The computation execution of each kernel is directly triggered by the data arriving at its input queue. The kernel only executes when the data is ready, and the execution result is directly pushed to the input queue of the downstream kernel, forming a self-triggering execution chain based on data flow. The data flow between kernels adopts a streaming transmission mode, and the data is directly transferred between the hardware queues of adjacent kernels on demand and in real time. This significantly reduces the operation of writing intermediate results to off-chip memory and reading them again, thereby effectively reducing the bandwidth requirements and access latency of off-chip memory access.

[0025] In practice, in addition to the FPGA inference acceleration module, there is usually a host function module. The host function module includes an encoding / decoding unit, a word segmentation unit, and a sampler unit. The host function module is used to preprocess the input data (text) and transmit it to the FPGA inference acceleration module for inference to obtain the inference output result. Specifically, the sampler unit samples the inference result, the encoding / decoding unit decodes the sampled result, and the word segmentation unit processes it to obtain readable text.

[0026] The encoding and decoding unit of the host functional module includes: encoding the input text or inference output results using byte pair encoding rules; and decoding the inference sampling results using the word segmentation unit to output readable text. The functions of the word segmenter unit include: reading the word segmenter parameter file, loading lexical units and vocabulary merging scores; the word segmenter unit uses the BPE rule for word segmentation.

[0027] The sampler unit includes: creating a sampler and setting initial parameters for the sampler; the sampler unit uses kernel sampling to perform sampling.

[0028] The host functional module and the FPGA inference acceleration module communicate via an XRT interface. The FPGA inference acceleration module interacts with high-bandwidth memory, while the host functional module interacts and communicates with high-bandwidth memory via a PCIe bus. Data exchange between the two modules occurs through data interaction with the high-bandwidth memory. Data is transferred between the host functional module and the FPGA inference acceleration module via the PCIe bus to the high-bandwidth memory on the FPGA side, and data is read from the high-bandwidth memory to achieve data transfer between the two modules. This embodiment implements the inference process on the FPGA, reducing the latency of the first word output through design optimization, while significantly improving throughput, thus reducing hardware resource usage while ensuring fast inference output speed.

[0029] In summary, this embodiment provides a hardware acceleration method for large model inference. This method includes a host functional module and an FPGA inference acceleration module. The host functional module is responsible for encoding the input data, transmitting the encoded tokens to the FPGA inference acceleration module for inference, obtaining the inference results, and performing a series of operations such as sampling, decoding, and word segmentation on the inference results to obtain readable results. The FPGA inference acceleration module executes the entire process from input tokens to output inference results. Throughout the inference process, multiple identical logical inference layers are passed. Each layer includes operations such as normalization and quantization, matrix projection calculation of the data, multi-head attention calculation, feedforward neural network layer projection calculation, and finally, normalization and other processing before outputting the inference results.

[0030] In the hardware acceleration method provided in this embodiment, when the FPGA inference acceleration module is faced with the situation that a single SLR region cannot fully accommodate the entire inference process, it divides the computation into several core functional modules for matrix and projection calculations of different dimensions, based on the different computational dimensions. These several core functional modules are distributed in different SLR regions. Furthermore, the hardware acceleration system provided in this embodiment includes an FPGA inference acceleration module comprising parallel functional cores deployed across multiple SLR (Super Logic Region) regions. Data transmission and execution control between functional cores are achieved through the following mechanisms: Hardware queue direct connection mechanism: A hardware-level FIFO queue direct connection interface is established between cores across SLR regions to achieve lock-free real-time transmission of data streams between SLR boundaries; HLS blocking control mechanism: Based on blocking flow control logic automatically generated by high-level synthesis tools, the core blocks reading when the input queue is empty and blocks writing when the output queue is full; Self-synchronization without dedicated control logic is achieved through collaborative action: Each core autonomously starts and stops computation based solely on its local queue state (data readable / space writable), completing global execution synchronization across SLRs without a central scheduler or additional control circuitry; Construction of a data-driven execution chain: Core computation is strictly triggered by the arrival of data in the input queue, and the output result is pushed to the downstream core queue in real time, forming a streaming execution pipeline across SLR regions; The mechanism enables all intermediate data to be directly transmitted between SLRs via hardware queues, completely avoiding access to off-chip memory.

[0031] As a preferred implementation, the complete inference process architecture executed by the FPGA inference acceleration module is the Transformer architecture or a variant thereof.

[0032] As a preferred embodiment, the linear computation kernel is a computational unit adapted to n * m matrix projection computation, used to compute a set of Q, K, V vectors; where n is the dimension of the activation vector that serves as the input to the linear computation kernel, and n and m are also the dimensions of the two dimensions of the weight matrix in the matrix projection computation, respectively. The computing unit includes k matrix projection calculation parts and one summation part; when each vector calculation is performed, the calculation results of the k matrix projection calculation parts are summed by the summation part to realize parallel computing in the computing unit, where k represents the number of data transmission channels between the HBM located on the FPGA and the linear computing core.

[0033] As a preferred implementation, when the linear computation kernel calculates a set of Q, K, V vectors, it first performs matrix projection calculation on the K and V vectors, and while inputting the calculation results into the multi-head attention computation kernel for position encoding, it then calculates the Q vector.

[0034] The FPGA inference acceleration module makes reasonable use of spatial and temporal architectures for each core; when the linear computing core performs linear calculations on K and V and passes the data to the multi-head attention computing core for position encoding, it writes the data back to high-bandwidth memory and begins synchronous calculation of Q in the linear computing core.

[0035] As a preferred implementation, the feedforward neural network layer computation kernel is used for the feedforward neural network layer dimensionality-up projection calculation and dimensionality-down projection calculation, as well as the calculation of nonlinear activation functions specific to the inference architecture to be executed; In the feedforward neural network layer computation kernel, each projection calculation is implemented by a dedicated computation unit that adapts to an n * m matrix projection calculation; where n is the dimension of the activation vector that serves as the input to the computation unit, and n and m are also the dimensions of the two dimensions of the weight matrix in the corresponding projection calculation. The computing unit contains k projection calculation parts and one summation part; when each corresponding projection calculation is performed, the calculation results of the k projection calculation parts are summed by the summation part to realize parallel computing in the computing unit. k represents the number of data transmission channels between the HBM located on the FPGA and the computing kernel of the feedforward neural network layer.

[0036] The hardware acceleration method provided in this embodiment designs dedicated hardware units for matrix projection calculations and uses multiple HBM channels to transmit data. The parallel transmission of multiple channels improves transmission and calculation efficiency. When calculating matrix multiplication, the matrix multiplication is divided according to the number of HBM channels, calculated separately, and then summed. When transmitting data, burst transmission is used, and the amount of data transmitted each time reaches the maximum bandwidth that the HBM channel can transmit, thus improving the efficiency of data transmission.

[0037] As a preferred implementation, the normalization function used by the Tranformer architecture or its variants is implemented in the quantization, normalization, and residual connection kernel. This kernel is used to quantize the activation vectors to be projected into the linear computation kernel and the feedforward neural network layer computation kernel for computation; it is also used to perform residual connection and normalization based on the normalization function on the output results calculated by the multi-head attention computation kernel and the feedforward neural network layer computation kernel, respectively.

[0038] As a preferred embodiment, the multi-head attention computation kernel is used to perform position encoding on a set of K and V vectors output by the linear computation kernel, and to store the position-encoded K and V vectors in the K cache and V cache located in HBM; it is also used to divide the Q vector and the position-encoded K and V vectors into multiple heads, and to perform attention computation on a set of Q, K, and V vectors corresponding to each head after division, thereby realizing multi-head attention computation as a whole.

[0039] As a preferred implementation, the multi-head attention computation kernel includes: a word-level sparse attention prediction unit and a position encoding unit; The word-level sparse attention prediction unit adopts Top-K sparse attention computation. In the multi-layer logical reasoning of a word, during the multi-head attention computation of the first layer, the Q vector corresponding to the word, which is input from the linear computation kernel to the multi-head attention computation kernel, is combined with the K and V vectors of each word stored in the cache to perform multi-head attention computation. The K words with the greatest influence of the K and V vectors in the first layer multi-head attention computation result are labeled. In the multi-head attention computation of subsequent layers, only the labeled K words are multi-head attention computationed. The choice of K value is not limited to certain specific values ​​and can be changed according to the actual situation.

[0040] The positional encoding unit is used to execute various positional encoding algorithms contained in the complete inference process architecture. In implementation, multiple dedicated positional encoding hardware units corresponding to the specific inference architecture to be executed can be designed, and multiple positional encoding hardware units can be computed and processed in parallel when performing positional encoding on a set of vectors corresponding to each word.

[0041] The hardware acceleration method provided in this embodiment adopts the Top-K sparse attention prediction mechanism in the multi-head attention calculation kernel. When calculating multi-head attention, only the most important K attention scores are calculated, which greatly reduces the amount of computation in multi-head attention calculation and improves the speed of inference.

[0042] As a preferred implementation, the part of the FPGA inference acceleration module that involves reading data from the high-bandwidth memory (HBM) all adopt burst transmission. Each time data is transmitted, the entire transmission bandwidth of the specific channel corresponding to the data to be transmitted between the kernel of this part and the high-bandwidth memory (HBM) is used to fully improve the transmission efficiency.

[0043] To better illustrate the implementation of this invention, as follows: Figure 1 The figure shown is the overall structure diagram of the hardware acceleration system for large model inference proposed in this embodiment.

[0044] The off-chip memory of an FPGA is a high-bandwidth memory (HBM). After the host functional module preprocesses the data, it is transmitted to the HBM via the PCIe bus, waiting for the cores of the FPGA inference acceleration module to retrieve the data for inference.

[0045] Specifically, the hardware acceleration system for large model inference includes a host functional module and an FPGA inference acceleration module; The host module reads and preprocesses the weight data and transmits it to HBM for the FPGA inference acceleration module to read and use. It performs word segmentation and encoding on the prompt words input for generating text, and transmits the processed prompt words to HBM. The FPGA inference acceleration module reads the preprocessed results of the prompt words from HBM, performs inference, obtains the inference output, and writes the result data to be transmitted across modules into HBM. The host module reads the inference results stored in HBM, performs sampling and decoding operations, selects a word from the inference results according to a reasonable sampling method as the final output, and decodes it into readable text.

[0046] The FPGA inference acceleration module is used to execute the complete inference process. First, it reads the known terms of the inference sequence from HBM, and performs the entire inference process on the terms, including normalization, quantization, matrix projection calculation, multi-head attention calculation, feedforward neural network calculation, residual connection and other operations. After the complete inference process is calculated, the result is written into HBM and waits for the host functional module to read it.

[0047] The hardware-accelerated method executes word terms sequentially during the entire inference process, requiring the inference result of each word term to be obtained and the inference result to be output in turn. Since each inference requires the use of the entire sequence of known words terms up to the current time step, the result of the entire inference process will affect the result of subsequent inferences, and each result cannot exist independently.

[0048] like Figure 2 The diagram shown is a structural diagram and flowchart of the host functional modules.

[0049] The host functional module will execute the corresponding tasks, and the data of each functional unit in the host functional module is stored in the corresponding structure.

[0050] The system reads the parameter file required to build the word segmenter from the hard drive. Based on the word segmenter file read from the hard drive, it loads the vocabulary and word frequency scores and constructs the mapping relationship. Then, it constructs a sampler and uses kernel sampling (Top-p) to obtain the input lexical units for the next inference based on the inference results.

[0051] During the inference process, the host functional module first constructs a tokenizer and a sampler to encode the input characters into corresponding word sequences. It then sets the FPGA kernel parameters and calls the FPGA kernel, passing the word sequences as parameters for inference. After inference, the inference result is read from the HBM on the FPGA side and transferred to the host. The inference result is then normalized and processed before sampling. Sampling analysis yields the next word for inference, and the current word is decoded to obtain the output result in string form.

[0052] like Figure 3 The diagram shows the design of each core of the FPGA inference acceleration module.

[0053] The normalization, quantization, and residual connection kernel (RQA kernel) reads weight data and embedding vectors from HBM, performs normalization and quantization on the embedding vectors, and then passes them to the linear computation kernel to calculate the Q, K, and V vectors. Since the kernels are distributed in different SLR regions, they cannot communicate directly and need to use the HLS::stream mechanism, i.e., the hardware queue mechanism, to transmit data across regions in the form of a stream.

[0054] When calculating Q, K, and V vectors, a customized dedicated hardware unit is used based on the calculation dimension. Weight data is transmitted through multiple HBM ports and is divided into multiple blocks for parallel computation. The calculation results of multiple blocks are accumulated to obtain the final result, which is then passed to the multi-head attention calculation kernel.

[0055] After receiving data, the multi-head attention computation kernel performs rotational position encoding on the Q and K vectors, and stores the encoded vectors and the V vector in a key-value cache for subsequent multi-head attention computation. After completing the multi-head attention computation at layer 0, it selects the words corresponding to the top k most important attention scores and their corresponding positions in the word sequence for TOP-K sparse attention prediction computation. After the multi-head attention computation and the weighted summation of the V vector are completed, the results are again transmitted to the linear computation kernel through a hardware queue, where the hardware unit is reused to perform matrix projection computation for the attention output.

[0056] The result of the linear computation kernel is fed into the RQA kernel (normalization, quantization, and residual connection kernel) for residual connection and normalization. The result, along with the activation vector, is then fed into the feedforward neural network computation kernel for matrix projection dimensionality upscaling and nonlinear function activation calculations. Finally, dimensionality downscaling is performed to obtain the features. The matrix projection calculation here is identical to the matrix projection calculation in the linear computation kernel, except for the different computational dimension.

[0057] The result from the feedforward neural network kernel is then fed back into the RQA kernel for residual connections and normalization calculations. After the calculation is complete, this process is repeated, executing the above inference process n times until n inferences are completed. Then, the result is written back into the RQA kernel for final calculation. After final normalization, projection calculations, and other operations, the output result is obtained and written into HBM, awaiting reading by the host functional module.

[0058] Example 2 A hardware acceleration method for large model inference includes: Based on the large model inference architecture to be executed, a hardware acceleration system as described in Example 1 is configured accordingly, and the large model inference is executed using the configured hardware acceleration system.

[0059] Specifically, the first step is to complete the specialized configuration of the hardware acceleration system for the large model inference architecture to be executed. For example, this involves configuring the nonlinear activation functions specific to the inference architecture within the computational kernel of the feedforward neural network layer. Afterward, the workflow of the hardware acceleration method for large model inference includes: Step S1: The host functional module reads the weight data, allocates it reasonably, and then transmits it to the off-chip memory HBM on the FPGA for storage.

[0060] Step S2: The host functional module uses a word segmenter to segment and encode the input prompt words, and then passes the encoded tokens into HBM, waiting for the FPGA inference acceleration module to read and execute the inference.

[0061] Step S3: The FPGA inference acceleration module reads the tokens and completes the entire inference process through each core. The RQA core first reads the tokens and converts them into embedding vectors. After performing quantization, normalization, and other operations, it transmits the vectors to the linear computation core. The linear computation core performs matrix projection calculations on the tokens to obtain Q, K, and V vectors, and transmits them to the multi-head attention computation core. The multi-head attention computation core performs positional encoding on the Q and K vectors and stores the K and V vectors in the designed key-value cache storage space, waiting to be retrieved when calculating attention. After the first layer of attention calculation, the top k tokens are selected based on their attention scores for sparse attention calculation in subsequent layers. The calculation results of the multi-head attention computation core are transmitted to the linear computation core for projection calculation to obtain the output, and then written back to the RQA core for residual connection and normalization. The processed results are transmitted to the feedforward neural network core for dimensionality-up projection calculation, nonlinear function activation calculation, and dimensionality-down projection calculation. The results are then written back to the RQA core for residual connection and normalization.

[0062] Step S4: Repeat step S3 n times, where n is the number of layers in the model. However, when calculating multi-head attention, it is not necessary to calculate all attentions again, but only the k most important attentions. After the calculation is completed, it is not necessary to select the word units corresponding to the top k most important attentions and their word unit sequence positions again.

[0063] Step S5: After the n-layer inference calculation is completed, the final result is normalized again and matrix projection is calculated to obtain the inference result and write it into HBM to wait for the host functional module to read it.

[0064] Step S6: The host functional module reads the inference results from HBM, calls the sampler and word segmenter to perform sampling and word segmentation, and then calls the decoding unit to decode the results and output readable text.

[0065] The relevant technical solutions are the same as above, and will not be repeated here.

[0066] In summary, the hardware acceleration system of this invention comprises two main parts: a host functional module and an FPGA inference acceleration module. The host functional module encodes the input data, transmits the encoded tokens to the FPGA inference acceleration module for inference, and obtains the inference results. It then performs a series of operations on the inference results, including sampling, decoding, and word segmentation, to obtain readable results. The FPGA inference acceleration module executes the entire process from input tokens to output inference results. This process involves multiple identical inference layers, each including normalization, quantization, matrix projection calculation, multi-head attention calculation, and feedforward neural network layer projection calculation. Finally, after normalization and other processing, the inference results are output. The host functional module and the FPGA inference acceleration module transmit data to the high-bandwidth memory on the FPGA via the PCIe bus, and read data from the high-bandwidth memory to achieve data transmission between the two modules. This invention implements the inference process on the FPGA, reduces the latency of the first token output through design optimization, and significantly improves throughput, reducing hardware resource usage while ensuring fast inference output speed.

[0067] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hardware acceleration system for large model inference, the system comprising: Comprise: An FPGA inference acceleration module for performing an entire complete inference process from input data to output inference results; Wherein, according to the calculation dimension of different stages in the inference process, it is divided into linear calculation kernel, multi-head attention calculation kernel, feedforward neural network layer calculation kernel and quantization normalization and residual connection kernel; The four kernels are distributed in different SLR areas of the FPGA, and the whole of each kernel is located in the same SLR area, and each kernel has a plurality of hardware units with the same calculation dimension in the SLR area where it is located, and each hardware unit is multiplexed by time architecture; The data transmission and execution scheduling between the four kernels are cooperatively realized through the following mechanisms: Hardware queue mechanism: FIFO queues are integrated at the data transmission interfaces between the kernels for low-latency transmission of data to be processed between the kernels; HLS blocking mechanism: A blocking data flow control logic is adopted, so that each kernel is automatically blocked when its input queue is empty when reading data; When writing data, if its output queue is full, it is automatically blocked; Each kernel autonomously starts or pauses its calculation operation according to the availability of data in its input queue and the availability of space in its output queue, and the calculation result is directly pushed to the input queue of the downstream kernel, forming a self-triggered execution chain based on data flow.

2. The hardware acceleration system of claim 1, wherein, The complete inference process performed by the FPGA inference acceleration module is a Transformer architecture or a variant thereof.

3. The hardware acceleration system of claim 1, wherein, The linear calculation kernel is a calculation unit adapted to matrix projection calculation of n*m, used to calculate a group of Q, K, V vectors; Wherein, n is the dimension of the activation vector as the input of the linear calculation kernel, and n and m are the dimensions of the two dimensions of the weight matrix in the matrix projection calculation; The calculation unit contains k matrix projection calculation parts and one accumulation summation part; When each vector calculation is performed, the calculation results of the k matrix projection calculation parts are accumulated and summed by the accumulation summation part to realize parallel calculation in the calculation unit, and k represents the number of data transmission channels for data transmission between the HBM on the FPGA and the linear calculation kernel.

4. The hardware acceleration system of claim 3, wherein, When calculating a group of Q, K, V vectors, the linear calculation kernel first performs matrix projection calculation on K and V vectors, and at the same time, calculates Q vector after transmitting the calculation result to the multi-head attention calculation kernel for position encoding.

5. The hardware acceleration system of claim 1, wherein, The feedforward neural network layer calculation kernel is used for feedforward neural network layer dimension projection calculation and dimension reduction projection calculation, and non-linear activation function calculation dedicated to the inference architecture to be executed; Wherein, each projection calculation in the feedforward neural network layer calculation kernel is realized by a special calculation unit adapted to matrix projection calculation of n*m; Wherein, n is the dimension of the activation vector as the input of the calculation unit, and n and m are the dimensions of the two dimensions of the weight matrix in the corresponding projection calculation; The calculation unit comprises k projection calculation parts and one accumulation summation part; the calculation results of the k projection calculation parts are accumulated and summed by the accumulation summation part during the execution of corresponding projection calculation, realizing parallel calculation in the calculation unit, and k represents the number of data transmission channels for data transmission between the HBM on the FPGA and the feedforward neural network layer calculation kernel.

6. The hardware acceleration system of claim 1, wherein, The quantization, normalization and residual connection kernel is used for quantizing each activation vector to be input into the linear calculation kernel and the feedforward neural network layer calculation kernel for projection calculation; and is also used for performing residual connection and normalization processing based on a normalization function on the results calculated and output by the multi-head attention calculation kernel and the feedforward neural network layer calculation kernel respectively.

7. The hardware acceleration system of claim 1, wherein, The multi-head attention calculation kernel is used for performing position encoding on a group of K and V vectors output by the linear calculation kernel, and storing the K and V vectors after position encoding in the K cache and V cache of the HBM respectively; and is also used for performing multi-head division on the Q vector and the K and V vectors after position encoding respectively, performing one-head attention calculation on a group of Q, K and V vectors corresponding to each head after division, and realizing multi-head attention calculation as a whole.

8. The hardware acceleration system of claim 7, wherein, The multi-head attention calculation kernel comprises a word-level sparse attention prediction unit and a position encoding unit. The word-level sparse attention prediction unit adopts Top-K sparse attention calculation; in the multi-layer logical reasoning of one word, during the multi-head attention calculation of the first layer, the Q vector corresponding to the word and input into the multi-head attention calculation kernel by the linear calculation kernel is combined with the K and V vectors of each word stored in the cache to perform multi-head attention calculation, and the K words with the greatest influence on the K and V vectors in the first layer multi-head attention calculation result are marked, and only the K words are subjected to multi-head attention calculation during the multi-head attention calculation of the subsequent layers. The position encoding unit is used for performing various position encoding algorithms contained in the complete reasoning process architecture.

9. The hardware acceleration system of claim 1, wherein, The part of the FPGA reasoning acceleration module related to reading data from the high bandwidth memory HBM adopts burst transmission, and each time data is transmitted, the full transmission bandwidth of the specific channel corresponding to the to-be-transmitted data between the kernel corresponding to the part and the high bandwidth memory HBM is used.

10. A hardware acceleration method for large model inference, characterized in that, The method comprises the following steps: According to the large model reasoning architecture to be executed, the hardware acceleration system as claimed in any one of claims 1 to 9 is configured, and the hardware acceleration system after configuration is used to execute large model reasoning.

Images