An acceleration system on an edge computing device performing causal attention computation

Abstract

This application discloses an acceleration system and method for performing causal attention computation on an edge computing device. The system includes: a hardwired scheduling controller for generating a control signal sequence based on the lower triangular structure of causal attention; a matrix multiplication unit for reading corresponding query blocks and key blocks according to the control signal sequence, performing block matrix multiplication operations, and calculating score blocks; a vector computation unit configured to receive the score blocks and value blocks in a streaming manner and perform Softmax normalization and weighted accumulation operations; and a triangular mask generation circuit connected to the hardwired scheduling controller and integrated between the matrix multiplication unit and the vector computation unit, for masking elements in the score block whose row index is less than the column index when the data block indicated by the control signal sequence is a diagonal block. The system of this application reduces the computational load and power consumption on the edge computing device and improves resource utilization.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip control technology, and in particular to an acceleration system and method for performing causal attention computation on an edge computing device. Background Technology

[0002] In the field of AI chip design and accelerated computing, the causal attention mechanism in Transformer neural networks is the core computational unit for autoregressive generation tasks (such as text continuation in large language models). This mechanism requires the model to focus only on the current position and preceding information in the sequence when generating the current output. This constraint is reflected in the algorithm by applying a causal mask to the attention score matrix, that is, setting the upper triangular region (corresponding to "future" information) of an N×N matrix to negative infinity. This results in the effective attention matrix (the result of QK^T and its weight matrix P after Softmax normalization) having a strict lower triangular structure, in which theoretically about half of the elements are invalid. The subsequent output calculation O = P·V also has structural sparsity. This computational characteristic provides a clear direction for hardware optimization, that is, when processing such tasks, ideal hardware should be able to directly avoid rather than passively process these invalid upper triangular regions.

[0003] To address the high complexity of attention computation, existing technologies primarily focus on software optimization. Software optimization approaches, such as FlashAttention and its derivatives (like Skip Softmax), are designed with the core principle of extreme tuning for general-purpose GPU architectures. They utilize sophisticated block-based computation, dynamic mask generation, and fusion kernel techniques to keep the computation process largely within the GPU's cache, aiming to reduce access to high-bandwidth memory and mask its latency.

[0004] However, when adapting to edge computing scenarios, these existing solutions rely on complex scheduling to "mask" rather than "physically eliminate" inefficient computation and data movement. They still require resource allocation for the complete N×N matrix, and their dynamic block partitioning and conditional skipping strategies generate significant control overhead and branch prediction burdens. Furthermore, low-precision computation may introduce biased numerical errors. More importantly, these solutions heavily depend on the GPU's many-core parallel architecture and large-capacity on-chip shared memory. Their complex thread synchronization and data movement logic is severely mismatched with the edge chip's design paradigm of "single large core, small cache, and simple control," resulting in low hardware utilization and poor energy efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide an acceleration system and method for performing causal attention computation on an edge computing device, thereby reducing the computational load and power consumption on the edge computing device and improving resource utilization.

[0006] To achieve the above objectives, this application provides an acceleration system on an edge computing device performing causal attention computation, comprising: A hardwired scheduling controller is used to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a first nesting manner, based on the lower triangular structure of causal attention, wherein the first nesting manner is a row-first-column nesting manner. The matrix multiplication unit, connected to the hardwired scheduling controller, is used to read the corresponding query block and key block according to the control signal sequence, perform block matrix multiplication operations, and calculate the score block; The vector computation unit, connected to the matrix multiplication unit, is configured to receive the score blocks and value blocks in a streaming manner and perform Softmax normalization and weighted accumulation operations. A triangular mask generation circuit, connected to the hard-wired scheduling controller, is integrated between the matrix multiplication unit and the vector calculation unit. It is used to mask the elements in the data block whose row index is less than the column index when the data block indicated by the control signal sequence is a diagonal block. On-chip memory unit, connected to the hard-wired scheduling controller, includes: A query buffer is used to cache query blocks, a key / value buffer is used to cache key / value blocks, and an output buffer is used to cache output accumulation blocks.

[0007] Furthermore, the hardwired scheduling controller is a hardware state machine with pre-set fixed state transition logic, configured to generate a control signal sequence for accessing data blocks located in the lower triangular structure in a first nested manner based on block coordinates, block size, and sequence length.

[0008] Furthermore, the triangular mask generator is configured to perform arithmetic gating on the multiplication-accumulation path corresponding to the upper triangular position in the data block when the data block indicated by the control signal sequence is a diagonal block.

[0009] Furthermore, the triangular mask generator is configured to set the elements at the upper triangular positions in the scoring block to negative infinity and set the corresponding weights to zero when the data block indicated by the control signal sequence is a diagonal block.

[0010] Furthermore, the vector computation unit includes a dedicated register set maintained for each data row; the dedicated register set includes a row maximum register, a normalized denominator cumulative l register, and an output cumulative register; the dedicated register set is configured to perform row maximum update, normalized denominator cumulative l update, and output cumulative update for each input score block.

[0011] Furthermore, the vector computation unit is configured to perform the following updates: m_new = max(m_old, max(s)); alpha = exp(m_old - m_new); p = exp(s - m_new); l_new = alpha · l_old + sum(p); o_new = alpha · o_old + (p · V); O = o_final / l_final; Where s is the logits segment vector in the scoring block, m_old is the historical cumulative maximum value of the row, initially set to -∞; m_new is the new cumulative maximum value of the row after including the current segment; alpha is the scaling factor; p is the unnormalized weight vector under m_new; V is the value vector; l_old is the historical cumulative value of the softmax denominator of the row, initially set to 0; l_new is the updated cumulative value of the denominator; o_old is the historical cumulative vector of the output numerator of the row; o_new is the updated cumulative output numerator; l_final and o_final are the final cumulative denominator and numerator of the row after processing all accessed segments, respectively; and O is the final attention output vector of the row.

[0012] Furthermore, at least one of the query buffer and the key / value buffer adopts a double buffer structure, enabling prefetching and overlap between external memory reads and matrix multiplication unit calculations.

[0013] Furthermore, the matrix multiplication unit is a fixed-size pulsating array of 64×64, and a zero-padding and vector processing mask strategy is used to process boundary blocks in the case of insufficient blocks.

[0014] Furthermore, the hardwired scheduling controller is also configured to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a second nesting manner, based on the lower triangular structure of causal attention, so that each loaded key-value block in the on-chip storage unit can be continuously computed with multiple different query blocks before it is replaced out of the on-chip storage unit; the second nesting manner is a column-row-first nesting manner.

[0015] Furthermore, the hardwired scheduling controller includes a counter and a comparator. By comparing control signal sequences, it generates a control signal sequence for accessing data blocks that satisfy the condition that column index ≤ row index; and when column index = row index, it determines the invalid position of the upper triangular structure by comparing the row and column indices within the block.

[0016] Furthermore, the triangular mask generation circuit includes gating logic, which is used to compare the row and column indices within the block and to padded or disable scoring elements whose column indices are greater than their row indices before writing back or sending to the VPU.

[0017] Furthermore, the matrix multiplication unit is configured to sequentially output the score vector of each column in a streaming manner when calculating the score block, so that the vector calculation unit obtains the local maximum vector of each row by performing element-wise maximum value operation.

[0018] Furthermore, it is also equipped with multiplexing logic for handling multi-head attention. This multiplexing logic enables computation on multiple heads by serial multiplexing or interleaved time-slice multiplexing when the resources of a single matrix multiplication unit are limited.

[0019] To achieve the above objectives, this application also provides a hardware acceleration method for performing causal attention computation on an edge computing device, employing the acceleration system on the edge computing device performing causal attention computation as described in any one of the above claims, comprising: Divide the query data, key data, and value data into fixed-size blocks along the sequence dimension; In response to performing causal attention computation, the hardwired scheduling controller generates a sequence of control signals to access data blocks located in the lower triangular structure in a first nested manner; According to the control signal sequence, the matrix multiplication unit reads the corresponding query block and key block, performs block matrix multiplication operation, and calculates the score block; When the data block indicated by the control signal sequence is a diagonal block, the triangular mask generation circuit will mask the elements in the data block whose row index is less than the column index. The scoring blocks are normalized in the vector computation unit using an online normalization row-by-row update method, and the weighted results of the value blocks are accumulated and output.

[0020] The acceleration system and method provided in this application for edge computing devices performing causal attention computation avoids all computation and data transfer in the upper triangular invalid region of causal attention from the source by using a hard-wired scheduling controller, which greatly reduces computational complexity and memory bandwidth requirements.

[0021] The acceleration system and method provided in this application for edge computing devices performing causal attention computation, wherein the matrix multiplication unit and vector processing unit only process valid lower triangular data blocks, improve resource utilization, and significantly reduce control overhead and dynamic power consumption, thereby improving the overall energy efficiency ratio.

[0022] The acceleration system and method for performing causal attention computation on edge computing devices provided in this application offer an optimization path designed specifically for edge computing that is completely different from general GPU solutions. It does not rely on massive parallelism, high-bandwidth memory, and complex thread synchronization. Instead, it transforms fixed computation patterns into deterministic hardware behaviors through the co-design of algorithms and hardware. It is particularly suitable for deploying large language models on edge chips with few computing cores, limited storage resources, and strict power consumption constraints.

[0023] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of the acceleration system on the edge computing device that performs causal attention computation according to Embodiment 1 of this application; Figure 2 This is a schematic diagram of the block-level lower triangle traversal; Figure 3 This is a calculation diagram for the lower triangle of the diagonal piece; Figure 4 A schematic diagram of the hardware data flow for online Softmax and fused output; Figure 5 This is a schematic diagram of double buffering and prefetch timing; Figure 6 State machine transition diagram for performing causal attention computation Figure 7 This is a schematic diagram of the architecture of an acceleration system on an edge computing device performing causal attention computation according to Embodiment 2 of this application; In the diagram: 100 - Hardwired scheduling controller, 200 - Matrix multiplication unit, 300 - Vector calculation unit, 400 - Triangle mask generation circuit, 500 - On-chip memory unit, 301 - Row maximum value register, 302 - Normalized denominator cumulative sum register, 303 - Output accumulator register, 501 - External memory, 502 - Online cache. Detailed Implementation

[0025] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0026] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the terms "one" and "multiple" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0029] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0030] Example 1 One embodiment of this application provides an acceleration system on an edge computing device performing causal attention computation. Through a fixed-size M×M matrix multiplication unit, coupled with a vector processing unit, on-chip block memory, and a hard-wired deterministic scheduler, it computes only the effective lower triangular blocks of the attention matrix in a down-triangular traversal order, and performs online softmax numerical updates and instantaneous multiplication and addition with V at the block level. This avoids materializing the complete attention matrix and significantly reduces computational and bandwidth requirements. Figure 1 As shown, it includes: The hardwired scheduling controller 100 is used to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a first nesting manner, based on the lower triangular structure of causal attention, wherein the first nesting manner is a row-first-column nesting manner. In the embodiments of this application, the hardwired scheduling controller is not a programmable kernel, but a hardware state machine with pre-set fixed state transition logic.

[0031] Specifically, the hardwired scheduling controller generates a control signal sequence for accessing data blocks located in the lower triangular structure in a first nested manner based on block coordinates, block size, and sequence length. That is, it generates a deterministic, branchless control signal sequence that only accesses data blocks whose block coordinates satisfy column index j ≤ row index i, thereby directly skipping all data blocks in the upper triangular structure at the hardware level and eliminating the control overhead of mask metadata query, condition judgment, and association.

[0032] In this embodiment, the hardwired scheduling controller includes a counter and a comparator. By comparing the control signal sequence, it generates a control signal sequence for accessing data blocks that satisfy column index j ≤ row index i. When column index j = row index i, it determines the invalid position of the upper triangular structure by comparing the row and column indices within the block.

[0033] In other embodiments, the hardwired scheduling controller 100 is further configured to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a second nested manner, based on the lower triangular structure of causal attention, so that each loaded key-value block in the on-chip storage unit can be computed consecutively with multiple different query blocks before it is replaced out of the on-chip storage unit; the second nested manner is a column-row-first nested manner; to improve the reuse rate of key / value blocks without receiving or reading column-wise interval vectors, sparse index metadata, or derived data preprocessed therefrom used to describe the mask.

[0034] It is also equipped with multiplexing logic for handling multi-head attention. This multiplexing logic enables computation on multiple heads by serial multiplexing or staggered time slice multiplexing when the resources of a single matrix multiplication unit are limited.

[0035] The matrix multiplication unit 200, or MMU, is connected to the hard-wired scheduling controller and is used to read the corresponding query block and key block according to the control signal sequence, perform block matrix multiplication operations, and calculate the score block.

[0036] In this embodiment of the application, the matrix multiplication unit is A fixed-size pulsating array is used to handle boundary blocks by employing zero-padding and vector processing masking strategies when there are insufficient blocks.

[0037] Understandably, the matrix multiplication unit uses the following formula to calculate the scoring blocks. : ; in, This is the score block with row index i and column index j. For the query block with row index i. This is the key block with column index j.

[0038] In the embodiments of this application, for incomplete blocks at the end of the sequence, a strategy combining zero padding and the mask of the vector processing unit is adopted to ensure numerical equivalence while keeping the design of the matrix multiplication unit simple.

[0039] In other implementations, the matrix multiplication unit may also be another M×M fixed-size pulsating array.

[0040] In this embodiment of the application, a multiplexing logic for processing multi-head attention is also configured. The multiplexing logic realizes the computation of multiple heads by serial multiplexing or interleaved time slice multiplexing when the resources of a single matrix multiplication unit are limited.

[0041] In this embodiment of the application, the matrix multiplication unit 200 is further configured to sequentially output the score vector of each column in a streaming manner when calculating the score block, so that the vector calculation unit obtains the local maximum value vector of each row by performing element-wise maximum value operation.

[0042] For example, such as Figure 2 As shown, Figure 2 Represent matrices T and W using a complete N×N square matrix, with T and W divided into N rows and columns (e.g., 64). Following the rule of matrix multiplication unit 200, rows of T are multiplied and added to columns of W. A matrix can be multiplied by N rows and N columns at a time, i.e., N × k.

[0043] Traditional method: All blocks of T (A / B / C / D) are calculated with all blocks of W (a / b / c / d), and the gray part of the output is "the calculated but ultimately masked invalid region".

[0044] In the scheme of this application: the blocks of T (A / B / C / D) are read as long as they are in the same condition as the blocks of W (a / b / c / d) when j ≤ i. The hard-wired scheduling controller 100 skips invalid blocks that are not loaded or calculated, so only half of the input is needed, and there are fewer invalid calculations.

[0045] The triangular mask generation circuit 400 is integrated between the matrix multiplication unit 200 and the vector calculation unit 300. It is used to mask the elements in the data block whose row index is less than the column index when the data block indicated by the control signal sequence is a diagonal block. In this embodiment of the application, the triangular mask generator is configured to implement arithmetic gating logic on the multiplication-addition path corresponding to the upper triangular position in the scoring block when the data block indicated by the control signal sequence is a diagonal block. The arithmetic gating logic is used to compare the row and column indices within the block, and to fill or disable the scoring elements whose column index is greater than the row index before writing back or sending to the vector processing unit, so as to mask the elements in the scoring block whose row index is less than the column index.

[0046] In this embodiment of the application, when the data block indicated by the control signal sequence is a diagonal block, the element at the upper triangular position in the scoring block is set to negative infinity, and the corresponding weight is set to zero.

[0047] The vector computation unit 300, or VPU, is connected to the matrix multiplication unit and is configured to receive the score blocks and value blocks in a streaming manner, and perform Softmax normalization and weighted accumulation operations. Understandably, the online Softmax process of vector computation unit 300 only calculates the effective lower triangular region, and its output attention weight matrix P is a lower triangular matrix (the theoretical value for the upper triangular region is zero). In subsequent... In the multiplication and accumulation phase, following the same lower triangular block coordinate rule (j ≤ i), only the lower triangular block of the attention weight matrix P and its corresponding value matrix block V are calculated. The process is as follows: In sparse weight-aware data scheduling, when the controller schedules V_j of the value matrix block V, its triggering condition is related to the matrix multiplication unit 200. The phases remain consistent, meaning that V_j is only loaded if there is a query block Q_i with row index i ≥ column index j that needs to be processed. This avoids loading unnecessary value matrix block V data for the upper triangular region with all-zero weights from the outset.

[0048] In this embodiment of the application, when the vector processing unit 300 or the matrix multiplication unit 200 performs the accumulation (P_ij × V_j) of the score block and the value block, it can activate the corresponding optimization mode according to the type of the score block P_ij: When the scoring block is a diagonal block (i == j), that is, the scoring block P_ii is a lower triangular matrix. The matrix multiplication unit 200 can be configured to a sparse computation mode, that is, skip the multiplication and addition operations on the upper triangular zero weight, or the vector processing unit 300 can directly mask the V_j rows of data corresponding to the zero weight when organizing the data stream.

[0049] When the scoring block is a non-diagonal block (i > j): that is, the scoring block P_ij is a dense block, it is performed according to the regular matrix multiplication.

[0050] This ensures that with It achieves the same computational efficiency across all stages, saving on computation and memory access throughout the entire process from input to output.

[0051] For example, such as Figure 3 As shown, Figure 3 The attention weight matrix P and the value matrix V are represented by a complete N×N square matrix. The attention weight matrix P and the value matrix V are divided into N rows / columns (e.g., 64). According to the rules of matrix multiplication, the rows of P and the columns of V are multiplied and added together. A matrix multiplication unit 200 can perform N rows and N columns multiplication at a time.

[0052] In traditional technical solutions, all blocks (A / B / C / D) of the attention weight matrix P need to be calculated together with all blocks (a / b / c / d) of V.

[0053] In the technical solution of this application, the attention weight matrix P is a lower triangular matrix, which is divided into N (64) blocks. Only the valid blocks Ad / Bd / Cd / Dd need to be read for calculation. When j ≤ i, the blocks are read and skipped by the hard-wired scheduling controller. Invalid blocks are not loaded or calculated.

[0054] In this embodiment of the application, the vector calculation unit 300 includes a dedicated register group maintained for each data row; the dedicated register group includes a row maximum value register 301, a normalized denominator cumulative l register 302, and an output cumulative register 303; the dedicated register group is configured to perform row maximum value update, normalized denominator cumulative l update, and output cumulative update for each input score block.

[0055] The vector computation unit is configured to perform the following updates: m_new = max(m_old, max(s)); alpha = exp(m_old - m_new); p = exp(s - m_new); l_new = alpha · l_old + sum(p); o_new = alpha · o_old + (p · V); O = o_final / l_final; Where s is the current logits segment vector being processed, which in the hardware implementation usually comes from a certain score block; max(s) is the maximum value in the current segment logits segment vector (the maximum value is reduced for the vector s).

[0056] m_old is the historical cumulative maximum value of this row (the maximum value of the row saved after the previous round / segment of processing), initially set to -∞.

[0057] m_new is the new cumulative maximum value of the row after including the current segment: m_new = max(m_old, max(s)), which serves as a new numerical stability benchmark.

[0058] alpha is the historical scaling factor (baseline switching factor), alpha = exp(m_old - m_new), with a value range of (0,1).

[0059] P is the unnormalized weight vector under the new baseline m_new, p = exp(s - m_new) (element-wise), which corresponds to the expression of the softmax molecule in its numerically stable form.

[0060] l_old is the historical cumulative value of the softmax denominator of this row (which can also be understood as the cumulative value of the "unnormalized weighted sum"), and is initially 0.

[0061] l_new is the updated cumulative value in the denominator; l_new = + sum(p), where sum(p) is the sum of all elements in the current segment.

[0062] o_old is the historical "numerator cumulative vector" (not yet divided by the denominator) output for this line, with the dimension being the value dimension.

[0063] o_new is the cumulative value of the updated output numerator: .in Similarly, the historical accumulation is converted to the new benchmark m_new, where V is a value vector.

[0064] l_final / o_final is the final cumulative result of the denominator / numerator of the line after processing all visited segments (e.g., traversing all j≤i that satisfy the causal constraint and the valid positions in the diagonal block).

[0065] O is the final attention output vector for this row, and the vector is divided element by element by the scalar l_final.

[0066] The update process is executed by micro-sequences fixed in the vector processing unit 300 hardware, ensuring numerical stability.

[0067] like Figure 4 As shown, Figure 4The microarchitecture and data flow of the vector processing unit (VPU) are shown in detail. After the S_ij and the corresponding V_j block from the MMU enter the VPU, the maximum value, the normalized denominator accumulation l, and the output accumulation are updated as new blocks flow in.

[0068] After all j ≤ i blocks belonging to the output accumulation block O_i have been processed, the vector processing unit 300 performs the final normalization O_i = o / l, and the hardwired scheduling controller triggers the writing of O_i back to external memory.

[0069] On-chip storage unit 500, connected to the hard-wired scheduling controller, matrix multiplication unit, and vector computation unit, includes: A query buffer is used to cache query blocks, a key / value buffer is used to cache key / value blocks, and an output buffer is used to cache output accumulation blocks.

[0070] In this embodiment, the on-chip storage unit 500 includes external storage for DRAM query blocks Q / key blocks K / value blocks V and an online cache for caching query block elements Q_i, key block elements K_j, value block elements V_j and outputs.

[0071] In this embodiment, the on-chip storage unit 500 is designed with an on-chip double buffer mechanism and a minimum status register set to match the scheduling order of the hard-wired scheduler. Under the command of the hard-wired scheduler, a fully pipelined operation is achieved, including computation, data prefetching, and online Softmax updates, ensuring stable streaming output while avoiding the materialization of any large intermediate matrices.

[0072] In this embodiment of the application, at least one of the query buffer and the key / value buffer adopts a double buffer structure, so that prefetching and overlap are achieved between external memory reading and matrix multiplication unit calculation.

[0073] Figure 5 This is a timing diagram of double buffering and prefetching, as shown below. Figure 5 As shown, while the matrix multiplication unit is computing the current block, the DMA is prefetching the next block's "computation-memory access" overlap, as well as the pipelined operation of the vector processing unit and the matrix multiplication unit.

[0074] Figure 6 The state machine transition graph for performing causal attention computation, such as Figure 6As shown, the process begins in the IDLE state. Upon receiving the start signal, it first enters the LOAD_Q state, where the currently processed query block Q_i is loaded into the internal buffer via DMA. Subsequently, the state machine enters a cyclical process for all key / value blocks K_j / V_j: in the LOAD_KV state, the corresponding K_j and V_j blocks are loaded, and then in the MMU_COMPUTE state, the attention score blocks are calculated. The loop embeds crucial conditional logic: when j < i, it's typically used to implement a causal attention mask, ensuring the current position only focuses on past information; when j == i, it processes the current block itself, applying a special diagonal mask while simultaneously updating intermediate states used for Softmax normalization (such as the maximum value m and accumulated value l) online, and accumulating a portion of the output result o. After completing the computation of all necessary blocks, the state machine enters the WRITEBACK state, writing the final normalized output block O_i back to memory, and then returns to the idle state, ready to process the next query block.

[0075] Example 2 One embodiment of this application provides a hardware acceleration method for performing causal attention computation on an edge computing device, which will be referred to below. Figure 7 This application provides a detailed description of a hardware acceleration method for performing causal attention computation on an edge computing device. Embodiment 2 of this application describes a hardware acceleration method for performing causal attention computation on an edge computing device, applied to the aforementioned system, comprising: S101: Divide the query data, key data, and value data into fixed-size blocks along the sequence dimension; S102: In response to performing causal attention computation, the hardwired scheduling controller generates a sequence of control signals to access the data block located in the lower triangular structure in a first nested manner; S103: According to the control signal sequence, the matrix multiplication unit reads the corresponding query block and key block, performs block matrix multiplication operation, and calculates the score block; S104: When the data block indicated by the control signal sequence is a diagonal block, the triangular mask generation circuit will mask the elements in the data block whose row index is less than the column index. S105: The score blocks are normalized in the vector computation unit using an online normalization row-by-row update method, and the weighted results of the value blocks are accumulated and output.

[0076] The above description is merely a partial embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0077] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0078] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An acceleration system for performing causal attention computation on an edge computing device, characterized in that, A hardwired scheduling controller is used to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a first nesting manner, based on the lower triangular structure of causal attention, wherein the first nesting manner is a row-first-column nesting manner. The matrix multiplication unit, connected to the hardwired scheduling controller, is used to read the corresponding query block and key block according to the control signal sequence, perform block matrix multiplication operations, and calculate the score block; The vector computation unit, connected to the matrix multiplication unit, is configured to receive the score blocks and value blocks in a streaming manner and perform Softmax normalization and weighted accumulation operations. A triangular mask generation circuit, connected to the hard-wired scheduling controller, is integrated between the matrix multiplication unit and the vector calculation unit. It is used to mask the elements in the data block whose row index is less than the column index when the data block indicated by the control signal sequence is a diagonal block. On-chip memory unit, connected to the hard-wired scheduling controller, includes: A query buffer is used to cache query blocks, a key / value buffer is used to cache key / value blocks, and an output buffer is used to cache output accumulation blocks.

2. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The hardwired scheduling controller is a hardware state machine with pre-set fixed state transition logic. It is configured to generate a sequence of control signals to access data blocks located in the lower triangular structure in a first nested manner based on block coordinates, block size, and sequence length.

3. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The triangular mask generator is configured to perform arithmetic gating logic on the multiply-accumulate path corresponding to the upper triangular position in the data block when the data block indicated by the control signal sequence is a diagonal block.

4. The acceleration system on an edge computing device performing causal attention computation according to claim 3, characterized in that, The arithmetic gating logic is used to compare row and column indices within a block, and to pad or disable score elements whose column index is greater than their row index before writing them back or sending them to the vector processing unit.

5. The acceleration system on an edge computing device performing causal attention computation according to claim 4, characterized in that, The triangular mask generator is configured to set the elements at the upper triangular positions in the scoring block to negative infinity and set the corresponding weights to zero when the data block indicated by the control signal sequence is a diagonal block.

6. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The vector computation unit includes a dedicated register set maintained for each data row; the dedicated register set includes a row maximum register, a normalized denominator cumulative l register, and an output cumulative register; the dedicated register set is configured to perform row maximum update, normalized denominator cumulative l update, and output cumulative update for each input score block.

7. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The vector computation unit is configured to perform the following update: m_new = max(m_old, max(s)); alpha = exp(m_old - m_new); p = exp(s - m_new); l_new = alpha · l_old + sum(p); o_new = alpha · o_old + (p · V); O = o_final / l_final; Where s is the logits segment vector in the scoring block, m_old is the historical cumulative maximum value of the row, initially set to -∞; m_new is the new cumulative maximum value of the row after including the current segment; alpha is the scaling factor; p is the unnormalized weight vector under m_new; V is the value vector; l_old is the historical cumulative value of the softmax denominator of the row, initially set to 0; l_new is the updated cumulative value of the denominator; o_old is the historical cumulative vector of the output numerator of the row; o_new is the updated cumulative output numerator; l_final and o_final are the final cumulative denominator and numerator of the row after processing all accessed segments, respectively; and O is the final attention output vector of the row.

8. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, At least one of the query buffer and the key / value buffer adopts a double buffer structure, enabling prefetching and overlap between external memory reads and matrix multiplication unit calculations.

9. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The matrix multiplication unit is a fixed-size pulsating array of 64×64. In the case of insufficient blocks, a zero-filling and vector processing masking strategy is used to process the boundary blocks.

10. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The hardwired scheduling controller is also configured to generate a sequence of control signals for accessing data blocks located in the lower triangular structure in a second nesting manner, based on the lower triangular structure of causal attention, so that each loaded key-value block in the on-chip storage unit can be computed continuously with multiple different query blocks before it is replaced out of the on-chip storage unit; the second nesting manner is a column-row-first nesting manner.

11. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The hardwired scheduling controller includes a counter and a comparator. It generates a control signal sequence for accessing data blocks that satisfy the condition that column index ≤ row index by comparing control signal sequences. When column index = row index, it determines the invalid position of the upper triangular structure by comparing the row and column indices within the block.

12. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, The matrix multiplication unit is configured to output the score vector of each column in a streaming manner when calculating the score block, so that the vector calculation unit obtains the local maximum vector of each row by performing element-wise maximum value operation.

13. The acceleration system on an edge computing device performing causal attention computation according to claim 1, characterized in that, It is also equipped with multiplexing logic for handling multi-head attention. This multiplexing logic enables computation on multiple heads by serial multiplexing or staggered time slice multiplexing when the resources of a single matrix multiplication unit are limited.

14. An acceleration method on an edge computing device performing causal attention computation, employing the acceleration system on an edge computing device performing causal attention computation as described in any one of claims 1-13, characterized in that, include: Divide the query data, key data, and value data into fixed-size blocks along the sequence dimension; In response to performing causal attention computation, the hardwired scheduling controller generates a sequence of control signals to access data blocks located in the lower triangular structure in a first nested manner; According to the control signal sequence, the matrix multiplication unit reads the corresponding query block and key block, performs block matrix multiplication operation, and calculates the score block; When the data block indicated by the control signal sequence is a diagonal block, the triangular mask generation circuit will mask the elements in the data block whose row index is less than the column index. The scoring blocks are normalized in the vector computation unit using an online normalization row-by-row update method, and the weighted results of the value blocks are accumulated and output.

Images