Edge-side configurable GEMV acceleration method, device and medium
By employing a configurable parameterized two-dimensional PE network array and INT16 accumulator design controlled by a finite state machine on the edge device, the GEMV operation is optimized, solving the performance bottleneck problem of the Transformer model on the edge device and achieving efficient GEMV acceleration and resource saving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-04-07
AI Technical Summary
When deploying generative AI Transformer models on resource-constrained edge devices, existing technologies often encounter GEMV operations as a performance bottleneck. Furthermore, general-purpose hardware accelerators introduce additional overhead in resource-limited edge scenarios, impacting computational efficiency and accuracy.
A configurable parameterized two-dimensional PE network array controlled by a finite state machine is used to manage the GEMV calculation process through global state management. Combined with INT16 accumulator and saturation logic design, the GEMV operation is optimized, hardware resource consumption is reduced and calculation accuracy is guaranteed.
It significantly accelerates GEMV operations, improves Transformer performance, saves hardware resources, adapts to different application scenarios, and meets the actual deployment needs of edge devices.
Smart Images
Figure CN121807409A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence and large model inference hardware acceleration technology, specifically to an edge-side configurable GEMV acceleration method, device, and medium. Background Technology
[0002] Generative artificial intelligence, relying on its end-to-end content generation and semantic understanding capabilities, has demonstrated broad application potential in numerous fields. Its powerful generative effects mainly stem from the self-attention mechanism in the Transformer model, which supports both ultra-long context modeling and efficient parallel processing of sequence data.
[0003] With the improvement in the performance of generative models, deploying them on resource-constrained edge devices has become a major research trend. However, even considering only the forward propagation stage of Transformer inference, a large number of matrix-vector multiplication operations (GEMV / MVM) still exist in the input embedding layer, hidden layers, and attention mechanism, which has become one of the main performance bottlenecks on edge devices with limited computing and storage resources. To alleviate this problem, an emerging quantization-based approach has been widely adopted: quantizing floating-point operations into INT8 / INT32 format and applying it to both GEMV and nonlinear function computation, achieving significant hardware efficiency improvements with minimal precision loss. Furthermore, emerging models often stack hidden layers to enhance the effect of the self-attention mechanism, thereby better understanding the contextual semantics, which further increases the computational burden of GEMV operations.
[0004] Existing GEMV acceleration relies on general-purpose hardware accelerators, which often introduce complex control and interconnect logic to ensure versatility. In edge computing scenarios where resources are already limited, the impact of this additional overhead is further amplified.
[0005] Therefore, how to improve Transformer performance, save hardware resources, and ensure computational accuracy by accelerating GEMV operations to meet the actual deployment needs of edge devices is a technical problem that urgently needs to be solved. Summary of the Invention
[0006] The technical objective of this invention is to provide an edge-side configurable GEMV acceleration method, device, and medium to address the problem of how to improve Transformer performance, save hardware resources, and ensure computational accuracy by accelerating GEMV operations, thereby meeting the actual deployment requirements of edge-side devices.
[0007] The technical objective of this invention is achieved as follows: an edge-side configurable GEMV acceleration method, in which a finite state machine (FSM) controls the operation of a configurable parameterized two-dimensional PE network array through global states, as detailed below:
[0008] The entire configurable parameterized 2D PE network array is initialized starting from the Preload state phase when the global state is 2'b00. The controller sends a Cfg_Index address to all PEs. Each PE compares the Cfg_Index address with its own coordinates (Cur_Row, Cur_Col). If they match and the Cfg_Valid signal is valid, the incoming Cfg_Data is loaded into the internal Weight_Reg register. The Weight_Reg register stores the weight matrix values used for GEMV calculation.
[0009] Subsequently, the configurable parameterized two-dimensional PE network array enters the global state Global_State 2'b01, i.e., the Broadcast state. The input data is streamed into the array in a column-aligned manner. The PE writes the incoming Data_In register to Data_Reg to ensure that the value of the input data vector can propagate downwards along the column direction. Along with entering the broadcast state, an FSM built-in counter is also initialized. The counter increments by 1 for each row of data loaded by the PE after the column-aligned data stream completes.
[0010] After the built-in counter of the finite state machine (FSM) equals the number of rows in the array (Row_PE), the FSM enters the global state (Global_State) of 2'b10, which is the Pipeline state. In the Pipeline state, each PE adds the stored weight value and input data to a partial sum, and saves the final result to the accumulator register (Acc_Reg) and propagates it eastward along the row direction, thus continuing to propagate part of the result horizontally in the configurable parameterized two-dimensional PE network array. After each column of data completes the accumulation operation, the built-in counter of the FSM continues to increment by 1.
[0011] After the built-in counter of the finite state machine (FSM) equals the sum of the number of rows and columns of the configurable parameterized 2D PE network array (Row_PE + Col_PE), the FSM enters the global state (Global_State 2'b11, i.e., the Store state). At this point, the computation is complete, and the PE will not perform any additional computations or state updates. However, the Acc_Out column of the last column of each row will display the final result, and the output buffer will perform the final processing of the result, which takes one cycle. The final state jumps back to the Preload global state or Idle state and resets the built-in counter of the FSM.
[0012] Preferably, the network size of the configurable parameterized two-dimensional PE network array is configurable Row_PE×Col_PE. Each PE is connected to its adjacent north and west PEs or array units for input and to its south and east PEs or array units for output, forming a pulsating array structure.
[0013] More preferably, the configurable parameterized 2D PE network array is initialized in the Global_State of 2'b00 (Preload state). Configuration data is read from SRAM via the AXI bus and streamed into the configurable parameterized 2D PE network array in the form of a triplet signal of {address, data, enable}. After the weight data is reassembled into a weight matrix, it is fed to the Config_Index, Config_Data, and Config_Valid ports of each PE. The weight matrix is distributed in both row and column dimensions, while the reassembled input data vector is distributed only in the row dimension of the configurable parameterized 2D PE network. After the data is unpacked and aligned, it is input through the Data_In port on the north side of the PE and propagated along the column direction to the Data_Out port on the south side. At the same time, the Acc_In port on the west side of the PE receives part of the accumulation result from the upstream PE and transmits part of the accumulation result from the upstream PE to the Acc_Out port on the east side along the row direction.
[0014] More preferably, an output buffer is deployed on the easternmost side of the configurable parameterized two-dimensional PE network array. The output buffer receives and processes the accumulated results from the configurable parameterized two-dimensional PE network array, and then flattens the accumulated results from the configurable parameterized two-dimensional PE network array row by row and transmits them through the AXI bus.
[0015] More preferably, each PE uses the global clock and reset signal of the acceleration unit;
[0016] When in configuration mode, i.e., when the Cfg_Valid signal is high, the PE determines whether to load the configuration data Cfg_Data by comparing whether the coordinates to be configured, Cfg_Index, match the current PE coordinates (Cur_Row, Cur_Col).
[0017] If the PE is in a non-configured or data flow state, i.e., when the Cfg_Valid signal is low, the PE completes the calculation based on the data flow from the north side, Data_In.
[0018] When the global state Global_State is 2'b00, i.e., the Preload state, all PEs perform configuration loading or initialization, and the weight register Weight_Reg is combined with the configuration or non-configuration state registers Cfg_Data / Data_In;
[0019] When the global state Global_State is 2'b01, i.e., Broadcast state, all PEs perform data loading, the data register Data_Reg stores Data_In, and Data_In is forwarded to the downstream PE in the south through the Data_Out port.
[0020] More preferably, this method adopts the design paradigm of INT16 accumulator and saturation logic to save hardware resources and reduce wiring complexity as much as possible;
[0021] The saturation logic in each PE is as follows: when the two most significant bits (MSB) of the intermediate result are the same (00, 11), it means that no overflow has occurred and the result remains unchanged; if the MSB is 01, it means that a positive overflow has occurred and the output is clamped to Acc_Max; if the MSB is 10, it means that a negative overflow has occurred and the output is clamped to Acc_Min; finally, the output is selected as clamped or unclamped accumulated result outside the saturation logic by whether overflow occurs.
[0022] When the global state is 2'b10, i.e., the Pipeline state, the accumulated result is stored in the Acc_Reg register and output to the downstream PE via Acc_Out.
[0023] An electronic device includes: a memory and at least one processor;
[0024] The memory contains computer programs;
[0025] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the edge-side configurable GEMV acceleration method as described above.
[0026] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the edge-side configurable GEMV acceleration method as described above.
[0027] The edge-side configurable GEMV acceleration method, device, and medium of the present invention have the following advantages: The proposed method is compatible with common Transformer model deployment hardware based on INT8 for inference quantization, which can significantly accelerate GEMV operations and thus improve Transformer performance. At the same time, it adopts a simplified hardware architecture, which greatly saves hardware resources and ensures computational accuracy, meeting the actual deployment needs of edge measurement devices. In addition, the array has parameterization and high scalability, and can be flexibly modified to adapt to different application scenarios. Attached Figure Description
[0028] The invention will be further described below with reference to the accompanying drawings.
[0029] Appendix Figure 1 A schematic diagram of the INT8 / 16 scalar PE;
[0030] Appendix Figure 2 A schematic diagram of a parameterized two-dimensional PE network array for configuration;
[0031] Appendix Figure 3 This is a schematic diagram of the FSM state machine transition. Detailed Implementation
[0032] The edge-configurable GEMV acceleration method, device, and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0033] Example 1:
[0034] In the Transformer model, the input embedding layer multiplies the input data with the embedding matrix to obtain an integer vector. The hidden layer contains dot product operations between the embedding matrix and the weight matrix, while the attention layer contains a large number of GEMV operations. Emerging models often stack hidden layers to enhance the effectiveness of the self-attention mechanism, thereby better understanding the contextual semantics, which further increases the computational burden of GEMV operations. Therefore, Transformer models using INT8 quantization for inference are widely adopted. Since INT8 multipliers produce INT16 results, to prevent overflow caused by the accumulation of multiple parts, these models typically require ultrawide accumulators or overflow-preventing saturation logic designs. Based on this, the computational requirements of Transformer inference are simplified to intensive INT8 GEMV operations.
[0035] For this type of dense INT8 GEMV, this embodiment provides an edge-side configurable GEMV acceleration method, as shown in the attached figure. Figure 1As shown, unlike traditional coprocessor designs, the PE design proposed in this embodiment requires no instruction memory or instruction pipeline. Each PE uses the global clock and reset signal of the acceleration unit. In the configuration state (Cfg_Valid signal high), the PE determines whether to load configuration data Cfg_Data by comparing the coordinates to be configured (Cfg_Index) with the current PE coordinates (Cur_Row, Cur_Col); if in the non-configuration / data flow state (Cfg_Valid signal low), the PE completes the calculation based on the data inflow Data_In from the north. When the global state Global_State is 2'b00 (Preload state), all PE units perform configuration loading / initialization, and the weight register Weight_Reg stores Cfg_Data / Data_In in conjunction with the configuration / non-configuration state registers. When the global state is 2'b01 (Broadcast), all PE units perform data loading, the data register Data_Reg stores Data_In, and Data_In is forwarded to the downstream PE to the south through the Data_Out port.
[0036] After the 8-bit multiplication unit Mul (multiplier) completes the scalar multiplication of the weighted data and performs sign extension, the result is added to the partial accumulation result input Acc_In from the upstream PE in the west. Since multiple accumulations may cause overflow, this embodiment, after careful consideration, chooses an INT16 accumulator and saturation logic design paradigm to save hardware resources and reduce wiring complexity as much as possible. (See attached...) Figure 1 The portion within the dashed box represents the saturation logic in each PE. When the two most significant bits (MSB) of the intermediate result are the same (00, 11), it indicates no overflow has occurred, and the result remains unchanged. If the MSB is 01, it indicates a positive overflow, and the output is clamped to Acc_Max. If the MSB is 10, it indicates a negative overflow, and the output is clamped to Acc_Min. Finally, the output is selected as clamped / unclamped accumulated result outside the saturation logic based on whether overflow occurs. When the global state is 2'b10 (Pipeline), the accumulated result is stored in the Acc_Reg register and output to the downstream PE via Acc_Out.
[0037] As attached Figure 2 As shown, the network size of the configurable parameterized two-dimensional PE network array is configurable Row_PE×Col_PE. Each PE is connected to its adjacent north and west PEs or array units for input and to its south and east PEs or array units for output, forming a pulsating array structure.
[0038] In the Preload global state, the network array completes initialization. Configuration data is read from SRAM via the AXI bus and streamed into the array as a triplet signal of {address, data, enable}. Weight data is reassembled into a weight matrix and fed to the Config_Index, Config_Data, and Config_Valid ports of each PE. The weight matrix is distributed along both row and column dimensions, while the reassembled input data vector is distributed only along the row dimension of the network. After unpacking and aligning, the data is input through the Data_In port on the north side of the PE and propagated along the column direction to the Data_Out port on the south side. Simultaneously, the Acc_In port on the west side of the PE receives a portion of the accumulated result from the upstream PE and passes it along the row direction to the Acc_Out port on the east side. An output buffer is designed at the easternmost edge of the network. The output buffer receives and processes the accumulated result from the PE array, flattens it by row, and transmits it via the AXI bus.
[0039] As attached Figure 3 As shown, the Finite State Machine (FSM) controls the operation of the PE array through global states as follows:
[0040] The entire array will be initialized starting in the Preload phase, where the controller sends a Cfg_Index address to all PEs. Each PE compares this address with its own coordinates (Cur_Row, Cur_Col). If they match and the Cfg_Valid signal is valid, the incoming Cfg_Data is loaded into the internal Weight_Reg register, which stores the weight matrix values used for GEMV calculation.
[0041] The array then enters Broadcast state, where input data is streamed into the array in column alignment. The PE writes the incoming Data_In register to Data_Reg to ensure that the value of the input data vector can propagate downwards along the column direction. Along with entering Broadcast state, an FSM built-in counter is also initialized. The counter increments by 1 for each row of data loaded by the PE after the column-aligned data stream completes.
[0042] After the FSM's built-in counter equals the number of array rows (Row_PE), the finite state machine FSM enters the Pipeline state. In this state, each PE adds the stored weight value and input data to a partial sum, and saves this final result to the accumulator register Acc_Reg and propagates it eastward along the row direction, thus continuing to propagate the partial result horizontally in the grid. After each column of data completes the accumulation operation, the FSM's built-in counter continues to increment by 1.
[0043] After the FSM's built-in counter equals the sum of the array's row and column numbers (Row_PE + Col_PE), the FSM state machine enters the global state (Global_State 2'b11), i.e., the Store state. At this point, the computation is complete, and PE will not perform any additional computations or state updates. However, the Acc_Out column of each row will display the final result, and the output buffer will perform final processing of the result, taking one cycle. The final state then transitions back to the Preload state or Idle state, and resets the FSM's built-in counter.
[0044] Example 2:
[0045] This embodiment also provides an electronic device, including: a memory and a processor;
[0046] The memory stores the instructions executed by the computer.
[0047] The processor executes computer execution instructions stored in the memory, causing the processor to execute the edge-side configurable GEMV acceleration method in any embodiment of the present invention.
[0048] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.
[0049] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0050] Example 3:
[0051] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the edge-side configurable GEMV acceleration method of any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the above embodiments can be provided, and the computer (or CPU or MPU) of the system or apparatus can read and execute the program code stored in the storage medium.
[0052] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0053] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0054] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0055] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An edge-side configurable GEMV acceleration method, characterized in that, This method uses a finite state machine (FSM) to control a configurable, parameterized two-dimensional PE network array through global states, as detailed below: The entire configurable parameterized 2D PE network array is initialized starting from the Preload state phase when the global state is 2'b00. The controller sends a Cfg_Index address to all PEs. Each PE compares the Cfg_Index address with its own coordinates (Cur_Row, Cur_Col). If they match and the Cfg_Valid signal is valid, the incoming Cfg_Data is loaded into the internal Weight_Reg register. The Weight_Reg register stores the weight matrix values used for GEMV calculation. Subsequently, the configurable parameterized two-dimensional PE network array enters the global state Global_State 2'b01, i.e., the Broadcast state. The input data is streamed into the array in a column-aligned manner. The PE writes the incoming Data_In register to Data_Reg to ensure that the value of the input data vector can propagate downwards along the column direction. Along with entering the broadcast state, an FSM built-in counter is also initialized. The counter increments by 1 for each row of data loaded by the PE after the column-aligned data stream completes. After the built-in counter of the finite state machine (FSM) equals the number of rows in the array (Row_PE), the FSM enters the global state (Global_State) of 2'b10, which is the Pipeline state. In the Pipeline state, each PE adds the stored weight value and input data to a partial sum, and saves the final result to the accumulator register (Acc_Reg) and propagates it eastward along the row direction, thus continuing to propagate part of the result horizontally in the configurable parameterized two-dimensional PE network array. After each column of data completes the accumulation operation, the counter continues to increment by 1. After the built-in counter of the finite state machine (FSM) equals the sum of the number of rows and columns of the configurable parameterized 2D PE network array (Row_PE + Col_PE), the FSM enters the global state (Global_State 2'b11, i.e., the Store state). At this point, the computation is complete, and the PE will not perform any additional computations or state updates. However, the Acc_Out column of the last column of each row will display the final result, and the output buffer will perform the final processing of the result, which takes one cycle. The final state jumps back to the Preload global state or Idle state and resets the built-in counter of the FSM.
2. The edge-side configurable GEMV acceleration method according to claim 1, characterized in that, The configurable parameterized two-dimensional PE network array has a network size of configurable Row_PE×Col_PE. Each PE is connected to its adjacent north and west PEs or array elements for input and to its south and east PEs or array elements for output, forming a pulsating array structure.
3. The edge-side configurable GEMV acceleration method according to claim 1 or 2, characterized in that, The configurable parameterized 2D PE network array is initialized in the Global_State of 2'b00 (Preload state). Configuration data is read from SRAM via the AXI bus and streamed into the configurable parameterized 2D PE network array in the form of a triplet signal of {address, data, enable}. After the weight data is reassembled into a weight matrix, it is fed to the Config_Index, Config_Data, and Config_Valid ports of each PE. The weight matrix is distributed in both row and column dimensions, while the reassembled input data vector is distributed only in the row dimension of the configurable parameterized 2D PE network. After the data is unpacked and aligned, it is input through the Data_In port on the north side of the PE and propagated along the column direction to the Data_Out port on the south side. At the same time, the Acc_In port on the west side of the PE receives part of the accumulation result from the upstream PE and passes part of the accumulation result from the upstream PE to the Acc_Out port on the east side along the row direction.
4. The edge-side configurable GEMV acceleration method according to claim 3, characterized in that, An output buffer is deployed on the easternmost side of the configurable parameterized two-dimensional PE network array. The output buffer receives and processes the accumulated results from the configurable parameterized two-dimensional PE network array, and then flattens the accumulated results from the configurable parameterized two-dimensional PE network array row by row and transmits them through the AXI bus.
5. The edge-side configurable GEMV acceleration method according to claim 2, characterized in that, Each PE uses the global clock and reset signal of the acceleration unit; When in configuration mode, i.e., when the Cfg_Valid signal is high, the PE determines whether to load the configuration data Cfg_Data by comparing whether the coordinates to be configured, Cfg_Index, match the current PE coordinates (Cur_Row, Cur_Col). If the PE is in a non-configured or data flow state, i.e., when the Cfg_Valid signal is low, the PE completes the calculation based on the data flow from the north side, Data_In. When the global state Global_State is 2'b00, i.e., the Preload state, all PEs perform configuration loading or initialization, and the weight register Weight_Reg is combined with the configuration or non-configuration state registers Cfg_Data / Data_In; When the global state Global_State is 2'b01, i.e., Broadcast state, all PEs perform data loading, the data register Data_Reg stores Data_In, and Data_In is forwarded to the downstream PE in the south through the Data_Out port.
6. The edge-side configurable GEMV acceleration method according to claim 5, characterized in that, This method employs an INT16 accumulator and a saturation logic design paradigm; The saturation logic in each PE is as follows: when the two most significant bits (MSB) of the intermediate result are the same (00, 11), it means that no overflow has occurred and the result remains unchanged; if the MSB is 01, it means that a positive overflow has occurred and the output is clamped to Acc_Max; if the MSB is 10, it means that a negative overflow has occurred and the output is clamped to Acc_Min; finally, the output is selected as clamped or unclamped accumulated result outside the saturation logic by whether overflow occurs. When the global state is 2'b10, i.e., the Pipeline state, the accumulated result is stored in the Acc_Reg register and output to the downstream PE via Acc_Out.
7. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the edge-side configurable GEMV acceleration method as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the edge-side configurable GEMV acceleration method as described in any one of claims 1 to 6.