A heterogeneous system and method for implementing attention mechanisms in large language models

Abstract

This invention discloses a heterogeneous system for implementing an attention mechanism in a large language model, comprising: a controller, a computing unit, and a memory; the computing unit includes a neural network processor and a graphics processor; the memory includes a local high-speed storage module; the controller controls the computing unit through an access interface and a data bus, and is connected to the memory via the computing unit; the computing unit and the local high-speed storage module are bidirectionally connected via a storage interface and a data bus; the heterogeneous system performs data transmission and exchange with an external global storage module; the local high-speed storage module and the global storage module are connected and interact with each other via a direct memory access channel. This invention also discloses an inference optimization method implemented through the above heterogeneous system, which has broad application value.

Description

Technical Field

[0001] This invention belongs to the field of model reasoning optimization technology, and relates to a heterogeneous system and method for implementing the attention mechanism of large language models. Background Technology

[0002] Current mainstream large-scale model inference systems all employ heterogeneous architectures. For example, Nvidia's latest architecture integrates TensorCore and CUDA Core. TensorCore has high computing power and can handle low-precision data such as INT4 / INT8 / FP8 / FP16; CUDA Core has lower computing power but can handle high-precision data such as FP32, TF32, and FP64. Other manufacturers also use similar solutions, which can be GPU+NPU heterogeneous or DSP+NPU heterogeneous, where the GPU and DSP handle units with high-precision requirements, and the NPU handles units with low-precision requirements.

[0003] In the design of large language models, the Transformer architecture is adopted, which contains both large-scale linear operations such as matrix multiplication and non-linear operations such as RoPE, SiLU, and Softmax. Non-linear operations must be performed on computing units that meet the accuracy requirements of FP32 to meet the accuracy requirements. Heterogeneous systems can well meet the needs of large model inference deployment.

[0004] However, in the structure of a large language model, the low-precision and high-precision parts are sequential. If the model is executed in its original order, the low-precision and high-precision components in the inference system will run alternately, failing to fully utilize the mechanism that allows these two parts to work in parallel on a heterogeneous system.

[0005] In addition, FlashAttention and its successors FlashAttention2, FlashAttention3, etc., target Q. K, Softmax, QK The V part was optimized, but the matrix multiplication operations of Q, K, and V with the weights and RoPE on heterogeneous systems were not optimized. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a heterogeneous system and method for implementing the attention mechanism of a large language model. This invention implements the attention mechanism of a large language model based on Transformer, where the MatMul operation is performed on a high-performance, low-precision MAC (Macro- ...

[0007] This invention is achieved through the following technical solution:

[0008] This invention provides a heterogeneous system for implementing the attention mechanism of a large language model, the heterogeneous system comprising:

[0009] The system comprises a controller, a computing unit, and a memory; the computing unit includes a neural network processor (NPU) and a graphics processing unit (GPU); the memory includes a local high-speed storage module.

[0010] In one specific implementation, the heterogeneous system performs data transmission and exchange with an external global storage module;

[0011] The neural network processor is used to process low-precision data including INT4, INT8, FP8, FP16, etc., and the graphics processor is used to process high-precision data including FP32, TF32, FP64, etc.

[0012] The controller controls the computing unit through an access interface and a data bus, and connects to the memory via the computing unit;

[0013] The computing unit and the local high-speed storage module are connected bidirectionally via a storage interface and a data bus; the local high-speed storage module and the global storage module can generally be connected and interact with each other through direct memory access channels or other means.

[0014] In the specific implementation of this invention, the weights in the large model are stored in the global storage module, and the data transfer between various components in the heterogeneous system is realized by the local high-speed storage module inside the system.

[0015] In the heterogeneous system described in this invention, the neural network processor is a low-precision, high-computing-power computing unit used to perform matrix multiplication operations in a large language model;

[0016] The graphics processor is a high-precision, low-computing-power computing unit used to perform high-precision calculation tasks related to RoPE.

[0017] The controller is used to schedule tasks to different computing units based on a scheduling strategy that considers data dependencies and task priorities.

[0018] The local cache module in the storage module provides access to both low-precision and high-precision computing units.

[0019] In this invention, the descriptions of "high computing power", "low precision", "low computing power" and "high precision" are not absolute high or low, but relatively high or relatively low in the system.

[0020] In one specific embodiment of the heterogeneous system described in this invention, the controller statically binds specific types of inference computation tasks to preset computing units, wherein tasks related to weight matrix multiplication are mapped to the neural network processor, and tasks related to RoPE function computation are mapped to the graphics processor.

[0021] This invention also provides a method for optimizing attention mechanism inference in large language models on heterogeneous systems. The method decomposes the model structure through dependency relationships and rationally allocates computational tasks to different computational units for execution; including:

[0022] Step 1: Assign matrix multiplication and RoPE calculation to computation units in different task queues; the matrix multiplication includes: MatMul(input, q_proj), MatMul(input, k_proj), MatMul(input, v_proj), and the RoPE calculation includes: loading the RoPE cos / sin matrix, RoPE(query), and RoPE(key);

[0023] Step 2: The neural network processor executes the MatMul(input, q_proj) operation, while the graphics processor loads the RoPE cos / sin matrix.

[0024] Step 3: After MatMul(input, q_proj) is completed, the neural network processor executes MatMul(input, k_proj); at the same time, after the RoPE cos / sin matrix is ​​loaded, the graphics processor executes RoPE(query).

[0025] Step 4: After MatMul(input, k_proj) is completed, the neural network processor executes MatMul(input, v_proj); at the same time, after RoPE(query) is completed, the graphics processor executes RoPE(key).

[0026] After the above calculations are completed, the calculation results are returned.

[0027] In the implementation of the method of this invention, computing units are allocated according to the computing task and its required computing power and accuracy requirements, including the following:

[0028] MatMul(input, q_proj) runs on the high-performance, low-precision computing unit (NPU).

[0029] MatMul(input, k_proj) runs on the high-performance, low-precision component (NPU).

[0030] MatMul(input, v_proj) runs on the high-computing-power, low-precision component (NPU);

[0031] RoPE cos / sin matrix loading runs on low-computing-power, high-precision components (GPUs).

[0032] RoPE (query) runs on low-computing-power, high-precision components (GPUs);

[0033] RoPE(key) runs on low-computing-power, high-precision components (GPUs).

[0034] The dependencies include strong dependencies and weak dependencies; the strong dependencies are data dependencies, where the computation of one task depends on the output of another computation task; the weak dependencies are device dependencies, where different tasks are executed in the same device according to their order of execution.

[0035] In one specific implementation, the operation of RoPE(query) depends on loading the RoPE cos / sin matrix, which depends on MatMul(input, q_proj); the operation of RoPE(key) depends on loading the RoPE cos / sin matrix, which depends on MatMul(input, k_proj).

[0036] In one specific implementation, the matrix multiplication operations MatMul(input, q_proj), MatMul(input, k_proj), and MatMul(input, v_proj) are generally performed using a low-precision data format.

[0037] While the neural network processor (low-precision, high-computing-power computing unit) performs matrix multiplication, the graphics processor (high-precision, low-computing-power computing unit) loads the high-precision data required for RoPE calculation and performs RoPE calculation in parallel.

[0038] In the implementation of the method of the present invention, the controller performs priority calculation based on the dependency relationship, and the calculation rules include the following:

[0039] 1) Assign a priority value of 0 to all tasks first;

[0040] 2) Tasks that are dependent on other tasks in the task queue of their respective computing units have their priority increased by 1;

[0041] 3) If a task is dependent on a task outside the task queue of its computing unit, its priority is increased by 2;

[0042] 4) The task queue corresponding to the neural network processor has a higher priority.

[0043] During the implementation of the method of the present invention, the controller dynamically adjusts the allocation of computing tasks according to the priority of computing tasks and data dependencies to ensure the parallel execution of low-precision and high-precision computing parts.

[0044] When assigning tasks, the controller also considers the input and / or output data dependencies of the computation tasks to avoid task execution conflicts.

[0045] In the implementation of the method of the present invention, the execution time on the neural network processor computing unit covers the execution time on the graphics processor computing unit.

[0046] In one specific implementation, the controller can further dynamically adjust the parallelism based on the idle level of hardware resources, computing load, and task priority.

[0047] This invention also provides applications of the above-mentioned heterogeneous systems or methods in the deployment of large language model inference within the Transformer architecture.

[0048] The present invention also provides a hardware system for implementing the above method, the hardware system comprising: a memory and a processor; the memory storing a computer program, which, when executed by the processor, implements the above method.

[0049] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0050] The beneficial effects of this invention include: Through task decomposition and scheduling optimization, this invention enables the GPU to perform RoPE-related cos / sin matrix loading and RoPE(query) and RoPE(key) calculation tasks in parallel during the NPU execution of MatMul(input, q_proj), MatMul(input, k_proj), and MatMul(input, v_proj), thereby avoiding waiting between the GPU and NPU, improving the MAC utilization of the NPU, and meeting the accuracy requirements of RoPE calculation; after optimization, the total execution time is shortened from T1+T2+T3+T4+T5+T6 to T1+T2+T3, significantly improving the overall inference efficiency.

[0051] Compared to existing technologies where Attention subtasks are executed sequentially in heterogeneous systems, leading to alternating idle periods for the GPU and NPU, this invention achieves staggered parallelism between NPU and GPU tasks by dividing task queues across different computing units and setting priorities based on dependencies. RoPE-related tasks are completed during the NPU's execution of the main matrix multiplication task, fully utilizing the GPU's computing power during idle periods and achieving collaborative use of heterogeneous computing resources. While maintaining RoPE computational accuracy, this significantly reduces the total inference time and improves the overall operating efficiency of large language models on heterogeneous systems. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a diagram of a heterogeneous system architecture suitable for implementing attention mechanism computation in large language models.

[0054] Figure 2 This is a flowchart of the Attention calculation (optimization part) in the Transformer architecture of this invention.

[0055] Figure 3 This is a graph representing the computational tasks, dependencies, and priorities of this invention.

[0056] Figure 4 This is a flowchart illustrating the parallel execution of the attention mechanism based on the Transformer architecture of this invention on a heterogeneous system (GPU+NPU). Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0058] Although the matrix multiplication operations of the input with weights q_proj, k_proj, and v_proj in the Attention model of a large language system appear to have no data dependency relationship, their actual execution on the hardware side, due to the limitations of computing power and bandwidth, is still serial for the NPU. Furthermore, the matrix multiplication operations of Query, Key, and Value with weights and the RoPE operation are not strongly data dependent. If the dependencies are not decomposed and properly planned for execution, they can only be executed sequentially. First, the matrix multiplication part is executed, where the low-precision components of the heterogeneous system are working (i.e., the NPU executes Query > Key > Value sequentially); then, the RoPE part is executed, where the high-precision components of the heterogeneous system are working (i.e., the GPU executes RoPE(query) and RoPE(key)). This fails to fully utilize the parallelizable characteristics of the low-precision and high-precision components of the heterogeneous system.

[0059] This invention decomposes the structure of the model based on data dependencies and plans it onto reasonable components, making full use of low-precision, high-computing-power components to improve the utilization rate of MAC (multiply-accumulate unit); during the intervals of calculation by high-computing-power components, high-precision, low-computing-power components are used to process RoPE, thereby improving the utilization rate of MAC while meeting the accuracy requirements.

[0060] To improve the execution efficiency of different components in a heterogeneous system, high data exchange efficiency between them is also required. The specialized heterogeneous system designed in this invention uses a large language model attention mechanism, pre-configured task static mapping at the hardware level, and a shared local cache module exchange path to exchange data, enabling NPU matrix multiplication and GPU RoPE high-precision operations to be performed in parallel for a long time.

[0061] like Figure 1 As shown, to adapt to the parallel and high-speed execution efficiency of large language models, the heterogeneous system in this invention has the following key designs:

[0062] 1. Controller / Scheduler: Responsible for coordinating GPU / NPU work, loading and writing local high-speed memory (SRAM) and global memory (DDR);

[0063] 2. Computing Units: These include high-performance, low-precision NPUs and low-performance, high-precision GPUs. These units can work independently, including computation, loading, and writing to SRAM / DDR. The low-precision, high-performance units process data in formats such as INT4, INT8, FP8, and FP16, while the high-precision, low-performance units process data in formats such as FP32, TF32, and FP64.

[0064] 3. Memory: The system has local high-speed storage (SRAM), which can be read and written by both the NPU and GPU; in addition, there is global storage (DDR) outside the heterogeneous system. DDR storage is not used very frequently but has a large data volume, and is used for data with relatively low requirements for data read / write speed; SRAM is used very frequently but has a smaller data volume, and is used for data with high requirements for data read / write speed. In the large model application scenario of this invention, DDR is used to store weight data; SRAM is used to store intermediate data after each computing unit (NPU and GPU) has completed its own calculations, for data exchange between computing units. In addition, SRAM is also used to preload the cos / sin matrix required for RoPE calculation, so that subsequent computing units do not need to read data from global data, thus accelerating the calculation.

[0065] In this invention, the descriptions of "high computing power", "low precision", "low computing power" and "high precision" are not absolute high or low, but relatively high and relatively low in the system.

[0066] In one specific embodiment of the present invention, the large language model to be processed is Figure 2 , Figure 2 This paper demonstrates the generation process of Query, Key, and Value in a multi-head attention mechanism based on the Transformer architecture. The input tensor is sequentially multiplied by three weight matrices to obtain Q, K, and V tensors, which are then rearranged in dimension to adapt for subsequent attention calculations. Each operation in the diagram is labeled with the input and output data dimensions, visually reflecting the multi-head grouping method and the matrix dimension changes. The three matrix multiplication tasks shown in the diagram are preferentially assigned to the low-precision, high-performance computing unit (NPU) in this invention and scheduled in parallel with the high-precision computing RoPE GPU task to achieve Attention inference in a heterogeneous system.

[0067] Specifically, RoPE depends on MatMul(input, q_proj) and MatMul(input, k_proj); MatMul(input, v_proj) has no data dependency on RoPE.

[0068] This invention first executes MatMul(input, q_proj) and MatMul(input, k_proj) on the NPU, while simultaneously using the GPU to load the cos / sin matrix required for RoPE calculation into the local high-speed storage space; after the NPU finishes its calculation, the NPU calculates MatMul(input, v_proj), while the GPU calculates RoPE(query) and RoPE(key) after loading the cos / sin matrix.

[0069] This fully utilizes the NPU, which is constantly working, while the GPU performs the high-precision calculations for RoPE simultaneously. This improves the NPU's MAC utilization and meets the model's high accuracy requirements for RoPE calculations.

[0070] The computational tasks in the attention mechanism inference process are decomposed and bound to the same or different computational units, and can be dynamically scheduled according to data dependencies and specific implementation latency:

[0071] The computing units are allocated according to the computing task and its required computing power and accuracy:

[0072] #1: MatMul(input, q_proj) runs on a high-performance, low-precision computing unit (NPU);

[0073] #2: MatMul(input, k_proj) runs on a high-performance, low-precision computing unit (NPU);

[0074] #3: MatMul(input, v_proj) runs on a high-computing-power, low-precision component (NPU);

[0075] #4: RoPE cos / sin matrix loading runs on low-computing-power, high-precision components (GPUs).

[0076] #5: RoPE (query) runs on low-computing-power, high-precision components (GPU);

[0077] #6: RoPE (key) runs on low-computing-power, high-precision components (GPU);

[0078] The dependencies in this invention include the following:

[0079] Strong dependency - data dependency: The computation of one task depends on the output of another computation task; including the following:

[0080] #5 depends on #4; #5 depends on #1;

[0081] #6 depends on #4; #6 depends on #2

[0082] Weak dependency - Component dependency: Although there is no obvious dependency between tasks, they run on the same component and there is an execution order relationship:

[0083] This establishes two task queues: #1, #2, and #3 in one task queue; #4, #5, and #6 in another task queue, as follows. Figure 3 As shown, the tasks in the task queue do not have a clear order of priority.

[0084] Based on strong and weak dependencies, the scheduler implements scheduling as follows:

[0085] 1. Assignment precedence (the higher the precedence value, the higher the precedence)

[0086] 1.1 Assign a priority value of 0 to all tasks first;

[0087] 1.2 Tasks that are dependent on other tasks in the task queue of their respective computing units will have their priority increased by one.

[0088] 1.3 Tasks that are dependent on tasks outside the task queue of their respective computing units have their priority increased by two; (to maximize parallelism, tasks that are dependent on tasks from another execution unit need to be processed with even greater priority).

[0089] 1.4 Task queue hierarchy: The computational core of the large model is the NPU, and the NPU task queue takes priority;

[0090] 2. Scheduling: Priority-based scheduling based on device task queues.

[0091] 2.1 From the high-priority task queue (here, the NPU task queue), the highest-priority task is selected for execution. If there are tasks with the same priority in the task queue, they are executed sequentially according to their recorded order; in the task settings above, tasks #1 and #2 will be executed sequentially.

[0092] 2.2 Select the highest priority task from another task queue (here, the GPU task queue) for execution. If there are tasks with the same priority in the task queue, execute them sequentially according to their recorded order; in the task settings above, execute task #4.

[0093] 2.3 Follow 2.1 and 2.2 until all tasks (task #3, task #5 and task #6) are completed.

[0094] The heterogeneous system and inference optimization method in this invention can be applied to variants of the Transformer large language model, including MHA, GQA, MLA, etc.

[0095] In one specific implementation, the time calculation for each part is assumed to be statistically as follows:

[0096] T1 = time {MatMul(input, q_proj)}

[0097] T2 = time {MatMul(input, k_proj)}

[0098] T3 = time {MatMul(input, v_proj)}

[0099] T4 = time {Loading RoPE matrix}

[0100] T5 = time {calculate RoPE(q)}

[0101] T6 = time {calculate RoPE(k)}

[0102] Therefore, the time before optimization is: T1+T2+T3+T4+T5+T6

[0103] The optimal time after optimization is: T1 + T2 + T3, with T4 / T5 / T6 all completed within the interval time.

[0104] While meeting accuracy requirements, the serial operation is parallelized, thereby improving MAC utilization.

[0105] In one specific implementation, such as Figure 4 The allocation algorithm is as follows:

[0106] 1. The controller (Scheduler) selects the highest priority task #1 from the high-priority task queue (here, the NPU task queue) and executes it, with a computation time of T1;

[0107] 2. The controller (Scheduler) selects the highest priority task (task #4, loading the RoPE matrix) from another task queue (here, the GPU task queue) for execution, with a computation time of T4;

[0108] 3. The controller (Scheduler) continues to allocate computation task #2 MatMul(k) from the NPU task queue to the NPU, with computation time T2;

[0109] 4. Tasks #1 and #4 are complete;

[0110] 5. The controller (Scheduler) continues to allocate computation task #5, RoPE(q), from the GPU task queue to the GPU, with computation time T5;

[0111] 6. Task #2 is complete (Task #4 has already been completed in step 4);

[0112] 7. The controller (Scheduler) continues to allocate computation task #6, RoPE(k), from the GPU task queue to the GPU, with computation time T6;

[0113] 8. The controller (Scheduler) continues to allocate computation task #3 MatMul(v) from the NPU task queue to the NPU, with computation time T3;

[0114] Assuming the NPU is the largest computational task and its execution time is always longer than that of the GPU, then the total execution time is: T1 + T2 + T3;

[0115] The computation time of T4, T5, and T6 on the GPU is overwhelmed by the computation time of the NPU.

[0116] Example 1# This invention is used in the Llama-2-7B model:

[0117] Llama-2-7B parameters:

[0118] n_embd = 4096; n_head = 32; n_embd_head = 128

[0119] n_gpq = 1; n_head_kv = 32

[0120] During the large model inference prefill phase, n_tokens are set to 512, 256, 128, 64, 32, and 16 respectively. Similarly, during the large model inference decoding phase, the same implementation is achieved by setting n_tokens = 1.

[0121] The obtained data is shown in Table 1 below:

[0122] Table 1. Task time consumption of the proposed method and traditional method in the Llama-2-7B model.

[0123]

[0124] Example 2# This invention is used in the Llama-3-8B model:

[0125] Llama-3-8B parameters:

[0126] n_embd = 4096; n_head = 32; n_embd_head = 128

[0127] n_gpq = 4; n_head_kv = 8

[0128] During the large model inference prefill phase, n_tokens are set to 512, 256, 128, 64, 32, and 16 respectively. Similarly, during the large model inference decoding phase, the same implementation is achieved by setting n_tokens = 1.

[0129] The obtained data is shown in Table 2 below:

[0130] Table 2. Task time consumption of the method in this application and the traditional method in the Llama-3-8B model.

[0131]

[0132] Data Analysis:

[0133] As can be seen from the specific implementations of the Llama-2-7B and Llama-3-8B models above:

[0134] 1. As the invention is based on: T4 < T1; T5 < T2; T6 < T3. That is, the computation on the GPU is performed simultaneously with the computation on the NPU and is submerged in the NPU's computation process.

[0135] 2. Whether in the Prefill or Decode stage, the system and implementation method used in this invention are optimized to varying degrees compared to traditional solutions.

[0136] 3. When the context is relatively long (n_tokens = 512, 256), the effect is not as obvious because the main time-consuming operation in this case is the matrix multiplication operation on the NPU. This calculation includes the NPU reading weight data from global storage and the specific MAC operation, which is affected by factors such as computing power and bandwidth.

[0137] 4. When Decode (n_tokens = 1), the performance decreases again. Although Decode is a token, the computation unit in our heterogeneous system is calculated in 4-bit units, which means the time consumption is the same as n_tokens = 4.

[0138] 5. The implementation effect of Llama-3-8B is better than that of Llama-2-7B. This is mainly due to the adoption of GQA in Llama-3-8B, which reduces the time spent on both MatMul operation on the NPU and RoPE (key) operation on the GPU. The MatMul operation is still the most time-consuming operation, so the optimization ratio is higher.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0144] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0145] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A heterogeneous system for implementing the attention mechanism of a large language model, characterized in that, include: Controller, computing unit, memory; The computing unit includes a neural network processor and a graphics processor; the memory includes a local high-speed storage module. The controller controls the computing unit through an access interface and a data bus, and is connected to the memory via the computing unit; The computing unit and the local high-speed storage module are connected bidirectionally via a storage interface and a data bus. The neural network processor is a low-precision, high-computing-power computing unit used to perform matrix multiplication tasks in large language models; The graphics processor is a high-precision, low-computing-power computing unit used to perform high-precision calculation tasks related to RoPE. The controller is used to schedule tasks to different computing units based on a scheduling strategy that considers data dependencies and task priorities. The local cache module in the storage module provides common access for both low-precision and high-precision computing units; The controller statically binds specific types of inference computation tasks to preset computing units, wherein tasks related to weight matrix multiplication are mapped to the neural network processor, and tasks related to RoPE function computation are mapped to the graphics processor. The heterogeneous system performs data transmission and exchange with the external global storage module; The local high-speed storage module and the global storage module are connected and interact with each other via a direct memory access channel.

2. A method for optimizing attention mechanism inference in a large language model on a heterogeneous system as described in claim 1, characterized in that, The method decomposes the model structure through dependencies and assigns computational tasks to different computational units for execution. include: Step 1: Assign matrix multiplication and RoPE calculation to computation units in different task queues; the matrix multiplication includes: MatMul(input, q_proj), MatMul(input, k_proj), MatMul(input, v_proj), and the RoPE calculation includes: loading the RoPE cos / sin matrix, RoPE(query), and RoPE(key); Step 2: The neural network processor executes the MatMul(input, q_proj) operation, while the graphics processor loads the RoPEcos / sin matrix. Step 3: After MatMul(input, q_proj) is completed, the neural network processor executes MatMul(input, k_proj); at the same time, after the RoPE cos / sin matrix is ​​loaded, the graphics processor executes RoPE(query). Step 4: After MatMul(input, k_proj) is completed, the neural network processor executes MatMul(input, v_proj); at the same time, after RoPE(query) is completed, the graphics processor executes RoPE(key).

3. The method as described in claim 2, characterized in that, The allocation of computing units is based on the computing task and its required computing power and accuracy, including the following: MatMul(input, q_proj) runs on a neural network processor; MatMul(input, k_proj) runs on a neural network processor; MatMul(input, v_proj) runs on a neural network processor; RoPE cos / sin matrix loading runs on the graphics processor; RoPE (query) runs on the graphics processor; RoPE(key) runs on the graphics processor.

4. The method as described in claim 2, characterized in that, The dependencies include strong dependencies and weak dependencies; wherein, the strong dependency is a data dependency, where the computation of one task depends on the output of another computation task; the weak dependency is a device dependency, where different tasks are executed in the same device according to their order of execution. The operation of RoPE(query) depends on loading the RoPE cos / sin matrix, which depends on MatMul(input, q_proj); the operation of RoPE(key) depends on loading the RoPE cos / sin matrix, which depends on MatMul(input, k_proj).

5. The method as described in claim 2, characterized in that, The execution time on the neural network processor computing unit overlaps with the execution time on the graphics processor computing unit.

6. A hardware system for implementing the method as described in any one of claims 2-5, characterized in that, The hardware system includes a memory and a processor; the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 2-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 2-5.

Images