Heterogeneous system and method for efficiently realizing attention mechanism of large language model and application

Abstract

The invention discloses a heterogeneous system for efficiently realizing an attention mechanism of a large language model. The heterogeneous system comprises a controller, a computing unit and a memory, the computing unit comprises a neural network processor and a graphics processor; the memory comprises a local high-speed storage module; the controller controls the computing unit through an access interface and a data bus, and is connected with the memory through the computing unit; the computing unit and the local high-speed storage module are in bidirectional data connection through a storage interface and a data bus; the heterogeneous system performs data transmission and exchange with an external global storage module; and the local high-speed storage module and the global storage module carry out connection and data interaction in a direct memory access channel form. The invention further discloses a reasoning optimization method implemented through the heterogeneous system, and the reasoning optimization method has wide application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model inference optimization, and relates to a heterogeneous system, method and application for efficiently implementing attention mechanism of a large language model. BACKGROUND

[0002] Current mainstream large model inference systems all adopt a heterogeneous architecture, for example, the latest architecture of Nvidia integrates TensorCore and CUDA Core, TensorCore has high computing power and can process INT4 / INT8 / FP8 / FP16 low-precision data; the computing power of CUDA Core is not so high, but it can process FP32, TF32, FP64 and other high-precision data. Other manufacturers also adopt similar solutions, which can be GPU+NPU heterogeneous or DSP+NPU heterogeneous, wherein GPU and DSP process high-precision units, and NPU processes low-precision units.

[0003] In the design of a large language model, a Transformer architecture is adopted, which has both large-scale matrix multiplication linear operations and RoPE, SiLU, Softmax and other nonlinear operations. Nonlinear operations must be completed on an operation unit meeting the FP32 precision requirement in order to meet the precision requirement. The heterogeneous system can well meet the needs of large model inference deployment.

[0004] However, the low-precision part and the high-precision part in the structure of a large language model are serial, and if only the original order of the model is executed, the low-precision device and the high-precision device on the inference system will run alternately, and the parallel working mechanism of the two parts on the heterogeneous system cannot be fully utilized.

[0005] In addition, FlashAttention and subsequent FlashAttention2, FlashAttention3 and the like optimize Q K, Softmax, QK V and the like, but do not optimize the matrix multiplication operation of Q, K, V and the weight and RoPE on the heterogeneous system. SUMMARY

[0006] In order to solve the problems in the prior art, the purpose of the present application is to provide a heterogeneous system, method and application for efficiently implementing attention mechanism of a large language model. In the attention mechanism part of the large language model realized based on the Transformer, the MatMul operation is performed on the MAC with high computing power and low precision, and the RoPE part operation is performed on the floating point calculation unit with lower computing power and high precision. The two parts are run in parallel, which improves the utilization of MAC and meets the precision requirement of large model inference.

[0007] The application is achieved by the following technical solutions:

[0008] The application provides a heterogeneous system for efficiently implementing an attention mechanism of a large language model, and the heterogeneous system comprises:

[0009] a controller, a computing unit and a memory; the computing unit comprises a neural network processor (NPU) and a graphics processing unit (GPU); and the memory comprises a local high-speed storage module;

[0010] In one specific embodiment, the heterogeneous system performs data transmission and exchange with an externally connected global storage module.

[0011] The neural network processor is used for processing low-precision data including INT4, INT8, FP8 and FP16, and the graphics processing unit is used for processing high-precision data including FP32, TF32 and FP64.

[0012] The controller controls the computing unit through an access interface and a data bus, and is connected to the memory via the computing unit.

[0013] The computing unit and the local high-speed storage module are connected through a storage interface and a data bus for bidirectional data connection, and the local high-speed storage module and the global storage module can be connected and exchange data through a direct memory access channel or the like.

[0014] In the specific implementation process of the application, the weights in the large model are stored in the global storage module, and the data transmission between the components in the heterogeneous system is realized by using the local high-speed storage module inside the system.

[0015] In the heterogeneous system, the neural network processor is a low-precision high-computing-power computing unit, which is used for performing a matrix multiplication task in the large language model.

[0016] The graphics processing unit is a high-precision low-computing-power computing unit, which is used for performing a RoPE-related high-precision operation task.

[0017] The controller is used for scheduling tasks to different computing units based on a scheduling strategy of data dependency relationship and task priority.

[0018] The local cache module in the storage module provides common access of the low-precision and high-precision computing units.

[0019] In the application, the descriptions of "high computing power", "low precision", "low computing power" and "high precision" are not absolute high or low, but relatively high or low in the system.

[0020] In one specific embodiment of the heterogeneous system, the controller statically binds specific types of inference computing tasks to preset computing units, wherein tasks related to weight matrix multiplication are mapped to the neural network processor, and tasks related to RoPE function calculation are mapped to the graphics processor.

[0021] The application also provides a method for optimizing large language model attention mechanism inference on a heterogeneous system. The method breaks down the model structure by relying on the relationship and reasonably allocates computing tasks to different computing units for execution. The method comprises the following steps:

[0022] Step one, allocate matrix multiplication and RoPE calculation to computing units in different task queues; the matrix multiplication includes MatMul(input, q_proj), MatMul(input, k_proj), and MatMul(input, v_proj), and the RoPE calculation includes loading RoPE cos / sin matrix, RoPE(query), and RoPE(key);

[0023] Step two, the neural network processor performs MatMul(input, q_proj) operation, and the graphics processor loads RoPE cos / sin matrix at the same time;

[0024] Step three, after the MatMul(input, q_proj) operation is completed, the neural network processor performs MatMul(input, k_proj) operation; at the same time, after the RoPE cos / sin matrix is loaded, the graphics processor operates RoPE(query);

[0025] Step four, after the MatMul(input, k_proj) operation is completed, the neural network processor performs MatMul(input, v_proj); at the same time, after the RoPE(query) operation is completed, the graphics processor operates RoPE(key).

[0026] After the above calculation is completed, the calculation result is returned.

[0027] In the implementation process of the method, the computing units are allocated according to the computing tasks and the required computing power and accuracy requirements, including the following:

[0028] MatMul(input, q_proj) runs on a high-computing-power low-precision component (NPU);

[0029] MatMul(input, k_proj) runs on a high-computing-power low-precision component (NPU);

[0030] MatMul(input, v_proj) runs on a high-performance low-precision component (NPU);

[0031] RoPE cos / sin matrix loading runs on a low-performance high-precision component (GPU);

[0032] RoPE(query) runs on a low-performance high-precision component (GPU);

[0033] RoPE(key) runs on a low-performance high-precision component (GPU);

[0034] The dependency relationship includes a strong dependency relationship and a weak dependency relationship; wherein the strong dependency relationship is a data dependency, and the calculation of one task depends on the output of another calculation task; and the weak dependency relationship is a device dependency, and when different tasks run in the same device, they are executed according to the order.

[0035] In one specific embodiment, the operation of RoPE(query) depends on RoPE cos / sin matrix loading and depends on MatMul(input, q_proj); and the operation of RoPE(key) depends on RoPE cos / sin matrix loading and depends on MatMul(input, k_proj).

[0036] In one specific embodiment, the matrix multiplication operations MatMul(input, q_proj), MatMul(input, k_proj) and MatMul(input, v_proj) are generally operated in a low-precision data format when executed;

[0037] While the neural network processor (low-precision high-performance calculation unit) performs matrix multiplication, the graphics processor (high-precision low-performance calculation unit) loads high-precision data required for RoPE calculation and performs parallel RoPE calculation.

[0038] In the implementation process of the method, the controller performs priority calculation according to the dependency relationship, and the calculation rules include the following:

[0039] 1) The priority of all tasks is first assigned to 0;

[0040] 2) The priority of a task is increased by 1 if the task is dependent on a task in the task queue of the calculation unit;

[0041] 3) The priority of a task is increased by 2 if the task is dependent on a task outside the task queue of the calculation unit;

[0042] 4) The priority of the task queue corresponding to the neural network processor is higher.

[0043] In the implementation process of the method, the controller dynamically adjusts the allocation of the computing tasks according to the priority and data dependency of the computing tasks, to ensure parallel execution of the low-precision and high-precision computing parts.

[0044] In the task allocation, the controller also considers the input and / or output data dependency of the computing tasks to avoid task execution conflicts.

[0045] In the implementation process of the method, the execution time on the neural network processor computing unit covers the execution time on the graphics processor computing unit.

[0046] In one specific embodiment, the controller can further dynamically adjust the parallelism according to the degree of idleness of the hardware resources, the computing load and the task priority.

[0047] The application also provides the above-mentioned heterogeneous system or the application of the above-mentioned method in the inference deployment of a large language model with a Transformer architecture.

[0048] The application also provides a hardware system for implementing the above-mentioned method, which comprises a memory and a processor, and the memory stores a computer program which, when executed by the processor, implements the above-mentioned method.

[0049] The application also provides a computer readable storage medium which stores a computer program, and the computer program, when executed by a processor, implements the above-mentioned method.

[0050] The application has the following beneficial effects: the application optimizes task decomposition and scheduling, so that the GPU can complete the RoPE-related cos / sin matrix loading and RoPE(query) and RoPE(key) computing tasks in parallel during the execution of MatMul(input, q_proj), MatMul(input, k_proj) and MatMul(input, v_proj) by the NPU, thereby avoiding waiting between the GPU and the NPU, improving the MAC utilization rate of the NPU, and meeting the accuracy requirement of RoPE computation; after optimization, the total execution time is shortened from T1+T2+T3+T4+T5+T6 to T1+T2+T3, which significantly improves the overall inference efficiency.

[0051] Compared with the prior art, the tasks of Attention in the prior art are executed in sequence and in series in a heterogeneous system, resulting in alternating idle conditions of GPU and NPU, the application divides the task queues on different computing units and sets priorities based on the dependency relationship, so that the tasks of NPU and GPU can be staggered and parallel in time. RoPE related tasks are completed during the execution of the main matrix multiplication task of NPU, which fully utilizes the computing power of GPU in the idle period, realizes the efficient cooperation of heterogeneous computing resources, and significantly reduces the total time consumption of reasoning while ensuring the calculation accuracy of RoPE, improving the overall operation efficiency of large language models on the heterogeneous system. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0053] Figure 1 is a heterogeneous system structure diagram suitable for implementing the attention mechanism calculation of large language models of the present application.

[0054] Figure 2 is a flowchart of Attention calculation (optimized part) in the Transformer architecture of the present application.

[0055] Figure 3 is a diagram showing the calculation tasks, dependency relationships and priorities of the present application.

[0056] Figure 4 is a parallel execution flowchart of the attention mechanism based on the Transformer architecture of the present application on a heterogeneous system (GPU+NPU). DETAILED DESCRIPTION

[0057] The present application will be further described in detail in combination with the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present application are the general knowledge and common sense of the art, and the present application does not have special restrictions.

[0058] Although the input in the large language model Attention is respectively multiplied by the matrix of weights q_proj, k_proj and v_proj, there is no dependency between them from the data dependency point of view, but the specific deployment and running on hardware are still serial for NPU due to the limitation of computing power and bandwidth. The matrix multiplication of Query, Key and Value with the weight and the RoPE operation are not strong data dependent, if the dependency relationship is not disassembled and reasonably planned to the execution component, only sequential execution can be performed, first the matrix multiplication part is executed, the low-precision component of the heterogeneous system is working, that is, NPU sequentially executes Query > Key > Value; the RoPE part is executed, the high-precision component of the heterogeneous system is working, that is, GPU executes RoPE(query), RoPE(key). The parallel characteristics of the low-precision component and the high-precision component of the heterogeneous system cannot be fully utilized.

[0059] According to the data dependency relationship of the structure of the model, the structure of the model is disassembled and reasonably planned to the component, the low-precision high-computing component is fully utilized, and the MAC (multiply-add operation unit) utilization rate is improved; in the gap of high-computing component calculation, the RoPE is processed by using the high-precision low-computing component, the MAC utilization rate is improved, and the precision requirement is met.

[0060] In order to improve the execution efficiency of different components of the heterogeneous system, the data exchange efficiency between them is required to be high, the special heterogeneous system designed in the application exchanges data through the large language model attention mechanism, the hardware layer pre-sets the task static mapping and the shared local cache module exchange channel, so that the NPU matrix multiplication and the high-precision GPU RoPE operation can be long-term parallel.

[0061] As shown in Figure 1 To adapt to the parallel and high-speed large language model execution efficiency, the heterogeneous system in the application has the following key designs:

[0062] 1. Controller / Scheduler: responsible for coordinating GPU / NPU work, loading and writing local high-speed storage (SRAM) and global storage (DDR);

[0063] 2. Computing unit: including high-computing low-precision component NPU and low-computing high-precision component GPU; these components can work independently, including calculation, loading and writing SRAM / DDR work; the low-precision high-computing component processes data including INT4, INT8, FP8, FP16, etc., and the high-precision low-computing component processes data including FP32, TF32, FP64, etc.

[0064] 3. Memory: There is local high-speed storage (SRAM) in the system, and NPU and GPU can read and write; in addition, there is global storage (DDR) outside the heterogeneous system. The DDR storage uses a relatively low frequency, but the data volume is large, and the data read / write speed requirement is relatively not so high; the SRAM storage uses a high frequency, and the data volume is not so large, and the data read / write speed requirement is fast. In the large model use scenario of the application, the DDR is used to store weight data; the SRAM is used to store the intermediate data of each computing unit (NPU and GPU) after completing the respective calculation, for data exchange after the computing unit, in addition, the SRAM is also used to pre-load the cos / sin matrix required for RoPE calculation, and the subsequent computing unit does not need to read data from the global data again to speed up the calculation.

[0065] In the application, the descriptions of “high computing power”, “low precision”, “low computing power” and “high precision” are not absolute high and low, but relatively high and relatively low in the system.

[0066] In one specific embodiment of the application, the large language model to be processed is Figure 2 , Figure 2 The generation process of Query, Key and Value in the multi-head attention mechanism based on the Transformer architecture is shown. After the input tensor is subjected to matrix multiplication operation with three weight matrices in turn, Q, K and V tensors are obtained, and the subsequent attention calculation is adapted through dimension rearrangement. Each operation in the figure is marked with the data dimensions of the input and output, which can intuitively reflect the multi-head grouping mode and the matrix dimension change process. The three matrix multiplication tasks shown in the figure are preferentially allocated to the low-precision high-computing-power computing unit (NPU) for execution, and are scheduled in parallel with the GPU task of high-precision computing RoPE to realize efficient Attention inference under a heterogeneous system.

[0067] Specifically, RoPE depends on MatMul(input, q_proj) and MatMul(input, k_proj); MatMul(input, v_proj) has no data dependency relationship with RoPE.

[0068] In the application, MatMul(input, q_proj) and MatMul(input, k_proj) are first executed on the NPU, and at the same time, the cos / sin matrix required for RoPE calculation is loaded into the local high-speed storage space by using the GPU; after the NPU completes the calculation, the NPU calculates MatMul(input, v_proj), and at the same time, the GPU calculates RoPE(query) and RoPE(key) after loading the cos / sin matrix.

[0069] In this way, the NPU is fully utilized, the NPU is always working, and the GPU calculates the high-precision calculation part of the RoPE while the NPU is working. Both the MAC utilization of the NPU and the high requirement of the model on the RoPE calculation precision are improved.

[0070] The calculation task in the attention mechanism reasoning process is decomposed and bound on the same or different calculation units, and is dynamically scheduled according to the data dependency relationship and the delay in the specific implementation:

[0071] Among them, the calculation unit is allocated according to the calculation task and the required computing power and precision requirement:

[0072] #1: MatMul (input, q_proj) is run on a high-computing-power low-precision component (NPU);

[0073] #2: MatMul (input, k_proj) is run on a high-computing-power low-precision component (NPU);

[0074] #3: MatMul (input, v_proj) is run on a high-computing-power low-precision component (NPU);

[0075] #4: RoPE cos / sin matrix loading is run on a low-computing-power high-precision component (GPU);

[0076] #5: RoPE (query) is run on a low-computing-power high-precision component (GPU);

[0077] #6: RoPE (key) is run on a low-computing-power high-precision component (GPU);

[0078] The dependency relationship in the application includes the following:

[0079] Strong dependency relationship - data dependency: the calculation of one task depends on the output of another calculation task; including the following:

[0080] #5 depends on #4; #5 depends on #1;

[0081] #6 depends on #4; #6 depends on #2

[0082] Weak dependency relationship - device dependency: there is no obvious dependency relationship between tasks, but they run on the same component, and there is an execution sequence:

[0083] In this way, two task queues are established: #1, #2 and #3 are in one task queue; #4, #5 and #6 are in one task queue, as shown in Figure 3 The tasks in the task queue have no obvious sequence.

[0084] According to the strong dependency relationship and the weak dependency relationship, the scheduler schedules the implementation as follows:

[0085] 1. Assignment priority (the greater the priority value, the higher the priority)

[0086] 1.1 Assign all task priorities to 0 first;

[0087] 1.2 The priority of the task is increased by one if the task is dependent on the task in the task queue of the computing unit;

[0088] 1.3 The priority of the task is increased by two if the task is dependent on the task outside the task queue of the computing unit; (in order to better play the parallelism, the task dependent on another execution unit needs to be prioritized)

[0089] 1.4 Task queue grading: the computing core of the large model is NPU, and the NPU task queue is prioritized;

[0090] 2. Scheduling: scheduling based on device task queue priority

[0091] 2.1 Select the task with the highest priority from the task queue with the highest priority (here, the NPU task queue). If there are tasks with the same priority in the task queue, execute them in the order of record; in the above task setting, tasks #1 and #2 will be executed in turn;

[0092] 2.2 Select the task with the highest priority from another task queue (here, the GPU task queue). If there are tasks with the same priority in the task queue, execute them in the order of record; in the above task setting, task #4 will be executed;

[0093] 2.3 Perform according to 2.1 and 2.2 until all tasks (task #3, task #5 and task #6) are completed.

[0094] The heterogeneous system and the reasoning optimization method in the application can be applied to variants of the Transformer-based large language model, including MHA, GQA, MLA, etc.

[0095] In one specific embodiment, the time of each part is assumed to be statistically calculated as:

[0096] T1 = time {MatMul(input, q_proj)}

[0097] T2 = time {MatMul(input, k_proj)}

[0098] T3 = time {MatMul(input, v_proj)}

[0099] T4 = time {load RoPE matrix}

[0100] T5 = time {compute RoPE(q)}

[0101] T6 = time {compute RoPE(k)}

[0102] So, the time before optimization is: T1+T2+T3+T4+T5+T6

[0103] The optimal time after optimization is: T1+T2+T3, T4 / T5 / T6 are all completed in the gap time

[0104] Parallelize the serial work under the premise of meeting the accuracy requirement, thereby improving the utilization rate of MAC.

[0105] In one embodiment, as Figure 4 The allocation algorithm is as follows:

[0106] 1. The controller (Scheduler) selects the highest priority task #1 from the priority task queue (here, the NPU task queue) for execution, and the calculation time is T1;

[0107] 2. The controller (Scheduler) selects the highest priority task (#4 task to load RoPE matrix) from another task queue (here, the GPU task queue) for execution, and the calculation time is T4;

[0108] 3. The controller (Scheduler) continues to allocate the calculation task #2 MatMul(k) on the NPU from the NPU task queue, and the calculation time is T2;

[0109] 4. Task #1 and task #4 are completed;

[0110] 5. The controller (Scheduler) continues to allocate the calculation task #5 compute RoPE(q) on the GPU from the GPU task queue, and the calculation time is T5;

[0111] 6. Task #2 is completed (task #4 has been completed in step 4);

[0112] 7. The controller (Scheduler) continues to allocate the calculation task #6 compute RoPE(k) on the GPU from the GPU task queue, and the calculation time is T6;

[0113] 8. The controller (Scheduler) continues to allocate the calculation task #3 MatMul(v) on the NPU from the NPU task queue, and the calculation time is T3;

[0114] Assuming the NPU is the largest computational task and its execution time is always longer than that of the GPU, then the total execution time is: T1 + T2 + T3;

[0115] The computation time of T4, T5, and T6 on the GPU is overwhelmed by the computation time of the NPU.

[0116] Example 1# This invention is used in the Llama-2-7B model:

[0117] Llama-2-7B parameters:

[0118] n_embd = 4096; n_head = 32; n_embd_head = 128

[0119] n_gpq = 1; n_head_kv = 32

[0120] During the large model inference prefill phase, n_tokens are set to 512, 256, 128, 64, 32, and 16 respectively. Similarly, during the large model inference decoding phase, the same implementation is achieved by setting n_tokens = 1.

[0121] The obtained data is shown in Table 1 below:

[0122] Table 1. Task time consumption of the proposed method and traditional method in the Llama-2-7B model.

[0123]

[0124] Example 2# This invention is used in the Llama-3-8B model:

[0125] Llama-3-8B parameters:

[0126] n_embd = 4096; n_head = 32; n_embd_head = 128

[0127] n_gpq = 4; n_head_kv = 8

[0128] During the large model inference prefill phase, n_tokens are set to 512, 256, 128, 64, 32, and 16 respectively. Similarly, during the large model inference decoding phase, the same implementation is achieved by setting n_tokens = 1.

[0129] The obtained data is shown in Table 2 below:

[0130] Table 2 Task time consumption table of the method of the application and the traditional method in Llama-3-8B model

[0131]

[0132] Data analysis:

[0133] From the above specific implementation on Llama-2-7B and Llama-3-8B models, it can be seen that:

[0134] 1. As the premise of the invention: T4 < T1; T5 < T2; T6 < T3. That is, the calculation on the GPU is carried out at the same time as the NPU calculation and is submerged in the NPU calculation process.

[0135] 2. Whether in the Prefill stage or the Decode stage, the system and implementation method adopted in the application have different degrees of optimization compared with the traditional scheme.

[0136] 3. When the context is long (n_tokens = 512, 256), the effect is not so obvious because in this case the main body of the time consumption is the matrix multiplication operation on the NPU. This calculation includes the NPU reading weight data from global storage and specific MAC operation, which is affected by factors such as computing power and bandwidth.

[0137] 4. When Decode (n_tokens = 1), the effect decreases because although Decode is a token, the calculation unit in our heterogeneous system operates according to 4 bits, that is, the time consumption is as if n_tokens = 4.

[0138] 5. The implementation effect of Llama-3-8B is better than that of Llama-2-7B, mainly because the adoption of Llama-3-8B GQA makes the Key calculation time consumption of both NPU MatMul operation and GPU RoPE(key) operation decrease, and the main time consumption is still the MatMul operation, so the optimization ratio is higher.

[0139] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the present application can be implemented in software and can be stored on a computer readable medium, which can include random access memory (RAM), read only memory (ROM), magnetic disk or optical disk, or the like. The software implementation can comprise one or more computer program components embodied on one or more computer readable medium(s). The computer readable medium can be resident within the computing device or external to the computing device. The computer program components can also be downloaded into the computing device from an external computer or external storage device.

[0140] The present application is described in reference to the flowchart illustrations and / or block diagrams according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0141] These computer program instructions can also be stored in a computer readable medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0143] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Accordingly, the appended claims are intended to encompass all modifications and variations of the preferred embodiments which fall within the scope of the application.

[0144] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

[0145] The present application is not limited to the above embodiments. Changes and modifications that can be conceived by those skilled in the art without departing from the spirit and scope of the present application are included in the present application and are protected by the scope of the appended claims.

Claims

1. A heterogeneous system for efficiently implementing attention mechanisms of large language models, characterized in that, Comprise: Controller, computing unit, memory; The computing unit comprises a neural network processor and a graphics processor; the memory comprises a local high-speed storage module; The controller controls the computing unit through an access interface and a data bus, and is connected with the memory via the computing unit; The computing unit and the local high-speed storage module are bidirectionally connected through a storage interface and a data bus; The heterogeneous system transmits and exchanges data with an external global storage module; The local high-speed storage module and the global storage module are connected and interact data through a direct memory access channel.

2. The heterogeneous system of claim 1, wherein, The neural network processor is a low-precision high-computing-power computing unit, used for performing matrix multiplication operation tasks in a large language model; The graphics processor is a high-precision low-computing-power computing unit, used for performing RoPE related high-precision operation tasks; The controller is configured to schedule tasks to different computing units based on a scheduling strategy of data dependency relationship and task priority; The local cache module in the storage module provides common access for low-precision and high-precision computing units.

3. The heterogeneous system of claim 2, wherein, The controller statically binds specific types of inference computing tasks to preset computing units, wherein tasks related to weight matrix multiplication are mapped to the neural network processor, and tasks related to RoPE function calculation are mapped to the graphics processor.

4. A method for optimizing large language model attention mechanism inference on a heterogeneous system, characterized in that, The method decomposes the model structure by dependency relationship, and allocates computing tasks to different computing units for execution. Comprise: Step one, allocate matrix multiplication operation and RoPE calculation to computing units in different task queues; the matrix multiplication operation comprises MatMul(input, q_proj), MatMul(input, k_proj) and MatMul(input, v_proj), and the RoPE calculation comprises loading RoPE cos / sin matrix, RoPE(query) and RoPE(key); Step two, the neural network processor executes MatMul(input, q_proj) operation, and the graphics processor loads RoPE cos / sin matrix at the same time; Step three, after the MatMul(input, q_proj) operation is completed, the neural network processor executes MatMul(input, k_proj) operation; at the same time, after the RoPE cos / sin matrix is loaded, the graphics processor operates RoPE(query); Step four, after the MatMul(input, k_proj) operation is completed, the neural network processor executes MatMul(input, v_proj); at the same time, after the RoPE(query) operation is completed, the graphics processor operates RoPE(key).

5. The method of claim 4, wherein, According to the computing task and the required computing power and precision requirement, the computing unit is allocated, including the following: MatMul(input, q_proj) runs on the neural network processor; MatMul(input, k_proj) runs on the neural network processor; MatMul(input, v_proj) runs on a neural network processor; RoPE cos / sin matrix load runs on a graphics processor; RoPE(query) runs on a graphics processor; RoPE(key) runs on a graphics processor.

6. The method of claim 4, wherein, The dependency relationship includes a strong dependency relationship and a weak dependency relationship; wherein the strong dependency relationship is a data dependency, and the calculation of one task depends on the output of another calculation task; the weak dependency relationship is a device dependency, and when different tasks run in the same device, they are executed according to the sequence; The operation of RoPE(query) depends on RoPE cos / sin matrix load and MatMul(input, q_proj); the operation of RoPE(key) depends on RoPE cos / sin matrix load and MatMul(input, k_proj).

7. The method of claim 4, wherein, The time of execution on the neural network processor calculation unit covers the execution time on the graphics processor calculation unit.

8. The use of the heterogeneous system of any one of claims 1-3, or the method of any one of claims 4-7, in inference deployment of a large language model of a Transformer architecture.

9. A hardware system for implementing the method of any of claims 4-7, characterized by, The hardware system comprises a memory and a processor; the memory stores a computer program, and when the computer program is executed by the processor, the method of any one of claims 4-7 is implemented.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 4-7.

Images