A model reasoning optimization method, device, storage medium and program product
By generating a calculation unit index table to allocate the calculation unit index corresponding to each attention head, the repeated calculation problem caused by the calculation unit's inability to separate the number of attention heads is solved, the data reading performance and throughput are improved, and the artificial intelligence chip is fully utilized. hardware features.
Patent Information
- Application Number
- CN202411127891.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-16
AI Technical Summary
In the calculation of multi-head attention mechanism of large language models, the calculation unit cannot decipher the number of attention heads, resulting in repeated calculations of some calculation units, resulting in poor read data performance and low throughput.
By generating a computing unit index table, allocating the computing unit index corresponding to each attention head, avoiding repeated calculations of the computing unit, improving the parallel service capabilities of the computing unit, and making full use of the hardware characteristics of the artificial intelligence chip.
Improves the read data performance and throughput, avoiding the case where the computing unit repeatedly calculates the attention head, thereby better making use of the hardware characteristics of the artificial intelligence chip.
Smart Images

Figure CN118656218B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a model reasoning optimization method, device, storage medium and program product. Background Art
[0002] In the field of model deployment optimization of large language models, with the emergence of PagedAttention technology, the reasoning throughput of large language models has been improved. In practical applications, it is of great significance to support this technology in the reasoning deployment of large language models based on the characteristics of acceleration hardware and make it closer to hardware design principles to improve throughput.
[0003] In related technologies, an AI chip includes multiple computing units. During the calculation process of the multi-head attention mechanism, each computing unit can independently access and calculate the data of an attention head, and multiple computing units need to access and calculate synchronously (that is, the behavior of multiple computing units needs to be consistent). In practical applications, it is inevitable that the number of attention heads cannot be divided by the number of computing units. At this time, in order to make the behavior of multiple computing units consistent, some computing units will repeatedly calculate the attention heads, resulting in poor data reading performance and low throughput. Summary of the invention
[0004] The embodiments of the present application provide a model reasoning optimization method, device, storage medium and program product for giving full play to the hardware characteristics of artificial intelligence chips and improving data reading performance and throughput.
[0005] On the one hand, an embodiment of the present application provides a model reasoning optimization method, including:
[0006] Receive prompt content, and obtain the target slot index corresponding to the prompt content in the slot;
[0007] Generate a computing unit index table corresponding to the prompt content according to the target slot index, wherein the computing unit index table includes: an index of a computing unit for calculating each attention head when performing a multi-head attention mechanism on the prompt content;
[0008] Locking a corresponding target physical storage block for the prompt content;
[0009] During the model inference process, the prompt content is calculated based on the computing unit index table and the target physical storage block to obtain a calculation result.
[0010] On the one hand, an embodiment of the present application provides a model reasoning optimization device, including:
[0011] An acquisition module, used for receiving prompt content and acquiring a target slot index corresponding to the prompt content in the slot;
[0012] An allocation module is used to generate a computing unit index table corresponding to the prompt content according to the target slot index, wherein the computing unit index table includes: when performing a multi-head attention mechanism calculation on the prompt content, an index of a computing unit for each attention head is calculated; and a corresponding target physical storage block is locked for the prompt content;
[0013] The inference module is used to calculate the prompt content based on the computing unit index table and the target physical storage block to obtain a calculation result during the model inference process.
[0014] Optionally, the acquisition module is specifically used for:
[0015] If the prompt content is in the pre-filling stage, an empty slot in the slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content;
[0016] If the prompt content is in a decoding stage, a target slot index of a slot in the slot that has been allocated to the prompt content is obtained.
[0017] Optionally, the acquisition module is specifically used for:
[0018] If the prompt content is in the pre-filling stage, traverse the multiple slots included in the slot in order from the beginning to the end;
[0019] When an empty slot among the multiple slots is traversed, the empty slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content.
[0020] Optionally, the allocation module is specifically used for:
[0021] From the remaining physical storage blocks in the video memory, a corresponding target physical storage block is allocated for the prompt content, and the target physical storage block is locked.
[0022] Optionally, the allocation module is specifically used for:
[0023] According to the target slot index, the number of attention heads included in the multi-head attention mechanism, and the number of computing units, a computing unit index table corresponding to the prompt content is generated.
[0024] Optionally, the reasoning module is specifically used for:
[0025] During the model inference process, the following operations are performed for the multiple attention heads in the multi-head attention mechanism:
[0026] Based on the index of the computing unit corresponding to an attention head in the computing unit index table, scheduling the computing task of the attention head to the corresponding computing unit;
[0027] The computing unit performs attention mechanism calculation on the head data corresponding to the attention head in the prompt content to obtain a corresponding calculation result, wherein the key-value data generated during the attention mechanism calculation process is cached in the target physical storage block.
[0028] Optionally, each of the multiple attention heads corresponds to an index of a computing unit.
[0029] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned model reasoning optimization method when executing the computer program.
[0030] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned model reasoning optimization method.
[0031] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned model reasoning optimization method.
[0032] In an embodiment of the present application, after receiving the prompt content, a computing unit index table corresponding to the prompt content is generated according to the target slot index corresponding to the prompt content in the slot. The computing unit index table includes: when calculating the prompt content with a multi-head attention mechanism, the index of the computing unit for each attention head is calculated. In this way, during the model reasoning process, based on the computing unit index table, the computing task of each attention head can be assigned to the corresponding computing unit for calculation. This method adjusts the control granularity from the original data block (Block) granularity to each attention head under the Block, so as to avoid the situation where the computing unit repeatedly calculates the attention head. From a macro perspective, it can better allow the computing units to serve in parallel, thereby making full use of the hardware characteristics of the artificial intelligence chip and improving the data reading performance and throughput. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0034] Figure 1 A schematic diagram of a key-value cache provided in an embodiment of the present application Figure 1 ;
[0035] Figure 2 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;
[0036] Figure 3 A flowchart of a model reasoning optimization method provided in an embodiment of the present application;
[0037] Figure 4 A schematic diagram of a slot provided in an embodiment of the present application Figure 1 ;
[0038] Figure 5 A schematic diagram of a slot provided in an embodiment of the present application Figure 2 ;
[0039] Figure 6 A schematic diagram of a key-value cache provided in an embodiment of the present application Figure 2 ;
[0040] Figure 7 A schematic diagram of the structure of a model reasoning optimization device provided in an embodiment of the present application;
[0041] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0043] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.
[0044] Token: refers to a word or tag, which is the smallest unit into which text data is segmented. It can be a single character, a whole word or a phrase. A complete sentence includes multiple tokens. In natural language processing, tokens are the basic units for models to process and understand text.
[0045] Multi-Head Attention: Each Token in the input data is split into multiple smaller parts for calculation, and each part corresponds to an attention head. The split parts have lower dimensions, which allows the model to learn data representation at a finer granularity. The step of splitting Tokens in the multi-head attention mechanism is the basis for realizing the core function of multi-head attention, which allows the model to process information in parallel in multiple different representation subspaces.
[0046] PagedAttention technology: Drawing on the idea of virtual memory page management, the key-value cache (KVCache) in the self-attention mechanism calculation is regarded as a collection of dynamically managed data blocks (Blocks). A Block includes: key-value data associated with multiple (for example, 16) tokens. Accordingly, a Block can be mapped to a physical storage block in the video memory, so that the key-value data of multiple tokens in a Block can be actually stored in the mapped physical storage block.
[0047] In actual applications, a complete sentence includes multiple tokens. A complete sentence is split into a sequence of blocks with a minimum granularity of Block. Each Block contains key-value data associated with multiple tokens. Then the key-value data of multiple tokens in each Block is mapped to a physical storage block in the video memory for storage. The Block management method can effectively reduce the waste of video memory in KVCache, thereby improving throughput and memory utilization in multi-request concurrent reasoning scenarios.
[0048] For example, see Figure 1 The video memory includes 6 physical storage blocks, namely physical storage block 0, physical storage block 1, physical storage block 2, physical storage block 3, physical storage block 4, and physical storage block 5. Each physical storage block can store the key-value data of 4 tokens, among which physical storage block 0, physical storage block 1, physical storage block 3, and physical storage block 5 have already stored other key-value data.
[0049] Assume that a complete sentence consists of 6 tokens, namely "I", "am", "fine", "it", "is", and "OK".
[0050] If the key value data of the above 6 tokens are not saved according to the Block management method, the key value data of the 6 tokens need to be saved continuously in the video memory. Among the 6 physical storage blocks of the video memory, only physical storage block 2 and physical storage block 4 are currently idle, and each physical storage block can save the key value data of 4 tokens. Although the total storage space of physical storage block 2 and physical storage block 4 can meet the storage requirements of the key value data of 6 tokens, physical storage block 2 and physical storage block 4 are not continuous. Therefore, the key value data of the above 6 tokens cannot be saved in the video memory, and can only wait for other physical storage blocks to release storage space.
[0051] If the key-value data of the above 6 tokens are saved according to the Block management method, the key-value data of the above 6 tokens can be divided into two blocks. The first block includes the key-value data of the four tokens: "I", "am", "fine", and "it"; the second block includes the key-value data of the two tokens: "is" and "OK".
[0052] When mapping two blocks to video memory, physical storage block 2 and physical storage block 4 in the video memory are in idle state. Therefore, the key value data of the four tokens in the first block are saved in physical storage block 2 in sequence, and the key value data of the two tokens in the second block are saved in physical storage block 4 in sequence, thereby effectively reducing the waste of video memory in KVCache.
[0053] The following is a brief introduction to the system architecture diagram applicable to the technical solution of the embodiment of the present application. It should be noted that the system architecture diagram introduced below is only used to illustrate the embodiment of the present application and is not a limitation.
[0054] refer to Figure 2 , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application. The artificial intelligence chip 200 at least includes: a video memory 201 and multiple computing units 202.
[0055] The video memory 201 may be a high bandwidth memory (HBM for short) or other types of memory.
[0056] The computing unit 202 can be used to execute various operators, such as a multi-head attention mechanism operator, a convolution operator, etc. When executing an operator, the computing unit 202 can load data from the video memory 201 for calculation to obtain a calculation result. The calculation result or the intermediate result generated during the calculation process can be written back to the video memory 201.
[0057] In addition to the above-mentioned structure, the artificial intelligence chip 200 in the present application may also include other structures, which are not specifically limited in the present application.
[0058] The artificial intelligence chip 200 may be: a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a domain specific architecture (DSA), etc.
[0059] In practical applications, in the field of model deployment optimization of large language models, with the emergence of PagedAttention technology, the reasoning throughput of large language models has been improved. In practical applications, it is of great significance to support this technology in the reasoning deployment of large language models based on the characteristics of acceleration hardware and make it closer to hardware design principles to improve throughput.
[0060] In related technologies, an AI chip includes multiple computing units. During the calculation process of the multi-head attention mechanism, each computing unit can independently access and calculate the data of an attention head, and multiple computing units need to access and calculate synchronously (that is, the behavior of multiple computing units needs to be consistent). In practical applications, it is inevitable that the number of attention heads cannot be divided by the number of computing units. At this time, in order to make the behavior of multiple computing units consistent, some computing units will repeatedly calculate the attention heads, resulting in poor data reading performance and low throughput.
[0061] For example, the multi-head attention mechanism is set to include 40 attention heads, and the artificial intelligence chip includes 16 computing units. At this time, the number of attention heads cannot be divided by the number of computing units, but multiple computing units need to access memory and perform calculations synchronously (that is, the behavior of multiple computing units needs to be consistent). Therefore, each computing unit can only be set to perform calculations on three attention heads, and 16 computing units perform calculations on a total of 48 attention heads. In this case, there must be 8 attention heads that are repeatedly calculated, which makes it impossible to fully utilize the hardware characteristics of the artificial intelligence chip, resulting in poor data reading performance and low throughput.
[0062] In view of this, this application provides a process of a model reasoning optimization method, such as Figure 3 As shown, the process of the method is executed by a computer device, the computer device comprising Figure 2 The artificial intelligence chip shown includes the following steps:
[0063] Step 301, receiving prompt content, and obtaining the target slot index corresponding to the prompt content in the slot.
[0064] Specifically, one or more requests are received, each request including prompt content and a request number (ID), wherein the prompt content may be a sentence including multiple tokens.
[0065] The received requests are scheduled according to the remaining physical storage blocks in the video memory, and the requests that meet the conditions are encapsulated, wherein the requests that meet the conditions refer to requests whose current remaining physical storage blocks can meet the storage requirements.
[0066] For the prompt content in the request that meets the conditions, the target slot index corresponding to the prompt content in the slot is obtained. In addition, the scheduling result of the request that meets the conditions also includes: the execution stage of the prompt content, wherein the execution stage includes: a prefill stage and a decode stage.
[0067] In some embodiments, if the prompt content is in the pre-filling stage, an empty slot in the slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content. If the prompt content is in the decoding stage, the target slot index of the slot in the slot that has been allocated to the prompt content is obtained.
[0068] Specifically, a slot refers to an array. After the slot is initialized, the slot is an empty array. A data bit in the array is also a slot, and the position of a slot in the array is represented by a slot index. When a placeholder corresponding to the prompt content is inserted into an empty slot (i.e., a blank data bit), the empty slot is changed to a slot allocated to the prompt content.
[0069] When the scheduling result indicates that the prompt content needs to be pre-filled, the prompt content is in the pre-filling stage. At this time, there is no slot in the slot allocated to the prompt content. Therefore, an empty slot is selected from the slot, and a placeholder corresponding to the prompt content is added to the empty slot, indicating that the empty slot is allocated to the prompt content, and the slot index of the empty slot is recorded as the target slot index corresponding to the prompt content.
[0070] For example, see Figure 4 , set the slot to include 10 slots, and the slot indexes of the 10 slots are: slot index 0 to slot index 9. All 10 slots are empty slots. Receive 2 sentences that need to be pre-filled, sentence 0 and sentence 1, then add a placeholder to the slot corresponding to slot index 0, and associate sentence 0 with slot index 0; add a placeholder to the slot corresponding to slot index 1, and associate sentence 1 with slot index 1.
[0071] In some embodiments, multiple slots included in the slot are traversed in order from the beginning to the end. When an empty slot among the multiple slots is traversed, the empty slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content.
[0072] Specifically, for multiple prompt contents received continuously, the calculation speeds for these multiple prompt contents may be different. After the prompt content with fast calculation is completed, the slot corresponding to the prompt content in the slot will be vacated and reallocated to other prompt contents. Therefore, discontinuous empty slots will appear in the slot. In view of this situation, when allocating empty slots for prompt contents, the present application can traverse each slot in order from the beginning to the end, and when traversing to the first empty slot, the empty slot is allocated to the prompt content, so that the prompt content that needs to be pre-filled is allocated to the empty slot at the front as much as possible, thereby making full use of each slot in the slot.
[0073] For example, see Figure 5 , the slot is set to include 10 slots, and the slot indexes of the 10 slots are: slot index 0 to slot index 9. The slots corresponding to slot index 0 to slot index 5 have been allocated to other sentences, and the other slots are empty slots.
[0074] After a period of time, the sentences associated with slot index 2 and slot index 4 are calculated, so the placeholders in the slots corresponding to slot index 2 and slot index 4 are cleared, that is, the slots corresponding to slot index 2 and slot index 4 are changed to empty slots.
[0075] At this time, two sentences that need to be pre-filled are received, namely sentence 0 and sentence 1.
[0076] For sentence 0, traverse the multiple slots contained in the slot in order from the beginning to the end. When traversing to the first empty slot (corresponding to slot index 2), add a placeholder to the empty slot and associate sentence 0 with slot index 2 for recording.
[0077] In the same way, the empty slot corresponding to slot index 4 can be traversed, and then a placeholder is added to the empty slot, and sentence 1 is associated with slot index 4 and recorded.
[0078] When the scheduling result indicates that the prompt content needs to be decoded, the prompt content is in the decoding stage. At this time, the slot contains the slot allocated to the prompt content (that is, the slot allocated in the pre-filling stage). Therefore, there is no need to allocate a slot to the prompt content again, but only to obtain the target slot index of the slot allocated to the prompt content in the slot.
[0079] For example, see Figure 4, it is set that the slot includes 10 slots, and the slot indexes of the 10 slots are: slot index 0 to slot index 9. The slots corresponding to slot index 0 and slot index 1 have been allocated to sentence 0 and sentence 1.
[0080] In the new round of scheduling, sentence 0 to be decoded is received. Since the slot includes the slot assigned to sentence 0 (corresponding to slot index 0), slot index 0 corresponding to sentence 0 is directly obtained.
[0081] Step 302: Generate a computing unit index table corresponding to the prompt content according to the target slot index.
[0082] Specifically, the computing unit index table includes: when performing a multi-head attention mechanism on the prompt content, calculating the index of the computing unit of each attention head, each attention head corresponds to the index of a computing unit.
[0083] In some embodiments, a computing unit index table corresponding to the prompt content is generated based on the target slot index, the number of attention heads included in the multi-head attention mechanism, and the number of computing units.
[0084] In the specific implementation, the following formula (1) is first used to calculate the index k of the computing unit used to calculate the attention head i:
[0085]
[0086] Among them, batch_id represents the target slot index, h represents the number of attention heads, and s represents the number of computing units. .
[0087] Based on the obtained index k of the computational unit for computing h attention heads, a computational unit index table is constructed.
[0088] For example, the target slot index is set to 0, the number of attention heads is 40, namely attention head 0 to attention head 39. The number of computation units is 16, namely computation unit 0 to computation unit 15.
[0089] Using the above formula (1), it can be obtained that the calculation units for calculating attention heads 0 to 15 are respectively calculation units 0 to 15 (i.e., the first round of calculation); the calculation units for calculating attention heads 16 to 31 are respectively calculation units 0 to 15 (i.e., the second round of calculation); the calculation units for calculating attention heads 32 to 39 are respectively calculation units 0 to 7 (i.e., the third round of calculation). In the third round of calculation, calculation units 8 to 15 can calculate the prompt content associated with other slot indexes, without having to repeatedly calculate some attention heads of the prompt content associated with the target slot index.
[0090] Step 303: Lock the corresponding target physical storage block for the prompt content.
[0091] Specifically, from the remaining physical storage blocks in the video memory, a corresponding target physical storage block is allocated for the prompt content, and the target physical storage block is locked. In practical applications, one or more physical storage blocks are locked for the prompt content. Specifically, when the prompt content is split into one Block with Block as the minimum granularity, one physical storage block is locked for the prompt content; when the prompt content is split into multiple Blocks with Block as the minimum granularity, multiple physical storage blocks are locked for the prompt content.
[0092] Step 304, during the model reasoning process, the prompt content is calculated based on the computing unit index table and the target physical storage block to obtain a calculation result.
[0093] Specifically, steps 301 to 303 are the execution steps of the scheduling phase, which are specifically executed by the central processing unit (CPU) in the computer device. After the scheduling phase is completed, the artificial intelligence chip in the computer device executes the model reasoning process (i.e., step 304). The artificial intelligence chip performs multi-head attention mechanism calculation on the prompt content according to the generated computing unit index table and the locked target physical storage block to obtain the calculation result.
[0094] In actual applications, the CPU uses the method of dynamically executing Python scripts to implement the scheduling phase from step 301 to step 303, and does not modify the user script. Therefore, the user will not perceive the newly added computing unit index table; accordingly, for the user, the model reasoning process executed by the artificial intelligence chip has not changed, which allows the user to have a non-perceptual use experience.
[0095] When the scheduling stage is for pre-filling of prompt content, after the scheduling stage is completed, the artificial intelligence chip performs a pre-filling operation on the prompt content according to the generated computing unit index table and the locked target physical storage block to obtain the calculation result.
[0096] When the scheduling stage is for scheduling the decoding of the prompt content, after the scheduling stage is completed, the artificial intelligence chip performs a decoding operation on the prompt content according to the generated computing unit index table and the locked target physical storage block to obtain the calculation result.
[0097] In an embodiment of the present application, after receiving the prompt content, a computing unit index table corresponding to the prompt content is generated according to the target slot index corresponding to the prompt content in the slot, and the computing unit index table includes: when calculating the prompt content with a multi-head attention mechanism, the index of the computing unit for each attention head is calculated, so that in the model reasoning process, based on the computing unit index table, the computing task of each attention head can be assigned to the corresponding computing unit for calculation. This method adjusts the control granularity from the original Block granularity to each attention head under the Block, so as to avoid the situation where the computing unit repeatedly calculates the attention head. From a macro perspective, it can better allow the computing units to serve in parallel, thereby making full use of the hardware characteristics of the artificial intelligence chip, thereby improving the data reading performance and throughput.
[0098] In some embodiments, during the model inference process, for multiple attention heads in the multi-head attention mechanism, the following operations are performed respectively:
[0099] Based on the index of a computing unit corresponding to an attention head in the computing unit index table, the computing task of the attention head is dispatched to the corresponding computing unit. The computing unit performs attention mechanism calculation on the head data corresponding to the attention head in the prompt content to obtain the corresponding calculation result, wherein the key value data generated during the attention mechanism calculation process is cached in the target physical storage block.
[0100] Specifically, the prompt content includes multiple tokens. When performing the calculation of the multi-head attention mechanism on the prompt content, each token is split into multiple partial data according to the number of attention heads, and each partial data corresponds to an attention head. The partial data corresponding to multiple tokens constitute the head data in the prompt content that needs to be calculated by a calculation unit.
[0101] After scheduling the computation task of an attention head to the corresponding computation unit according to the computation unit index table, the computation unit performs attention mechanism calculation on partial data of multiple tokens contained in the prompt content in turn. When the attention mechanism calculation is performed on partial data of each token, corresponding key-value (KV) data will be generated. The generated key-value data is saved in the target physical storage block locked for the prompt content.
[0102] When the prompt content is pre-filled, the key-value data related to the partial data of each Token is stored in the target physical storage block. When the prompt content is decoded, the computing unit can read the key-value data cached in the pre-filling phase from the target physical storage block to decode the prompt content to generate a new Token. In the process of generating a new Token, key-value data associated with the new Token is also generated, and the key-value data is also saved in the target physical storage block. New Tokens are subsequently generated one by one in the same manner until the preset stop condition is met.
[0103] In some embodiments, for each storage space in the target physical storage block (the storage space is used to store key-value data associated with a Token), the storage space can be further divided into multiple subspaces according to the number of attention heads, each subspace corresponds to an attention head, and is used to store the key-value data corresponding to an attention head (that is, the key-value data generated when the attention mechanism is calculated on part of the Token data).
[0104] For example, see Figure 6 The video memory includes 6 physical storage blocks, namely physical storage block 0, physical storage block 1, physical storage block 2, physical storage block 3, physical storage block 4, and physical storage block 5. Each physical storage block can store the key-value data of 4 tokens, among which physical storage block 0, physical storage block 1, physical storage block 3, and physical storage block 5 have already stored other key-value data.
[0105] The prompt content is set to include 3 tokens, namely "I", "am", and "fine", and the first storage space to the third storage space in physical storage block 2 are locked for the prompt content. Taking the first storage space as an example, the first storage space can be further divided into 8 subspaces, each of which is used to store the key value data corresponding to Token "I" in an attention head. For example, subspace 0 of the 8 subspaces is set to be locked to store the key value data corresponding to Token "I" in attention head 0.
[0106] Then, in the model inference stage, after the calculation task of attention head 0 of Token "I" is scheduled to the corresponding calculation unit 0 according to the calculation unit index table, the attention mechanism calculation is performed on the partial data corresponding to attention head 0 of Token "I" through calculation unit 0 to obtain the calculation result of attention head 0; the key-value data generated during the attention mechanism calculation process is saved in subspace 0.
[0107] In an embodiment of the present application, based on the virtual memory paging management technology of the operating system, the key-value cache is further divided into blocks according to different attention heads, and the deployment control granularity is adjusted to each attention head, thereby allowing more flexible key-value cache management, thereby improving the video memory utilization and processing speed of large language model reasoning.
[0108] Based on the same technical concept, the present application embodiment provides a structural diagram of a model reasoning optimization device, such as Figure 7 As shown, the model reasoning optimization device 700 includes:
[0109] The acquisition module 701 is used to receive the prompt content and obtain the target slot index corresponding to the prompt content in the slot;
[0110] The allocation module 702 is used to generate a computing unit index table corresponding to the prompt content according to the target slot index, wherein the computing unit index table includes: when performing multi-head attention mechanism calculation on the prompt content, the index of the computing unit for each attention head is calculated; and locking the corresponding target physical storage block for the prompt content;
[0111] The reasoning module 703 is used to calculate the prompt content based on the computing unit index table and the target physical storage block to obtain a calculation result during the model reasoning process.
[0112] Optionally, the acquisition module 701 is specifically used for:
[0113] If the prompt content is in the pre-filling stage, an empty slot in the slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content;
[0114] If the prompt content is in a decoding stage, a target slot index of a slot in the slot that has been allocated to the prompt content is obtained.
[0115] Optionally, the acquisition module 701 is specifically used for:
[0116] If the prompt content is in the pre-filling stage, traverse the multiple slots included in the slot in order from the beginning to the end;
[0117] When an empty slot among the multiple slots is traversed, the empty slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content.
[0118] Optionally, the allocation module 702 is specifically configured to:
[0119] From the remaining physical storage blocks in the video memory, a corresponding target physical storage block is allocated for the prompt content, and the target physical storage block is locked.
[0120] Optionally, the allocation module 702 is specifically configured to:
[0121] According to the target slot index, the number of attention heads included in the multi-head attention mechanism, and the number of computing units, a computing unit index table corresponding to the prompt content is generated.
[0122] Optionally, the reasoning module 703 is specifically used for:
[0123] During the model inference process, the following operations are performed for the multiple attention heads in the multi-head attention mechanism:
[0124] Based on the index of the computing unit corresponding to an attention head in the computing unit index table, scheduling the computing task of the attention head to the corresponding computing unit;
[0125] The computing unit performs attention mechanism calculation on the head data corresponding to an attention head in the prompt content to obtain a corresponding calculation result, wherein the key-value data generated during the attention mechanism calculation process is cached in the target physical storage block.
[0126] Optionally, each of the multiple attention heads corresponds to an index of a computing unit.
[0127] In an embodiment of the present application, after receiving the prompt content, a computing unit index table corresponding to the prompt content is generated according to the target slot index corresponding to the prompt content in the slot, and the computing unit index table includes: when calculating the prompt content with a multi-head attention mechanism, the index of the computing unit for each attention head is calculated, so that in the model reasoning process, based on the computing unit index table, the computing task of each attention head can be assigned to the corresponding computing unit for calculation. This method adjusts the control granularity from the original Block granularity to each attention head under the Block, so as to avoid the situation where the computing unit repeatedly calculates the attention head. From a macro perspective, it can better allow the computing units to serve in parallel, thereby making full use of the hardware characteristics of the artificial intelligence chip and improving the data reading performance and throughput.
[0128] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 8 As shown, it includes at least one artificial intelligence chip 200 and a memory 801 connected to the at least one artificial intelligence chip 200. The specific connection medium between the artificial intelligence chip 200 and the memory 801 is not limited in the embodiment of the present application. Figure 8For example, the artificial intelligence chip 200 and the memory 801 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0129] In an embodiment of the present application, the memory 801 stores instructions that can be executed by at least one artificial intelligence chip 200. At least one artificial intelligence chip 200 can execute the steps of the above-mentioned model reasoning optimization method by executing the instructions stored in the memory 801.
[0130] Among them, the artificial intelligence chip 200 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the model reasoning optimization can be realized by running or executing instructions stored in the memory 801 and calling the data stored in the memory 801. Optionally, the artificial intelligence chip 200 may include one or more processing units, and the artificial intelligence chip 200 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 200. In some embodiments, the artificial intelligence chip 200 and the memory 801 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.
[0131] The artificial intelligence chip 200 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.
[0132] The memory 801 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 801 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 801 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 801 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0133] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned model reasoning optimization method.
[0134] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned model reasoning optimization method.
[0135] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0137] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0138] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0139] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0140] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A model reasoning optimization method, It is characterized in that include: Receive prompt content, and obtain the target slot index corresponding to the prompt content in the slot; Generate a computing unit index table corresponding to the prompt content according to the target slot index, wherein the computing unit index table includes: an index of a computing unit for calculating each attention head when performing a multi-head attention mechanism on the prompt content; The corresponding target physical storage block is locked for the prompt content; the prompt content includes multiple tokens; each storage space in the target physical storage block is used to store key value data associated with a token; During the model inference process, for the multiple attention heads in the multi-head attention mechanism, the following operations are performed respectively: based on the index of the computing unit corresponding to an attention head in the computing unit index table, the computing task of the attention head is scheduled to the corresponding computing unit; The computing unit performs attention mechanism calculation on the head data corresponding to the attention head in the prompt content to obtain a corresponding calculation result, wherein the key-value data generated during the attention mechanism calculation process is cached in the target physical storage block.
2. The method according to claim 1, It is characterized in that The obtaining of the target slot index corresponding to the prompt content in the slot includes: If the prompt content is in the pre-filling stage, an empty slot in the slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content; If the prompt content is in a decoding stage, a target slot index of a slot in the slot that has been allocated to the prompt content is obtained.
3. The method according to claim 2, It is characterized in that If the prompt content is in the pre-filling stage, allocating an empty slot in the slot to the prompt content, and using the slot index of the empty slot as the target slot index corresponding to the prompt content, including: If the prompt content is in the pre-filling stage, traverse the multiple slots included in the slot in order from the beginning to the end; When an empty slot among the multiple slots is traversed, the empty slot is allocated to the prompt content, and the slot index of the empty slot is used as the target slot index corresponding to the prompt content.
4. The method according to claim 1, It is characterized in that The step of locking a corresponding target physical storage block for the prompt content includes: From the remaining physical storage blocks in the video memory, a corresponding target physical storage block is allocated for the prompt content, and the target physical storage block is locked.
5. The method according to claim 1, It is characterized in that The step of generating a computing unit index table corresponding to the prompt content according to the target slot index includes: According to the target slot index, the number of attention heads included in the multi-head attention mechanism, and the number of computing units, a computing unit index table corresponding to the prompt content is generated.
6. The method according to claim 1, It is characterized in that Each of the multiple attention heads corresponds to an index of a computing unit.
7. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, It is characterized in that When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium, It is characterized in that It stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of the method as claimed in any one of claims 1 to 6.
9. A computer program product, It is characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 6.
