Methods, computing devices, media, and program products for inference scheduling of artificial intelligence models

Abstract

The present application relates to a method, a computing device, a medium and a program product for inference scheduling of an artificial intelligence model. The method comprises: determining a first predetermined token capacity for a pre-filling stage and a second predetermined token capacity for a decoding stage, the first predetermined token capacity being associated with a first cache, and the second predetermined token capacity being associated with a second cache; a scheduler determining, in a first predetermined token capacity range, a memory block size in a predetermined token number unit, for performing pre-filling based on the calculated memory block size for a pre-filling request of a corresponding size; and triggering a decoding stage once for each time the pre-filling of the first predetermined token capacity range is completed, a decoding request of the triggered decoding stage occupying a slot of the second predetermined token capacity. The present application can avoid affecting performance due to excessive splitting of memory blocks, and can avoid inference delay.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of artificial intelligence technology, and more specifically to a method, computing device, computer-readable storage medium, and computer program product for inference scheduling of an artificial intelligence model. Background Technology

[0002] Traditional AI model inference scheduling methods primarily involve sharing the same memory pool on the AI ​​chip (e.g., a graphics processing unit) for both prefill and decode requests. Memory allocation is then performed uniformly based on a scheduling policy that considers the maximum number of tokens requested (max_num_tokens). However, because prefill and decode requests share the same memory pool, there's a problem of prefill and decode tokens competing for the same cache segment. For example, competition for decode tokens can lead to excessive memory block splitting during prefill token allocation, impacting execution performance. Furthermore, memory block splitting is based solely on the remaining space in the current memory pool, with the last chunk allowed to be of arbitrary length. This can easily cause attention operators to require dynamic padding or fallback to slower computation paths (such as a fallback kernel), wasting computational resources and memory bandwidth, thus increasing inference latency.

[0003] In summary, the shortcomings of traditional artificial intelligence model inference scheduling methods are: the performance is affected by the splitting of many memory blocks, and inference latency is easily caused. Summary of the Invention

[0004] This invention provides a method, computing device, computer-readable storage medium, and computer program product for inference scheduling of artificial intelligence models, which can avoid performance impact caused by excessive memory block splitting and can avoid inference latency.

[0005] According to a first aspect of the present invention, a method for inference scheduling of an artificial intelligence model is provided, the method comprising: determining a first predetermined token capacity for a pre-filling phase and a second predetermined token capacity for a decoding phase, the first predetermined token capacity being associated with a first cache and the second predetermined token capacity being associated with a second cache; a scheduler determining a memory block size within the first predetermined token capacity, in units of predetermined tokens, for performing pre-filling for pre-filling requests of corresponding sizes based on the calculated memory block sizes; and

[0006] Each time the first predetermined token capacity range is pre-filled, a decoding phase is triggered, and the decoding request of the triggered decoding phase occupies a slot of the second predetermined token capacity.

[0007] According to a second aspect of the present invention, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the computing device to perform the method of the first aspect of the present invention.

[0008] According to a third aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0009] According to a fourth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0010] In some embodiments, performing prefilling for prefilling requests of a corresponding size based on the calculated memory block size includes: suspending any remaining prefilling requests that exceed the memory block size, pending splicing.

[0011] In some embodiments, the first cache and the second cache are buffer areas that are physically isolated from each other.

[0012] In some embodiments, the scheduler determines the memory block size in units of a predetermined number of tokens within a first predetermined token capacity range by: calculating the memory block size based on the first predetermined token capacity and the predetermined number of tokens, such that the calculated memory block size does not exceed the first predetermined token capacity and is a maximum integer multiple of the predetermined number of tokens.

[0013] In some embodiments, the size of the predetermined number of tokens is associated with the granularity of the tensor kernel used to perform inference computations for the artificial intelligence model.

[0014] In some embodiments, performing pre-filling for a pre-filling request of a corresponding size based on the calculated memory block size includes: determining whether the length of the pre-filling request is less than or equal to a first predetermined token capacity; directly pre-filling the pre-filling request in response to the length of the pre-filling request being less than or equal to the first predetermined token capacity; and calculating the memory block size based on the first predetermined token capacity and a predetermined number of tokens in response to the length of the pre-filling request being greater than the first predetermined token capacity, so as to perform pre-filling for the pre-filling request of the corresponding size based on the calculated memory block size; and suspending any remaining pre-filling requests exceeding the memory block size to await splicing.

[0015] In some embodiments, suspending a remaining prefill request that exceeds the memory block size to await splicing includes splicing the suspended remaining prefill request with the next prefill request.

[0016] In some embodiments, triggering a decoding phase includes: in response to determining that the slots of the second predetermined token capacity are full and the previous decoding phase has not yet exited, waiting until the previous decoding phase exits before triggering the current decoding phase.

[0017] This invention avoids resource contention between the pre-filling and decoding stages through physical isolation. Therefore, the allocation of pre-filling and decoding tokens in their respective caches does not result in excessive memory block splitting, thus preventing performance degradation due to excessive memory block splitting. Furthermore, by using memory blocks aligned to a predetermined number of tokens, dynamic filling is avoided, ensuring that the attention operator always runs on the optimal path. This helps optimize the computational efficiency of the graphics processing unit, reducing memory waste and inference latency. Therefore, this invention can prevent performance degradation due to excessive memory block splitting and can also avoid inference latency.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0020] Figure 1 The diagram illustrates a computing device for implementing a method for inference scheduling of an artificial intelligence model according to an embodiment of the present invention.

[0021] Figure 2 A flowchart of a method for inference scheduling for an artificial intelligence model according to some embodiments of the present invention is shown.

[0022] Figure 3 A schematic diagram of a method for inference scheduling of an artificial intelligence model according to some embodiments of the present invention is shown.

[0023] Figure 4 A flowchart is shown for a method for pre-filling memory blocks based on a calculated memory block size, according to some embodiments of the present invention.

[0024] Figure 5 A schematic diagram of a prior art method for inference scheduling of an artificial intelligence model is shown.

[0025] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0026] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0027] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0028] The following text combines Figure 5 This describes existing methods for inference scheduling in artificial intelligence models. In traditional methods, prefill requests and decoding requests compete for cache resources. For example, in the first round of scheduling, the cache allocation is as shown in Figure 510, where prefill requests (such as...) compete for cache resources. Figure 5 The "P" indicates that 1 includes 16 tokens P1-1 to P1-16, which are allocated to cache slot 512. The remaining slot 514 is allocated to a portion of prefill request 2, P2-1 to P2-4. In the second round of scheduling, cache allocation is as shown in 520, which first allocates slot 522 to the remaining portion of prefill request 2, P2-5 to P2-16; the remaining slot 524 is allocated to a portion of prefill request 3, P3-1 to P3-7. In addition, the remaining slot 526 is allocated to decoding requests (such as...). Figure 5The first token (D1-1) of the prefill request 1 is indicated by "D". Similarly, in the third round of scheduling, the cache allocation is as shown in 530. Cache slot 532 is allocated to the remaining portion P3-8 to P3-16 of prefill request 3; cache slot 534 is allocated to a portion of prefill request 4, namely P4-1 to P4-9; cache slot 536 is allocated to the second token D1-2 of decoding request 1. Cache slot 538 is allocated to the first token D2-1 of decoding request 2. It can be seen that due to the free competition between prefill and decoding requests in the cache, the cache allocation is fragmented, resulting in short remaining portions, which affects the time consumption of the attention mechanism. In addition, the competition for Decode tokens may lead to more memory block splits in the allocation of Prefill tokens in the cache, and memory block splits affect execution performance. Furthermore, the splitting of memory blocks (chunks) is only based on the remaining space in the current memory pool, and the last chunk can be of arbitrary length. Therefore, attention operators may need to be dynamically padded or fall back to slower computation paths (such as fallback kernels), thus wasting computational resources and memory bandwidth, thereby increasing inference latency.

[0029] As described above, the shortcomings of traditional artificial intelligence model inference scheduling methods are that they affect execution performance due to the splitting of many memory blocks and are prone to inference latency.

[0030] To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of the present invention provide at least one artificial intelligence chip and a method for inference scheduling for artificial intelligence models. In this method, by determining a first predetermined token capacity (associated with a first cache) for the pre-filling phase and a second predetermined token capacity (associated with a second cache) for the decoding phase, the present invention avoids resource contention between the pre-filling and decoding phases through physical isolation. The allocation of pre-filling tokens and decoding tokens in their respective caches does not result in excessive memory block splitting, thereby avoiding performance degradation due to excessive memory block splitting. Furthermore, within the first predetermined token capacity range, the scheduler determines the memory block size in units of predetermined tokens for pre-filling requests of corresponding sizes based on the calculated memory block size. Each time pre-filling within the first predetermined token capacity range is completed, a decoding phase is triggered, with the decoding requests in the triggered decoding phase occupying slots in the second predetermined token capacity. Because the present invention uses memory blocks aligned to predetermined token numbers, it avoids dynamic filling, allowing the attention operator to always run on the optimal path, thus optimizing the computational efficiency of the graphics processor and reducing memory waste and inference latency. Therefore, the present invention can avoid performance impact caused by excessive memory block splitting and can avoid inference latency.

[0031] The following combination Figure 1 A computing apparatus 100 according to an embodiment of the present invention is described, which implements a method for inference scheduling of an artificial intelligence model. For example... Figure 1 As shown, the computing device 100 may have one or more processing units and a memory communicatively connected to the one or more processing units; wherein the memory stores instructions executable by the at least one processor. The processing units include dedicated processing units such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), general-purpose computing on graphics processing units (GPGPUs), and general-purpose processing units such as CPUs. The computing device 100 also includes at least: first and second predetermined token capacity determination modules 102, a memory block determination module 104, and a decoding phase triggering module 106.

[0032] Regarding the first and second predetermined token capacity determination modules 102, they are used to determine the first predetermined token capacity for the pre-filling stage and the second predetermined token capacity for the decoding stage. The first predetermined token capacity is associated with the first cache, and the second predetermined token capacity is associated with the second cache.

[0033] Regarding the memory block determination module 104, it is used by the scheduler to determine the memory block size in units of a predetermined number of tokens within a first predetermined token capacity range, so as to perform pre-filling for pre-filling requests of the corresponding size based on the calculated memory block size.

[0034] Regarding the decoding phase triggering module 106, it is used to trigger a decoding phase once each time the pre-filling of the first predetermined token capacity range is completed, and the decoding request of the triggered decoding phase occupies the slot of the second predetermined token capacity.

[0035] The following will combine Figure 2 and Figure 3 This invention describes a method 200 for inference scheduling of an artificial intelligence model, according to embodiments of the present invention. It should be understood that method 200 can, for example, be used in... Figure 1 The described computing device 100 performs the operation. Method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0036] At step 202, computing device 100 determines a first predetermined token capacity for the pre-filling phase and a second predetermined token capacity for the decoding phase, the first predetermined token capacity being associated with a first cache and the second predetermined token capacity being associated with a second cache.

[0037] Regarding the prefill stage, it should be understood that all tokens from the input prompt are processed in parallel, the first token is output, and a key-value cache (KV Cache) is generated for future decoding.

[0038] Regarding the decoding phase, it uses a KV cache to progressively generate new tokens in an autoregressive manner. The decoding phase generates only one token at each autoregressive step, even though the input sequence length for the decoding phase is always 1.

[0039] Regarding the first predetermined token capacity, it is, for example, the token capacity P_max used only in the pre-filling phase. In some embodiments, the first predetermined token capacity is, for example, but not limited to, 1024 (i.e., P_max = 1024).

[0040] Regarding the second predetermined token capacity, it is, for example, the token capacity D_max used only in the decoding phase. The first predetermined token capacity and the second predetermined token capacity constitute the total token capacity. In some embodiments, the second predetermined token capacity is, for example, 128 (i.e., D_max = 128).

[0041] The first cache is, for example, a buffer area used only for Prefill requests (e.g., initial computation for handling input prompts). The second cache is, for example, a buffer area used only for Decode requests (e.g., iterative computation for generating tokens). In some embodiments, the first and second caches are physically isolated buffer areas from each other. For example, a Prefill request cannot occupy the Decode request buffer area, and vice versa. It should be understood that Prefill and Decode requests have their own buffer areas, and the allocation of prefill and decode tokens in their respective caches does not result in excessive memory block splits, thus avoiding performance impact due to excessive memory block splits. Test results show that the Decode latency is zero jitter, and P99 is reduced by more than 25%.

[0042] In some embodiments, the present invention will process Prefill and Decode requests in the same batch to improve the utilization of the graphics processor.

[0043] At step 204, the scheduler of the computing device 100 determines the memory block size in units of a predetermined number of tokens within a first predetermined token capacity range, for use in performing pre-filling for pre-filling requests of the corresponding size based on the calculated memory block size.

[0044] Regarding the number of predefined tokens, it is associated with the alignment size of the memory blocks that are only accepted, as indicated in the attention operator declaration. In some embodiments, the size of the predefined token number is associated with the granularity of the tensor kernel used to perform inference computations for the AI ​​model. For example, N represents the number of predefined tokens. The attention operator declaration only accepts partial 128 alignment of chunks. N=128.

[0045] A method for performing pre-filling on pre-fill requests of a corresponding size based on a calculated memory block size includes, for example: determining whether the length of the pre-fill request is less than or equal to a first predetermined token capacity; directly pre-filling the pre-fill request in response to the length of the pre-fill request being less than or equal to the first predetermined token capacity; and calculating the memory block size based on the first predetermined token capacity and a predetermined number of tokens in response to the length of the pre-fill request being greater than the first predetermined token capacity, so as to perform pre-filling on pre-fill requests of a corresponding size based on the calculated memory block size; and suspending any remaining pre-fill requests exceeding the memory block size to await concatenation. The method 400 for performing pre-filling on pre-fill requests of a corresponding size based on a calculated memory block size will be discussed below in conjunction with... Figure 4 A detailed explanation will not be repeated here.

[0046] The following example illustrates the method for calculating memory block size using formula (1).

[0047] chunk_size = floor(P_max / N N (1)

[0048] In formula (1) above, chunk_size represents the memory block size. P_max represents the first pre-determined token capacity. N represents the number of pre-determined tokens. floor(P_max / N) represents the function that rounds down the result of the division between P_max and N. For example, P_max = 1500, N = 128. Then floor(1500 / 128) = floor(11.71875) = 11. At this time, chunk_size In some embodiments, P_max=1024, N=128, and chunk_size=1024.

[0049] It should be understood that by determining the memory block size in units of a predetermined number of tokens within a first predetermined token capacity range through the scheduler, and then performing pre-padding for pre-padding requests of corresponding sizes based on the calculated memory block size, the pre-padding data block sizes can be made uniform, which is beneficial for the parallel computing of the graphics processor. This reduces memory fragmentation, improves memory bandwidth utilization, and avoids computational waste caused by dynamic padding. Tests also show that at the end of this invention, the memory block padding ratio is 0, and the attention operator always follows the fast computation path (lieu, FastKernel) without backoff.

[0050] In step 206, each time the computing device 100 completes a pre-filling of the first predetermined token capacity range, it triggers a decoding phase, and the decoding request of the triggered decoding phase occupies a slot of the second predetermined token capacity.

[0051] For example, after each prefill request for a corresponding size based on the calculated memory block size is completed (e.g., after the Prefill chunk is executed), a decoding request is immediately initiated to fill the second predetermined token capacity. It should be understood that each time a prefill within the range of P_max is completed (i.e., when the token capacity indicated by P_max is full), a decoding phase is triggered, occupying a slot in D_max. This forms a "prefill → decode" cycle, thus making latency modelable.

[0052] The following combination Figure 3 A method for inference scheduling for an artificial intelligence model according to some embodiments of the present invention is described. Cache 301 includes a first cache 302 and a second cache 303, the first cache 302 and the second cache 303 being, for example, buffer regions physically isolated from each other.

[0053] Specifically, the first pre-determined token capacity is associated with, for example, a first cache. The second pre-determined token capacity is associated with a second cache. Figure 3 As shown in the example, the first cache 302 is used for the pre-filling stage (such as...). Figure 3 (As indicated by "P"). The first predetermined token capacity is, for example, 16, and the second cache 303 is used for the decoding stage (e.g., Figure 3 (As indicated by "D"), the second predetermined token capacity is, for example, 4. It should be understood that the above-described first and second predetermined token capacities are merely examples. The pre-filling and decoding phases are each allocated corresponding cache resources without competing for them. In some embodiments, for a case of 4 sentences, each with 16 tokens, the cache allocation for each round is as follows.

[0054] For example, in the first round of scheduling, cache allocation is as shown in 310, where the first cache 302 is allocated to 16 tokens P1-1 to P1-16 for prefill request 1. More specifically, even if the second cache 303 is not occupied, it is not allocated to the remaining prefill requests. Furthermore, based on the determination that a first predetermined token capacity of 16 prefilling has been completed, a decoding phase is triggered. In the second round of scheduling, cache allocation is as shown in 320, where the first cache 302 is allocated to 16 tokens P2-1 to P2-16 for prefill request 2. Simultaneously, decoding phase 1 generates one token, and a slot 321 in the second cache is allocated to the first token D1-1 generated by decoding phase 1. The remaining slot 322 remains unallocated. Furthermore, based on the determination that another first predetermined token capacity of 16 prefilling has been completed, another decoding phase is triggered. Similarly, in the third round of scheduling, the cache allocation is as shown in 330. The first cache 302 is allocated to the 16 tokens P3-1 to P3-16 of the prefill request 3. At the same time, decoding phase 1 continues to generate 1 token, and decoding phase 2 generates 1 token. Then, in the second cache, slot 331 is allocated to the second token D1-2 generated by decoding phase 1, slot 332 is allocated to the first token D2-1 generated by decoding phase 2, and the remaining slot 333 remains unallocated. In addition, based on the determination that another prefill of the first predetermined token capacity 16 is completed, another decoding phase is triggered. In the fourth round of scheduling, the cache allocation is as shown in 340. The first cache 302 is allocated to the 16 tokens P4-1 to P4-16 of prefill request 4. At the same time, decoding stage 1 continues to generate 1 token, decoding stage 2 continues to generate 1 token, and decoding stage 3 generates 1 token. Then, in the second cache, slot 341 is allocated to the third token D1-3 generated by decoding stage 1, slot 342 is allocated to the second token D2-2 generated by decoding stage 2, slot 343 is allocated to the first token D3-1 generated by decoding stage 3, and the remaining slot 344 remains unallocated. At this time, prefill requests 1-4 have all been processed. In the fifth round of scheduling, the cache allocation is as shown in 350. Since all prefill requests have been processed, the first cache 302 is unallocated. At the same time, decoding stage 4 generates 1 token. Then, in the second cache, slot 351 is allocated to the first token D4-1 generated by decoding stage 4, and the remaining slots in the second cache remain in the state of the previous round of scheduling. Therefore, since each stage of the pre-filling and decoding phases has its own corresponding cache resources, the pre-filling tokens and decoding tokens will not generate many memory block splits in their respective cache allocations, thus avoiding inference latency.

[0055] In the above scheme, by determining a first predetermined token capacity (associated with a first cache) for the pre-filling stage and a second predetermined token capacity (associated with a second cache) for the decoding stage, this invention avoids resource contention between the pre-filling and decoding stages through physical isolation. Therefore, the allocation of pre-filling tokens and decoding tokens in their respective caches does not result in excessive memory block splitting, thus avoiding performance impact caused by excessive memory block splitting. Furthermore, within the first predetermined token capacity range, the scheduler determines the memory block size in units of predetermined tokens, for pre-filling requests of corresponding sizes based on the calculated memory block size. Each time pre-filling within the first predetermined token capacity range is completed, a decoding stage is triggered, and the decoding requests in the triggered decoding stage occupy slots in the second predetermined token capacity. Because this invention uses memory blocks aligned to predetermined token numbers, it avoids dynamic filling, allowing the attention operator to always run on the optimal path, thus optimizing the computational efficiency of the graphics processor and reducing memory waste and inference latency. Therefore, it avoids performance impact caused by excessive memory block splitting and prevents inference latency.

[0056] In addition, the memory layout of this invention is fixed, eliminating the need for dynamic caching and simplifying the fusion of kernel functions.

[0057] The following will combine Figure 3 and Figure 4 This describes an embodiment of the present invention, a method 400 for pre-filling memory blocks based on a calculated memory block size. It should be understood that method 400 can, for example, be implemented in... Figure 1 The described computing device 100 performs the operation. Method 400 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0058] At step 402, the computing device 100 determines whether the length of the pre-fill request is less than or equal to the first predetermined token capacity.

[0059] For example, the length of the prefill phase request is request.prefill_len.

[0060] At step 404, in response to the pre-fill request being less than or equal to the first predetermined token capacity, pre-filling is performed directly for the pre-fill request.

[0061] For example, if the length of the prefill phase request (request.prefill_len) is less than or equal to the first pre-defined token capacity (P_max), as indicated by the expression "request.prefill_len ≤ P_max", prefilling is performed directly on the prefill phase request. That is, the entire prefill request is processed at once.

[0062] At step 406, in response to the pre-fill request being longer than the first predetermined token capacity, the memory block size is calculated based on the first predetermined token capacity and the predetermined number of tokens, so that pre-filling is performed for the corresponding size of the pre-fill request based on the calculated memory block size.

[0063] If the length of the prefill phase request is greater than or equal to the first pre-defined token capacity, as indicated by the expression “request.prefill_len>P_max”, the memory block size is calculated based on the first pre-defined token capacity and the number of pre-defined tokens, such that the calculated memory block size is equal to an integer multiple of 128, not exceeding the maximum of P_max, so that prefilling is performed for prefill requests of the corresponding size based on the calculated memory block size.

[0064] At step 408, any remaining pre-fill requests exceeding the memory block size are suspended pending splicing.

[0065] In some embodiments, remaining pre-fill requests exceeding the memory block size are suspended. For example, if a remaining pre-fill request is less than 128, it is not immediately pre-filled but suspended. This suspension could occur, for example, waiting to be concatenated with the remainder of other pre-fill requests or waiting for the next scheduling opportunity. It should be understood that determining the memory block size in units of a predetermined number of tokens for pre-fill requests ensures that each block fed to the graphics processor for computation is aligned to a predetermined number of tokens (e.g., 128), thus maximizing the utilization of the tensor kernel and avoiding computational resource waste caused by dynamic padding.

[0066] Test results show that the throughput of this invention is about 15% higher than that of typical inference models, and the performance curve is stable and predictable.

[0067] In the above scheme, the present invention can maximize the utilization of tensor kernels, avoid the waste of computing resources caused by dynamic filling, and significantly improve inference efficiency.

[0068] The various processes and procedures described above, such as methods 200 and 400, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 200 and 400 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 200 and 400 described above can be performed.

[0069] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0070] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0073] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0074] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. A method for inference scheduling in an artificial intelligence model, characterized in that, The method includes: A first predetermined token capacity for the pre-filling phase and a second predetermined token capacity for the decoding phase are determined, wherein the first predetermined token capacity is associated with a first cache and the second predetermined token capacity is associated with a second cache; The scheduler determines the memory block size within a first predetermined token capacity, in units of a predetermined number of tokens, for use in performing pre-filling for pre-filling requests of the corresponding size based on the calculated memory block size; and Each time the first predetermined token capacity range is pre-filled, a decoding phase is triggered, and the decoding request of the triggered decoding phase occupies a slot of the second predetermined token capacity.

2. The method according to claim 1, characterized in that, Pre-filling for pre-fill requests of the corresponding size, based on the calculated memory block size, includes: Remaining pre-pad requests that exceed the memory block size are suspended pending splicing.

3. The method according to claim 1, characterized in that, The first cache and the second cache are buffer areas that are physically isolated from each other.

4. The method according to claim 1, characterized in that, Within the first predetermined token capacity, the scheduler determines the memory block size in units of the predetermined token quantity, including: The memory block size is calculated based on the first predetermined token capacity and the number of predetermined tokens, such that the calculated memory block size does not exceed the first predetermined token capacity and is a maximum integer multiple of the number of predetermined tokens.

5. The method according to claim 1, characterized in that, The size of the predetermined number of tokens is related to the granularity of the tensor kernel used to perform inference computations for the artificial intelligence model.

6. The method according to claim 1, characterized in that, Pre-filling for pre-fill requests of the corresponding size, based on the calculated memory block size, includes: Determine whether the length of the pre-filled request is less than or equal to the first pre-defined token capacity; In response to a pre-fill request whose length is less than or equal to the first predetermined token capacity, pre-fill the request directly; and In response to a pre-fill request length exceeding a first predetermined token capacity, a memory block size is calculated based on the first predetermined token capacity and the predetermined number of tokens, so that pre-filling is performed for pre-fill requests of the corresponding size based on the calculated memory block size; and Remaining pre-fill requests that exceed the memory block size are suspended pending splicing.

7. The method according to claim 6, characterized in that, Suspend any remaining pre-padding requests that exceed the memory block size, pending splicing, including: For any remaining pre-filled requests that have been suspended, concatenate them with the next pre-filled request.

8. The method according to claim 1, characterized in that, Also includes: Triggering a decoding phase includes: In response to the determination that the slot for the second predetermined token capacity is full and the previous decoding phase has not yet ended, a wait is initiated until the previous decoding phase ends before the current decoding phase is triggered.

9. A computing device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-8.

11. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-8.

Images