Artificial intelligence AI model reasoning method and device, equipment, medium and product
By dynamically scheduling decoding tasks to fill task nodes in the AI cluster, the problem of uneven resource utilization during the inference process of artificial intelligence models is solved, thereby improving resource utilization and inference efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, artificial intelligence models cannot fully utilize resources during the inference process due to uneven resource utilization in the filling and decoding stages, and it is difficult to achieve dynamic load balancing.
In the AI cluster, after the first node completes the fill task, it is dynamically determined whether there is a decoding node with idle resources. If not, the first node executes the decoding task. If so, the fill task result is transmitted to the second node for execution, or the decoding task is migrated to a node with idle resources.
It improves resource utilization and efficiency in the reasoning process of artificial intelligence models, and achieves more efficient task processing.
Smart Images
Figure CN121766423A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this application primarily relate to the field of artificial intelligence technology. More specifically, the embodiments of this application relate to artificial intelligence (AI) model inference methods, apparatuses, devices, storage media, and products. Background Technology
[0002] With the advancement of computer technology, artificial intelligence models, especially neural network models, have developed rapidly. AI models are now being increasingly applied across various fields to process complex information. For example, in the field of artificial intelligence technology, AI models are gradually becoming the primary means of information processing.
[0003] With the development of artificial intelligence model technology, the amount of data that needs to be processed is increasing, and the types of data are becoming more complex, resulting in increasingly massive computational resources required for current AI models during inference. For example, inference in large language models based on transformers consists of a prefill phase and a decoding phase. Related technologies typically deploy the prefill and decoding phases separately. However, because prefilling and decoding have significantly different execution characteristics—for example, prefilling is computationally intensive, while decoding is memory-intensive—resources in the AI cluster cannot be fully utilized. Summary of the Invention
[0004] The embodiments of this application provide a scheme for AI model inference.
[0005] According to a first aspect of this application, an artificial intelligence (AI) model inference method is provided. The method includes determining whether a second node in the AI cluster, used for executing a decoding task of the AI model, has idle resources after a first node in the AI cluster completes a first prefill task for a first inference request. The method further includes that if the second node does not have idle resources, the first node executes the first decoding task of the first inference request based on the execution result of the first prefill task. The method also includes that if the second node has idle resources, the execution result of the first prefill task is transmitted to the second node, and the second node executes the first decoding task based on the execution result of the first prefill task.
[0006] In this way, during the inference process of the artificial intelligence model, when the node used to perform the decoding task has no idle resources, the decoding task is scheduled to be executed by the node that performs the filling task, thereby improving the resource utilization rate during the inference process of the artificial intelligence model and improving the inference efficiency of the artificial intelligence model at the same time.
[0007] In some embodiments, transmitting the execution result of the first fill task to the second node includes: retrieving the execution result stored in the cache of the first node, the execution result including key-value pairs for the first fill task; and transmitting the execution result from the first node to the second node. This method enables rapid retrieval of the execution result of the fill task and quick provision to the second node, improving the efficiency of task transmission.
[0008] In some embodiments, the method further includes: determining whether a second node has idle resources during the execution of a first decoding task on the first node; and migrating the first decoding task to the second node when the second node has idle resources. In this way, decoding tasks can be dynamically migrated to the second node based on the resource utilization status of the second node.
[0009] In some embodiments, migrating the first decoding task to the second node includes: determining a storage location in the first node for storing intermediate results of the first decoding task; and sending the storage location to the second node so that the second node can retrieve the intermediate results based on the storage location. This method enables the second node to accurately retrieve the intermediate results from the first node, improving data retrieval efficiency.
[0010] In some embodiments, the second node is one of a plurality of nodes used to perform the decoding task of the AI model, and the method further includes: migrating the decoding task in the fourth node of the plurality of nodes to the third node for execution when the third node of the plurality of nodes has idle resources. This approach enables dynamic task transfer between the plurality of nodes used to perform the decoding task, improving node resource utilization.
[0011] In some embodiments, the AI cluster includes multiple sub-clusters, and the first node and the second node are included in the first sub-cluster. This approach allows tasks to be completed through multiple sub-clusters, improving task processing efficiency.
[0012] According to a second aspect of this application, an artificial intelligence (AI) model inference apparatus is provided. The apparatus includes: an idle resource determination unit configured to determine whether a second node in the AI cluster, used for executing a decoding task of the AI model, has idle resources after a first node in the AI cluster has completed a first prefill task for a first inference request; a first task execution unit configured to, when the second node has no idle resources, have the first node execute the first decoding task of the first inference request based on the execution result of the first prefill task; and a second task execution unit configured to, when the second node has idle resources, transmit the execution result of the first prefill task to the second node, and have the second node execute the first decoding task based on the execution result of the first prefill task.
[0013] In some embodiments, the second task execution unit includes: an execution result acquisition unit configured to acquire the execution result stored in the cache of the first node, the execution result including key-value pairs for the first filling task; and an execution node transmission unit configured to transmit the execution result from the first node to the second node.
[0014] In some embodiments, the apparatus further includes: an idle resource determination unit configured to determine whether a second node has idle resources during the execution of a first decoding task by a first node; and a task migration unit configured to migrate the first decoding task to the second node when the second node has idle resources.
[0015] In some embodiments, the task migration unit includes: a storage location determination unit configured to determine a storage location in a first node for storing intermediate results of a first decoding task; and a storage location sending unit configured to send the storage location to a second node so that the second node obtains the intermediate results based on the storage location.
[0016] In some embodiments, the second node is one of a plurality of nodes for performing decoding tasks of an AI model, and the apparatus further includes a decoding task migration unit configured to migrate the decoding task in the fourth node of the plurality of nodes to the third node for execution when the third node of the plurality of nodes has idle resources.
[0017] In some embodiments, the AI cluster includes multiple sub-clusters, and the first node and the second node are included in the first sub-cluster among the multiple sub-clusters.
[0018] According to a third aspect of this application, an electronic device is also provided, comprising: at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, causing the device to perform the method according to the first aspect of this application.
[0019] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method described according to a first aspect of this application.
[0020] According to a fifth aspect of this application, a computer program product is also provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method described according to a first aspect of this application.
[0021] Understandably, the apparatus of the second aspect, the electronic device of the third aspect, the computer storage medium of the fourth aspect, or the computer program product of the fifth aspect provided above are used to perform the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, and fifth aspects. Furthermore, the beneficial effects achieved by the second, third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description
[0022] The above and other features, advantages and aspects of the embodiments of this application will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description.
[0023] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0024] Figure 1 A schematic diagram illustrates an example environment in which several embodiments of this application can be implemented;
[0025] Figure 2 A schematic diagram of an example architecture for an inference cluster according to some embodiments of this application is shown;
[0026] Figure 3 A schematic flowchart of a method for processing a task according to some embodiments of this application is shown;
[0027] Figure 4 A schematic diagram of another example architecture of an inference cluster according to some embodiments of this application is shown;
[0028] Figure 5 A schematic diagram illustrating an example of processing a reasoning request according to some embodiments of this application is shown;
[0029] Figure 6 A schematic diagram of a migration node for inference requests is shown according to some embodiments of this application;
[0030] Figure 7 A schematic diagram illustrating an example of an allocation processing request according to some embodiments of this application is shown;
[0031] Figure 8 Block diagrams of apparatuses according to some embodiments of this application are shown; and
[0032] Figure 9 A block diagram of a computing device capable of implementing several embodiments of the present application is shown. Detailed Implementation
[0033] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.
[0034] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0035] As mentioned above, inference tasks for artificial intelligence models are generally performed by separating the filling and decoding stages. Therefore, when using AI models for inference, inference requests are typically assigned to a group of nodes in the inference cluster for computation, parsing the requests, and then sending the parsed data to the decoding nodes for decoding computation. However, in this traditional approach, the resources of the filling and decoding nodes are prone to idleness, resulting in low resource utilization.
[0036] To address the traditional approach of separating fill and decode nodes, a further improvement is proposed. This improvement involves setting up a hybrid node pool to handle sudden task bursts. However, this approach also suffers from an overabundance of fill or decode node resources, leading to low resource utilization. Furthermore, when there is an imbalance in decode node execution across the inference cluster, this traditional approach struggles to achieve dynamic load balancing.
[0037] To address at least some of the aforementioned problems and other potential issues, in the embodiments of this application, after the computing device completes the first prefill task of the first inference request on the first node in the AI cluster, it determines whether the second node in the AI cluster, which is used to execute the decoding task of the AI model, has idle resources. If the second node does not have idle resources, the first node can execute the first decoding task of the first inference request based on the execution result of the first prefill task. If the second node has idle resources, the execution result of the first prefill task can be transmitted to the second node, and the second node can execute the first decoding task based on the execution result of the first prefill task. In this way, during the AI model inference process, when the node used to execute the decoding task does not have idle resources, the decoding task is scheduled to be executed on the node executing the prefill task, thereby improving the resource utilization during the AI model inference process and simultaneously improving the inference efficiency of the AI model.
[0038] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this application can be implemented is shown. For example... Figure 1 As shown, the example environment 100 includes an artificial intelligence cluster 102, which may include multiple nodes. Additionally, the multiple nodes are divided into multiple sub-clusters, each of which can be used to execute AI model inference tasks. Each sub-cluster has a control device for controlling the scheduling of tasks across different nodes. In some embodiments, the nodes in the artificial intelligence cluster 102 may be dedicated computing devices that can be used to perform inference of artificial intelligence models, such as neural network processing unit (NPU) devices or graphics processing unit (GPU) devices. In some embodiments, the nodes in the artificial intelligence cluster 102 include, but are not limited to, personal computers, servers, minicomputers, mainframes, and distributed computing environments including any of the above systems or devices.
[0039] like Figure 1 As shown, the nodes in AI cluster 102 are nodes used to process AI model inference tasks, such as inference tasks for large language models. The nodes in AI cluster 102 are divided into fill nodes, which can be used to process fill and decode tasks in AI model inference, and decode nodes, which are used to process decode tasks in AI model inference. For example, Figure 1The first node 104 shown is a fill node, which can be used to handle fill and decoding tasks in the inference request of the artificial intelligence model; the second node 108 is a decoding node, which is used to handle decoding tasks in the inference request of the artificial intelligence model. In some embodiments, the first node 104 and the second node 108 are located in the same sub-cluster, and the execution of the inference tasks is scheduled by the coordinator of the same sub-cluster.
[0040] The first node 104 is a filler node configured to handle filler tasks corresponding to inference requests from the AI model. After it completes its filler task, the subsequent decoding tasks for that inference request can typically be scheduled by the control device within the AI cluster to a node dedicated to handling decoding tasks, such as the second node 108.
[0041] like Figure 1 As shown, after receiving the first inference request 106 for the artificial intelligence model, the AI cluster 102 can assign the corresponding first filling task to the first node 104 for execution. After the first filling task 110 is completed, if the second node 108 has resources available to execute the first decoding task 112 corresponding to the first filling task 110, it can be assigned to the second node 108 for execution. If the second node 108 does not have resources available to process the decoding task, the first decoding task 112 can continue to be processed in the first node 104.
[0042] In some embodiments, if the first decoding task 112 has already been executed in the first node 104, during this period, the control device of the artificial intelligence cluster 102 can monitor whether there are idle resources in the second node. If there are idle resources, the first decoding task 112 can be migrated to the second node 108.
[0043] In some embodiments, if the second node has no more resources available for processing the decoding task, and other decoding tasks processed in the second node 108 have been completed, the second node 108 will then have resources available for executing the decoding task. At this time, the second node 108 can send a notification to the control device of the AI cluster 102 to indicate that the second node 108 has resources available for executing the decoding task. The control device of the AI cluster 102 can then determine whether a decoding task is being executed in the first node 104. If a decoding task is being processed in the first node 104, such as the first decoding task 112, the decoding task can be migrated to the second node 108 so that the second node 108 can continue processing the decoding task. When migrating the decoding task, in addition to migrating the decoding task to the second node, the intermediate data calculated by the decoding task in the second node also needs to be transmitted to the second node. At this time, the first node 104 can send the storage location of the intermediate data, such as the corresponding data storage location in KVcache, to the second node 108. Then, after the second node 108 starts executing the decoding task, it pulls intermediate data from the data storage location of the first node 104 to continue the decoding task.
[0044] After the decoding task in the first node 104 is migrated to the second node 108, the first node can be used to continue processing the subsequent padding task or other decoding tasks.
[0045] exist Figure 1 The AI cluster 102, including the first node 104 and the second node 108, is merely an example and not a specific limitation of this disclosure. The AI cluster can include any suitable number of nodes. In this AI cluster 102, in addition to dynamically adjusting decoding tasks between fill nodes and decoder nodes, decoding tasks can also be dynamically adjusted among decoder nodes, and the allocation ratio between fill nodes and decoder nodes can be adjusted to improve task processing efficiency. These adjustments will be described in detail below.
[0046] In this way, during the inference process of the artificial intelligence model, when the node used to perform the decoding task has no idle resources, the decoding task is scheduled to be executed by the node that performs the filling task, thereby improving the resource utilization rate during the inference process of the artificial intelligence model and improving the inference efficiency of the artificial intelligence model at the same time.
[0047] The above combination Figure 1 A schematic diagram illustrating an example environment 100 in which embodiments of this application can be implemented is described. The following is in conjunction with... Figure 2 This application describes a schematic diagram of the configuration of an inference cluster 202 according to some embodiments thereof. Figure 1An example of an artificial intelligence cluster 102. As shown in Example 200, the inference cluster 202 for processing inference requests from artificial intelligence models includes inference service group 204 and inference service group 206.
[0048] In some embodiments, the inference service group 204 includes a coordinator 208, a plurality of fill nodes (P nodes), and a plurality of decode nodes (D nodes), with the coordinator serving as the control device for the inference service group. The coordinator 208 is mainly used to receive status parameters of the plurality of P nodes, including P nodes 210, and the plurality of D nodes, including D nodes 212, and to allocate inference requests or adjust the real-time load of each node according to the different status parameters of the nodes.
[0049] In some embodiments, the number of inference requests that P nodes and D nodes can process within a predetermined time is the same, for example, both can process 100 inference requests per second. In some embodiments, the number of inference requests that P nodes and D nodes can process within a predetermined time is different, for example, P nodes can process 100 inference requests per second, while D nodes can process 50 inference requests per second.
[0050] In some embodiments, the inference task being processed in the P node can be sent to the D node via the coordinator, so that the D node can specialize in handling the inference task, enabling the P node to process the fill task in subsequent received inference requests more efficiently.
[0051] In some embodiments, the number of P nodes and D nodes in inference service groups 204 and 206 is the same, for example, both contain 10 P nodes and 20 D nodes. In some embodiments, the number of P nodes and D nodes in inference service groups 204 and 206 is different; for example, inference service group 204 contains 10 P nodes and 20 D nodes, while inference service group 206 contains 5 P nodes and 10 D nodes. It is understood that the above description is for illustrative purposes only and is not intended to limit the scope of this disclosure. In this way, the coordinator can dynamically allocate fill tasks and inference tasks in inference requests within its own inference service group, thereby improving the inference efficiency of the artificial intelligence model.
[0052] The above combination Figure 2 A schematic diagram illustrating the configuration of an inference cluster according to some embodiments of this application is described below. Figure 3 This is a schematic flowchart illustrating Example 300 of a task processing method according to some embodiments of this application. Example 300 can be derived from... Figure 1 It executes on the control devices and any suitable computing devices in the artificial intelligence cluster.
[0053] At box 302, after the first node 104 in the AI cluster 102 completes the first prefill task 110 of the first inference request 106, it is determined whether the second node 108 in the AI cluster, which is used to perform the decoding task of the AI model, has idle resources. Since the second node 108 is the node used to process the decoding task for inference of the artificial intelligence model, it can be determined by the controller after the first node 104 completes the prefill task for the inference request.
[0054] The controller in the AI cluster 102 can receive a first inference request 106 for the AI model, which includes data that needs to be processed by the AI model. Based on this request, a corresponding first filling task 110 and a first decoding task 112 can be determined. The control device can then determine the allocation of the first decoding task 112 based on the execution status of the first filling task 110 and the resources available to the second node 108. For example, the control device can determine whether the first filling task 110 for the first inference request 106 has been completed. If the first filling task 110 is not completed, processing continues on the first node 104. If the first filling task 110 is completed, the control device can further determine whether the second node 108 has resources available to process the first decoding task 112. For example, the second node 108 can report its available computing and / or storage resources to the control device, or the control device can collect the computing and / or storage resources of the second node 108 in real time.
[0055] At box 304, if the second node 108 has no free resources, the first node 104 executes the first decoding task 112 of the first inference request 106 based on the execution result of the first padding task 110. Since it is necessary to check whether the second node 108 has free resources to process the first decoding task 112 after the first padding task 110 is executed, if the second node 108 does not have resources available to process the first decoding task 112, then the first decoding task 112 is not assigned to the second node. Additionally, if other nodes have suitable resources, the first decoding task 112 can be assigned to other decoding nodes.
[0056] In some embodiments, during the execution of the first decoding task 112 on the first node 104, the controller of the AI cluster may further determine whether the second node 108 or other nodes used to process the decoding task have idle resources. If there are no idle resources, the first decoding task 112 continues to be executed on the first node 104. If idle resources are detected on the second node 108, the first decoding task 112 is then migrated to the second node.
[0057] In some embodiments, when migrating the first decoding task 112 from the first node 104 to the second node 108, the controller or the first node 104 can determine the storage location in the first node for storing the intermediate results of the first decoding task. This storage location can then be sent to the second node 108, so that the second node 108 can obtain the storage location for storing the intermediate results of the first decoding task 112, such as the storage location of key-value pairs in the cache space. The second node 108 can then retrieve the intermediate results for the first decoding task from this storage location in the first node 104.
[0058] Additionally, the first node can first send the storage location to the control device in the AI cluster, and then the control device can send the storage location to the second node 108. Subsequently, the second node 108 can use the storage location to obtain the intermediate results for the first decoding task 112.
[0059] At box 306, when the second node 112 has free resources, the execution result of the first filling task 110 is transmitted to the second node 108, and the second node 108 executes the first decoding task 112 according to the execution result of the first filling task 110.
[0060] In some embodiments, when transmitting the execution result of the first padding task 112 to the second node 108, in addition to transmitting the data to be processed by the first decoding task, it is also necessary to obtain the execution result of the first padding task 110 stored in the cache of the first node 104. This execution result includes key-value pairs for the first padding task, such as key-value pairs stored in a key-value cache (KV cache). The controller can then control the transmission of this execution result from the first node to the second node.
[0061] After the first decoding task 112 is completed, idle resources become available in the second node 108. To improve resource utilization, if there are still decoding tasks in the first node 104, the decoding tasks can be migrated to the second node 108.
[0062] In some embodiments, after the first decoding task 112 is executed by the second node 108 or after the first decoding task 112 is migrated from the first node 104 to the second node 108, since the first node 104 has idle resources, a new inference request is then provided to the first node 104 so that the first node 104 can handle the padding task for the subsequent inference request. For example, a second inference request is provided to the first node 104 so that the first node 104 can handle the second padding task for the second inference request.
[0063] In some embodiments, the AI cluster 102 has a large number of nodes, which can be divided into nodes for processing decoding tasks in inference requests from the AI model, and nodes for processing both filling and decoding tasks in inference requests from the AI model. Additionally, the nodes in the AI cluster can be divided into multiple distinct sub-clusters, each sub-cluster including a set of nodes for processing decoding tasks in inference requests from the AI model and nodes for processing both filling and decoding tasks in inference requests from the AI model. The first node 104 and the second node 102 reside within one sub-cluster.
[0064] In some embodiments, the second node 108 is one of a plurality of nodes used to perform the decoding task of the AI model. The tasks can also be evenly distributed among the plurality of nodes. For example, if the third node has idle resources, the decoding task in the fourth node can be migrated to the third node for execution.
[0065] In some embodiments, the controller of the AI cluster 102 determines the state parameters of each node. These state parameters include at least one of the following: the node's batch size, the node's block resource usage, the number of requests processed by the node, or the output length requested by the node. After determining the node's state parameters, these parameters can be further used to determine the node's load. The controller can then use these state parameters to identify nodes among the multiple nodes used to process decoding tasks that require adjustment of the decoding tasks, such as the third and fourth nodes mentioned above. The decoding task is then transferred between these identified nodes.
[0066] The controller can employ any suitable strategy to determine a fourth node suitable for transferring decoding tasks and a third node suitable for receiving decoding tasks from a plurality of nodes. In some embodiments, the computing device can use obtained state parameters to determine the load of each of the plurality of nodes. Then, a fourth node is determined from the plurality of nodes based on the load of each node, the fourth node having a load greater than a first threshold. The computing device can also determine a third node from the plurality of nodes based on state parameters, the third node having a load less than a second threshold, where the first threshold is greater than the second threshold. In one example, the load of a node can be determined by the ratio of the number of inference requests the node is currently processing to the maximum number of inference requests the node itself can handle. For example, if the fourth node is processing 80 inference requests, and the maximum number of inference requests the fourth node itself can handle is 100, the ratio is 0.8. If the highest threshold for the node's load is 0.7, then 0.8 is already greater than that threshold load, and the excess inference requests in the fourth node need to be transferred to other nodes. This highest threshold can be called the first threshold. When the load of the third node determined by the control device is 0.2, which is lower than the minimum threshold of 0.5 for the node load, the computing device can transfer excess inference requests from the fourth node to the third node for processing. This minimum threshold can be referred to as the second threshold. Additionally, if the load of the third node also exceeds the maximum threshold load after the transfer, the remaining inference requests are transferred to other nodes that have not reached the maximum threshold load while keeping the third node at the maximum threshold load state. The above is merely a description of the capabilities of this disclosure and is not intended to limit the disclosure. In another example, the node load can be determined by the number of inference requests processed by the node. The above examples are merely for describing this disclosure and are not intended to specifically limit the disclosure. Additionally, the number of inference requests corresponds to the number of decoding tasks or padding tasks.
[0067] In some embodiments, the third node and the fourth node can handle the same number of inference requests, for example, a maximum of 100 inference requests. In other embodiments, the third node and the fourth node can handle different numbers of inference requests; for example, the third node can handle a maximum of 100 inference requests, and the fourth node can handle a maximum of 10 inference requests. Furthermore, the maximum and minimum threshold loads for each node can be set as needed.
[0068] In some embodiments, the load of the nodes among multiple nodes can be sorted, with the node having the highest value designated as the fourth node and the node having the lowest value designated as the third node. Then, the task corresponding to the inference request in the fourth node is transferred to the third node.
[0069] In some embodiments, the computing device can determine the number of idle resource blocks in a node based on the acquired node status parameters. The number of idle resource blocks can also indicate the node's processing capacity. Therefore, the computing device can identify a fourth node with a number of idle resource blocks less than a third threshold and a third node with a number of idle resource blocks greater than the fourth threshold, where the third threshold is less than the fourth threshold. For example, if the fourth node has 100 resource blocks and 10 idle resource blocks, the third threshold indicating fewer idle resources is 15, and the fourth node has fewer idle resource blocks than the third threshold. If the third node has 50 resource blocks and 40 idle resource blocks, and the fourth threshold is 30, the third node has more idle resource blocks than the fourth threshold. In this case, some inference requests from the fourth node can be transferred to the third node. Additionally, after transferring inference task requests, if the third node reaches its load threshold, the remaining inference requests from the fourth node are transferred to other nodes below their load threshold.
[0070] In some embodiments, the computing device may use state parameters for multiple nodes used to process padding and decoding tasks to determine which nodes require adjustments to the padding and decoding tasks. Task transfer is then implemented between these determined nodes.
[0071] In some embodiments, if the nodes in a sub-cluster that handle filling and decoding tasks are designated as a first plurality of nodes, and the nodes that handle decoding tasks are designated as a second plurality of nodes, the control device can further adjust the first plurality of nodes and the second plurality of nodes. For example, the first plurality of nodes and the second plurality of nodes can be adjusted based on a first number of inference requests that nodes in the first plurality of nodes can handle within a predetermined time period and a second number of inference requests that nodes in the second plurality of nodes can handle within the predetermined time period. In one example, if one node in the second plurality of nodes can handle 100 decoding tasks for inference requests within a predetermined time, such as 1 second, while one node in the first plurality of nodes can handle 50 filling tasks for inference requests within 1 second, the ratio of the number of the second plurality of nodes to the number of the first plurality of nodes can be adjusted to 1:2, thereby coordinating task processing as a whole.
[0072] In some embodiments, the computing device also acquires the state parameters of each of the first plurality of nodes and the second plurality of nodes, and then uses these state parameters to further determine the number of inference requests that need to be allocated to the second plurality of nodes, i.e., the third number of decoding tasks allocated to the second plurality of nodes. For example, based on the number of requests being processed by each node and the processing capacity of the nodes, the number of decoding tasks that can be executed in the second plurality of nodes can be calculated. After determining the third number, when processing an inference request, a scheduling decision for that inference request is further determined based on the third number. For example, based on the third number of inference requests allocated to the second plurality of nodes and the number of inference requests already allocated to the second plurality of nodes, it can be determined whether the decoding task of the current inference request should continue to be executed in the first plurality of nodes or needs to be allocated to the second plurality of nodes for processing. For ease of description, the current inference request can also be referred to as the third inference request. In some embodiments, the computing device can send the third inference request to the target node in the second plurality of nodes to perform a fill task according to the scheduling decision. After the fill task is completed, if the scheduling decision instructs the third decoding task of the third inference request to continue to be executed in the target node in the first plurality of nodes, then the third decoding task continues to be executed in the target node. If the scheduling decision instructs the third decoding task to be executed on one of the second plurality of nodes, then after the third padding task for the third inference request has been processed in the target node, the third decoding task is transmitted to the second plurality of nodes for processing.
[0073] In this way, during the inference process of the artificial intelligence model, when the node used to perform the decoding task has no idle resources, the decoding task is scheduled to be executed by the node that performs the filling task, thereby improving the resource utilization rate during the inference process of the artificial intelligence model and improving the inference efficiency of the artificial intelligence model at the same time.
[0074] The above combination Figure 3 A schematic flowchart illustrating a task processing method according to some embodiments of this application is described below. Figure 4 Another schematic diagram illustrating the configuration of an inference cluster according to some embodiments of this application is shown.
[0075] As shown in Example 400, the inference cluster is allocated by the Application Programming Interface gate (APIG) 402, resulting in inference service group 1 404, inference service group 2 406, a first set 408 including a coordinator, P nodes and D nodes, and a second set 410 including a coordinator, P nodes and D nodes.
[0076] In some embodiments, the APIG may allocate one inference service group. In other embodiments, the APIG may allocate multiple inference service groups. Allocation can be based on the actual number of inference requests. In one example, when the number of inference requests is small and the required computing resources are low, only one inference service group may be allocated. In another example, when the number of inference requests is large and the required computing resources are significant, multiple inference service groups are allocated to handle the inference requests. Coordinators 408 and 410 can perform load balancing and dynamic allocation of inference requests for the P-nodes and D-nodes they manage. When processing inference requests, the coordinators also determine the status parameters of the multiple nodes they manage. These status parameters include at least one of the following: the node's batch size, the node's block resource usage, the number of requests processed by the node, or the output length requested by the node.
[0077] The above combination Figure 4 Another schematic diagram illustrating the configuration of an inference cluster according to some embodiments of this application is described below, in conjunction with... Figure 5 A schematic diagram illustrating one embodiment of processing a reasoning request according to some embodiments of this application.
[0078] In Example 500, when processing an inference request begins, both the first node 504 and the second node 502 are idle at stage R1 506. The computing device first processes the fill task of the received inference request at the second node 502. After the fill task is completed, the inference task in the inference request is sent to the first node 504 for processing. Then, while the first node 504 processes the received inference task at stage R2 508, the second node will continue to receive new inference requests and process the fill task in the new inference requests.
[0079] Then, after the second node finishes processing the population task for the new inference request, it will continue processing the decoding task in the new inference request until it receives an updated inference request. When there are idle resources in the first node, for example after phase 508 in R2, it will send the decoding task being processed in the second node to the first node so that the first node can execute the relevant inference task.
[0080] At this point, in RN phase 510, the node that originally handled the decoding task in the new inference request in the second node can be used to handle the padding task in the updated inference request.
[0081] This approach allows for greater utilization of the first and second nodes, enabling a more efficient allocation of padding and decoding tasks within inference requests and improving the overall efficiency of inference request processing.
[0082] The above combination Figure 5A schematic diagram of one embodiment of processing inference requests according to some embodiments of this application is described below, in conjunction with... Figure 6 A schematic diagram illustrating a migration node for processing inference requests according to some embodiments of this application.
[0083] In Example 600, the coordinator 602 collects status parameters of the decoder nodes 604, 606, and 608, which include at least one of the following: the node's batch size, the node's block resource usage, the number of requests processed by the node, or the length of the output requested by the node.
[0084] Then, the coordinator 602 can select a relatively idle node 604 or 608 based on the status parameters of each decoding node to migrate the inference task in decoding node 606 to decoding node 604 or 608. For example, it can select decoding node 608 whose load has not reached the threshold or decoding node 604 whose number of idle resource blocks is greater than the threshold to migrate the inference task to at least one of these two nodes. Finally, the idle decoding node 604 and idle decoding node 608 receive the migrated inference task.
[0085] This approach utilizes a coordinator to dynamically allocate inference tasks within inference requests, ensuring a relatively balanced load across nodes, preventing excessive unused nodes, improving resource utilization, and enhancing the efficiency of processing inference requests.
[0086] The above combination Figure 6 A schematic diagram of a migration node for processing inference requests according to some embodiments of this application is described below, in conjunction with... Figure 7 A schematic diagram illustrating the determination of the number of requests to be processed by a target node according to some embodiments of this application.
[0087] In Example 700, at 706, the coordinator 702 collects the status parameters of all fill nodes and multiple decode nodes, including resource usage and the number of inference requests being processed.
[0088] For example, after obtaining the number of inference requests currently being processed in the inference cluster and the status parameters of the decoding nodes, the coordinator 702 can determine the number of inference requests that can be executed on multiple decoding nodes, for example, based on the processing capacity of the decoding nodes. Then, based on the number of inference requests executed on multiple decoding nodes, a scheduling decision for the allocation of inference requests is generated. This scheduling decision is used to determine whether the decoding task is processed by the filling node or by the decoding node.
[0089] Then, at 704, the coordinator 702 sends the generated scheduling decision to the fill node 708 of the current inference cluster. After receiving the scheduling decision, if the scheduling decision indicates that the decoding task is executed at the corresponding fill node (i.e., at one of the nodes used for the first plurality of nodes), the fill node 708 executes the inference task directly at the fill node.
[0090] If the scheduling decision instructs the decoding task to be executed in a decoding node, then at 710, the decoding task is sent to an idle decoding node among the second or more decoding nodes 712 for processing. At this time, the storage address (e.g., the location in the KVCache) of the intermediate data to be filled in node 708 is also sent to that decoding node. Then, the decoding node can pull the data from the cache address of the intermediate data to be filled in node 708.
[0091] Additionally, when multiple nodes in an inference cluster exceed their threshold load, the scheduling decision instructs the inference task to be migrated to an idle node in another inference cluster for processing.
[0092] This approach utilizes a coordinator to dynamically allocate inference tasks within inference requests, ensuring a relatively balanced load across nodes, preventing excessive unused nodes, improving resource utilization, and enhancing the efficiency of processing inference requests.
[0093] Figure 8 A block diagram of a task processing apparatus 800 according to an embodiment of this application is further shown. The apparatus 800 is applied to a computing device and may include multiple modules for performing tasks such as... Figure 2 The corresponding steps in process 200 discussed herein. For example... Figure 8 As shown, the apparatus 800 includes: an idle resource determination unit 802, configured to determine whether a second node in the AI cluster, used for executing a decoding task of an AI model, has idle resources after the first node in the AI cluster has completed the first prefill task of the first inference request; a first task execution unit 804, configured to execute the first decoding task of the first inference request by the first node based on the execution result of the first prefill task when the second node has no idle resources; and a second task execution unit 806, configured to transmit the execution result of the first prefill task to the second node when the second node has idle resources, and the second node executes the first decoding task based on the execution result of the first prefill task.
[0094] In some embodiments, the second task execution unit 806 includes: an execution result acquisition unit configured to acquire the execution result stored in the cache of the first node, the execution result including key-value pairs for the first filling task; and an execution node transmission unit configured to transmit the execution result from the first node to the second node.
[0095] In some embodiments, the apparatus further includes: an idle resource determination unit configured to determine whether a second node has idle resources during the execution of a first decoding task by a first node; and a task migration unit configured to migrate the first decoding task to the second node when the second node has idle resources.
[0096] In some embodiments, the task migration unit includes: a storage location determination unit configured to determine a storage location in a first node for storing intermediate results of a first decoding task; and a storage location sending unit configured to send the storage location to a second node so that the second node obtains the intermediate results based on the storage location.
[0097] In some embodiments, the second node is one of a plurality of nodes for performing decoding tasks of an AI model, and the apparatus further includes a decoding task migration unit configured to migrate a decoding task in a fourth node of the plurality of nodes to the third node for execution when the third node of the plurality of nodes has idle resources.
[0098] In some embodiments, the AI cluster includes multiple sub-clusters, and the first node and the second node are included in the first sub-cluster among the multiple sub-clusters.
[0099] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present application is shown. For example, according to embodiments of the present application... Figure 1 The nodes and controllers, such as coordinators, in the artificial intelligence cluster 102 are implemented by example device 900. As shown, device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. Various programs and data required for the operation of device 900 can also be stored in RAM 903. CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.
[0100] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0101] The various processes and handling described above, such as method 300, can be executed by processing unit 901. For example, in some embodiments, method 300 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method 300 described above can be performed.
[0102] This application may be a method, apparatus, system, chip, and / or computer program product. A chip may include a processing unit and a communication interface, the processing unit being capable of processing program instructions received from the communication interface. A computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of this application are stored.
[0103] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0105] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.
[0106] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0110] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. An artificial intelligence (AI) model inference method, comprising: The method comprises: after a first node in an AI cluster completes a first prefill task of a first inference request, determining whether a second node in the AI cluster for performing a decode task of the AI model has idle resources; when the second node does not have idle resources, then performing, by the first node, a first decode task of the first inference request according to an execution result of the first prefill task; and when the second node has idle resources, then transmitting the execution result of the first prefill task to the second node, and performing, by the second node, the first decode task according to the execution result of the first prefill task.
2. The method of claim 1, wherein, Transmitting the execution result of the first prefill task to the second node comprises: obtaining the execution result stored in a cache of the first node, the execution result comprising key-value pairs for the first prefill task; and transmitting the execution result from the first node to the second node.
3. The method of claim 1, wherein, The method further comprises: during execution of the first decode task by the first node, determining whether the second node has idle resources; and when the second node has idle resources, migrating the first decode task to the second node.
4. The method of claim 3, wherein, Migrating the first decode task to the second node comprises: determining a storage location in the first node for storing an intermediate result of the first decode task; and sending the storage location to the second node to enable the second node to obtain the intermediate result based on the storage location.
5. The method of claim 1, wherein, The second node is one of a plurality of nodes for performing decode tasks of the AI model, and the method further comprises: when a third node of the plurality of nodes has idle resources, migrating a decode task in a fourth node of the plurality of nodes to the third node for execution.
6. The method of claim 1, wherein, The AI cluster comprises a plurality of sub-clusters, and the first node and the second node are included in a first sub-cluster of the plurality of sub-clusters.
7. An artificial intelligence (AI) model inference apparatus, comprising: The apparatus comprises: an idle resource determination unit configured to, after a first node in an AI cluster completes a first prefill task of a first inference request, determine whether a second node in the AI cluster for performing a decode task of the AI model has idle resources; a first task execution unit configured to, when the second node does not have idle resources, perform, by the first node, a first decode task of the first inference request according to an execution result of the first prefill task; and a second task execution unit configured to, when the second node has idle resources, transmit the execution result of the first prefill task to the second node, and perform, by the second node, the first decode task according to the execution result of the first prefill task.
8. The apparatus of claim 7, wherein, The second task execution unit comprises: an execution result obtaining unit configured to obtain the execution result stored in a cache of the first node, the execution result comprising key-value pairs for the first prefill task; and an execution result transmitting unit configured to transmit the execution result from the first node to the second node. An execution node transmission unit configured to transmit the execution result from the first node to the second node.
9. The apparatus of claim 7, wherein, The apparatus further includes: An idle resource determination unit configured to determine whether the second node has idle resources during execution of the first decoding task by the first node; and A task migration unit configured to migrate the first decoding task to the second node when the second node has idle resources.
10. The apparatus of claim 9, wherein, The task migration unit includes: A storage location determination unit configured to determine a storage location in the first node for storing an intermediate result of the first decoding task; and A storage location sending unit configured to send the storage location to the second node to enable the second node to obtain the intermediate result based on the storage location.
11. The apparatus of claim 7, wherein, The second node is one of a plurality of nodes for performing decoding tasks of the AI model, and the apparatus further includes: A decoding task migration unit configured to migrate a decoding task in a fourth node of the plurality of nodes to a third node of the plurality of nodes for execution when the third node has idle resources.
12. The apparatus of claim 7, wherein, The AI cluster includes a plurality of sub-clusters, and the first node and the second node are included in a first sub-cluster of the plurality of sub-clusters.
13. A computer readable storage medium having stored thereon a computer program, the program being executed by a processor to implement the method according to any one of claims 1-6.
14. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1-6.