Load balancing method, apparatus and electronic device

Abstract

The present disclosure relates to a load balancing method, device and electronic equipment, the method comprising: obtaining a target service request; obtaining state information of each GPU node in a first set of GPU nodes; the first set of GPU nodes comprising: GPU nodes currently available for processing the target service request; wherein the state information of any GPU node comprises: the longest duration of each inference task currently executed by the GPU node and the number of historical service requests in a session to which the target service request belongs; based on the state information of each GPU node in the first set of GPU nodes, at least one target GPU node is selected from the first set of GPU nodes; and the target large language model is used to process the target service request based on the at least one target GPU node. Through the present disclosure, the probability of hitting the large language model cache data is effectively improved, the computing power is saved, the efficiency of the GPU node is improved, and the quality of the large language model providing inference service is greatly improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a load balancing method, apparatus and electronic device. Background Technology

[0002] Large Language Models (LLMs) are predictive models in the field of artificial intelligence and natural language processing. They are capable of handling dialogue tasks and providing logical and comprehensive answers to human-posed questions. They can be applied to scenarios such as chatbots, document rewriting, and intelligent search. The prediction output process of a large language model can be called the inference process. Although large language models have good overall prediction performance, their large parameter scale leads to long inference times, and they require certain hardware resources, such as graphics processing units (GPUs), to provide inference services, thus placing high demands on hardware resources. Because scenarios such as chatbots, document rewriting, and intelligent search require low latency and high availability, large language models currently face certain challenges in practical applications. Currently, existing acceleration techniques for large language models, addressing their long inference times and limited practical application capabilities, can be divided into two categories: model computation-side optimization and I / O (input / output) optimization. Among them, computational optimization refers to reducing the waste of storage space during the computation of large language models through methods such as KV Cache (key-value pair cache), distributed inference, operator fusion and quantization compression; I / O optimization refers to reducing the speed bottleneck of storage access by utilizing the speed and storage capacity differences of multi-level storage such as cache, memory and video memory in GPU storage. The main technologies include Flash Attention and Countinous Batching.

[0003] In real-world application scenarios, for example, when handling high-volume concurrent service requests, large language model inference services need to receive dozens or even hundreds of service requests within seconds. With limited hardware resources (such as GPU resources), the aforementioned model acceleration methods cannot meet the low-latency requirements, which greatly affects the quality of inference services provided by large language models. Summary of the Invention

[0004] In view of this, this disclosure provides a load balancing method, apparatus, electronic device, storage medium, and computer program product.

[0005] According to one aspect of this disclosure, a load balancing method is provided, comprising:

[0006] Obtain the target service request;

[0007] Obtain the status information of each GPU node in the first GPU node set; the first GPU node set includes: GPU nodes currently available for processing the target service request; wherein, the status information of any GPU node includes: the longest duration of each inference task currently executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs;

[0008] Based on the status information of each GPU node in the first GPU node set, at least one target GPU node is selected from the first GPU node set;

[0009] The target service request is processed using at least one target GPU node based on the target large language model.

[0010] In one possible implementation, the step of selecting at least one target GPU node from the first GPU node set based on the state information of each GPU node in the first GPU node set includes:

[0011] Based on the status information of each GPU node in the first GPU node set, a priority score is calculated for each GPU node, and the priority score indicates the priority of processing the target service request;

[0012] According to the priority scores of each GPU node from high to low, at least one GPU node is selected from the first GPU node set as the at least one target GPU node.

[0013] In one possible implementation, the priority score of the first GPU node is positively correlated with the longest duration of each inference task currently executed by the first GPU node and with the number of historical service requests in the session to which the target service request belongs processed by the first GPU node; wherein, the first GPU node is any GPU node in the first GPU node set.

[0014] In one possible implementation, calculating the priority score of each GPU node based on the state information of each GPU node in the first GPU node set includes:

[0015] Calculate the absolute value of the difference between the longest duration of each inference task currently being executed by the first GPU node and the average duration of processing historical inference tasks; wherein, the average duration of processing historical inference tasks represents the average time from when the GPU node acquires a historical inference task to when it completes the historical inference task.

[0016] Calculate the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved for one historical service request, where the time saved for one historical service request represents the time saved by the GPU node in calculating the cached information by directly using the cached information corresponding to one historical service request during the execution of the inference task.

[0017] The priority score of the first GPU node is calculated based on the absolute value of the difference and the product; wherein the priority score of the first GPU node is positively correlated with the absolute value of the difference and positively correlated with the product.

[0018] In one possible implementation, calculating the priority score of the first GPU node based on the absolute value of the difference and the product includes:

[0019] The priority score of the first GPU node is obtained by weighted summation of the absolute value of the difference and the product.

[0020] In one possible implementation, the method further includes:

[0021] Determine the type of resource that can be used to process the target service request;

[0022] Obtain the second set of GPU nodes, which includes: GPU nodes currently capable of processing service requests;

[0023] In the second set of GPU nodes, a GPU node that matches the resource type is identified as the GPU node currently available to process the target service request.

[0024] In one possible implementation, the resource type includes: the type of GPU node and / or the type of large language model.

[0025] In one possible implementation, the method further includes:

[0026] Determine the minimum number of GPU nodes corresponding to the target service request;

[0027] The step of selecting at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set includes:

[0028] If the minimum number of GPU nodes does not exceed the number of GPU nodes in the first GPU node set, the GPU node with the minimum number of GPU nodes is selected from the first GPU node set as the at least one target GPU node based on the status information of each GPU node in the first GPU node set.

[0029] And / or,

[0030] If the minimum number of GPU nodes exceeds the number of GPU nodes in the first GPU node set, the first GPU node set is updated; and if the number of GPU nodes in the updated first GPU node set is not less than the minimum number of GPU nodes, the GPU nodes with the minimum number of GPU nodes are selected from the updated first GPU node set as the at least one target GPU node based on the status information of each GPU node in the updated first GPU node set.

[0031] In one possible implementation, when the longest duration of each inference task currently being executed by the first GPU node exceeds the average duration of processing historical inference tasks, the weight of the absolute value of the difference is positively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node.

[0032] And / or,

[0033] If the longest duration of each inference task currently being executed by the first GPU node does not exceed the average duration of processing historical inference tasks, the weight of the absolute value of the difference is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is positively correlated with the longest duration of each inference task currently being executed by the first GPU node.

[0034] In one possible implementation, obtaining the second set of GPU nodes includes:

[0035] Based on the distributed semaphore corresponding to each candidate GPU node in the candidate GPU node set, the GPU node currently capable of processing service requests is selected from the candidate GPU node set.

[0036] The method further includes:

[0037] Update the distributed semaphore corresponding to the at least one target GPU node.

[0038] In one possible implementation, the method further includes: using a key-value database as a message middleware to synchronize messages between the at least one target GPU node and the node that issued the target service request.

[0039] According to another aspect of this disclosure, a load balancing device is provided, the device comprising:

[0040] The acquisition module is used to acquire the target service request;

[0041] The acquisition module is further configured to acquire the status information of each GPU node in the first GPU node set; the first GPU node set includes: GPU nodes currently available for processing the target service request; wherein, the status information of any GPU node includes: the longest duration of each inference task currently executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs;

[0042] The filtering module is used to filter out at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set.

[0043] A processing module is used to process the target service request based on the target large language model using the at least one target GPU node.

[0044] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0045] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0046] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0047] Through the embodiments of this disclosure, based on the caching characteristics of the large language model during the service process, at least one target GPU node is selected from the first GPU node set according to the status information of each GPU node in the first GPU node set, namely the longest duration of each inference task currently being executed by the GPU node and the number of historical service requests in the session to which the target service request belongs. Since the number of historical service requests in the session to which the target service request belongs is relatively large, the probability of hitting cached data when processing the target service request is higher. At the same time, the longer the longest duration of each inference task currently being executed, the faster more computing resources can be released to execute other inference tasks. In this way, the selected target GPU node has a higher probability of hitting the cached data of the target GPU node when processing the target service request based on the target large language model, saving computing power and improving the efficiency of the GPU node. It reduces the computing power bottleneck under limited hardware resources. When processing high-volume concurrent service requests, hardware resources can be made more fully utilized, and the throughput of the large language model can be effectively utilized, thereby avoiding frequent queuing and waiting of user service requests, resulting in low service latency and greatly improving the quality of inference services provided by the large language model.

[0048] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0050] Figure 1 This diagram illustrates the structure of a large language model service system according to an embodiment of the present disclosure.

[0051] Figure 2 A flowchart of a load balancing method according to an embodiment of the present disclosure is shown.

[0052] Figure 3 A flowchart illustrating a target GPU node selection method according to an embodiment of the present disclosure is shown.

[0053] Figure 4 A flowchart of a load balancing method according to an embodiment of the present disclosure is shown.

[0054] Figure 5 A flowchart of a load balancing method according to an embodiment of the present disclosure is shown.

[0055] Figure 6 A structural diagram of a load balancing device according to an embodiment of the present disclosure is shown.

[0056] Figure 7A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0057] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0058] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "exemplary," "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0059] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0060] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0061] Figure 1 A schematic diagram of the structure of a large language model service system according to an embodiment of the present disclosure is shown. Figure 1 As shown, the large language model service system may include: a GPU node set 101 and a load balancing device 102.

[0062] The GPU node set 101 may include one or more GPU nodes (also known as service nodes or nodes). Figure 1 The GPU node set 101 shown includes N GPU nodes, namely GPU node 1, GPU node 2, ..., GPU node N, where N is a positive integer; these GPU nodes are used to perform inference tasks for large language models.

[0063] For example, a GPU node can be a physical entity device configured with GPUs. This physical entity can be a computer device, such as a terminal or server, or a part of a computer device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.; a single server can serve as a GPU node. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, etc.; a single terminal can serve as a GPU node. As an example, a computer device such as a terminal or server can be configured with one or more processors. When a computer device is configured with multiple processors, all or part of those processors can serve as a GPU node; for example, if a computer device is configured with multiple GPUs, each GPU can serve as a GPU node, or a number of GPUs can serve as a single GPU node.

[0064] The load balancing device 102 is used to receive service requests from users and, through a load balancing method, determine some or all of the GPU nodes in the GPU node set 101 as GPU nodes to process the service request. Then, it uses these GPU nodes to process the service request based on a large language model. For example, each of these GPU nodes performs a corresponding inference task to complete the processing of the service request.

[0065] For example, the load balancing device 102 can be software, hardware, or a combination of software and hardware, without limitation.

[0066] To improve the quality of inference services provided by large language models, related technologies, such as the vLLM framework, can configure the load balancer 102 with traditional load balancing methods. Upon receiving a user's service request, it can directly determine the GPU node to handle the request based on the traffic of each GPU node in the GPU node set 101. During the GPU node's processing of the service request, page attention is used to achieve discontinuous large language model cache data storage, which improves hardware resource utilization and reduces latency to some extent. However, the vLLM framework, using traditional load balancing methods, only distributes service requests based on GPU node traffic. With limited hardware resources, computing power becomes a bottleneck, and frequent queuing of user service requests still occurs when handling high-volume concurrent service requests. Furthermore, although large language model cache data is stored during GPU node processing to accelerate the process, the probability of this cached data being hit in subsequent processing is extremely low.

[0067] To address the aforementioned technical issues, this disclosure provides a dynamic load balancing method for large language model inference services (detailed description below). Considering the actual load capacity of GPU nodes and the caching characteristics of the large language model during service processing, the method schedules GPU nodes to handle service requests. This effectively increases the probability of cached data being hit, saves computing power, and improves the efficiency of GPU nodes. It also reduces the computing power bottleneck under limited hardware resources. When handling high-volume concurrent service requests, it can make fuller use of hardware resources, effectively maximizing the throughput of the large language model, thereby avoiding frequent queuing of user service requests, resulting in low service latency and significantly improving the quality of inference services provided by the large language model.

[0068] It should be noted that the large language model service system described in this disclosure is for the purpose of more clearly illustrating the technical solutions of this disclosure, and does not constitute a limitation on the application scenarios of the technical solutions provided in this disclosure. As those skilled in the art will know, the technical solutions provided in this disclosure are also applicable to similar technical problems in the face of other similar or new scenarios.

[0069] The load balancing method provided in the embodiments of this disclosure will be described in detail below.

[0070] Figure 2 A flowchart illustrating a load balancing method according to an embodiment of the present disclosure is shown. Exemplarily, the method can be derived from the above... Figure 1 The language model service system 10 or load balancer 102 at Zhongda University will execute, such as Figure 2 As shown, the method may include the following steps:

[0071] Step 201: Obtain the target service request.

[0072] Service requests can also be called inference requests, and for example, they can include requests for natural language processing tasks such as text generation, translation, and question answering.

[0073] For example, the large language model service system can provide multiple access interfaces and support multiple access protocols to receive different types of service requests. For instance, it can handle HTTP or gRPC service requests. As an example, users or applications can send service requests to the load balancer 102 through interactive interfaces or other interfaces. For instance, in a chatbot scenario, a user can input a question via voice or text through an interactive interface, triggering a service request to answer that question. This service request is the target service request. After receiving the service request triggered by the user, the load balancer 102 executes the following steps to call a suitable GPU node to process the service request. It can be understood that in a chatbot scenario, a user can ask multiple questions in a single conversation with the machine. Each question asked by the user triggers a service request; that is, in this scenario, the user can trigger multiple service requests, and the target service request is any one of these multiple service requests.

[0074] For example, after receiving a target service request, the load balancer 102 can parse and preprocess the target service request. For instance, it can perform data format conversion on the target service request. As an example, the load balancer 102 can use the Golang language and its ecosystem components to implement authentication and parsing of the target service request.

[0075] For example, after the load balancer 102 parses the target service request, it can check the relevant parameters of the target service request. For example, the relevant parameters may include keywords and phrases indicated by the target service request.

[0076] For example, a globally unique identifier can be configured for the target service request within the large language model service system to identify the target service request and distinguish it from other service requests.

[0077] Step 202: Obtain the status information of each GPU node in the first GPU node set.

[0078] The first set of GPU nodes includes GPU nodes currently available to process the target service request. As an example, when the large language model service system is configured with homogeneous GPU nodes, the GPU nodes currently with available load can be considered as those currently available to process the target service request. As another example, when the large language model service system is configured with heterogeneous GPU nodes, the GPU nodes currently available to process the target service request can be selected from those with available load; the specific selection method is described below.

[0079] In one possible implementation, a monitoring and reporting mechanism can be used to monitor the stability of the large language model service system's network links and related indicators (e.g., signal strength, bit error rate, latency, etc.), measure the overall pressure on the large language model service system, and configure new GPU nodes or delete existing GPU nodes in the large language model service system according to changes in overall pressure, thereby realizing the dynamic expansion or reduction of the large language model service system.

[0080] In the first set of GPU nodes, the status information of any GPU node includes: the longest duration of each inference task currently being executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs.

[0081] For example, in a clustered deployment of GPU nodes, a large language model service system can be configured with multiple model instances (also called service instances). Each model instance can accept a certain number of concurrent service requests. At any given time, multiple model instances may run simultaneously to handle high-volume concurrent service requests. Here, a model instance represents a concrete implementation of a large language model base, containing the model structure file and model weights. The model structure includes the hierarchical construction information of the model, and the model weights are the parameter information learned during the model pre-training process. Each model instance can be used to generate text, answer questions, or perform general natural language processing tasks. Each model instance is consistent, using the same large language model architecture and parameter set, thus ensuring consistent performance and behavior when processing the same task. Each model instance can be implemented by one or more GPU nodes, meaning one or more GPU nodes can handle a single service request. Meanwhile, each GPU node can implement one or more model instances. That is, for each GPU node, one or more model instances can be implemented at the same time, which means that multiple large language model inference tasks can be executed at the same time. It can be understood that the process of the GPU node executing each inference task needs to last for a period of time. The longest period of time is the longest duration of each inference task currently executed by the GPU node, which can also be called the longest inference task time of the GPU node. As an example, a GPU node can be configured to execute a maximum of four inference tasks simultaneously. For instance, the GPU node is currently executing inference task A and inference task B. Inference task A lasts for 5 seconds, meaning that as of the current moment, the GPU node has been executing inference task A for 5 seconds and is still executing it. Inference task B lasts for 8 seconds, meaning that as of the current moment, the GPU node has been executing inference task B for 8 seconds. Therefore, the longest duration of each inference task currently being executed by the GPU node is 8 seconds. As another example, the GPU node is currently executing inference task C, which lasts for 10 seconds, meaning that as of the current moment, the GPU node has been executing inference task C for 10 seconds and is still executing it. Therefore, the longest duration of each inference task currently being executed by the GPU node is 10 seconds.

[0082] In a chatbot scenario, users can engage in conversations with a large language model service system. During these conversations, users can ask and answer multiple questions. Each question represents a service request. Upon receiving a question, the large language model service system can generate a response through inference and present it to the user. Within the same conversation, there may be connections between the user's questions. For example, a user might ask questions about the same thing from different angles or in a progressive manner. Therefore, after each inference task, the GPU node caches the corresponding data (such as the inference result or intermediate data) so that when processing subsequent service requests within the same conversation, it can directly access the cached data, thereby improving its inference efficiency. For instance, after a conversation ends, the large language model service system can release the cached data from each GPU node. Understandably, for each service request, a suitable GPU node is selected from the first set of GPU nodes as the target GPU node to handle the current service request. Thus, different GPU nodes may be selected for different service requests in the same session; that is, the GPU nodes handling different service requests may be different or the same. For any given GPU node, the number of service requests processed by that GPU node within the session to which the target service request belongs is the number of historical service requests processed by that GPU node within the session to which the target service request belongs. If the GPU node has not processed any historical service request within the session to which the target service request belongs, the corresponding count is 0; if the GPU node has processed one historical service request within the session to which the target service request belongs, the corresponding count is 1, and so on.

[0083] Since the service request volume in the large language model service system is usually very large, each GPU node in the large language model service system will perform a large number of inference tasks, and the data cached by each GPU node is relatively complex, the state information of each GPU node in the large language model service system changes frequently. For example, the state information of each GPU node in the large language model service system can be dynamically updated, so that the latest state information of each GPU node in the large language model service system can be obtained at any time.

[0084] Step 203: Based on the status information of each GPU node in the first GPU node set, select at least one target GPU node from the first GPU node set.

[0085] In this step, the load balancer 102 can select at least one target GPU node from the first GPU node set based on the longest duration of each currently executed inference task of each GPU node in the first GPU node set and the number of times it has processed historical service requests in the session to which the target service request belongs. Considering that when processing service requests based on a large language model, there is usually a limit to the duration of each inference task executed by a GPU node, for any GPU node, the longer the longest duration of each currently executed inference task, the more likely the inference task corresponding to the longest duration is to be nearing completion, thus allowing the GPU node to release more computing resources to execute other inference tasks more quickly. At the same time, for a certain service request, if a GPU node processes more historical service requests in the session to which the service request belongs, the more relevant the data cached in the GPU node is to the data involved in processing the service request, the more times the GPU node processes historical service requests in the session to which the target service request belongs, the higher the efficiency of the GPU node in processing the target service request. Therefore, suitable target GPU nodes can be selected based on the longest duration of each inference task currently being executed by each GPU node in the first GPU node set and the number of historical service requests in the session to which the target service request belongs, which can effectively improve the efficiency of subsequent target GPU nodes in processing target service requests.

[0086] In one possible implementation, before performing step 203, the minimum number of GPU nodes corresponding to the target service request can be determined. Then, based on the relative size of the minimum number of GPU nodes corresponding to the target service request and the number of GPU nodes in the first GPU node set, different strategies can be flexibly executed to select at least one target GPU node from the first GPU node set. The minimum number of GPU nodes can be configured according to requirements; for example, it can be 1.

[0087] Scenario 1: When the minimum number of GPU nodes does not exceed the number of GPU nodes in the first GPU node set, based on the status information of each GPU node in the first GPU node set, the minimum number of GPU nodes is selected from the first GPU node set as the at least one target GPU node. In this scenario, the minimum number of GPU nodes corresponding to the target service request does not exceed the number of GPU nodes in the first GPU node set, that is, there are currently enough GPU node computing resources in the large language model service system to process the target service request. Therefore, the strategy of directly selecting target GPU nodes is executed.

[0088] Scenario 2: When the minimum number of GPU nodes exceeds the number of GPU nodes in the first GPU node set, the first GPU node set is updated. And when the number of GPU nodes in the updated first GPU node set is not less than the minimum number of GPU nodes, based on the status information of each GPU node in the updated first GPU node set, the GPU nodes with the minimum number of GPU nodes are selected from the updated first GPU node set as the at least one target GPU node. In this scenario, the minimum number of GPU nodes corresponding to the target service request exceeds the number of GPU nodes in the first GPU node set, meaning that the computing resources of all currently idle GPU nodes in the large language model service system are insufficient to process the target service request. Therefore, a queuing strategy is required. The target service request enters a queuing queue and waits, and the first GPU node set is dynamically updated. As it is continuously updated, the number of GPU nodes in the first GPU node set changes in real time. When the minimum number of GPU nodes corresponding to the target service request does not exceed the number of GPU nodes in the first GPU node set, the target GPU node is selected.

[0089] For example, the load status of each GPU node in the large language model service system can be updated in real time. Correspondingly, the number of GPU nodes currently available to process the target service request in the first GPU node set will also change dynamically. In this way, by dynamically updating the first GPU node set, when there are enough GPU nodes available to process the target service request in the first GPU node set, the target GPU node for processing the target service request can be selected in a timely manner.

[0090] Step 204: Use the at least one target GPU node to process the target service request based on the target large language model.

[0091] In this step, the target GPU nodes among the at least one target GPU nodes determined in step 203 are used to process the target service request, thereby scheduling the at least one target GPU node to execute the corresponding inference task of the target large language model. During the inference process, the target large language model can be used to provide language generation capabilities. Since each target GPU node is determined based on the longest duration of each inference task currently being executed by each GPU node in the first GPU node set and the number of historical service requests in the session to which the target service request belongs, the probability of the target GPU node hitting cached data when executing the inference task is increased, thereby saving the computing resources of the target GPU node and improving the efficiency of target service request processing.

[0092] For example, the target large language model can be pre-configured or determined based on the relevant parameters of the target service request. For instance, if the relevant parameters of the target service request include the type of the large language model, then the large language model of that type can be used as the target large language model.

[0093] In one possible implementation, the method further includes: using a key-value (KV) database as a message broker to synchronize messages between the at least one target GPU node and the node that issued the target service request. The node issuing the target service request is the load balancer 102; for example, the KV database can be Redis. Thus, the load balancer 102 can schedule target GPU nodes through the message broker, and the target GPU nodes can pull messages from the load balancer 102 through the message broker and execute corresponding inference tasks. While the target GPU node is executing the target service request, the large language model service system can subscribe to the target service request and continue to acquire new service requests. For each new service request, the system can perform the steps described above to determine the GPU node that will handle the new service request, thereby achieving asynchronous scheduling.

[0094] As an example, a target service request can be distributed to multiple target GPU nodes. These nodes can then execute the corresponding inference tasks in parallel and obtain the inference results. Finally, after each target GPU node completes its inference task, it reports its inference result. Once all target GPU nodes have completed their inference tasks, the reported results are aggregated to generate the processing result for the target service request. This processing result can then be fed back to the user through a pre-defined interface. This allows users to access the large language model system and obtain the processing results of their service requests using a pre-defined interface, offering convenience, simplicity, and greater versatility.

[0095] In this embodiment, based on the caching characteristics of the large language model during service, at least one target GPU node is selected from the first GPU node set according to the status information of each GPU node in the first GPU node set, namely the longest duration of each inference task currently executed by the GPU node and the number of historical service requests in the session to which the target service request belongs. Since the number of historical service requests in the session to which the target service request belongs is relatively large, the probability of hitting cached data when processing the target service request is higher. At the same time, the longer the longest duration of each inference task currently executed, the faster more computing resources can be released to execute other inference tasks. In this way, the selected target GPU node has a higher probability of hitting the cached data of the target GPU node when processing the target service request based on the target large language model, saving computing power and improving the efficiency of the GPU node. It reduces the computing power bottleneck under limited hardware resources. When processing high-volume concurrent service requests, it can make fuller use of hardware resources and effectively leverage the throughput of the large language model, thereby avoiding frequent queuing and waiting of user service requests, resulting in low service latency and greatly improving the quality of inference services provided by the large language model.

[0096] The following provides an exemplary description of possible implementations for selecting target GPU nodes in step 203 above.

[0097] Figure 3 A flowchart illustrating a target GPU node selection method according to an embodiment of the present disclosure is shown. Exemplarily, the method can be derived from the above... Figure 1 The language model service system 10 or load balancer 102 at Zhongda University will execute, such as Figure 3 As shown, the method may include the following steps:

[0098] Step 301: Based on the status information of each GPU node in the first GPU node set, calculate the priority score of each GPU node, whereby the priority score represents the priority of processing the target service request.

[0099] Based on the foregoing, for any given GPU node, the longer the longest duration of each currently executed inference task, the more likely the corresponding inference task is to near completion. This allows the GPU node to release computing resources more quickly, enabling it to better handle the target service request. Furthermore, for any given GPU node, the more times it processes historical service requests within the session to which the target service request belongs, the higher its efficiency in processing the target service request. Therefore, the priority score of each GPU node can be calculated based on the longest duration of each currently executed inference task and the number of times it processes historical service requests within the session to which the target service request belongs.

[0100] For example, the priority score of the first GPU node is positively correlated with the longest duration of each inference task currently being executed by the first GPU node, and positively correlated with the number of times the first GPU node processes historical service requests in the session to which the target service request belongs; wherein, the first GPU node is any GPU node in the first GPU node set. That is, the longer the longest duration of each inference task currently being executed by the GPU node and the more times the GPU node processes historical service requests in the session to which the target service request belongs, the higher the corresponding priority score, that is, the higher the priority of processing the target service request; conversely, the lower the corresponding priority score, that is, the lower the priority of processing the target service request.

[0101] In one possible implementation, step 301 may include: calculating the absolute value of the difference between the longest duration of each inference task currently executed by the first GPU node and the average duration of processing historical inference tasks; calculating the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved corresponding to one historical service request; and calculating the priority score of the first GPU node based on the absolute value of the difference and the product; wherein the priority score of the first GPU node is positively correlated with the absolute value of the difference and positively correlated with the product.

[0102] The average processing time for historical inference tasks refers to the average time from when the GPU node acquires the historical inference task to when it completes the historical inference task. This average processing time can be a quantitative statistical value. For example, information on multiple historical inference tasks can be collected. For any historical inference task, the collected information can include the time from when the GPU node acquires the historical inference task to when it completes the historical inference task. Then, the average of the corresponding times of multiple historical inference tasks is calculated as the average processing time for historical inference tasks. As an example, the average processing time for historical inference tasks by the GPU node can be 20 seconds.

[0103] The time saved for a single historical service request refers to the time saved by directly utilizing the cached information corresponding to a single historical service request during the execution of an inference task by the GPU node compared to the time saved by the GPU node in calculating the cached information. This time saving can be a quantitative statistical value. For example, processing information from multiple historical service requests can be collected. For each historical service request, the collected processing information can include the time saved by directly utilizing the cached information corresponding to that historical service request compared to the time saved by the GPU node in calculating the cached information.

[0104] For example, calculating the priority score of the first GPU node based on the absolute value of the difference and the product may include: taking the absolute value of the difference between the longest duration of each inference task currently executed by the first GPU node and the average duration of processing historical inference tasks, and the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved corresponding to one historical service request, and then performing a weighted summation to obtain the priority score of the first GPU node.

[0105] For example, the priority score of the first GPU node can be calculated using the following formula (1):

[0106]

[0107] Where T represents the time saved for a single historical service request, n represents the number of times the first GPU node processes historical service requests in the session to which the target service request belongs, then T×n represents the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved for a single historical service request, a is the weight of the product T×n, t represents the longest duration of each inference task currently being executed by the first GPU node, M represents the average time for processing historical inference tasks, |t M| represents the absolute value of the difference between the longest duration of each inference task currently being executed by the first GPU node and the average duration of processing historical inference tasks, where b is the absolute value of the difference |t. The weight of M |.

[0108] In one possible implementation, if the longest duration of each inference task currently being executed by the first GPU node exceeds the average duration of processing historical inference tasks, the weight of the absolute value of the difference is positively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node; and / or, if the longest duration of each inference task currently being executed by the first GPU node does not exceed the average duration of processing historical inference tasks, the weight of the absolute value of the difference is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is positively correlated with the longest duration of each inference task currently being executed by the first GPU node.

[0109] Because the average processing time of any GPU node for historical inference tasks in the large language model service system generally follows a normal distribution, there is a certain long-tail effect. For the first GPU node, if the longest duration of each inference task currently being executed by the first GPU node exceeds the average processing time of historical inference tasks, then the larger the value of the longest duration, the more likely the first GPU node is to be close to completing the corresponding inference task, and thus the more likely it is to quickly allocate the computing resources of the first GPU node to execute other inference tasks. In this case, relative to the product of the number of historical service requests in the session to which the first GPU node processes the target service request and the time saved for one historical service request, the absolute value of the difference between the longest duration of each inference task currently being executed by the first GPU node and the average processing time of historical inference tasks is more important than the product of the number of historical service requests processed by the first GPU node and the time saved for one historical service request. The priority score of the GPU node has a greater impact, so the absolute value of the difference has a larger weight, and correspondingly, the weight of the product is smaller. If the longest duration of each inference task currently being executed by the first GPU node does not exceed the average duration of processing historical inference tasks, then the smaller the value of the longest duration, the lower the probability that the first GPU node is close to finishing the corresponding inference task. In this case, relative to the absolute value of the difference between the longest duration of each inference task currently being executed by the first GPU node and the average duration of processing historical inference tasks, the product of the number of historical service requests processed by the first GPU node in the session to which the target service request belongs and the time saved corresponding to one historical service request has a greater impact on the priority score of the first GPU node, so the absolute value of the difference has a smaller weight, and correspondingly, the weight of the product is larger.

[0110] For example, taking the above formula (1) as an example, the values ​​of weights a and b can be adjusted according to the relative size of the longest duration t of each inference task currently executed by the first GPU node and the average duration M of processing historical inference tasks. When t exceeds M and the larger t is, a is smaller and b is larger. When t exceeds M and the smaller t is, a is larger and b is smaller.

[0111] Step 302: Select at least one GPU node from the first GPU node set as the at least one target GPU node according to the priority scores of each GPU node from high to low.

[0112] Thus, by executing steps 301-302, the priority scores of GPU nodes are quantitatively calculated. Based on these priority scores, suitable GPU nodes are selected as target GPU nodes. For example, in scenario one, if the first GPU node set has sufficient GPU computing resources to process the target service request, step 301 can be executed to calculate the priority score of each GPU node. Then, step 302 is executed to select target GPU nodes in descending order of their priority scores. In scenario two, since the first GPU node set does not have sufficient GPU nodes, the target service request will be queued through an internally maintained queue. Simultaneously, the large language model service system will maintain a certain frequency of GPU node load monitoring and update the first GPU node set. Once the first GPU node set has sufficient GPU nodes, step 301 can be executed to calculate the priority score of each GPU node. Then, step 302 is executed to select target GPU nodes in descending order of their priority scores to provide services to the queued target service requests.

[0113] Furthermore, considering the existence of heterogeneous GPU nodes in the large language model service system, for example, in a scenario where heterogeneous GPU nodes are deployed in a cluster, taking one GPU as a GPU node as an example, different GPU nodes correspond to different GPU types. In related technologies such as the vLLM framework, the service capability differences of different types of GPUs are not considered when distributing service requests to GPU nodes. In this embodiment of the disclosure, based on the above consideration of the actual load capacity of GPU nodes and the caching characteristics of the large language model in the service process, the service capability differences of heterogeneous GPU nodes are combined to make more refined use of computing power.

[0114] Figure 4 A flowchart illustrating a load balancing method according to an embodiment of the present disclosure is shown. Exemplarily, the method can be derived from the above... Figure 1 The language model service system 10 or load balancer 102 at Zhongda University will execute, such as Figure 4 As shown, the method may include the following steps:

[0115] Step 401: Obtain the target service request;

[0116] This step is the same as the above. Figure 2 The same applies to step 201.

[0117] As an example, users can also specify the large language model that handles the target service request, i.e., the target large language model. For instance, multiple large language models can be pre-configured, and users can choose the large language model that provides the inference service according to their needs. As another example, a default large language model, i.e., the target large language model, can be pre-configured to provide the inference service to the user.

[0118] For example, the relevant parameters of the target service request may also include: the type of the large language model and the type of the GPU node.

[0119] As an example, when configuring different types of GPUs on different GPU nodes, the type of GPU node can be represented by the GPU model. For example, a GPU node configured with type A GPUs, a GPU node configured with type B GPUs, etc.

[0120] For example, there is a correspondence between GPU node types and different large language models. For instance, for multiple large language models: large language model A, large language model B, and large language model C, there are four types of GPU nodes: type a, type b, type c, and type d. Each type of GPU node can be configured to have one or more nodes. Thus, type a GPU nodes can be configured to correspond to large language model A, meaning type a GPU nodes can perform inference tasks for large language model A; type b GPU nodes can correspond to large language model B, meaning type b GPU nodes can perform inference tasks for large language model B; and type c and type d GPU nodes can correspond to large language model C, meaning type c and type d GPU nodes can perform inference tasks for large language model C.

[0121] Step 402: Determine the resource type that can be used to process the target service request.

[0122] For example, the resource type that can be used to process the target service request can be determined by the relevant parameters of the target service request; as an example, the resource type may include the type of GPU node and / or the type of large language model; wherein, the type of large language model that can be used to process the target service request is the type of large language model indicated by the relevant parameters of the target service request, and the type of GPU node that can be used to process the target service request is the type of GPU node indicated by the relevant parameters of the target service request.

[0123] Step 403: Obtain the second set of GPU nodes.

[0124] The second set of GPU nodes includes GPU nodes that currently have the capability to process service requests. Specifically, GPU nodes that currently have the capability to process service requests are those that are currently accessible and whose load is either idle or not fully loaded.

[0125] For example, a list of accessible GPU nodes in the large language model service system can be obtained. This list can record the load status of each GPU node in the large language model service system. Then, GPU nodes with service request processing capabilities can be filtered out based on the load status of each GPU node to obtain a second set of GPU nodes. The load status of each GPU node can be determined by its available load capacity. For example, the load status can include idle, partially loaded, and fully loaded. Idle indicates that the GPU node is not currently executing an inference task, and the maximum load capacity of the GPU node is the current available load capacity. Partially loaded indicates that the GPU node is executing an inference task, and a portion of the load capacity has been occupied, but some available load capacity remains. Fully loaded indicates that the GPU node is executing an inference task, and the currently occupied load capacity has reached the maximum load capacity of the GPU node, i.e., the available load capacity is zero.

[0126] As an example, load can be identified using distributed semaphores. For instance, the distributed semaphore S corresponding to a GPU node can be N, indicating that the GPU node can currently handle N inference tasks, where N is an integer ranging from 0 to M, and M represents the maximum number of inference tasks that the GPU node can handle simultaneously. For any GPU node, if the value of the distributed semaphore S corresponding to the GPU node is 0, it means that the GPU node currently has no available load, i.e., the GPU node is fully loaded. If the value of the distributed semaphore S corresponding to the GPU node is M, it means that the GPU node currently has the maximum available load, i.e., the GPU node is idle. If the value of the distributed semaphore S corresponding to the GPU node is between 0 and M, it means that the GPU node still has some available load, i.e., the GPU node is not fully loaded.

[0127] Because the service request volume in a large language model service system is typically very high, the load status of each GPU node in the system changes frequently. For example, the load status of each GPU node in the large language model service system can be dynamically updated. For instance, the load balancer 102 can synchronize the load of each GPU node in real time through a distributed semaphore mechanism, thus obtaining the latest load status of each GPU node in the large language model service system at any given time. For example, for any GPU node, when the GPU node processes an inference task, the inference task requires a certain amount of load on the GPU node, and the value of the corresponding distributed semaphore S of the GPU node will decrease by one; when the GPU node completes an inference task, the GPU node will release the load occupied by processing the inference task, and the value of the corresponding distributed semaphore S of the GPU node will increase by one. In this way, using distributed semaphores as an indicator of the load of each GPU node, and updating the corresponding distributed semaphores of each GPU node, the current load status of each GPU node can be grasped, improving the maintainability of the load capacity of each GPU node in the large language model service system.

[0128] In one possible implementation, the step of obtaining the second GPU node set includes: filtering out the GPU nodes currently capable of processing service requests from the candidate GPU node set based on the distributed semaphores corresponding to each candidate GPU node in the candidate GPU node set. For example, the candidate GPU node set may include the GPU nodes configured in the large language model service system; by using the distributed semaphores corresponding to each candidate GPU node, the GPU nodes currently capable of processing service requests can be quickly and accurately filtered out.

[0129] For example, the resource type of each GPU node in the second GPU node set can also be obtained, such as the type of each GPU node and / or the type of the large language model adapted by each GPU node; as an example, the accessible GPU node list can also record the resource type of each GPU node, so that GPU nodes with service request processing capabilities can be obtained and filtered through the GPU node list.

[0130] In this way, the service capabilities of heterogeneous GPU nodes can be distinguished by the resource types of the GPU nodes.

[0131] Step 404: Determine the GPU node in the second GPU node set that matches the resource type as the GPU node currently available to process the target service request.

[0132] For example, based on the resource type of each GPU node in the second GPU node set, as an example, the resource type may include: the type of GPU node and / or the type of large language model; then the GPU node type can be filtered to be the type of GPU node that can be used to process the target service request, and the GPU node of the adapted large language model is the type of the target large language model, as all GPU nodes that can currently be used to process the target service request, thereby obtaining the first GPU node set.

[0133] Step 405: Obtain the status information of each GPU node in the first GPU node set.

[0134] This step is the same as the above. Figure 2 The same applies to step 202.

[0135] Step 406: Based on the status information of each GPU node in the first GPU node set, select at least one target GPU node from the first GPU node set.

[0136] This step is the same as the above. Figure 2 The same applies to step 203.

[0137] In one possible implementation, after selecting at least one target GPU node from the first set of GPU nodes, the distributed semaphore of the at least one target GPU node is updated. For example, when there are multiple selected target GPU nodes, the distributed semaphore of each target GPU node is updated; for instance, the value of the distributed semaphore of each target GPU node can be reduced by 1.

[0138] Step 407: Use the at least one target GPU node to process the target service request based on the target large language model.

[0139] This step is the same as the above. Figure 2 The same applies to step 204.

[0140] In one possible implementation, this step involves updating the distributed semaphore of the at least one target GPU node after it has completed processing the target service request. For example, when there are multiple target GPU nodes processing the target service request, the distributed semaphore of any target GPU node is updated when it completes its corresponding inference task; for instance, the value of the distributed semaphore of the target GPU node can be increased by 1.

[0141] In this embodiment, the resource type of the GPU node that can be used to process the target service request is determined; a GPU node matching the resource type is determined from the second GPU node set as the GPU node currently available to process the target service request; then, based on the status information of the GPU node currently available to process the target service request, i.e., the longest duration of each inference task currently executed by the GPU node and the number of historical service requests in the session to which the target service request belongs, at least one target GPU node is selected. Thus, considering the actual load capacity of the GPU nodes and the caching characteristics of the large language model during service, and combining the differences in service capabilities of heterogeneous GPU nodes, computing power is used more granularly. This allows for scientific scheduling of heterogeneous GPU nodes based on multi-dimensional factors, evenly distributing service requests, and further improving the quality of the large language model inference service.

[0142] As an example, the load balancing device 102 may include a GPU node registration module, a message synchronization module, a node status maintenance module, and a load balancing scheduling module.

[0143] The GPU node registration module manages the GPU nodes in the large language model service system and allows other functional modules to query information about each GPU node, reducing the coupling of the large language model service system. For example, information about each GPU node can be managed based on a Redis key-value database, such as the type of each GPU node and the large language model it adapts to. For example, an interface can be provided for GPU node registration by encapsulating the key-value database, allowing GPU nodes to register their own information for easier management of GPU nodes with different characteristics. To reduce the storage load on the key-value database, snapshots, read-write isolation, subscriptions, and notifications can be used to implement GPU node registration and query functions while maintaining data correctness and consistency.

[0144] The message synchronization module handles the one-to-many working relationship between model instances and GPU nodes. It enables asynchronous message passing between different GPU nodes and between the load balancing scheduling module and GPU nodes, eliminating the overhead of communication time. For example, the message synchronization module is implemented using a KV database's Stream method, synchronizing inference results generated by the load balancing scheduling module and each GPU node through a push-pull message mechanism.

[0145] The node status maintenance module is used to dynamically update and maintain the working status and load of GPU nodes. For example, it can periodically detect the status information and load of each GPU node in the large language model service system. Alternatively, GPU nodes can report changes in their status information or load when those changes occur. For example, the status information and load of GPU nodes can be synchronized using a distributed semaphore mechanism.

[0146] The load balancing and scheduling module is the central entry point for users to obtain services through the large language model service system. As an example, Figure 5 A flowchart of a load balancing method according to an embodiment of the present disclosure is shown, as follows: Figure 5 As shown, the load balancing module can receive service requests, obtain GPU node information from the GPU node registration module, and determine whether there are enough GPU nodes available to process the service request. If not, the service request is queued and waited for. If so, the module obtains the status information of GPU nodes available to process the service request from the node status maintenance module, thereby filtering out target GPU nodes based on the GPU node status information, and distributing the service request to each target GPU node through the message synchronization module. After receiving the inference results from each target GPU node, the module summarizes these inference results to generate the processing result of the service request and feeds back the processing result to the user.

[0147] Based on the same inventive concept as the above method embodiments, embodiments of this disclosure also provide a load balancing device, which can be used to execute the technical solutions described in the above method embodiments. For example, it can execute the above... Figures 2-5 The steps of the load balancing method shown are illustrated.

[0148] Figure 6 This diagram illustrates a structural diagram of a load balancing device according to an embodiment of the present disclosure, as follows: Figure 6 As shown, the apparatus includes: an acquisition module 601, configured to acquire a target service request; the acquisition module 601 is further configured to acquire the status information of each GPU node in a first GPU node set; the first GPU node set includes: GPU nodes currently available for processing the target service request; wherein the status information of any GPU node includes: the longest duration of each inference task currently executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs; a filtering module 602, configured to filter at least one target GPU node in the first GPU node set based on the status information of each GPU node in the first GPU node set; and a processing module 603, configured to process the target service request based on a target large language model using the at least one target GPU node.

[0149] In this embodiment, based on the caching characteristics of the large language model during service, at least one target GPU node is selected from the first GPU node set according to the status information of each GPU node in the first GPU node set, namely the longest duration of each inference task currently executed by the GPU node and the number of historical service requests in the session to which the target service request belongs. Since the number of historical service requests in the session to which the target service request belongs is relatively large, the probability of hitting cached data when processing the target service request is higher. At the same time, the longer the longest duration of each inference task currently executed, the faster more computing resources can be released to execute other inference tasks. In this way, the selected target GPU node has a higher probability of hitting the cached data of the target GPU node when processing the target service request based on the target large language model, saving computing power and improving the efficiency of the GPU node. It reduces the computing power bottleneck under limited hardware resources. When processing high-volume concurrent service requests, it can make fuller use of hardware resources and effectively leverage the throughput of the large language model, thereby avoiding frequent queuing and waiting of user service requests, resulting in low service latency and greatly improving the quality of inference services provided by the large language model.

[0150] In one possible implementation, the filtering module 602 is further configured to: calculate the priority score of each GPU node based on the status information of each GPU node in the first GPU node set, wherein the priority score represents the priority of processing the target service request; and select at least one GPU node from the first GPU node set as the at least one target GPU node in descending order of the priority scores of each GPU node.

[0151] In one possible implementation, the priority score of the first GPU node is positively correlated with the longest duration of each inference task currently executed by the first GPU node and with the number of historical service requests in the session to which the target service request belongs processed by the first GPU node; wherein, the first GPU node is any GPU node in the first GPU node set.

[0152] In one possible implementation, the filtering module 602 is further configured to: calculate the absolute value of the difference between the longest duration of each inference task currently executed by the first GPU node and the average duration of processing historical inference tasks; wherein the average duration of processing historical inference tasks represents the average duration from when the GPU node acquires a historical inference task to when it completes the historical inference task; calculate the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved corresponding to one historical service request, wherein the time saved corresponding to one historical service request represents the time saved by the GPU node in directly utilizing the cached information corresponding to one historical service request relative to the GPU node's calculation of the cached information during the execution of the inference task; and calculate the priority score of the first GPU node based on the absolute value of the difference and the product; wherein the priority score of the first GPU node is positively correlated with the absolute value of the difference and positively correlated with the product.

[0153] In one possible implementation, the filtering module 602 is further configured to: perform a weighted summation of the absolute value of the difference and the product to obtain the priority score of the first GPU node.

[0154] In one possible implementation, the acquisition module 601 is further configured to: determine the resource type that can be used to process the target service request; acquire a second GPU node set, the second GPU node set including: GPU nodes currently capable of processing service requests; and determine, in the second GPU node set, a GPU node that matches the resource type as the GPU node currently available to process the target service request.

[0155] In one possible implementation, the resource type includes: the type of GPU node and / or the type of large language model.

[0156] In one possible implementation, the filtering module 602 is further configured to: determine the minimum number of GPU nodes corresponding to the target service request; the step of filtering at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set includes: when the minimum number of GPU nodes does not exceed the number of GPU nodes in the first GPU node set, filtering the GPU nodes with the minimum number of GPU nodes as the at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set; and / or, when the minimum number of GPU nodes exceeds the number of GPU nodes in the first GPU node set, updating the first GPU node set; and when the number of GPU nodes in the updated first GPU node set is not less than the minimum number of GPU nodes, filtering the GPU nodes with the minimum number of GPU nodes as the at least one target GPU node from the updated first GPU node set based on the status information of each GPU node in the updated first GPU node set.

[0157] In one possible implementation, if the longest duration of each inference task currently being executed by the first GPU node exceeds the average duration of processing historical inference tasks, the weight of the absolute value of the difference is positively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node; and / or, if the longest duration of each inference task currently being executed by the first GPU node does not exceed the average duration of processing historical inference tasks, the weight of the absolute value of the difference is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is positively correlated with the longest duration of each inference task currently being executed by the first GPU node.

[0158] In one possible implementation, the acquisition module 601 is further configured to: filter out the GPU nodes currently capable of processing service requests from the candidate GPU node set based on the distributed semaphores corresponding to each candidate GPU node in the candidate GPU node set; the processing module 603 is further configured to: update the distributed semaphores corresponding to the at least one target GPU node.

[0159] In one possible implementation, the processing module 603 is further configured to: use a key-value database as a message middleware to synchronize messages between the at least one target GPU node and the node that issued the target service request.

[0160] The above Figure 6 The technical effects and specific descriptions of the load balancing devices and their various possible implementations can be found in the above-mentioned load balancing methods, and will not be repeated here.

[0161] It should be understood that the division of modules in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the modules in the device can be implemented by a processor calling software; for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each module in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the modules in the device can be implemented as hardware circuits. The functionality of some or all modules can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC). The functionality of some or all of the modules is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD). Taking a field-programmable gate array (FPGA) as an example, it can include a large number of logic gates. The connection relationships between these logic gates are configured through configuration files, thereby achieving the functionality of some or all of the modules. All modules of the above device can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0162] In this embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a CPU, microprocessor, graphics processing unit (GPU), digital signal processor (DSP), neural network processing unit (NPU), tensor processing unit (TPU), etc. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships of hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules.

[0163] As can be seen, each module in the above apparatus can be one or more processors (or processing circuits) configured to implement the methods of the above embodiments, such as: CPU, GPU, NPU, TPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types. Furthermore, each module in the above apparatus can be integrated in whole or in part, or can be implemented independently; there is no limitation on this.

[0164] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0165] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0166] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0167] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 7The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0168] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0169] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0170] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0171] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0172] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0173] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0174] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0175] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0176] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0178] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A load balancing method, characterized in that, include: Obtain the target service request; Obtain the status information of each GPU node in the first GPU node set; The first set of GPU nodes is the set of GPU nodes currently available for processing the target service request; wherein, the status information of any GPU node includes: the longest duration of each inference task currently being executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs; the longest duration is determined based on the execution time of each inference task currently being executed by the GPU node; the number of times it has processed historical service requests in the session to which the target service request belongs is the number of service requests in the session to which the GPU node has already processed in the session to which the target service request belongs; Based on the status information of each GPU node in the first GPU node set, at least one target GPU node is selected from the first GPU node set; The target service request is processed using at least one target GPU node based on the target large language model.

2. The method according to claim 1, characterized in that, The step of selecting at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set includes: Based on the status information of each GPU node in the first GPU node set, a priority score is calculated for each GPU node, and the priority score indicates the priority of processing the target service request; According to the priority scores of each GPU node from high to low, at least one GPU node is selected from the first GPU node set as the at least one target GPU node.

3. The method according to claim 2, characterized in that, The priority score of the first GPU node is positively correlated with the longest duration of each inference task currently executed by the first GPU node and with the number of historical service requests in the session to which the target service request belongs processed by the first GPU node; wherein, the first GPU node is any GPU node in the first GPU node set.

4. The method according to claim 2, characterized in that, The step of calculating the priority score of each GPU node based on the state information of each GPU node in the first GPU node set includes: Calculate the absolute value of the difference between the longest duration of each inference task currently being executed by the first GPU node and the average duration of processing historical inference tasks; wherein, the average duration of processing historical inference tasks represents the average time from when the GPU node acquires a historical inference task to when it completes the historical inference task. Calculate the product of the number of times the first GPU node processes historical service requests in the session to which the target service request belongs and the time saved for one historical service request, where the time saved for one historical service request represents the time saved by the GPU node in calculating the cached information by directly using the cached information corresponding to one historical service request during the execution of the inference task. The priority score of the first GPU node is calculated based on the absolute value of the difference and the product; wherein the priority score of the first GPU node is positively correlated with the absolute value of the difference and positively correlated with the product.

5. The method according to claim 4, characterized in that, The calculation of the priority score of the first GPU node based on the absolute value of the difference and the product includes: The priority score of the first GPU node is obtained by weighted summation of the absolute value of the difference and the product.

6. The method according to claim 1, characterized in that, The method further includes: Determine the type of resource that can be used to process the target service request; Obtain the second set of GPU nodes, which is the set of GPU nodes currently capable of processing service requests; In the second set of GPU nodes, a GPU node that matches the resource type is identified as the GPU node currently available to process the target service request.

7. The method according to claim 6, characterized in that, The resource types include: GPU node types and / or large language model types.

8. The method according to claim 1, characterized in that, The method further includes: Determine the minimum number of GPU nodes corresponding to the target service request; The step of selecting at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set includes: If the minimum number of GPU nodes does not exceed the number of GPU nodes in the first GPU node set, the GPU node with the minimum number of GPU nodes is selected from the first GPU node set as the at least one target GPU node based on the status information of each GPU node in the first GPU node set. And / or, If the minimum number of GPU nodes exceeds the number of GPU nodes in the first GPU node set, the first GPU node set is updated; and if the number of GPU nodes in the updated first GPU node set is not less than the minimum number of GPU nodes, the GPU nodes with the minimum number of GPU nodes are selected from the updated first GPU node set as the at least one target GPU node based on the status information of each GPU node in the updated first GPU node set.

9. The method according to claim 5, characterized in that, When the longest duration of each inference task currently being executed by the first GPU node exceeds the average duration of processing historical inference tasks, the weight of the absolute value of the difference is positively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node. And / or, If the longest duration of each inference task currently being executed by the first GPU node does not exceed the average duration of processing historical inference tasks, the weight of the absolute value of the difference is negatively correlated with the longest duration of each inference task currently being executed by the first GPU node, and the weight of the product is positively correlated with the longest duration of each inference task currently being executed by the first GPU node.

10. The method according to claim 6, characterized in that, The process of obtaining the second GPU node set includes: Based on the distributed semaphore corresponding to each candidate GPU node in the candidate GPU node set, the GPU node currently capable of processing service requests is selected from the candidate GPU node set. The method further includes: Update the distributed semaphore corresponding to the at least one target GPU node.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: using a key-value database as a message middleware to synchronize messages between the at least one target GPU node and the node that issued the target service request.

12. A load balancing device, characterized in that, The device includes: The acquisition module is used to acquire the target service request; The acquisition module is further configured to acquire the status information of each GPU node in the first GPU node set; the first GPU node set is a set of GPU nodes currently available for processing the target service request; wherein, the status information of any GPU node includes: the longest duration of each inference task currently executed by the GPU node and the number of times it has processed historical service requests in the session to which the target service request belongs; the longest duration is determined based on the execution time of each inference task currently executed by the GPU node; the number of times it has processed historical service requests in the session to which the target service request belongs is the number of service requests in the session to which the GPU node has processed in the session to which the target service request belongs; The filtering module is used to filter out at least one target GPU node from the first GPU node set based on the status information of each GPU node in the first GPU node set. A processing module is used to process the target service request based on the target large language model using the at least one target GPU node.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 11 when executing instructions stored in the memory.

14. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.

Images