A load balancing method and device for large models

Abstract

The present application relates to the field of data processing technology, and in particular to a load balancing method and device for large models. In an embodiment of the present application, the electronic device determines the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node deployed with a large model based on relevant process parameters and monitoring indicators in the large model inference process, and determines the load index of each large model inference node based on the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node, so that inference requests are allocated to each large model inference node according to the load index, realizing the optimization of the large model multi-node inference load balancing strategy, improving resource utilization, and improving inference efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a load balancing method and device for large models. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the rapid development of large model technology, artificial intelligence has opened up a new paradigm and has the potential to penetrate into thousands of industries. It has become an industry trend to realize scenario-based applications based on large models. Due to the large model resource usage and low reasoning efficiency in the actual deployment and application of large models, it is inevitable to meet the needs of high throughput and high real-time scenario applications through distributed deployment of large models, that is, to build large model reasoning nodes, deploy large models in each large model reasoning node, and then schedule the large model by scheduling the large model reasoning nodes.

[0003] However, in the scenario of distributed deployment of large models, traditional load balancing strategies cannot effectively distribute the load, cannot fully utilize the resource capabilities of each large model, and cannot achieve optimal reasoning performance. Based on this, how to achieve efficient reasoning of large models in the application process through effective load balancing strategies has become a technical problem to be solved. Summary of the invention

[0004] The present application provides a load balancing method and device for large models, which is used to solve the problem that traditional load balancing strategies in existing scenarios of distributed deployment of large models cannot effectively distribute loads, cannot fully utilize resource capabilities, and cannot achieve optimal reasoning performance.

[0005] In a first aspect, an embodiment of the present application provides a load balancing method for a large model, the method comprising:

[0006] Obtaining operating parameters of each large model inference node where the large model is deployed; wherein the operating parameters are determined according to relevant process parameters and monitoring indicators in the large model inference process, including real-time load parameters, predicted load parameters, resource utilization, and processing capacity parameters;

[0007] Determine the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization rate and processing capacity parameter corresponding to each large model inference node; wherein the larger the predicted load parameter and the real-time load parameter, the larger the load index; the larger the resource utilization rate and the processing capacity parameter, the smaller the load index;

[0008] The inference request to be processed is distributed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

[0009] The above technical solution has the following advantages or beneficial effects: In the embodiment of the present application, the electronic device determines the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node deployed with the large model based on the relevant process parameters and monitoring indicators in the large model inference process, and determines the load index of each large model inference node based on the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node, so that inference requests are allocated to each large model inference node according to the load index, thereby optimizing the large model multi-node inference load balancing strategy, improving resource utilization, and improving inference efficiency.

[0010] In some embodiments of the present application, the process of determining the real-time load parameter includes:

[0011] For each large model inference node, a first proportion of the number of requests in the maximum number of requests is determined based on the number of requests contained in each queue in the large model inference node and the corresponding maximum number of requests allowed to be contained; a second proportion of the storage space value in the maximum storage space value is determined based on the currently occupied storage space value of the large model inference node and the maximum storage space value allowed to be occupied; the real-time load parameter is determined based on the forward transfer token rate of the large model deployed in the large model inference node, the first proportion and the second proportion; wherein each queue includes a waiting queue, a preemptive queue and a running queue, and the larger the first proportion, the second proportion and the forward transfer token rate, the larger the real-time load parameter.

[0012] The above technical solution has the following advantages or beneficial effects: it determines the real-time load parameters of the large model based on the information of the inference queue in the large model inference node, the storage space value currently occupied by the large model inference node, the maximum storage space value allowed to be occupied, and the forward pass token rate, and realizes load balancing based on relevant process parameters and monitoring indicators in the large model inference process, making the load balancing method more suitable for large models.

[0013] In some embodiments of the present application, the process of determining the predicted load parameter includes:

[0014] For each large model inference node, obtain the current output length of each request being processed by the large model inference node and the average output length of the large model inference node; determine the predicted remaining output length corresponding to each request based on the relationship between the current output length corresponding to each request and the average output length; determine the predicted load parameter based on the predicted remaining output length corresponding to each request and the number of each request.

[0015] The above technical solution has the following advantages or beneficial effects: it implements the prediction of the load parameters of the large model inference node to complete the request currently being processed based on the current output length and the average output length in the large model inference node, and takes into account the impact of pending requests in the load balancing process, making the load balancing result more accurate.

[0016] In some embodiments of the present application, determining the predicted remaining output length corresponding to each request according to the size relationship between the current output length corresponding to each request and the average output length includes:

[0017] For each request, if the target current output length corresponding to the request is not less than the average output length, the predicted remaining output length is determined based on the maximum output length corresponding to the saved large model inference node and the target current output length; if the target current output length corresponding to the request is less than the average output length, the predicted remaining output length is determined based on the target current output length and the average output length.

[0018] The above technical solution has the following advantages or beneficial effects: according to the size relationship between the current output length corresponding to the request and the average output length, the predicted remaining output length of each large model inference node can be predicted more accurately.

[0019] In some embodiments of the present application, the process of determining the resource utilization rate includes:

[0020] For each large model inference node, obtain the saved CPU utilization, GPU utilization and memory utilization of the large model inference node; determine the resource utilization based on the CPU utilization, GPU utilization and memory utilization.

[0021] The above technical solution has the following advantages or beneficial effects: based on the CPU utilization, GPU utilization and memory utilization of the large model inference node, the calculation of the resource utilization of the large model inference node is realized.

[0022] In some embodiments of the present application, the process of determining the processing capability parameter includes:

[0023] For each large model inference node, determine the first average token generation time and throughput of the large model inference node that are saved, and determine the second average token generation time and average throughput of each large model inference node that are saved; determine the third proportion of the token generation time of the large model inference node in each large model inference node based on the first average token generation time and the second average token generation time; determine the fourth proportion of the throughput of the large model inference node in each large model inference node based on the throughput and the average throughput; determine the processing capacity parameter based on the third proportion and the fourth proportion; wherein, the larger the third proportion, the smaller the processing energy parameter; the larger the fourth proportion, the larger the processing capacity parameter.

[0024] The above technical solution has the following advantages or beneficial effects: based on the relevant process parameters and monitoring indicators in the large model reasoning process, the calculation of the processing capacity parameters of the large model reasoning node is realized, making the load balancing method more suitable for large models.

[0025] In some embodiments of the present application, before determining the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization and processing capacity parameter corresponding to each large model inference node, the method further includes:

[0026] According to the real-time load parameters of each large model inference node, determining whether there is an idle large model inference node with no load at present;

[0027] If so, the inference request is distributed to the idle large model inference node;

[0028] If it does not exist, a subsequent process is performed to determine the load index corresponding to each large model inference node based on the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node.

[0029] The above technical solution has the following advantages or beneficial effects: first determine whether there is an idle large model inference node, and if so, directly distribute the inference request to the idle large model inference node, avoiding load balancing calculation and improving the efficiency of inference request allocation.

[0030] In some embodiments of the present application, the determination process of each large model reasoning node includes:

[0031] Obtain the latest sending time of the node information of each saved candidate large model inference node; wherein the node information includes an access path and a node identifier;

[0032] Determine each target candidate large model inference node whose time difference between the corresponding latest sending time and the current time is within a preset difference range;

[0033] Each target large model inference node is determined as each large model inference node.

[0034] The above technical solution has the following advantages or beneficial effects: ensuring that each large model inference node participating in load balancing is online and can receive and process inference requests.

[0035] In some embodiments of the present application, distributing the inference request to the target large model inference node with the smallest load index includes:

[0036] The saved target access path corresponding to the target large model inference node is obtained, and the inference request is distributed to the target large model inference node through the target access path.

[0037] The above technical solution has the following advantages or beneficial effects: distribution of inference requests is achieved.

[0038] In a second aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned load balancing methods for large models when executing a computer program stored in the memory.

[0039] In an embodiment of the present application, an electronic device obtains operating parameters of each large model inference node deployed with a large model; wherein the operating parameters are determined according to the parameters of the inference stage of the large model, including real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters; according to the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node, the load index corresponding to each large model inference node is determined; wherein, the larger the predicted load parameters and the real-time load parameters, the larger the load index; the larger the resource utilization and the processing capacity parameters, the smaller the load index; the inference request to be processed is distributed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request. In an embodiment of the present application, the electronic device determines the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node deployed with the large model based on relevant process parameters and monitoring indicators in the large model inference process, and determines the load index of each large model inference node based on the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node, so that inference requests are allocated to each large model inference node according to the load index, thereby optimizing the large model multi-node inference load balancing strategy, improving resource utilization, and at the same time improving inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of a load balancing process for a large model provided in an embodiment of the present application;

[0042] Figure 2 A business process diagram provided for an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a page of a large model reasoning platform provided in an embodiment of the present application;

[0044] Figure 4 The overall architecture diagram of the load balancing provided in the embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a load balancing device for large models provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0048] In order to achieve effective load distribution in the scenario of distributed deployment of large models, improve the resource utilization of large models, and at the same time improve the reasoning efficiency, the embodiments of the present application provide a load balancing method, device, equipment and medium for large models.

[0049] In an embodiment of the present application, an electronic device obtains operating parameters of each large model inference node deployed with a large model; wherein the operating parameters are determined according to the parameters of the inference stage of the large model, including real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters; according to the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node, the load index corresponding to each large model inference node is determined; wherein, the larger the predicted load parameters and the real-time load parameters, the larger the load index; the larger the resource utilization and the processing capacity parameters, the smaller the load index; the inference request to be processed is distributed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

[0050] Figure 1 A schematic diagram of a load balancing process for a large model provided in an embodiment of the present application, the process includes:

[0051] S101: Obtain the operating parameters of each large model inference node where the large model is deployed; wherein the operating parameters are determined based on relevant process parameters and monitoring indicators in the large model inference process, including real-time load parameters, predicted load parameters, resource utilization, and processing capacity parameters.

[0052] A large-model oriented load balancing method provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.

[0053] The embodiment of the present application provides a load balancing method for large model reasoning, which is a general method for large model reasoning and can be applied to large models in different scenarios. First, the large model is distributed and deployed to form multiple large model reasoning nodes. The electronic device implements the reasoning process by scheduling each large model reasoning node.

[0054] Among them, in the embodiment of the present application, the electronic device can be composed of multiple modules, including but not limited to a registration center, an operation monitoring module and a load balancing module. Among them, the registration center receives the registration information of each large model inference node, providing a basis for load balancing; the operation monitoring module is responsible for real-time monitoring of the operation of each large model inference node, and determining the operation parameters of each large model inference node, which includes not only general indicators such as resource utilization, but also real-time load parameters, processing capacity parameters and predicted load parameters designed for large model inference; the load balancing module is used to perform relevant calculations according to the load balancing strategy, obtain load distribution decisions, and then distribute the inference requests.

[0055] In addition, in the embodiments of the present application, the above-mentioned registration center, operation monitoring module and load balancing module can also be deployed in different electronic devices or systems, or two of them can be deployed in one electronic device or system, and the other one can be deployed in other electronic devices or other systems, without limitation here.

[0056] It should be noted that in the embodiment of the present application, at least one large model is deployed in each large model inference node, and the large model can be a large language model or other multimodal large model, which is not limited here. In addition, the operating parameters of each large model inference node are determined by the electronic device based on the relevant process parameters and monitoring indicators in the inference process of each large model deployed in the large model inference node.

[0057] Among them, the operating parameters of the large model inference node include but are not limited to real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters.

[0058] Specifically, for each large model inference node, the electronic device uses a monitoring operation module to obtain relevant process parameters and monitoring indicators in the large model inference process deployed in the large model inference node, and determines and saves the operation parameters of the large model inference node based on the relevant process parameters and the monitoring indicators.

[0059] When an electronic device receives an inference request to be processed by a large model sent by a business system through a gateway or other channels, the electronic device obtains the saved operating parameters of each large model inference node and performs load balancing based on the operating parameters of each large model inference node.

[0060] S102: Determine the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization and processing capacity parameter corresponding to each large model inference node; wherein, the larger the predicted load parameter and the real-time load parameter, the larger the load index; the larger the resource utilization and the processing capacity parameter, the smaller the load index.

[0061] In an embodiment of the present application, the electronic device can determine the load index corresponding to each large model inference node based on the predicted load parameters, real-time load parameters, resource utilization and processing power parameters of each large model inference node, and then allocate inference tasks based on the load index corresponding to each large model inference node.

[0062] Specifically, in an embodiment of the present application, for each large model inference node, the electronic device determines the sum of the predicted load parameters of each large model inference node, and determines the ratio of the large model inference node to the sum of the predicted load parameters; the electronic device determines the product A of the real-time load parameter of the large model inference node and the ratio; the electronic device determines the logarithm of the resource utilization of the large model inference node with the irrational number e as the base, performs a square root operation on the absolute value of the logarithm, and determines the product B of the value obtained by the square root operation and the influencing factor of the preset resource utilization parameter and the processing capacity parameter of the large model inference node; the electronic device determines the ratio of the product A to the product B as the load index of the large model inference node.

[0063] Specifically, the electronic device can determine the load index of each large model inference node by the following formula:

[0064]

[0065] Among them, W(N i ) represents the load index of the i-th large model inference node, R(N i ) represents the real-time load parameter of the i-th large model inference node, P(N i ) represents the predicted load parameter of the i-th large model inference node, D(N i ) represents the processing capacity parameter of the large model inference node, L(N i ) represents the resource utilization of the i-th large model inference node, τ is the influencing factor of the resource utilization parameter, which can be configured by the technicians according to actual needs. Generally, τ = 0.2.

[0066] It should be noted that in the embodiment of the present application, the real-time load parameters of the large model inference node are calculated by the inference queue length, forward transfer token rate and Kv-cache occupied space of the large model deployed in the large model inference node, representing the current real-time load situation of the large model inference; the predicted load parameters of the large model inference node are calculated by the current output length of the large model, representing the predicted load situation that the large model needs to occupy to complete the current inference; the resource utilization of the large model inference node is basically consistent with the statistical method of the traditional load balancing method; the processing capacity parameters of the large model inference node are calculated by the average token generation time and average throughput of the large models deployed in each large model inference node.

[0067] Among them, the larger the predicted load parameter and the real-time load parameter of the large model inference node, the larger the load index corresponding to the large model inference node; the larger the resource utilization and the processing capacity parameters of the large model inference node, the smaller the load index corresponding to the large model inference node.

[0068] S103: Distribute the inference request to be processed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

[0069] In an embodiment of the present application, after determining the load index corresponding to each large model inference node, the electronic device determines a target large model inference node with the smallest load index among each large model inference node. The smallest load index indicates that the load pressure of the target large model inference node is smaller.

[0070] Based on this, the electronic device distributes the inference request to be processed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

[0071] In an embodiment of the present application, the electronic device determines the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node deployed with the large model based on relevant process parameters and monitoring indicators in the large model inference process, and determines the load index of each large model inference node based on the real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters of each large model inference node, so that inference requests are allocated to each large model inference node according to the load index, thereby optimizing the large model multi-node inference load balancing strategy, improving resource utilization, and at the same time improving inference efficiency.

[0072] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the process of determining the real-time load parameters includes:

[0073] For each large model inference node, a first proportion of the number of requests in the maximum number of requests is determined based on the number of requests contained in each queue in the large model inference node and the corresponding maximum number of requests allowed to be contained; a second proportion of the storage space value in the maximum storage space value is determined based on the currently occupied storage space value of the large model inference node and the maximum storage space value allowed to be occupied; the real-time load parameter is determined based on the forward transfer token rate of the large model deployed in the large model inference node, the first proportion and the second proportion; wherein each queue includes a waiting queue, a preemptive queue and a running queue, and the larger the first proportion, the second proportion and the forward transfer token rate, the larger the real-time load parameter.

[0074] In the embodiment of the present application, the large model reasoning node is a unit of large model distributed reasoning, and each node has independent and complete reasoning capabilities. The large model reasoning node can be a separate server, or a containerized reasoning service deployed based on cloud native technology, or other deployment methods.

[0075] Among them, the large model inference node will collect each relevant process parameter and each monitoring indicator in the large model inference process in real time, and send them to the resource monitoring module of the electronic device in a unified manner. The electronic device calculates and stores the operating parameters of the large model inference node.

[0076] Among them, the large model inference node sends the following relevant process parameters and monitoring indicators to the electronic equipment:

[0077] 1. Parameters of the inference queue of the large model: The inference queue of the large model includes a waiting queue, a running queue and a preemptive queue. Among them, after the large model inference node receives the inference request, it will first add the inference request to the waiting queue. After the round of inference is completed, the inference request in the waiting queue will be input into the running queue, and the unfinished inference request in the running queue will be input into the preemptive queue. When there is free space in the large model, the inference request in the preemptive queue will be put into the waiting queue. Based on this, the large model inference node will monitor the number of requests contained in each queue and the corresponding maximum number of requests allowed to be included, and send the number of requests contained in each queue and the corresponding maximum number of requests allowed to be included to the electronic device.

[0078] 2. Parameters for single-node running status statistics: mainly for the statistics of large model inference data in a single node, including the collection of real-time parameters such as forward pass token rate, kv-cache space occupied, and historical parameters such as inference request result length. At the same time, the token generation time, node throughput, etc. are counted and sent.

[0079] In addition, large model inference nodes can also use inference acceleration technologies such as continuous batch processing and dynamic segmentation to perform large model inference.

[0080] Since the electronic device has the same calculation process for the real-time load parameters of each large model inference node, in the embodiment of the present application, a large model inference node is taken as an example to illustrate the real-time load parameter calculation process.

[0081] Specifically, when the electronic device is determining the real-time load parameters of a large model inference node, the electronic device can determine a first proportion of the number of requests in the maximum number of requests based on the number of requests contained in each queue in the large model inference node and the corresponding maximum number of requests allowed to be contained; determine a second proportion of the storage space value in the maximum storage space value based on the currently occupied storage space value of the large model inference node and the maximum storage space value allowed to be occupied; and determine the real-time load parameters based on the forward transfer token rate, the first proportion, and the second proportion of the large model deployed in the large model inference node.

[0082] Among them, the storage space value is the Kv-cache space currently used by the large model, divided by the unit space size to calculate the value, which can also be called the Kv-cache quantity; each queue includes a waiting queue, a preemptive queue and a running queue. The maximum number of requests allowed to be included in each queue is a pre-configured item and does not change with the operation of the large model inference node. In addition, the number of requests contained in the queue can also be the real-time length of the queue, and the maximum number of requests allowed to be included in the queue can also be the maximum length of the queue; the larger the first proportion, the second proportion and the forward transfer token rate, the larger the real-time load parameter.

[0083] Specifically, the electronic device determines a first sub-ratio of the number of requests included in the waiting queue and the corresponding maximum number of requests allowed to be included, determines a second sub-ratio of the number of requests included in the running queue and the corresponding maximum number of requests allowed to be included, and determines a third sub-ratio of the number of requests included in the preemption queue and the corresponding maximum number of requests allowed to be included. The electronic device also stores a weight factor for each queue when calculating the real-time load parameter, and the weight factor corresponding to each queue adds up to 1. Generally, the weight factor corresponding to the waiting queue stored in the electronic device is 0.4, the weight factor corresponding to the running queue is 0.4, and the weight factor corresponding to the preemption queue is 0.2. The electronic device performs a weighted sum of the sub-ratios corresponding to each queue based on the weight factor corresponding to each queue, and determines the weighted sum of each sub-ratio as the first ratio.

[0084] The electronic device determines the product of the first proportion and the forward transfer token rate of the large model deployed in the large model inference node; the electronic device determines a power with an irrational number e as the base and a negative value of the second proportion as the exponent, and determines the sum of a preset value and the power; the electronic device determines the ratio of the product to the sum as the real-time load parameter of the large model inference node.

[0085] Specifically, the electronic device can determine the real-time load parameters of the large model inference node according to the following formula:

[0086]

[0087] Among them, R(N i ) represents the real-time load parameter of the i-th large model inference node; l w , l r and l s They represent the number of requests contained in the waiting queue, running queue, and preemption queue of the i-th large model inference node; l w_max , l r_max and l s_max They represent the maximum number of requests allowed in the waiting queue, running queue, and preemption queue of the i-th large model inference node; σ w , σ r and σ s They represent the weight factors of the waiting queue, running queue and preemption queue of the i-th large model inference node when calculating the real-time load parameters, σ w +σ r +σ s =1; V i represents the forward pass token rate of the large model in the large model inference node; C kv_t represents the storage space currently occupied by the i-th large model inference node, C kv Indicates the maximum storage space value allowed to be occupied by the i-th large model inference node.

[0088] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the process of determining the predicted load parameters includes:

[0089] For each large model inference node, obtain the current output length of each request being processed by the large model inference node and the average output length of the large model inference node; determine the predicted remaining output length corresponding to each request based on the relationship between the current output length corresponding to each request and the average output length; determine the predicted load parameter based on the predicted remaining output length corresponding to each request and the number of each request.

[0090] In an embodiment of the present application, the predicted load parameter is a load value required for predicting the large model inference node to complete each request currently being processed.

[0091] Since the electronic device has the same calculation process for the predicted load parameters of each large model inference node, in the embodiment of the present application, a large model inference node is taken as an example to illustrate the calculation process of the predicted load parameters.

[0092] In an embodiment of the present application, the electronic device determines the predicted remaining output length corresponding to each request based on the relationship between the current output length corresponding to each request and the average output length; and determines the predicted load parameter based on the predicted remaining output length corresponding to each request and the number of each request.

[0093] Specifically, if the current output length corresponding to the request is not less than the average output length, the preset output length is determined to be a first value, wherein the first value is determined based on the maximum output length of the large model and the current output length; if the current output length corresponding to the request is less than the average output length, the preset output length is determined to be a second value, wherein the second value is determined based on the average output length of the large model and the current output length.

[0094] The electronic device determines the root mean square of each average output length, that is, first performs a square operation on each average output length, then determines the mean of the square values ​​of each average output length, and then performs a square root operation on the mean, and determines the result of the square root operation as the predicted load parameter.

[0095] Among them, the electronic device can determine the predicted load parameter based on the following formula:

[0096]

[0097] Among them, P(N i ) represents the predicted load parameter of the i-th large model inference node; m represents that there are m requests being processed in the i-th large model inference node; p j Indicates the predicted output length corresponding to the j-th request.

[0098] Among them, in the embodiment of the present application, the current output length corresponding to each request is the token length currently calculated for each request, and the lengths in the maximum output length, average output length and predicted output length are all token lengths.

[0099] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the predicted remaining output length corresponding to each request is determined according to the size relationship between the current output length corresponding to each request and the average output length, including:

[0100] For each request, if the target current output length corresponding to the request is not less than the average output length, the predicted remaining output length is determined based on the maximum output length corresponding to the saved large model inference node and the target current output length; if the target current output length corresponding to the request is less than the average output length, the predicted remaining output length is determined based on the target current output length and the average output length.

[0101] In an embodiment of the present application, for each request, if the target current output length corresponding to the request is not less than the average output length, the predicted remaining output length is determined based on the maximum output length and the target current output length corresponding to the saved large model inference node; if the target current output length corresponding to the request is less than the average output length, the predicted remaining output length is determined based on the target current output length and the average output length.

[0102] Specifically, if the target current output length corresponding to the request is not less than the average output length, a first absolute value of the difference between the maximum output length and the target current output length is determined, and the product of the first absolute value and a preset output length prediction influence factor is determined as the predicted remaining length; if the target current output length corresponding to the request is less than the average output length, a second absolute value of the difference between the target current output length and the average output length is determined, and the second absolute value is determined as the predicted remaining length

[0103] The electronic device may determine the predicted remaining length of each request based on the following formula:

[0104]

[0105] Among them, p j represents the predicted remaining length of the jth request, L max Indicates the maximum remaining length, L j Indicates the current output length of the jth request, L avg represents the average output length, μ represents the preset output length prediction impact factor, generally, μ=0.3.

[0106] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the process of determining the resource utilization includes:

[0107] For each large model inference node, obtain the saved CPU utilization, GPU utilization and memory utilization of the large model inference node; determine the resource utilization based on the CPU utilization, GPU utilization and memory utilization.

[0108] In the embodiment of the present application, resource utilization is the statistics of CPU utilization, GPU utilization and memory utilization of the large model inference node.

[0109] Since the electronic device has the same process for calculating the resource utilization of each large model inference node, in the embodiment of the present application, a large model inference node is taken as an example to illustrate the resource utilization calculation process.

[0110] Specifically, the electronic device obtains the CPU utilization, GPU utilization and memory utilization of the saved large model inference node; performs weighted summation based on the CPU utilization, GPU utilization and memory utilization to determine the resource utilization corresponding to the large model inference node.

[0111] Among them, the electronic device can determine the resource utilization rate through the following formula:

[0112] L(N i )=δ c ·C i +δ g ·G i +δ m ·M i

[0113] Among them, L(N i ) represents the resource utilization of the i-th large model inference node, C i represents the CPU utilization of the i-th large model inference node, G i represents the GPU utilization of the i-th large model inference node, M i represents the memory utilization of the i-th large model inference node, δ c ,δ g ,δ m They represent the weight factors corresponding to CPU utilization, GPU utilization, and memory utilization, which add up to 1. They are generally 0.3, 0.4, and 0.3 respectively.

[0114] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the process of determining the processing capacity parameter includes:

[0115] For each large model inference node, determine the saved first average token generation time and first throughput of the large model inference node, and determine the saved second average token generation time and second throughput of each large model inference node; determine the third proportion of the token generation time of the large model inference node in each large model inference node based on the first average token generation time and the second average token generation time; determine the fourth proportion of the throughput of the large model inference node in each large model inference node based on the first throughput and the second throughput; determine the processing capacity parameter based on the third proportion and the fourth proportion; wherein, the larger the third proportion, the smaller the processing energy parameter; the larger the fourth proportion, the larger the processing capacity parameter.

[0116] In an embodiment of the present application, the large model inference node processing capacity parameter is calculated based on the average token generation time and average throughput of each large model inference node.

[0117] Since the electronic device has the same calculation process for the processing capability parameters of each large model inference node, in the embodiment of the present application, a large model inference node is taken as an example to illustrate the calculation process of the processing capability parameters.

[0118] Specifically, in an embodiment of the present application, the electronic device determines the first average token generation time and the first throughput of the saved large model inference node, and determines the second average token generation time and the average throughput of each saved large model inference node.

[0119] It should be noted that the first average token generation time is different from the second average token generation time. The first average token generation time is obtained by averaging the token generation time of each request processed by the large model inference node, while the second average token generation time is obtained by averaging the first average token generation time of each large model inference node.

[0120] Throughput and average throughput are also different. Throughput is the throughput of a large model inference node, while average throughput is the average throughput of each large model inference node.

[0121] The electronic device determines the third proportion of the token generation time of the large model inference node in each large model inference node based on the first average token generation time and the second average token generation time. According to the throughput and the average throughput, the fourth proportion of the throughput of the large model inference node in each large model inference node is determined; according to the third proportion and the fourth proportion, the processing capacity parameter is determined. Among them, the larger the third proportion, the smaller the processing capacity parameter; the larger the fourth proportion, the larger the processing capacity parameter.

[0122] Specifically, the electronic device uses a preset value to perform a power operation on the third ratio, and determines the product of the power operation result and the fourth ratio, and determines the product as the processing capability parameter.

[0123] The preset value may be (-θe), where θ is an adjustment factor of a preset parameter weight.

[0124] Specifically, the electronic device can determine the processing capability parameter of the large model inference node based on the following formula:

[0125]

[0126] Among them, D(N i ) represents the processing capacity parameter of the i-th large model inference node, T i is the first average token generation time, T avg The second average token generation time for each large model inference node, is the third proportion of the token generation time of the i-th large model inference node in each large model inference node, -θe is the preset value, θ is the adjustment factor of the preset parameter weight, Q i is the throughput of the i-th large model inference node, Q avg represents the average throughput of each large model inference node, It is the fourth proportion of the throughput of the i-th large model inference node in each large model inference node.

[0127] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of the large model, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, before determining the load index corresponding to each large model reasoning node according to the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model reasoning node, the method also includes:

[0128] According to the real-time load parameters of each large model inference node, determining whether there is an idle large model inference node with no load at present;

[0129] If so, the inference request is distributed to the idle large model inference node;

[0130] If it does not exist, a subsequent process is performed to determine the load index corresponding to each large model inference node based on the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node.

[0131] In an embodiment of the present application, in order to improve the efficiency of load balancing, after receiving an inference request, the electronic device will traverse the current inference node status and adopt different load balancing strategies according to the inference node status.

[0132] Specifically, the electronic device determines whether there is an idle large model inference node with no load based on the real-time load parameters of each large model inference node; if so, the inference request is distributed to the idle large model inference node; if not, the subsequent process of determining the load index corresponding to each large model inference node based on the predicted load parameters, real-time load parameters, resource utilization and processing power parameters corresponding to each large model inference node is executed.

[0133] Among them, if the electronic device determines that there are currently multiple large model inference nodes without load, the electronic device randomly selects from the large model inference nodes without load for inference until all large model inference nodes have tasks to execute; if the electronic device determines that all current large model inference nodes have inference loads, it is necessary to calculate the load index of each large model inference node and select the large model inference node with the lowest load index for task dispatch.

[0134] Figure 2 The business process diagram provided in the embodiment of the present application is as follows: Figure 2 As shown, the process includes: the business system initiates an inference request, the load balancing module calculates the load index, and selects the assigned target large model inference node, and then sends the request to the target large model inference node for model inference. After the target large model inference node completes the inference, it returns the inference result to the business system.

[0135] In an embodiment of the present application, when using the large model inference platform, the user can select the load balancing strategy to be used on the configuration page. In addition to the load balancing method provided in the embodiment of the present application, traditional load balancing methods will also be provided for the user to choose.

[0136] Figure 3 A schematic diagram of a page of a large model reasoning platform provided in an embodiment of the present application, such as the Figure 3 As shown, in this large model inference platform, users can choose the number of inference nodes, the type of inference nodes, the node deployment method, and the load balancing strategy. Figure 3 The "dedicated load balancing strategy for large model reasoning" is a load balancing method provided in an embodiment of the present application. The user can also select one or more parameters for calculating the load index, which include the real-time load index of the inference node (real-time load parameter), the predicted load index of the inference node (predicted load parameter), the processing capacity index of the inference node (processing capacity parameter) and the resource utilization index of the inference node (resource utilization).

[0137] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the determination process of each large model reasoning node includes:

[0138] Obtain the latest sending time of the node information of each saved candidate large model inference node; wherein the node information includes an access path and a node identifier;

[0139] Determine each target candidate large model inference node whose time difference between the corresponding latest sending time and the current time is within a preset difference range;

[0140] Each target large model inference node is determined as each large model inference node.

[0141] In order to ensure that the selected target large model inference node can respond to the inference request, in an embodiment of the present application, the electronic device will first screen out each large model inference node that is currently online before determining the target large model inference node, and then determine the target inference node from each large model inference node.

[0142] Among them, in the embodiment of the present application, each large model reasoning node sends node information to the electronic device at a set time interval, and the node information includes an access path and a node identifier. The electronic device can determine whether the large model reasoning node is online based on the time difference between the latest sending time of the node information corresponding to each large model reasoning node and the current time.

[0143] In order to distinguish the description, in the embodiment of the present application, the large model inference node before online screening is referred to as a candidate large model inference node. Based on this, the electronic device obtains the latest sending time of the node information of each candidate large model inference node saved; the electronic device determines each target candidate large model inference node whose time difference between the corresponding latest sending time and the current time is within a preset difference range, and determines each target large model inference node as each large model inference node to be load balanced.

[0144] In order to effectively distribute the load in the scenario of distributed deployment of large models, improve the resource utilization of large models, and improve the reasoning efficiency, based on the above embodiments, in the embodiment of the present application, the inference request is distributed to the target large model reasoning node with the smallest load index, including:

[0145] The saved target access path corresponding to the target large model inference node is obtained, and the inference request is distributed to the target large model inference node through the target access path.

[0146] In an embodiment of the present application, after receiving the node information sent by the large model inference node, the electronic device will save the node information and can subsequently schedule the corresponding large model inference node based on the saved node information.

[0147] Specifically, in an embodiment of the present application, when the electronic device distributes an inference request to the target large model inference node with the smallest load index, the electronic device will first obtain the target access path corresponding to the saved target large model inference node, and distribute the inference request to the target large model inference node through the target access path.

[0148] It should be noted that in the embodiments of the present application, it includes but is not limited to a registration center, an operation monitoring module and a load balancing module. Among them, the registration center receives the registration information of each large model reasoning node, and the registration information is the node information. The node information corresponding to each large model reasoning node is uniformly managed and maintained by the registration center. When performing load balancing and request distribution, it is necessary to obtain access through the registration center. The operation monitoring module is mainly responsible for collecting the operation data collected by each large model reasoning node and performing corresponding calculations. The operation monitoring module has the ability to manage multiple large model reasoning nodes at the same time, and can process the reported data of multiple large model reasoning nodes. Its main functions include: operation parameter processing, processing the operation parameters reported by the large model reasoning node, including storing real-time parameters, statistically calculating the processing capacity parameters, and completing the prediction of the output length. These processing processes are carried out in real time and provided to the load balancing module for load index calculation, and load balancing processing is performed based on this. The load balancing module is mainly responsible for calculating the load index based on the various parameters obtained and counted by the operation monitoring module after receiving the inference request, and obtaining the most suitable target large model inference node based on the calculation results, and sending the inference request to the target large model inference node for inference.

[0149] Figure 4 The overall architecture diagram of the load balancing provided in the embodiment of the present application is as follows: Figure 4 As shown, the architecture includes a business system, a gateway, a registration center, a load balancing module, a resource monitoring module, a large model reasoning node 1, a large model reasoning node 2 and a large model reasoning node 3.

[0150] Among them, the business system sends an inference request to the load balancing module through the gateway; the resource monitoring module obtains the operating parameters of each large model inference node, and determines the real-time load parameters, predicted load parameters, processing capacity parameters and resource utilization of each large model inference node; the registration center obtains the node information of each large model inference node; the load balancing module determines the target large model inference node with the smallest inference index based on the real-time load parameters, predicted load parameters, processing capacity parameters and resource utilization of each large model inference node, and distributes the inference request to the target large model inference node through the access path in the node information saved by the registration center. Among them, each large model inference node includes: inference queue, inference module and device monitoring.

[0151] Based on the above embodiments, Figure 5 A schematic diagram of a load balancing device for a large model provided in an embodiment of the present application, the device comprising:

[0152] The acquisition module 501 is used to acquire the operating parameters of each large model inference node where the large model is deployed; wherein the operating parameters are determined according to relevant process parameters and monitoring indicators in the large model inference process, including real-time load parameters, predicted load parameters, resource utilization and processing capacity parameters;

[0153] The load balancing module 502 is used to determine the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization rate and processing capacity parameter corresponding to each large model inference node; wherein, the larger the predicted load parameter and the real-time load parameter, the larger the load index; the larger the resource utilization rate and the processing capacity parameter, the smaller the load index;

[0154] The distribution module 503 is used to distribute the inference request to be processed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

[0155] In a possible implementation, the load balancing module 502 is also used to determine, for each large model inference node, a first proportion of the number of requests in the maximum number of requests based on the number of requests contained in each queue in the large model inference node and the corresponding maximum number of requests allowed to be contained; determine a second proportion of the storage space value in the maximum storage space value based on the currently occupied storage space value of the large model inference node and the maximum storage space value allowed to be occupied; determine the real-time load parameter based on the forward transfer token rate of the large model deployed in the large model inference node, the first proportion and the second proportion; wherein each queue includes a waiting queue, a preemptive queue and a running queue, and the larger the first proportion, the second proportion and the forward transfer token rate, the larger the real-time load parameter.

[0156] In a possible implementation, the load balancing module 502 is also used to obtain, for each large model inference node, each current output length of each request being processed by the large model inference node and the average output length of the large model inference node; determine the predicted remaining output length corresponding to each request based on the relationship between the current output length corresponding to each request and the average output length; determine the predicted load parameter based on the predicted remaining output length corresponding to each request and the number of each request.

[0157] In one possible implementation, the load balancing module 502 is specifically used to determine, for each request, the predicted remaining output length based on the maximum output length and the target current output length corresponding to the saved large model inference node if the target current output length corresponding to the request is not less than the average output length; if the target current output length corresponding to the request is less than the average output length, the predicted remaining output length is determined based on the target current output length and the average output length.

[0158] In a possible implementation, the load balancing module 502 is also used to obtain the saved CPU utilization, GPU utilization and memory utilization of each large model inference node for each large model inference node; and determine the resource utilization based on the CPU utilization, GPU utilization and memory utilization.

[0159] In a possible implementation, the load balancing module 502 is also used to determine, for each large model inference node, the first average token generation time and throughput of the large model inference node saved, and to determine the second average token generation time and average throughput of each large model inference node saved; determine the third proportion of the token generation time of the large model inference node in each large model inference node based on the first average token generation time and the second average token generation time; determine the fourth proportion of the throughput of the large model inference node in each large model inference node based on the throughput and the average throughput; determine the processing capacity parameter based on the third proportion and the fourth proportion; wherein, the larger the third proportion, the smaller the processing energy parameter; and the larger the fourth proportion, the larger the processing capacity parameter.

[0160] In a possible implementation, the load balancing module 502 is also used to determine whether there is an idle large model inference node with no load at present according to the real-time load parameters of each large model inference node; if so, the inference request is distributed to the idle large model inference node; if not, a subsequent process of determining the load index corresponding to each large model inference node according to the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node is executed.

[0161] In a possible implementation, the load balancing module 502 is also used to obtain the latest sending time of the node information of each saved candidate large model inference node; wherein the node information includes an access path and a node identifier; determine each target candidate large model inference node whose time difference between the corresponding latest sending time and the current time is within a preset difference range; and determine each target large model inference node as each large model inference node.

[0162] In a possible implementation, the distribution module 503 is specifically used to obtain the saved target access path corresponding to the target large model inference node, and distribute the inference request to the target large model inference node through the target access path.

[0163] Based on the above embodiments, the present application also provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, it includes: a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604;

[0164] The memory 603 stores a computer program. When the program is executed by the processor 601, the processor 601 executes the steps of the load balancing method for large models provided in the above embodiments.

[0165] Since the principle of solving the problem by the above electronic device is similar to the load balancing method for large models, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0166] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 602 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0168] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the load balancing method for large models provided in the above embodiments.

[0169] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the load balancing method for large models, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A load balancing method for large models, characterized in that: The method comprises: Obtaining operating parameters of each large model inference node where the large model is deployed; wherein the operating parameters are determined according to relevant process parameters and monitoring indicators in the large model inference process, including real-time load parameters, predicted load parameters, resource utilization, and processing capacity parameters; Determine the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization rate and processing capacity parameter corresponding to each large model inference node; wherein the larger the predicted load parameter and the real-time load parameter, the larger the load index; the larger the resource utilization rate and the processing capacity parameter, the smaller the load index; The inference request to be processed is distributed to the target large model inference node with the smallest load index, so that the large model deployed in the target large model inference node responds to the inference request.

2. The method according to claim 1, characterized in that The process of determining the real-time load parameters includes: For each large model inference node, a first proportion of the number of requests in the maximum number of requests is determined based on the number of requests contained in each queue in the large model inference node and the corresponding maximum number of requests allowed to be contained; a second proportion of the storage space value in the maximum storage space value is determined based on the currently occupied storage space value of the large model inference node and the maximum storage space value allowed to be occupied; the real-time load parameter is determined based on the forward transfer token rate of the large model deployed in the large model inference node, the first proportion and the second proportion; wherein each queue includes a waiting queue, a preemptive queue and a running queue, and the larger the first proportion, the second proportion and the forward transfer token rate, the larger the real-time load parameter.

3. The method according to claim 1, characterized in that The process of determining the predicted load parameters includes: For each large model inference node, obtain the current output length of each request being processed by the large model inference node and the average output length of the large model inference node; determine the predicted remaining output length corresponding to each request based on the relationship between the current output length corresponding to each request and the average output length; determine the predicted load parameter based on the predicted remaining output length corresponding to each request and the number of each request.

4. The method according to claim 3, characterized in that Determining the predicted remaining output length corresponding to each request according to the size relationship between the current output length corresponding to each request and the average output length includes: For each request, if the target current output length corresponding to the request is not less than the average output length, the predicted remaining output length is determined based on the maximum output length corresponding to the saved large model inference node and the target current output length; if the target current output length corresponding to the request is less than the average output length, the predicted remaining output length is determined based on the target current output length and the average output length.

5. The method according to claim 1, characterized in that The process of determining the resource utilization rate includes: For each large model inference node, obtain the saved CPU utilization, GPU utilization and memory utilization of the large model inference node; determine the resource utilization based on the CPU utilization, GPU utilization and memory utilization.

6. The method according to claim 1, characterized in that The process of determining the processing capacity parameter includes: For each large model inference node, determine the first average token generation time and throughput of the large model inference node that are saved, and determine the second average token generation time and average throughput of each large model inference node that are saved; determine the third proportion of the token generation time of the large model inference node in each large model inference node based on the first average token generation time and the second average token generation time; determine the fourth proportion of the throughput of the large model inference node in each large model inference node based on the throughput and the average throughput; determine the processing capacity parameter based on the third proportion and the fourth proportion; wherein, the larger the third proportion, the smaller the processing energy parameter; the larger the fourth proportion, the larger the processing capacity parameter.

7. The method according to claim 1, characterized in that Before determining the load index corresponding to each large model inference node according to the predicted load parameter, real-time load parameter, resource utilization rate and processing capacity parameter corresponding to each large model inference node, the method further includes: According to the real-time load parameters of each large model inference node, determining whether there is an idle large model inference node with no load at present; If so, the inference request is distributed to the idle large model inference node; If it does not exist, a subsequent process is performed to determine the load index corresponding to each large model inference node based on the predicted load parameters, real-time load parameters, resource utilization and processing capacity parameters corresponding to each large model inference node.

8. The method according to claim 1, characterized in that The determination process of each large model reasoning node includes: Obtain the latest sending time of the node information of each saved candidate large model inference node; wherein the node information includes an access path and a node identifier; Determine each target candidate large model inference node whose time difference between the corresponding latest sending time and the current time is within a preset difference range; Each target large model inference node is determined as each large model inference node.

9. The method according to claim 1 or 8, characterized in that: The distributing the inference request to the target large model inference node with the smallest load index comprises: The saved target access path corresponding to the target large model inference node is obtained, and the inference request is distributed to the target large model inference node through the target access path.

10. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the large model-oriented load balancing method as claimed in any one of claims 1 to 9 when executing a computer program stored in the memory.

Images