Load balancing method of heterogeneous computing power cluster based on large model and related device

Abstract

The invention provides a load balancing method of a heterogeneous computing power cluster based on a large model and a related device, and relates to the technical field of computers, in particular to the technical fields of large models, artificial intelligence, load balancing and the like. According to the specific implementation scheme, the average level range of expected indexes of the heterogeneous computing power cluster based on the large model is determined; based on a comparison result of the expected indexes of the plurality of to-be-optimized model instances and the average level range, determining a target model instance meeting a control condition in the plurality of to-be-optimized model instances; the management and control conditions are used for screening out to-be-optimized model instances of which expected indexes need to be reduced; and in response to the task acquisition request of the target model instance, controlling the task acquisition speed of the target model instance.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of large models, artificial intelligence, and load balancing. Background Technology

[0002] With the continuous development of artificial intelligence technology, the reasoning capabilities of large models have been improved and perfected, and the application of large models in various fields is becoming increasingly widespread. Large model inference systems in related technologies are often deployed in heterogeneous computing clusters composed of various types of hardware accelerator cards. To ensure the efficient and stable operation of large model inference tasks, it is necessary to rationally allocate tasks through load balancing. Summary of the Invention

[0003] This disclosure provides a load balancing method and related apparatus for heterogeneous computing power clusters based on large models.

[0004] According to one aspect of this disclosure, a load balancing method for heterogeneous computing power clusters based on large models is provided, comprising: Determine the average range of expected metrics for heterogeneous computing power clusters based on large models; Based on the comparison results of the expected indicators and the average level range of multiple model instances to be optimized, the target model instances that meet the control conditions are identified among the multiple model instances to be optimized; the control conditions are used to filter out the model instances to be optimized that need to reduce the expected indicators. In response to a task retrieval request from a target model instance, control the task retrieval speed of the target model instance.

[0005] According to another aspect of this disclosure, a load balancing device for a heterogeneous computing power cluster based on a large model is provided, comprising: The first determining module is used to determine the average level range of expected indicators for heterogeneous computing power clusters based on large models. The second determination module is used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized based on the comparison results of the expected indicators and the average level range of multiple model instances to be optimized; the control conditions are used to filter out the model instances to be optimized that need to reduce the expected indicators. The control module is used to respond to the task retrieval request of the target model instance and control the task retrieval speed of the target model instance.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a load balancing method for a heterogeneous computing power cluster based on a large model according to an embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the determination of a target model instance that meets the control conditions among a plurality of model instances to be optimized, according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram illustrating another method for determining a target model instance that meets the control conditions among a plurality of model instances to be optimized, according to an embodiment of the present disclosure. Figure 4 This is a schematic diagram illustrating another method for determining a target model instance that meets the control conditions among a plurality of model instances to be optimized, according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the task acquisition speed of a target model instance according to an embodiment of the present disclosure; Figure 6 A schematic diagram of the overall process structure of a load balancing method for heterogeneous computing power clusters based on a large model according to an embodiment of this disclosure. Figure 7 A schematic diagram of the structure of a load balancing device for a heterogeneous computing power cluster based on a large model according to an embodiment of the present disclosure; Figure 8 This is a block diagram of an electronic device used to implement the load balancing method for a heterogeneous computing power cluster based on a large model according to embodiments of this disclosure. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0013] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0014] With the rapid development of artificial intelligence technology, inference tasks are being used more and more widely in various fields. In order to complete these inference tasks efficiently, reasonable load balancing is crucial.

[0015] However, load balancing methods for inference tasks in related technologies have many limitations. For example: Related Method 1: In static weight-based task allocation methods, some large-model inference systems statically allocate tasks to different hardware accelerator cards (such as GPUs (Graphics Processing Units)) during deployment based on their theoretical computing power or preset weights. This method is simple to implement but cannot adapt to load changes during inference. For example, when some hardware accelerator cards experience task backlog, some large-model inference systems cannot adjust their allocation strategies in a timely manner, leading to a performance degradation of the entire large-model inference system.

[0016] Related Method 2: In round-robin or random uniform allocation methods, common scheduling approaches include using round-robin, random allocation, or consistent hashing strategies to evenly distribute tasks across different hardware accelerator cards in a large model inference system. This approach does not consider the performance differences and real-time status of different hardware accelerator cards, easily leading to situations where high-performance cards are idle while low-performance cards are overloaded.

[0017] Therefore, while Method 1 considers the performance differences between different hardware accelerator cards, it only relies on their theoretical computing power, i.e., ideal conditions, for load balancing. Method 2, on the other hand, does not consider the differences between different hardware accelerator cards.

[0018] In view of this, the present disclosure provides a novel load balancing method for heterogeneous computing power clusters based on large models.

[0019] The heterogeneous computing power cluster based on a large model, also known as a multi-model inference system, provided in this disclosure, is a computing cluster system composed of various types of computing resources. These computing resources primarily include GPUs. Different types of computing resources possess varying computing capabilities, storage capacities, and energy consumption characteristics. Multiple model instances are deployed within this heterogeneous computing power cluster. Each model instance supports different computing resources or different neural network model architectures to adapt to diverse task requirements. Each model instance is responsible for running the entire large model or for running a portion of the large model's inference functionality. Each model instance in this heterogeneous computing power cluster requires at least one GPU card to run. The multiple GPU cards relied upon by the same model instance can have different performance levels.

[0020] This disclosure provides a load balancing method for a heterogeneous computing power cluster based on a large model. The method is executed by a device responsible for load balancing within the heterogeneous computing power cluster. This load balancing device can be any electronic device within the heterogeneous computing power cluster that meets set conditions. Specific set conditions can be configured according to actual needs, and this disclosure does not limit their application.

[0021] like Figure 1 The diagram shown is a flowchart of the method, including the following: S101, determine the average range of expected metrics for heterogeneous computing power clusters based on large models.

[0022] Users can send tasks to be processed to the heterogeneous computing power cluster via terminal devices. These tasks are then distributed to the corresponding model instances to be optimized for execution using the load balancing method described in this embodiment.

[0023] The expected metrics for this heterogeneous computing cluster are used to measure the response speed of the cluster to the tasks to be processed, and / or to measure the load on the computing resources of the cluster. Resource utilization is used to balance the load on computing resources occupied by different model instances, thereby improving the overall system performance. In this embodiment, load balancing is used to adjust the average response speed of the heterogeneous computing cluster to the tasks to be processed, and / or to balance the load on computing resources of different model instances to be optimized.

[0024] The average range of the expected indicators of the heterogeneous computing power cluster represents the average level of the expected indicators of all model instances to be optimized in the heterogeneous computing power cluster, and can be used as a reference standard for subsequent judgment on whether each model instance needs to be adjusted.

[0025] S102, Based on the comparison results of the expected indicators and the average level range of multiple model instances to be optimized, determine the target model instance that meets the control conditions among the multiple model instances to be optimized; the control conditions are used to screen out the model instances to be optimized that need to reduce the expected indicators. Among them, multiple model instances to be optimized refer to all model instances in the heterogeneous computing power cluster that need to be load balanced and can operate normally.

[0026] The expected metrics of the model instance to be optimized, used to describe the response speed of the model instance to be optimized, and / or the resource utilization of the computing resources on which the model instance to be optimized depends.

[0027] The control conditions are rules set based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range. Different control conditions can be used to select target model instances that need to be optimized for different comparison results. The control conditions corresponding to different comparison results all aim to select target model instances whose expected indicators need to be reduced, so that the target model instance meets the following expected state: after the operation of step S103, or even after multiple iterative executions of the load balancing method provided in this embodiment (such as S101-S103), the expected indicator of the target model instance can be improved (mainly reduced). For example, if the expected indicator of the target model instance was originally higher than the upper limit of the average level range, after adjustment, the expected indicator of the target model instance is reduced to the average level range, thereby achieving the purpose of load balancing for different model instances to be optimized. Therefore, the control conditions corresponding to different comparison results can be adjusted according to specific application scenarios and business needs, and this embodiment does not limit this.

[0028] S103, responding to the task acquisition request of the target model instance, controls the task acquisition speed of the target model instance.

[0029] Among them, the target acquisition request is used to obtain the task to be executed by the target model instance, that is, the task to be processed.

[0030] During implementation, since the target model instance meets the control conditions, the amount of tasks that the target model instance needs to process is controlled by controlling its task acquisition speed, thereby adjusting the load balance of the target model instance.

[0031] In this embodiment, a heterogeneous computing power cluster based on a large model and each instance of the model to be optimized are compared using their respective expected indicators, enabling a quantitative measurement of the load status of each instance. Based on the comparison results of the expected indicators of multiple instances of the model to be optimized with the average level range, the actual operation of different instance models can be understood, and the load differences between the computing resources of different instance models can be measured or reflected, thereby identifying target instance models that meet the control conditions for control. During control, target instance models whose expected indicators need to be reduced are selected, and their task acquisition speed is controlled. The task acquisition speed can be dynamically adjusted according to the actual state of the target instance models, achieving load balancing under the guidance of expected indicators. In summary, compared to related method 1, this embodiment can achieve load balancing based on the actual situation (i.e., expected indicators) of different instance models to be optimized; compared to related method 2, this embodiment can consider the differences between different instance models to be optimized, i.e., the differences between different hardware.

[0032] In this embodiment of the disclosure, for the heterogeneous computing power cluster and the model instance to be optimized, the respective expected indicators are used to measure the response speed to the task to be processed, and / or, the expected indicators are used to measure the load on computing resources.

[0033] When using this expected metric to measure response speed, TTFT (Time To First Token) can be used as the expected metric. Response speed also reflects the load balancing of different instance models to be optimized.

[0034] When measuring response speed, the expected metrics should be based on the response time of different users' pending tasks. This allows for adjustments to the load balancing of different model instances to be optimized in a heterogeneous computing cluster, facilitating a more balanced distribution of tasks across model instances in a heterogeneous environment.

[0035] This expected metric is used to measure the load on computing resources. It can dynamically adjust the load balancing between different model instances to be optimized based on their actual load, thereby improving the overall performance of the system.

[0036] In this embodiment of the disclosure, the expected index is obtained based on at least one of the following statistical information: (1) Concurrency of each instance of the model to be optimized; Concurrency is calculated individually for each instance of the model to be optimized. Concurrency refers to the number of tasks that each instance of the model to be optimized can process simultaneously within the same time period, reflecting its parallel processing capability. Higher concurrency means that the instance of the model to be optimized can process more tasks per unit of time, and correspondingly, the performance of its configured computing resources is higher.

[0037] (2) Memory utilization of each model instance to be optimized; Memory utilization is calculated individually for each instance of the model to be optimized. Memory is the memory space used by the graphics processor to store data and programs. Memory utilization represents the proportion of memory actually used by the instance of the model to be optimized within a single statistical period. If the memory utilization is too high, it may lead to insufficient memory, affecting the normal operation of the model; conversely, if the memory utilization is too low, it indicates that memory resources are not being fully utilized.

[0038] (3) Resource utilization of the graphics processor for each instance of the model to be optimized; Resource utilization is calculated individually for each model instance to be optimized. The graphics processor (GPU) resource utilization reflects the target model instance's use of the overall GPU resources, comprehensively considering factors such as GPU computing power and data transfer capabilities. High resource utilization indicates that all GPU resources are being fully utilized.

[0039] (4) First word generation time, which is used to represent the time required from the submission of the task to be processed by the target object to the generation of the first word based on the large model; The first word generation time refers to the time elapsed from when the target object (such as a user) submits a task to when the large model obtains the first word. For example, in a natural language processing scenario, when a user inputs a question, the large model needs to process the question and generate an answer. The first word generation time is the time from when the user inputs the question to when the large model outputs the first word. The first word generation time reflects the response speed of the large model; a shorter first word generation time allows the user to receive feedback faster.

[0040] For each model instance to be optimized, assuming that the model instance can run the large model completely, the time taken to generate the first word unit is calculated. The average TTFT within each statistical period can be used as the first word unit generation time.

[0041] For each model instance to be optimized, if the model instance to be optimized only performs a part of the functions of the large model, the multiple model instances to be optimized that execute the entire large model can be regarded as a group of instances. The average TTFT of the group of instances is calculated as the first word generation time of each model instance to be optimized in the group.

[0042] (5) Task execution time, which is used to indicate the time required for the task to be processed to be executed by the large model until the task is completed.

[0043] Task execution time refers to the total time required from when the large model begins executing the task until the task is completed. It includes the time spent from when the large model receives the task, through a series of calculations and processing, until the final result is output.

[0044] For each model instance to be optimized, assuming it can run the larger model completely, calculate the task execution time of that model instance. The average task execution time over each statistical period can be used to determine the task execution time required to achieve the desired metric.

[0045] For each model instance to be optimized, if the model instance to be optimized only executes a part of the functions of the large model, the multiple model instances to be optimized that execute the entire large model can be regarded as a group of instances. The average task execution time of the group of instances is calculated as the task execution time of each model instance to be optimized in the group.

[0046] In this embodiment of the disclosure, in the load balancing method for heterogeneous computing power clusters based on large models, at least one of the following statistical information—concurrency, memory utilization, graphics processor resource utilization, first-word generation time, and task execution time—is used as the expected indicator for the granular statistics of the model instance to be optimized. When using a single statistical information, the load balancing of the heterogeneous computing power cluster can be optimized based on the dimension of that statistical information. When using multiple statistical information, the load and performance of the model instance to be optimized can be comprehensively understood, thereby identifying the target model instance that needs adjustment, achieving reasonable task allocation, and load balancing of the heterogeneous computing power cluster based on large models.

[0047] Among them, at least one of concurrency, memory utilization, and resource utilization can enable the desired metric to be used to measure the load on computing resources.

[0048] At least one of the time taken to generate the first word and the time taken to execute the task can make the expected metric a measure of response speed.

[0049] In cases involving multiple statistical information, each statistical information can be used as a desired indicator, and each desired indicator can be adjusted individually based on the method provided in the embodiments of this disclosure until all desired indicators reach a desired state position.

[0050] In other embodiments, when multiple statistical information is involved, a weighted summation method can be used to determine the final expected indicator. The weights can be allocated according to the indicators emphasized by the business, and this disclosure does not limit this approach.

[0051] In this embodiment, because the load of each model instance to be optimized is adjusted in real time, the expected metrics of each model instance to be optimized will change. To dynamically and adaptively achieve load balancing based on actual conditions, in this embodiment, the average level range is updated periodically, and the average level range is determined based on the mean and variance of the expected metrics of multiple model instances to be optimized in the previous statistical period. That is, the average level range obtained in the previous statistical period will affect the load balancing of each model instance to be optimized in the current statistical period.

[0052] In implementation, a module (such as a thread) can periodically collect the expected metrics of each instance of the model to be optimized according to a statistical period. Then, after summing the expected metrics of all instances of the model to be optimized, its mean and variance are calculated. A reasonable average level range is determined based on this mean and variance. In implementation, the average level range described by the mean and variance can be determined through load testing.

[0053] The mean of the expected metrics for multiple model instances to be optimized is obtained by averaging the expected metrics of the multiple model instances to be optimized. For example, the mean of the expected metrics for all model instances to be optimized can be obtained by adding up the expected metrics of all model instances to be optimized and then dividing by the number of model instances to be optimized.

[0054] The variance of the expected metric across multiple instances of the model to be optimized measures the degree of dispersion of the expected metric values ​​relative to the mean. To calculate the variance, first calculate the square of the difference between each expected metric value and the mean, and then calculate the average of these squared values.

[0055] The following expression (1) is an example of this average level range: Avg=(μ±nσ) (1) In expression (1), Avg represents the average level range; μ represents the mean of the expected indicators of multiple model instances to be optimized; σ represents the variance of the expected indicators of multiple model instances to be optimized; and n is a parameter, which is obtained based on experimental data, for example, by stress testing different model instances to be optimized.

[0056] In this embodiment, periodic updates ensure that the average level range dynamically changes to keep pace with the real-time operating status and task load changes of the heterogeneous computing power cluster, avoiding the inability to adapt to actual conditions due to static settings. Determining this average level range using the mean and variance of the desired indicators comprehensively considers the overall characteristics and dispersion of multiple model instances to be optimized, reflecting the actual performance of the model instances to be optimized in the heterogeneous computing power cluster based on a large model. This provides a more reliable reference for subsequently determining target model instances that meet control conditions, making the control of the task acquisition speed of target model instances more aligned with actual needs, and ultimately achieving more efficient and reasonable load balancing.

[0057] In this embodiment of the disclosure, based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range, the target model instance that meets the control conditions among the multiple model instances to be optimized may include the following: (1) When the expected indicators of multiple model instances to be optimized are all within the average level range, the model instances to be optimized with expected indicators higher than the first reference value are determined as target model instances that meet the control conditions; the difference between the first reference value and the median of the average level range is less than the first preset difference, and the first reference value is greater than the lower limit of the average level range.

[0058] The median of the average level range refers to the mean of that average level range. When the average level range is expressed using expression (1), the median is the mean μ.

[0059] For example, such as Figure 2 As shown, when the expected metrics of multiple model instances to be optimized are all within the average level range, the difference between the first reference value and the median of the average level range is less than the first preset difference, and the first reference value is greater than the lower limit of the average level range. The first preset difference is a value pre-set based on the actual situation of the cluster and the load balancing strategy.

[0060] During implementation, model instances whose expected metrics are higher than the first reference value can be identified as target model instances that meet the control conditions. This means that even if the expected metrics of all model instances are within the average range, those model instances whose expected metrics are higher than the first reference value may be at risk of further load increase, or their expected metrics may be close to the upper limit, and therefore need to be controlled.

[0061] In certain special cases, where the average expected metric of multiple model instances to be optimized exceeds a first preset value, it indicates that each model instance in the heterogeneous computing power cluster is operating at near full capacity or has a high load. Therefore, these multiple model instances will not be subject to control. For example, if the average expected metric of all model instances to be optimized is greater than 80%, it means that each model instance in the heterogeneous computing power cluster is operating at near full capacity, and there is no significant uneven load or idle resources among the computing resources of the model instances (e.g., σ is less than a second preset value). In such cases, these multiple model instances will not be subject to control.

[0062] In this embodiment, when the average of the expected indicators of multiple model instances to be optimized is within the average level range, the model instances whose expected indicators are higher than a first reference value are set to meet the control conditions, and the difference between the first reference value and the average level range is less than a first preset difference and greater than a lower limit. This allows for the further identification of target model instances that may have a potential risk of overload when the overall performance of the model instances to be optimized is within a relatively efficient and balanced range, avoiding localized overload problems that cannot be detected solely based on the average level range.

[0063] (2) If there is at least one model instance to be optimized with an expected indicator that is lower than the lower limit of the average level range, and there is no model instance to be optimized with an expected indicator that exceeds the upper limit of the average level range, the model to be optimized with an expected indicator that is within the average level range shall be determined as the target model instance that meets the control conditions.

[0064] For example, such as Figure 3 As shown, when there is at least one model instance to be optimized with an expected metric lower than the lower limit of the average range, and no model instance to be optimized with an expected metric higher than the upper limit of the average range, the model to be optimized with an expected metric within the average range is controlled in order to reduce its expected metric by controlling its task acquisition speed, thereby achieving the purpose of load balancing.

[0065] For example, by expanding capacity, a new instance of a model to be optimized is added. The expected metrics of this new instance will be smaller. In this way, the tasks to be processed can be adjusted as quickly as possible, and the corresponding tasks can be allocated to the new instance of the model to be optimized to achieve load balancing as soon as possible.

[0066] In this embodiment of the disclosure, if there is at least one model instance to be optimized whose expected index is below the lower limit of the average level range, it means that the model instance to be optimized may be in an idle or low-load state. At this time, identifying the model instance to be optimized that is within the average level range as the target model instance that meets the control conditions allows for reasonable control of the target model instance, thereby optimizing the task allocation of the entire heterogeneous computing power cluster based on the large model, improving the resource utilization of idle model instances to be optimized, and making the load of the heterogeneous computing power cluster based on the large model more balanced.

[0067] (3) If there is at least one model instance to be optimized with an expected index lower than the lower limit of the average range and at least one model instance to be optimized with an expected index higher than the upper limit of the average range, the model to be optimized with an expected index higher than the second reference value is determined as the target model instance that meets the control conditions; the second reference value is within the average range.

[0068] During implementation, the second reference value can be smaller than the first reference value to increase the range of target model instances that need to be adjusted and optimized, so as to achieve the purpose of load balancing as soon as possible.

[0069] For example, such as Figure 4 As shown, if there is at least one model instance to be optimized with an expected index lower than the lower limit of the average range, and at least one model instance to be optimized with an expected index higher than the upper limit of the average range, the model to be optimized with an expected index higher than the second reference value will be controlled.

[0070] For example, by expanding capacity, a new model instance to be optimized is added. The expected metrics of the newly added model instance to be optimized will be smaller. In this way, the tasks to be processed can be adjusted as soon as possible, the burden on the high-load target model instance can be reduced, and the corresponding tasks to be processed can be allocated to the newly added model instance to be optimized, so as to achieve the goal of load balancing as soon as possible.

[0071] In this embodiment of the disclosure, by determining that the model to be optimized, whose expected index is higher than the second reference value, meets the control conditions, it is possible to identify model instances that, although within the average range, have relatively high loads and are at risk of overload. By controlling these model instances, the task load within the cluster can be effectively balanced, preventing high-load model instances from further overloading and causing performance degradation.

[0072] In this embodiment of the disclosure, in a heterogeneous computing power cluster based on a large model, in order to avoid some model instances from being overloaded due to the task acquisition speed being too fast, while allowing other model instances to have the opportunity to acquire tasks and make full use of resources, the task acquisition speed of the target model instance can be controlled.

[0073] During implementation, the method for controlling the task retrieval speed of the target model instance in response to the task retrieval request is as follows: Figure 5 As shown, it includes: S501, in response to the task retrieval request for the target model instance, retrieves the pre-determined delay duration for the target model instance.

[0074] S502, if the delay duration is delayed, send the pending task to the target model instance.

[0075] Upon receiving a task retrieval request from a target model instance, the instance is not immediately assigned the necessary tasks. Instead, it waits for a predetermined delay. During this delay, the target model instance remains in a waiting state and cannot acquire new tasks. Only after the delay has elapsed will the tasks be sent to the target model instance for execution.

[0076] Specifically, the delay duration can be defined as the sleep duration for sending the pending task to the target model instance. When a task retrieval request is received from the target model instance, the pending task is sent to the target model instance after the sleep duration begins from the time the task retrieval request is received.

[0077] In this embodiment of the disclosure, by responding to the task acquisition request of the target model instance, a pre-determined delay time is obtained, and the task to be processed is sent to the target model instance after the delay time. This is to regulate the time allocation of tasks, avoid the target model instance receiving tasks in a concentrated period of time, which would lead to excessive processing pressure, achieve more reasonable task scheduling, and thus effectively balance the task load in the heterogeneous computing power cluster based on the large model.

[0078] It is understood that the target model instance is based on the task acquisition request, and the number of tasks acquired can be determined according to the business situation. This disclosure embodiment does not limit this.

[0079] In this embodiment of the disclosure, the delay duration of the target model instance can be dynamically determined. In practice, the delay duration of the target model instance can be determined based on a comparison of the expected metrics of multiple model instances to be optimized with the average level range. The delay duration of the target model instance is used to ensure that the expected metrics of the target model instance in the next statistical period approach the average level range relative to the previous statistical period.

[0080] For example, if the expected metric of the target model instance is within the average range, its latency may be fine-tuned based on its relative position to the mean μ.

[0081] If the expected metrics of the target model instance exceed the upper limit of the average range, it indicates that the model instance is under high load. To reduce its load, a relatively long latency period can be determined.

[0082] If the expected metrics of the target model instance are below the lower limit of the average range, it indicates that the model instance is under low load and resources are not being fully utilized.

[0083] In this embodiment, a suitable latency is determined for the target model instance based on a comparison of the expected metrics of multiple model instances to be optimized with the average level range. This latency can regulate the rhythm of task reception and processing of the target model instance, causing the expected metrics of the target model instance to gradually approach the average level range, thereby optimizing the load balancing of the heterogeneous computing power cluster of the large model.

[0084] In implementation, for the target model instance determined under the above different conditions, the implementation method for determining the latency of the target model instance based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range may include the following: (1) When the expected indicators of multiple model instances to be optimized are all within the average level range and the lower limit of the average level range is lower than the preset threshold, the delay duration of the target model instance is generated based on the first constraint strategy; wherein, the first constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the first cutoff value; the first cutoff value is determined based on the difference between the expected indicators of the target model instance and the first reference value.

[0085] The following expression (2) is an example of the delay time for generating the target model instance based on the first constraint strategy: T1=sleep((x-μ) / σ*N) (2) In expression (2), T1 represents the delay duration for generating the target model instance based on the first constraint strategy; sleep() represents the first constraint strategy; x represents the expected index of the target model instance; μ represents the first reference value; x-μ represents the first cutoff value; σ represents the variance of the expected index of multiple model instances to be optimized; N represents the parameter that can transform the first cutoff value into the time space of the executable delay duration of the first constraint strategy.

[0086] Here, N represents a transformation parameter with physical meaning, used to transform a value determined based on the desired index into a physical meaning space representing duration. The value of N can be determined through stress testing on different instances of the model to be optimized, and this disclosure does not limit this.

[0087] The first constraint strategy controls the latency of the target model instance to have a positive correlation with the first cutoff value. That is, the larger the first cutoff value, the longer the latency of the target model instance; the smaller the first cutoff value, the shorter the latency of the target model instance.

[0088] In this embodiment of the disclosure, when the expected metrics of multiple model instances to be optimized are all within the average level range, and the lower limit of the average level range is lower than a preset threshold, the latency of the target model instance is generated based on a first constraint strategy. By determining a first cutoff value based on the difference between the expected metrics of the target model instance and a first reference value, and making the latency positively correlated with the first cutoff value, the timing of the target model instance receiving tasks to be processed can be accurately controlled, thereby effectively balancing the task load within the heterogeneous computing power cluster based on a large model.

[0089] (2) If there is at least one model instance to be optimized with an expected index lower than the lower limit of the average range, and there is no model instance to be optimized with an expected index higher than the upper limit of the average range, the delay duration of the target model instance is generated based on the second constraint strategy; wherein, the second constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the second cutoff value; the second cutoff value is determined based on the gap between the expected index of the target model instance and the third reference value; the gap between the third reference value and the lower limit of the average range is less than the second preset gap.

[0090] The following expression (3) is an example of the delay time for generating the target model instance based on the second constraint strategy: T2=sleep((xA) / σ*N) (3) In expression (3), T2 represents the delay duration for generating the target model instance based on the second constraint strategy; sleep() represents the second constraint strategy; x represents the expected index of the target model instance; A represents the third reference value; xA represents the second cutoff value; σ represents the variance of the expected index of multiple model instances to be optimized; N represents the parameter that can transform the second cutoff value into the duration space of the executable delay duration of the second constraint strategy. It is used to transform the value determined based on the expected index into the physical meaning space representing the duration. The value of N can be determined by stress testing different model instances to be optimized, and this embodiment of the present disclosure does not limit this.

[0091] In this embodiment, a second cutoff value is obtained by calculating the difference between the expected index of the target model instance and the third reference value, and the delay duration is positively correlated with the second cutoff value. This allows for reasonable adjustment of the task reception time of target model instances within the average range, enabling reasonable control over the target model instance, thereby optimizing the task allocation of the entire heterogeneous computing power cluster based on the large model, improving resource utilization, and making the load of the heterogeneous computing power cluster based on the large model more balanced.

[0092] (3) If there is at least one model instance to be optimized with an expected index lower than the lower limit of the average range and at least one model instance to be optimized with an expected index higher than the upper limit of the average range, the delay duration of the target model instance is generated based on the third constraint strategy; wherein, the third constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the third cutoff value; the third cutoff value is determined based on the gap between the expected index of the target model instance and the fourth reference value; the gap between the fourth reference value and the lower limit of the average range is less than the third preset gap.

[0093] The following expression (4) is an example of the delay time for generating target model instances based on the third constraint strategy: T3=sleep((xB) / σ*N) (4) In expression (4), T3 represents the delay duration for generating the target model instance based on the third constraint strategy; sleep() represents the third constraint strategy; x represents the expected index of the target model instance; B represents the fourth reference value, which can be the same as the third reference value in expression (3); xB represents the third cutoff value; σ represents the variance of the expected index of multiple model instances to be optimized; N represents the parameter that can convert the third cutoff value to the duration space of the executable delay duration of the third constraint strategy. It is used to convert the value determined based on the expected index to the physical meaning space representing the duration. The value of N can be determined by stress testing different model instances to be optimized, and this embodiment does not limit this.

[0094] In this embodiment, a third cutoff value is determined based on the difference between the expected metric of the target model instance and a fourth reference value where the difference between the expected metric and the average level is less than a third preset difference. This makes the latency positively correlated with the third cutoff value. This allows for accurate control over the time the target model instance receives tasks, reducing the deviation between the expected metric and the average level, balancing the performance of each target model instance, and preventing some instances from performing too poorly or too well. This effectively balances the task load within a heterogeneous computing cluster based on a large model.

[0095] In summary, the overall process of the load balancing method for heterogeneous computing power clusters based on large models provided in this disclosure is as follows: Figure 6 As shown.

[0096] Figure 6 The diagram shows a partial structural schematic of a heterogeneous computing power cluster based on a large model. It includes components such as: client, queue service, IC (inference-controller) instance, model instance to be optimized, and puller module.

[0097] The queue service serves as the entry point for the entire large-scale model inference system, responsible for receiving requests from external users (such as pending tasks). It buffers and manages these requests, ensuring the system can efficiently handle high-concurrency requests. A queue service contains multiple IC instances, and the queue service is aware in real-time of all request inflows, processing, and the status of each model instance to be optimized. As an intermediary component between the queue service and the model instances, the IC instance is responsible for retrieving requests from the queue service and distributing them to the appropriate model instance for processing. Furthermore, the IC instance employs a multi-replica design, meaning one IC instance can correspond to one or more model instances to be optimized.

[0098] The model instance to be optimized is the component that actually performs the task (such as inference or computation). It can retrieve requests from the IC instance and return the processing results. The model instance to be optimized can be heterogeneous, supporting different hardware or different model architectures to adapt to diverse task requirements.

[0099] The Puller module is a component deployed on the model instance to be optimized, with one Puller module corresponding to each model instance. The Puller module can periodically collect the running status of the hardware (such as GPU) in the corresponding model instance to be optimized. This running status can include metrics such as utilization, memory usage, task execution time, and concurrency. The IC instance achieves load balancing by controlling the task acquisition speed of the target model instance.

[0100] Specifically, the queue service can collect the TTFT of pending requests for each model instance to be optimized. It can also obtain the utilization rate, memory usage, task execution time, and concurrency parameters of each model instance through the Puller module. Finally, based on these parameters, it generates the expected metrics for each model instance to be optimized, and the average range of the heterogeneous computing cluster. The expected metrics and the average range for each model instance to be optimized can be calculated by each IC instance, or synchronized to each IC instance by the storage middleware (Redis).

[0101] For each IC instance, based on the expected metrics of at least one model instance to be optimized and the average level range, the load balancing method for heterogeneous computing power clusters based on large models provided in this disclosure is executed.

[0102] Specifically, in Figure 6 Architectural load balancing methods include: S601, the target object initiates a processing request for the pending task.

[0103] The target object can uniformly access the pending task requests through the API (Application Programming Interface) gateway or load balancer entry point via the client, and then enter the queue service. This queue service has high availability and persistence capabilities, which can ensure that requests are not lost from the bottom layer; at the same time, it supports a priority scheduling mechanism, which can accurately tag and sort the pending tasks entering the service queue based on information such as user level and task urgency.

[0104] S602, IC instance determines the target model instance that meets the control conditions among multiple model instances to be optimized.

[0105] During implementation, each IC instance determines the target model instance that meets the control conditions from among multiple model instances to be optimized, based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range. This has been explained above and will not be repeated here.

[0106] S603, IC instance controls the task acquisition speed of target model instances for load balancing.

[0107] Specifically, each IC instance selects target model instances that meet the control conditions based on the average level range and the expected metrics of each model instance to be optimized, and determines the latency of the target model instance. Upon receiving a task retrieval request from the target model instance, the pending task is sent to the target model instance only after the latency period is suspended.

[0108] In summary, the load balancing method for heterogeneous computing power clusters based on large models provided in this disclosure can optimize throughput by controlling the task acquisition speed of the target model instance based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range. Simultaneously, it enables online scaling up and down of heterogeneous computing power clusters based on large models and automatic adaptation to heterogeneous hardware. It can be applied to various scenarios, including but not limited to large language model inference platforms, multimodal AI (Artificial Intelligence) service platforms, intelligent search and recommendation systems, and enterprise-level AISaaS (Artificial Intelligence Software as a Service) services, which involve the mixed deployment of multiple types of GPUs / AI accelerator cards.

[0109] In heterogeneous computing power inference scenarios, the problems of load balancing methods 1 and 2 for inference tasks become more prominent, and the following shortcomings still exist: Shortcoming (1): The static weight allocation method relies on preset computing power parameters and cannot reflect the hardware operating status in real time. Different models of hardware accelerator cards have differences in video memory, bandwidth, parallel computing units, etc. It is impossible to fully measure the actual performance by simply allocating tasks based on FLOPS (Floating Point Operations Per Second) or video memory capacity.

[0110] Shortcoming (2): Load balancing methods based on round-robin, random, and other strategies do not consider the load differences of GPUs, which can easily cause some hardware accelerator cards to be overloaded and others to be idle. Although methods based on monitoring indicators can perceive the load to a certain extent, the scheduling frequency is low and they are not combined with the computing power model, resulting in scheduling lag.

[0111] Shortcoming (3): In heterogeneous environments, the execution time of the same inference request varies greatly on different hardware. The above methods lack optimization for latency-sensitive tasks (such as online interactive services), and the inference task may experience significant latency jitter due to being assigned to a low-performance hardware accelerator card.

[0112] To address the aforementioned shortcomings (1), in this embodiment, for each service instance to be optimized, dynamic indicators such as resource utilization, memory usage, TTFT, task execution time, and concurrency are periodically sensed, no longer relying on fixed FLOPS or preset memory capacity values. The IC instance in the queue service can calculate desired indicators based on these dynamic indicators to optimize the task acquisition speed of each service instance to be optimized, thereby dynamically adjusting the load of each service instance to be optimized based on actual conditions, and thus dynamically balancing the load in a dynamic manner, combined with the status of each service instance to be optimized.

[0113] To address the aforementioned shortcomings (2), this embodiment of the present disclosure can dynamically perceive the expected metrics of each model instance to be optimized, and periodically update the average level range. These expected metrics, including TTFT and task execution time, can fully consider the execution of large model computing power on different hardware, and can adjust the workload on different model instances to be optimized in a timely manner. When the scale of the inference task changes or the hardware cluster expands or shrinks, the solution provided by this embodiment of the present disclosure can dynamically adjust the workload of each model instance to be optimized by updating the expected metrics, achieving automatic convergence to the global optimum. The entire process requires no manual parameter tuning, and expansion and contraction can be flexibly supported, thereby further improving dynamism.

[0114] To address the above shortcomings (3), when a user makes an inference request, the queue service can tag the request based on the user's level and the urgency of the task (such as online interactive delay-sensitive tasks) and prioritize scheduling high-priority tasks to avoid them being assigned to low-performance hardware.

[0115] Based on the same technical concept, this disclosure also provides a load balancing device 700 for a heterogeneous computing power cluster based on a large model, such as... Figure 7 As shown, it includes: The first determining module 701 is used to determine the average level range of the expected indicators of the heterogeneous computing power cluster based on the large model; The second determining module 702 is used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized based on the comparison results of the expected indicators and the average level range of the multiple model instances to be optimized; the control conditions are used to filter out the model instances to be optimized that need to reduce the expected indicators. The control module 703 is used to control the task acquisition speed of the target model instance in response to the task acquisition request of the target model instance.

[0116] In some embodiments, the second determining module includes: The first determining unit is used to determine the model instance whose expected index is higher than the first reference value as the target model instance that meets the control conditions, when the expected indices of multiple model instances to be optimized are all within the average level range. The difference between the first reference value and the median of the average level range is less than the first preset difference, and the first reference value is greater than the lower limit of the average level range.

[0117] In some embodiments, the second determining module includes: The second determining unit is used to determine the model to be optimized whose expected index is within the average level range as the target model instance that meets the control conditions, when there is at least one model instance to be optimized whose expected index is less than the lower limit of the average level range, and there is no model instance to be optimized whose expected index exceeds the upper limit of the average level range.

[0118] In some embodiments, the second determining module includes: The third determining unit is used to determine the model to be optimized with an expected index higher than the second reference value as the target model instance that meets the control conditions when there is at least one model instance to be optimized with an expected index lower than the lower limit of the average level range and at least one model instance to be optimized with an expected index higher than the upper limit of the average level range. The second reference value is within the average range.

[0119] In some embodiments, the control module includes: The acquisition unit is used to acquire the latency duration pre-determined for the target model instance in response to the task acquisition request of the target model instance; The sending unit is used to send the task to be processed to the target model instance when the delay duration is delayed.

[0120] In some embodiments, a third determining module is further included, configured to: Based on the comparison results of the expected metrics of multiple model instances to be optimized with the average level range, the latency of the target model instance is determined. The delay duration of the target model instance is used to ensure that the expected metric of the target model instance in the next statistical period is closer to the average level range of the previous statistical period.

[0121] In some embodiments, the third determining module is specifically used for: When the expected metrics of multiple model instances to be optimized are all within the average level range and the lower limit of the average level range is lower than the preset threshold, the delay time for generating the target model instance based on the first constraint strategy is determined. The first constraint strategy is used to control the positive correlation between the delay duration of the target model instance and the first cutoff value; the first cutoff value is determined based on the gap between the expected index of the target model instance and the first reference value.

[0122] In some embodiments, the third determining module is specifically used for: The delay time for generating the target model instance based on the second constraint strategy is as follows: There is at least one model instance to be optimized whose expected index is lower than the lower limit of the average level range, and there is no model instance to be optimized whose expected index exceeds the upper limit of the average level range. The second constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the second cutoff value; the second cutoff value is determined based on the gap between the expected index of the target model instance and the third reference value; the gap between the third reference value and the lower limit of the average level range is less than the second preset gap.

[0123] In some embodiments, the third determining module is specifically used for: The delay time for generating the target model instance based on the third constraint strategy when there is at least one model instance to be optimized with an expected index lower than the lower limit of the average range and at least one model instance to be optimized with an expected index higher than the upper limit of the average range. The third constraint strategy is used to control the positive correlation between the delay duration of the target model instance and the third cutoff value; the third cutoff value is determined based on the gap between the expected index of the target model instance and the fourth reference value; the gap between the fourth reference value and the lower limit of the average level range is less than the third preset gap.

[0124] In some embodiments, the average level range is updated periodically, and the average level range is determined based on the mean and variance of the expected metrics of multiple model instances to be optimized within the previous statistical period.

[0125] In some embodiments, the desired indicator is obtained based on at least one of the following statistical information: Concurrency of each model instance to be optimized; The memory utilization rate of each model instance to be optimized; Resource utilization of the graphics processor for each instance of the model to be optimized; The first word generation time is used to represent the time required from the submission of the task to the target object to the generation of the first word based on the large model; Task execution duration is used to represent the time required from when the task to be processed is started by the large model until the task is completed.

[0126] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0127] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0128] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0129] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0131] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a load balancing method for a large-model-based heterogeneous computing cluster. For example, in some embodiments, the load balancing method for a large-model-based heterogeneous computing cluster can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the load balancing method for a large-model-based heterogeneous computing cluster described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform a load balancing method for heterogeneous computing power clusters based on large models.

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0138] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A load balancing method for heterogeneous computing power clusters based on large models, comprising: Determine the average range of expected metrics for heterogeneous computing power clusters based on large models; Based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range, the target model instance that meets the control conditions among the multiple model instances to be optimized is determined; the control conditions are used to filter out the model instances to be optimized that need to reduce the expected indicators. In response to the task acquisition request of the target model instance, control the task acquisition speed of the target model instance.

2. The method according to claim 1, wherein, The comparison results between the expected indicators of multiple model instances to be optimized and the average level range are used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized, including: If the expected indicators of the multiple model instances to be optimized are all within the average level range, the model instance to be optimized with an expected indicator higher than the first reference value is determined as the target model instance that meets the control conditions. The difference between the first reference value and the median of the average level range is less than a first preset difference, and the first reference value is greater than the lower limit of the average level range.

3. The method according to claim 2, wherein, The comparison results between the expected indicators of multiple model instances to be optimized and the average level range are used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized, including: If there is at least one model instance to be optimized with an expected indicator lower than the lower limit of the average level range, and there is no model instance to be optimized with an expected indicator higher than the upper limit of the average level range, then the model to be optimized with an expected indicator within the average level range is determined as the target model instance that satisfies the control conditions.

4. The method according to claim 2, wherein, The comparison results between the expected indicators of multiple model instances to be optimized and the average level range are used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized, including: If there is at least one model instance to be optimized with an expected index lower than the lower limit of the average level range, and at least one model instance to be optimized with an expected index higher than the upper limit of the average level range, then the model to be optimized with an expected index higher than the second reference value is determined as the target model instance that meets the control conditions. The second reference value is within the range of the average level.

5. The method according to any one of claims 1-4, wherein, The step of controlling the task acquisition speed of the target model instance in response to the task acquisition request of the target model instance includes: In response to the task acquisition request of the target model instance, obtain the delay duration pre-determined for the target model instance; If the delay duration is delayed, the task to be processed is sent to the target model instance.

6. The method according to claim 5, further comprising: Based on the comparison results of the expected metrics of multiple model instances to be optimized with the average level range, the latency of the target model instance is determined. The delay duration of the target model instance is used to ensure that the expected index of the target model instance in the next statistical period approaches the average level range relative to the previous statistical period.

7. The method according to claim 6, wherein, The determination of the latency of the target model instance based on the comparison results of the expected metrics of multiple model instances to be optimized with the average level range includes: When the expected metrics of the multiple model instances to be optimized are all within the average level range and the lower limit of the average level range is lower than a preset threshold, the delay duration of the target model instance is generated based on the first constraint strategy. The first constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the first cutoff value; the first cutoff value is determined based on the gap between the expected index of the target model instance and the first reference value.

8. The method according to claim 6, wherein, The determination of the latency of the target model instance based on the comparison results of the expected metrics of multiple model instances to be optimized with the average level range includes: If there is at least one model instance to be optimized with an expected metric lower than the lower limit of the average level range, and there is no model instance to be optimized with an expected metric higher than the upper limit of the average level range, the delay duration of the target model instance is generated based on the second constraint strategy. Wherein, the second constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the second cutoff value; the second cutoff value is determined based on the gap between the expected index of the target model instance and the third reference value; the gap between the third reference value and the lower limit of the average level range is less than the second preset gap.

9. The method according to claim 6, wherein, The determination of the latency of the target model instance based on the comparison results of the expected metrics of multiple model instances to be optimized with the average level range includes: If there is at least one model instance to be optimized with an expected metric lower than the lower limit of the average level range, and at least one model instance to be optimized with an expected metric higher than the upper limit of the average level range, the delay duration of the target model instance is generated based on the third constraint strategy. The third constraint strategy is used to control the delay duration of the target model instance to have a positive correlation with the third cutoff value; the third cutoff value is determined based on the gap between the expected index of the target model instance and the fourth reference value; the gap between the fourth reference value and the lower limit of the average level range is less than the third preset gap.

10. The method according to any one of claims 1-9, wherein, The average level range is updated periodically, and the average level range is determined based on the mean and variance of the expected indicators of the multiple model instances to be optimized in the previous statistical period.

11. The method according to any one of claims 1-9, wherein, The expected metrics are used to measure the response speed to the task to be processed, and / or, The expected metrics are used to measure the load on computing resources.

12. The method according to claim 11, wherein, The expected index is obtained based on at least one of the following statistical information: Concurrency of each model instance to be optimized; The memory utilization rate of each model instance to be optimized; Resource utilization of the graphics processor for each instance of the model to be optimized; The time taken to generate the first word element is used to represent the time required from the submission of the task to be processed by the target object to the generation of the first word element based on the large model. Task execution duration is used to represent the time required from when the task to be processed is started by the large model until the task is completed.

13. A load balancing device for a heterogeneous computing power cluster based on a large model, comprising: The first determining module is used to determine the average level range of expected indicators for heterogeneous computing power clusters based on large models. The second determining module is used to determine the target model instance that meets the control conditions among the multiple model instances to be optimized based on the comparison results of the expected indicators of multiple model instances to be optimized with the average level range; the control conditions are used to filter out the model instances to be optimized that need to reduce the expected indicators. The control module is used to control the task acquisition speed of the target model instance in response to the task acquisition request of the target model instance.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images