Large model reasoning load balancing method and device based on iterative search, equipment and storage medium

Abstract

The embodiment of the invention provides a large model reasoning load balancing method and device based on iterative search, equipment and a storage medium, and the method comprises the steps that a reasoning task set and the number of calculation units are obtained, according to the reasoning task set, determining the total calculation amount of all reasoning tasks in the reasoning task set and the maximum single task calculation amount; based on the total calculation amount of all the reasoning tasks, the maximum single task calculation amount and the number of the calculation units, the maximum distribution capacity threshold value of the calculation units and the reasoning tasks distributed to the calculation units and corresponding to the maximum distribution capacity threshold value are determined through a binary search iterative algorithm; the distributed reasoning tasks are sent to all the calculation units, so that all the calculation units execute the distributed reasoning tasks and output reasoning results, load planning is continuously optimized through global iterative search, loads among all the calculation units are better balanced, the maximum scheduling length is reduced, and idle resources are reduced.

Description

Technical Field

[0001] This application relates to the field of model inference technology, and more specifically, to a method, apparatus, device, and storage medium for load balancing of large model inference based on iterative search. Background Technology

[0002] Currently, the explosive development of artificial intelligence technology is driving the large-scale application of large-model technologies (such as GPT and LLaMA models) in fields like natural language processing, computer vision, and intelligent interaction. The industry has an urgent need to improve the performance and optimize the cost of large-model inference. Performance improvements in large-model inference systems rely on techniques such as operator computation, data scheduling, and load balancing. Breakthroughs have been made in operator-level optimization; FlashInfer, a high-performance GPU kernel library specifically designed for large language model inference, is one of the most representative optimization solutions in the industry and has been widely adopted. However, practice shows that FlashInfer and other mainstream large-model inference frameworks still suffer from issues such as rigid strategies and insufficient adaptability in load balancing and scheduling. Optimizing load balancing is an important direction for large-model inference.

[0003] To address the load balancing problem in the inference process of large models (including but not limited to large-scale language models and visual language models), existing solutions generally employ greedy algorithms, sequentially allocating tasks to the computing units with the lowest current total load. These algorithms only focus on the instantaneous load state of nodes and do not optimize load distribution balance from a global perspective. In real-world scenarios where large model inference tasks dynamically change, an imbalance can easily occur where "local node overload and congestion" coexist with "some node resource idleness." Experimental data shows that the scheduling length obtained using this algorithm is about 20% higher than the ideal state, resulting in wasted computing resources and a decrease in inference speed.

[0004] The core implementation idea of ​​the open-source inference framework FlashInfer is as follows: First, we obtain the data size constraints. The basic partitioning unit length with a fixed Q dimension is 64. Based on the total KV length and hardware scale, we calculate the upper limit of the KV length to constrain the KV data block size.

[0005] Next, data segmentation is performed using the Q dimension as the core. The Q-packing length (Q data length multiplied by the number of attention heads) of each sample is calculated, and the number of sub-blocks that the Q data of that sample needs to be segmented is calculated, splitting the Q data into multiple contiguous sub-blocks. For each Q sub-block, its corresponding KV data effective length is determined. If the effective KV length exceeds the length limit, the KV data is split into multiple sub-blocks, ensuring that the length of each KV sub-block does not exceed the threshold.

[0006] Then, load balancing is performed. The load of a single task is quantified using the Q-packet length multiplied by the KV sub-block length as the cost function. The current cumulative load of all computational units is maintained using a min-heap. All split tasks are traversed, assigned to the unit with the lowest current load, and the cumulative load of that unit is updated.

[0007] Therefore, the open-source inference framework FlashInfer achieves efficient scheduling of attention computation for large models through data partitioning and greedy load distribution, making it one of the most widely used inference optimization solutions in the industry.

[0008] The core drawback of existing technologies, such as FlashInfer, lies in their greedy algorithms, which focus only on the least loaded computational unit at any given time, without considering the global distribution of the task set. This short-sighted strategy leads to uneven task allocation in large-scale model inference scenarios, causing load fragmentation. This manifests as a coexistence of numerous lightly loaded computational units (loads far below average) and a few heavily loaded computational units (loads significantly above average), resulting in low resource utilization. Experimental data shows that in a typical inference scenario with a batch size of 64, using a greedy strategy results in a maximum scheduling length that is approximately 20% higher than the theoretical optimal value. Summary of the Invention

[0009] This application provides a method, apparatus, device, and storage medium for large model inference load balancing based on iterative search.

[0010] The first aspect of this application provides a large model inference load balancing method based on iterative search, comprising: Obtain the set of inference tasks and the number of computing units. The set of inference tasks includes multiple inference tasks. Determine the total computing power of all inference tasks in the set of inference tasks and the maximum computing power of a single task based on the set of inference tasks. Based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computing units, a binary search iterative algorithm is used to determine the maximum allocation capacity threshold of computing units and the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold. The assigned inference tasks are sent to each computing unit, enabling each computing unit to execute the assigned inference tasks and output the inference results.

[0011] In an optional embodiment of this application, the step of determining the maximum allocation capacity threshold for each computing unit and the inference task allocated to each computing unit based on the total computational amount of all inference tasks, the maximum computational amount of a single task, and the number of computing units, using a binary search iterative algorithm, includes: Step 1: Initialize the binary search boundary based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computational units n. Step 2: Determine the current intermediate allocation capacity of the computing unit based on the initial binary search boundary, verify whether the current intermediate allocation capacity can complete the partitioning of the inference task set using n subsets, and divide the inference task set into multiple inference task subsets; Step 3: Adjust the binary search boundary based on the verification results. If the adjusted binary search boundary meets the preset conditions, use the current intermediate allocation capacity as the maximum allocation capacity threshold, and use multiple inference task subsets as the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold. If the adjusted binary search boundary does not meet the preset conditions, use the adjusted binary search boundary as the initial binary search boundary and re-execute Step 2.

[0012] In an optional embodiment of this application, the binary search boundary is initialized based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computational units n, using the following expression:

[0013]

[0014] in, The lower realm The upper bound is n, where n is the number of computational units. The total computational cost for all reasoning tasks. This represents the maximum computational cost for a single task.

[0015] In an optional embodiment of this application, the current intermediate allocation capacity of the computing unit is determined based on the initialized binary search boundary using the following expression:

[0016] in, Allocate capacity in the intermediate stage of the t-th iteration of the computation unit. This is the lower bound for the (t-1)th iteration. It is the upper bound of the (t-1)th iteration.

[0017] In an optional embodiment of this application, the step of verifying whether the current intermediate allocation capacity is divided into n subsets to complete the partitioning of the inference task set, and dividing the inference task set into multiple inference task subsets, includes: Rearrange all reasoning tasks S in descending order of element size to obtain S'={s1',s2',...,sm'}, satisfying s1'≥s2'... ≥sm'; Initialize empty subset collection ; The following steps are used to pack elements in descending order: For each si', iterate through the created subsets B∈Bins and check whether sum(B) + si'≤Ct is satisfied; If the first subset B that satisfies the condition is found, add si' to B; If no subset that meets the conditions is found, create a new subset Bnew = {si'} and add it to the set Bins; The number of subsets is calculated as bin_count = |Bins|, which is the number of subsets required under the current intermediate allocation capacity Ct. If bin_count ≤ n, the verification passes; otherwise, the verification fails, where n is the number of computational units.

[0018] In an optional embodiment of this application, adjusting the binary search boundary based on the verification result includes: If bin_count≤n, then update the upper bound Ut = Ct, where bin_count is the number of subsets of inference tasks, n is the number of computation units, Ut is the upper bound, and Ct is the current intermediate allocation capacity; If bin_count > n, then update the lower bound Lt = Ct, where Lt is the lower bound.

[0019] In an optional embodiment of this application, the preset condition is: Ut == Lt or the number of iterations reaches the preset parameter k, where Ut is the upper bound and Lt is the lower bound.

[0020] A second aspect of this application provides a large model inference load balancing device based on iterative search, comprising: The acquisition module is used to acquire the inference task set and the number of computing units. The inference task set includes multiple inference tasks. Based on the inference task set, the total computing power of all inference tasks in the inference task set and the maximum computing power of a single task are determined. The determination module is used to determine the maximum allocation capacity threshold of the computing unit and the inference task allocated to each computing unit based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computing units, using a binary search iterative algorithm. The sending module is used to send the assigned inference tasks to each computing unit, so that each computing unit can execute the assigned inference tasks and output the inference results.

[0021] A third aspect of this application provides a computer device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-mentioned large model inference load balancing methods based on iterative search.

[0022] A fourth aspect of the embodiments of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the large model inference load balancing method based on iterative search as described above.

[0023] Compared with the prior art, the technical solutions provided in this application have at least some or all of the following advantages: This application describes a large-model inference load balancing method based on iterative search. The inference task set includes multiple inference tasks. The total computational load and maximum single-task computational load of all inference tasks in the inference task set are determined. Based on the total computational load, maximum single-task computational load, and number of computing units, a binary search iterative algorithm is used to determine the maximum allocation capacity threshold for each computing unit and the corresponding inference tasks allocated to each computing unit. The allocated inference tasks are sent to each computing unit, enabling each computing unit to execute its assigned inference tasks and output inference results. Through global iterative search, the load planning is continuously optimized to better balance the load among computing units, reduce the maximum scheduling length, and decrease resource idleness, thereby improving the throughput of large-model inference, reducing latency, improving inference service quality, and reducing costs. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart of a large model inference load balancing method based on iterative search provided in one embodiment of this application; Figure 2 A large model inference load balancing system architecture diagram based on iterative search is provided for one embodiment of this application; Figure 3 A schematic diagram of a large model inference load balancing device based on iterative search provided in one embodiment of this application; Figure 4 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0025] In the process of developing this application, the inventors discovered that current methods of load balancing for large-scale model inference are inferior in terms of throughput, latency, inference service quality, and cost.

[0026] To address the aforementioned issues, this application provides a method, apparatus, device, and storage medium for load balancing of large model inference based on iterative search, in order to improve the throughput of large model inference, reduce latency, enhance the quality of inference services, and reduce costs.

[0027] The solutions in this application embodiment can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0028] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0029] Please see Figure 1 The large model inference load balancing method based on iterative search provided in this application includes the following steps S100~S300: S100, obtain the inference task set and the number of computing units, wherein the inference task set includes multiple inference tasks, and determine the total computing amount of all inference tasks in the inference task set and the maximum computing amount of a single task based on the inference task set; S200, based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computing units, determines the maximum allocation capacity threshold of computing units and the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold through a binary search iterative algorithm. S300 sends the assigned inference tasks to each computing unit, enabling each computing unit to execute the assigned inference tasks and output the inference results.

[0030] The large-model inference load balancing method based on iterative search proposed in this application is applicable to static batch processing or quasi-dynamic scheduling scenarios. For online inference, tasks can be accumulated to a certain number before triggering the algorithm for batch processing, or a sliding window mechanism can be used to periodically reschedule. The large models in this application include, but are not limited to, the beanbag model.

[0031] This application presents a load balancing method for large-model inference based on iterative search. Focusing on load balancing strategies for large-model inference, it achieves significant competitive advantages in performance improvement and cost control through innovative iterative search design, demonstrating broad commercial prospects. Specifically, this is reflected in the following aspects: First, cost reduction and efficiency improvement, addressing core industry pain points: Improving performance and reducing costs are crucial demands for the commercial promotion of large-model inference. This application precisely addresses these two needs, significantly reducing computing power costs while improving hardware resource utilization. The load balancing algorithm reduces the idle ratio of computing resources, supporting higher inference throughput with the same computing power investment. For cloud vendors, the improved concurrent service capabilities of GPU servers directly increase revenue efficiency per unit of hardware. Second, inference latency optimization, improving service quality and user experience: For cache-dependent scenarios (such as real-time dialogue and intelligent customer service), this application reduces cache access fragmentation through data flow scheduling strategies, lowering inference latency and shortening dialogue response time, meeting the commercial needs of low-latency scenarios. Low-latency experiences can help businesses gain a differentiated competitive advantage in areas such as intelligent interaction and real-time content generation, improving user retention and conversion rates. Third, broad coverage: This application can be applied to various large-scale model inference scenarios, including natural language processing, computer vision, and multimodal generation, meeting the diverse needs of industries such as internet, finance, healthcare, and manufacturing. Core target customers include cloud service providers, large-scale model R&D companies, industry solution providers, and hardware manufacturers.

[0032] In an optional embodiment of this application, step S200, which involves determining the maximum allocation capacity threshold for each computing unit and the corresponding inference task allocated to each computing unit based on the total computational amount of all inference tasks, the maximum computational amount of a single task, and the number of computing units, using a binary search iterative algorithm, includes: Step 1: Initialize the binary search boundary based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computational units n. Step 2: Determine the current intermediate allocation capacity of the computing unit based on the initial binary search boundary, verify whether the current intermediate allocation capacity can complete the partitioning of the inference task set using n subsets, and divide the inference task set into multiple inference task subsets; Step 3: Adjust the binary search boundary based on the verification results. If the adjusted binary search boundary meets the preset conditions, use the current intermediate allocation capacity as the maximum allocation capacity threshold, and use multiple inference task subsets as the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold. If the adjusted binary search boundary does not meet the preset conditions, use the adjusted binary search boundary as the initial binary search boundary and re-execute Step 2.

[0033] The large-model inference load balancing method based on iterative search proposed in this application is characterized by iterative search-based task partitioning, specifically including the following three interrelated technical points: First, load capacity optimization based on iterative search: The set of task loads to be allocated is initialized with lower and upper bounds for capacity. In each iteration, the median value is taken as the "hypothetical maximum capacity limit," and it is verified whether the task can be packed into a limited number of computing units under this capacity. Second, descending bin packing: In each iterative search, based on the principle of "prioritizing large tasks," all tasks to be allocated are first sorted by load value from largest to smallest, prioritizing large tasks to avoid overload risk. Then, the sorted tasks are traversed, and the first computing unit whose cumulative load does not exceed the current hypothetical capacity limit is found. The current task is allocated to this unit, and its load is updated. If all computing units cannot accommodate the current task, the hypothetical capacity is deemed infeasible, triggering iterative search boundary adjustment. Third, dynamic boundary update: After multiple iterations, the final upper bound of the capacity limit is taken as the optimal capacity limit, and the descending first-fit algorithm is executed again. The resulting subset partition is the task allocation scheme. The above three points together form a "search-verification-convergence" closed loop: iterative search controls and optimizes the accuracy, descending bin packing ensures the quality of single-step solutions, and dynamic boundary updates drive global convergence.

[0034] In an optional embodiment of this application, the task partitioning problem can be modeled as a one-dimensional bin packing problem. The optimal solution is approximated in polynomial time using a First-Fit Decreasing (FFD) algorithm combined with binary search. The input is: a set of numbers S = {s1,s2,...,sm} (m ≥ 1, si ≥ 0); the target subset n (positive integers); and a preset precision control parameter k. S is partitioned into Π = {B1,B2,...,Bn}, such that each subset is pairwise disjoint and their union is S. The completion time is defined as makespan(Π) = max1≤i≤nsum(Bi). The final objective is to find the optimal completion time OPT(S, n) = minΠmakespan(Π).

[0035] In an optional embodiment of this application, the binary search boundary is initialized based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computational units n, using the following expression:

[0036]

[0037] in, The lower realm The upper bound is n, where n is the number of computational units. The total computational cost for all reasoning tasks. This represents the maximum computational cost for a single task.

[0038] In an optional embodiment of this application, the current intermediate allocation capacity of the computing unit is determined based on the initialized binary search boundary using the following expression:

[0039] in, Allocate capacity in the intermediate stage of the t-th iteration of the computation unit. This is the lower bound for the (t-1)th iteration. Let t be the upper bound of the (t-1)th iteration, where t = 1, 2, ..., k, and k is a preset parameter.

[0040] In an optional embodiment of this application, the step of verifying whether the current intermediate allocation capacity is divided into n subsets to complete the partitioning of the inference task set, and dividing the inference task set into multiple inference task subsets, includes: Rearrange all reasoning tasks S in descending order of element size to obtain S'={s1',s2',...,sm'}, satisfying s1'≥s2'... ≥sm'; Initialize the empty subset collection Bins = []. The following steps are used to pack elements in descending order: For each si', iterate through the created subsets B∈Bins and check whether sum(B) + si'≤Ct is satisfied; If the first subset B that satisfies the condition is found, add si' to B; If no subset that meets the conditions is found, create a new subset Bnew = {si'} and add it to the set Bins; The number of subsets is calculated as bin_count = |Bins|, which is the number of subsets required under the current intermediate allocation capacity Ct. If bin_count ≤ n, the verification passes; otherwise, the verification fails, where n is the number of computational units.

[0041] In an optional embodiment of this application, adjusting the binary search boundary based on the verification result includes: If bin_count ≤ n (Ct capacity is sufficient), then update the upper bound Ut = Ct, where bin_count is the number of inference task subsets, n is the number of computation units, Ut is the upper bound, and Ct is the current intermediate allocation capacity; If bin_count > n (Ct capacity is insufficient), then update the lower bound Lt = Ct, where Lt is the lower bound.

[0042] In an optional embodiment of this application, the preset condition is: Ut == Lt or the number of iterations reaches the preset parameter k, where Ut is the upper bound and Lt is the lower bound.

[0043] In an optional embodiment of this application, the method further includes: After k iterations, the capacity interval length is reduced to the initial range. Take the final capacity C after iteration. final =U k ; The capacity limit is C. final The final subset set of Bins final ; Calculate completion time (makespan) final =maxB∈Πfinalsum(B).

[0044] In an optional embodiment of this application, the method further includes: The algorithm logic is verified through the following steps: Objective: To confirm that the core mechanisms such as binary search, descending bin packing, and boundary update are implemented correctly.

[0045] Test content: 1) Convergence Test: Input three sets of task loads of different sizes (|S|=100 / 1000 / 10000), with n=64 and k=20. Record the convergence of each iteration. , The judgment criteria include: a. Monotonic non-decreasing a. monotonically non-increasing; b. final | - | ≤ ε (ε is a preset precision threshold); c. bin_count t When n ≤ Update is required, otherwise Update is required.

[0046] 2) Constraint Satisfaction Test: For the final partition Πfinal, verify sum(Bᵢ) ≤ 0 for each computational unit. And confirm that the union of all subsets equals the original set S without omission. Judgment criterion: constraint violation rate is 0%.

[0047] In an optional embodiment of this application, comparative experimental analysis is conducted through the following steps to quantify the performance improvement of this application compared to FlashInfer in load balancing and inference: In the LLaMA-70B inference scenario, with a batch size of 64 and input lengths randomly distributed between 128 and 2048, the hardware is an 8×H100 GPU. The method of this application and FlashInfer were run separately, repeated three times, and the average was taken. Evaluation criteria: a. Makespan reduction rate: The total scheduling time of this application is reduced by 10% compared to FlashInfer; b. Load balancing: LBF = (max load - min load) / average load is calculated, and the LBF of this application is reduced by ≥10% compared to FlashInfer; c. P99 latency (maximum latency for 99% of requests): End-to-end latency is reduced by ≥10%.

[0048] Figure 2 The diagram shows the system architecture of the large model inference load balancing method based on iterative search, illustrating the connection relationships between the functional modules. This allows the large model inference load balancing method based on iterative search to be used in an independent inference framework or to replace the load balancing module of the Flashinfer framework.

[0049] In an optional embodiment of this application, the large model inference load balancing method based on iterative search is applied to the computing nodes of a molecular dynamics computing cluster. The molecular dynamics computing task is treated as an inference task, the computing nodes of the molecular dynamics computing cluster are treated as computing units, and before performing load balancing calculations on the computing nodes of the molecular dynamics computing cluster, the method further includes: performing hierarchical configuration management of the molecular dynamics computing cluster. For each node in the three-tier architecture of the molecular dynamics computing cluster, when the current node is started, it obtains configuration information from the parent node of the current node, establishes the version number, change log and event subscription relationship in the configuration information, wherein the three-tier architecture includes management nodes, service nodes and computing nodes; When the configuration information of the current node changes, a change event and its corresponding version number are generated, a change log is recorded based on the change event, and the change event is propagated through the event bus; Periodically check whether the configuration version number of the current node is consistent with that of its parent node; If the configuration version number of the current node is inconsistent with that of its parent node, a synchronization request is sent to the parent node, wherein the synchronization request carries the configuration version number of the current node; Based on the configuration version number of the current node, incremental data is extracted from the parent node, enabling the current node to apply incremental configuration changes to its configuration information. The incremental data is determined based on the configuration version numbers of the current node and its parent node.

[0050] In an optional embodiment of this application, for each node in the three-tier architecture of the molecular dynamics computing cluster, when the current node is started, obtaining configuration information from the parent node of the current node and establishing the version number, change log, and event subscription relationship in the configuration information includes: In the three-tier architecture of the molecular dynamics computing cluster, the management node layer, service node layer, and computing node layer each manage their own configuration domain independently. The service node configuration inherits the configuration template of the management node and overrides the first preset configuration item, while the computing node configuration inherits the configuration template of the service node and overrides the second preset configuration item.

[0051] In an optional embodiment of this application, the step of generating a change event and its corresponding version number when the configuration information of the current node changes, recording a change log based on the change event, and propagating the change event through the event bus includes: When the configuration information of the management node changes, a first change event is generated and pushed to all service nodes through the event bus. The service nodes receive the first change event, update their local configuration, and pass the first change event to the compute nodes for local configuration updates. After modifying their local configuration, each node replies with a confirmation message. The first change event includes a version number, change content, and timestamp. The change content is recorded in the change log. When the configuration information of the compute node changes, a second change event is generated and pushed to all service nodes through the event bus. The service nodes determine whether to synchronize with the service nodes, compute nodes and management nodes based on the changes in the second change event.

[0052] In an optional embodiment of this application, the process of determining whether to synchronize with the service node, compute node, and management node based on the change content of the second change event at the service node includes: If the change in the second change event is to expand the memory of the compute node, the change content of the second change event will be synchronized to the management node; If the change in the second change event is a change in the location of the temporary directory of the compute node, there is no need to synchronize the change in the second change event to the management node.

[0053] In an optional embodiment of this application, incremental data is extracted from the parent node based on the configuration version number of the current node, including: Extract all changes from the current node onwards from the change log of the parent node, and treat all changes as incremental data.

[0054] In an optional embodiment of this application, the method further includes: When the configuration items of the first compute node are modified, the version number and timestamp of the first compute node are V1 and T1, respectively. After the modified configuration items are propagated to the second compute node, the second compute node modifies the configuration items based on the version number V1. The version number and timestamp of the second compute node are V2 and T2, respectively. V2>V1 and T2>T1. The version numbers V1 and V2 are in an increasing relationship, and it is determined that there is no conflict in the configuration of the compute node. When the configuration items of the first computing node are modified, the version number and timestamp of the first computing node are V1 and T1 respectively. After the modified configuration items are propagated to the second computing node, the second computing node modifies the configuration items based on the version number V1. The version number and timestamp of the second computing node are V1 and T2 respectively. It is determined that there is a first conflict in the configuration of the computing node. When the global version number is V4, the third compute node receives an event with version number V3. It determines that V4 > V3, and therefore determines that there is a second conflict in the compute node's configuration. There is no need to modify the configuration items of the third compute node, and it obtains the latest configuration information from the superior node.

[0055] In an optional embodiment of this application, the method further includes: In the event of a first conflict in the configuration of compute nodes, the first conflict can be resolved using one of the following strategies: timestamp priority strategy, node priority strategy, merging strategy, or manual resolution.

[0056] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0057] Please see Figure 3 One embodiment of this application provides a large model inference load balancing device 300 based on iterative search, comprising: The acquisition module 310 is used to acquire the inference task set and the number of computing units. The inference task set includes multiple inference tasks. The total computing amount of all inference tasks in the inference task set and the maximum computing amount of a single task are determined based on the inference task set. The determination module 320 is used to determine the maximum allocation capacity threshold of the computing unit and the inference task allocated to each computing unit based on the total amount of computation of all inference tasks, the maximum amount of computation of a single task, and the number of computing units, through a binary search iterative algorithm. The sending module 330 is used to send the assigned inference tasks to each computing unit, so that each computing unit can execute the assigned inference tasks and output the inference results.

[0058] Specific limitations of the aforementioned device 300 can be found in the above description of the limitations of the large model inference load balancing method based on iterative search, and will not be repeated here. Each module in the aforementioned device 300 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0059] In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described iterative search-based large model inference load balancing method. It includes: memory and a processor; the memory stores the computer program; and the processor executes the computer program to implement any step of the above-described iterative search-based large model inference load balancing method.

[0060] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, can implement any of the steps in the large model inference load balancing method based on iterative search described above.

[0061] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0066] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A large-model inference load balancing method based on iterative search, characterized in that, include: Obtain the set of inference tasks and the number of computing units. The set of inference tasks includes multiple inference tasks. Determine the total computing power of all inference tasks in the set of inference tasks and the maximum computing power of a single task based on the set of inference tasks. Based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computing units, a binary search iterative algorithm is used to determine the maximum allocation capacity threshold of computing units and the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold. The assigned inference tasks are sent to each computing unit, enabling each computing unit to execute the assigned inference tasks and output the inference results.

2. The method according to claim 1, characterized in that, The method, based on the total computational load of all inference tasks, the maximum computational load of a single task, and the number of computing units, uses a binary search iterative algorithm to determine the maximum allocation capacity threshold for computing units and the corresponding inference tasks allocated to each computing unit, including: Step 1: Initialize the binary search boundary based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computational units n. Step 2: Determine the current intermediate allocation capacity of the computing unit based on the initial binary search boundary, verify whether the current intermediate allocation capacity can complete the partitioning of the inference task set using n subsets, and divide the inference task set into multiple inference task subsets; Step 3: Adjust the binary search boundary based on the verification results. If the adjusted binary search boundary meets the preset conditions, use the current intermediate allocation capacity as the maximum allocation capacity threshold, and use multiple inference task subsets as the inference tasks allocated to each computing unit corresponding to the maximum allocation capacity threshold. If the adjusted binary search boundary does not meet the preset conditions, use the adjusted binary search boundary as the initial binary search boundary and re-execute Step 2.

3. The method according to claim 2, characterized in that, The binary search boundary is initialized based on the total computational cost of all inference tasks, the maximum computational cost of a single task, and the number of computational units n, using the following expression: ； ； in, The lower realm The upper bound is n, where n is the number of computational units. The total computational cost for all reasoning tasks. This represents the maximum computational cost for a single task.

4. The method according to claim 2, characterized in that, The current intermediate allocation capacity of the computing unit is determined based on the initial binary search boundary using the following expression: ； in, Allocate capacity in the intermediate stage of the t-th iteration of the computation unit. This is the lower bound for the (t-1)th iteration. It is the upper bound of the (t-1)th iteration.

5. The method according to claim 2, characterized in that, The step of verifying whether the current intermediate allocation capacity is used to complete the partitioning of the inference task set using n subsets, and partitioning the inference task set into multiple inference task subsets, includes: Rearrange all reasoning tasks S in descending order of element size to obtain S'={s1',s2',...,sm'}, satisfying s1'≥s2'... ≥sm'; Initialize the empty subset collection Bins = []. The following steps are used to pack elements in descending order: For each si', iterate through the created subsets B∈Bins and check whether sum(B) + si'≤Ct is satisfied; If the first subset B that satisfies the condition is found, add si' to B; If no subset that meets the conditions is found, create a new subset Bnew = {si'} and add it to the set Bins; The number of subsets is calculated as bin_count = |Bins|, which is the number of subsets required under the current intermediate allocation capacity Ct. If bin_count ≤ n, the verification passes; otherwise, the verification fails, where n is the number of computational units.

6. The method according to claim 2, characterized in that, The step of adjusting the binary search boundary based on the verification results includes: If bin_count≤n, then update the upper bound Ut = Ct, where bin_count is the number of subsets of inference tasks, n is the number of computation units, Ut is the upper bound, and Ct is the current intermediate allocation capacity; If bin_count > n, then update the lower bound Lt = Ct, where Lt is the lower bound.

7. The method according to claim 2, characterized in that, The preset conditions are: Ut == Lt or the number of iterations reaches the preset parameter k, where Ut is the upper bound and Lt is the lower bound.

8. A large-model inference load balancing device based on iterative search, characterized in that, include: The acquisition module is used to acquire the inference task set and the number of computing units. The inference task set includes multiple inference tasks. Based on the inference task set, the total computing power of all inference tasks in the inference task set and the maximum computing power of a single task are determined. The determination module is used to determine the maximum allocation capacity threshold of the computing unit and the inference task allocated to each computing unit based on the total computation of all inference tasks, the maximum computation of a single task, and the number of computing units, using a binary search iterative algorithm. The sending module is used to send the assigned inference tasks to each computing unit, so that each computing unit can execute the assigned inference tasks and output the inference results.

9. A computer device, comprising: A memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the large model inference load balancing method based on iterative search as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the large model inference load balancing method based on iterative search as described in any one of claims 1 to 7.

Images