Optimization design method for GPU server computing power of large model training

Abstract

The application discloses a GPU server computing power optimization design method suitable for large model training, relates to the technical field of GPU server computing power optimization, and comprises the following steps: constructing a hierarchical topological cost graph reflecting the physical interconnection topology of a GPU cluster; identifying a seed operator from a to-be-trained model calculation graph and preferentially mapping the seed operator to a GPU area with the lowest communication cost; starting from the seed operator, gradually expanding along the data dependency direction, naturally closing at a weak connection position with the minimum communication amount to form multiple calculation areas; after boundary refinement and topological proximity mapping, pre-generating elastic scaling rules based on load distribution; and dynamically borrowing adjacent GPUs in a low communication cost domain according to the rules to eliminate pipeline bubbles during training operation. Through rule pre-generation in the planning stage, the application realizes the organic cooperation of static strategies and dynamic scheduling, and solves the problem that the power utilization rate is limited due to the separation of the two and the lack of topological perception in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of GPU server computing power optimization technology, and in particular relates to a GPU server computing power optimization design method adapted for large model training. Background Technology

[0002] As the parameter scale of large language models and multimodal models reaches hundreds of billions or even trillions, a single GPU is far from sufficient to carry out the complete training task. Distributed training must rely on GPU server clusters ranging from thousands to tens of thousands of GPUs. However, the actual computing power utilization of large-scale clusters is usually only between 35% and 45%, with a large amount of computing power consumed by communication waiting, uneven load, and pipeline bubbles. How to maximize training throughput with limited GPU resources has become the core challenge in the design of large model training infrastructure.

[0003] Currently, distributed parallel training typically divides the model into multiple operators and distributes them to various GPUs for execution using a hybrid strategy of data parallelism, tensor parallelism, and pipeline parallelism. Some methods plan static parallel strategies before training using cost models and search algorithms, and then combine 1F1B or dynamic micro-batch scheduling to reduce pipeline bubbles. However, static strategy search and dynamic bubble elimination are disconnected from each other. The former does not consider the feasibility of bubble elimination at runtime when planning, and the latter lacks fine-grained awareness of high-speed interconnect topologies such as NVLink inside the GPU during execution. This makes it difficult for the generated solution to achieve optimal coordination between communication cost and bubble elimination effect. In addition, existing methods are mostly based on global search and lack focus on computationally intensive operators and communication structures in the model, which limits both search efficiency and solution quality.

[0004] Existing computing power optimization methods cannot automatically perceive the hierarchical topology differences within GPU clusters, making it difficult to collaboratively generate static hybrid parallel strategies and dynamic elastic scheduling rules. This results in the model's parallel partition boundaries not being naturally located in communication-sparse areas, and the inability to elastically borrow computing power within high-speed topology domains to eliminate pipeline bubbles during runtime. Consequently, these methods severely restrict the further improvement of computing power utilization and training throughput for large model training. To address this, this invention proposes a GPU server computing power optimization design method adapted for large model training. Summary of the Invention

[0005] The purpose of this invention is to provide a GPU server computing power optimization design method adapted to large model training, which solves the problem that the static parallel strategy and dynamic scheduling are separated in the existing methods, and that the lack of awareness of the hierarchical topology differences of the GPU cluster leads to limited computing power utilization.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0007] This invention provides a GPU server computing power optimization design method adapted for large model training, comprising the following steps:

[0008] S1. Detect the physical interconnection relationship between GPUs in the target GPU cluster, obtain communication bandwidth and communication latency, and construct a hierarchical topology cost map that reflects the communication cost between GPUs.

[0009] S2. Obtain the computation graph of the large model to be trained, identify seed operators based on the computational cost, communication cost and data dependency of each operator, and map the seed operators to the GPU region with the lowest communication cost in the hierarchical topology cost graph to form the initial computation region.

[0010] S3. Starting from the seed operator, expand adjacent operators step by step along the data dependency direction of the computation graph. Based on the communication volume between the current operator and the upstream operator and the remaining computing capacity of the current computing region, decide step by step whether to merge the current operator into the current computing region. When the communication volume exceeds the preset condition, close the current computing region at the weak connection position before the current operator, and generate a new computing region in the topologically adjacent GPU region, so that the region boundary is formed at the weak connection position with the minimum communication volume.

[0011] S4. Adjust the boundaries of each computing region and perform topological proximity mapping on operators within the region, prioritizing the mapping of operator pairs with communication volume exceeding a preset threshold to adjacent GPUs with low communication cost.

[0012] S5. Generate elastic scaling rules for computing regions based on the load distribution and topological adjacency of each computing region. When the load of the target computing region exceeds the preset threshold, adjacent GPUs will be dynamically merged to take over some computing tasks. The merging relationship will be terminated after the load is restored.

[0013] S6. During training, the buffer status and computing load of each computing area are monitored in real time. When a pipeline bubble risk is detected, adjacent GPUs are triggered to merge and execute computing tasks according to the elastic scaling rules. After completion, the merging relationship is released.

[0014] Furthermore, the specific process of constructing the hierarchical topology cost graph in step S1 includes:

[0015] S1.1 Read the physical location identifier and interconnect port connection record of each GPU in the target GPU cluster to determine the physical connection path between GPUs;

[0016] S1.2 For GPU pairs with physical connections, initiate a bidirectional data transmission test to measure the communication bandwidth and round-trip communication latency between the GPU pairs;

[0017] S1.3. Based on the measured communication bandwidth and communication latency values, GPU pairs with communication bandwidth greater than the preset communication bandwidth threshold and communication latency lower than the preset communication latency threshold are classified into the first communication level; GPU pairs that do not meet the conditions of the first communication level but are located within the same host are classified into the second communication level; and GPU pairs that need to be connected via external network devices are classified into the third communication level.

[0018] S1.4 Construct a hierarchical topology cost graph with GPUs as nodes and communication levels as edge attributes, and label each edge with the actual measured communication bandwidth and communication latency values ​​as communication costs.

[0019] Furthermore, the specific process of identifying the seed operator and preferentially mapping the seed operator to the GPU region with the lowest communication cost in step S2 includes:

[0020] S2.1 Extract the number of floating-point operations of each operator in the computation graph of the large model to be trained as the computational quantity, extract the amount of data transmission between each operator and its predecessor and successor operators as the communication quantity, and extract the number of direct predecessor operators and the number of direct successor operators of each operator as the data dependency tightness.

[0021] S2.2 Normalize the computational cost, communication cost, and data dependency tightness of each operator, and sum the three normalized values ​​according to preset weights to obtain the seed score of the operator.

[0022] S2.3. Identify operators whose seed scores exceed a preset score threshold as seed operators;

[0023] S2.4. In the hierarchical topology cost map, find the GPU region with the lowest communication cost. Map the seed operators to each GPU in the GPU region with the lowest communication cost in descending order of seed score. During mapping, the remaining computing capacity of each GPU in the region is evenly distributed as a constraint to form the initial computing region.

[0024] Furthermore, the specific process in step S3 of gradually expanding adjacent operators from the seed operator to form multiple computational regions includes:

[0025] S3.1. Take the GPU region where various sub-operators are located as the current computation region, and add the direct predecessor and direct successor operators of the seed operator in the computation graph to the queue of operators to be expanded.

[0026] S3.2. Take an operator from the queue of operators to be expanded according to the data dependency direction as the current operator, obtain the data transfer amount between the current operator and its upstream operator in the current computing region, and obtain the remaining computing capacity of each GPU in the current computing region.

[0027] S3.3 If the data transmission volume does not exceed the preset communication volume threshold and the remaining computing capacity is not lower than the preset capacity threshold, then the current operator is merged into the current computing region, and the unvisited operators in the direct predecessor and direct successor operators of the current operator are added to the queue of operators to be expanded.

[0028] S3.4 If the data transmission volume exceeds the preset communication volume threshold, then among all connections between the current operator and the upstream operator in the current computing region, select the connection with the smallest data transmission volume as the weak connection position, close the current computing region at the weak connection position, take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region.

[0029] S3.5 If the data transmission volume does not exceed the preset communication volume threshold but the remaining computing capacity is lower than the preset capacity threshold, then close the current computing region at the last connection before the current operator; take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region.

[0030] S3.6 Repeat steps S3.2 to S3.5 until the queue of operators to be expanded is empty, resulting in multiple computation regions. The boundaries between each computation region are formed at the weak connection position with the least communication traffic.

[0031] Furthermore, the specific process of adjusting the boundaries of each computational region and performing topological proximity mapping on the operators within the region in step S4 includes:

[0032] S4.1. Traverse the boundaries of each computational region generated in step S3. For each operator located at the boundary, calculate the total communication volume between the operator and each operator in the current computational region, and at the same time calculate the total communication volume between the operator located at the boundary and each operator in the adjacent computational regions.

[0033] S4.2 If the total communication volume between the operator located at the boundary position and the adjacent computing region is greater than the total communication volume with the current computing region, and the remaining computing capacity of the adjacent computing region can accommodate the operator located at the boundary position, then the operator located at the boundary position is transferred from the current computing region to the adjacent computing region, and the boundary between the two computing regions is updated.

[0034] S4.3 Repeat S4.1 to S4.2 until all boundary operators no longer meet the relocation conditions, and the boundary adjustment is complete;

[0035] S4.4 After the boundary adjustment is completed, all operators within each computational region are traversed in pairs to obtain the communication volume between each pair of operators. The operator pairs are sorted in descending order of communication volume to form a sequence of operator pairs to be mapped.

[0036] S4.5. Take out operator pairs sequentially from the sequence of operator pairs to be mapped, query the communication cost between any two GPUs in the GPU set occupied by the current computing region in the hierarchical topology cost graph, and map the two operators in the operator pair to the two GPUs with the lowest communication cost respectively; if there are multiple GPU pairs with the same communication cost, select the GPU pair with more remaining computing capacity for mapping.

[0037] S4.6 Repeat S4.5 until all operator pairs in the sequence of operator pairs to be mapped that have a communication volume exceeding a preset threshold have been mapped.

[0038] Furthermore, the specific process of generating the elastic scaling rules for the computing region based on the load distribution and topological adjacency relationships of each computing region in step S5 includes:

[0039] S5.1 For each computing region formed after the mapping in step S4, the total amount of computation of the operators allocated on each GPU in each computing region is calculated. The ratio of the total amount of computation to the theoretical peak computing power of the GPU is used as the estimated load rate of the GPU. The average value of the estimated load rates of all GPUs in the computing region is taken as the load value of the computing region.

[0040] S5.2 Query the topological adjacency relationship of the GPU sets occupied by each computing region in the hierarchical topological cost map, determine the computing regions that are directly adjacent to each computing region in the topological structure, and record the communication level and communication cost between the GPUs occupied by adjacent computing regions.

[0041] S5.3 For each computing region, generate the corresponding elastic scaling rule entry. The rule entry includes: target computing region identifier, list of adjacent computing region identifiers that can be requisitioned, specific GPU identifiers that can be requisitioned within the adjacent computing regions, requisition trigger conditions, and requisition release conditions.

[0042] The requisition triggering conditions are: the real-time load rate of any GPU in the target computing area exceeds the preset load rate threshold, and the input buffer length of the GPU is lower than the preset buffer threshold.

[0043] The conditions for lifting the requisition are: the real-time load rate of the GPU that was requisitioned in the target computing area falls below the preset load rate threshold, and the input buffer length is restored to above the preset buffer threshold.

[0044] S5.4 Prioritize the list of requisitionable adjacent computing regions. The ranking is based on the communication cost between the adjacent computing region and the target computing region. The adjacent computing region with the lower the communication cost is ranked higher in the list.

[0045] Furthermore, the specific process of real-time monitoring of the buffer status and computing load of each computing region and triggering elastic scaling rules in step S6 includes:

[0046] S6.1 During training, the input buffer length and real-time computing load rate of each GPU in each computing region are periodically collected at preset time intervals. The input buffer length is the number of micro-batch computing tasks to be executed by the GPU, and the real-time computing load rate of the GPU is the ratio of the actual computing throughput of the GPU at the current moment to its theoretical peak computing throughput.

[0047] S6.2 For each computing region, the GPU with the shortest input buffer length in that computing region is identified as a potential bottleneck GPU. When the input buffer length of that GPU is lower than a preset warning threshold, it is determined that there is a pipeline bubble risk in that computing region.

[0048] S6.3 When it is determined that there is a pipeline bubble risk in the target calculation region, retrieve the rule entry corresponding to the target calculation region from the elastic scaling rule table generated in S5.

[0049] S6.4 From the list of available adjacent computing regions in the rule entries, query in order of priority from high to low whether the available GPUs in each adjacent computing region are currently in an idle state. The idle state is when the real-time computing load rate of the GPU is lower than a preset idle threshold.

[0050] S6.5. The first available GPU that is idle is identified as the requisitioned GPU. The micro-batch computing tasks to be executed in the target computing area and their dependent data are sent to the requisitioned GPU through the channel with the lowest communication cost between the two in the hierarchical topology cost graph.

[0051] S6.6. Requisition the GPU to execute the received micro-batch computing tasks, and after the computing is completed, send the computing results back to the original GPU in the target computing area through the same channel.

[0052] S6.7 When the length of the input buffer of the target computing region recovers to above the preset warning threshold, a requisition release instruction is sent to the requisitioned GPU. The requisitioned GPU stops receiving new micro-batch computing tasks, and after completing the currently received tasks, it returns to an idle state, and the requisition relationship is released.

[0053] Furthermore, it also includes step S7:

[0054] S7. During the training process, the actual computation time and actual communication time of each GPU are periodically collected and compared with the estimated computation time and estimated communication time generated in the planning stage. When the deviation exceeds the preset deviation threshold, local reconstruction is performed on the affected computation area. The local reconstruction is performed online during the training process without interrupting the training task.

[0055] Furthermore, in step S3, when there are multiple parallel branches in the computation graph, an independent water droplet immersion growth process is started simultaneously for each branch path, and the progress of each branch is coordinated during the growth process so that the time deviation of each branch reaching the convergence node does not exceed a preset threshold.

[0056] The present invention has the following beneficial effects:

[0057] 1. This invention starts with a seed operator and gradually expands along the data dependency direction of the computation graph. It naturally closes the computation region at the weak connection position with the least communication, so that the inter-segment boundary of the pipeline parallelism naturally falls in the communication sparse area, reducing cross-regional communication overhead and pipeline bubble generation.

[0058] 2. This invention uses a seed operator identification and priority mapping strategy to prioritize the deployment of computationally intensive and communication-intensive core operators in the GPU region with the lowest communication cost. This ensures that the communication on the critical path of model training is naturally under optimal topological conditions, and avoids the core operators being dispersed to high communication cost regions in subsequent partitioning.

[0059] 3. This invention optimizes the assignment of boundary operators based on skeleton partitioning by refining the boundary and mapping the topological proximity. It also maps operators with high communication volume within the region to adjacent GPUs in the physical topology, further reducing communication latency within the region and aligning the logical structure of the computing region with the physical topology.

[0060] 4. This invention pre-generates elastic scaling rules during the planning phase, organically combining static parallel strategies with dynamic bubble elimination. During runtime, there is no need to temporarily search for borrowed GPUs; a direct table lookup can complete millisecond-level borrowing responses within a low-communication-cost domain. The borrowing process itself does not introduce additional communication bottlenecks.

[0061] 5. This invention uses an online adaptive local reconstruction mechanism to sense the deviation between actual performance and prediction during training and automatically correct the boundary of the computational region and the mapping relationship without interrupting training, thus ensuring the robustness of the solution in long-term training tasks. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0064] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0065] Example 1

[0066] See Figure 1 This invention provides a GPU server computing power optimization design method adapted for large model training, comprising the following steps:

[0067] S1. Detect the physical interconnection relationship between GPUs in the target GPU cluster, obtain communication bandwidth and communication latency, and construct a hierarchical topology cost map that reflects the communication cost between GPUs.

[0068] It should be noted that the specific process of constructing the hierarchical topology cost graph in step S1 includes:

[0069] S1.1 Read the physical location identifier and interconnect port connection record of each GPU in the target GPU cluster to determine the physical connection path between GPUs;

[0070] S1.2 For GPU pairs with physical connections, initiate a bidirectional data transmission test to measure the communication bandwidth and round-trip communication latency between the GPU pairs;

[0071] S1.3. Based on the measured communication bandwidth and communication latency values, GPU pairs with communication bandwidth greater than the preset communication bandwidth threshold and communication latency lower than the preset communication latency threshold are classified into the first communication level; GPU pairs that do not meet the conditions of the first communication level but are located within the same host are classified into the second communication level; and GPU pairs that need to be connected via external network devices are classified into the third communication level.

[0072] S1.4 Construct a hierarchical topology cost graph with GPUs as nodes and communication levels as edge attributes, and label each edge with the actual measured communication bandwidth and communication latency values ​​as communication costs.

[0073] S2. Obtain the computation graph of the large model to be trained, identify seed operators based on the computational cost, communication cost and data dependency of each operator, and map the seed operators to the GPU region with the lowest communication cost in the hierarchical topology cost graph to form the initial computation region.

[0074] It should be further explained that the specific process of identifying the seed operator and preferentially mapping the seed operator to the GPU region with the lowest communication cost in step S2 includes:

[0075] S2.1 Extract the number of floating-point operations of each operator in the computation graph of the large model to be trained as the computational quantity, extract the amount of data transmission between each operator and its predecessor and successor operators as the communication quantity, and extract the number of direct predecessor operators and the number of direct successor operators of each operator as the data dependency tightness.

[0076] S2.2 Normalize the computational cost, communication cost, and data dependency tightness of each operator, and sum the three normalized values ​​according to preset weights to obtain the seed score of the operator.

[0077] S2.3. Identify operators whose seed scores exceed a preset score threshold as seed operators;

[0078] S2.4. In the hierarchical topology cost map, find the GPU region with the lowest communication cost. Map the seed operators to each GPU in the GPU region with the lowest communication cost in descending order of seed score. During mapping, the remaining computing capacity of each GPU in the region is evenly distributed as a constraint to form the initial computing region.

[0079] S3. Starting from the seed operator, expand adjacent operators step by step along the data dependency direction of the computation graph. Based on the communication volume between the current operator and the upstream operator and the remaining computing capacity of the current computing region, decide step by step whether to merge the current operator into the current computing region. When the communication volume exceeds the preset condition, close the current computing region at the weak connection position before the current operator, and generate a new computing region in the topologically adjacent GPU region, so that the region boundary is formed at the weak connection position with the minimum communication volume.

[0080] It should be further explained that the specific process of gradually expanding adjacent operators from the seed operator to form multiple computational regions in step S3 includes:

[0081] S3.1. Take the GPU region where various sub-operators are located as the current computation region, and add the direct predecessor and direct successor operators of the seed operator in the computation graph to the queue of operators to be expanded.

[0082] S3.2. Take an operator from the queue of operators to be expanded according to the data dependency direction as the current operator, obtain the data transfer amount between the current operator and its upstream operator in the current computing region, and obtain the remaining computing capacity of each GPU in the current computing region.

[0083] S3.3 If the data transmission volume does not exceed the preset communication volume threshold and the remaining computing capacity is not lower than the preset capacity threshold, then the current operator is merged into the current computing region, and the unvisited operators in the direct predecessor and direct successor operators of the current operator are added to the queue of operators to be expanded.

[0084] S3.4 If the data transmission volume exceeds the preset communication volume threshold, then among all connections between the current operator and the upstream operator in the current computing region, select the connection with the smallest data transmission volume as the weak connection position, close the current computing region at the weak connection position, take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region.

[0085] S3.5 If the data transmission volume does not exceed the preset communication volume threshold but the remaining computing capacity is lower than the preset capacity threshold, then close the current computing region at the last connection before the current operator; take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region.

[0086] S3.6 Repeat steps S3.2 to S3.5 until the queue of operators to be expanded is empty, resulting in multiple computation regions. The boundaries between each computation region are formed at the weak connection position with the least communication traffic.

[0087] S4. Adjust the boundaries of each computing region and perform topological proximity mapping on operators within the region, prioritizing the mapping of operator pairs with communication volume exceeding a preset threshold to adjacent GPUs with low communication cost.

[0088] It should be noted that the specific process of adjusting the boundaries of each computational region and performing topological proximity mapping on the operators within the region in step S4 includes:

[0089] S4.1. Traverse the boundaries of each computational region generated in step S3. For each operator located at the boundary, calculate the total communication volume between the operator and each operator in the current computational region, and at the same time calculate the total communication volume between the operator located at the boundary and each operator in the adjacent computational regions.

[0090] S4.2 If the total communication volume between the operator located at the boundary position and the adjacent computing region is greater than the total communication volume with the current computing region, and the remaining computing capacity of the adjacent computing region can accommodate the operator located at the boundary position, then the operator located at the boundary position is transferred from the current computing region to the adjacent computing region, and the boundary between the two computing regions is updated.

[0091] S4.3 Repeat S4.1 to S4.2 until all boundary operators no longer meet the relocation conditions, and the boundary adjustment is complete;

[0092] S4.4 After the boundary adjustment is completed, all operators within each computational region are traversed in pairs to obtain the communication volume between each pair of operators. The operator pairs are sorted in descending order of communication volume to form a sequence of operator pairs to be mapped.

[0093] S4.5. Take out operator pairs sequentially from the sequence of operator pairs to be mapped, query the communication cost between any two GPUs in the GPU set occupied by the current computing region in the hierarchical topology cost graph, and map the two operators in the operator pair to the two GPUs with the lowest communication cost respectively; if there are multiple GPU pairs with the same communication cost, select the GPU pair with more remaining computing capacity for mapping.

[0094] S4.6 Repeat S4.5 until all operator pairs in the sequence of operator pairs to be mapped that have a communication volume exceeding a preset threshold have been mapped.

[0095] S5. Generate elastic scaling rules for computing regions based on the load distribution and topological adjacency of each computing region. When the load of the target computing region exceeds the preset threshold, adjacent GPUs will be dynamically merged to take over some computing tasks. The merging relationship will be terminated after the load is restored.

[0096] It should be further explained that the specific process of generating the elastic scaling rules for the computing region based on the load distribution and topological adjacency relationships of each computing region in step S5 includes:

[0097] S5.1 For each computing region formed after the mapping in step S4, the total amount of computation of the operators allocated on each GPU in each computing region is calculated. The ratio of the total amount of computation to the theoretical peak computing power of the GPU is used as the estimated load rate of the GPU. The average value of the estimated load rates of all GPUs in the computing region is taken as the load value of the computing region.

[0098] S5.2 Query the topological adjacency relationship of the GPU sets occupied by each computing region in the hierarchical topological cost map, determine the computing regions that are directly adjacent to each computing region in the topological structure, and record the communication level and communication cost between the GPUs occupied by adjacent computing regions.

[0099] S5.3 For each computing region, generate the corresponding elastic scaling rule entry. The rule entry includes: target computing region identifier, list of adjacent computing region identifiers that can be requisitioned, specific GPU identifiers that can be requisitioned within the adjacent computing regions, requisition trigger conditions, and requisition release conditions.

[0100] The requisition triggering conditions are: the real-time load rate of any GPU in the target computing area exceeds the preset load rate threshold, and the input buffer length of the GPU is lower than the preset buffer threshold.

[0101] The conditions for lifting the requisition are: the real-time load rate of the GPU that was requisitioned in the target computing area falls below the preset load rate threshold, and the input buffer length is restored to above the preset buffer threshold.

[0102] S5.4 Prioritize the list of requisitionable adjacent computing regions. The ranking is based on the communication cost between the adjacent computing region and the target computing region. The adjacent computing region with the lower the communication cost is ranked higher in the list.

[0103] S6. During training, the buffer status and computing load of each computing area are monitored in real time. When a pipeline bubble risk is detected, adjacent GPUs are triggered to merge and execute computing tasks according to the elastic scaling rules. After completion, the merging relationship is released.

[0104] It should be further explained that the specific process of real-time monitoring of the buffer status and computing load of each computing region and triggering the elastic scaling rules in step S6 includes:

[0105] S6.1 During training, the input buffer length and real-time computing load rate of each GPU in each computing region are periodically collected at preset time intervals. The input buffer length is the number of micro-batch computing tasks to be executed by the GPU, and the real-time computing load rate of the GPU is the ratio of the actual computing throughput of the GPU at the current moment to its theoretical peak computing throughput.

[0106] S6.2 For each computing region, the GPU with the shortest input buffer length in that computing region is identified as a potential bottleneck GPU. When the input buffer length of that GPU is lower than a preset warning threshold, it is determined that there is a pipeline bubble risk in that computing region.

[0107] S6.3 When it is determined that there is a pipeline bubble risk in the target calculation region, retrieve the rule entry corresponding to the target calculation region from the elastic scaling rule table generated in S5.

[0108] S6.4 From the list of available adjacent computing regions in the rule entries, query in order of priority from high to low whether the available GPUs in each adjacent computing region are currently in an idle state. The idle state is when the real-time computing load rate of the GPU is lower than a preset idle threshold.

[0109] S6.5. The first available GPU that is idle is identified as the requisitioned GPU. The micro-batch computing tasks to be executed in the target computing area and their dependent data are sent to the requisitioned GPU through the channel with the lowest communication cost between the two in the hierarchical topology cost graph.

[0110] S6.6. Requisition the GPU to execute the received micro-batch computing tasks, and after the computing is completed, send the computing results back to the original GPU in the target computing area through the same channel.

[0111] S6.7 When the length of the input buffer of the target computing region recovers to above the preset warning threshold, a requisition release instruction is sent to the requisitioned GPU. The requisitioned GPU stops receiving new micro-batch computing tasks, and after completing the currently received tasks, it returns to an idle state, and the requisition relationship is released.

[0112] S7. During the training process, the actual computation time and actual communication time of each GPU are periodically collected and compared with the estimated computation time and estimated communication time generated in the planning stage. When the deviation exceeds the preset deviation threshold, local reconstruction is performed on the affected computation area. The local reconstruction is performed online during the training process without interrupting the training task.

[0113] Example 2

[0114] In the above embodiments, the water droplet immersion growth process in step S3 adopts a strategy of gradually expanding along a single data dependency path, which is suitable for scenarios where the model computation graph is mainly linear chain structure. However, when the computation graph of the large model to be trained has a multi-branch structure, such as the text encoding branch and image encoding branch converging at the cross attention layer in a multimodal model, or multiple expert network branches converging at the gate node in a hybrid expert model, the single-chain growth strategy can only process each branch one by one in sequence. After the branch processed first reaches the convergence node, the GPU resources it occupies are in a waiting state, while the branch processed later has not yet completed growth and mapping. This results in a time difference between the computation regions on both sides of the convergence node during the planning stage, which will inevitably generate structural waiting bubbles during runtime, weakening the optimization effect of immersion growth partitioning. In order to solve this problem, in this embodiment, when there are multiple parallel branches in the computation graph, an independent water droplet immersion growth process is started simultaneously for each branch path in step S3, and the progress of each branch is coordinated during the growth process so that the time deviation of each branch reaching the convergence node does not exceed a preset threshold.

[0115] It should be noted that the specific process includes the following:

[0116] A. Perform topological analysis on the computation graph to identify all branch nodes with an out-degree greater than or equal to 2 and sink nodes with an in-degree greater than or equal to 2. For each pair of branch nodes and sink nodes, extract all independent computation paths between them to form a set of branch paths, and determine the seed operator for each branch path.

[0117] B. For all branch paths within the same branch path set, start an independent water droplet immersion growth process simultaneously. Each branch path starts from its first operator and gradually expands adjacent operators along the data dependency direction to form its own branch computation region chain in the manner described in S3. The growth process of each branch path shares progress information through the growth coordination controller. The progress information includes the number of expanded operators, the cumulative computation amount, and the number of computation regions formed.

[0118] C. For each branch path, based on its established branch computation region chain and operator mapping relationship, predict the total time required for the branch path to reach the convergence node, and calculate the maximum deviation of the predicted arrival time for each branch path.

[0119] D. When the maximum deviation exceeds the preset allowable deviation threshold, growth adjustment is performed on the leading branch with the earliest arrival time and the bottleneck branch with the latest arrival time respectively; for the leading branch, the computation region is closed in advance at the weak connection position of its current growth front, and the computational wait generated by the newly added pipeline segment is used to delay the arrival time of the branch; for the bottleneck branch, the current computation region of the branch is expanded to the topologically adjacent idle GPU to increase the parallelism and shorten the arrival time of the branch.

[0120] E. When all branch paths grow to the convergence node, the convergence node operator is mapped to the low-communication-cost GPU that is topologically adjacent to the last computational region of each branch path, completing the region stitching of each branch computational region chain and the convergence node, forming a multi-branch computational region, and the time deviation of each branch to the convergence node does not exceed the preset allowable deviation threshold.

[0121] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A GPU server computing power optimization design method adapted for large model training, characterized in that, Includes the following steps: S1. Detect the physical interconnection relationship between GPUs in the target GPU cluster, obtain communication bandwidth and communication latency, and construct a hierarchical topology cost map that reflects the communication cost between GPUs. S2. Obtain the computation graph of the large model to be trained, identify seed operators based on the computational cost, communication cost and data dependency of each operator, and map the seed operators to the GPU region with the lowest communication cost in the hierarchical topology cost graph to form the initial computation region. S3. Starting from the seed operator, gradually expand the adjacent operators along the data dependency direction of the computation graph. Based on the communication volume between the current operator and the upstream operator and the remaining computation capacity of the current computation region, gradually decide whether to merge the current operator into the current computation region. When the traffic exceeds the preset condition, the current computation region is closed at the weak connection position before the current operator, and a new computation region is generated in the topologically adjacent GPU region, so that the region boundary is formed at the weak connection position with the least traffic. S4. Adjust the boundaries of each computing region and perform topological proximity mapping on operators within the region, prioritizing the mapping of operator pairs with communication volume exceeding a preset threshold to adjacent GPUs with low communication cost. S5. Generate elastic scaling rules for computing regions based on the load distribution and topological adjacency of each computing region. When the load of the target computing region exceeds the preset threshold, adjacent GPUs will be dynamically merged to take over some computing tasks. The merging relationship will be terminated after the load is restored. S6. During training, the buffer status and computing load of each computing area are monitored in real time. When a pipeline bubble risk is detected, adjacent GPUs are triggered to merge and execute computing tasks according to the elastic scaling rules. After completion, the merging relationship is released.

2. The GPU server computing power optimization design method for adapting to large model training according to claim 1, characterized in that, The specific process of constructing the hierarchical topology cost graph in step S1 includes: S1.1 Read the physical location identifier and interconnect port connection record of each GPU in the target GPU cluster to determine the physical connection path between GPUs; S1.2 For GPU pairs with physical connections, initiate a bidirectional data transmission test to measure the communication bandwidth and round-trip communication latency between the GPU pairs; S1.

3. Based on the measured communication bandwidth and communication latency values, GPU pairs with communication bandwidth greater than the preset communication bandwidth threshold and communication latency lower than the preset communication latency threshold are classified into the first communication level; GPU pairs that do not meet the conditions of the first communication level but are located within the same host are classified into the second communication level; and GPU pairs that need to be connected via external network devices are classified into the third communication level. S1.4 Construct a hierarchical topology cost graph with GPUs as nodes and communication levels as edge attributes, and label each edge with the actual measured communication bandwidth and communication latency values ​​as communication costs.

3. The GPU server computing power optimization design method for adapting to large model training according to claim 2, characterized in that, The specific process of identifying the seed operator and preferentially mapping the seed operator to the GPU region with the lowest communication cost in step S2 includes: S2.1 Extract the number of floating-point operations of each operator in the computation graph of the large model to be trained as the computational quantity, extract the amount of data transmission between each operator and its predecessor and successor operators as the communication quantity, and extract the number of direct predecessor operators and the number of direct successor operators of each operator as the data dependency tightness. S2.2 Normalize the computational cost, communication cost, and data dependency tightness of each operator, and sum the three normalized values ​​according to preset weights to obtain the seed score of the operator. S2.

3. Identify operators whose seed scores exceed a preset score threshold as seed operators; S2.

4. In the hierarchical topology cost map, find the GPU region with the lowest communication cost. Map the seed operators to each GPU in the GPU region with the lowest communication cost in descending order of seed score. During mapping, the remaining computing capacity of each GPU in the region is evenly distributed as a constraint to form the initial computing region.

4. The GPU server computing power optimization design method for adapting to large model training according to claim 3, characterized in that, The specific process in step S3, which involves gradually expanding adjacent operators from the seed operator to form multiple computational regions, includes: S3.

1. Take the GPU region where various sub-operators are located as the current computation region, and add the direct predecessor and direct successor operators of the seed operator in the computation graph to the queue of operators to be expanded. S3.

2. Take an operator from the queue of operators to be expanded according to the data dependency direction as the current operator, obtain the data transfer amount between the current operator and its upstream operator in the current computing region, and obtain the remaining computing capacity of each GPU in the current computing region. S3.3 If the data transmission volume does not exceed the preset communication volume threshold and the remaining computing capacity is not lower than the preset capacity threshold, then the current operator is merged into the current computing region, and the unvisited operators in the direct predecessor and direct successor operators of the current operator are added to the queue of operators to be expanded. S3.4 If the data transmission volume exceeds the preset communication volume threshold, then among all connections between the current operator and the upstream operator in the current computing region, select the connection with the smallest data transmission volume as the weak connection position, close the current computing region at the weak connection position, take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region. S3.5 If the data transmission volume does not exceed the preset communication volume threshold but the remaining computing capacity is lower than the preset capacity threshold, then close the current computing region at the last connection before the current operator; take the current operator as the starting point of the new computing region, map it to the GPU region that is topologically adjacent to the current computing region, and set the new computing region as the current computing region. S3.6 Repeat steps S3.2 to S3.5 until the queue of operators to be expanded is empty, resulting in multiple computation regions. The boundaries between each computation region are formed at the weak connection position with the least communication traffic.

5. The GPU server computing power optimization design method for adapting to large model training according to claim 4, characterized in that, The specific process of adjusting the boundaries of each computational region and performing topological proximity mapping on the operators within the region in step S4 includes: S4.

1. Traverse the boundaries of each computational region generated in step S3. For each operator located at the boundary, calculate the total communication volume between the operator and each operator in the current computational region, and at the same time calculate the total communication volume between the operator located at the boundary and each operator in the adjacent computational regions. S4.2 If the total communication volume between the operator located at the boundary position and the adjacent computing region is greater than the total communication volume with the current computing region, and the remaining computing capacity of the adjacent computing region can accommodate the operator located at the boundary position, then the operator located at the boundary position is transferred from the current computing region to the adjacent computing region, and the boundary between the two computing regions is updated. S4.3 Repeat S4.1 to S4.2 until all boundary operators no longer meet the relocation conditions, and the boundary adjustment is complete; S4.4 After the boundary adjustment is completed, all operators within each computational region are traversed in pairs to obtain the communication volume between each pair of operators. The operator pairs are sorted in descending order of communication volume to form a sequence of operator pairs to be mapped. S4.

5. Take out operator pairs sequentially from the sequence of operator pairs to be mapped, query the communication cost between any two GPUs in the GPU set occupied by the current computing region in the hierarchical topology cost graph, and map the two operators in the operator pair to the two GPUs with the lowest communication cost respectively; if there are multiple GPU pairs with the same communication cost, select the GPU pair with more remaining computing capacity for mapping. S4.6 Repeat S4.5 until all operator pairs in the sequence of operator pairs to be mapped that have a communication volume exceeding a preset threshold have been mapped.

6. The GPU server computing power optimization design method for adapting to large model training according to claim 5, characterized in that, The specific process of generating the elastic scaling rules for the computing region based on the load distribution and topological adjacency of each computing region in step S5 includes: S5.1 For each computing region formed after the mapping in step S4, the total amount of computation of the operators allocated on each GPU in each computing region is calculated. The ratio of the total amount of computation to the theoretical peak computing power of the GPU is used as the estimated load rate of the GPU. The average value of the estimated load rates of all GPUs in the computing region is taken as the load value of the computing region. S5.2 Query the topological adjacency relationship of the GPU sets occupied by each computing region in the hierarchical topological cost map, determine the computing regions that are directly adjacent to each computing region in the topological structure, and record the communication level and communication cost between the GPUs occupied by adjacent computing regions. S5.3 For each computing region, generate the corresponding elastic scaling rule entry. The rule entry includes: target computing region identifier, list of adjacent computing region identifiers that can be requisitioned, specific GPU identifiers that can be requisitioned within the adjacent computing regions, requisition trigger conditions, and requisition release conditions. The requisition triggering conditions are: the real-time load rate of any GPU in the target computing area exceeds the preset load rate threshold, and the input buffer length of the GPU is lower than the preset buffer threshold. The conditions for lifting the requisition are: the real-time load rate of the GPU that was requisitioned in the target computing area falls below the preset load rate threshold, and the input buffer length is restored to above the preset buffer threshold. S5.4 Prioritize the list of requisitionable adjacent computing regions. The ranking is based on the communication cost between the adjacent computing region and the target computing region. The adjacent computing region with the lower the communication cost is ranked higher in the list.

7. The GPU server computing power optimization design method for adapting to large model training according to claim 6, characterized in that, The specific process of real-time monitoring of the buffer status and computing load of each computing region and triggering elastic scaling rules in step S6 includes: S6.1 During training, the input buffer length and real-time computing load rate of each GPU in each computing region are periodically collected at preset time intervals. The input buffer length is the number of micro-batch computing tasks to be executed by the GPU, and the real-time computing load rate of the GPU is the ratio of the actual computing throughput of the GPU at the current moment to its theoretical peak computing throughput. S6.2 For each computing region, the GPU with the shortest input buffer length in that computing region is identified as a potential bottleneck GPU. When the input buffer length of that GPU is lower than a preset warning threshold, it is determined that there is a pipeline bubble risk in that computing region. S6.3 When it is determined that there is a pipeline bubble risk in the target calculation region, retrieve the rule entry corresponding to the target calculation region from the elastic scaling rule table generated in S5. S6.4 From the list of available adjacent computing regions in the rule entries, query in order of priority from high to low whether the available GPUs in each adjacent computing region are currently in an idle state. The idle state is when the real-time computing load rate of the GPU is lower than a preset idle threshold. S6.

5. The first available GPU that is idle is identified as the requisitioned GPU. The micro-batch computing tasks to be executed in the target computing area and their dependent data are sent to the requisitioned GPU through the channel with the lowest communication cost between the two in the hierarchical topology cost graph. S6.

6. Requisition the GPU to execute the received micro-batch computing tasks, and after the computing is completed, send the computing results back to the original GPU in the target computing area through the same channel. S6.7 When the length of the input buffer of the target computing region recovers to above the preset warning threshold, a requisition release instruction is sent to the requisitioned GPU. The requisitioned GPU stops receiving new micro-batch computing tasks, and after completing the currently received tasks, it returns to an idle state, and the requisition relationship is released.

8. The GPU server computing power optimization design method for adapting to large model training according to claim 7, characterized in that, It also includes step S7: S7. During the training process, the actual computation time and actual communication time of each GPU are periodically collected and compared with the estimated computation time and estimated communication time generated in the planning stage. When the deviation exceeds the preset deviation threshold, local reconstruction is performed on the affected computation area. The local reconstruction is performed online during the training process without interrupting the training task.

9. The GPU server computing power optimization design method for adapting to large model training according to claim 1, characterized in that, In step S3, when there are multiple parallel branches in the computation graph, an independent water droplet immersion growth process is started simultaneously for each branch path, and the progress of each branch is coordinated during the growth process so that the time deviation of each branch reaching the convergence node does not exceed a preset threshold.

Images