An AI model automatic parallel cutting method for multi-GPU server

Abstract

The application discloses a kind of AI model automatic parallel cutting methods for multi-GPU server, belong to artificial intelligence and computer system architecture technical field, it includes obtaining the layer structure parameter of hardware connection parameter and AI model of multi-GPU server, constructs hardware topological graph and model calculation graph;Determine the cost of calculation and communication cost, generate cost evaluation result;Through the basic cutting scheme generated by greedy algorithm, optimization is adjusted by iterative redistribution algorithm, and initial cutting strategy is generated;Collect the real-time running data of each GPU to generate running state data;Analyze performance deviation, generate re-optimization trigger signal when meeting re-optimization condition;Running state data is used to update cost evaluation result, and update cutting strategy is generated.The application adopts closed-loop optimization mechanism combining static modeling and online dynamic feedback, can realize adaptive adjustment and continuous optimization of model parallel cutting strategy in training process, to improve the overall efficiency and resource utilization of distributed training.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer system architecture technology, and in particular to an automatic parallel partitioning method for AI models for multi-GPU servers. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI models, represented by deep neural networks, are becoming increasingly massive, with their computational complexity and number of parameters growing exponentially. The computing power and memory capacity of a single GPU are far from sufficient to handle the training tasks of these giant models. Therefore, using high-performance servers with multiple GPUs and employing distributed parallel training techniques to accelerate model iteration has become a mainstream technical solution in the industry. Parallel partitioning methods for AI models are one of the core technologies in this field, studying how to rationally distribute different computational parts of a large model across multiple GPUs for collaborative execution.

[0003] In existing technologies, parallel partitioning of AI models primarily relies on human expert configuration or automated tools based on simple rules. Human configuration requires developers to have a deep understanding of the model structure, hardware topology, and distributed training principles, finding optimal partitioning schemes through repeated trials and manual adjustments. Some deep learning frameworks also provide automated tools, but they typically employ coarse strategies, such as dividing the model into several equal parts or partitioning based solely on the number of model layers or parameters, without deeply considering the complex coupling between computation and communication. Furthermore, there are also deployment planning methods based on static analysis, which perform a one-time analysis of the model and hardware before training begins, generating a fixed partitioning strategy.

[0004] However, the aforementioned existing technologies have significant technical drawbacks. Manual configuration is inefficient, highly dependent on expert experience, and difficult to scale and reuse. Automated tools based on simple rules ignore the heterogeneous communication topology within multi-GPU servers and the significant differences in computational load between different model layers, easily leading to severe computational load imbalance and communication bottlenecks. Meanwhile, planning methods based on static analysis rely entirely on the accuracy of offline evaluation for their generated partitioning strategies, failing to cope with dynamic changes such as performance fluctuations and resource contention during training. If the initial evaluation is flawed or the system's operating state changes, its fixed deployment strategy will cause the entire training process to remain in a suboptimal state, resulting in a severe waste of computational resources. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an automatic parallel partitioning method for AI models on multi-GPU servers. It employs a closed-loop optimization mechanism that combines static modeling with online dynamic feedback, enabling adaptive adjustment and continuous optimization of the model parallel partitioning strategy during training, thereby improving the overall efficiency and resource utilization of distributed training.

[0006] The above objectives can be achieved through the following approach: An automatic parallel partitioning method for AI models on multi-GPU servers includes: acquiring hardware connection parameters of the multi-GPU server and layer structure parameters of the AI ​​model to construct a hardware topology graph and a model computation graph; determining the computational and communication costs related to the distributed deployment of the model based on the hardware topology graph and the model computation graph, and generating a cost evaluation result; generating a basic partitioning scheme using a greedy algorithm based on the cost evaluation result, adjusting and optimizing the basic partitioning scheme using an iterative redistribution algorithm to generate an initial partitioning strategy; during training using the initial partitioning strategy, collecting real-time running data from each GPU for timestamp alignment and integration to generate running status data; analyzing performance deviations based on the running status data, and generating a re-optimization trigger signal when the performance deviations meet the re-optimization conditions; responding to the re-optimization trigger signal, updating the cost evaluation result using the running status data, and generating an updated partitioning strategy based on the updated cost evaluation result.

[0007] Optionally, the construction of the hardware topology graph and model computation graph includes: identifying the physical connection types between GPUs in a multi-GPU server and obtaining bandwidth parameters to generate connection relationship data; constructing a hardware topology graph with GPUs as nodes and physical connections as edges based on the connection relationship data; parsing the AI ​​model structure, extracting the computation operation types, computational quantity parameters, parameter scales, and inter-layer data flow dependencies of each model layer to generate layer structure parameters; and constructing a directed acyclic graph with model layers as nodes and data flow dependencies as edges as the model computation graph based on the layer structure parameters.

[0008] Optionally, the generation of cost evaluation results includes: calculating the communication paths between GPU nodes based on the hardware topology graph, quantifying the communication costs of the communication paths, and generating a communication cost matrix; parsing the model computation graph, extracting the computational quantity parameters and parameter scale parameters of each model layer, and forming a model layer attribute set; calculating the estimated computation time for each model layer deployed on different GPUs based on the communication cost matrix and the model layer attribute set; and integrating the estimated computation time of all model layers and the communication time caused by inter-layer dependencies to generate cost evaluation results.

[0009] Optionally, generating the initial partitioning strategy includes: constructing a multi-objective optimization function based on the cost evaluation result; using a greedy algorithm to traverse the model layers in the model computation graph based on the multi-objective optimization function, selecting GPUs for deployment in each layer, and generating a basic partitioning scheme; applying an iterative reallocation algorithm to the basic partitioning scheme, adjusting it under the conditions of satisfying GPU memory capacity constraints and communication bandwidth constraints, and generating the initial partitioning strategy.

[0010] Optionally, generating runtime status data includes: applying the initial segmentation strategy for training, deploying monitoring agents on each GPU, and collecting computational utilization data and memory usage data; obtaining cross-GPU communication data volume by monitoring the underlying communication library and calculating the actual bandwidth usage of the communication link; and aligning and integrating the computational utilization data, the memory usage data, and the actual bandwidth usage with timestamps to generate runtime status data.

[0011] Optionally, generating the re-optimization trigger signal includes: extracting the computational utilization rate and the actual bandwidth utilization rate of the communication link from the operating status data, calculating the utilization variance index characterizing the degree of load balancing; obtaining the utilization variance threshold and the bandwidth utilization threshold used to determine whether the performance deviates from the normal range; and generating the re-optimization trigger signal when the utilization variance index exceeds the utilization variance threshold or the actual bandwidth utilization rate exceeds the bandwidth utilization threshold.

[0012] Optionally, updating the cost evaluation result using the running status data includes: receiving the re-optimization trigger signal, extracting the running status data within the time window before the trigger time; using the actual computation time and communication time in the running status data to calibrate the estimated computation time and communication time of the cost evaluation result, and generating an updated cost evaluation result.

[0013] Optionally, generating an updated partitioning strategy based on the updated cost evaluation results includes: taking the currently executing partitioning strategy as a starting point, and based on the updated cost evaluation results, applying an iterative redistribution algorithm to evaluate the improvement effect of migrating the model layer from the source GPU to the target GPU on the global optimization objective function, generating candidate migration operations; selecting migration operations from the candidate migration operations whose improvement effect meets predetermined conditions and satisfies hardware constraints, executing the migration, and generating an updated partitioning strategy.

[0014] Optionally, the method further includes: at the synchronization point of the training iteration, identifying the model layer that needs to be migrated based on the difference between the update splitting strategy and the current strategy; transferring the model parameter state of the model layer that needs to be migrated from the source GPU to the target GPU; dynamically modifying the data flow dependency in the model computation graph to match the new layer position; and taking effect at the start of the next training iteration.

[0015] Based on the same inventive concept, this invention also provides an automatic parallel partitioning system for AI models on multi-GPU servers. The system includes: a graph construction module for acquiring hardware connection parameters of the multi-GPU server and layer structure parameters of the AI ​​model, constructing a hardware topology graph and a model computation graph; a cost evaluation module for determining the computational and communication costs related to the distributed deployment of the model based on the hardware topology graph and the model computation graph, and generating a cost evaluation result; a strategy generation module for generating a basic partitioning scheme using a greedy algorithm based on the cost evaluation result, applying an iterative redistribution algorithm to adjust and optimize the basic partitioning scheme, and generating an initial partitioning strategy; a runtime monitoring module for collecting real-time runtime data from each GPU during training using the initial partitioning strategy, performing timestamp alignment and integration, and generating runtime status data; a re-optimization decision module for analyzing performance deviations based on the runtime status data, and generating a re-optimization trigger signal when the performance deviations meet the re-optimization conditions; and a strategy update module for responding to the re-optimization trigger signal, updating the cost evaluation result using the runtime status data, and generating an updated partitioning strategy based on the updated cost evaluation result.

[0016] Compared with the prior art, the present invention has the following advantages: This invention achieves fully automated generation of parallel deployment schemes for complex AI models on multi-GPU servers by jointly modeling hardware topology and model structure, combined with an efficient initial strategy generation algorithm. This reduces reliance on human expert experience, improves the scientificity and rationality of initial deployment, and shortens the debugging cycle of model deployment.

[0017] This invention establishes a closed-loop dynamic optimization mechanism from deployment, monitoring to feedback. By collecting runtime status data during the training process in real time, it can capture the performance deviation between static prediction and actual operation and actively trigger the re-optimization process, enabling the parallel partitioning strategy to adapt to the dynamically changing system environment and solving the problem of efficiency decay of static deployment schemes during long-term training.

[0018] This invention utilizes real-world operational data to calibrate the cost evaluation model online, ensuring that re-optimization decisions are made based on data reflecting the current real performance bottlenecks of the system, thereby achieving continuous refinement of the parallel scheme.

[0019] This invention designs a migration mechanism that performs policy updates at training iteration synchronization points, which can complete the dynamic redeployment of the model layer without interrupting the main training process, ensuring the stability and continuity of the training process, effectively transforming the dynamic optimization decision into actual performance gains, and realizing online adaptive parallel training.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an automatic parallel segmentation method for AI models on multi-GPU servers according to an embodiment of the present invention.

[0023] Figure 2 This is a hardware topology diagram of an embodiment of the present invention.

[0024] Figure 3 This is a heatmap of the cost evaluation results of an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the timing alignment of multi-GPU runtime status data according to an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of an AI model automatic parallel segmentation system for multi-GPU servers according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1 One embodiment of the present invention proposes an automatic parallel partitioning method for AI models on multi-GPU servers. It adopts a closed-loop optimization mechanism that combines static modeling with online dynamic feedback, which can adaptively adjust and continuously optimize the parallel partitioning strategy of the model during training, thereby improving the overall efficiency and resource utilization of distributed training.

[0029] The method described in this embodiment specifically includes: Obtain the hardware connection parameters of the multi-GPU server and the layer structure parameters of the AI ​​model, and construct the hardware topology diagram and the model computation diagram; Based on the hardware topology diagram and the model computation diagram, the computational and communication costs associated with the distributed deployment of the model are determined, and cost evaluation results are generated. Based on the cost evaluation results, a basic partitioning scheme is generated using a greedy algorithm, and an iterative redistribution algorithm is applied to adjust and optimize the basic partitioning scheme to generate an initial partitioning strategy. During the training process using the initial splitting strategy, real-time running data from each GPU is collected, timestamp aligned, and integrated to generate running status data. Based on the analysis of the operating status data, a performance deviation is identified, and a re-optimization trigger signal is generated when the performance deviation meets the re-optimization conditions. In response to the re-optimization trigger signal, the cost evaluation result is updated using the running status data, and an updated segmentation strategy is generated based on the updated cost evaluation result.

[0030] Specifically, firstly, this method lays the foundation for quantitative analysis by abstracting the hardware structure of multi-GPU servers and the computational structure of AI models into hardware topology graphs and model computation graphs, respectively. Then, based on these two graphs, a cost evaluation result incorporating computational and communication overhead is generated, transforming the complex parallel partitioning problem into an optimizable mathematical model. On this basis, a theoretically superior initial partitioning strategy is generated through a hybrid strategy combining greedy algorithms and iterative redistribution algorithms. The innovation of this method lies in its dynamic feedback and adjustment mechanism. After the model actually runs according to the initial strategy, it continuously monitors the real-world operating status of each GPU. Once performance deviations caused by uneven load or communication bottlenecks are detected, re-optimization is triggered. The re-optimization process uses collected real-time running data to calibrate and update the original cost evaluation result, and based on this, generates a better updated partitioning strategy, thus forming a complete adaptive closed loop of "modeling-deployment-monitoring-feedback-optimization". Through the initial static modeling and optimization, this method effectively avoids the blindness and inefficiency of manual configuration, improving the starting efficiency of distributed training from the outset. More importantly, the introduced online monitoring and dynamic re-optimization mechanisms enable the parallel partitioning strategy to adapt to the complex and ever-changing runtime environment during training, proactively identifying and resolving performance bottlenecks caused by inaccurate static predictions or state changes. This continuous self-adjustment capability ensures that computing resources are utilized throughout the entire training task, ultimately improving the overall training throughput of the AI ​​model, shortening the model development and iteration cycle, and enhancing the stability and performance of the large-scale distributed training system.

[0031] Optionally, the construction of the hardware topology graph and model computation graph includes: Identify the physical connection types between GPUs in a multi-GPU server and obtain bandwidth parameters to generate connection relationship data; Based on the connection relationship data, a hardware topology graph is constructed with GPUs as nodes and physical connections as edges; Analyze the AI ​​model structure, extract the computational operation types, computational parameters, parameter scales, and inter-layer data flow dependencies of each model layer, and generate layer structure parameters. Based on the layer structure parameters, a directed acyclic graph is constructed as the model computation graph, with model layers as nodes and data flow dependencies as edges.

[0032] Specifically, the first step is to construct a hardware topology map. This process aims to digitize and graphically represent the physical interconnect structure of the multi-GPU server. Operationally, this is done by executing low-level hardware probing commands on the server or calling hardware management libraries, such as the NVIDIA Management Library (NVML), to systematically identify the physical connection topology between all GPU devices within the server. This process requires accurately determining the connection type between each pair of GPUs, such as a high-speed NVLink bus or a general-purpose PCIe bus, and obtaining its corresponding theoretical maximum bandwidth parameters. This information is integrated into a structured connection relationship data set, which explicitly describes the connection attributes between any two GPU nodes. Based on this connection relationship data, a hardware topology map can be constructed. In this map, each GPU is abstracted as a node, and the physical connections between GPUs are abstracted as edges connecting nodes. The weights or attributes of the edges can be assigned to the obtained bandwidth parameters, thereby quantifying the communication capabilities between nodes. Figure 2As shown, the hardware topology diagram clearly illustrates the heterogeneous connections within the multi-GPU server. Nodes represent GPUs, and different edge patterns distinguish connection types with varying bandwidths, providing an intuitive and accurate data foundation for subsequent communication cost calculations. Next, the model computation graph is constructed, transforming the AI ​​model's computational flow into an analyzable directed acyclic graph. Operationally, static analysis tools or the framework's built-in tracing functionality are used to parse the AI ​​model's network structure definition. For each layer in the model, its core attributes need to be extracted, including the type of computational operation (e.g., convolution or fully connected), computational complexity parameters (usually quantified in floating-point operations (FLOPs), parameter size (i.e., the storage space occupied by trainable parameters such as weights and biases), and crucial inter-layer data flow dependencies, indicating which subsequent layers' inputs are the output of a particular layer's computation. These attribute information from all layers are then aggregated to form a set of layer structure parameters. Based on these layer structure parameters, a model computation graph can be constructed, where each layer of the AI ​​model is abstracted as a node, and the data flow dependencies between layers are abstracted as directed edges connecting the nodes. The direction of the edges represents the direction of data flow, thus forming a directed acyclic graph that accurately reflects the forward propagation computation order and dependencies of the model. This method improves the scientific nature and initial rationality of subsequent partitioning schemes, avoiding blind or experience-based model deployment, thereby laying the theoretical and data foundation for distributed training and realizing heterogeneous hardware-aware model analysis and representation, which is a prerequisite for the entire automated parallel optimization process.

[0033] Optionally, the generation cost evaluation result includes: Based on the hardware topology, the communication paths between each GPU node are calculated, and the communication cost of the communication paths is quantified to generate a communication cost matrix. Analyze the model computation graph, extract the computational quantity parameters and parameter scale parameters of each model layer, and form a model layer attribute set; Based on the communication cost matrix and the model layer attribute set, the estimated computation time for each model layer deployed on different GPUs is calculated. The estimated computation time of all model layers and the communication time caused by inter-layer dependencies are integrated to generate cost evaluation results.

[0034] Specifically, firstly, based on the constructed hardware topology graph, the accurate communication cost for distributed deployment needs to be calculated. This process uses graph theory algorithms, such as Dijkstra's or Floyd-Warshall, to calculate the optimal communication path between any two GPU nodes. The selection criterion for the optimal path is usually to maximize the end-to-end effective bandwidth. The communication cost of the path is quantified and stored in a communication cost matrix, where each element represents the time required to transfer a unit amount of data between two GPU nodes. Next, the model computation graph is analyzed in depth to evaluate the computational cost. For each node in the graph, i.e., each model layer, its core computational parameters and parameter size parameters are extracted. These attributes are organized into a model layer attribute set, providing the basis for subsequent computation time estimation. Based on the above communication cost matrix and model layer attribute set, the scenario of each model layer being deployed on each available GPU is simulated, and its computation execution time is estimated. This estimated computation time can be obtained using the following formula: , in, Representation model layer Estimated computation time on GPU j; It is the model layer The total number of floating-point operations, which is obtained from the model layer attribute set; This represents the theoretical or measured computational performance of GPU j, measured in floating-point operations per second. This parameter is used to support accurate evaluation in heterogeneous GPU environments. Finally, the estimated computation times of all model layers and the cross-GPU communication times that may arise due to data dependencies between layers are integrated to form the final cost evaluation result. This result is a comprehensive data structure that not only contains the computation time matrix for each model layer on each GPU, but also includes a function or lookup table describing the communication overhead required for cross-device deployment between any two layers. This communication overhead is calculated based on the amount of data transferred and the communication cost matrix between them. Figure 3 As shown, the core of this cost evaluation result is a cost matrix heatmap. The horizontal axis represents GPUs, and the vertical axis represents model layers. The color intensity of the cells represents the estimated computation time for deploying a specific model layer on a specific GPU. This method establishes a bridge connecting hardware topology characteristics and model computation requirements, enabling the prediction of the performance of potential deployment schemes. This predictive capability allows the optimization process to identify and avoid computational or communication bottlenecks in advance, thereby improving the quality of the initial splitting strategy and training efficiency.

[0035] Optionally, the initial segmentation strategy includes: Based on the cost evaluation results, a multi-objective optimization function is constructed; Based on the multi-objective optimization function, a greedy algorithm is used to traverse the model layers in the model computation graph, select GPUs for deployment in each layer, and generate a basic partitioning scheme. An iterative redistribution algorithm is applied to the basic partitioning scheme to adjust it under the conditions of satisfying GPU memory capacity constraints and communication bandwidth constraints, thereby generating an initial partitioning strategy.

[0036] Specifically, firstly, based on the cost evaluation results, a multi-objective optimization function is constructed. This function aims to quantify the merits of a specific partitioning scheme, typically minimizing the total execution time (pipeline latency) of a single training iteration of the entire model while maximizing load balancing across all GPUs. This function comprehensively considers the estimated computation time of each model layer in the cost evaluation results, as well as the communication time incurred due to cross-GPU deployment, formalizing a complex partitioning problem into a solvable mathematical optimization problem. Next, based on this multi-objective optimization function, a greedy algorithm performs a fast traversal of the model computation graph. This traversal process typically follows the topological sorting of the graph, processing each model layer according to the order of data flow dependencies. For each model layer to be allocated, the greedy algorithm evaluates the local impact of deploying it on each available GPU on the multi-objective optimization function and selects the GPU that maximizes the benefit or minimizes the cost of the current optimization objective as the deployment location for that layer. This decision is a locally optimal choice. After completing the traversal and allocation of all model layers, a complete but potentially non-globally optimal deployment scheme is formed; this scheme is called the basic partitioning scheme. Finally, to compensate for the potential for greedy algorithms to get stuck in local optima, an iterative redistribution algorithm is applied to refine the basic partitioning scheme. This algorithm attempts to migrate single or mini-batch model layers from the currently allocated GPU to other GPUs in multiple iterations, re-evaluating the impact of each potential migration on the global multi-objective optimization function value. Any migration operation that improves the optimization function value is adopted only after rigorous constraint checks. These constraints include that the target GPU's memory capacity must be sufficient to accommodate the parameters, gradients, and intermediate activation values ​​of the migrated layers (i.e., satisfying GPU memory capacity constraints), and that the increased cross-GPU communication traffic after migration must not exceed the effective carrying capacity of the relevant physical connections (i.e., satisfying communication bandwidth constraints). This iterative process continues until no migration operation that satisfies the constraints and brings performance improvements can be found, or the preset iteration limit is reached. The final optimized scheme output is the initial partitioning strategy. This method can generate an initial parallel deployment scheme that balances computational and communication overhead, load balancing, and hardware resource adaptation through offline simulation before the model is actually run. This provides a starting point for subsequent online training and dynamic optimization, and shortens the tuning cycle of distributed AI model deployment.

[0037] Optionally, the generation of runtime status data includes: Training is performed using the initial partitioning strategy described above, and monitoring agents are deployed on each GPU to collect computational utilization data and memory usage data. By monitoring the underlying communication library, the amount of cross-GPU communication data is obtained, and the actual bandwidth utilization of the communication link is calculated. The computational utilization data, the video memory usage data, and the actual bandwidth usage are timestamped and integrated to generate runtime status data.

[0038] Specifically, firstly, the initial partitioning strategy is implemented by deploying each layer of the AI ​​model to designated GPUs and initiating the distributed training process. During this training, a lightweight monitoring agent is deployed on each participating GPU. This agent periodically collects the core utilization of its GPU by calling underlying hardware monitoring interfaces, such as the NVIDIA Management Library (NVML), generating computational utilization data, and simultaneously collects real-time GPU memory usage data. Meanwhile, to understand the efficiency of data flow between GPUs, the monitoring system instrumentes or listens to underlying communication libraries such as NCCL to obtain the total amount of data actually transferred between any two GPUs, i.e., the amount of cross-GPU communication data. Based on this data, the actual bandwidth utilization of the communication link can be calculated using the following formula: , in, This represents the actual bandwidth utilization of the communication link. By monitoring the communication library within the time window The amount of cross-GPU communication data acquired internally; This is the theoretical maximum bandwidth of the physical connection between the two GPUs, a parameter obtained from the previously constructed hardware topology. Finally, the computational utilization data, memory usage data, and the calculated actual bandwidth utilization of each communication link, collected independently from each GPU, are timestamped and integrated. Figure 4 As shown, this graph integrates performance data from different sources through a shared timeline, forming synchronous, structured time-series data. This process ensures that data from different sources and devices can be correlated onto a unified timeline, thus forming a comprehensive, synchronous, and structured time-series database. This database is the runtime data, which accurately records the dynamic behavior of the entire multi-GPU system during training under the initial partitioning strategy. This method enables subsequent re-optimization decisions to be based on actual performance rather than static assumptions, thereby improving the accuracy and effectiveness of dynamic adjustments. This allows the entire automatic parallel partitioning system to adapt to complex runtime changes and achieve continuous performance self-optimization.

[0039] Optionally, updating the cost evaluation result using the operational status data includes: Receive the re-optimization trigger signal and extract the running status data within the time window preceding the trigger time; Using the actual computation time and communication time from the operational status data, the estimated computation time and communication time of the cost evaluation results are calibrated to generate updated cost evaluation results.

[0040] Specifically, firstly, upon receiving the re-optimization trigger signal, this signal is used as an instruction to initiate cost model calibration. The system immediately locks and extracts runtime data within a preset time window prior to the trigger signal. Through fine-grained analysis of kernel execution during training, the precise execution time of each model layer on its actual deployed GPU can be obtained—the actual computation time—as well as the actual latency of any cross-GPU data transfer—the communication time. The core step is to use this measured data to calibrate and replace the theoretical estimates in the initial cost evaluation results. Specifically, the runtime data within this time window is traversed, and for each model layer, its recorded average actual computation time is updated to the corresponding entry in the cost evaluation results. Similarly, for cross-GPU data transfers caused by inter-layer dependencies, the portion of the cost evaluation results describing the communication cost between the corresponding GPU pairs is updated based on the actual communication time recorded in the runtime data. After this series of calibration operations based on real runtime feedback, the generated data structure is the updated cost evaluation result. The data structure of this result remains consistent with the initial evaluation result, but its internal values ​​have been updated from static predictions to dynamic measured values. This method provides a basis for decision-making in subsequent segmentation strategy updates, ensuring that the re-optimization process is based on the current system's performance bottlenecks rather than outdated static assumptions. This improves the pertinence and effectiveness of dynamic adjustments, enabling the entire system to have the ability to learn and evolve continuously.

[0041] Optionally, the step of generating an updated segmentation strategy based on the updated cost evaluation results includes: Starting with the currently executing splitting strategy and based on the updated cost evaluation results, an iterative redistribution algorithm is applied to evaluate the improvement effect of migrating the model layer from the source GPU to the target GPU on the global optimization objective function, and candidate migration operations are generated. Select migration operations from the candidate migration operations that meet the predetermined conditions for improvement and satisfy hardware constraints, execute the migration, and generate an updated splitting strategy.

[0042] Specifically, based on the updated cost evaluation results, an iterative reallocation algorithm is first initiated. The core of this algorithm is to systematically evaluate the potential impact on global performance of migrating any model layer in the model computation graph from its current source GPU to another possible target GPU. For each potential migration operation, the global optimization objective function is recalculated using precise computation and communication time data from the updated cost evaluation results. This global optimization objective function is consistent with the objective of the function used when generating the initial policy, namely, minimizing the total training iteration time and optimizing load balancing. By comparing the objective function values ​​before and after the migration, the improvement effect of each migration operation on global performance can be quantified. All migration operations with positive improvement effects are collected to form a candidate migration operation set. Subsequently, each candidate migration operation in this set undergoes rigorous screening. There are two selection criteria: first, the improvement effect must meet predetermined conditions, such as the estimated performance improvement brought by migration must exceed a minimum threshold to avoid unnecessary adjustments due to minor fluctuations; second, the migration operation must strictly meet hardware constraints, namely, the remaining memory capacity of the target GPU must be sufficient to support the migrated model layer and its related computational overhead, and the increased communication load after migration must not cause any communication link saturation. From all the candidate migration operations that pass the screening, the operation that can bring the greatest improvement effect is selected. Finally, this selected optimal migration operation is applied to the current partitioning strategy, that is, logically updating the position of the model layer from the source GPU to the target GPU, thereby generating the updated partitioning strategy. This method responds to runtime performance changes by incrementally and targetedly fine-tuning based on the current strategy, rather than costly global recalculation. It utilizes a cost model calibrated based on real runtime data to ensure that every adjustment decision is data-driven and performance-oriented.

[0043] Optionally, the method further includes: At the synchronization point of the training iteration, the model layers that need to be transferred are identified based on the difference between the updated splitting strategy and the current strategy. The model parameter states of the model layers that need to be migrated are transferred from the source GPU to the target GPU, the data flow dependencies in the model computation graph are dynamically modified to match the new layer positions, and the changes take effect at the start of the next training iteration.

[0044] Specifically, this process is triggered at a specific point in a training iteration, known as the synchronization point. The synchronization point refers to the moment when all GPUs complete one forward propagation, backpropagation, gradient synchronization, and parameter update in distributed training. At this synchronization point, the current splitting strategy is first compared with the newly generated updated splitting strategy. By comparing the deployment position of each model layer in the model computation graph one by one, it is possible to accurately identify those model layers that have been assigned to different GPUs in the new strategy; these are the model layers that need to be migrated. For each identified model layer that needs to be migrated, a controlled data transfer process is initiated. The core of this process is to transfer the complete model parameter state of the model layer from its current source GPU to the target GPU specified by the updated splitting strategy. The model parameter state includes not only trainable parameters such as layer weights and biases, but also the internal state maintained by the optimizer for these parameters, such as momentum buffers or first and second moment estimates of the Adam optimizer, to ensure the continuity and correctness of the training process. After the transfer is completed, the logical structure of the model computation graph is dynamically modified. Specifically, this involves adjusting the data flow dependencies related to the transferred model layers. This means modifying the connections of their input and output edges to ensure that upstream layers send activation values ​​to the new target GPU, while downstream layers receive input from the new target GPU. All these migration and graph modification operations are completed at this synchronization point and will officially take effect at the start of the next training iteration. This means that new data batches will be computed according to the updated splitting strategy. This method enables the system to adapt online to runtime performance fluctuations and bottleneck changes, translating dynamically generated optimization strategies into actual performance gains. This ensures that the AI ​​model can consistently maintain optimal or near-optimal operating efficiency in long-term, large-scale distributed training tasks.

[0045] Based on the same inventive concept, such as Figure 5 As shown, the present invention also provides an automatic parallel segmentation system for AI models on multi-GPU servers, the system comprising: The graph construction module is used to obtain the hardware connection parameters of the multi-GPU server and the layer structure parameters of the AI ​​model, and to construct the hardware topology graph and the model computation graph. The cost evaluation module is used to determine the computational and communication costs related to the distributed deployment of the model based on the hardware topology diagram and the model computation diagram, and generate cost evaluation results. The strategy generation module is used to generate a basic partitioning scheme based on the cost evaluation result using a greedy algorithm, and to adjust and optimize the basic partitioning scheme using an iterative redistribution algorithm to generate an initial partitioning strategy. The runtime status monitoring module is used to collect real-time runtime data of each GPU during the training process of applying the initial splitting strategy, perform timestamp alignment and integration, and generate runtime status data. The re-optimization decision module is used to analyze performance deviations based on the operating status data, and generate a re-optimization trigger signal when the performance deviations meet the re-optimization conditions. The strategy update module is used to respond to the re-optimization trigger signal, update the cost evaluation result using the running status data, and generate an updated segmentation strategy based on the updated cost evaluation result.

[0046] To verify the feasibility of this invention in practice, it was applied to a large-scale AI model training center. This center needs to perform distributed training on a large language model containing hundreds of billions of parameters on a multi-GPU server equipped with eight heterogeneous GPUs. Traditional parallel partitioning schemes rely on manual configuration, are time-consuming, and struggle to adapt to dynamic load changes during training, often leading to uneven GPU resource utilization and communication bottlenecks, severely impacting training efficiency. The center aims to use the method of this invention to achieve efficient, automated parallel deployment and dynamic optimization of the model.

[0047] In this embodiment, the training center first uses this invention to obtain the hardware connection parameters of the 8-GPU server and the layer structure parameters of the large language model. The system identifies the NVLink and PCIe connection types between GPUs and obtains the bandwidth, constructing a hardware topology graph; simultaneously, it parses the model structure, extracts the computational cost, parameter scale, and data dependencies of each Transformer layer, and constructs a model computation graph. Based on this, the cost evaluation module generates cost evaluation results, quantifying the estimated computation time and inter-layer communication overhead of each model layer deployed on different GPUs.

[0048] Subsequently, based on the cost evaluation results, the strategy generation module generated an initial partitioning strategy using a greedy algorithm and an iterative reallocation algorithm. This strategy comprehensively considered computational and communication costs and rationally allocated different parts of the model across the eight GPUs while satisfying the memory capacity constraints of each GPU. On a certain training task, the model began training according to this initial partitioning strategy.

[0049] During training, the runtime monitoring module deployed monitoring agents on each GPU to collect real-time data on computational utilization, memory usage, and actual bandwidth utilization of cross-GPU communication links, aligning the data with timestamps to generate runtime status data. Approximately two hours after training began, the re-optimization decision module analyzed the runtime data and found that some GPUs responsible for computationally intensive operations consistently maintained utilization above 95%, while others hovered around 60%. The calculated load balancing utilization variance was 0.09, exceeding the preset threshold of 0.05. The system determined that the re-optimization conditions were met and generated a re-optimization trigger signal.

[0050] In response to this signal, the policy update module extracted the runtime status data from the time window prior to the trigger, used the recorded actual computation and communication times to calibrate the cost evaluation results, and generated updated cost evaluation results. Based on this more accurate cost data, the system, starting with the current splitting strategy, applied an iterative reallocation algorithm to evaluate multiple candidate migration operations. Ultimately, it was determined that migrating a model layer from the overloaded GPU-1 to the less overloaded GPU-4 would bring the greatest global performance improvement. At the synchronization point of the next training iteration, the system executed this migration operation, transferring the relevant model parameter states from the source GPU to the target GPU, dynamically modifying the data flow dependencies of the computation graph, generating an updated splitting strategy, and taking effect immediately.

[0051] Data shows that the overall performance of the training system has been improved through the dynamic optimization of this invention. The problem of uneven load between GPUs has been effectively alleviated, and the average utilization rate has increased from 78% under the initial strategy to 91%. The training throughput has also increased from processing 45 samples per second to 52 samples per second, with an overall efficiency improvement of approximately 15.6%.

[0052] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0053] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A method for automatic parallel partitioning of AI models for multi-GPU servers, characterized in that, The method includes: Obtain the hardware connection parameters of the multi-GPU server and the layer structure parameters of the AI ​​model, and construct the hardware topology diagram and the model computation diagram; Based on the hardware topology graph and the model computation graph, the computational and communication costs associated with the distributed deployment of the model are determined, and a cost evaluation result is generated. This generation of the cost evaluation result includes: calculating the communication paths between GPU nodes based on the hardware topology graph and quantifying the communication costs of these paths to generate a communication cost matrix; parsing the model computation graph to extract computational parameters and parameter scale parameters for each model layer, forming a model layer attribute set; calculating the estimated computation time for each model layer deployed on different GPUs based on the communication cost matrix and the model layer attribute set; and integrating the estimated computation time for all model layers and the communication time caused by inter-layer dependencies to generate the cost evaluation result. Based on the cost evaluation results, a basic partitioning scheme is generated using a greedy algorithm, and an iterative redistribution algorithm is applied to adjust and optimize the basic partitioning scheme to generate an initial partitioning strategy. During the training process using the initial splitting strategy, real-time running data from each GPU is collected, timestamp aligned, and integrated to generate running status data. Based on the analysis of performance deviations from the operational status data, when the performance deviations meet the re-optimization conditions, a re-optimization trigger signal is generated. The generation of the re-optimization trigger signal includes: extracting and calculating utilization and actual bandwidth utilization of the communication link from the operational status data; calculating a utilization variance index characterizing the degree of load balancing; obtaining a utilization variance threshold and a bandwidth utilization threshold to determine whether performance deviates from the normal range; and generating a re-optimization trigger signal when the utilization variance index exceeds the utilization variance threshold or the actual bandwidth utilization exceeds the bandwidth utilization threshold. In response to the re-optimization trigger signal, the cost evaluation result is updated using the running status data, and an updated partitioning strategy is generated based on the updated cost evaluation result. The generation of the updated partitioning strategy based on the updated cost evaluation result includes: taking the currently executing partitioning strategy as a starting point, and based on the updated cost evaluation result, applying an iterative redistribution algorithm to evaluate the improvement effect of migrating the model layer from the source GPU to the target GPU on the global optimization objective function, generating candidate migration operations; selecting migration operations from the candidate migration operations whose improvement effect meets predetermined conditions and satisfies hardware constraints, executing the migration, and generating the updated partitioning strategy.

2. The automatic parallel segmentation method for AI models on multi-GPU servers according to claim 1, characterized in that, The construction of the hardware topology diagram and model computation diagram includes: Identify the physical connection types between GPUs in a multi-GPU server and obtain bandwidth parameters to generate connection relationship data; Based on the connection relationship data, a hardware topology graph is constructed with GPUs as nodes and physical connections as edges; Analyze the AI ​​model structure, extract the computational operation types, computational parameters, parameter scales, and inter-layer data flow dependencies of each model layer, and generate layer structure parameters. Based on the layer structure parameters, a directed acyclic graph is constructed as the model computation graph, with model layers as nodes and data flow dependencies as edges.

3. The automatic parallel segmentation method for AI models on multi-GPU servers according to claim 2, characterized in that, The initial segmentation strategy includes: Based on the cost evaluation results, a multi-objective optimization function is constructed; Based on the multi-objective optimization function, a greedy algorithm is used to traverse the model layers in the model computation graph, select GPUs for deployment in each layer, and generate a basic partitioning scheme. An iterative redistribution algorithm is applied to the basic partitioning scheme to adjust it under the conditions of satisfying GPU memory capacity constraints and communication bandwidth constraints, thereby generating an initial partitioning strategy.

4. The automatic parallel segmentation method for AI models on multi-GPU servers according to claim 3, characterized in that, The generated runtime status data includes: Training is performed using the initial partitioning strategy described above, and monitoring agents are deployed on each GPU to collect computational utilization data and memory usage data. By monitoring the underlying communication library, the amount of cross-GPU communication data is obtained, and the actual bandwidth utilization of the communication link is calculated. The computational utilization data, the video memory usage data, and the actual bandwidth usage are timestamped and integrated to generate runtime status data.

5. The automatic parallel segmentation method for AI models on multi-GPU servers according to claim 4, characterized in that, The step of updating the cost assessment result using the operational status data includes: Receive the re-optimization trigger signal and extract the running status data within the time window preceding the trigger time; Using the actual computation time and communication time from the operational status data, the estimated computation time and communication time of the cost evaluation results are calibrated to generate updated cost evaluation results.

6. The automatic parallel segmentation method for AI models on multi-GPU servers according to claim 5, characterized in that, The method further includes: At the synchronization point of the training iteration, the model layers that need to be transferred are identified based on the difference between the updated splitting strategy and the current strategy. The model parameter states of the model layers that need to be migrated are transferred from the source GPU to the target GPU, the data flow dependencies in the model computation graph are dynamically modified to match the new layer positions, and the changes take effect at the start of the next training iteration.

7. An automatic parallel segmentation system for AI models on multi-GPU servers, applied to the automatic parallel segmentation method for AI models on multi-GPU servers as described in any one of claims 1-6, characterized in that, The system includes: The graph construction module is used to obtain the hardware connection parameters of the multi-GPU server and the layer structure parameters of the AI ​​model, and to construct the hardware topology graph and the model computation graph. The cost evaluation module is used to determine the computational and communication costs related to the distributed deployment of the model based on the hardware topology graph and the model computation graph, and generate cost evaluation results. The generation of cost evaluation results includes: calculating the communication paths between GPU nodes based on the hardware topology graph and quantifying the communication costs of the communication paths to generate a communication cost matrix; parsing the model computation graph and extracting the computational quantity parameters and parameter scale parameters of each model layer to form a model layer attribute set; calculating the estimated computation time for each model layer deployed on different GPUs based on the communication cost matrix and the model layer attribute set; and integrating the estimated computation time of all model layers and the communication time caused by inter-layer dependencies to generate cost evaluation results. The strategy generation module is used to generate a basic partitioning scheme based on the cost evaluation result using a greedy algorithm, and to adjust and optimize the basic partitioning scheme using an iterative redistribution algorithm to generate an initial partitioning strategy. The runtime status monitoring module is used to collect real-time runtime data of each GPU during the training process of applying the initial splitting strategy, perform timestamp alignment and integration, and generate runtime status data. The re-optimization decision module is used to analyze performance deviations based on the operating status data. When the performance deviations meet the re-optimization conditions, a re-optimization trigger signal is generated. The generation of the re-optimization trigger signal includes: extracting and calculating the utilization rate and the actual bandwidth utilization rate of the communication link from the operating status data; calculating the utilization rate variance index, which characterizes the degree of load balancing; obtaining the utilization rate variance threshold and the bandwidth utilization rate threshold, which are used to determine whether the performance deviates from the normal range; and generating the re-optimization trigger signal when the utilization rate variance index exceeds the utilization rate variance threshold or the actual bandwidth utilization rate exceeds the bandwidth utilization rate threshold. The strategy update module is used to respond to the re-optimization trigger signal, update the cost evaluation result using the running status data, and generate an updated partitioning strategy based on the updated cost evaluation result. The generation of the updated partitioning strategy based on the updated cost evaluation result includes: taking the currently executing partitioning strategy as a starting point, and based on the updated cost evaluation result, applying an iterative redistribution algorithm to evaluate the improvement effect of migrating the model layer from the source GPU to the target GPU on the global optimization objective function, generating candidate migration operations; selecting migration operations from the candidate migration operations whose improvement effect meets predetermined conditions and satisfies hardware constraints, executing the migration, and generating the updated partitioning strategy.

Images