Distributed large model training environment construction system based on heterogeneous computing power pooling

Abstract

This invention discloses a distributed large-scale model training environment construction system based on heterogeneous computing power pooling, relating to the field of resource scheduling technology. It includes: a task profiling module, which receives large-scale model training task requests, constructs a multi-dimensional task resource profile vector based on historical training operation data, and combines it with a computing power capability vector matrix to form a supply-demand matching data structure; an evolution module, which performs resource demand evolution on the supply-demand matching data structure, generating the resource demand time-varying relationship and dynamic resource matching matrix corresponding to different training stages of the training task; and an execution module, which performs comprehensive constraint decisions based on the resource demand time-varying relationship and the dynamic resource matching matrix, generating a distributed training execution orchestration plan; simultaneously, it collects the operating status of heterogeneous computing power nodes, generating a real-time computing power operating status vector; this improves training throughput efficiency and environment construction stability, ensuring the long-term and efficient operation of distributed large-scale model training.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a system for building a distributed large-scale model training environment based on heterogeneous computing power pooling. Background Technology

[0002] With the evolution of deep neural network architectures, especially the explosive growth in computing power demands of ultra-large-scale parameter models centered around Transformers, distributed parallel training has become a fundamental infrastructure support for the field of artificial intelligence. Currently, mainstream training environment construction technologies mostly rely on homogeneous computing power clusters, using integrated communication libraries (such as NCCL) to perform gradient synchronization and parameter updates between acceleration chips with a unified architecture. Related research focuses on achieving virtualization and elastic scaling of hardware resources through computing power pooling technology, and using container orchestration systems (such as Kubernetes) to schedule computing nodes, aiming to improve resource utilization in multi-tenant environments, and has initially formed a unified management and task distribution system for high-performance computing resources.

[0003] However, in real-world large-scale training scenarios, existing computing power pooling scheduling schemes still exhibit significant limitations when dealing with the collaborative work of heterogeneous computing power nodes across architectures and vendors. Specifically, due to the inherent performance heterogeneity of different acceleration chips in terms of floating-point computing power, memory bandwidth, and interconnect topology, existing static scheduling algorithms struggle to accurately quantify the dynamic mapping relationship between task requirements and hardware capabilities. This leads to a high likelihood of mismatch between computing power supply and task load at different stages of model training, resulting in significant communication deficits and computing power stalling issues. Consequently, the overall throughput efficiency of the distributed training environment cannot increase linearly with the increase in node scale. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a distributed large-scale model training environment construction system based on heterogeneous computing power pooling to solve the problem of supply and demand mismatch in the training stage caused by the difficulty in quantifying and mapping the heterogeneity of heterogeneous resource performance.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a system for building a distributed large-scale model training environment based on heterogeneous computing power pooling, which includes: The modeling module encodes the capabilities of heterogeneous computing power nodes to form computing power capability vectors, and constructs a heterogeneous computing power capability vector pool and a computing power capability vector matrix. The task profiling module receives large-scale model training task requests, constructs a multi-dimensional task resource profiling vector based on historical training operation data, and combines it with the computing power vector matrix to form a supply and demand matching data structure. The evolution module performs resource demand evolution on the supply and demand matching data structure, generating the time change relationship of resource demand for training tasks at different training stages and a dynamic resource matching matrix. The execution module makes comprehensive constraint decisions based on the relationship between resource demand changes over time and the dynamic resource matching matrix, and generates a distributed training execution orchestration plan; at the same time, it collects the running status of heterogeneous computing power nodes, generates a real-time computing power running status vector, and aggregates it with spatiotemporal attributes to form a status dataset. The optimization module detects deviations between the status dataset and the time-varying relationship of resource requirements. When the deviation threshold is exceeded, adaptive rescheduling is triggered, and the task resource profile vector is updated based on the rescheduling data.

[0007] Preferably, the method for forming the computing power vector includes: Obtain the static physical parameters and real-time operating indicators of heterogeneous computing power nodes; map them to the computing power equivalence metric matrix for normalization processing to obtain the nominal computing power value in a unified unit, standardized storage and communication characteristics; encapsulate them into a computing power capability vector according to a predetermined dimension.

[0008] Preferably, the method for constructing the heterogeneous computing power capability vector pool and computing power capability vector matrix includes: Cluster the computing power capability vectors, divide the heterogeneous computing power nodes into logical resource groups, and form a hierarchical heterogeneous computing power capability vector pool. Extract the computing power capability vectors from the heterogeneous computing power capability vector pool, represent them in matrix form according to the physical topology order, and generate a computing power capability vector matrix.

[0009] Preferably, the method for constructing the multi-dimensional task resource profile vector includes: Parse the large model training task request and retrieve and extract the corresponding historical training operation data from the historical large model operation log library; We perform weighted calculations on historical training operation data, extract nominal feature values ​​of resource requirements, and encapsulate them into a multi-dimensional task resource profile vector.

[0010] Preferably, the method for forming the supply and demand matching data structure includes: Based on the number of column dimensions of the computing power vector matrix, the multidimensional task resource profile vector is expanded to the same dimension, so that the multidimensional task resource profile vector is aligned with the dimension index of the computing power vector matrix. The multidimensional task resource profile vector after dimension expansion and alignment is matched and measured with the computing power vector matrix to determine the correlation degree between the multidimensional task resource profile vector and each row vector in the computing power vector matrix, and a matching correlation score sequence is generated. The matching association score sequence is used as the association weight, and combined with the task resource constraints represented by the multi-dimensional task resource profile vector, it is mounted into the attribute topology composed of the multi-dimensional task resource profile vector and the computing power capability vector matrix to form a supply and demand matching data structure.

[0011] Preferably, the method for generating the time-varying relationship of resource requirements and the dynamic resource matching matrix corresponding to different training stages of the training task includes: The task resource constraints in the supply and demand matching data structure are analyzed, and the resource consumption is predicted and decomposed according to the forward propagation, backward propagation and gradient communication stages of the large model training task. The time occupied by each stage for nominal computing power, standardized storage and communication characteristics is determined. The decomposed resource consumption prediction data of each stage is mapped to the time axis, and the fluctuation curve of task resource constraints in the time dimension is smoothly fitted to form the time change relationship of resource demand for the entire cycle of the large model training task. By combining the temporal changes in resource demand with the supply-demand correlation weights in the supply-demand matching data structure, the computing power capability vector matrix is ​​state-trimmed along the time step dimension. The heterogeneous computing power node capability vectors that meet the performance boundaries are recombined with the task resource constraints at the corresponding time point to generate a dynamic resource matching matrix.

[0012] Preferably, the method for generating the distributed training execution orchestration plan includes: The extreme values ​​of resource demand are extracted from the temporal relationship of resource demand, and the real-time availability boundary is extracted from the dynamic resource matching matrix to construct the decision space; Within the decision space, the relationship between resource demand changes over time is used as the time-series benchmark. By comparing the affinity values ​​of heterogeneous computing power nodes in the dynamic resource matching matrix, candidate heterogeneous computing power node links that meet physical limits and have the best communication overhead are retrieved. The combination of heterogeneous computing power nodes and the resource allocation schedule are determined, and the comprehensive constraint decision is completed to form the decision result. The code execution logic and decision results of large-scale model training tasks are mapped, and the time sequence is encapsulated according to the cluster management protocol format to generate a distributed training execution orchestration plan.

[0013] Preferably, the method for forming the state dataset includes: Collect real-time load parameters and hardware health indicators of heterogeneous computing power nodes in the execution state; form a real-time computing power operation status vector through vectorization mapping; The real-time computing power operation status vector is associated with timestamp information and spatial index features of heterogeneous computing power nodes to form a status dataset with spatiotemporal attributes.

[0014] Preferably, the method for triggering adaptive rescheduling when the deviation threshold is exceeded includes: Extract the real-time computing power operation state vector at the corresponding moment from the state dataset, and perform a difference mapping with the resource demand estimate at the corresponding moment in the resource demand time change relationship to determine the real-time offset degree. When the weighted sum of the real-time offset exceeds the deviation threshold, the resource demand evolution is performed by re-calling the supply and demand matching data structure, triggering adaptive rescheduling and generating rescheduling operation data.

[0015] Preferably, the method of updating the task resource profile vector based on rescheduling operation data includes: Collect resource consumption characteristics from rescheduled operation data and incrementally supplement the historical large model operation log library by combining task configuration feature groups; Based on the updated historical large model runtime log library, the nominal feature values ​​are recalculated to achieve closed-loop update of the multi-dimensional task resource profile vector.

[0016] The beneficial effects of this invention are as follows: By performing equivalence measurement mapping and matrix representation on heterogeneous computing power nodes, the standardized alignment of hardware performance of different architectures is achieved, solving the problem of fragmented allocation caused by the unmeasurability of heterogeneous resources and improving the resource utilization rate of computing power pooling. On this basis, by performing staged consumption decomposition and time-series smoothing fitting on training tasks, a technical leap from static allocation to dynamic time-series evolution of resource requirements is achieved, effectively coping with the characteristics of drastic fluctuations in the training load of large models and greatly reducing communication latency and computing power stranding losses. Combining comprehensive constraint decision-making and adaptive rescheduling feedback, a closed-loop management system from perception, evolution to self-correction is constructed. Through incremental completion of rescheduling data, continuous iteration of profile accuracy is achieved, improving training throughput efficiency and environment construction stability in complex heterogeneous environments, and ensuring the long-term and efficient operation of distributed large model training. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the distributed large model training environment construction system based on heterogeneous computing power pooling in this invention.

[0019] Figure 2 This is a flowchart illustrating the generation of the computing power capability vector matrix in this invention.

[0020] Figure 3 This is a flowchart of the process for generating a dynamic resource matching matrix in this invention.

[0021] Figure 4 This is a flowchart of the closed-loop update of the multi-dimensional task resource profile vector in this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 This is one embodiment of the present invention, which provides a system for building a distributed large model training environment based on heterogeneous computing power pooling, including the following steps:

[0026] The methods for forming computing power vectors include: Obtain the static physical parameters and real-time operating indicators of heterogeneous computing power nodes; map them to the computing power equivalence metric matrix for normalization processing to obtain the nominal computing power value in a unified unit, standardized storage and communication characteristics; encapsulate them into a computing power capability vector according to a predetermined dimension.

[0027] Specifically, the theoretical peak performance of the acceleration chip, physical capacity of the video memory, and physical link bandwidth contained in the static physical parameters of the heterogeneous computing power nodes are extracted through the hardware abstraction layer interface, and the real-time load rate of the acceleration chip, the occupied space of the video memory, and the real-time utilization rate of the link are collected simultaneously in the real-time operation indicators of the heterogeneous computing power nodes. The theoretical peak performance and real-time load rate of the accelerator chip are substituted into the computing power equivalence measurement matrix. The performance differences of different chip architectures are offset by the preset architecture conversion coefficient in the computing power equivalence measurement matrix. The nominal computing power value of a unified unit reflecting the actual computing power of heterogeneous computing power nodes is obtained through linear scaling calculation. Simultaneously, the physical capacity of the video memory and the occupied space of the video memory are converted into standardized storage in percentage form through normalization processing, and the physical link bandwidth and link real-time utilization rate are converted into standardized communication characteristics reflecting communication capabilities through normalization processing. The nominal computing power value, standardized storage, standardized communication characteristics and real-time operation indicators are concatenated according to a predetermined dimension order, which is computing dimension, storage dimension, communication dimension and load dimension, and encapsulated to form a computing power capability vector of heterogeneous computing power node performance distribution. It should be noted that the underlying hardware description files of heterogeneous computing nodes are accessed to obtain performance benchmark data, which is then used as the initialization parameter for the computing power equivalence metric matrix. The measured execution time of different chip architectures is obtained by running a set of standard operators on the heterogeneous computing nodes. The ratio of the measured execution time to the execution time of the standard reference chip is used as a preset architecture conversion coefficient. For example, the value is 1.2 for NVIDIA architecture chips and 0.95 for Ascend architecture chips. The preset architecture conversion coefficient is then filled into the diagonal position of the computing power equivalence metric matrix to complete the construction of the computing power equivalence metric matrix.

[0028] The methods for constructing heterogeneous computing power vector pools and computing power vector matrices include: Cluster the computing power capability vectors, divide the heterogeneous computing power nodes into logical resource groups, and form a hierarchical heterogeneous computing power capability vector pool.

[0029] It should be noted that a predetermined number of vector center points are randomly selected, and the Euclidean distance between each computing power vector and each vector center point is calculated. The computing power vectors are then assigned to the cluster to which the vector center point with the smallest Euclidean distance belongs. The vector center point is recalculated based on the mean of all vectors within the cluster. This process is repeated iteratively until the center point position no longer shifts. Heterogeneous computing power nodes with an Euclidean distance less than a preset similarity threshold are identified as having similar nominal computing power values, standardized storage, standardized communication characteristics, and real-time operating indicators. An example preset similarity threshold value is 0.15. The identified heterogeneous computing power nodes are then aggregated into a single set. Based on the physical distribution of the heterogeneous computing power nodes in data center racks, server rooms, and regions, the aggregated set is divided into rack-level resource groups, server room-level resource groups, and region-level resource groups, forming a hierarchical heterogeneous computing power vector pool.

[0030] Extract the computing power capability vectors from the heterogeneous computing power capability vector pool, represent them in matrix form according to the physical topology order, and generate a computing power capability vector matrix.

[0031] Specifically, the physical topology order of heterogeneous computing power nodes is obtained by querying the physical topology tree diagram of the data center management platform. The computing power capability vectors belonging to the rack-level resource group are extracted from the heterogeneous computing power capability vector pool with a hierarchical structure. Each computing power capability vector is arranged horizontally as a row vector according to the physical topology order. The nominal computing power value, standardized storage, standardized communication characteristics and real-time operation indicators inside the computing power capability vector are mapped to the column dimension of the matrix and matrix-based representation processing is performed to generate a computing power capability vector matrix.

[0032] The query data center management platform establishes a communication link with the data center resource scheduling system through an application programming interface, and accesses the resource topology database stored in the background in real time to parse and obtain a hierarchical logical structure tree containing information on computer rooms, racks, and rack positions, thereby determining the coordinate sequence and interconnection hierarchy of heterogeneous computing power nodes in the physical space that are in the execution state.

[0033] Methods for constructing multi-dimensional task resource profile vectors include: Parse the large-scale model training task requests and retrieve and extract the corresponding historical training operation data from the historical large-scale model operation log library.

[0034] Specifically, by receiving the interface and parsing the large model training task request, the model parameter count, training parallel strategy and iteration steps contained in the large model training task request are extracted as task configuration feature groups; using the task configuration feature groups as index keywords, correlation matching retrieval is performed in the historical large model running log library to locate and extract historical training running data that have the same task configuration feature groups as the large model training task request. The construction process of the historical large model operation log library is as follows: by collecting the large model training task requests that have been completed in the past and their corresponding training operation indicators, the model parameter quantity, training parallel strategy and iteration steps of each past task are extracted and encapsulated as index features. At the same time, the actual peak memory usage, gradient synchronization bandwidth consumption and computation operator throughput rate during the execution of the past task are recorded. The index features and the associated training operation indicators are stored in a non-relational database in the form of key-value pairs to form the historical large model operation log library.

[0035] We perform weighted calculations on historical training operation data, extract nominal feature values ​​of resource requirements, and encapsulate them into a multi-dimensional task resource profile vector.

[0036] Specifically, the process involves obtaining the record generation time of each historical operation record in the historical training operation data, calculating the absolute time difference between the record generation time and the current system time, substituting the absolute time difference into a preset exponential decay formula (with the example value based on the base of the natural logarithm and the ratio of the absolute time difference to the preset half-life constant as the negative exponent), and calculating a value reflecting the timeliness of the historical operation record, which serves as the time decay factor for various indicators in the historical training operation data.

[0037] By using the time decay factor, the historical peak memory usage, historical gradient synchronization bandwidth consumption, and historical computation operator throughput rate contained in the historical training operation data are multiplied, summarized, and averaged to extract the nominal feature value reflecting the resource requirements of the large model training task. According to the predetermined dimensional order consistent with the computing power capability vector, namely computing dimension, storage dimension, communication dimension, and load dimension, the nominal feature value is numerically serialized and encapsulated to construct a multi-dimensional task resource profile vector.

[0038] The ways to form a supply and demand matching data structure include: Based on the number of column dimensions in the computing power vector matrix, the multidimensional task resource profile vector is expanded to the same dimension, so that the multidimensional task resource profile vector is aligned with the dimension index of the computing power vector matrix.

[0039] Specifically, by obtaining the number of column dimensions of the computing power capability vector matrix, the number of heterogeneous computing power nodes contained in the computing power capability vector matrix is ​​identified. The multi-dimensional task resource profile vector is used as the original vector, and the matrix copy operator is used to repeatedly stack it along the row direction. This makes the number of rows of the multi-dimensional task resource profile vector after repeated stacking equal to the number of rows of the computing power capability vector matrix, and the column dimension indicators are completely consistent with the computing power capability vector matrix in terms of computing dimension, storage dimension, communication dimension, and load dimension. This completes the dimensional expansion of the multi-dimensional task resource profile vector and realizes the alignment of the dimensional indicators of the multi-dimensional task resource profile vector and the computing power capability vector matrix.

[0040] The multidimensional task resource profile vector after dimension expansion and alignment is matched and measured with the computing power vector matrix to determine the correlation degree between the multidimensional task resource profile vector and each row vector in the computing power vector matrix, and a matching correlation score sequence is generated.

[0041] Specifically, the multidimensional task resource profile vector and the computing power vector matrix after dimension expansion and alignment are input into the cosine similarity calculation operator. The cosine value of the angle between each row vector in the multidimensional task resource profile vector and the corresponding row vector in the computing power vector matrix is ​​calculated. The obtained cosine value is defined as the correlation value, which reflects the degree of matching between the performance distribution of heterogeneous computing power nodes and the resource requirements of large model training tasks. The correlation values ​​are linearly arranged according to the row index order of the computing power vector matrix to generate a matching correlation score sequence.

[0042] The matching association score sequence is used as the association weight, and combined with the task resource constraints represented by the multi-dimensional task resource profile vector, it is mounted into the attribute topology composed of the multi-dimensional task resource profile vector and the computing power capability vector matrix to form a supply and demand matching data structure.

[0043] Specifically, a bidirectional graph structure containing task nodes and resource nodes is constructed. The multi-dimensional task resource profile vector is used as the vertex attribute of the task node, and the computing power vector matrix is ​​used as the vertex attribute of the resource node. Large model parameters are extracted from the multi-dimensional task resource profile vector, and the minimum space required to store the model weights is calculated based on a preset floating-point precision byte count (2 bytes for 16-bit floating-point in the example), serving as the minimum requirement for GPU memory. Simultaneously, tensor parallelism and pipeline parallelism dimensions are extracted from the training parallelism strategy, and combined with a preset communication volume formula, the minimum bandwidth expression for cross-node data exchange is calculated as follows: ;in This is the minimum bandwidth requirement. It has a large number of model parameters. It refers to the number of bytes for floating-point precision. This refers to the number of computing nodes within the tensor parallel dimension. Specifically, it refers to the total number of heterogeneous computing nodes to which the parameter tensors of a large model network layer are partitioned and mapped within a single tensor parallel group. It is a preset iteration cycle time threshold (0.5 seconds for example), and the minimum bandwidth for cross-node data exchange is taken as the bandwidth baseline requirement and uniformly encapsulated as task resource constraints; The numerical values ​​in the matching association score sequence are mapped to the weights of the connection edges between task nodes and each resource node. The task resource constraints, task node attributes, resource node attributes, and connection edge weights are uniformly mounted into the attribute topology represented by the bidirectional graph structure to form a supply and demand matching data structure.

[0044] Existing technologies primarily employ a static resource allocation model, locking resources once based on estimated peak values. This approach ignores the significant temporal differences in computing power, memory, and bandwidth requirements during the forward propagation, backpropagation, and gradient communication phases of large-scale model training. This leads to severe resource waste during off-peak periods or task crashes in core computation phases due to insufficient resource allocation. Furthermore, the inability to capture the dynamic fluctuations in memory release and gradient synchronization makes it difficult to achieve refined peak shaving and valley filling in heterogeneous environments, severely limiting the resource throughput and parallel efficiency of heterogeneous computing pools. Therefore, this invention designs a dynamic scheduling mechanism based on temporal evolution characteristics, specifically through refined prediction and fitting of resource consumption throughout the entire training cycle, as follows: The methods for generating the time-varying relationship of resource requirements for training tasks at different training stages and the dynamic resource matching matrix include: The task resource constraints in the supply and demand matching data structure are analyzed, and the resource consumption is predicted and decomposed according to the forward propagation, backward propagation and gradient communication stages of the large model training task. The time occupied by the nominal computing power, standardized storage and communication characteristics of each stage is determined.

[0045] Specifically, task resource constraints are extracted from the supply and demand matching data structure. Based on the structural characteristics of the large-scale model training task, the computational complexity is estimated during the forward propagation phase using the number of large-scale model parameters and a pre-defined operator computational complexity formula. The expression for calculating the computational complexity is as follows: ;in Forward propagation computation For large model parameter count, To determine the total number of features in the training data batch, the duration of time occupied for the nominal computing power value is determined by dividing the forward propagation computation by the nominal computing power value in the computing power capability vector. At the same time, the duration of time occupied for standardized storage is determined based on the activation value storage requirements. During the backpropagation phase, the secondary occupancy time of the nominal computing power is predicted based on the backpropagation computational load. The calculation of the backpropagation computational load is expressed as follows: ;in The computational cost of backpropagation (including gradient computation and error backpropagation computation) is determined by combining the pre-set memory release logic to determine the memory release rhythm of standardized storage after the gradient calculation of each layer is completed during backpropagation. The pre-set memory release logic is as follows: the gradient calculation state of each layer of neurons is identified in reverse order according to the computation graph of the large model training task. After the gradient vector of the current layer of neurons is calculated, the memory reclamation operator is immediately triggered to release the forward propagation activation value tensor corresponding to the layer, and the storage cost of updating the gradient tensor of the layer in the standardized storage is updated synchronously. During the gradient communication phase, the communication time required for gradient synchronization is calculated using the minimum bandwidth expression for cross-node data exchange. The expression for calculating the communication time is as follows: ;in For communication time, For floating-point precision in bytes, This refers to the actual number of tensor parallel computing nodes allocated in the dynamic resource matching matrix. The real-time bandwidth value of the standardized communication feature corresponding to the supply and demand matching data structure is physically consistent with the expression of communication time consumption and the expression of minimum bandwidth. By replacing the minimum bandwidth requirement with the real-time bandwidth value, the mapping from the demand constraint space to the actual evolution time domain is realized, thereby determining the duration of occupation of the standardized communication feature.

[0046] The decomposed resource consumption prediction data for each stage is mapped to the time axis, and the fluctuation curve of task resource constraints in the time dimension is smoothly fitted to form the time change relationship of resource demand for the entire cycle of the large model training task.

[0047] Specifically, the resource consumption prediction data of the forward propagation, backward propagation and gradient communication stages are arranged in the order of execution and mapped to the time axis. The instantaneous resource demand jump value at each stage switching point is obtained. The cubic spline interpolation algorithm is used to smoothly fit the fluctuation curve of task resource constraints in the time dimension, eliminate the numerical mutation caused by stage switching, and form the time change relationship of resource demand covering the entire cycle of large model training tasks.

[0048] By combining the temporal changes in resource demand with the supply-demand correlation weights in the supply-demand matching data structure, the computing power capability vector matrix is ​​state-trimmed along the time step dimension. The heterogeneous computing power node capability vectors that meet the performance boundaries are recombined with the task resource constraints at the corresponding time point to generate a dynamic resource matching matrix.

[0049] Specifically, the time axis is moved at a preset time step, with an example value of 10 milliseconds. The task resource constraints at the current moment are obtained by combining the relationship between resource demand and time changes. At the same time, the supply and demand correlation weights in the supply and demand matching data structure are retrieved to perform state trimming on the computing power vector matrix. The specific pruning process is as follows: the task resource constraints at the current moment are compared with the real-time operation indicators corresponding to each row vector in the computing power capability vector matrix, and a selection operator vector with the same number of rows as the computing power capability vector matrix is ​​constructed. If the remaining video memory or remaining bandwidth of the heterogeneous computing power node is lower than the task resource constraints at the current moment, the element at the corresponding index position in the selection operator vector is set to 0, otherwise it is set to 1. The selection operator vector is multiplied element-wise with the computing power capability vector matrix, so as to remove the capability vectors of heterogeneous computing power nodes whose remaining video memory or remaining bandwidth is lower than the task resource constraints at the current moment in real time while maintaining the consistency of dimensions, thus completing the state pruning. The heterogeneous computing power node capability vector, which has been state-trimmed and meets the performance boundary, is multiplied with the task resource constraints at the corresponding time to achieve affinity reorganization. The performance boundary example is that the real-time utilization rate of the heterogeneous computing power node's video memory is less than 90% and the real-time utilization rate of the link is less than 85%. A dynamic resource matching matrix reflecting the optimal resource allocation state at each time is generated. By designing resource requirement evolution for large-scale model training tasks, the transformation from static stacking to dynamic evolution of these tasks was achieved. By capturing computational fluctuations and memory release nodes at each training stage, resource reservation overhead during off-peak periods was reduced, effectively preventing training interruptions caused by inaccurate memory predictions. Simultaneously, cubic spline interpolation ensured communication stability during resource switching, deeply optimizing the overall utilization of heterogeneous computing pools and laying an accurate timing foundation for generating high-precision training orchestration plans.

[0050] The methods for generating distributed training execution orchestration plans include: The extreme values ​​of resource demand are extracted from the temporal relationship of resource demand, and the real-time availability boundary is extracted from the dynamic resource matching matrix to construct the decision space.

[0051] Specifically, by traversing the relationship between resource demand changes over time, the peak values ​​of nominal computing power, standardized storage, and standardized communication features covering the entire lifecycle of large-scale model training tasks are identified as extreme values ​​of resource demand. Simultaneously, extract the upper limits of real-time memory utilization and link utilization of each heterogeneous computing node at different time steps from the dynamic resource matching matrix, and use them as real-time availability boundaries; map the extreme values ​​of resource demand and real-time availability boundaries as the boundary range of multi-dimensional constraints to construct a decision space for resource scheduling optimization.

[0052] Within the decision space, using the temporal relationship of resource demand as a time-series benchmark, by comparing the affinity values ​​of heterogeneous computing power nodes in the dynamic resource matching matrix, candidate heterogeneous computing power node links that meet physical limits and have the best communication overhead are retrieved, the combination of heterogeneous computing power nodes and the resource allocation schedule are determined, and the comprehensive constraint decision is completed to form the decision result.

[0053] Specifically, within the decision space, the temporal logic provided by the relationship between resource demand and time changes is used as the vertical benchmark. By comparing the affinity values ​​of heterogeneous computing power nodes represented by each row vector in the dynamic resource matching matrix, a set of heterogeneous computing power nodes with affinity values ​​higher than a preset affinity threshold is selected. An example of the preset affinity threshold is 0.85. The Dijkstra algorithm is used to retrieve candidate heterogeneous computing node links that are physically adjacent and have the optimal communication overhead from a set of heterogeneous computing nodes. Specifically, the heterogeneous computing nodes in the set are treated as graph nodes, and the communication delay and physical topological distance between nodes are used as edge weights. Starting from the initial heterogeneous computing node, the shortest path estimate of adjacent nodes is continuously updated through relaxation operations. Nodes with determined shortest paths are added to the visited set in ascending order of weight, until all nodes in the candidate links are traversed to find the path with the minimum cumulative communication overhead. The entry and release times of each selected heterogeneous computing node are determined based on the resource occupation time of each stage, forming a heterogeneous computing node combination and resource allocation schedule, and completing the comprehensive constraint decision to form the decision result.

[0054] The code execution logic and decision results of large-scale model training tasks are mapped, and the time sequence is encapsulated according to the cluster management protocol format to generate a distributed training execution orchestration plan.

[0055] Specifically, by extracting the operator call sequence and synchronization primitives contained in the code execution logic of the large model training task, specific operators for each stage of the forward propagation, backward propagation and gradient communication tasks are identified from the operator call sequence. The specific operators are derived from the matrix multiplication operator, activation function operator and set communication operator obtained by parsing the computation graph of the deep learning model network structure defined in the request of the large model training task. Specific operators in the code execution logic are logically bound to the resource allocation schedule in the decision results to ensure that specific operators start in sequence on the corresponding heterogeneous computing power node combination. According to the cluster management protocol format, the bound instruction sequence, heterogeneous computing power node identifier and resource quota parameters are encapsulated into a binary stream in time step order to generate a distributed training execution orchestration plan with time-series execution capability.

[0056] The ways to form a state dataset include: Collect real-time load parameters and hardware health indicators of heterogeneous computing power nodes in the execution state; form a real-time computing power operation status vector through vectorization mapping.

[0057] Specifically, the real-time load parameters of the heterogeneous computing nodes in the execution state are read in real time through the hardware abstraction layer interface. The real-time load parameters include the real-time load rate of the acceleration chip, the occupied space of the video memory, and the real-time utilization rate of the link. At the same time, the self-test data of the underlying driver of the heterogeneous computing nodes in the execution state are retrieved to obtain hardware health indicators. The hardware health indicators are specifically derived from the physical status data fed back in real time by the underlying firmware of the acceleration chip, including the acceleration chip temperature, fan speed, and voltage stability status. The real-time load rate of the accelerator chip, the occupied space of the video memory, the real-time utilization rate of the link, and the hardware health indicators are arranged numerically according to a predetermined dimension order, namely the load dimension and the health dimension. The load dimension consists of the real-time load rate of the accelerator chip, the occupied space of the video memory, and the real-time utilization rate of the link, while the health dimension consists of the temperature of the accelerator chip, the fan speed, and the voltage stability status. These are converted into numerical vectors through a mapping function to form a real-time computing power operation status vector.

[0058] The real-time computing power operation status vector is associated with timestamp information and spatial index features of heterogeneous computing power nodes to form a status dataset with spatiotemporal attributes.

[0059] Specifically, the precise moment of the real-time computing power operation status vector generation is obtained and converted into a standard format timestamp for association and encapsulation; the physical topology tree diagram of the data center management platform is retrieved simultaneously, and the region, computer room, rack, and rack number of the heterogeneous computing power node in the execution state are extracted and encoded to obtain the spatial index feature of the heterogeneous computing power node; the record containing the timestamp information, the spatial index feature of the heterogeneous computing power node, and the real-time computing power operation status vector is stored in a distributed time-series database to form a state dataset with spatiotemporal attributes.

[0060] In existing technologies, the monitoring of distributed large-scale model training operations mostly remains at the passive fault-tolerance stage, meaning that simple fault recovery is only performed after a complete hardware failure or memory overflow causes task interruption. This mechanism ignores operational deviations caused by performance fluctuations of computing nodes, network congestion, or hidden faults during training, and cannot provide predictive intervention before task crashes. Furthermore, traditional resource scheduling is often a one-off process, lacking the ability to absorb actual operational data, resulting in a long-term static deviation in task resource profiles. This makes it difficult to self-correct based on real-time feedback from heterogeneous computing environments, limiting the system's robustness under high-pressure operation. Therefore, this invention designs an adaptive rescheduling and profile closed-loop update mechanism based on real-time status feedback, as follows:

[0061] The ways to trigger adaptive rescheduling when the deviation threshold is exceeded include: Extract the real-time computing power operation state vector at the corresponding moment from the state dataset, and perform a difference mapping with the estimated resource demand at the corresponding moment in the resource demand time change relationship to determine the real-time offset degree.

[0062] Specifically, using timestamp information as an index, the real-time computing power operation status vector at the current moment is extracted from the state dataset with spatiotemporal attributes, and the corresponding estimated resource demand value is retrieved from the resource demand time change relationship. The real-time load rate of the acceleration chip, the occupied space of the video memory, and the real-time utilization rate of the link contained in the real-time computing power operation status vector are subtracted item by item from the theoretical demand values ​​of the nominal computing power value, standardized storage, and standardized communication characteristics in the estimated resource demand value to obtain the numerical difference of each dimension. By mapping the numerical difference to the standard score range, the degree of deviation of the actual operation status of the heterogeneous computing power node from the theoretical prediction value is quantified to determine the real-time offset degree.

[0063] When the weighted sum of the real-time offset exceeds the deviation threshold, the resource demand evolution is performed by re-calling the supply and demand matching data structure, triggering adaptive rescheduling and generating rescheduling operation data.

[0064] Specifically, the real-time offset is weighted and summed to calculate a weighted summation value that reflects the stability of the overall task operation status. When the weighted summation value of the real-time offset exceeds the preset deviation threshold (example value is 0.2), it is determined that the current large model training task is facing resource shortage or node failure risk. The large model training task request is immediately re-parsed and the supply and demand matching data structure is called. The affected heterogeneous computing power nodes are removed from the computing power capability vector matrix. The resource occupation time of each training stage is recalculated through resource demand evolution. Adaptive rescheduling is triggered and rescheduling operation data containing new heterogeneous computing power node combinations and new resource allocation schedules is generated.

[0065] The methods for updating task resource profile vectors based on rescheduled runtime data include: Collect resource consumption characteristics from rescheduled operation data and incrementally supplement the historical large model operation log library by combining task configuration feature groups.

[0066] Specifically, during the adaptive rescheduling process, the resource consumption characteristics of heterogeneous computing nodes when actually executing large model training tasks are collected in real time from the rescheduling operation data. Specifically, the peak value of actual video memory usage after rescheduling, the gradient synchronization bandwidth consumption after rescheduling, and the throughput rate of computing operators after rescheduling are extracted.

[0067] The extracted resource consumption features are associated with the task configuration feature group consisting of the number of model parameters, training parallel strategies, and iteration steps extracted from the large model training task request. This group is then encapsulated into new key-value pairs and stored in a non-relational database to achieve incremental completion of the historical large model running log library.

[0068] Based on the updated historical large model runtime log library, the nominal feature values ​​are recalculated to achieve closed-loop update of the multi-dimensional task resource profile vector.

[0069] Specifically, the updated historical large-scale model operation log library is retrieved, and the generation time of all historical training operation data, including rescheduled operation data, is re-acquired. The latest time decay factor of each historical operation record is calculated using a preset exponential decay formula. The latest time decay factor is used to perform a weighted summation and average of the historical peak memory usage, historical gradient synchronization bandwidth consumption, and historical computation operator throughput rate contained in the historical large-scale model operation log library. The nominal feature value reflecting the resource requirements of the large-scale model training task is then recalculated. The recalculated nominal feature value is serialized and encapsulated according to the computation dimension, storage dimension, communication dimension, and load dimension to achieve a closed-loop update of the multi-dimensional task resource profile vector.

[0070] Through adaptive rescheduling and profile updates, a leap from "passive operation and maintenance" to "proactive closed-loop" has been achieved. By quantifying operational deviations in real time, early intervention can be made when potential risks arise on heterogeneous computing power nodes. Adaptive rescheduling ensures the continuity of training tasks and reduces the frequency of task interruptions caused by node fluctuations. By feeding rescheduling data back to the log library and re-encapsulating profile vectors, continuous accumulation of system experience is achieved. This local effect allows the prediction accuracy of task profiles to be continuously optimized with the number of executions, enhancing the system's adaptability to heterogeneous computing power environments and forming a virtuous cycle of feedback between resource allocation and actual operating status. This provides a reliable and stable guarantee for large-scale, long-term model training.

[0071] In summary, this invention achieves standardized alignment of hardware performance across different architectures by performing equivalence measurement mapping and matrix representation on heterogeneous computing power nodes. This solves the fragmentation problem caused by the unmeasurability of heterogeneous resources and improves the resource utilization of computing power pooling. Furthermore, by performing phased consumption decomposition and time-series smoothing fitting on training tasks, it achieves a technological leap from static allocation to dynamic time-series evolution of resource requirements, effectively addressing the drastic fluctuations in large model training loads and significantly reducing communication latency and computing power stalling losses. Combining comprehensive constraint decision-making and adaptive rescheduling feedback, a closed-loop management system from perception and evolution to self-correction is constructed. Through incremental completion of rescheduling data, continuous iteration of profile accuracy is achieved, improving training throughput efficiency and environment construction stability in complex heterogeneous environments, and ensuring the long-term, efficient operation of distributed large model training.

[0072] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A distributed large model training environment construction system based on heterogeneous computing power pooling, characterized in that, include: The modeling module encodes the capabilities of heterogeneous computing power nodes to form computing power capability vectors, and constructs a heterogeneous computing power capability vector pool and a computing power capability vector matrix. The task profiling module receives large-scale model training task requests, constructs a multi-dimensional task resource profiling vector based on historical training operation data, and combines it with the computing power vector matrix to form a supply and demand matching data structure. The evolution module performs resource demand evolution on the supply and demand matching data structure, generating the time change relationship of resource demand for training tasks at different training stages and a dynamic resource matching matrix. The execution module makes comprehensive constraint decisions based on the relationship between resource demand changes over time and the dynamic resource matching matrix, and generates a distributed training execution orchestration plan; at the same time, it collects the running status of heterogeneous computing power nodes, generates a real-time computing power running status vector, and aggregates it with spatiotemporal attributes to form a status dataset. The optimization module detects deviations between the status dataset and the time-varying relationship of resource requirements. When the deviation threshold is exceeded, adaptive rescheduling is triggered, and the task resource profile vector is updated based on the rescheduling data.

2. The distributed large model training environment construction system based on heterogeneous computing power pooling according to claim 1, wherein, The methods for forming the computing power vector include: Obtain the static physical parameters and real-time operating indicators of heterogeneous computing power nodes; map them to the computing power equivalence metric matrix for normalization processing to obtain the nominal computing power value in a unified unit, standardized storage and communication characteristics; encapsulate them into a computing power capability vector according to a predetermined dimension.

3. The distributed large model training environment construction system based on heterogeneous computing power pooling according to claim 2, wherein, The methods for constructing the heterogeneous computing power capability vector pool and computing power capability vector matrix include: Cluster the computing power capability vectors, divide the heterogeneous computing power nodes into logical resource groups, and form a hierarchical heterogeneous computing power capability vector pool. Extract the computing power capability vectors from the heterogeneous computing power capability vector pool, represent them in matrix form according to the physical topology order, and generate a computing power capability vector matrix.

4. The distributed large model training environment construction system based on heterogeneous computing power pooling according to claim 3, wherein, The methods for constructing multi-dimensional task resource profile vectors include: Parse the large model training task request and retrieve and extract the corresponding historical training operation data from the historical large model operation log library; We perform weighted calculations on historical training operation data, extract nominal feature values ​​of resource requirements, and encapsulate them into a multi-dimensional task resource profile vector.

5. The distributed large model training environment construction system based on heterogeneous computing power pooling according to claim 4, wherein, The methods for forming the supply and demand matching data structure include: Based on the number of column dimensions of the computing power vector matrix, the multidimensional task resource profile vector is expanded to the same dimension, so that the multidimensional task resource profile vector is aligned with the dimension index of the computing power vector matrix. The multidimensional task resource profile vector after dimension expansion and alignment is matched and measured with the computing power vector matrix to determine the correlation degree between the multidimensional task resource profile vector and each row vector in the computing power vector matrix, and a matching correlation score sequence is generated. The matching association score sequence is used as the association weight, and combined with the task resource constraints represented by the multi-dimensional task resource profile vector, it is mounted into the attribute topology composed of the multi-dimensional task resource profile vector and the computing power capability vector matrix to form a supply and demand matching data structure.

6. The distributed large model training environment construction system based on heterogeneous computing power pooling according to claim 5, wherein, The methods for generating the time-varying relationship of resource requirements and the dynamic resource matching matrix corresponding to different training stages of the training task include: The task resource constraints in the supply and demand matching data structure are analyzed, and the resource consumption is predicted and decomposed according to the forward propagation, backward propagation and gradient communication stages of the large model training task. The time occupied by each stage for nominal computing power, standardized storage and communication characteristics is determined. The decomposed resource consumption prediction data of each stage is mapped to the time axis, and the fluctuation curve of task resource constraints in the time dimension is smoothly fitted to form the time change relationship of resource demand for the entire cycle of the large model training task. By combining the temporal changes in resource demand with the supply-demand correlation weights in the supply-demand matching data structure, the computing power capability vector matrix is ​​state-trimmed along the time step dimension. The heterogeneous computing power node capability vectors that meet the performance boundaries are recombined with the task resource constraints at the corresponding time point to generate a dynamic resource matching matrix.

7. The distributed large-scale model training environment construction system based on heterogeneous computing power pooling as described in claim 6, characterized in that, The methods for generating distributed training execution orchestration plans include: The extreme values ​​of resource demand are extracted from the temporal relationship of resource demand, and the real-time availability boundary is extracted from the dynamic resource matching matrix to construct the decision space. Within the decision space, the relationship between resource demand changes over time is used as the time-series benchmark. By comparing the affinity values ​​of heterogeneous computing power nodes in the dynamic resource matching matrix, candidate heterogeneous computing power node links that meet physical limits and have the best communication overhead are retrieved. The combination of heterogeneous computing power nodes and the resource allocation schedule are determined, and the comprehensive constraint decision is completed to form the decision result. The code execution logic and decision results of large-scale model training tasks are mapped, and the time sequence is encapsulated according to the cluster management protocol format to generate a distributed training execution orchestration plan.

8. The distributed large-scale model training environment construction system based on heterogeneous computing power pooling as described in claim 7, characterized in that, The methods for forming the state dataset include: Collect real-time load parameters and hardware health indicators of heterogeneous computing power nodes in the execution state; form a real-time computing power operation status vector through vectorization mapping; The real-time computing power operation status vector is associated with timestamp information and spatial index features of heterogeneous computing power nodes to form a status dataset with spatiotemporal attributes.

9. The distributed large-scale model training environment construction system based on heterogeneous computing power pooling as described in claim 8, characterized in that, The methods for triggering adaptive rescheduling when the deviation threshold is exceeded include: Extract the real-time computing power operation state vector at the corresponding moment from the state dataset, and perform a difference mapping with the resource demand estimate at the corresponding moment in the resource demand time change relationship to determine the real-time offset degree. When the weighted sum of the real-time offset exceeds the deviation threshold, the resource demand evolution is performed by re-calling the supply and demand matching data structure, triggering adaptive rescheduling and generating rescheduling operation data.

10. The distributed large-scale model training environment construction system based on heterogeneous computing power pooling as described in claim 9, characterized in that, The method of updating the task resource profile vector based on rescheduled runtime data includes: Collect resource consumption characteristics from rescheduled operation data and incrementally supplement the historical large model operation log library by combining task configuration feature groups; Based on the updated historical large model runtime log library, the nominal feature values ​​are recalculated to achieve closed-loop update of the multi-dimensional task resource profile vector.

Images