Heterogeneous cluster resource allocation method and storage medium

Abstract

The application relates to a heterogeneous cluster resource allocation method and a storage medium, wherein the heterogeneous cluster resource allocation method comprises the following steps: obtaining the computing performance parameters and the communication performance parameters of each computing node, constructing a resource allocation model with the target of minimizing the single training iteration time, and obtaining the target data allocation amount capable of covering the communication time of each computing node based on the resource allocation model. Through the application, the problems of resource idling, waiting for synchronization and the "barrel effect" caused by the mismatch between the computing capacity and the network bandwidth in the heterogeneous cluster under the parameter server architecture are solved, the communication cost is completely hidden in the computing process under the premise that the total training data amount is unchanged, and the overall resource utilization and the distributed training efficiency of the heterogeneous cluster are significantly improved.

Description

Technical Field

[0001] This application relates to the field of distributed machine learning, and in particular to methods for allocating resources in heterogeneous clusters and storage media. Background Technology

[0002] As the scale of deep learning models continues to expand, single computing devices can no longer meet the computational and memory demands of the training process, making distributed training an indispensable technical means. However, in actual industrial deployment environments, computing clusters generally exhibit significant heterogeneity, mainly in two aspects: First, there are significant differences in computing capabilities between nodes, with different models of CPUs, GPUs, or TPUs possibly being deployed in the cluster, each with varying processing speeds and architectural characteristics; second, network bandwidth is asymmetrical, with significant differences in communication link bandwidth between nodes due to factors such as network topology, physical distance, and network device performance.

[0003] Traditional resource allocation methods typically optimize computation and communication separately, lacking a coordination mechanism, which leads to numerous problems: high-performance computing nodes, after quickly completing local computation, must wait for slower-communicating nodes to complete gradient synchronization, resulting in idle computing resources; while high-bandwidth nodes, due to unreasonable computational load allocation, cannot fully utilize their communication advantages, resulting in low communication resource utilization. More seriously, the entire training iteration time is limited by the slowest node, forming a "weakest link" effect, significantly reducing the overall efficiency of the cluster.

[0004] Currently, no effective solution has been proposed for the problem of low training efficiency caused by the ineffective coordination of computing and communication resources in heterogeneous environments. Summary of the Invention

[0005] This application provides a heterogeneous cluster resource allocation method and storage medium to at least solve the problem in related technologies where computing and communication resources fail to coordinate effectively in a heterogeneous environment under a parameter server architecture, resulting in low training efficiency.

[0006] In a first aspect, embodiments of this application provide a heterogeneous cluster resource allocation method, applied to a parameter server architecture, including:

[0007] Obtain computational performance parameters that characterize the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and communication performance parameters that characterize the network communication capabilities of each computing node; the computing node is the working node in the heterogeneous cluster that undertakes distributed training computation tasks.

[0008] Based on the communication performance parameters, the communication time required for each computing node to complete one gradient synchronization is determined.

[0009] Based on the computational performance parameters and the communication time, a resource allocation model is constructed;

[0010] Based on the resource allocation model, the target data allocation amount for each computing node is obtained; wherein, the product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node.

[0011] Based on the target data allocation amount, the training data in the heterogeneous cluster is divided, and the divided data subsets are allocated to the corresponding computing nodes.

[0012] In some embodiments, the resource allocation model is represented as:

[0013] ;

[0014] ;

[0015] ;

[0016] in, This represents the target data allocation amount for the i-th computing node. This represents the computational performance parameters of the i-th computing node. This represents the communication time of the i-th computing node. This represents the total amount of data in a single training iteration; the total amount of data is determined based on the total number of computing nodes in the heterogeneous cluster and the memory capacity of each computing node.

[0017] In some embodiments, the communication time of the i-th computing node is determined in the following way:

[0018] ;

[0019] Where M represents the amount of gradient parameter data of the deep learning model to be trained. and These represent the uplink bandwidth and downlink bandwidth from the i-th computing node to the corresponding parameter server, respectively. The parameter server is a worker node in the heterogeneous cluster that undertakes the function of global parameter aggregation and distribution.

[0020] In some embodiments, obtaining the target data allocation amount for each computing node based on the resource allocation model includes:

[0021] Calculate the theoretical equilibrium time ;

[0022] Compare the theoretical equilibrium time Communication time with the i-th computing node Size;

[0023] if If a balanced allocation strategy is adopted, the target data allocation amount for the i-th computing node is... ;

[0024] if Then based on the constraints and total data volume constraints Based on the resource allocation model, the target data allocation amount is obtained.

[0025] In some embodiments, the if Then based on the constraints and total data volume constraints Based on the resource allocation model, the target data allocation amount is obtained, including:

[0026] The constraint conditions Introduce slack variables to construct the Lagrange function;

[0027] Based on the introduced slack variables, the Lagrangian function is solved using an iterative weighted method to obtain the candidate data allocation amount;

[0028] Verify whether the candidate data allocation quantity meets the total data volume constraint, and determine the candidate data allocation quantity that meets the total data volume constraint as the target data allocation quantity.

[0029] In some embodiments, after allocating the partitioned data subsets to the corresponding computing nodes, the method further includes:

[0030] Based on the data subset, distributed training is performed, and the actual computing time and actual communication time of each computing node in the current process of the distributed training are monitored in real time.

[0031] When the deviation between the actual computing time and the expected computing time obtained based on the computing performance parameters exceeds a first preset threshold, or the deviation between the actual communication time and the expected communication time obtained based on the communication performance parameters exceeds a second preset threshold, the actual computing performance parameters are obtained based on the actual computing time, and a new resource allocation model is constructed based on the actual computing performance parameters and the actual communication time.

[0032] Based on the new resource allocation model, a new target data allocation amount is obtained for each computing node; wherein, the product of the new target data allocation amount for each computing node and the actual computing performance parameter of each computing node is greater than or equal to the actual communication time of each computing node.

[0033] Based on the new target data allocation, the training data in the heterogeneous cluster is divided, and the new data subsets are allocated to the corresponding computing nodes.

[0034] In some embodiments, the real-time monitoring of the actual computation time and actual communication time of each computing node in the current process of the distributed training further includes:

[0035] Count the number of iterations of the distributed training in the current process;

[0036] When the number of execution iterations reaches a preset period threshold, the actual computing time and actual communication time of each computing node are obtained.

[0037] In some embodiments, before obtaining the computing performance parameters characterizing the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and the communication performance parameters characterizing the network communication capabilities of each computing node, the method further includes:

[0038] Perform network topology discovery on the heterogeneous cluster and construct a cluster communication topology graph;

[0039] Based on the cluster communication topology diagram, the parameter server nodes and computing nodes in the heterogeneous cluster are identified, and the connection relationship between each computing node and the parameter server is determined.

[0040] Based on the connection relationship, the communication performance parameters of each computing node are calculated.

[0041] In some embodiments, obtaining computational performance parameters characterizing the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and communication performance parameters characterizing the network communication capabilities of each computing node, includes:

[0042] The computing performance parameters of each computing node are measured by running benchmark tests on each computing node.

[0043] Based on the determined connection relationship between each computing node and the parameter server, a point-to-point bandwidth test is performed between the computing node and the parameter server to measure the communication performance parameters of each computing node, including uplink bandwidth and downlink bandwidth.

[0044] Secondly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the heterogeneous cluster resource allocation method as described in the first aspect above.

[0045] Compared to related technologies, the heterogeneous cluster resource allocation method provided in this application is applied to a parameter server architecture. By accurately obtaining the computing performance parameters and communication performance parameters of each computing node, a resource allocation model is constructed with the goal of minimizing the time of a single training iteration. Based on the resource allocation model, the target data allocation amount is obtained so that the computing time of each computing node can cover its communication time. This solves the problems of resource idleness, waiting for synchronization, and the "barrel effect" caused by the mismatch between computing power and network bandwidth in heterogeneous clusters under the parameter server architecture. It achieves that, under the premise that the total amount of training data remains unchanged, the communication overhead is completely hidden in the computing process, which significantly improves the overall resource utilization and distributed training efficiency of the heterogeneous cluster.

[0046] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a hardware structure block diagram of a heterogeneous cluster resource allocation method according to an embodiment of this application;

[0049] Figure 2 This is a flowchart of a heterogeneous cluster resource allocation method according to an embodiment of this application;

[0050] Figure 3 This is a structural block diagram of a heterogeneous cluster resource allocation device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiments provided in this application can be executed in a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the heterogeneous cluster resource allocation method according to an embodiment of this application. Figure 1 The terminal may include a processor 102, an internal bus 104, a network interface 106, memory 108, and non-volatile memory 110, and may also include other hardware required for the services. The processor 102 reads the corresponding computer program from the non-volatile memory 110 into memory and then runs it to implement the heterogeneous cluster resource allocation method. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0055] This embodiment provides a method for allocating resources in a heterogeneous cluster. In this application, "resource allocation" refers to the optimized allocation of training data as a key, controllable resource. Figure 2 This is a flowchart of a heterogeneous cluster resource allocation method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0056] Step S201: Obtain the computing performance parameters that characterize the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and the communication performance parameters that characterize the network communication capabilities of each computing node; the aforementioned computing nodes are the working nodes in the heterogeneous cluster that undertake distributed training computing tasks.

[0057] In the parameter server architecture, the nodes in the heterogeneous cluster are divided into compute nodes and parameter server nodes, which are interconnected through a high-speed network. The compute nodes are used to execute distributed training tasks in parallel, including forward propagation, back propagation, and local gradient calculation. The parameter server nodes are used to perform tasks such as global gradient aggregation, model parameter updating, and distribution.

[0058] Specifically, for heterogeneous clusters under a parameter server architecture, the first step is to obtain two types of hardware performance parameters for each computing node: computing performance parameters and communication performance parameters.

[0059] Among them, the computing performance parameter is used to characterize the computing time required for each computing node to process a unit of data (such as a sample or a minimum batch). It represents the absolute computing power of each computing node and is usually related to the model, frequency, architecture and current load of the processor (CPU / GPU / TPU).

[0060] Communication performance parameters are used to characterize the network communication capabilities of each computing node under the above parameter server architecture, and are usually reflected in the uplink and downlink bandwidth from each computing node to the parameter server.

[0061] In some embodiments, before obtaining the computing performance parameters and communication performance parameters of each computing node, the method further includes: performing network topology discovery on the heterogeneous cluster under the above-mentioned parameter server architecture and constructing a cluster communication topology map; based on the cluster communication topology map, identifying the parameter server nodes and computing nodes in the heterogeneous cluster and determining the connection relationship between each computing node and the parameter server node; and calculating the communication performance parameters of each node based on the connection relationship.

[0062] Specifically, network topology discovery is performed on the heterogeneous cluster under the aforementioned parameter server architecture. Cluster management tools (e.g., based on the LLDP protocol or cluster management API) are used to automatically detect the physical connections and logical relationships between nodes, constructing an accurate cluster communication topology map. Based on this topology map, nodes playing different logical roles in the cluster can be clearly identified: namely, parameter server nodes and compute nodes, and the specific connection relationships between each compute node and the parameter server node can be further determined. These steps form the basis for subsequent targeted performance testing, ensuring the accuracy and relevance of communication performance parameter measurements.

[0063] Furthermore, the acquisition of computational performance parameters can be achieved as follows: After identifying the computing nodes based on the topology graph, a standard deep learning benchmark program (e.g., the MLPerf benchmark suite or custom tests for core operations such as matrix multiplication and convolution) can be run on each computing node. By measuring the stable computation time required to process a series of batches of data of known size, the average time required for each node to process a unit amount of data can be calculated, which is its computational performance parameter. (where i is the index of the compute node).

[0064] Furthermore, the acquisition of communication performance parameters can be achieved as follows: Based on the aforementioned established connection relationships, point-to-point bandwidth tests are performed between each computing node and its corresponding parameter server node (e.g., using tools such as iperf, netperf, or custom testing programs) to measure and obtain the uplink bandwidth from each computing node to the parameter server. (For gradient upload) and downlink bandwidth (Used for parameter download), these bandwidth parameters are the communication performance parameters. The discovery of the above network topology can be completed automatically through the Link Layer Discovery Protocol (LLDP) or by calling the application programming interface (API) of the cluster management system.

[0065] Through the above steps, the system has completed a precise analysis of the hardware performance of the heterogeneous cluster, providing reliable data input for subsequent collaborative optimization.

[0066] Step S202: Based on communication performance parameters, determine the communication time required for each computing node to complete one gradient synchronization; based on computing performance parameters and communication time, construct a resource allocation model; based on the resource allocation model, obtain the target data allocation amount for each computing node; wherein, the product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node.

[0067] The purpose of the resource allocation model is to intelligently allocate the target data volume based on the computing performance parameters and communication time of each computing node, so that when each computing node processes the target data allocation, the product of its target data allocation and its computing performance parameters is greater than or equal to its communication time, so that the communication time of each computing node can be completely covered by its own computing time.

[0068] Specifically, step S202 includes the following three sub-steps.

[0069] (1) Determine the communication time.

[0070] Based on the communication performance parameters (uplink / downlink bandwidth) obtained in step S201, and combined with the characteristics of the training task, the communication time required for each node to complete one gradient synchronization is calculated. In a typical embodiment, for the i-th computing node:

[0071] ;

[0072] Where M represents the total amount of gradient parameters of the deep learning model to be trained. Its value is determined by the structure of the deep learning model (number of parameters, data type) and is a fixed value. and This represents the measured uplink and downlink bandwidths; i is a positive integer (i.e., the index of the compute node). This formula accurately reflects the network time required for one gradient synchronization under a parameter server architecture.

[0073] (2) Construct a resource allocation model.

[0074] The objective is to minimize the total time of a single training iteration, taking into account the computational performance parameters of each node. and communication time A resource allocation model is constructed. The purpose of this model is to find an optimal set of data allocation quantities. }( Let represent the target data allocation amount assigned to the i-th node, minimizing the processing time (the larger of computation and communication time) of the slowest node while satisfying the total data volume constraint. The above resource allocation model can be expressed as:

[0075] ;

[0076] ;

[0077] ;

[0078] in, This represents the target data allocation amount for the i-th computing node; This represents the computational performance parameters of the i-th computing node; This represents the communication time of the i-th computing node; This represents the total amount of data in a single training iteration. This total amount of data is determined based on the total number of computing nodes in the heterogeneous cluster and the memory capacity of each computing node. In practical applications, its typical value is between 1024 and 8192 samples.

[0079] (3) Calculate the target data allocation amount.

[0080] Based on the above resource allocation model, the target data allocation amount for each computing node is calculated. }; where the product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node.

[0081] Through the above steps, by co-optimizing the computation and communication times, the communication overhead is completely hidden within the computation process; by intelligently redistributing the data load of heterogeneous nodes, the "barrel effect" in the traditional synchronization mechanism is effectively broken; and finally, the core goal of minimizing the single training iteration time and maximizing the overall resource utilization of the cluster is achieved while keeping the total amount of data constant.

[0082] Step S203: Based on the target data allocation amount, the training data in the heterogeneous cluster is divided, and the divided data subsets are allocated to the corresponding computing nodes.

[0083] Specifically, based on the target data allocation obtained in step S202, the training data required for this training iteration is dynamically divided, and the divided data subsets are distributed (transmitted) in parallel to the corresponding computing nodes. After receiving the specified data subset, each computing node begins to execute the distributed training task in parallel. Since the data allocation is collaboratively optimized, each computing node is expected to complete (or be overwritten) gradient synchronization with the parameter server while completing its local computation, thereby greatly reducing the idle waiting time between nodes.

[0084] Through the steps S201 to S203 described above, the heterogeneous cluster resource allocation method provided in this embodiment effectively solves the "barrel effect" and idle computing / communication resources caused by the heterogeneity of cluster hardware under the parameter server architecture. By accurately characterizing the computing and communication capabilities of heterogeneous computing nodes and establishing a resource allocation model for intelligent data load allocation, it achieves the technical effect that communication overhead is almost completely hidden by the computing process under the parameter server architecture, thereby significantly improving the overall resource utilization and training throughput of heterogeneous clusters in distributed deep learning training.

[0085] In some embodiments, the target data allocation amount for each computing node is obtained based on a resource allocation model, including:

[0086] Calculate the theoretical equilibrium time ;

[0087] Comparison of theoretical equilibrium time Communication time with the i-th computing node Size;

[0088] if Then, a balanced allocation strategy is adopted, and the target data allocation amount for the i-th computing node is determined. ;

[0089] if Then based on the constraints and total data volume constraints By combining the resource allocation model, the target data allocation amount is obtained.

[0090] Among them, theoretical equilibrium time It is based on the computing power of each node ( (reciprocal) and total data volume The calculated physical meaning is as follows: In an ideal situation, if the total amount of data is distributed inversely proportional to the computing power of each computing node, and all communication overhead is ignored, then all computing nodes will complete the computation simultaneously. In this case, the theoretical time for the entire iteration is minimized, i.e., [the time is missing from the original text]. After calculating Then, the communication time with each computing node is recorded. Comparisons, especially with the maximum value, are used to assess whether communication time will become a bottleneck for system performance at that computing node.

[0091] Specifically, the theoretical equilibrium time is calculated. and compare Communication time with the i-th computing node Size.

[0092] when At this point, it indicates that even under the most ideal computational equilibrium state, the computation time required is [not specified]. It is also not shorter than the communication time of the slowest computing node. This means that for all computing nodes, their theoretical computation time can cover their own communication time, and the communication process will not become a key factor slowing down the entire iteration. In this case, there is no need for complex joint optimization; a balanced allocation strategy can be directly adopted, that is, let This allocation scheme naturally satisfies the total data volume constraint. ), and because It can be deduced that This holds true for all computing nodes i, thus achieving the goal of "computation covering communication". Furthermore, the above-mentioned balanced allocation strategy is computationally simple and extremely efficient.

[0093] when At this time, it indicates that there is at least one computing node communicating. Exceeded the theoretical equilibrium calculation time This also means that if data allocation is still simply based on computing power, the communication time of the computing node (and possibly other computing nodes) will be longer than its computing time. This will force it to wait for communication to complete after completing its computation, thus lengthening the entire iteration time. In this case, communication becomes the main bottleneck of system performance. Therefore, proactive optimization is necessary, with a corresponding "tilt" in data allocation. That is, more data should be allocated to nodes with longer communication times, thereby increasing their computing time so that they can "catch up" with and cover their longer communication time. Thus, the above process is transformed into a constrained optimization problem: to satisfy the total data volume... Under the premise of not changing, find a set of { }, making the constraints This holds true for all computation nodes, while minimizing the maximum value. (i.e., actual iteration time).

[0094] In some embodiments, the aforementioned target data allocation amount { } can be obtained in the following way:

[0095] Constraints Introduce slack variables to construct the Lagrange function;

[0096] Based on the introduced slack variables, the Lagrangian function is solved using an iterative weighted method to obtain the candidate data allocation.

[0097] Verify whether the candidate data allocation satisfies the total data volume constraint, and determine the candidate data allocation that satisfies the total data volume constraint as the target data allocation.

[0098] Through the above steps, this application embodiment introduces "theoretical equilibrium time" as a fast criterion, intelligently selecting between simple and efficient equilibrium allocation and fine and complex constraint optimization. This not only ensures a high-quality data allocation scheme under various cluster configurations, but also significantly reduces the average running cost of the algorithm, achieving a balance between optimization effect and execution efficiency. As a result, this resource allocation method can adapt to dynamically changing cluster environments.

[0099] In some embodiments, after allocating the partitioned data subsets to the corresponding computing nodes, the method further includes:

[0100] Based on a subset of data, distributed training is performed, and the actual computation time and actual communication time of each computing node in the current process of distributed training are monitored in real time.

[0101] When the deviation between the actual computing time and the expected computing time obtained based on the computing performance parameters exceeds the first preset threshold, or the deviation between the actual communication time and the expected communication time obtained based on the communication performance parameters exceeds the second preset threshold, the actual computing performance parameters are obtained based on the actual computing time, and a new resource allocation model is constructed based on the actual computing performance parameters and the actual communication time.

[0102] Based on the new resource allocation model, a new target data allocation amount for each computing node is obtained; wherein, the product of the new target data allocation amount for each computing node and the actual computing performance parameter of each computing node is greater than or equal to the actual communication time of each computing node.

[0103] Based on the new target data allocation, the training data in the heterogeneous cluster is divided, and the new data subsets are allocated to the corresponding computing nodes.

[0104] Specifically, to address potential performance fluctuations in heterogeneous cluster environments, such as resource contention by other tasks, momentary network congestion, and hardware frequency reduction, and to ensure that the resource allocation strategy remains optimal, this application introduces a dynamic adjustment mechanism. This mechanism is activated after the partitioned data subset is allocated to the corresponding computing nodes and distributed training is initiated, and specifically includes the following steps:

[0105] First, during the execution of distributed training tasks, the system continuously monitors the actual computation time and actual communication time of each computing node in each training iteration cycle. Actual computation time refers to the actual time spent by a computing node to complete the corresponding task for its assigned data subset; actual communication time refers to the actual network time spent by the computing node and the parameter server to synchronize gradient upload and parameter download. In addition, the system can also monitor node idle / waiting time, that is, through timeline analysis, identify the idle time periods when a computing node completes computation but waits for communication or completes communication but waits for computation, which is used to characterize whether resource allocation is balanced.

[0106] Meanwhile, monitoring data is collected and aggregated in real time for comparison with expected computation time and expected communication time, where expected computation time is a computational performance parameter based on the computing node. and its current target data allocation amount The calculated theoretical computation time, i.e. The expected communication time is based on the communication performance parameters (uplink bandwidth) of this computing node. and downlink bandwidth The theoretical communication duration calculated using the gradient parameter data volume M is... .

[0107] Then, when the system detects that the deviation between the actual computing time and the expected computing time of any computing node exceeds a first preset threshold (e.g., 15%), or the deviation between the actual communication time and the expected communication time exceeds a second preset threshold (e.g., 15%), it determines that the performance of the computing node has changed significantly, the initial allocation scheme may no longer be optimal, and immediately triggers a reallocation process. The first and second preset thresholds can be configured according to the system's stability requirements.

[0108] In addition, as a supplement to the performance deviation trigger, the system also sets a periodic threshold (for example, every 100 training iterations). When the number of training iterations reaches the periodic threshold, regardless of whether the performance deviation exceeds the limit, the system will force the start of a comprehensive performance evaluation and reallocation judgment process to accommodate possible slow performance drift (such as frequency reduction caused by heat dissipation, or the cumulative impact of background tasks).

[0109] Once any of the above triggering conditions are met, the system initiates a reallocation process, using the latest real-time performance data, namely the actual computational performance parameters and actual communication time, combined with the above resource allocation model, to obtain the latest target data allocation amount. Based on the new target data allocation amount, the system divides the training data for the new round and allocates the divided data subsets to each computing node, thereby starting a new round of distributed training.

[0110] Through the aforementioned dynamic adjustment process, this method achieves an evolution from static optimization to dynamic adaptation. It not only provides optimal allocation based on the cluster's static characteristics at the start of training but also continuously senses changes in cluster performance during long-term operation and responds quickly through online reallocation. This approach effectively solves the performance degradation problem caused by runtime interference, ensuring that heterogeneous clusters maintain near-optimal resource utilization and training efficiency even in complex production environments, significantly improving the robustness and long-term performance of the distributed training system.

[0111] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0112] It is understandable that the above-mentioned heterogeneous cluster resource allocation method, by optimizing the data load of each computing node, lays a solid foundation for achieving time matching between computation and communication at the allocation level. To achieve even better training efficiency, in the actual training execution process, overlapping scheduling techniques for computation and communication can be further combined to tap the hardware concurrency potential at the execution level. The following provides two overlapping scheduling embodiments that can work in conjunction with the resource allocation method of this application.

[0113] Example A: Inter-batch overlapping scheduling, which overlaps the gradient communication process of the Kth training batch with the forward and backward propagation calculation process of the (K+1)th batch, where K is a positive integer.

[0114] Specifically, after applying the resource allocation method of this application, the cluster allocates a target data allocation amount to each computing node. The system has initiated data-parallel training. Following this, the system executes the following overlapping scheduling steps:

[0115] (1) The computing node completes the forward and backward propagation of the current batch (Batch K) and generates the local gradient.

[0116] (2) Immediately initiate global synchronous communication of the gradients in this batch asynchronously (e.g., push gradients to the parameter server in a parameter server architecture).

[0117] (3) At the same time, the computation thread does not wait for the communication to complete, but immediately begins to process the forward propagation computation of the next batch (BatchK+1).

[0118] (4) Before updating the model parameters (e.g., before starting the calculation of Batch K+2), it is necessary to ensure that the gradient communication of Batch K has been completed and the back propagation of Batch K+1 has been completed.

[0119] (5) After the conditions are met, update the model parameters and continue to start asynchronous communication for the gradient of Batch K+1, while starting the calculation of Batch K+2, and so on, forming a “computation-communication” pipeline.

[0120] Through the above steps, the resource allocation method of this application ensures that the computation time and communication time of each computing node in processing a batch are balanced, providing a good time window basis for the overlap between batches, making it more likely that communication will be completely hidden in the computation of subsequent batches, and further reducing the end-to-end iteration time.

[0121] In addition to the batch-level implementation example A described above, a more granular layer-level overlapping scheduling approach can also be adopted to tap into potential, as shown in the following implementation example B.

[0122] Example B: In-iteration overlapping scheduling, which means that within a training batch, the gradient communication of different network layers is overlapped with the backpropagation calculation of subsequent layers in a more granular way.

[0123] Specifically, after applying the resource allocation method of this application, the cluster allocates a target data allocation amount to each computing node. The data-parallel training has been initiated, and the trained deep learning model consists of multiple sequential layers. At this point, the following steps can be used to achieve computation-communication overlap within the iteration:

[0124] (1) The computing node completes the full forward propagation of the current batch.

[0125] (2) Start backpropagation. After calculating the gradient of a certain layer (e.g., the Lth layer), start the synchronous communication of the gradient of the parameters of that layer asynchronously.

[0126] (3) At the same time, the computation thread continues the backpropagation process to calculate the gradient of the previous layer.

[0127] (4) This process is repeated to form a fine-grained pipeline of “calculate the gradient of one layer -> asynchronously communicate the gradient of that layer -> calculate the gradient of the next layer”.

[0128] (5) After the gradient calculation of all layers is completed and all layers are submitted for communication, wait for the gradient of all layers to be synchronized, and then update the entire model parameters at once.

[0129] In practical system deployments, due to constraints such as the integer granularity of data allocation and dynamic fluctuations in performance parameters, even with the aforementioned resource allocation model for optimization, the total computation time and total communication time of each node within a single batch may still be difficult to achieve a theoretically perfect balance. In this case, the aforementioned iterative overlapping scheduling technique can be further applied. By breaking down communication tasks into smaller parts and interleaving them with computation, idle time slices can be further filled. Combined with the resource allocation method of this application, optimization can be achieved from both macro-level load allocation and micro-level operation scheduling, maximizing cluster utilization.

[0130] It should be noted that the computation and communication overlap scheduling techniques involved in the above embodiments A and B can be used as independent optimization methods or in combination with the resource allocation method based on the performance model of this application. The two work together from different dimensions (load allocation and execution scheduling) to jointly solve the efficiency bottleneck problem in distributed training of heterogeneous clusters.

[0131] This embodiment also provides a heterogeneous cluster resource allocation device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0132] Figure 3 This is a structural block diagram of a heterogeneous cluster resource allocation device according to an embodiment of this application, such as... Figure 3 As shown, this device is applied to a parameter server architecture and includes:

[0133] The acquisition module 10 is used to acquire computing performance parameters that characterize the time required for each computing node to process a unit amount of data in a heterogeneous cluster under the parameter server architecture, as well as communication performance parameters that characterize the network communication capabilities of each computing node; the computing node is the working node in the heterogeneous cluster that undertakes distributed training computing tasks.

[0134] The computing module 20 is used to determine the communication time required for each computing node to complete one gradient synchronization based on the communication performance parameters; to construct a resource allocation model based on the computing performance parameters and the communication time; and to obtain the target data allocation amount for each computing node based on the resource allocation model. The product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node.

[0135] The allocation module 30 is used to divide the training data in the heterogeneous cluster according to the target data allocation amount, and allocate the divided data subsets to the corresponding computing nodes.

[0136] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0137] Furthermore, the above-mentioned device can be deployed and operated in the following exemplary system environments:

[0138] (1) Hardware environment example: The above heterogeneous cluster can contain various computing devices (such as graphics processing units (GPUs), artificial intelligence accelerator cards, etc.) as computing nodes, as well as dedicated parameter server nodes. The nodes are interconnected through high-speed networks (such as InfiniBand) and can access distributed storage systems to process large-scale training data.

[0139] (2) Software stack configuration example: The above device can be built on mainstream deep learning frameworks (such as PyTorch, TensorFlow), high-performance communication libraries (such as NCCL, OpenMPI), and containerization and resource management platforms (such as Kubernetes) to support efficient resource scheduling, task distribution, and communication. In addition, the above performance profiling process can be carried out with the help of benchmark test suites such as MLPerf; optimization of the resource allocation model can be carried out with modeling frameworks such as CVXPY; and task scheduling can be integrated with schedulers such as Kubernetes or Slurm to realize an automated pipeline from resource discovery, model solving to task execution.

[0140] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0141] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0142] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0143] S1, obtain the computing performance parameters that characterize the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and the communication performance parameters that characterize the network communication capability of each computing node; the above computing nodes are the working nodes in the heterogeneous cluster that undertake distributed training computing tasks.

[0144] S2, based on communication performance parameters, determine the communication time required for each computing node to complete one gradient synchronization; based on computing performance parameters and communication time, construct a resource allocation model; based on the resource allocation model, obtain the target data allocation amount for each computing node; wherein, the product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node.

[0145] S3 divides the training data in the heterogeneous cluster according to the target data allocation amount, and then allocates the partitioned data subsets to the corresponding computing nodes.

[0146] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0147] Furthermore, in conjunction with the heterogeneous cluster resource allocation method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the heterogeneous cluster resource allocation methods in the above embodiments.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0150] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for resource allocation in a heterogeneous cluster, characterized in that, Applied to parameter server architectures, including: Obtain computational performance parameters that characterize the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and communication performance parameters that characterize the network communication capabilities of each computing node; the computing node is the working node in the heterogeneous cluster that undertakes distributed training computation tasks. Based on the communication performance parameters, the communication time required for each computing node to complete one gradient synchronization is determined. Based on the computational performance parameters and the communication time, a resource allocation model is constructed; Based on the resource allocation model, the target data allocation amount for each computing node is obtained; wherein, the product of the target data allocation amount for each computing node and the computing performance parameter of each computing node is greater than or equal to the communication time of each computing node. Based on the target data allocation amount, the training data in the heterogeneous cluster is divided, and the divided data subsets are allocated to the corresponding computing nodes. The resource allocation model is expressed as follows: ； ； ； in, This represents the target data allocation amount for the i-th computing node. This represents the computational performance parameters of the i-th computing node. This represents the communication time of the i-th computing node. This represents the total amount of data in a single training iteration; the total amount of data is determined based on the total number of computing nodes in the heterogeneous cluster and the memory capacity of each computing node.

2. The heterogeneous cluster resource allocation method according to claim 1, characterized in that, The communication time of the i-th computing node is determined in the following way: ； Where M represents the amount of gradient parameter data of the deep learning model to be trained. and These represent the uplink bandwidth and downlink bandwidth from the i-th computing node to the corresponding parameter server, respectively. The parameter server is the node in the heterogeneous cluster that is responsible for global parameter maintenance and synchronization.

3. The heterogeneous cluster resource allocation method according to claim 1 or 2, characterized in that, The process of obtaining the target data allocation amount for each computing node based on the resource allocation model includes: Calculate the theoretical equilibrium time ; Compare the theoretical equilibrium time Communication time with the i-th computing node Size; if If a balanced allocation strategy is adopted, the target data allocation amount for the i-th computing node is... ; if Then based on the constraints and total data volume constraints Based on the resource allocation model, the target data allocation amount is obtained.

4. The heterogeneous cluster resource allocation method according to claim 3, characterized in that, The if Then based on the constraints and total data volume constraints Based on the resource allocation model, the target data allocation amount is obtained, including: The constraint conditions Introduce slack variables to construct the Lagrange function; Based on the introduced slack variables, the Lagrangian function is solved using an iterative weighted method to obtain the candidate data allocation amount; Verify whether the candidate data allocation quantity meets the total data volume constraint, and determine the candidate data allocation quantity that meets the total data volume constraint as the target data allocation quantity.

5. The heterogeneous cluster resource allocation method according to claim 1, characterized in that, After allocating the partitioned data subsets to the corresponding computing nodes, the method further includes: Based on the data subset, distributed training is performed, and the actual computing time and actual communication time of each computing node in the current process of the distributed training are monitored in real time. When the deviation between the actual computing time and the expected computing time obtained based on the computing performance parameters exceeds a first preset threshold, or the deviation between the actual communication time and the expected communication time obtained based on the communication performance parameters exceeds a second preset threshold, the actual computing performance parameters are obtained based on the actual computing time, and a new resource allocation model is constructed based on the actual computing performance parameters and the actual communication time. Based on the new resource allocation model, a new target data allocation amount is obtained for each computing node; wherein, the product of the new target data allocation amount for each computing node and the actual computing performance parameter of each computing node is greater than or equal to the actual communication time of each computing node. Based on the new target data allocation, the training data in the heterogeneous cluster is divided, and the new data subsets are allocated to the corresponding computing nodes.

6. The heterogeneous cluster resource allocation method according to claim 5, characterized in that, The real-time monitoring of the actual computation time and actual communication time of each computing node in the current process of the distributed training also includes: Count the number of iterations of the distributed training in the current process; When the number of execution iterations reaches a preset period threshold, the actual computing time and actual communication time of each computing node are obtained.

7. The heterogeneous cluster resource allocation method according to claim 1, characterized in that, Before obtaining the computing performance parameters characterizing the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and the communication performance parameters characterizing the network communication capabilities of each computing node, the method further includes: Perform network topology discovery on the heterogeneous cluster and construct a cluster communication topology graph; Based on the cluster communication topology diagram, the parameter server nodes and computing nodes in the heterogeneous cluster are identified, and the connection relationship between each computing node and the parameter server is determined. Based on the connection relationship, the communication performance parameters of each computing node are calculated.

8. The heterogeneous cluster resource allocation method according to claim 7, characterized in that, The acquisition of computational performance parameters, representing the time required for each computing node to process a unit amount of data in the heterogeneous cluster under the parameter server architecture, and communication performance parameters, representing the network communication capabilities of each computing node, includes: The computing performance parameters of each computing node are measured by running benchmark tests on each computing node. Based on the determined connection relationship between each computing node and the parameter server, a point-to-point bandwidth test is performed between the computing node and the parameter server to measure the communication performance parameters of each computing node, including uplink bandwidth and downlink bandwidth.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the heterogeneous cluster resource allocation method according to any one of claims 1 to 8 when it runs.

Images