A GPU cluster scheduling method, system, device, medium and program product

Abstract

This invention discloses a GPU cluster scheduling method, system, device, medium, and program product. By obtaining the regularity entropy, which characterizes the aggregation degree of idle computing nodes in the GPU cluster, and when the regularity entropy is greater than a preset regularity entropy threshold, a reserved set of supernodes can be set in the GPU cluster based on the current distributed training task information and the number of idle computing nodes of each supernode. The idle computing nodes in the reserved set are constrained to be used only for executing distributed training tasks. This can lock the complete resource space for the target distributed training task to be executed in advance, effectively reducing the degree of resource fragmentation and avoiding deadlock problems caused by severe resource fragmentation. As a result, the resource utilization of the GPU cluster is significantly improved, ensuring the effective and efficient execution of distributed training tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a GPU cluster scheduling method, system, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence, cloud computing, and edge computing technologies, large-scale GPU (Graphics Processing Unit) clusters have become crucial for supporting deep learning tasks. To efficiently manage heterogeneous computing resources, it is necessary to fully utilize the cluster's computing, storage, and network resources to improve task execution efficiency. Existing GPU cluster scheduling methods suffer from high resource fragmentation and deadlock issues when facing distributed training tasks, resulting in low resource utilization of GPU clusters and making it difficult to guarantee the effective and efficient execution of distributed training tasks. Summary of the Invention

[0003] This invention provides a GPU cluster scheduling method, system, device, medium, and program product to solve the technical problem that existing GPU cluster scheduling methods cannot guarantee the effective and efficient execution of distributed training tasks.

[0004] To address the aforementioned technical problems, a first aspect of this invention provides a GPU cluster scheduling method, comprising: The regularity entropy of the GPU cluster is determined based on the number of idle computing nodes in each supernode of the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. When the regularity entropy is greater than the preset regularity entropy threshold, the current set of reserved supernodes is determined based on the current distributed training task information and the number of idle computing nodes; wherein, the idle computing nodes in the set of reserved supernodes are used to execute distributed training tasks. Based on the preset resource mapping strategy of each candidate supernode for the target distributed training task to be executed and the resource information of each candidate supernode, the target distributed training task is scheduled; wherein each candidate supernode includes the set of reserved supernodes.

[0005] As a preferred embodiment, determining the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster specifically includes: The total number of idle computing nodes in the GPU cluster is determined based on the number of idle computing nodes in each of the supernodes. The regularity entropy is determined based on the ratio of the number of idle computing nodes to the total number of idle computing nodes for each supernode and the logarithm of the ratio.

[0006] As a preferred embodiment, determining the current set of retained supernodes based on the current distributed training task information and the number of idle computing nodes specifically includes: Based on the distributed training task information, determine the required number of idle computing nodes corresponding to the target distributed training task; The set of reserved supernodes is determined based on the required number of idle computing nodes and the number of idle computing nodes.

[0007] As a preferred embodiment, scheduling the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode specifically includes: Based on the communication cost between the various mapping computing devices indicated by the preset resource mapping strategy, the amount of cross-supernode communication data of the target distributed training task in each of the candidate supernodes is obtained; wherein, the computing node includes multiple computing devices; the communication cost is used to represent the amount of training task communication data between any two computing devices per unit time. The resource information and the amount of cross-supernode communication data are weighted to obtain the scheduling score corresponding to each candidate supernode; wherein, the resource information includes the number of idle computing nodes and the average remaining communication bandwidth within the supernode under the preset resource mapping strategy; The target distributed training task is scheduled based on the target supernode with the highest scheduling score.

[0008] As a preferred embodiment, the preset resource mapping strategy is determined with the goal of minimizing the total communication cost penalty; the total communication cost penalty is determined based on the product of the communication cost between each of the mapping computing devices and the communication penalty coefficient; the communication penalty coefficient is determined based on the communication bandwidth between each of the mapping computing devices.

[0009] As a preferred embodiment, scheduling the target distributed training task based on the target supernode with the highest scheduling score specifically includes: According to the required number of idle computing nodes, pre-occupancy information is sent to several physically contiguous idle computing nodes in the target supernode within a preset pre-occupancy time window, so that the idle computing nodes are in a pre-occupancy state. When the number of idle computing nodes with the pre-occupancy information is always less than the required number of idle computing nodes within the preset pre-occupancy time window, obtain the set of training tasks that have been assigned to the target supernode and do not belong to the distributed training tasks. The hot migration target is set as the number of physically contiguous idle computing nodes in the target supernode being greater than or equal to the required number of idle computing nodes. The hot migration strategy and hot migration time of the training task set are then determined. The hot migration strategy is executed when the hot migration time is less than the preset waiting time threshold corresponding to the target distributed training task. When the hot migration strategy is detected to be completed, the pre-occupancy information is sent to the idle computing nodes in the target supernode so that the number of idle computing nodes with the pre-occupancy information and physical contiguousness is equal to the number of idle computing nodes required, and the target distributed training task is assigned to each of the idle computing nodes with the pre-occupancy information and physical contiguousness for execution.

[0010] As a preferred embodiment, the hot migration time is determined based on the number of source computing devices corresponding to the training task set, the amount of task data to be migrated on each source computing device corresponding to the training task set, the additional task migration overhead corresponding to each training task in the training task set, and the effective bandwidth of the migration link.

[0011] As a preferred embodiment, the method further includes: When the hot migration time is greater than or equal to the preset waiting time threshold, a release operation is performed on each computing node that has the pre-occupancy information.

[0012] As a preferred embodiment, the method further includes: When the hot migration conditions are not met, a preset degraded scheduling strategy is executed; wherein, the hot migration conditions include the training task set supporting hot migration, the hot migration time being less than the preset waiting time threshold, and having at least one target computing device capable of migrating into the training task set; the degraded scheduling strategy includes at least one of the following: cross-supernode resource allocation strategy, computing node preemption strategy, idle computing node expansion strategy, and parallel execution strategy adjustment strategy. When the degraded scheduling strategy is completed, and the total number of idle computing nodes with the pre-occupancy information and the number of idle computing nodes determined based on the degraded scheduling strategy is greater than or equal to the required number of idle computing nodes or the adjusted number of idle computing nodes corresponding to the target distributed training task adjusted by the parallel execution strategy, the pre-occupancy information is sent to the idle computing nodes determined based on the degraded scheduling strategy, and the target distributed training task or the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution. When the downgrade scheduling strategy is completed and the number of idle computing nodes with the pre-occupancy information is greater than or equal to the adjusted number of idle computing nodes required, the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution.

[0013] As a preferred embodiment, the cross-hypernode resource allocation strategy is an idle computing node allocation strategy aimed at minimizing the cross-hypernode communication cost; the cross-hypernode communication cost is used to represent the amount of training task communication data between any two computing devices located on different hypernodes per unit time. The computing node preemption strategy is a strategy executed based on the preset priority of each training task in the training task set and the preset priority of the target distributed training task; The idle computing node expansion strategy is a strategy implemented based on the difference between the required number of idle computing nodes and the number of physically contiguous idle computing nodes in the target supernode. The parallel execution strategy adjustment strategy is used to adjust the parallel execution strategy of the target distributed training task when the minimum number of idle computing nodes required supported by the target distributed training task is less than the current number of idle computing nodes required, so as to reduce the number of idle computing nodes required to the minimum number of idle computing nodes required, and use the minimum number of idle computing nodes required as the adjusted number of idle computing nodes required.

[0014] A second aspect of this invention provides a GPU cluster scheduling system, comprising: The regularity entropy acquisition module is used to determine the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. The module for determining the set of retained supernodes is used to determine the current set of retained supernodes based on the current distributed training task information and the number of idle computing nodes when the regularity entropy is greater than a preset regularity entropy threshold; wherein the idle computing nodes in the set of retained supernodes are used to execute distributed training tasks. The scheduling module is used to schedule the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode; wherein each candidate supernode includes the set of reserved supernodes.

[0015] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the GPU cluster scheduling method described in any of the first aspects.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the GPU cluster scheduling method described in any of the first aspects.

[0017] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, wherein when the computer program / instructions are executed by a processor, they implement the steps of the GPU cluster scheduling method described in any of the first aspects.

[0018] Compared to existing technologies, the beneficial effects of this invention are as follows: by obtaining the regularity entropy, which characterizes the aggregation degree of idle computing nodes in a GPU cluster, and when the regularity entropy is greater than a preset regularity entropy threshold, a set of reserved supernodes can be set in the GPU cluster based on the current distributed training task information and the number of idle computing nodes of each supernode. The idle computing nodes in the set of reserved supernodes are constrained to be used only for executing distributed training tasks. This allows for the early locking of complete resource space for the target distributed training task to be executed, effectively reducing resource fragmentation and avoiding deadlock problems caused by severe resource fragmentation. Consequently, the resource utilization of the GPU cluster is significantly improved, ensuring the effective and efficient execution of distributed training tasks. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the GPU cluster scheduling method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the GPU cluster scheduling system in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 The first aspect of this invention provides a GPU cluster scheduling method, comprising the following steps S1 to S3: Step S1: Determine the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. Step S2: When the regularity entropy is greater than the preset regularity entropy threshold, determine the current set of reserved supernodes based on the current distributed training task information and the number of idle computing nodes; wherein, the idle computing nodes in the set of reserved supernodes are used to execute distributed training tasks. Step S3: Based on the preset resource mapping strategy of each candidate supernode for the target distributed training task to be executed and the resource information of each candidate supernode, schedule the target distributed training task; wherein each candidate supernode includes the set of reserved supernodes.

[0022] It is worth noting that this embodiment strictly maps the physical resources of the GPU cluster to the network layer, abstracting them into three core layers: computing devices, computing nodes, and supernodes. A computing device is the smallest hardware unit of the GPU cluster, which can be an AI (Artificial Intelligence) acceleration chip such as a GPU or NPU (Neural Processing Unit). A computing node is a complete machine containing multiple computing devices; for example, each computing node includes eight computing devices. A supernode is a cluster of nodes containing multiple computing nodes. The communication network in the GPU cluster includes: Intra-Node communication, which interconnects cards through a single-machine internal bus; Intra-Supernode communication, which interconnects computing nodes through a Scale-Up network, possessing near-single-machine bus-like non-blocking bandwidth and extremely low latency; and Inter-Supernode communication, which achieves large-scale horizontal scaling interconnection between multiple supernodes through a Scale-Out network (such as an InfiniBand network or RoCE network), with relatively limited bandwidth, higher latency, and susceptibility to congestion.

[0023] Specifically, when a distributed training task scheduling request containing K Pods (each Pod requires G computing devices) is received, in order to prevent small-scale training tasks that do not belong to distributed training tasks from breaking up the continuous idle computing node resources within the supernode, thus forcing large-scale distributed training tasks to rely on inter-supernode communication for execution, this embodiment needs to determine the regularity entropy of the GPU cluster based on the number of idle computing nodes in each supernode in the GPU cluster. It can be understood that the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. If the regularity entropy is lower, the degree of aggregation of idle computing nodes is higher, indicating that there are more cases of multiple physically continuous idle computing nodes in the same supernode, that is, the degree of resource fragmentation is lower. Therefore, in this embodiment, when the regularity entropy is greater than the preset regularity entropy threshold, it indicates that there is a certain degree of resource fragmentation. At this time, based on the current distributed training task information, with the goal of maximizing the number of idle computing nodes in the supernodes, a set of reserved supernodes is set in the GPU cluster. Training tasks that do not belong to the distributed training tasks are prohibited from being assigned to idle computing nodes in the set of reserved supernodes. This locks the complete resource space within the supernodes for the target distributed training task to be executed in advance, effectively preventing the degree of resource fragmentation from worsening. At the same time, since the target distributed training task has sufficient complete resource space, deadlock problems caused by severe resource fragmentation can be avoided.

[0024] Furthermore, for each candidate supernode with idle computing nodes, these candidate supernodes include the reserved supernode set determined above. In this embodiment, based on the preset resource mapping strategy of the target distributed training task to be executed on each candidate supernode, the amount of cross-supernode communication data when the target distributed training task is executed on each candidate supernode can be clearly determined. The larger the amount of cross-supernode communication data, the higher the communication latency, which makes the execution time of the target distributed training task longer. In addition, this embodiment also considers the resource information of each candidate supernode to select the optimal target supernode from each candidate supernode to realize the scheduling of the target distributed training task.

[0025] It is worth noting that when the regularity entropy is less than or equal to the preset regularity entropy threshold, it indicates that the resource fragmentation is low. There is no need to set up a set of reserved supernodes. The target distributed training task can be scheduled directly based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode. The candidate supernodes are still supernodes with idle computing nodes.

[0026] The GPU cluster scheduling method provided in this invention obtains the regularity entropy, which characterizes the aggregation degree of idle computing nodes in the GPU cluster. When the regularity entropy is greater than a preset regularity entropy threshold, it can set a reserved set of supernodes in the GPU cluster based on the current distributed training task information and the number of idle computing nodes of each supernode. The idle computing nodes in the reserved set are constrained to be used only for executing distributed training tasks. This can lock the complete resource space for the target distributed training task to be executed in advance, effectively reducing the degree of resource fragmentation and avoiding deadlock problems caused by severe resource fragmentation. As a result, it significantly improves the resource utilization of the GPU cluster and ensures the effective and efficient execution of distributed training tasks.

[0027] As a preferred embodiment, determining the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster specifically includes: The total number of idle computing nodes in the GPU cluster is determined based on the number of idle computing nodes in each of the supernodes. The regularity entropy is determined based on the ratio of the number of idle computing nodes to the total number of idle computing nodes for each supernode and the logarithm of the ratio.

[0028] Specifically, this embodiment determines the total number of idle computing nodes in the current GPU cluster by summing the number of idle computing nodes of each supernode, and then calculates the regularity entropy using the following expression: ; in, Represents the regularity entropy; Indicates the number of supernodes; This represents the ratio of the number of idle compute nodes in the i-th supernode to the total number of idle compute nodes. It can be understood that if a supernode has a large number of idle compute nodes, then its corresponding... A larger value indicates that there are more idle computing nodes clustered within it, resulting in a smaller regularity entropy.

[0029] As a preferred embodiment, determining the current set of retained supernodes based on the current distributed training task information and the number of idle computing nodes specifically includes: Based on the distributed training task information, determine the required number of idle computing nodes corresponding to the target distributed training task; The set of reserved supernodes is determined based on the required number of idle computing nodes and the number of idle computing nodes.

[0030] Specifically, upon receiving a distributed training task scheduling request, the corresponding distributed training task information can be clearly identified, thereby determining the required number of idle computing nodes for the target distributed training task. For example, based on the distributed training task information, the parallel strategy of the target distributed training task can be determined, such as TP (Tensor Parallelism), PP (Pipeline Parallelism), and DP (Data Parallelism). If the target distributed training task contains K Pods, and the number of computing devices required by each Pod is consistent with the number of computing devices contained in a single computing node, then the required number of idle computing nodes for the target distributed training task is K. Then, this embodiment uses the constraint that the number of idle computing nodes in a supernode is greater than or equal to the required number of idle computing nodes, and aims to maximize the number of idle computing nodes in a supernode. Several supernodes are selected as the current set of reserved supernodes according to a preset selection number, thus satisfying the requirements of the target distributed training task while ensuring a large number of physically contiguous idle computing nodes within each supernode.

[0031] As a preferred embodiment, scheduling the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode specifically includes: Based on the communication cost between the various mapping computing devices indicated by the preset resource mapping strategy, the amount of cross-supernode communication data of the target distributed training task in each of the candidate supernodes is obtained; wherein, the computing node includes multiple computing devices; the communication cost is used to represent the amount of training task communication data between any two computing devices per unit time. The resource information and the amount of cross-supernode communication data are weighted to obtain the scheduling score corresponding to each candidate supernode; wherein, the resource information includes the number of idle computing nodes and the average remaining communication bandwidth within the supernode under the preset resource mapping strategy; The target distributed training task is scheduled based on the target supernode with the highest scheduling score.

[0032] Understandably, since the number of idle computing nodes varies among different candidate supernodes, the target distributed training task may involve cross-supernode task data communication under the preset resource mapping strategy in some candidate supernodes. To avoid the target distributed training task being significantly affected by cross-supernode communication latency, it is necessary to obtain the amount of cross-supernode communication data corresponding to the target distributed training task in each candidate supernode as one of the scheduling criteria, as shown in the following expression: ; in, This indicates that the target distributed training task is in the candidate supernode. The corresponding amount of cross-supernode communication data; This represents the communication cost between mapping computing device i and mapping computing device j; and Based on candidate supernodes The mapping computing device i and mapping computing device j are determined by the corresponding preset resource mapping strategy; This is an indicator function used to determine whether the communication between mapping computing device i and mapping computing device j is cross-hypernode communication. When the communication between mapping computing device i and mapping computing device j is cross-hypernode communication, the indicator function takes the value of 1, and when the communication between mapping computing device i and mapping computing device j is not cross-hypernode communication, the indicator function takes the value of 0.

[0033] Understandably, the communication cost is used to represent the amount of training task communication data between any two computing devices per unit time, that is, the product of the frequency of task communication and the amount of data in a single communication.

[0034] Furthermore, this embodiment uses the following expression to weight the resource information of candidate supernodes and the amount of cross-supernode communication data: ; in, Indicates candidate supernodes The corresponding scheduling score; Indicates candidate supernodes The number of idle computing nodes, which can be understood as, if candidate supernodes If the number of idle computing nodes is greater than or equal to the number of idle computing nodes required for the target distributed training task, it indicates that the target distributed training task can be fully accomplished using this candidate supernode. Within the closed loop of the scale-up network, the scheduling score receives a significant weighting bonus; Indicates candidate supernodes The average remaining communication bandwidth within a supernode under the preset resource mapping strategy; , and They are respectively , and The corresponding weight values, where, by setting Much larger This makes the penalty for cross-supernode communication much higher than the benefit of the remaining communication bandwidth within a supernode, thus helping to select the optimal target supernode.

[0035] The candidate supernode with the highest scheduling score is selected as the current target supernode, and the target distributed training task is scheduled.

[0036] As a preferred embodiment, the preset resource mapping strategy is determined with the goal of minimizing the total communication cost penalty; the total communication cost penalty is determined based on the product of the communication cost between each of the mapping computing devices and the communication penalty coefficient; the communication penalty coefficient is determined based on the communication bandwidth between each of the mapping computing devices.

[0037] Specifically, at the physical level, this embodiment defines the communication penalty coefficient between any two computing devices as follows: They exhibit a strict hierarchical distribution; if two computing devices belong to the same computing node, they communicate via an internal bus. Minimal; if two computing devices belong to two different computing nodes within the same supernode, they communicate via a scale-up network. The value is relatively small, greater than but close to the communication penalty coefficient when two computing devices belong to the same computing node; if the two computing devices belong to two different supernodes, they communicate through a scale-out network. Then it is extremely large. The communication penalty coefficient is determined based on the communication bandwidth between the various mapping computing devices, and its expression is: = Unit time / B, where B is the communication bandwidth. Its initial value can be set according to the link specifications or historical measured bandwidth. During operation, it is updated periodically based on the average network throughput of the most recent 5 to 10 minutes, and smoothed by moving average or EMA to reduce the impact of short-term fluctuations.

[0038] In this embodiment, the sum of the products of the communication cost and the communication penalty coefficient between each mapping computing device is the total communication cost penalty. Since the communication penalty coefficient of two computing devices belonging to two different supernodes is usually much larger than the communication penalty coefficient of two computing devices belonging to two different computing nodes in the same supernode, minimizing the total communication cost penalty can bring communication-intensive parallel tasks, especially distributed training tasks under TP and PP parallel strategies, into the scale-up network domain of a single supernode.

[0039] As a preferred embodiment, scheduling the target distributed training task based on the target supernode with the highest scheduling score specifically includes: According to the required number of idle computing nodes, pre-occupancy information is sent to several physically contiguous idle computing nodes in the target supernode within a preset pre-occupancy time window, so that the idle computing nodes are in a pre-occupancy state. When the number of idle computing nodes with the pre-occupancy information is always less than the required number of idle computing nodes within the preset pre-occupancy time window, obtain the set of training tasks that have been assigned to the target supernode and do not belong to the distributed training tasks. The hot migration target is set as the number of physically contiguous idle computing nodes in the target supernode being greater than or equal to the required number of idle computing nodes. The hot migration strategy and hot migration time of the training task set are then determined. The hot migration strategy is executed when the hot migration time is less than the preset waiting time threshold corresponding to the target distributed training task. When the hot migration strategy is detected to be completed, the pre-occupancy information is sent to the idle computing nodes in the target supernode so that the number of idle computing nodes with the pre-occupancy information and physical contiguousness is equal to the number of idle computing nodes required, and the target distributed training task is assigned to each of the idle computing nodes with the pre-occupancy information and physical contiguousness for execution.

[0040] Specifically, in this embodiment, according to the required number of idle computing nodes, pre-occupancy information is sent to several physically contiguous idle computing nodes in the target supernode within a preset pre-occupancy time window, so that the idle computing nodes are in a pre-occupancy state and will not be occupied by other training tasks. If the number of idle computing nodes with pre-occupancy information is always less than the required number of idle computing nodes within the preset pre-occupancy time window, the set of training tasks currently allocated to the target supernode that do not belong to distributed training tasks is scanned. This is because these fragmented training tasks may occupy different computing nodes, resulting in an insufficient number of idle computing nodes. In this case, they need to be treated as a whole for hot migration. The hot migration target is that the number of physically contiguous idle computing nodes in the target supernode is greater than or equal to the required number of idle computing nodes. The target computing device to be migrated into the training task set is preferably a computing device in the target supernode to reduce the hot migration time. Only when there is no computing device to be migrated into the target supernode is hot migration to computing devices in other supernodes considered to free up new idle computing nodes in the target supernode, thereby forming a hot migration strategy for the training task set and determining its corresponding hot migration time.

[0041] When the hot migration time is less than the preset waiting time threshold (e.g., 200s) corresponding to the target distributed training task, it indicates that the current hot migration strategy has not timed out for the target distributed training task, and therefore the hot migration strategy is executed. When the hot migration strategy is detected to be completed, pre-occupancy information is sent to the idle computing nodes in the target supernode to ensure that the number of idle computing nodes with pre-occupancy information and physical contiguousness equals the number of idle computing nodes required. Then, the target distributed training task is assigned to each idle computing node with pre-occupancy information and physical contiguousness for execution. Specifically, locking information can be sent to each idle computing node with pre-occupancy information to change each idle computing node from the pre-occupancy state to the locked state, and then the target distributed training task is assigned to each idle computing node in the locked state for execution.

[0042] As a preferred embodiment, the hot migration time is determined based on the number of source computing devices corresponding to the training task set, the amount of task data to be migrated on each source computing device corresponding to the training task set, the additional task migration overhead corresponding to each training task in the training task set, and the effective bandwidth of the migration link.

[0043] Specifically, this embodiment calculates the thermal migration time using the following expression: ; in, Indicates the time taken for thermal migration; This indicates the number of source computing devices corresponding to the set of training tasks; This represents the amount of task data to be migrated from the g-th source computing device corresponding to the training task set; Indicates the effective bandwidth of the migration link; This represents the additional task migration overhead corresponding to each training task in the training task set, such as state synchronization, communication coordination, task pause / resumption, etc.

[0044] As a preferred embodiment, the method further includes: When the hot migration time is greater than or equal to the preset waiting time threshold, a release operation is performed on each computing node that has the pre-occupancy information.

[0045] It is worth noting that when the hot migration time is greater than or equal to the preset waiting time threshold, it indicates that the current hot migration strategy has timed out for the target distributed training task. In order to avoid affecting the execution of other training tasks, the computing nodes with pre-occupancy information are released at this time, so that they exit the pre-occupancy state and the target distributed training task returns to sleep.

[0046] As a preferred embodiment, the method further includes: When the hot migration conditions are not met, a preset degraded scheduling strategy is executed; wherein, the hot migration conditions include the training task set supporting hot migration, the hot migration time being less than the preset waiting time threshold, and having at least one target computing device capable of migrating into the training task set; the degraded scheduling strategy includes at least one of the following: cross-supernode resource allocation strategy, computing node preemption strategy, idle computing node expansion strategy, and parallel execution strategy adjustment strategy. When the degraded scheduling strategy is completed, and the total number of idle computing nodes with the pre-occupancy information and the number of idle computing nodes determined based on the degraded scheduling strategy is greater than or equal to the required number of idle computing nodes or the adjusted number of idle computing nodes corresponding to the target distributed training task adjusted by the parallel execution strategy, the pre-occupancy information is sent to the idle computing nodes determined based on the degraded scheduling strategy, and the target distributed training task or the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution. When the downgrade scheduling strategy is completed and the number of idle computing nodes with the pre-occupancy information is greater than or equal to the adjusted number of idle computing nodes required, the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution.

[0047] Specifically, in this embodiment, when the hot migration conditions are not met, such as the training task set not supporting hot migration, the hot migration time being greater than or equal to a preset waiting time threshold, or the number of target computing devices capable of migrating into the training task set is insufficient, this embodiment executes a preset degraded scheduling strategy to ensure that the target distributed training task can be executed. This strategy includes at least one of the following: a cross-supernode resource allocation strategy, a compute node preemption strategy, an idle compute node expansion strategy, and a parallel execution strategy adjustment strategy. The cross-supernode resource allocation strategy allows the target distributed training task to be deployed in a controlled manner across multiple supernodes. The compute node preemption strategy performs whole-machine preemption on low-priority training tasks to form complete compute node resources. The idle compute node expansion strategy aims to expand the number of idle compute nodes within the target supernode. The parallel execution strategy adjustment strategy reduces the number of idle compute nodes required by adjusting the parallel execution strategy of the target distributed training task.

[0048] Therefore, by employing cross-supernode resource allocation strategies, computing node preemption strategies, and idle computing node expansion strategies, new idle computing nodes can be determined based on the currently pre-occupied idle computing nodes. When the degradation scheduling strategy is completed, and the total number of idle computing nodes with pre-occupied information and the idle computing nodes determined based on the degradation scheduling strategy is greater than or equal to the original number of idle computing nodes required for the target distributed training task, pre-occupied information continues to be sent to the idle computing nodes determined based on the degradation scheduling strategy. This ensures that the number of idle computing nodes in the pre-occupied state equals the original number of idle computing nodes required for the target distributed training task. Then, the target distributed training task is allocated to each idle computing node with pre-occupied information for execution.

[0049] If the total number of idle computing nodes with pre-occupancy information and the number of idle computing nodes determined by the degraded scheduling strategy is still less than the original number of idle computing nodes required for the target distributed training task, but the number of idle computing nodes required for the target distributed training task can be reduced by adjusting the parallel execution strategy, then it can be determined whether the total number of idle computing nodes with pre-occupancy information and the number of idle computing nodes determined by the degraded scheduling strategy is greater than or equal to the adjusted number of idle computing nodes required for the target distributed training task after the adjustment by the parallel execution strategy. If the total number is greater than or equal to the adjusted number of idle computing nodes required, then pre-occupancy information continues to be sent to the idle computing nodes determined by the degraded scheduling strategy, so that the number of idle computing nodes in the pre-occupancy state is equal to the adjusted number of idle computing nodes required. Then, the target distributed training task adjusted by the parallel execution strategy is allocated to each of the idle computing nodes with pre-occupancy information for execution.

[0050] If, after implementing the parallel execution strategy adjustment strategy, the number of idle computing nodes with pre-occupancy information is greater than or equal to the adjusted number of idle computing nodes required, then the target distributed training task adjusted by the parallel execution strategy adjustment strategy will be directly assigned to each idle computing node with pre-occupancy information for execution.

[0051] If the resource requirements of the target distributed training task still cannot be met after the execution of the above-mentioned degradation scheduling strategies, the computing nodes that currently have pre-occupancy information will be released to exit the pre-occupancy state. The target distributed training task will then return to sleep and enter the delayed retry mechanism to ensure the forward progress and stability of the overall scheduling of the GPU cluster.

[0052] As a preferred embodiment, the cross-hypernode resource allocation strategy is an idle computing node allocation strategy aimed at minimizing the cross-hypernode communication cost; the cross-hypernode communication cost is used to represent the amount of training task communication data between any two computing devices located on different hypernodes per unit time. The computing node preemption strategy is a strategy executed based on the preset priority of each training task in the training task set and the preset priority of the target distributed training task; The idle computing node expansion strategy is a strategy implemented based on the difference between the required number of idle computing nodes and the number of physically contiguous idle computing nodes in the target supernode. The parallel execution strategy adjustment strategy is used to adjust the parallel execution strategy of the target distributed training task when the minimum number of idle computing nodes required supported by the target distributed training task is less than the current number of idle computing nodes required, so as to reduce the number of idle computing nodes required to the minimum number of idle computing nodes required, and use the minimum number of idle computing nodes required as the adjusted number of idle computing nodes required.

[0053] It is worth noting that this embodiment is based on a cross-supernode resource allocation strategy, aiming to minimize cross-supernode communication costs. According to the required number of idle computing nodes, it continues to send pre-occupancy information to several idle computing nodes among the remaining supernodes (excluding the target supernode). Minimizing cross-supernode communication costs ensures that cross-supernode communication latency is kept as low as possible.

[0054] Based on the computing node preemption strategy, this embodiment suspends training tasks in the training task set whose preset priority is lower than the preset priority of the target distributed training task, thereby increasing the number of idle computing nodes in the target supernode.

[0055] Based on the idle computing node expansion strategy, this embodiment first clarifies the difference between the current required number of idle computing nodes and the number of idle computing nodes in the target supernode. For example, if the required number of idle computing nodes is K, and the number of idle computing nodes in the target supernode is N, where N is less than K, then the resource gap is: Defict = max(0, KN), and the target number of idle computing nodes is: min(N... max ,max(N min ,λDeficit), where λ is the expansion coefficient, which can be set to 0.5. It can be understood that if small-scale training tasks within the target supernode support hot migration, these training tasks can be hot-migrated to the local computing node range, thereby reducing resource fragmentation within the target supernode and expanding the range of idle computing nodes. The local computing node range can be 1 computing node, 2 computing nodes, etc., depending on the amount of task data that can be migrated into the computing node and the total amount of task data for the training tasks to be migrated. This embodiment does not impose specific limitations here. If small-scale training tasks within the target supernode do not support hot migration, these training tasks can be waited for to complete, and the newly formed idle computing nodes can be included in the reserved supernode set. The reserved supernode set is actually a set of idle computing nodes in each supernode, thus also achieving the effect of expanding the range of idle computing nodes, reducing the failure of distributed training task scheduling due to insufficient candidate resource space, and reducing the number of fragmentation reorganizations to meet distributed training task requirements, thereby suppressing the generation of new fragments and accelerating resource regularization recovery.

[0056] Based on the parallel execution strategy adjustment strategy, if the target distributed training task supports elastic parallelism, and thus the minimum number of idle computing nodes it supports is less than the current number of idle computing nodes it requires, then the parallel execution strategy of the target distributed training task is adjusted to reduce the current number of idle computing nodes required to the minimum number of idle computing nodes it supports, so that the target distributed training task can be rescheduled after the parallel execution strategy adjustment.

[0057] Please see Figure 2 A second aspect of the present invention provides a GPU cluster scheduling system 100, comprising: The regularity entropy acquisition module 11 is used to determine the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. The reserved supernode set determination module 12 is used to determine the current reserved supernode set based on the current distributed training task information and the number of idle computing nodes when the regularity entropy is greater than a preset regularity entropy threshold; wherein the idle computing nodes in the reserved supernode set are used to execute distributed training tasks. The scheduling module 13 is used to schedule the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode; wherein each candidate supernode includes the set of reserved supernodes.

[0058] As a preferred embodiment, the regularity entropy acquisition module 11 is used to determine the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster, specifically including: The total number of idle computing nodes in the GPU cluster is determined based on the number of idle computing nodes in each of the supernodes. The regularity entropy is determined based on the ratio of the number of idle computing nodes to the total number of idle computing nodes for each supernode and the logarithm of the ratio.

[0059] As a preferred embodiment, the reserved supernode set determination module 12 is used to determine the current reserved supernode set based on the current distributed training task information and the number of idle computing nodes, specifically including: Based on the distributed training task information, determine the required number of idle computing nodes corresponding to the target distributed training task; The set of reserved supernodes is determined based on the required number of idle computing nodes and the number of idle computing nodes.

[0060] As a preferred embodiment, the scheduling module 13 is used to schedule the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode, specifically including: Based on the communication cost between the various mapping computing devices indicated by the preset resource mapping strategy, the amount of cross-supernode communication data of the target distributed training task in each of the candidate supernodes is obtained; wherein, the computing node includes multiple computing devices; the communication cost is used to represent the amount of training task communication data between any two computing devices per unit time. The resource information and the amount of cross-supernode communication data are weighted to obtain the scheduling score corresponding to each candidate supernode; wherein, the resource information includes the number of idle computing nodes and the average remaining communication bandwidth within the supernode under the preset resource mapping strategy; The target distributed training task is scheduled based on the target supernode with the highest scheduling score.

[0061] As a preferred embodiment, the preset resource mapping strategy is determined with the goal of minimizing the total communication cost penalty; the total communication cost penalty is determined based on the product of the communication cost between each of the mapping computing devices and the communication penalty coefficient; the communication penalty coefficient is determined based on the communication bandwidth between each of the mapping computing devices.

[0062] As a preferred embodiment, the scheduling module 13 is used to schedule the target distributed training task based on the target supernode with the highest scheduling score, specifically including: According to the required number of idle computing nodes, pre-occupancy information is sent to several physically contiguous idle computing nodes in the target supernode within a preset pre-occupancy time window, so that the idle computing nodes are in a pre-occupancy state. When the number of idle computing nodes with the pre-occupancy information is always less than the required number of idle computing nodes within the preset pre-occupancy time window, obtain the set of training tasks that have been assigned to the target supernode and do not belong to the distributed training tasks. The hot migration target is set as the number of physically contiguous idle computing nodes in the target supernode being greater than or equal to the required number of idle computing nodes. The hot migration strategy and hot migration time of the training task set are then determined. The hot migration strategy is executed when the hot migration time is less than the preset waiting time threshold corresponding to the target distributed training task. When the hot migration strategy is detected to be completed, the pre-occupancy information is sent to the idle computing nodes in the target supernode so that the number of idle computing nodes with the pre-occupancy information and physical contiguousness is equal to the number of idle computing nodes required, and the target distributed training task is assigned to each of the idle computing nodes with the pre-occupancy information and physical contiguousness for execution.

[0063] As a preferred embodiment, the hot migration time is determined based on the number of source computing devices corresponding to the training task set, the amount of task data to be migrated on each source computing device corresponding to the training task set, the additional task migration overhead corresponding to each training task in the training task set, and the effective bandwidth of the migration link.

[0064] As a preferred embodiment, the scheduling module 13 is further configured to: When the hot migration time is greater than or equal to the preset waiting time threshold, a release operation is performed on each computing node that has the pre-occupancy information.

[0065] As a preferred embodiment, the scheduling module 13 is further configured to: When the hot migration conditions are not met, a preset degraded scheduling strategy is executed; wherein, the hot migration conditions include the training task set supporting hot migration, the hot migration time being less than the preset waiting time threshold, and having at least one target computing device capable of migrating into the training task set; the degraded scheduling strategy includes at least one of the following: cross-supernode resource allocation strategy, computing node preemption strategy, idle computing node expansion strategy, and parallel execution strategy adjustment strategy. When the degraded scheduling strategy is completed, and the total number of idle computing nodes with the pre-occupancy information and the number of idle computing nodes determined based on the degraded scheduling strategy is greater than or equal to the required number of idle computing nodes or the adjusted number of idle computing nodes corresponding to the target distributed training task adjusted by the parallel execution strategy, the pre-occupancy information is sent to the idle computing nodes determined based on the degraded scheduling strategy, and the target distributed training task or the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution. When the downgrade scheduling strategy is completed and the number of idle computing nodes with the pre-occupancy information is greater than or equal to the adjusted number of idle computing nodes required, the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution.

[0066] As a preferred embodiment, the cross-hypernode resource allocation strategy is an idle computing node allocation strategy aimed at minimizing the cross-hypernode communication cost; the cross-hypernode communication cost is used to represent the amount of training task communication data between any two computing devices located on different hypernodes per unit time. The computing node preemption strategy is a strategy executed based on the preset priority of each training task in the training task set and the preset priority of the target distributed training task; The idle computing node expansion strategy is a strategy implemented based on the difference between the required number of idle computing nodes and the number of physically contiguous idle computing nodes in the target supernode. The parallel execution strategy adjustment strategy is used to adjust the parallel execution strategy of the target distributed training task when the minimum number of idle computing nodes required supported by the target distributed training task is less than the current number of idle computing nodes required, so as to reduce the number of idle computing nodes required to the minimum number of idle computing nodes required, and use the minimum number of idle computing nodes required as the adjusted number of idle computing nodes required.

[0067] The GPU cluster scheduling system 100 provided in this embodiment of the invention obtains the regularity entropy, which characterizes the aggregation degree of idle computing nodes in the GPU cluster. When the regularity entropy is greater than a preset regularity entropy threshold, it can set a reserved supernode set in the GPU cluster based on the current distributed training task information and the number of idle computing nodes of each supernode. The idle computing nodes in the reserved supernode set are constrained to be used only for executing distributed training tasks. This can lock the complete resource space in advance for the target distributed training task to be executed, effectively reducing the degree of resource fragmentation and avoiding deadlock problems caused by severe resource fragmentation. As a result, it significantly improves the resource utilization of the GPU cluster and ensures the effective and efficient execution of distributed training tasks.

[0068] Please see Figure 3 The third aspect of the present invention provides an electronic device 200, including a memory 22, a processor 21, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the GPU cluster scheduling method described in any embodiment of the first aspect.

[0069] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 200.

[0070] The electronic device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 200 and does not constitute a limitation on the electronic device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device 200 may also include input / output devices, network access devices, buses, etc.

[0071] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 21 can be any conventional processor 21. The processor 21 is the control center of the electronic device 200, connecting various parts of the electronic device 200 via various interfaces and lines.

[0072] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the electronic device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the GPU cluster scheduling method described in any embodiment of the first aspect.

[0074] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, wherein when the computer program / instructions are executed by a processor, the steps of the GPU cluster scheduling method described in any embodiment of the first aspect are implemented.

[0075] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0076] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A GPU cluster scheduling method, characterized in that, include: The regularity entropy of the GPU cluster is determined based on the number of idle computing nodes in each supernode of the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. When the regularity entropy is greater than the preset regularity entropy threshold, the current set of reserved supernodes is determined based on the current distributed training task information and the number of idle computing nodes; wherein, the idle computing nodes in the set of reserved supernodes are used to execute distributed training tasks. Based on the preset resource mapping strategy of each candidate supernode for the target distributed training task to be executed and the resource information of each candidate supernode, the target distributed training task is scheduled; wherein each candidate supernode includes the set of reserved supernodes.

2. The GPU cluster scheduling method as described in claim 1, characterized in that, The step of determining the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster specifically includes: The total number of idle computing nodes in the GPU cluster is determined based on the number of idle computing nodes in each of the supernodes. The regularity entropy is determined based on the ratio of the number of idle computing nodes to the total number of idle computing nodes for each supernode and the logarithm of the ratio.

3. The GPU cluster scheduling method as described in claim 1, characterized in that, The step of determining the current set of retained supernodes based on the current distributed training task information and the number of idle computing nodes specifically includes: Based on the distributed training task information, determine the required number of idle computing nodes corresponding to the target distributed training task; The set of reserved supernodes is determined based on the required number of idle computing nodes and the number of idle computing nodes.

4. The GPU cluster scheduling method as described in claim 3, characterized in that, The scheduling of the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode specifically includes: Based on the communication cost between the various mapping computing devices indicated by the preset resource mapping strategy, the amount of cross-supernode communication data of the target distributed training task in each of the candidate supernodes is obtained; wherein, the computing node includes multiple computing devices; the communication cost is used to represent the amount of training task communication data between any two computing devices per unit time. The resource information and the amount of cross-supernode communication data are weighted to obtain the scheduling score corresponding to each candidate supernode; wherein, the resource information includes the number of idle computing nodes and the average remaining communication bandwidth within the supernode under the preset resource mapping strategy; The target distributed training task is scheduled based on the target supernode with the highest scheduling score.

5. The GPU cluster scheduling method as described in claim 4, characterized in that, The preset resource mapping strategy is determined with the goal of minimizing the total communication cost penalty; the total communication cost penalty is determined based on the product of the communication cost between each of the mapping computing devices and the communication penalty coefficient; the communication penalty coefficient is determined based on the communication bandwidth between each of the mapping computing devices.

6. The GPU cluster scheduling method as described in claim 4, characterized in that, The scheduling of the target distributed training task based on the target supernode with the highest scheduling score specifically includes: According to the required number of idle computing nodes, pre-occupancy information is sent to several physically contiguous idle computing nodes in the target supernode within a preset pre-occupancy time window, so that the idle computing nodes are in a pre-occupancy state. When the number of idle computing nodes with the pre-occupancy information is always less than the required number of idle computing nodes within the preset pre-occupancy time window, obtain the set of training tasks that have been assigned to the target supernode and do not belong to the distributed training tasks. The hot migration target is set as the number of physically contiguous idle computing nodes in the target supernode being greater than or equal to the required number of idle computing nodes. The hot migration strategy and hot migration time of the training task set are then determined. The hot migration strategy is executed when the hot migration time is less than the preset waiting time threshold corresponding to the target distributed training task. When the hot migration strategy is detected to be completed, the pre-occupancy information is sent to the idle computing nodes in the target supernode so that the number of idle computing nodes with the pre-occupancy information and physical contiguousness is equal to the number of idle computing nodes required, and the target distributed training task is assigned to each of the idle computing nodes with the pre-occupancy information and physical contiguousness for execution.

7. The GPU cluster scheduling method as described in claim 6, characterized in that, The hot migration time is determined based on the number of source computing devices corresponding to the training task set, the amount of task data to be migrated on each source computing device corresponding to the training task set, the additional task migration overhead corresponding to each training task in the training task set, and the effective bandwidth of the migration link.

8. The GPU cluster scheduling method as described in claim 6, characterized in that, The method further includes: When the hot migration time is greater than or equal to the preset waiting time threshold, a release operation is performed on each computing node that has the pre-occupancy information.

9. The GPU cluster scheduling method as described in claim 6, characterized in that, The method further includes: When the hot migration conditions are not met, a preset degraded scheduling strategy is executed; wherein, the hot migration conditions include the training task set supporting hot migration, the hot migration time being less than the preset waiting time threshold, and having at least one target computing device capable of migrating into the training task set; the degraded scheduling strategy includes at least one of the following: cross-supernode resource allocation strategy, computing node preemption strategy, idle computing node expansion strategy, and parallel execution strategy adjustment strategy. When the degraded scheduling strategy is completed, and the total number of idle computing nodes with the pre-occupancy information and the number of idle computing nodes determined based on the degraded scheduling strategy is greater than or equal to the required number of idle computing nodes or the adjusted number of idle computing nodes corresponding to the target distributed training task adjusted by the parallel execution strategy, the pre-occupancy information is sent to the idle computing nodes determined based on the degraded scheduling strategy, and the target distributed training task or the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution. When the downgrade scheduling strategy is completed and the number of idle computing nodes with the pre-occupancy information is greater than or equal to the adjusted number of idle computing nodes required, the target distributed training task adjusted by the parallel execution strategy is assigned to each of the idle computing nodes with the pre-occupancy information for execution.

10. The GPU cluster scheduling method as described in claim 9, characterized in that, The cross-hypernode resource allocation strategy is an idle computing node allocation strategy aimed at minimizing the cross-hypernode communication cost; The cross-supernode communication cost is used to represent the amount of training task communication data per unit time between any two computing devices located on different supernodes. The computing node preemption strategy is a strategy executed based on the preset priority of each training task in the training task set and the preset priority of the target distributed training task; The idle computing node expansion strategy is a strategy implemented based on the difference between the required number of idle computing nodes and the number of physically contiguous idle computing nodes in the target supernode. The parallel execution strategy adjustment strategy is used to adjust the parallel execution strategy of the target distributed training task when the minimum number of idle computing nodes required supported by the target distributed training task is less than the current number of idle computing nodes required, so as to reduce the number of idle computing nodes required to the minimum number of idle computing nodes required, and use the minimum number of idle computing nodes required as the adjusted number of idle computing nodes required.

11. A GPU cluster scheduling system, characterized in that, include: The regularity entropy acquisition module is used to determine the regularity entropy of the GPU cluster based on the number of idle computing nodes of each supernode in the GPU cluster; wherein, the supernode includes multiple computing nodes, and the regularity entropy is used to characterize the degree of aggregation of idle computing nodes in the GPU cluster. The module for determining the set of retained supernodes is used to determine the current set of retained supernodes based on the current distributed training task information and the number of idle computing nodes when the regularity entropy is greater than a preset regularity entropy threshold; wherein the idle computing nodes in the set of retained supernodes are used to execute distributed training tasks. The scheduling module is used to schedule the target distributed training task based on the preset resource mapping strategy of each candidate supernode and the resource information of each candidate supernode; wherein each candidate supernode includes the set of reserved supernodes.

12. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the GPU cluster scheduling method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the GPU cluster scheduling method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the steps of the GPU cluster scheduling method according to any one of claims 1 to 10.

Images