Task scheduling method and apparatus, electronic device, medium, and computer program product

Abstract

The embodiment discloses a task scheduling method and device, electronic equipment, medium and computer program product, wherein the task scheduling method comprises: determining an internal correlation degree of each topology structure in at least two topology structures; the internal correlation degree represents the correlation degree between each computing node in the topology structure; determining a topology weight of each topology structure according to the internal correlation degree of each topology structure; the topology weight is positively correlated with the internal correlation degree; determining a target topology structure for processing a task cluster in the at least two topology structures based on the topology weight of each topology structure; there are at least two to-be-processed tasks in the task cluster.

Description

Technical Field

[0001] This application belongs to the field of big data technology, and specifically relates to a task scheduling method, apparatus, electronic device, medium and computer program product. Background Technology

[0002] Distributed training task scheduling systems for intelligent computing primarily schedule tasks based on idle Graphics Processing Unit (GPU) resources. This scheduling method fails to consider the impact of topology on task scheduling and data interaction efficiency, resulting in insufficient intelligence in task scheduling. Furthermore, the efficiency of data interaction between tasks needs improvement. Summary of the Invention

[0003] This application provides a task scheduling method, apparatus, electronic device, medium, and computer program product.

[0004] This application provides a task scheduling method, the method comprising:

[0005] Determine the degree of internal correlation for each of at least two topologies; the degree of internal correlation represents the degree of correlation between the computing nodes in the topology.

[0006] The topological weight of each topology is determined based on the degree of internal correlation of each topology; the topological weight is positively correlated with the degree of internal correlation.

[0007] Based on the topology weight of each topology, a target topology for processing the task cluster is determined from the at least two topologies; the task cluster contains at least two tasks to be processed.

[0008] As can be seen, the method provided in this application embodiment can reflect the degree of correlation between various computing nodes within the topology through topological weights. Since the greater the degree of correlation between computing nodes within the topology, the faster the data interaction speed between computing nodes, determining the target topology weight for processing the task cluster through topological weights is beneficial to improve the data interaction speed of the tasks to be processed through the computing nodes in the target topology, thereby improving the overall processing efficiency of the task cluster.

[0009] In some embodiments, determining a target topology for processing the task cluster among the at least two topologies based on the topology weights of each topology includes: determining a target topology for processing the task cluster among the at least two topologies based on the topology weights of each topology and a first value of each topology; wherein the first value is determined based on the difference between the number of idle computing nodes in each topology and a first number; the first number is the number of computing nodes required to process the task cluster when computing nodes are allocated for a first task to be processed; wherein the first task to be processed is the first scheduled task to be processed in the task cluster.

[0010] It can be seen that by combining the topological weight and the first value to determine the target topology, it is beneficial to ensure that the number of idle computing nodes in the target topology meets the processing requirements of the task cluster, thereby achieving efficient processing of the task cluster.

[0011] In some embodiments, determining a target topology for processing the task cluster among the at least two topologies based on the topology weights of each topology and a first value of each topology includes: determining a first score for each topology based on the topology weights of each topology and the first value of each topology; wherein the first score is positively correlated with the topology weights and positively correlated with the first value; the first value is negatively correlated with the absolute value of the difference between the number of idle computing nodes in each topology and the first number; and determining a target topology for processing the task cluster based on the first score among the at least two topologies.

[0012] It can be seen that when the number of idle computing nodes in the topology is equal to the first number, the topology has the highest first score. This allows us to select the topology with the number of idle computing nodes closest to the first number as the target topology, which is beneficial for improving the resource utilization of the target topology.

[0013] In some embodiments, the method further includes: processing the m-th pending task in the task cluster via the n-th computing node in the target topology; wherein m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1; determining a second score for each computing node based on a first association degree between each computing node in the at least two topologies and the n-th computing node, and the idle computing resources of each computing node; and determining a computing node in the at least two topologies for processing the (m+1)-th pending task in the task cluster based on the second score of each computing node; wherein the (m+1)-th pending task represents a pending task scheduled after the m-th pending task.

[0014] As can be seen, the method in this embodiment is beneficial for obtaining computing nodes related to the computing nodes after the scheduling of the previous task, realizing efficient data interaction between tasks, and enabling the computing nodes of the next task to meet the task processing requirements of the next task, thereby improving task processing efficiency.

[0015] In some embodiments, determining the topology weight of each topology based on the degree of internal association of each topology includes: determining the topology weight of each topology based on the degree of internal association of each topology and a first similarity; wherein the topology weight is positively correlated with the degree of internal association and positively correlated with the first similarity; the first similarity represents the highest similarity among the similarities between the switch identifier in each topology and a preset switch identifier.

[0016] It can be seen that when the number of data interactions between tasks to be processed is low, determining the topology weight by combining the switch identifier is beneficial to use the topology containing the switch as the target topology to realize task processing, thereby improving the resource utilization of computing nodes in different topologies.

[0017] In some embodiments, when the task cluster is a data-parallel task, the preset switch identifier represents the Leaf switch identifier; when the task cluster is a pipelined parallel task, the preset switch identifier represents the Spine switch identifier.

[0018] As can be seen, since the number of data interactions in data-parallel tasks is greater than that in pipelined parallel tasks, the method presented in this embodiment helps to improve the processing efficiency of data-parallel tasks with a large amount of data interactions and enhances the intelligence of scheduling different parallel tasks.

[0019] In some embodiments, the method further includes: determining a target computing node for processing each of at least two target tasks in the task cluster; wherein there is data interaction between each target task; and in the event of a failure of any one of the target computing nodes, migrating the target task corresponding to the failed target computing node to a target computing node that has not failed.

[0020] As can be seen, the method presented in this embodiment helps to restore the task processing of faulty computing nodes in a timely manner and improves task processing efficiency.

[0021] This application embodiment also provides a task scheduling device, the device comprising:

[0022] A processing module is configured to determine the degree of internal correlation of each topology in at least two topologies; and to determine the topological weight of each topology based on the degree of internal correlation of each topology; wherein the degree of internal correlation represents the degree of correlation between computing nodes in the topology; and the topological weight is positively correlated with the degree of internal correlation.

[0023] The scheduling module is used to determine a target topology for processing the task cluster from the at least two topologies based on the topology weight of each topology; the task cluster contains at least two tasks to be processed.

[0024] This application provides an electronic device, which includes a processor and a memory for storing computer programs capable of running on the processor; wherein,

[0025] The processor is used to run the computer program to perform any of the above-described task scheduling methods.

[0026] This application provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described task scheduling methods.

[0027] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described task scheduling methods. Attached Figure Description

[0028] Figure 1 A flowchart of a task scheduling method provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of model segmentation for three-dimensional parallel training provided in an embodiment of this application;

[0030] Figure 3 A schematic diagram of a leaf ridge topology provided in an embodiment of this application;

[0031] Figure 4 A communication diagram of a cross-spine switch provided in an embodiment of this application;

[0032] Figure 5 A schematic diagram of a K8S architecture provided in an embodiment of this application;

[0033] Figure 6 A flowchart of a fault handling method provided in an embodiment of this application;

[0034] Figure 7 This is a schematic diagram of the structure of a task scheduling device provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] Currently, distributed training task scheduling systems for intelligent computing mainly schedule tasks based on idle GPU resources. This scheduling method does not take into account the impact of topology on scheduling, nor does it provide a method for recovering from computing node failures.

[0037] Additionally, while the open-source container orchestration platform Kubernetes (K8S)'s scheduler, Volcano, provides support for scheduling algorithms such as gang and binpack in distributed training scenarios for Artificial Intelligence (AI), Volcano cannot perform scheduling based on network topology. Furthermore, it cannot support automatic recovery of training tasks when underlying devices experience hardware failures.

[0038] To overcome the problems existing in related technologies and improve the intelligence of task scheduling and task processing efficiency, embodiments of this application provide a task scheduling method, apparatus, electronic device, medium, and computer program product. The task scheduling method provided in this application can determine the target topology for task processing by combining the degree of correlation between computing nodes in the topology, thereby improving the intelligence of task scheduling and task processing efficiency.

[0039] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0040] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other related elements in the method or apparatus that includes that element (e.g., steps in the method or units / modules in the apparatus; for example, units / modules in the apparatus may be portions of circuitry, processors, programs, or software, etc.).

[0041] The task scheduling method provided in this application includes a series of steps, but the task scheduling method provided in this application is not limited to the steps described. Similarly, the task scheduling device provided in this application includes a series of modules, but the device provided in this application is not limited to the modules explicitly described, and may also include modules that need to be set for obtaining relevant information or processing based on information.

[0042] This application provides a task scheduling method, such as... Figure 1 As shown, Figure 1 A flowchart of a task scheduling method is shown. Figure 1 The task scheduling methods shown include:

[0043] Step 101: Determine the degree of internal association between each of the at least two topologies.

[0044] The degree of internal correlation represents the degree of correlation between various computing nodes in the topology.

[0045] In this embodiment, computing nodes are the basic units in the topology. Computing nodes can communicate directly to exchange data, or they can communicate through a central node, such as a switch. The topology in this embodiment reflects the connection method between the computing nodes and can represent the physical layout of connecting computing nodes together using transmission media (such as switches). The topologies involved in this embodiment can include different forms such as star topology, ring topology, and mesh topology.

[0046] The computing nodes in this application embodiment can be personal computers, servers, virtual machines, workstations, etc. In cloud computing, computing nodes can specifically be physical machines, virtual computing resource units (Nodes), etc. This application embodiment does not specifically limit the type of computing nodes.

[0047] In at least two topologies, the degree of association between computing nodes within each topology can be determined based on the computing nodes contained in each topology, thus obtaining the internal association degree of each topology. Specifically, the degree of association between computing nodes can be determined by combining the node information and affiliation information of each computing node in each topology, thus obtaining the internal association degree of each topology. Node information may include computing node identifiers (IDs), computing node types, etc. Affiliation information may include the computing node's relationship, location, role, and management scope in the network or system, etc. Affiliation information may specifically include the computing node's geographical location, network affiliation, cluster affiliation, data center affiliation, availability zone affiliation, etc.

[0048] For example, the degree of internal correlation can be determined using calculation methods such as network density, average clustering coefficient, and connectivity. For instance, the average clustering coefficient measures the degree of clustering or separation of computing nodes in each topology; a higher average clustering coefficient indicates a stronger overall correlation between computing nodes in the topology. In practical applications, each computing node can also be labeled according to the topology. For example, a node in cloud computing, such as Node A, can be labeled "NodeA: Switch=Leaf1, Zone=ZoneA," indicating that for computing node Node A, the corresponding switch is Leaf1, and the availability zone is ZoneA. Based on the labeling of each computing node, the degree of correlation between computing nodes can be determined, thus obtaining the degree of internal correlation.

[0049] Step 102: Determine the topological weight of each topology based on the degree of internal correlation of each topology.

[0050] Among them, the topological weight is positively correlated with the degree of internal association.

[0051] In this embodiment, the higher the degree of correlation between the computing nodes in the topology, the greater the degree of internal correlation of the topology, and the greater the corresponding topology weight.

[0052] For example, topological weights can be determined based on the following preset conditions:

[0053] Topologies:

[0054] Calculate the node label NodeLabel: Availability Zone A; corresponding weight: 50;

[0055] The node label is NodeLabel: Switch1; corresponding weight: 100.

[0056] Based on the preset conditions shown above, when the label of a compute node in the topology includes Zone A, the topology weight of the topology is increased by 50; when the label of a compute node in the topology includes Switch1, the topology weight of the topology is increased by 100. Finally, the topology weight of the topology is accumulated based on the label of each compute node in the topology.

[0057] As can be seen, the topology weight reflects the overall correlation between computing nodes in the topology. The higher the correlation between computing nodes in the topology, the greater the corresponding topology weight.

[0058] Step 103: Based on the topology weight of each topology, determine the target topology for processing the task cluster from at least two topologies.

[0059] There must be at least two tasks to be processed in the task cluster.

[0060] Once the topological weights of each topology are obtained, the target topology is determined based on these weights. Specifically, the target topology can be determined based on the processing requirements of the task cluster. For example, when the real-time requirements of the task cluster are high, or when the correlation between the tasks to be processed in the task cluster is greater than a correlation threshold, the topology with the larger topological weight can be used as the target topology.

[0061] When each task in the task cluster can run independently and no data interaction is required between tasks, the topology structure satisfying the first weight interval can be used as the target topology structure. When the correlation between tasks in the task cluster is low, such as when the correlation is less than or equal to the correlation threshold, and it is considered that only a small number of tasks need to interact, the topology structure satisfying the second weight interval can be used as the target topology structure. The value of the second weight interval is greater than the value of the first weight interval.

[0062] The task cluster is processed using computing nodes within the target topology. After determining the target topology, the tasks to be processed in the task cluster can be processed based on the computing nodes within the target topology. In this embodiment, all tasks to be processed in the task cluster can be processed preferentially using computing nodes within the target topology, and tasks with a correlation degree greater than the correlation degree threshold can be centrally allocated to a single computing node for processing.

[0063] In practical applications, when the idle computing resources of the computing nodes in the target topology cannot meet the processing needs of the task cluster, the tasks can be prioritized based on their urgency, importance, and interrelationship. The target topology then processes higher-priority tasks first, followed by lower-priority tasks. Alternatively, the target topology can handle higher-priority tasks, while new topologies (e.g., other topologies from at least two existing topologies) are added to process lower-priority tasks. This combined approach of using the target topology and other topologies effectively processes the task cluster.

[0064] This application provides a task scheduling method, specifically a topology affinity scheduling method. By combining the degree of association between computing nodes in the topology, the topology weight of the topology is determined. The target topology is then determined by referring to the topology weight for task processing. This method helps to process multiple tasks with high degree of association, such as in distributed parallel task processing, by using computing nodes in the topology with high internal association to process the task cluster. This improves the intelligence of task scheduling, enhances the efficiency of data interaction between tasks, and improves the processing efficiency of the task cluster.

[0065] In practical applications, steps 101 to 103 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor.

[0066] Based on the task scheduling method given in the above embodiments, in order to ensure that the computing nodes in the target topology meet the processing requirements of the task cluster, in some embodiments, the above-mentioned determination of the target topology for processing the task cluster based on the topology weight of each topology includes: determining the target topology for processing the task cluster based on the topology weight of each topology and a first value of each topology; wherein, the first value is determined based on the difference between the number of idle computing nodes in each topology and a first number; the first number is the number of computing nodes required to process the task cluster when computing nodes are allocated to the first task to be processed; wherein, the first task to be processed is the first scheduled task to be processed in the task cluster.

[0067] Since the number of compute nodes varies across topologies, and these nodes may contain tasks being processed, a preliminary number of compute nodes required to process the task cluster can be determined before finalizing the target topology—that is, when scheduling the first task to be processed. Here, the first task to be processed can be the first task scheduled in the cluster. For example, each task can be analyzed to assess its required memory capacity, network bandwidth, number of CPU cores, etc., and the preliminary number can be estimated based on the known configuration of the compute nodes.

[0068] A first value is determined based on the difference between the number of idle computing nodes in each topology and a first value. When the first value is 0, it indicates that the idle computing nodes in the topology exactly meet the processing requirements of the task cluster, and this topology can be used as the target topology.

[0069] When there is no topology with a first value of 0, a topology with more than a first number of idle computing nodes can be used as the target topology.

[0070] In order to achieve efficient utilization of computing nodes in the topology, in practical applications, when there is no topology with a first value of 0, the topology with a number of idle computing nodes greater than the first number and the smallest first value can be selected as the target topology.

[0071] It can be seen that by combining the first value to determine the target topology, it is beneficial to select the target topology with the number of idle computing nodes that meets the first requirement from at least two topologies, and at the same time, it is beneficial to improve the resource utilization of computing nodes in the target topology.

[0072] Based on the method given in the above embodiments, in some embodiments, determining the target topology for processing the task cluster among at least two topologies based on the topology weight of each topology and the first value of each topology includes: determining a first score for each topology based on the topology weight of each topology and the first value of each topology; wherein the first score is positively correlated with the topology weight and positively correlated with the first value; the first value is negatively correlated with the absolute value of the difference between the number of idle computing nodes and a first number in each topology; and determining the target topology for processing the task cluster among at least two topologies based on the first score.

[0073] In this embodiment, a first score for each topology can be determined using the first numerical value and topology weight described in the previous embodiments. The first score is used to determine the corresponding target topology for each task cluster. For example, based on the first score and a preset score range for each topology, each topology can be divided into three clusters: topologies with a first score greater than the score range are designated as the first topology cluster; topologies with a first score within the score range are designated as the second topology cluster; and topologies with a first score less than the score range are designated as the third topology cluster. Based on the degree of correlation between tasks to be processed within the task cluster and the real-time requirements of task processing, different task clusters are processed using the topologies within different topology clusters.

[0074] The first value can be determined based on the absolute value of the difference between the number of idle computing nodes in each topology and the first value. Based on the above discussion, when the difference between the number of idle computing nodes in the topology and the first value is 0, the absolute value is the smallest. In this case, the first value can be determined to be the largest, and it is considered that when the first value is 0, the utilization rate of computing nodes in the topology is the highest.

[0075] In practical applications, when the difference between the number of idle computing nodes in the topology and the initial number is not zero, two scenarios exist: First, the number of idle computing nodes in the topology is greater than the initial number; second, the number of idle computing nodes in the topology is less than the initial number. In the first scenario, when processing a task cluster using computing nodes in the topology, there will be surplus computing node resources, resulting in wasted computing resources in the topology. In the second scenario, when processing a task cluster using computing nodes in the topology, some pending tasks may not be processed by any computing nodes. In this case, the priority of the pending tasks can be considered for processing.

[0076] To achieve rapid processing of task clusters, especially when handling parallel tasks, the tasks are often highly correlated and need to be processed simultaneously. Therefore, the target topology can be determined first in the first scenario. Specifically, before determining the first score, the relationship between the number of idle computing nodes in the topology and the first threshold can be assessed. The first score of topologies with fewer idle computing nodes than the first threshold can be decreased, or the first score of topologies with more idle computing nodes than the first threshold can be increased. This ensures that when determining the target topology based on the first score, topologies with more idle computing nodes than the first threshold are preferentially selected as the target topology.

[0077] When the number of idle computing nodes in each topology is less than a first number, it is considered that the current topology cannot meet the task processing needs of the task cluster. In this case, based on the first value, the topology with the smallest absolute value of the difference between the number of idle computing nodes and the first number can be determined as the target topology, so that the target topology can handle more tasks in the task cluster and improve the processing efficiency of the task cluster.

[0078] For example, based on the method given in this embodiment, a further method is provided for calculating a first value of the topology based on the current number of idle computing nodes and a first number (expected). The calculation formula below is based on an exponential decay mathematical model, ensuring that the first value is higher when the number of idle computing nodes is close to the first number, and lower otherwise. The formula for calculating the first value is as follows:

[0079] When current == expected, it means that the idle computing nodes in the current topology just meet the processing needs of the task cluster. The first value, Score, is determined using the method in formula (1):

[0080] Score=Max (1)

[0081] Here, Max represents the maximum value of the preset first value. For example, Max can be equal to 100.

[0082] When current > expected, it means that the idle computing nodes in the current topology can meet the processing needs of the task cluster. The first value, Score, is determined using the method in formula (2):

[0083] (2)

[0084] When current < expected, it means that the number of idle computing nodes in the current topology cannot meet the processing requirements of the task cluster. However, the more idle computing nodes there are in the topology, the better the processing efficiency of the task cluster will be. In this case, the first value Score is determined by the method in formula (3):

[0085] (3)

[0086] In formulas (2) and (3) above, k represents the attenuation rate. The score threshold for computing nodes can be set based on empirical values. When the number of idle computing nodes is less than the first number, the value of k is the preset maximum value, that is, the value of k in formula (3) can be greater than the value of k in formula (2).

[0087] The method presented in this embodiment helps to improve the resource utilization of computing nodes in the target topology and reduce resource fragmentation in the target topology during the process of determining the target topology.

[0088] To determine the target computing node for each pending task in the task cluster and to schedule the pending tasks, in some embodiments, the method further includes: processing the m-th pending task in the task cluster through the n-th computing node in the target topology; where m is an integer greater than or equal to 1 and n is an integer greater than or equal to 1; determining a second score for each computing node based on a first association degree between each computing node and the n-th computing node in at least two topologies, and the idle computing resources of each computing node; and determining, based on the second score of each computing node, a computing node for processing the (m+1)-th pending task in the task cluster in at least two topologies; the (m+1)-th pending task represents a pending task scheduled after the m-th pending task.

[0089] When m equals 1, that is, when scheduling the first task to be processed, the target topology corresponding to the first task to be processed can be determined based on the method given in the above embodiments, based on the topology weight of each topology and the first value of each topology.

[0090] When scheduling tasks after the first task to be processed, the computing nodes for scheduling tasks can be determined based on the computing nodes of the previous task to be processed and the idle computing resources of each computing node in the entire topology.

[0091] For example, when the scheduled task to be processed is the next task to be processed after the first task to be processed, i.e., the second task to be processed, the computing node of the first task to be processed can be obtained, assuming it is the first computing node. Then, the first association degree between each computing node and the first computing node is calculated. Here, the first association degree can be determined based on one or more dimensions among the communication mode, data dependency, topology, and performance metrics of each computing node and the first computing node. Each computing node includes the first computing node and all other computing nodes except the first computing node.

[0092] It's understandable that a compute node in a topology can include multiple processing devices such as GPUs; therefore, a single compute node can run multiple tasks. Taking a compute node as an example in cloud computing, during task scheduling, each task can be assigned a Pod, and a Node can provide computing resources for one or more Pods. Therefore, when determining the compute node for the second task, the initial correlation between the first compute node and each other can be calculated. It can be seen that the initial correlation between the first compute node and each other is the highest. However, since the first compute node is already used to process the first task, its idle computing resources may be less than those of other compute nodes. In this case, based on the initial correlation between the first compute node and each other, combined with the idle computing resources of each compute node, it's possible to determine the compute node in the topology that has a high correlation with the first compute node and sufficient idle computing resources as the compute node for the second task, and then process the second task.

[0093] For any non-first scheduled task to be processed, based on the method given in this embodiment, when the (m+1)th task to be processed is determined, a computing node with sufficient idle computing resources related to the nth computing node can be determined to process the (m+1)th task to be processed, based on the nth computing node of the m-th task to be processed.

[0094] It is understandable that when scheduling the (m+1)th pending task, when determining the first affinity based on each computing node in at least two topologies, the first affinity corresponding to each computing node in the target topology is usually greater than or equal to the first affinity corresponding to any computing node in any of the other topologies in the at least two topologies. Therefore, based on the method of this embodiment, when the idle computing resources of the computing nodes in the target topology meet the processing requirements of the (m+1)th pending task, a computing node for processing the (m+1)th pending task will be determined in the target topology. Only when the idle computing resources of the computing nodes in the target topology cannot meet the processing requirements of the (m+1)th pending task will a computing node for processing the (m+1)th pending task be determined in other topologies.

[0095] When there is little data interaction between tasks in a task cluster, and the task cluster contains tasks that can be processed independently, in order to achieve balanced utilization of different topologies and improve the utilization rate of each computing node in a large-scale topology, in some embodiments, the above-mentioned determination of the topology weight of each topology based on the internal correlation degree of each topology includes: determining the topology weight of each topology based on the internal correlation degree of each topology and a first similarity; wherein, the topology weight is positively correlated with the internal correlation degree and positively correlated with the first similarity; the first similarity represents the highest similarity among the similarities between the switch identifier in each topology and the preset switch identifier.

[0096] If the number of data interactions between tasks in the task cluster is small, for example, if the number of data interactions between tasks is less than the first threshold, the topological weight of each topological structure can be determined based on the degree of internal association of each topological structure and the first similarity.

[0097] In the topology, switches are used to enable communication between different computing nodes. Within each computing node, inter-card communication (inter-processor communication) is used to achieve high-speed, low-latency data exchange between different functional modules, such as network cards, memory cards, and GPUs. Therefore, when the number of data interactions between pending tasks in the task cluster is greater than or equal to a first threshold, these tasks can be prioritized and scheduled to run within a computing node for data interaction via inter-card communication. When the number of data interactions between pending tasks in the task cluster is less than the first threshold, these tasks can be scheduled to a topology containing switches for processing.

[0098] In practical applications, different topologies may contain different numbers and types of switches. The more complex the topology and the more computing nodes a topology has, the more switches it may contain. When data is transmitted through switches, data transmission efficiency is inevitably affected. The more switches used in the data transmission process, the greater the data transmission latency. Therefore, based on the number of switches in each topology, we can determine the task clusters corresponding to topologies containing different numbers of switches. The number of data interactions between tasks in a task cluster is inversely proportional to the number of switches in the corresponding target topology.

[0099] When the topology weight is determined by combining the first similarity, different task clusters can be scheduled to a target topology containing different switches for processing by updating the preset switch identifier.

[0100] In this embodiment, task clusters with fewer data interactions can be scheduled to a topology containing switches for task processing. This is beneficial for improving the utilization rate of the topology containing switches when the number of data interactions for the tasks to be processed is small.

[0101] Furthermore, when task scheduling is required for parallel tasks, in some embodiments, when the task cluster is a data parallel task, the preset switch identifier represents the Leaf switch identifier; when the task cluster is a pipeline parallel task, the preset switch identifier represents the Spine switch identifier.

[0102] In model training, since large models often have hundreds of billions or even trillions of parameters, the processing power and memory of a single machine are often insufficient to accommodate the training of large models. Therefore, it is usually necessary to use a hybrid parallel strategy to cut the model into multiple dimensions before training. The three commonly used parallel strategies include data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP).

[0103] Figure 2 This diagram illustrates a model segmentation method for three-dimensional (3D) parallel training, which includes three parallel modes: DP, TP, and PP. Figure 2 All the cubes shown represent a complete model training task. For example, the model training task can be divided along the y-axis based on the specific parallel processing requirements of DP, along the x-axis based on the specific parallel processing requirements of PP, and along the z-axis based on the specific parallel processing requirements of TP. Ultimately, the model training task can be divided into the following segments: Figure 2 The 32 tasks to be processed shown form a task cluster. Here, Figure 2 The 32 tasks shown are exemplary. In practical applications, the model training task can be divided into different numbers of tasks based on specific parallel processing requirements. One task is... Figure 2 In a cube, a GPU needs to be allocated. In cloud computing, a Pod can be designated to execute a task to be processed, and computing resources can be provided to one or more Pods through compute nodes.

[0104] Model training often requires a large amount of data communication, and the amount of communication varies depending on the parallel strategy. Typically, parallel processing (TP) has the highest communication volume, followed by dynamic processing (DP), and parallel processing (PP) has the lowest. In terms of communication speed, intra-machine communication is generally implemented using NVLink or a high-speed cache coherence system like HCCs, and its speed is greater than that of inter-machine communication. Intra-machine communication can include communication between compute nodes, virtual machines, etc. In large-scale distributed training scenarios, inter-machine communication typically employs a multi-track networking approach, such as... Figure 3 As shown. Figure 3 A schematic diagram of a leaf ridge topology is shown, wherein, with Figure 3 Taking a compute node as an example, compute node 1 is Node 1, compute node 2 is Node 2, and compute node N is Node N. Different GPUs in the same Node can interact with each other through intra-machine communication (such as inter-card communication). GPUs with the same serial number in different Nodes can interact with each other through Leaf switches. GPUs with different serial numbers in different Nodes need to interact with each other through Leaf switches and Spine switches.

[0105] For example, in multi-track networking, processors with the same serial number within 16 or 32 computing nodes can be interconnected. Figure 3 On this basis, Figure 4 A communication diagram of a cross-ridge switch is shown, in order to Figure 4 Taking the computing nodes as an example, namely computing node 1 is Node 1 and computing node N is Node N, it can be seen that processors with the same sequence number in different computing nodes only need 2 hops to complete data interaction, but when processors with different sequence numbers in different groups communicate, 4 hops are required.

[0106] Based on the method presented in this embodiment, when the task cluster for model training includes both data-parallel tasks and pipelined parallel tasks, parallel tasks with higher communication volumes can be scheduled to the same switch, which can significantly improve communication speed and thus increase training efficiency. Specifically, data-parallel tasks can be placed under the same leaf switch, while pipelined parallel tasks, due to their lower data interaction volume, can be placed under a spine switch that spans leaf switches.

[0107] In practical applications, when the tasks to be processed in the task cluster are data parallel tasks, by setting the preset switch identifier to the Leaf switch identifier, a larger topology weight can be determined for the topology where the leaf switch is located, so that the data parallel tasks are scheduled to be processed in the same topology under the same leaf switch.

[0108] When the tasks to be processed in the task cluster are pipelined parallel tasks, by setting the preset switch identifier to the Spine switch identifier, a larger topology weight can be determined for the topology where the Spine switch is located, so that the pipelined parallel tasks are scheduled to be processed in the same topology under the same Spine switch.

[0109] When the tasks to be processed in the task cluster are tensor parallel tasks, the tensor parallel tasks can be preferentially placed in the computing nodes and data interaction can be achieved through inter-card communication based on the method of the above embodiments.

[0110] For example, when implementing large model training tasks (workers) through cloud computing, the tasks to be processed in the model training task are logically grouped according to the 3D parallel settings of the model training task, such as pipeline parallel groups, data parallel groups, tensor parallel groups, etc., to obtain different task clusters.

[0111] After that, for the grouped tasks to be processed, the group labels are marked on the Pod. For example, worker: TP=groupN, DP=groupM, PP=groupO. Here, N, M, and O are specifically used to mark the number of parallel tasks. For example, when the number of parallel tasks for TP is set to 6, N equals 6; when the number of parallel tasks for DP is set to 2, M equals 2; when the number of parallel tasks for PP is set to 3, O equals 3.

[0112] During task scheduling, the validity of the task cluster is checked first, and then the tasks to be processed are sorted according to the Rank function and added to the priority queue in turn.

[0113] Before scheduling the first task to be processed, based on the method given in the above embodiments, the topology weight of each topology can be determined, the first score of each topology can be determined, and the correspondence between topology and topology weight, and the correspondence between each topology and computing node can be initialized. Before scheduling the first task to be processed, the second score of each computing node can be initialized, for example, the second score of each computing node can be initialized to 0. Based on the determined first score, the target topology corresponding to the first Pod of the first task to be processed is determined, and computing resources are provided to the first Pod based on a computing node in the target topology.

[0114] After scheduling the first task to be processed, the second score of each computing node is updated based on the resource usage information of the computing nodes in the target topology for the first task to be processed.

[0115] When scheduling the second pending task, to avoid the impact of user changes to compute node affiliation on the topology, the mapping between each topology and compute node can be re-initialized. This involves iterating through the information of each compute node, such as resource utilization and the degree of association between compute nodes. Then, if a compute node is in the same topology as the compute node of the first pending task, its second score is updated; otherwise, the second score is not updated. Based on the final second score of each compute node, the compute node for the second Pod handling the second pending task is determined.

[0116] The following further describes a method for handling computing node failures. In some embodiments, the method further includes: determining a target computing node for processing each of at least two target tasks in a task cluster; wherein there is data interaction between each target task; and in the event of a failure of any target computing node, migrating the target task corresponding to the failed target computing node to a non-failed target computing node.

[0117] In this embodiment, the processing status of tasks in each computing node can be obtained in real time, along with the running information of each computing node. When a computing node experiences a hardware failure, the failure information is reported promptly, and the tasks and pending tasks in the computing node are migrated to computing nodes that have not experienced a failure. For multiple pending tasks that are related, such as when processing pipelined parallel tasks, if the computing node of the first task fails, the tasks related to the first task also need to be reprocessed to ensure task data synchronization. In this case, the first task can be migrated to a computing node that has not experienced a failure, and the first task can be processed based on the migrated computing node. At the same time, for tasks related to the first task in computing nodes that have not experienced a failure, the task can be restarted in place on the corresponding computing node that has not experienced a failure, and the first task and the tasks related to the first task can be processed synchronously. In this embodiment, the target pending task can be a task scheduled to a computing node that has not yet started processing. In practical applications, the target pending task can also be a task being processed in a computing node.

[0118] Taking a computing node failure in a cloud computing scenario as an example, Figure 5 A schematic diagram of a K8S architecture is shown. The scheduling method given in this application embodiment can be run as a container in a K8S cluster. Figure 5 The K8S architecture shown includes major components such as scheduler 501, monitor 502, and rescheduler 503, as well as kube-apiserver 504 and driver interface 505. Figure 5Containers serve as the carriers for task processing and application execution, while Pods are the encapsulation and management units of containers in Kubernetes. The scheduler 501 is responsible for receiving container events (such as new Pod creation) through kube-apiserver 504, monitoring and scheduling the task cluster, and allocating compute nodes to the Pods corresponding to pending tasks via kube-apiserver 504. The monitor 502 is responsible for collecting hardware fault information from devices, such as processor and network fault information. The rescheduler 503, based on an event processing mechanism, obtains compute node fault information from monitor 502 and reschedules affected or pending tasks via kube-apiserver 504. kube-apiserver 504 is a built-in component of Kubernetes, providing a task management interface, receiving container events, and notifying scheduler 501. The driver interface 505 shields underlying differences, enabling Kubernetes to schedule and manage diverse resources in a unified manner.

[0119] Based on the method of this embodiment, Figure 6 A flowchart of a fault handling method is shown, such as Figure 6 As shown, the monitor 502 collects fault information 601 from hardware of different manufacturers and types at the underlying level, and manages and maintains the fault information. After collecting the fault information, the monitor 502 reports the fault information to the rescheduler 503.

[0120] After receiving the fault information reported by the monitor 502, the rescheduler 503 marks the tasks that need to be rescheduled due to hardware failures or the tasks that are pending. The rescheduler 503 can also periodically check the processing status of each task and, in combination with the task status information and task job settings, determine whether the task needs to be rescheduled by software.

[0121] If the failure is a hardware fault in a compute node, the rescheduler 503 instructs the scheduler 501 to filter out the compute node or specific hardware devices within that compute node during the pre-selection phase, thus achieving hardware isolation of the fault. If the hardware fault is a chip fault, and the fault type is determined to be recoverable by restarting, the rescheduler 503 calls the processor driver to reset and restart the faulty chip.

[0122] The rescheduler 503 can also record information about normal compute nodes and processors.

[0123] When rescheduling tasks on a faulty compute node is required, all tasks needing rescheduling can be deleted first, and task re-creation can be triggered. Here, all tasks needing rescheduling include rescheduled task 602 (the target task to be processed corresponding to the faulty compute node) on the faulty compute node, and other rescheduled tasks related to rescheduled task 602 (other target tasks to be processed). For other rescheduled tasks, scheduler 501 launches the task in place based on its original compute node and processor information. For rescheduled task 602, due to compute node and processor isolation, rescheduled task 602 will be launched on a new compute node and processor.

[0124] This application provides a task scheduling method that, by combining the underlying topology, offers a topology affinity scheduling method. This method enhances the scheduling capabilities of specific processors within computing nodes, providing superior network performance and processing efficiency for large-scale task clusters. Furthermore, in the event of a computing node failure, the method described in this application enables rapid processor isolation, computing node isolation, and quick recovery of task rescheduling.

[0125] For large-scale distributed training tasks, it provides scheduling services for training jobs in the cloud-native ecosystem, extracts the common requirements of AI training jobs, and schedules tasks through topology. It can shield the differences between AI chips from different manufacturers and the differences between various model training frameworks, providing cloud-native training jobs with transparent, topology-optimized, and self-healing scheduling capabilities for node failures.

[0126] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0127] Based on the task scheduling method proposed in the foregoing embodiments, this application also provides a task scheduling device. Figure 7 A schematic diagram of a task scheduling device is shown, such as... Figure 7 As shown, the task scheduling device includes:

[0128] Processing module 701 is used to determine the degree of internal correlation of each topology in at least two topologies; and to determine the topological weight of each topology based on the degree of internal correlation of each topology; the degree of internal correlation represents the degree of correlation between each computing node in the topology; the topological weight is positively correlated with the degree of internal correlation.

[0129] The scheduling module 702 is used to determine the target topology for processing the task cluster from at least two topologies based on the topology weight of each topology; there are at least two tasks to be processed in the task cluster.

[0130] In practical applications, the processing module 701 and the scheduling module 702 can be implemented based on a processor and a communication device.

[0131] In some embodiments, the scheduling module 702 is specifically configured to determine a target topology for processing the task cluster in at least two topologies based on the topology weight of each topology and a first value of each topology; wherein the first value is determined based on the difference between the number of idle computing nodes in each topology and a first number; the first number is the number of computing nodes required to process the task cluster when computing nodes are allocated to a first task to be processed; wherein the first task to be processed is the first scheduled task to be processed in the task cluster.

[0132] In some embodiments, the scheduling module 702 is specifically configured to determine a first score for each topology based on the topology weight of each topology and a first value of each topology; wherein the first score is positively correlated with the topology weight and positively correlated with the first value; the first value is negatively correlated with the absolute value of the difference between the number of idle computing nodes and a first number in each topology; and based on the first score, determine a target topology for processing the task cluster in at least two topologies.

[0133] In some embodiments, the scheduling module 702 is further configured to process the m-th pending task in the task cluster through the n-th computing node in the target topology; where m is an integer greater than or equal to 1 and n is an integer greater than or equal to 1; determine a second score for each computing node based on a first association degree between each computing node in at least two topologies and the n-th computing node, and the idle computing resources of each computing node; and determine, based on the second score of each computing node, a computing node for processing the (m+1)-th pending task in the task cluster in at least two topologies; the (m+1)-th pending task represents a pending task scheduled after the m-th pending task.

[0134] In some embodiments, the processing module 701 is specifically used to determine the topological weight of each topological structure based on the degree of internal association of each topological structure and the first similarity; wherein the topological weight is positively correlated with the degree of internal association and positively correlated with the first similarity; the first similarity represents the highest similarity among the similarities between the switch identifier in each topological structure and the preset switch identifier.

[0135] In some embodiments, the task scheduling device further includes a rescheduling module; the rescheduling module is used to determine the target computing node for processing each of at least two target tasks in the task cluster; wherein there is data interaction between each target task; and in the event of a failure of any target computing node, the target task corresponding to the failed target computing node is migrated to the target computing node that has not failed.

[0136] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0137] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0138] This application also provides an electronic device. Figure 8 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 8 As shown, the electronic device 80 may include:

[0139] Memory 801 is used to store executable instructions.

[0140] The processor 802 is used to implement any of the above-described task scheduling methods when executing executable instructions stored in the memory 801.

[0141] The processor 802 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0142] The aforementioned computer-readable storage medium or memory 801 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0143] This application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the task scheduling methods provided in the above embodiments.

[0144] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the task scheduling methods provided in the above embodiments.

[0145] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0146] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0147] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0148] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0149] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of protection of this application, and these forms are all within the protection scope of this application.

Claims

1. A task scheduling method, characterized in that, The method includes: Determine the degree of internal correlation for each of at least two topologies; the degree of internal correlation represents the degree of correlation between the computing nodes in the topology. The topological weight of each topology is determined based on the degree of internal correlation of each topology; the topological weight is positively correlated with the degree of internal correlation. Based on the topology weight of each topology, a target topology for processing the task cluster is determined from the at least two topologies; the task cluster contains at least two tasks to be processed. The step of determining the target topology for processing the task cluster from the at least two topologies based on the topology weights of each topology includes: Based on the topology weight of each topology and a first value of each topology, a target topology for processing the task cluster is determined among the at least two topologies; wherein, the first value is determined based on the difference between the number of idle computing nodes in each topology and a first number; the first number is the number of computing nodes required to process the task cluster when computing nodes are allocated to a first task to be processed; wherein, the first task to be processed is the first scheduled task to be processed in the task cluster.

2. The method according to claim 1, characterized in that, The determination of the target topology for processing the task cluster from the at least two topologies based on the topology weight of each topology and the first value of each topology includes: Based on the topological weight of each topology and the first value of each topology, a first score is determined for each topology; wherein the first score is positively correlated with the topological weight and positively correlated with the first value; the first value is negatively correlated with the absolute value of the difference between the number of idle computing nodes in each topology and the first number. Based on the first score, a target topology for processing the task cluster is determined from the at least two topologies.

3. The method according to claim 1, characterized in that, The method further includes: The m-th task in the task cluster is processed by the n-th computing node in the target topology; where m is an integer greater than or equal to 1 and n is an integer greater than or equal to 1. Based on the first association degree between each computing node and the nth computing node in the at least two topologies, and the idle computing resources of each computing node, a second score is determined for each computing node; Based on the second score of each computing node, a computing node is determined in the at least two topologies to process the (m+1)th pending task in the task cluster; the (m+1)th pending task represents a pending task scheduled after the m-th pending task.

4. The method according to claim 1, characterized in that, The step of determining the topological weight of each topological structure based on the degree of internal correlation of each topological structure includes: The topological weight of each topology is determined based on the degree of internal association of each topology and the first similarity; wherein the topological weight is positively correlated with the degree of internal association and positively correlated with the first similarity; the first similarity represents the highest similarity among the similarities between the switch identifier in each topology and the preset switch identifier.

5. The method according to claim 4, characterized in that, In the case that the task cluster is a data-parallel task, the preset switch identifier represents the Leaf switch identifier; In the case that the task cluster is a pipelined parallel task, the preset switch identifier represents the Spine switch identifier.

6. The method according to claim 1, characterized in that, The method further includes: Identify target computing nodes for processing each of at least two target tasks in the task cluster; wherein data interaction exists between each target task. If any of the target computing nodes fails, the target tasks to be processed corresponding to the failed target computing node will be migrated to the target computing nodes that have not failed.

7. A task scheduling device, characterized in that, The device includes: A processing module is configured to determine the degree of internal correlation of each topology in at least two topologies; and to determine the topological weight of each topology based on the degree of internal correlation of each topology; wherein the degree of internal correlation represents the degree of correlation between computing nodes in the topology; and the topological weight is positively correlated with the degree of internal correlation. A scheduling module is configured to determine a target topology for processing the task cluster from among at least two topologies based on the topology weight of each topology and a first value of each topology; wherein the first value is determined based on the difference between the number of idle computing nodes in each topology and a first number; the first number is the number of computing nodes required to process the task cluster when computing nodes are allocated to a first task to be processed; wherein the first task to be processed is the first scheduled task to be processed in the task cluster; and there are at least two tasks to be processed in the task cluster.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing computer programs capable of running on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images