Rescheduling method, device, system and equipment for training tasks, medium and product

Abstract

The embodiment of the invention discloses a rescheduling method, device, system and equipment for training tasks, a medium and a product, and belongs to the technical field of artificial intelligence. The method comprises the steps that when it is determined that a rescheduling opportunity is reached, a first index of a current topology to which a training task is scheduled is determined, the current topology comprises a plurality of processing units, and the first index represents the communication cost of the current topology; determining a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; determining a second index of the rescheduling topology, wherein the second index represents the communication cost of the rescheduling topology; and rescheduling the training task as rescheduling topology when the second index is smaller than the first index. And the training task is rescheduled into a better topology with lower communication cost, so that the communication efficiency and the training efficiency can be improved. When an abnormality is detected, the processing unit is reconstructed instead of simple error reporting, and stability and reliability can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI) technology, and more specifically, to a method, device, system, equipment, medium and product for rescheduling training tasks. Background Art

[0002] ‌A large model refers to an AI model with a large number of parameters, such as billions or even hundreds of billions of parameters. Large models can handle complex tasks and generate high-quality results, and are widely used in fields such as natural language processing and computer vision. The parallel strategy for large model training has evolved from early data parallelism to the now widely used 3D parallelism (a combination of data parallelism, model parallelism, and pipeline parallelism), forming a multi-dimensional parallel training framework.

[0003] As the scale of models continues to increase, the clusters used to train large models have higher and higher requirements for communication bandwidth. In current large model training, training tasks are assigned to specific topologies in the cluster according to preset rules before training begins. However, as training continues, the communication efficiency of the topology that executes the training tasks may decrease, thereby affecting the training efficiency. In addition, when a failure occurs (for example, a GPU failure or a node failure), error processing is directly performed, causing the entire cluster to be affected, posing a challenge to the stability and reliability of training. Summary of the invention

[0004] The present invention provides a method, device, system, equipment, medium and product for rescheduling training tasks, which are helpful to improve training efficiency.

[0005] The technical solution of the embodiment of the present invention is as follows: A method for rescheduling a training task, comprising: When it is determined that the rescheduling opportunity has arrived, determining a first indicator of a current topology to which the training task has been scheduled, the current topology comprising a plurality of processing units, the first indicator representing a communication cost of the current topology; Determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; Determining a second indicator of the rescheduling topology, the second indicator representing a communication cost of the rescheduling topology; When the second indicator is less than the first indicator, the training task is rescheduled to the rescheduling topology.

[0006] In one embodiment, the determining that a rescheduling opportunity has been reached includes at least one of the following: Determining that a usage rate of a cluster used to execute the training task is lower than a predetermined threshold or an overall topology structure of the cluster changes; Determining that a rescheduling instruction triggered by a user is received; Determine that a scheduled rescheduling time point has been reached.

[0007] In one embodiment, determining a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated includes: Virtually migrate a processing unit in a first node from a current position in the first node to another position in the first node that is different from the current position; or A processing unit in a first node is virtually migrated from a current position in the first node to a second node.

[0008] In one embodiment, the virtually migrating the processing unit in the first node from the current position in the first node to the second node includes: Virtually migrate a processing unit in a first node from a current location in the first node to a second node connected to the same switch node as the first node; or A processing unit in a first node is virtually migrated from a current position in the first node to a second node connected to a switch node different from the first node.

[0009] In one embodiment, determining the first indicator of the current topology to which the training task has been scheduled includes: determining the individual communication cost between each processing unit in the current topology and the remaining processing units in the current topology; calculating a first summation result of the individual communication costs of all processing units in the current topology; determining the first indicator based on the first summation result; Determining the second indicator of the rescheduling topology includes: determining the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology; calculating the second summation result of the individual communication costs of all processing units in the rescheduling topology; and determining the second indicator based on the second summation result.

[0010] In one embodiment, rescheduling the training task to the rescheduling topology includes: evicting the at least one processing unit; Rebuilding the at least one processing unit at a location in the rescheduling topology to which the at least one processing unit is virtually migrated.

[0011] In one embodiment, it includes: When a processing unit in an abnormal state is detected in the current topology, the processing unit in the abnormal state is expelled; When the processing unit in the abnormal state is located at a current position of the current topology, the processing unit is rebuilt.

[0012] In one embodiment, it includes: When a processing unit in an abnormal state is detected in the current topology, determining an abnormal processing topology formed after the processing unit in the abnormal state is virtually migrated; Determining a third indicator of the exception handling topology, wherein the third indicator represents a communication cost of the exception handling topology; When the third indicator is less than the first indicator, the processing unit in the abnormal state is expelled, and the processing unit is rebuilt at the position in the abnormal processing topology to which the processing unit in the abnormal state is virtually migrated; When the third index is greater than or equal to the first index, the processing unit in the abnormal state is expelled, and the processing unit in the abnormal state is rebuilt at the current position of the current topology.

[0013] A training task rescheduling device, comprising: A first determination module is used to determine a first indicator of a current topology to which the training task has been scheduled when it is determined that a rescheduling opportunity has been reached, the current topology including a plurality of processing units, and the first indicator represents a communication cost of the current topology; A second determining module, used to determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; A third determination module is used to determine a second indicator of the rescheduling topology, where the second indicator represents a communication cost of the rescheduling topology; A rescheduling module is used to reschedule the training task to the rescheduling topology when the second indicator is less than the first indicator.

[0014] A training task rescheduling system, comprising: A cluster, which consists of multiple nodes; a scheduler for scheduling training tasks to the cluster to form a current topology, the current topology comprising a plurality of processing units in respective nodes; A rescheduler is used to determine a first indicator of the current topology when it is determined that a rescheduling opportunity has been reached, the first indicator representing the communication cost of the current topology; determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; determine a second indicator of the rescheduling topology, the second indicator representing the communication cost of the rescheduling topology; when the second indicator is less than the first indicator, rescheduling the training task to the rescheduling topology.

[0015] An electronic device, comprising: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to perform any of the above-described methods for rescheduling training tasks.

[0016] A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor executes any of the above-described methods for rescheduling training tasks.

[0017] A program product includes a computer program, which, when executed by a processor, implements any of the above-described methods for rescheduling training tasks.

[0018] It can be seen from the above technical solution that when it is determined that the time for rescheduling has arrived, the first indicator of the current topology to which the training task has been scheduled is determined, the current topology includes multiple processing units, and the first indicator represents the communication cost of the current topology; the rescheduling topology formed after at least one processing unit in the current topology is virtually migrated is determined; the second indicator of the rescheduling topology is determined, and the second indicator represents the communication cost of the rescheduling topology; when the second indicator is less than the first indicator, the training task is rescheduled to the rescheduling topology. It can be seen that by rescheduling the training task to be executed by a topology with lower communication cost, it is possible to adapt to many dynamic changes in the cluster (such as load changes and cluster resource changes, etc.), thereby improving communication efficiency and training efficiency. In addition, when an anomaly is detected, the processing unit is rebuilt based on a variety of mechanisms instead of simply reporting an error, thereby improving the stability and reliability of the training task. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 4 is an exemplary flowchart of a method for rescheduling training tasks according to an embodiment of the present invention.

[0020] Figure 2 FIG. 4 is a schematic diagram of migrating a virtual processing unit within a node according to an embodiment of the present invention.

[0021] Figure 3 FIG. 4 is a schematic diagram of virtually migrating processing units between nodes according to an embodiment of the present invention.

[0022] Figure 4 FIG. 4 is a schematic diagram of virtually migrating processing units in a cluster based on a leaf-spine structure according to an embodiment of the present invention.

[0023] Figure 5 Schematic diagram of an exemplary process of rescheduling training tasks according to an embodiment of the present invention.

[0024] Figure 6 is an exemplary structural diagram of a training task rescheduling system according to an embodiment of the present invention.

[0025] Figure 7 is an exemplary structural diagram of a training task rescheduling device according to an embodiment of the present invention.

[0026] Figure 8 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0028] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0029] The parallel strategy for large model training has evolved from early data parallelism to the now widely used 3D parallelism. 3D parallelism comprehensively utilizes three technologies: data parallelism, model parallelism, and pipeline parallelism to form a multi-dimensional parallel training framework. Each parallel strategy targets a specific bottleneck problem, and their combination can improve training efficiency and resource utilization. In data parallelism, gradient synchronization is mainly performed, and the communication volume is proportional to the model size. In model parallelism, communication is mainly achieved through tensor splitting, and the communication volume is large. In pipeline parallelism, activation value transfer is mainly performed, and the communication volume is relatively small. The scheduler in the cluster used to train large models can adopt differentiated bandwidth allocation strategies for different parallel dimensions based on these characteristics.

[0030] In the existing large-model 3D parallel training, as the training work continues, many dynamic changes may occur in the cluster (such as load changes and cluster resource changes, etc.), resulting in a decrease in the communication efficiency of the current topology that executes the training task, thereby affecting the training efficiency. Moreover, in the existing large-model 3D parallel training technology, there are limitations in fault tolerance. For example, taking the Spine-Leaf switch network architecture as an example, the Spine layer switch may become a single point of failure. Once a failure occurs, the entire cluster directly reports an error, resulting in challenges to the stability and reliability of training.

[0031] In the implementation mode of the present invention, a rescheduling strategy based on network topology awareness is used to overcome or alleviate the above problems. The operation status of the cluster is continuously monitored. When a fault is detected, it does not simply report an error directly, but expels the abnormal operation unit, and creates a new operation unit to replace the expelled operation unit. This can avoid the impact of single point failures on training and improve the stability and reliability of training tasks. Moreover, when the cluster utilization rate is low, the topological allocation status of the training task can be detected, and the training task can be rescheduled to a new topology with lower communication cost, so that resource allocation is more reasonable, can adapt to many changes in the training process, improve communication efficiency and training efficiency, and optimize resource utilization.

[0032] The above disclosure details the technical defects existing in the related art, the causes of the technical defects, and the thinking and analysis process of overcoming the technical defects. In fact, the cognition of the above technical defects is not common knowledge in the field, but a novel discovery made by the inventor in his research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the inventor in the actual research process, and are not common knowledge in the field.

[0033] Figure 1 FIG. 1 is an exemplary flow chart of a method for rescheduling a training task according to an embodiment of the present invention. The method may be executed by a controller in a cluster that executes a large model training task. Figure 1 As shown, the method includes: Step 101: When it is determined that a rescheduling opportunity has arrived, determine a first indicator of a current topology to which a training task has been scheduled, the current topology includes a plurality of processing units, and the first indicator represents a communication cost of the current topology.

[0034] In one embodiment, determining that a rescheduling opportunity has been reached includes at least one of the following: (1) Determine whether the utilization rate of the cluster used to execute the training task is lower than a predetermined threshold or whether the overall topology of the cluster has changed.

[0035] For example, cluster utilization can be measured based on cluster GPU computing power utilization, cluster GPU memory utilization, cluster throughput, and other indicators. In addition, changes in the overall cluster topology may include: increases or decreases in the number of nodes in the cluster, changes in node hardware performance, or changes in the switch network structure used to connect nodes, etc.

[0036] (2) Determine whether a rescheduling instruction triggered by a user has been received.

[0037] For example, a rescheduling instruction is received from a user through a human-computer interaction interface.

[0038] (3) Determine the scheduled rescheduling time point.

[0039] For example, a periodic rescheduling time point is set. When the periodic time point is reached each time, the rescheduling time is automatically determined.

[0040] It can be seen that by executing during the low-peak period or periodic time of the cluster Figure 1 The process can reduce the complexity and overhead of rescheduling and achieve an efficient and stable training process.

[0041] The scheduler in the cluster can schedule training tasks to the processing units in the cluster (the processing units are usually located in the nodes that serve as physical servers) based on various predetermined scheduling principles, thereby forming the current topology. For example, scheduling principles may include: resource matching principle, affinity and anti-affinity principle, priority and fairness principle, or load balancing and communication optimization principle, etc.

[0042] The current topology includes specific processing units in the cluster for executing the training task and the specific deployment locations of these specific processing units in the cluster (for example, in which specific node the processing unit is located). These specific processing units in the current topology execute the training task.

[0043] The nodes in the cluster act as physical servers, and through their powerful computing resources, they support the training tasks of large models and perform complex mathematical operations and data processing. In addition, the cluster implements communication between nodes via various types of switch network architectures. For example, the switch network architecture can be implemented as a fat tree structure, a leaf-spine structure, a multi-plane structure, a SuperPOD structure, a multi-track structure, and so on.

[0044] Processing units are usually located in nodes that are physical servers. A processing unit (e.g., a Pod in Kubernetes) is the smallest deployable and schedulable unit in a cluster and can be implemented as a software module. Specifically, a processing unit can contain one or more closely related containers that share network and storage resources. In large model training, the main functions of processing units include: (1) Encapsulating training tasks: A processing unit can encapsulate one or more containers for encapsulating and running training tasks. For example, a processing unit may contain a container for performing model training and another container for data loading or logging. (2) Resource sharing: Containers in a processing unit share the same network namespace and storage volume, which allows containers to communicate and share data easily. (3) Scheduling and management: The scheduler allocates running units to appropriate nodes to run based on resource requirements and cluster status. (4) Lifecycle management: A processing unit has its own lifecycle and can be created, started, stopped, and deleted.

[0045] The first indicator of the current topology to which the training task has been scheduled is used to characterize the communication cost of the current topology, where the communication cost of the current topology is the cumulative cost of communication between all processing units in the current topology. Similarly, the second indicator is used to characterize the communication cost of the rescheduled topology, where the communication cost of the rescheduled topology is the cumulative cost of communication between all processing units in the rescheduled topology.

[0046] In one embodiment, the individual communication cost between each processing unit in the current topology and the remaining processing units in the current topology is determined; based on a predetermined accumulation method and the individual communication cost between each processing unit in the current topology and the remaining processing units in the current topology, a first indicator is determined. Similarly, the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology is determined; based on a predetermined accumulation method and the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology, a second indicator is determined.

[0047] The individual communication costs of all processing units in the current topology and the rescheduling topology can be accumulated based on a plurality of optional types of accumulation methods to obtain the first index and the second index respectively. For example, the accumulation methods may include: a simple summation method; a weighted summation method; a linear summation method; an accumulation operation based on an exponential function; an accumulation operation based on a power function; an accumulation operation based on a logarithmic function; an accumulation operation based on a composite function, and the like.

[0048] Considering the advantage of simple summation being easy to implement, it is preferred to use a simple summation method to determine the first indicator and the second indicator. In one embodiment, determining the first indicator of the current topology to which the training task has been scheduled includes: determining the individual communication cost of each processing unit in the current topology with the remaining processing units in the current topology; calculating the first summation result of the individual communication costs of all processing units in the current topology; and determining the first indicator based on the first summation result.

[0049] In an embodiment of the present invention, the individual communication cost can be measured based on a variety of technical parameters. For example, the technical parameters may include: communication time, communication bandwidth occupancy, network delay, network jitter and network resource occupancy, etc. The individual communication cost between nodes can be determined based on a single or multiple technical parameters. In a specific implementation, the first indicator or the second indicator can be calculated through a matrix, and the specific process includes: determining a communication matrix based on a topological structure, wherein the communication matrix lists the communication relationship between all nodes in the topology, including the amount of data and the direction of communication; calculating the individual communication cost between each pair of nodes according to technical parameters such as communication time and bandwidth occupancy; accumulating the individual communication costs of all pairs of nodes to obtain the total communication cost of the topology, which is the first indicator or the second indicator.

[0050] For example, assume that the current topology to which the training task has been scheduled includes processing unit 1 to processing unit 4.

[0051] First, calculate: a1: the communication cost between processing unit 1 and processing unit 2, processing unit 3 and processing unit 4; a2: communication cost between processing unit 2 and processing unit 1, processing unit 3 and processing unit 4; a3: communication cost between processing unit 3 and processing unit 1, processing unit 2 and processing unit 4; a4: Communication cost between processing unit 4 and processing unit 1, processing unit 2 and processing unit 3.

[0052] Then, (a1+ a2+ a3+ a4) is determined as the first index.

[0053] Step 102: Determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated.

[0054] Here, at least one processing unit in the current topology may be virtually migrated to form a rescheduled topology. Wherein: any processing unit in the current topology may be virtually migrated to any node with an idle position. Preferably, the processing unit is virtually migrated to the same node as the rest of the processing units in the current topology in a best-effort manner, thereby minimizing the communication cost of the topology.

[0055] In one embodiment, step 102 includes: (1) Virtually migrate a processing unit in a first node from a current position in the first node to another position in the first node that is different from the current position.

[0056] Here, the processing unit nodes in the current topology are virtually moved to determine the rescheduling topology formed after the virtual migration.

[0057] Figure 2 FIG. 4 is a schematic diagram of migrating a virtual processing unit within a node according to an embodiment of the present invention.

[0058] exist Figure 2 In the example, the current topology includes processing units 0 to 3, where processing units 0 to 3 are all in the same node 0. Processing unit 3 can be virtually migrated to another processing unit 4 in node 0, thereby forming a rescheduling topology: processing unit 0, processing unit 1, processing unit 2, and processing unit 4.

[0059] (2) Virtually migrate the processing unit in the first node from its current location in the first node to the second node.

[0060] Here, the processing unit in the current topology is virtually moved between different nodes of the processing unit to determine a rescheduling topology formed after the virtual migration.

[0061] For example, the virtual movement of processing units between different nodes may include: (a) Virtually migrating a processing unit in a first node from a current location in the first node to a second node connected to the same switch node as the first node.

[0062] (b) Virtually migrating the processing unit in the first node from a current location in the first node to a second node connected to a switch node different from the first node.

[0063] Figure 3 FIG. 4 is a schematic diagram of virtually migrating processing units between nodes according to an embodiment of the present invention.

[0064] exist Figure 3 In the figure, the current topology includes processing units 0 to 3, where processing units 0 to 3 are all in the same node 0. Processing unit 3 can be virtually migrated to processing unit 4 in node 1, thereby forming a rescheduling topology: processing unit 0 in node 0, processing unit 1 in node 0, processing unit 2 in node 0, and processing unit 4 in node 1. Among them: node 0 and node 1 can be connected to a common switch node, or node 0 and node 1 are connected to different switch nodes respectively.

[0065] Step 103: Determine a second index of the rescheduling topology, where the second index represents the communication cost of the rescheduling topology.

[0066] In one embodiment, determining a second indicator of the rescheduling topology includes: determining the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology; calculating a second summation result of the individual communication costs of all processing units in the rescheduling topology; and determining a second indicator based on the second summation result.

[0067] For example: Figure 3 As shown, it is assumed that the rescheduling topology includes: processing unit 0 in node 0, processing unit 1 in node 0, processing unit 2 in node 0, and processing unit 4 in node 1.

[0068] First, calculate: b1: the communication cost between processing unit 0 in node 0 and processing unit 1 in node 0, processing unit 2 in node 0 and processing unit 4 in node 1; b2: the communication cost between processing unit 1 in node 0 and processing unit 0 in node 0, processing unit 2 in node 0 and processing unit 4 in node 1; b3: the communication cost between processing unit 2 in node 0 and processing unit 0 in node 0, processing unit 1 in node 0 and processing unit 4 in node 1; b4: Communication cost between processing unit 4 in node 1 and processing unit 0 in node 0, processing unit 1 in node 0 and processing unit 2 in node 0.

[0069] Then, (b1+ b2+ b3+ b4) is determined as the second index.

[0070] Step 104: When the second indicator is less than the first indicator, rescheduling the training task into a rescheduling topology.

[0071] Here, when the second indicator is less than the first indicator, the training task is rescheduled to the rescheduled topology, so that the rescheduled topology continues to execute the training task. Therefore, rescheduling the training task to a rescheduled topology with lower communication cost can improve communication efficiency and training efficiency. When an abnormality is detected, rebuilding the processing unit instead of simply reporting an error can improve stability and reliability.

[0072] In step 102, multiple rescheduling topologies may be formed based on different virtual migration locations of different processing units. At this time, the second index of each rescheduling topology can be calculated to form a second index set containing the second indexes of all rescheduling topologies. Then, the smallest second index is selected from the second index set. In step 104, when the smallest second index is less than the first index, the training task is rescheduled to the rescheduling topology corresponding to the smallest second index.

[0073] In one embodiment, step 104 includes: expelling at least one processing unit (that is, the at least one processing unit that is virtually migrated in step 102 to form a rescheduling topology); and rebuilding at least one processing unit at a location in the rescheduling topology to which the at least one processing unit is virtually migrated. Therefore, by rebuilding the processing unit based on the rescheduling topology, the training task can continue to be processed.

[0074] Figure 4 The diagram is a diagram of a virtual migration processing unit in a cluster based on a leaf-spine structure according to an embodiment of the present invention. In a leaf-spine structure, a leaf switch group generally refers to a group of switches located at the bottom of the network topology, directly connecting nodes and providing network access functions. The leaf switch group is connected to the spine switch through an uplink. The leaf switch group may include at least one leaf switch. Figure 4 In the example, it is assumed that a single processing unit runs in a single node. Leaf switch group 1 is connected to nodes 1 to 8. Leaf switch group 4 is connected to nodes 9 to 16. Leaf switch groups 1 to 4 are connected to spine switches respectively.

[0075] For the training task, the current topology scheduled by the scheduling unit includes: processing unit 1 in node 1, processing unit 2 in node 2, and processing unit 9 in node 9.

[0076] Calculate: d1: Communication cost between processing unit 1 and processing unit 2: (S1+S2); d2: Communication cost between processing unit 1 and processing unit 9: (S1+C1+C2+S9).

[0077] d3: Communication cost between processing unit 2 and processing unit 1: (S1+S2); d4: Communication cost between processing unit 2 and processing unit 9: (S2+C1+C2+S9).

[0078] Therefore, the communication cost of the current topology is: d1+d2+d3+d4. Among them: C1 is the communication cost between leaf switch group 1 and spine switch; C2 is the communication cost between leaf switch group 4 and spine switch; S2 is the communication cost between processing unit 2 and leaf switch group 1; S9 is the communication cost between processing unit 9 and leaf switch group 4.

[0079] During the rescheduling process, processing unit 9 is virtually moved to the same node as the rest of the processing units in the current topology (for example, the position of processing unit 8 in node 1), thereby obtaining a rescheduling topology: processing unit 1 in node 1, processing unit 2 in node 2, and processing unit 8 in node 1.

[0080] Calculate: e1: Communication cost between processing unit 1 and processing unit 2: (S1+S2); e2: Communication cost between processing unit 1 and processing unit 8: (S1+S8).

[0081] e3: Communication cost between processing unit 2 and processing unit 1: (S1+S2); e4: Communication cost between processing unit 2 and processing unit 8: (S1+S8).

[0082] Wherein: S8 is the communication cost between processing unit 8 and leaf switch group 1.

[0083] Therefore, the communication cost of rescheduling the topology includes: e1+e2+e3+e4.

[0084] When e1+e2+e3+e4 is greater than or equal to d1+d2+d3+d4, the training task is not rescheduled to be executed by the rescheduling topology, that is, the training task is still executed by processing unit 1 in node 1, processing unit 2 in node 2, and processing unit 9 in node 9. When e1+e2+e3+e4 is less than d1+d2+d3+d4, the training task can be rescheduled to be executed by the rescheduling topology. At this time, processing unit 9 in node 9 is first expelled, and the expelled processing unit is rebuilt at the position of processing unit 8 in node 1, so that the training task is executed by processing unit 1 in node 1, processing unit 2 in node 2, and processing unit 8 in node 1.

[0085] In the above Figure 1 In , the rescheduling method of training tasks is described. Considering that large model training usually includes multiple parallel training tasks (for example, data parallel tasks, model parallel tasks, and pipeline parallel tasks), each training task can be executed separately. Figure 1 The process shown is to complete the large model training.

[0086] In one embodiment, Figure 1 The method further includes: when a processing unit in an abnormal state is detected in the current topology, the processing unit in the abnormal state is expelled; and when the processing unit in the abnormal state is located at a current position of the current topology, the processing unit is rebuilt.

[0087] Therefore, when an abnormal processing unit is detected, it does not simply report an error, but instead expels the abnormal processing unit and rebuilds the processing unit at the current position in the current topology, thereby improving the stability and reliability of the training task.

[0088] In one embodiment, the method includes: when a processing unit in an abnormal state is detected in a current topology, determining an abnormal processing topology formed after the processing unit in the abnormal state is virtually migrated; determining a third indicator of the abnormal processing topology, the third indicator representing the communication cost of the abnormal processing topology; when the third indicator is less than the first indicator, expelling the processing unit in the abnormal state, and rebuilding the processing unit at the position in the abnormal processing topology to which the processing unit in the abnormal state is virtually migrated; when the third indicator is greater than or equal to the first indicator, expelling the processing unit in the abnormal state, and rebuilding the processing unit at the current position of the processing unit in the abnormal state in the current topology.

[0089] Therefore, when an abnormality is detected, it does not simply report an error, but expels the processing unit in the abnormal state, and calculates the third index of the abnormal processing topology based on the abnormal processing topology formed after the processing unit in the abnormal state is virtually migrated. Then, the reconstruction position of the processing unit is determined based on the third index, so as to further consider the communication cost factor in the reconstruction process and improve the communication efficiency and training efficiency.

[0090] Figure 5 FIG. 1 is a schematic diagram showing an exemplary process of rescheduling a training task according to an embodiment of the present invention. Figure 5 As shown, the method includes: Step 201: Detect the status of a cluster, where the cluster trains a large model in parallel with multiple training tasks.

[0091] Step 202: Determine whether there is an abnormal processing unit in the cluster. If yes, execute step 209 and its subsequent steps; otherwise, execute step 203 and its subsequent steps. For example, it is possible to determine whether a processing unit in a node is abnormal by determining whether there is an abnormality in a node in the cluster.

[0092] Step 203: Determine whether the rescheduling time has arrived. If so, execute step 204 and subsequent steps; otherwise, return to execute step 201.

[0093] Step 204: Traverse all training tasks.

[0094] Step 205: For the currently traversed training task, calculate the communication cost a1 of the current topology of the training task.

[0095] Step 206: For the currently traversed training task, calculate the communication cost a2 of the rescheduling topology of the training task. In the process of forming the rescheduling topology, the processing unit is virtually migrated to the same node as the rest of the processing units in the current topology in a best-effort manner.

[0096] Step 207: Determine whether a2 is less than a1. If so, execute step 208; otherwise, return to execute step 204 and subsequent steps.

[0097] Step 208: evict and reconstruct the processing unit so that the training task is executed by the rescheduled topology, and end this process.

[0098] Step 209: Expel the abnormal processing unit.

[0099] Step 210: Rebuild the abnormal processing unit.

[0100] Figure 6 FIG. 1 is an exemplary structural diagram of a rescheduling system for training tasks according to an embodiment of the present invention. Figure 6 As shown, the rescheduling system includes: a cluster, including multiple nodes; a scheduler, used to schedule training tasks to the cluster to form a current topology, the current topology includes multiple processing units in respective nodes; a rescheduler, used to determine a first indicator of the current topology when it is determined that the rescheduling time has arrived, the first indicator representing the communication cost of the current topology; determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; determine a second indicator of the rescheduling topology, the second indicator representing the communication cost of the rescheduling topology; when the second indicator is less than the first indicator, reschedule the training task to the rescheduling topology.

[0101] The nodes in the cluster act as physical servers, usually containing high-performance CPUs and GPUs for performing computationally intensive tasks. Each node can run one or more operating units. Each operating unit can contain a separate container for running training tasks, or multiple containers for different functions such as training, data loading, and logging. Based on the switch network architecture, the nodes in the cluster can communicate with each other.

[0102] The scheduler schedules training tasks to the cluster based on various predetermined scheduling principles to form the current topology. For example, the scheduling principles may include: (1) Resource matching principle: The scheduler first checks the resource status of each node in the cluster, including the availability of CPU, memory, GPU, etc., to ensure that the selected node can meet the resource requirements of the training task. For example, for deep learning training tasks that require a large amount of GPU resources, the scheduler will give priority to nodes with sufficient GPU resources. Moreover, under the premise of meeting the task resource requirements, the scheduler will try to select nodes with lower resource utilization to achieve balanced utilization of cluster resources. This helps to avoid the situation where some node resources are overcrowded while other node resources are idle, thereby improving the overall resource utilization efficiency. (2) Affinity and anti-affinity principle: Through node affinity rules, training tasks can be scheduled to nodes with specific labels. For example, tasks that require a specific operating system or software environment can be scheduled to nodes with corresponding labels to ensure that the tasks can run smoothly. (3) Priority and fairness principle: For different training tasks, different priorities can be set according to their importance and urgency. The scheduler will give priority to high-priority tasks to ensure that critical tasks can be executed in time. For example, for model training tasks that need to be completed quickly, a higher priority can be set. The scheduler will try to ensure that all tasks have a fair chance to be scheduled. Even if some tasks cannot be scheduled temporarily due to insufficient resources, they will be included in the scheduling queue and will be scheduled when resources are sufficient. (4): Load balancing and communication optimization principle: The scheduler will try to evenly distribute training tasks to each node in the cluster to avoid some nodes being overloaded and other nodes being underloaded. In a distributed training scenario, the scheduler will consider the communication overhead between nodes and try to schedule running units that need to communicate frequently to the same node or adjacent nodes. This can reduce network transmission delays and improve training efficiency.

[0103] In one embodiment, the rescheduler is used to evict the processing unit in the abnormal state when the processing unit in the abnormal state is detected in the current topology; and to reconstruct the processing unit at the current position of the processing unit in the abnormal state in the current topology.

[0104] In one embodiment, a rescheduler is used to, when a processing unit in an abnormal state is detected in a current topology, determine an exception processing topology formed after the processing unit in the abnormal state is virtually migrated; determine a third indicator of the exception processing topology, the third indicator representing the communication cost of the exception processing topology; when the third indicator is less than the first indicator, evict the processing unit in the abnormal state, and rebuild the processing unit at the location in the exception processing topology to which the processing unit in the abnormal state is virtually migrated; when the third indicator is greater than or equal to the first indicator, evict the processing unit in the abnormal state, and rebuild the processing unit at the current location of the processing unit in the abnormal state in the current topology.

[0105] When the switch architecture in the cluster is implemented as a Fat Tree structure or a Spine-Leaf structure, the rescheduler and the scheduler can be deployed on the leaf nodes.

[0106] In summary, when it is determined that the time for rescheduling has arrived, the first indicator of the current topology to which the training task has been scheduled is determined, the current topology includes multiple processing units, and the first indicator represents the communication cost of the current topology; the rescheduling topology formed after at least one processing unit in the current topology is virtually migrated is determined; the second indicator of the rescheduling topology is determined, and the second indicator represents the communication cost of the rescheduling topology; when the second indicator is less than the first indicator, the training task is rescheduled to the rescheduling topology. It can be seen that by rescheduling the training task to be executed by a topology with lower communication cost, many dynamic changes in the cluster (such as load changes and cluster resource changes, etc.) can be adapted, thereby improving communication efficiency and training efficiency. In addition, when an anomaly is detected, the processing unit is rebuilt based on a variety of mechanisms instead of simply reporting an error, thereby improving the stability and reliability of the training task.

[0107] Figure 7 FIG. 1 is an exemplary structural diagram of a rescheduling device for training tasks according to an embodiment of the present invention. Figure 7 As shown, the rescheduling device 700 for training tasks includes: a first determination module 701, which is used to determine the first indicator of the current topology to which the training task has been scheduled when it is determined that the rescheduling time has arrived, the current topology includes multiple processing units, and the first indicator represents the communication cost of the current topology; a second determination module 702, which is used to determine the rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; a third determination module 703, which is used to determine the second indicator of the rescheduling topology, and the second indicator represents the communication cost of the rescheduling topology; a rescheduling module 704, which is used to reschedule the training task to the rescheduling topology when the second indicator is less than the first indicator.

[0108] In one embodiment, determining that the time for rescheduling has been reached includes at least one of the following: determining that the utilization rate of the cluster used to perform the training task is lower than a predetermined threshold or that the overall topology of the cluster has changed; determining that a user-triggered rescheduling instruction has been received; and determining that a predetermined rescheduling time point has been reached.

[0109] In one embodiment, the second determination module 702 is used to virtually migrate the processing unit in the first node from the current position in the first node to another position in the first node different from the current position; or to virtually migrate the processing unit in the first node from the current position in the first node to the second node.

[0110] In one embodiment, the second determination module 702 is used to virtually migrate the processing unit in the first node from the current position in the first node to a second node connected to the same switch node as the first node; or to virtually migrate the processing unit in the first node from the current position in the first node to a second node connected to a different switch node than the first node.

[0111] In one embodiment, determining a first indicator of a current topology to which a training task has been scheduled includes: determining the individual communication cost between each processing unit in the current topology and the remaining processing units in the current topology; calculating a first summation result of the individual communication costs of all processing units in the current topology; determining the first indicator based on the first summation result; determining a second indicator of a rescheduling topology includes: determining the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology; calculating a second summation result of the individual communication costs of all processing units in the rescheduling topology; and determining a second indicator based on the second summation result.

[0112] In one embodiment, the rescheduling module 704 is configured to evict at least one processing unit; and recreate at least one processing unit at a location in the rescheduling topology to which the at least one processing unit is virtually migrated.

[0113] In one embodiment, the rescheduling module 704 is used to evict the processing unit in an abnormal state when a processing unit in an abnormal state is detected in the current topology; and to reconstruct the processing unit at the current position of the processing unit in the abnormal state in the current topology.

[0114] In one embodiment, the rescheduling module 704 is used to determine, when a processing unit in an abnormal state is detected in the current topology, an exception processing topology formed after the processing unit in the abnormal state is virtually migrated; determine a third indicator of the exception processing topology, the third indicator representing the communication cost of the exception processing topology; when the third indicator is less than the first indicator, evict the processing unit in the abnormal state, and rebuild the processing unit at the position in the exception processing topology to which the processing unit in the abnormal state is virtually migrated; when the third indicator is greater than or equal to the first indicator, evict the processing unit in the abnormal state, and rebuild the processing unit at the current position of the processing unit in the abnormal state in the current topology.

[0115] The embodiment of the present invention further provides an electronic device having a processor-memory architecture. Figure 8 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 8 As shown, the electronic device includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the computer program is executed by the processor 801, any of the above training task rescheduling methods is implemented. Among them, the memory 802 can be specifically implemented as a variety of storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor 801 can be implemented to include one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be implemented as a CPU, a GPU, a GPGPU, an MCU or a DSP, and so on.

[0116] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0117] The hardware modules in each embodiment may be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) to perform a specific operation. For example, specific operations can be performed in various types of chips (for example, artificial intelligence chips). The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform specific operations. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0118] The present invention also provides a machine-readable storage medium, storing instructions for a machine to perform a method as described in the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the embodiments described in the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system operated on the computer can also be made to complete part or all of the actual operations by instructions based on the program code. The program code read out from the storage medium can also be written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then the CPU installed on the expansion board or the expansion unit is made to perform part and all of the actual operations based on the instructions of the program code, thereby realizing the functions of any of the embodiments described in the above embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0119] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for rescheduling a training task, characterized in that: include: When it is determined that the rescheduling opportunity has arrived, determining a first indicator of a current topology to which the training task has been scheduled, the current topology comprising a plurality of processing units, the first indicator representing a communication cost of the current topology; Determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; Determining a second indicator of the rescheduling topology, the second indicator representing a communication cost of the rescheduling topology; When the second indicator is less than the first indicator, the training task is rescheduled to the rescheduling topology.

2. The method according to claim 1, characterized in that The determining that the rescheduling opportunity has arrived includes at least one of the following: Determining that a usage rate of a cluster used to execute the training task is lower than a predetermined threshold or an overall topology structure of the cluster changes; Determining that a rescheduling instruction triggered by a user is received; Determine that a scheduled rescheduling time point has been reached.

3. The method according to claim 1, characterized in that The determining of a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated comprises: Virtually migrate a processing unit in a first node from a current position in the first node to another position in the first node that is different from the current position; or A processing unit in a first node is virtually migrated from a current position in the first node to a second node.

4. The method according to claim 3, characterized in that The virtually migrating the processing unit in the first node from the current position in the first node to the second node comprises: Virtually migrate a processing unit in a first node from a current location in the first node to a second node connected to the same switch node as the first node; or A processing unit in a first node is virtually migrated from a current position in the first node to a second node connected to a switch node different from the first node.

5. The method according to claim 1, characterized in that Determining the first indicator of the current topology to which the training task has been scheduled includes: determining the individual communication cost between each processing unit in the current topology and the remaining processing units in the current topology; calculating a first summation result of the individual communication costs of all processing units in the current topology; and determining the first indicator based on the first summation result; Determining the second indicator of the rescheduling topology includes: determining the individual communication cost between each processing unit in the rescheduling topology and the remaining processing units in the rescheduling topology; calculating the second summation result of the individual communication costs of all processing units in the rescheduling topology; and determining the second indicator based on the second summation result.

6. The method according to any one of claims 1 to 5, characterized in that The rescheduling of the training task into the rescheduling topology comprises: evicting the at least one processing unit; Rebuilding the at least one processing unit at a location in the rescheduling topology to which the at least one processing unit is virtually migrated.

7. The method according to any one of claims 1 to 5, characterized in that include: When a processing unit in an abnormal state is detected in the current topology, the processing unit in the abnormal state is expelled; When the processing unit in the abnormal state is located at a current position of the current topology, the processing unit is rebuilt.

8. The method according to any one of claims 1 to 5, characterized in that include: When a processing unit in an abnormal state is detected in the current topology, determining an abnormal processing topology formed after the processing unit in the abnormal state is virtually migrated; Determining a third indicator of the exception handling topology, wherein the third indicator represents a communication cost of the exception handling topology; When the third indicator is less than the first indicator, the processing unit in the abnormal state is expelled, and the processing unit is rebuilt at the position in the abnormal processing topology to which the processing unit in the abnormal state is virtually migrated; When the third index is greater than or equal to the first index, the processing unit in the abnormal state is expelled, and the processing unit in the abnormal state is rebuilt at the current position of the current topology.

9. A training task rescheduling device, characterized in that: include: A first determination module is used to determine a first indicator of a current topology to which the training task has been scheduled when it is determined that a rescheduling opportunity has been reached, the current topology including a plurality of processing units, and the first indicator represents a communication cost of the current topology; A second determining module, used to determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; A third determination module is used to determine a second indicator of the rescheduling topology, where the second indicator represents a communication cost of the rescheduling topology; A rescheduling module is used to reschedule the training task to the rescheduling topology when the second indicator is less than the first indicator.

10. A training task rescheduling system, characterized in that: include: A cluster, which consists of multiple nodes; a scheduler for scheduling training tasks to the cluster to form a current topology, the current topology comprising a plurality of processing units in respective nodes; A rescheduler, configured to determine a first indicator of the current topology when it is determined that a rescheduling opportunity has arrived, wherein the first indicator represents a communication cost of the current topology; Determine a rescheduling topology formed after at least one processing unit in the current topology is virtually migrated; Determine a second indicator of the rescheduling topology, where the second indicator represents a communication cost of the rescheduling topology; when the second indicator is less than the first indicator, reschedule the training task to the rescheduling topology.

11. An electronic device, characterized in that: include: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to execute the rescheduling method for training tasks as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, cause the processor to execute the method for rescheduling training tasks as described in any one of claims 1 to 8.

13. A program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for rescheduling training tasks as described in any one of claims 1 to 8.

Images