Task scheduling method and device, equipment and medium

Abstract

The invention provides a task scheduling method and device, equipment and a medium, and the method comprises the steps: building an inter-task communication matrix based on the task communication traffic between every two sub-tasks in a plurality of to-be-scheduled sub-tasks; constructing a network affinity matrix based on the bandwidth and the congestion index between every two task processing nodes in the plurality of task processing nodes to be scheduled; performing clustering processing on the plurality of subtasks based on the inter-task communication matrix to obtain a task cluster set, and performing clustering processing on the plurality of task processing nodes based on the network affinity matrix to obtain a node cluster set; and according to the resource demand information of each sub-task and the resource supply information of each task processing node, mapping the task cluster set and the node cluster set, and determining a mapping result between the plurality of sub-tasks and the plurality of task processing nodes. Therefore, the task cluster with dense communication can be mapped to the node cluster with high network affinity, the communication overhead is reduced, and the end-to-end time delay of the task is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a task scheduling method, apparatus, device, and medium. Background Technology

[0002] With the proliferation of large-scale deep learning tasks (such as models with billions to hundreds of billions of parameters), a single machine often cannot meet the load requirements. Distributed parallelism has become a new option, which means that deep learning tasks are distributed across different machines to work together. Deep learning tasks are divided into multiple parallel methods and distributed training is carried out in a multi-machine environment.

[0003] When scheduling tasks, random allocation or matching based on computing resources is often used, resulting in poor task execution, affecting network affinity, causing communication bottlenecks, increased latency, and decreased resource utilization. Summary of the Invention

[0004] In view of this, this application provides a task scheduling method, apparatus, device and medium, which combines the task communication volume between subtasks and the bandwidth and congestion indicators between task processing nodes to perform a reasonable mapping between subtasks and task processing nodes, thereby improving the network affinity of task scheduling.

[0005] Specifically, this application is implemented through the following technical solution: According to a first aspect of this application, a task scheduling method is provided, the method comprising: Based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, an inter-task communication matrix is ​​constructed, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled. Based on the bandwidth and congestion metrics between every two task processing nodes in a plurality of task processing nodes to be scheduled, a network affinity matrix is ​​constructed. Based on the inter-task communication matrix, the multiple subtasks are clustered to obtain a set of task clusters, and based on the network affinity matrix, the multiple task processing nodes are clustered to obtain a set of node clusters. Each task cluster in the set of task clusters includes at least one subtask, and each node cluster in the set of node clusters includes at least one task processing node. Based on the resource requirement information of each subtask and the resource supply information of each task processing node, the task cluster set and the node cluster set are mapped to determine the mapping result between the multiple subtasks and the multiple task processing nodes.

[0006] In one optional implementation, the plurality of subtasks are obtained through the following steps: Based on the task structure of the deep learning distributed task to be scheduled and the predetermined parallel strategy, the deep learning distributed task is divided into multiple subtasks, and the parallel strategy is used to indicate the task division method.

[0007] In one optional implementation, the parallel strategy includes at least one of data parallelism, pipeline parallelism, and tensor parallelism; the task communication volume between the two subtasks is determined through the following steps: The execution position of each subtask under each strategy is determined. The execution position of the subtask under the data parallel strategy is used to indicate the data parallel group to which the subtask belongs. The execution position of the subtask under the pipeline parallel strategy is used to indicate the execution stage to which the subtask belongs. The execution position of the subtask under the tensor parallel strategy is used to indicate the execution link to which the subtask belongs. Based on the task execution positions of the two subtasks, the task communication volume between the two subtasks is determined.

[0008] In one alternative implementation, the bandwidth and congestion metrics between the two task processing nodes are determined through the following steps: Determine the connection relationship between two task processing nodes, including direct connection, indirect connection, and no connection; Based on the connection relationship between the two task processing nodes, determine the bandwidth and congestion indicators between the two task processing nodes.

[0009] In one optional implementation, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: When the connection relationship is a direct connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate between the two task processing nodes are normalized. The normalized original bandwidth is determined as the bandwidth between the two task processing nodes; Based on the normalized retransmission rate, the normalized network interface queue length, and the normalized packet loss rate, the congestion index between the two task processing nodes is determined.

[0010] In one optional implementation, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: In the case where the connection relationship is an indirect connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate corresponding to the connection path between the two task processing nodes are normalized. The normalized original bandwidth corresponding to the shortest connection path between two task processing nodes is determined as the bandwidth between the two task processing nodes. Based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate of the connection path between the two task processing nodes, the congestion index between the two task processing nodes is determined.

[0011] In one optional implementation, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: If the connection is not established, the bandwidth and congestion metrics between the two task processing nodes are set to 0.

[0012] In one optional implementation, the step of clustering the multiple subtasks based on the inter-task communication matrix to obtain a task cluster set includes: Based on the inter-task communication matrix, spectral clustering is performed on the multiple subtasks to obtain a task spectral clustering matrix. Normalize each row of the task spectrum clustering matrix to obtain a normalized task spectrum clustering matrix; K-means clustering is performed on the normalized task spectrum clustering matrix to obtain multiple task clusters; Based on the target communication volume of each task cluster, determine the communication index of each task cluster; The multiple task clusters are sorted in descending order according to the communication index to generate the task cluster set.

[0013] In one optional implementation, the step of performing spectral clustering processing on the plurality of subtasks based on the inter-task communication matrix to obtain a task spectral clustering matrix includes: The inter-task communication matrix is ​​used as the weight matrix for spectral clustering, and each element in the inter-task communication matrix is ​​used to indicate the weight between every two graph nodes in the undirected connected graph. Based on the inter-task communication matrix, determine the degree matrix of the undirected connected graph; Based on the degree matrix and the inter-task communication matrix, a symmetric normalized Laplace matrix is ​​determined. Calculate multiple eigenvalues ​​of the symmetric normalized Laplace matrix, and select the target eigenvalues ​​with the smallest predetermined number from the multiple eigenvalues; The task spectrum clustering matrix is ​​generated based on the feature vectors corresponding to the target feature values.

[0014] In one optional implementation, the target communication volume includes intra-cluster communication volume; determining the communication index of each task cluster based on the target communication volume of each task cluster includes: For each task cluster, the communication index of the task cluster is determined based on the intra-cluster communication volume of the task cluster, the target communication volume of the task cluster, and the maximum target communication volume in each task cluster.

[0015] In one optional implementation, the step of clustering the plurality of task processing nodes based on the network affinity matrix to obtain a node cluster set includes: Based on the network affinity matrix, spectral clustering is performed on the multiple task processing nodes to obtain a node spectral clustering matrix. The nodes are normalized by normalizing each row of the node spectrum clustering matrix to obtain the normalized node spectrum clustering matrix. K-means clustering is performed on the normalized node spectrum clustering matrix to obtain multiple node clusters; The node clusters are sorted in descending order according to the number of task processing nodes included in the node clusters to generate the node cluster set.

[0016] In one optional implementation, the step of mapping the task cluster set and the node cluster set according to the resource requirement information of each subtask and the resource supply information of each task processing node, and determining the mapping result between the plurality of subtasks and the plurality of task processing nodes, includes: Based on the resource requirements of each subtask and the resource supply information of each task processing node, each task cluster and each node cluster is mapped sequentially according to the arrangement order of each task cluster in the task cluster set and the arrangement order of each node cluster in the node cluster set, until the allocation of the multiple subtasks or the allocation of the multiple task processing nodes is completed, thereby obtaining the mapping result between the multiple subtasks and the multiple task processing nodes.

[0017] In one optional implementation, the mapping of each task cluster and each node cluster includes: If the resource supply of the node cluster can meet the resource requirements of the task cluster, the node cluster and the task cluster are mapped. If the resource supply of the node cluster cannot meet the resource requirements of the task cluster, the task cluster is divided into multiple sub-task clusters according to the intra-cluster communication density. The sub-task clusters are then mapped sequentially according to their resource requirements in descending order.

[0018] According to a second aspect of this application, a task scheduling apparatus is provided, the apparatus comprising: The task matrix construction module is used to construct an inter-task communication matrix based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled. The network matrix construction module is used to construct a network affinity matrix based on the bandwidth and congestion indicators between every two task processing nodes in a plurality of task processing nodes to be scheduled. The clustering module is used to cluster the multiple subtasks based on the inter-task communication matrix to obtain a set of task clusters, and to cluster the multiple task processing nodes based on the network affinity matrix to obtain a set of node clusters. Each task cluster in the set of task clusters includes at least one subtask, and each node cluster in the set of node clusters includes at least one task processing node. The set mapping module is used to map the task cluster set and the node cluster set according to the resource requirement information of each subtask and the resource supply information of each task processing node, and determine the mapping result between the multiple subtasks and the multiple task processing nodes.

[0019] According to a third aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the task scheduling method described in the first aspect above.

[0020] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the task scheduling method described in the first aspect above.

[0021] The task scheduling method, apparatus, device, and medium provided in this application construct an inter-task communication matrix based on the task communication volume between subtasks, and construct a network affinity matrix based on the bandwidth and congestion indicators between task processing nodes. Based on the inter-task communication matrix and the network affinity matrix, clustering processing is performed to obtain a set of task clusters and a set of node clusters with high network affinity, thereby improving the network affinity of task scheduling. Mapping the set of task clusters and the set of node clusters can map communication-intensive task clusters to node clusters with high network affinity, which helps to reduce communication overhead, reduce end-to-end latency of tasks, improve resource utilization, and improve the efficiency of the entire deep learning task.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0023] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a task scheduling method in an exemplary embodiment of this application; Figure 2 This is a communication diagram between subtasks illustrated in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the bandwidth between task processing nodes according to an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating a congestion index between task processing nodes according to an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the distribution of eigenvalues ​​of a symmetric normalized Laplace matrix according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram illustrating a method for determining a task cluster, as shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating a task scheduling process according to an exemplary embodiment of this application; Figure 8 It is a schematic diagram of random mapping between multiple subtasks and multiple task processing nodes; Figure 9 This is a schematic diagram illustrating a mapping between multiple subtasks and multiple task processing nodes, as shown in an exemplary embodiment of this application. Figure 10 This is a schematic diagram of a task scheduling device shown in an exemplary embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0028] Research has revealed that task scheduling often employs random allocation, simple rules such as greedy allocation algorithms, or allocation solely based on computing resources, neglecting network bandwidth differences and transient congestion. This results in communication-intensive tasks traversing low-bandwidth or highly congested links, leading to poor task performance and causing communication bottlenecks, increased latency, and decreased resource utilization. Furthermore, the network topology in data centers or cloud platforms with multiple machines is often diverse. When congestion or uneven load exists, nodes or traffic hotspots with high network affinity naturally emerge, but the aforementioned scheduling methods cannot automatically identify and utilize this network affinity.

[0029] Based on the above research, this application provides a task scheduling method that can identify task cluster sets and node cluster sets with high network affinity, improve the network affinity of task scheduling, and map task cluster sets and node cluster sets to map communication-intensive task clusters to node clusters with high network affinity, which helps to reduce communication overhead, reduce end-to-end task latency, and improve resource utilization.

[0030] To facilitate understanding of this embodiment, a task scheduling method disclosed in this application will first be described in detail. The execution entity of the task scheduling method provided in this application is generally an electronic device with a certain computing power. This electronic device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, this task scheduling method can be implemented by the processor calling computer-readable instructions stored in memory.

[0031] The following description, in conjunction with the accompanying drawings, illustrates a task scheduling method provided by an embodiment of this application.

[0032] See Figure 1 The diagram shown is a flowchart illustrating a task scheduling method according to an exemplary embodiment of this application. Figure 1 As shown in the figure, the task scheduling method provided in this embodiment includes steps S101 to S104, wherein: S101: Based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, construct an inter-task communication matrix, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled.

[0033] Here, the deep learning distributed task to be scheduled can be, for example, model training, neuromorphic computing, etc. The multiple subtasks obtained by dividing the deep learning distributed task can be assigned to different task processing nodes for collaborative work.

[0034] In some possible implementations, the plurality of subtasks are obtained through the following steps: Based on the task structure of the deep learning distributed task to be scheduled and the predetermined parallel strategy, the deep learning distributed task is divided into multiple subtasks, and the parallel strategy is used to indicate the task division method.

[0035] In this step, based on the task structure of the deep learning distributed task, the overall task execution flow and the dependencies between each execution step can be clarified. Then, a pre-determined parallel strategy is invoked, and the deep learning distributed task is divided into multiple subtasks according to the partitioning method indicated by the parallel strategy. Optionally, the input data, output data, and resource requirement information of each subtask can also be labeled for subsequent mapping.

[0036] In this way, dividing tasks according to their structure and parallel strategies in deep learning distributed tasks can improve the effectiveness of subtask partitioning and provide a high-quality data foundation for subsequent task scheduling.

[0037] In some possible implementations, the parallel strategy includes at least one of data parallelism, pipeline parallelism, and tensor parallelism. Data parallelism (DP) often requires global gradient synchronization, such as Ring-Allreduce, and under this strategy, multiple data parallel groups can be obtained. Pipeline parallelism (PP) often requires inter-stage activation or gradient exchange during the forward propagation and / or backward propagation stages, and under this strategy, multiple task execution stages can be obtained. Tensor parallelism (TP) involves dense communication in high-speed intra-machine channels (such as NVLink), and under this strategy, multiple task execution stages can be obtained.

[0038] Optionally, a single-strategy approach can be used for task partitioning. Alternatively, a hybrid strategy can be employed, combining the advantages of data parallelism, pipelined parallelism, and tensor parallelism to maximize computational resource utilization. The hybrid strategy approach is more complex than the single-strategy approach because it requires consideration of the interactions between different strategy dimensions.

[0039] For example, if a three-dimensional hybrid strategy of data parallelism, pipeline parallelism, and tensor parallelism is adopted, the number of subtasks is represented by the following formula (1): (1) in, Indicates the number of subtasks. This indicates the number of subtasks divided under a data parallelism strategy. This indicates the number of subtasks divided under the pipeline parallelism strategy. This represents the number of subtasks divided under the tensor parallel strategy.

[0040] Here, for strategies that are not adopted, the number of subtasks can be 1.

[0041] For example, a data parallel strategy can be divided into 3 data parallel groups, a pipelined parallel strategy into 3 task execution stages, and a tensor parallel strategy into 2 task execution stages, resulting in a total of 18 subtasks: A1 (A11, A12), A2 (A21, A22), A3 (A31, A32), B1 (B11, B12), B2 (B21, B22), B3 (B31, B32), C1 (C11, C12), C2 (C21, C22), and C3 (C31, C32). This is understandable. The first data parallel group includes A1 (A11, A12), A2 (A21, A22), and A3 (A31, A32); the second data parallel group includes B1 (B11, B12), B2 (B21, B22), and B3 (B31, B32); and the third data parallel group includes C1 (C11, C12), C2 (C21, C22), and C3 (C31, C32). Each data parallel group has three task execution stages. Taking the first data parallel group as an example, it includes the three task execution stages A1, A2, and A3. Each task execution stage has two task execution steps. Taking task execution stage A1 as an example, it includes the two task execution steps A11 and A12.

[0042] Specifically, the task communication between two subtasks can be determined through the following steps: The execution position of each subtask under each strategy is determined. The execution position of the subtask under the data parallel strategy indicates the data parallel group to which the subtask belongs. The execution position of the subtask under the pipeline parallel strategy indicates the execution stage to which the subtask belongs. The execution position of the subtask under the tensor parallel strategy indicates the execution link to which the subtask belongs. Based on the execution positions of the two subtasks, the task communication volume between the two subtasks is determined.

[0043] In the above steps, for each of the two subtasks, the task execution position of each subtask under each task strategy can be determined. Based on the respective task execution positions of the two subtasks, the basic communication volume between the two subtasks is estimated, and the basic communication volume is determined as the task communication volume.

[0044] In this way, when determining the task communication volume, the task execution position of the subtask under each strategy is taken into consideration, so that the subtasks with high communication volume can be assigned to the same cluster during subsequent clustering processing.

[0045] In some possible implementations, the method further includes: If two subtasks belong to different data parallel groups and have the same task execution stage, determine the data parallel communication volume between the two subtasks. The task communication volume between the two subtasks is determined based on the basic task communication volume and the data parallel communication volume between the two subtasks.

[0046] Following the example above, subtasks A11 and B11 belong to different data parallel groups and have the same task execution stage. When determining the task communication volume between subtasks A11 and B11, data parallel communication volume can be added on the basis of the basic task communication volume.

[0047] In this way, for two subtasks that belong to different data parallel groups and have the same task execution stage, adding data parallel communication volume on top of the basic communication volume helps to improve the comprehensiveness and accuracy of determining task communication volume.

[0048] In some other possible implementations, the method further includes: If two subtasks belong to the same data parallel group and their execution phases are adjacent, determine the pipeline parallel communication volume between the two subtasks. The task communication volume between the two subtasks is determined based on the basic task communication volume and the pipeline parallel communication volume between the two subtasks.

[0049] Following the example above, subtasks A11 and A21 belong to the same data parallel group and their task execution stages are adjacent. When determining the task communication volume between subtasks A11 and A21, pipeline parallel communication volume can be added on the basis of the basic task communication volume.

[0050] In this way, for two subtasks that are adjacent in execution phase and share the same data parallel group, adding pipelined parallel communication on top of the basic communication volume helps to improve the comprehensiveness and accuracy of determining task communication volume.

[0051] In the above embodiments, the data parallel group partitioning under the data parallel strategy and the task execution stage partitioning under the pipelined parallel strategy are considered. The task execution stage partitioning under the tensor parallel strategy is not considered because the tensor parallel strategy has higher communication requirements than the data parallel strategy and the pipelined parallel strategy, and is generally completed within a single machine. However, this embodiment is aimed at multiple task processing nodes, that is, multiple machines. At the machine level, the communication volume of the tensor parallel strategy can be ignored to reduce the scale.

[0052] For example, it can be done through Subtasks sub-tasks The task communication between each pair of subtasks By splicing them together, you can generate 3D inter-task communication matrix .

[0053] To more intuitively illustrate the communication volume between subtasks, please refer to [link / reference]. Figure 2 This is a communication diagram between subtasks illustrated in an exemplary embodiment of this application. Figure 2 As shown in the diagram, taking 61 subtasks as an example, the horizontal axis represents the target subtask, and the vertical axis represents the source subtask. During communication between subtasks, data is sent from the source subtask to the target subtask; that is, the source subtask acts as the starting point of communication, and the target subtask acts as the destination. To visually represent the communication volume between each pair of subtasks, the intensity of the color is used to indicate the volume; the darker the color, the greater the communication volume.

[0054] S102: Construct a network affinity matrix based on the bandwidth and congestion metrics between every two task processing nodes among the multiple task processing nodes to be scheduled.

[0055] In practical applications, each task processing node is a machine device, including but not limited to a Graphics Processing Unit (GPU), a Central Processing Unit (CPU), a Field Programmable Gate Array (FPGA), an Artificial Intelligence (AI) accelerator card, and an edge computing box. Multiple task processing nodes to be scheduled can be nodes in the same topology.

[0056] In some possible implementations, the bandwidth and congestion metrics between the two task processing nodes are determined through the following steps: Determine the connection relationship between two task processing nodes, including direct connection, indirect connection, and no connection; Based on the connection relationship between the two task processing nodes, determine the bandwidth and congestion indicators between the two task processing nodes.

[0057] In the above steps, within the topology formed by the multiple task processing nodes, if one of the two task processing nodes can directly reach the other, the connection between the two task processing nodes can be determined as a direct connection; if one of the two task processing nodes needs to go through other nodes to reach the other, the connection between the two task processing nodes can be determined as an indirect connection; if one of the two processing nodes is unreachable from the other, the connection between the two task processing nodes can be determined as a disconnection. Based on the connection relationships between the two task processing nodes, the bandwidth and congestion indicators between the two task processing nodes are determined.

[0058] In this way, when determining the bandwidth and congestion indicators between two task processing nodes, the connection between the two task processing nodes is taken into account, ensuring that the determined bandwidth and congestion indicators can reflect the actual node topology, improving the accuracy of the determined bandwidth and congestion indicators, and enhancing the adaptability of task scheduling.

[0059] In some possible implementations, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: When the connection relationship is a direct connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate between the two task processing nodes are normalized. The normalized original bandwidth is determined as the bandwidth between the two task processing nodes; Based on the normalized retransmission rate, the normalized network interface queue length, and the normalized packet loss rate, the congestion index between the two task processing nodes is determined.

[0060] In the above steps, if the connection relationship is a direct connection, the raw bandwidth, retransmission rate, network interface queue length, and packet loss rate between the two task processing nodes can be collected.

[0061] For example, see also Figure 3 This is a schematic diagram illustrating bandwidth between task processing nodes as an exemplary embodiment of this application. The bandwidth shown in this example is the raw bandwidth, such as... Figure 3 As shown, gray dots represent task processing nodes. To visually demonstrate the bandwidth between pairs of task processing nodes, the intensity of the line connecting the gray dots is used to represent the bandwidth; the darker the line, the greater the bandwidth value.

[0062] The raw bandwidth, retransmission rate, network interface queue length, and packet loss rate between the two task processing nodes are normalized.

[0063] When normalizing the original bandwidth, specifically, the ratio between the original bandwidth and the bandwidth threshold can be determined, and the smaller value between this ratio and 1 is determined as the original bandwidth after normalization, thereby normalizing the bandwidth to the range of [0,1].

[0064] For example, the original bandwidth after normalization can be determined by the following formula (2): (2) in, This represents the normalized raw bandwidth between task processing node u and task processing node v; a larger value indicates better bandwidth. This represents the raw bandwidth between task processing node u and task processing node v. This indicates the bandwidth threshold.

[0065] The specific value of the bandwidth threshold can be set according to the actual task scheduling needs, and there is no restriction here. For example, it can be 10Gbps. The original bandwidth exceeding the bandwidth threshold is considered to fully meet the requirements.

[0066] The specific methods for normalizing the retransmission rate, network interface queue length, and packet loss rate are similar to those for normalizing the original bandwidth, and can be referred to the description in the above embodiments, which will not be repeated here.

[0067] In this embodiment, the normalized original bandwidth can be determined as the bandwidth between the two task processing nodes. Based on the normalized retransmission rate, the normalized network interface queue length, and the normalized packet loss rate, the congestion index between the two task processing nodes can be determined.

[0068] When determining the congestion index between two task processing nodes based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate, specifically, the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate can be weighted and summed to obtain the congestion index between the two task processing nodes. The specific values ​​of the weights corresponding to the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate can be set according to the actual task scheduling needs and are not restricted here.

[0069] For example, the congestion index in the case of direct connection can be determined by the following formula (3): (3) in, This represents the congestion metric between directly connected task processing nodes u and v. This represents the normalized retransmission rate between directly connected task processing nodes u and v. This represents the weight corresponding to the retransmission rate after normalization. This represents the normalized network interface queue length between directly connected task processing nodes u and v. This represents the weight corresponding to the normalized network interface queue length. This represents the normalized packet loss rate between directly connected task processing nodes u and v. This represents the weight corresponding to the packet loss rate after normalization.

[0070] For example, see also Figure 4 This is a schematic diagram illustrating a congestion index between task processing nodes, as shown in an exemplary embodiment of this application. Figure 4 As shown, gray dots represent task processing nodes. To visually represent the congestion indicators between pairs of task processing nodes, the intensity of the lines connecting the gray dots is used to indicate the congestion level; the darker the line, the higher the congestion level.

[0071] In this way, for directly connected task processing nodes, the original bandwidth, retransmission rate, network interface queue length, and packet loss rate between the two task processing nodes are normalized to eliminate the difference in units. Then, the normalized original bandwidth is taken as the required bandwidth. The normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate are weighted and summed to obtain the congestion index, thus avoiding the quality of a certain value from affecting the accuracy of the congestion index.

[0072] In some possible implementations, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: In the case where the connection relationship is an indirect connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate corresponding to the connection path between the two task processing nodes are normalized. The normalized original bandwidth corresponding to the shortest connection path between two task processing nodes is determined as the bandwidth between the two task processing nodes. Based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate of the connection path between the two task processing nodes, the congestion index between the two task processing nodes is determined.

[0073] In the above steps, if the connection relationship is an indirect connection, such as communication between two task processing nodes requiring forwarding through a switch, a connection path between the two task processing nodes can be determined. The connection path includes multiple directly connected task processing nodes. The original bandwidth, retransmission rate, network interface queue length, and packet loss rate between each pair of directly connected task processing nodes on the connection path can be collected, and the original bandwidth, retransmission rate, network interface queue length, and packet loss rate between each pair of directly connected task processing nodes on the connection path can be normalized. Here, the specific method of normalization is similar to that in the aforementioned embodiments (e.g., formula (2)), and can be referred to the description in the aforementioned embodiments, which will not be repeated here.

[0074] The effective bandwidth between two indirectly connected task processing nodes depends on the bottleneck link on the shortest connection path. In this embodiment of the disclosure, the original bandwidth with the smallest value is selected from the normalized original bandwidth between every two directly connected task processing nodes on the connection path. That is, the normalized original bandwidth corresponding to the shortest connection path between the two task processing nodes is determined, and this value is determined as the bandwidth between the two task processing nodes.

[0075] For example, the bandwidth in the case of indirect connection can be determined by the following formula (4): (4) in, This represents the bandwidth between indirectly connected task processing nodes u and v, where P represents the connection path between task processing nodes u and v, and task processing nodes x and y are two directly connected task processing nodes on the connection path P. This represents the normalized raw bandwidth between task processing node x and task processing node y.

[0076] The congestion index between two indirectly connected task processing nodes depends on the congestion index between every two directly connected task processing nodes on the connection path between the two task processing nodes. When determining the congestion index between two task processing nodes based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate corresponding to the connection path between the two task processing nodes, specifically, for every two directly connected task processing nodes on the connection path between the two task processing nodes, the congestion index between the two task processing nodes can be determined based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate between the two task processing nodes. Here, the specific method for determining the congestion index is similar to that in the aforementioned embodiments (e.g., formula (3)), and can be referred to the description in the aforementioned embodiments, which will not be repeated here. Then, based on the congestion index between every two directly connected task processing nodes on the connection path, the congestion index between the two task processing nodes is determined.

[0077] For example, the congestion index in the case of indirect connection can be determined by the following formula (5): (5) in, This represents the congestion index between indirectly connected task processing nodes u and v, where P represents the connection path between task processing nodes u and v, and task processing nodes x and y are two directly connected task processing nodes on the connection path P. This represents the congestion index between task processing node x and task processing node y.

[0078] In this way, for indirectly connected task processing nodes, the bandwidth between the two task processing nodes is determined based on the normalized raw bandwidth corresponding to the shortest connection path between them, and the congestion index between the two task processing nodes is determined based on the congestion index between each pair of directly connected task processing nodes on the connection path between them. This helps to improve the accuracy of the determined bandwidth and congestion index and enhance network affinity.

[0079] In some possible implementations, determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: If the connection is not established, the bandwidth and congestion metrics between the two task processing nodes are set to 0.

[0080] Here, since the two task processing nodes are not connected, there will be no communication between them, and therefore no corresponding bandwidth and congestion indicators. Thus, the bandwidth and congestion indicators between the two task processing nodes can be set to 0. This way, subsequent clustering mapping will automatically skip such pairings, avoid assigning communication-intensive subtasks to task processing nodes that cannot communicate with each other, reduce invalid waiting and timeout retries, save detection overhead, and improve the stability of task scheduling.

[0081] In some possible implementations, constructing a network affinity matrix based on bandwidth and congestion metrics between every two task processing nodes among the multiple task processing nodes to be scheduled includes: Based on the bandwidth and congestion indicators between every two task processing nodes in the multiple task processing nodes to be scheduled, the network affinity score between every two task processing nodes in the multiple task processing nodes is determined. A network affinity matrix is ​​constructed based on the network affinity score between every two task processing nodes among the multiple task processing nodes.

[0082] Optionally, the bandwidth and congestion index between each pair of task processing nodes can be smoothed to obtain the smoothed bandwidth and smoothed congestion index between each pair of task processing nodes. Based on the smoothed bandwidth and smoothed congestion index between each pair of task processing nodes in the multiple task processing nodes, the network affinity score between each pair of task processing nodes in the multiple task processing nodes can be determined.

[0083] When smoothing the bandwidth between two task processing nodes, for each time step other than the first time step, the bandwidth of the current time step after smoothing is determined based on the bandwidth of that time step and the bandwidth of the previous time step. The bandwidth of the previous time step can be the bandwidth of the previous time step after smoothing.

[0084] For example, bandwidth smoothing can be performed as shown in the following formula (6): (6) in, This represents the bandwidth at the current time step after smoothing. Represents the smoothing factor. Indicates the bandwidth at the current time step. This indicates the bandwidth of the previous time step.

[0085] The specific value of the smoothing factor can be set according to the actual task scheduling needs, and there is no restriction here.

[0086] The specific method for smoothing the congestion index between two task processing nodes is similar to the method for smoothing the bandwidth between two task processing nodes, and can be referred to the description in the above embodiments, which will not be repeated here.

[0087] For example, the network affinity score between two task processing nodes can be determined by the following formula (7): (7) in, This represents the network affinity score between task processing node u and task processing node v. The network affinity score ranges from [0,1]. A higher network affinity score indicates better network quality. This represents the congestion penalty coefficient, which indicates the degree of negative impact on network quality. This represents the bandwidth after smoothing between task processing node u and task processing node v. This represents the congestion index after smoothing between task processing node u and task processing node v.

[0088] The specific value of the congestion penalty coefficient can be set according to the actual task scheduling needs, and there is no restriction here.

[0089] The network affinity score between each pair of task processing nodes is calculated. By splicing them together, you can generate The network affinity matrix is ​​Q, and A is the number of task processing nodes.

[0090] S103: Cluster the multiple subtasks based on the inter-task communication matrix to obtain a task cluster set, and cluster the multiple task processing nodes based on the network affinity matrix to obtain a node cluster set. Each task cluster in the task cluster set includes at least one subtask, and each node cluster in the node cluster set includes at least one task processing node.

[0091] In the above steps, the inter-task communication matrix can be used to cluster the multiple subtasks to determine the task cluster set, and the network affinity matrix can be used to cluster the multiple task processing nodes to determine the node cluster set.

[0092] In some possible implementations, the step of clustering the multiple subtasks based on the inter-task communication matrix to obtain a task cluster set includes: Based on the inter-task communication matrix, spectral clustering is performed on the multiple subtasks to obtain a task spectral clustering matrix. Normalize each row of the task spectrum clustering matrix to obtain a normalized task spectrum clustering matrix; K-means clustering is performed on the normalized task spectrum clustering matrix to obtain multiple task clusters; Based on the target communication volume of each task cluster, determine the communication index of each task cluster; The multiple task clusters are sorted in descending order according to the communication index to generate the task cluster set.

[0093] In the above steps, the inter-task communication matrix can be used to perform spectral clustering on the multiple subtasks to obtain a task spectral clustering matrix. Spectral clustering is a graph-based clustering algorithm that mainly utilizes the similarity between data points to construct a graph structure, and then performs clustering based on the eigenvectors of the Laplacian matrix.

[0094] In some possible implementations, the step of performing spectral clustering on the plurality of subtasks based on the inter-task communication matrix to obtain a task spectral clustering matrix includes: The inter-task communication matrix is ​​used as the weight matrix for spectral clustering, and each element in the inter-task communication matrix is ​​used to indicate the weight between every two graph nodes in the undirected connected graph. Based on the inter-task communication matrix, determine the degree matrix of the undirected connected graph; Based on the degree matrix and the inter-task communication matrix, a symmetric normalized Laplace matrix is ​​determined. Calculate multiple eigenvalues ​​of the symmetric normalized Laplace matrix, and select the target eigenvalues ​​with the smallest predetermined number from the multiple eigenvalues; The task spectrum clustering matrix is ​​generated based on the feature vectors corresponding to the target feature values.

[0095] The process of determining the spectral clustering matrix will be explained next.

[0096] Specifically, we can define an undirected connected graph G = (V, E), where V represents the set of vertices and E represents the set of edges. Then, we determine a weight matrix W, where each element of the weight matrix W indicates the weight between any two nodes in the undirected connected graph G. For example... W represents the weight between graph node a and graph node b in an undirected connected graph G. Based on the weight matrix W, the degree matrix D of the undirected connected graph G can be determined. In the degree matrix D, only the elements on the diagonal from the top left to the bottom right have specific values; all other elements are 0.

[0097] For example, the degree matrix can be determined by the following formula (8): (8) in, This represents the element at the a-th row and a-th column in the degree matrix D. Let represent the weight between graph node a and graph node b in an undirected connected graph G, and n represent the number of graph nodes in the undirected connected graph G.

[0098] After determining the degree matrix D, the symmetric normalized Laplace matrix can be determined based on the degree matrix D and the weight matrix W. For example, the symmetric normalized Laplace matrix can be determined by the following formula (9): (9) in, Let L denote the symmetric normalized Laplacian matrix, D denote the degree matrix, and L denote the unnormalized Laplacian matrix. The unnormalized Laplacian matrix is ​​determined based on the difference between the degree matrix and the weight matrix (L=DW).

[0099] In this way, the orthogonality of eigenvectors can be guaranteed by using a symmetric normalized Laplace matrix.

[0100] Determining the symmetric normalized Laplace matrix Then, calculate the symmetric normalized Laplace matrix. Multiple eigenvalues.

[0101] For example, the eigenvalues ​​of a symmetric normalized Laplace matrix can be determined by the following formula (10): (10) in, Represents the symmetric normalized Laplace matrix. Let x represent the eigenvalues ​​of the symmetric normalized Laplacian matrix, and let x represent the eigenvectors corresponding to the eigenvalues.

[0102] Determining the symmetric normalized Laplace matrix After obtaining multiple feature values, the target feature value with the smallest preset number k is selected from the multiple feature values.

[0103] The preset quantity k is based on the symmetric normalized Laplace matrix. The characteristic gap is determined, and the characteristic gap is used to represent the symmetric normalized Laplacian matrix. The difference between two adjacent feature values ​​is calculated after sorting multiple feature values ​​in ascending order of numerical value. Specifically, the largest feature gap can be determined from multiple feature gaps. This can be understood as the difference between two feature values. The feature value that appears earlier in the ascending order of numerical value is then identified, and its corresponding order value is set as a preset quantity k.

[0104] For example, see also Figure 5 This is a schematic diagram illustrating the distribution of eigenvalues ​​of a symmetric normalized Laplace matrix, as shown in an exemplary embodiment of this application. Figure 5 As shown in the figure, the horizontal axis represents the index of the feature value, and the vertical axis represents the specific value of the feature value. In this example, the maximum feature gap is the feature gap between the 5th and 6th feature values. Therefore, the preset quantity k=5, and the first 5 feature values ​​in ascending order of numerical value are used as the target feature values.

[0105] Based on the above, it can be seen that in calculating the symmetric normalized Laplacian matrix... When there are multiple eigenvalues, the eigenvector corresponding to each eigenvalue can also be calculated (refer to formula (10)). Each eigenvector is a... A dimensional vector. The feature vectors corresponding to the target feature values ​​are used as column vectors to form... A spectral clustering matrix U of dimension, where k is the preset number.

[0106] When determining the task spectral clustering matrix, the inter-task communication matrix is ​​used as the weight matrix for spectral clustering processing. The spectral clustering matrix U obtained according to the above example is the task spectral clustering matrix.

[0107] In this way, spectral clustering can capture the natural grouping structure in the network (such as nonlinearity and nonconvexity), adapt to dynamic and irregular traffic scenarios, and is particularly effective when there is congestion or traffic hotspots. Using spectral clustering helps to improve the clustering effect, enhance the accuracy of the task spectral clustering matrix, and spectral clustering is fast, making it suitable for online rapid scheduling decisions.

[0108] After obtaining the task spectrum clustering matrix, each row of the task spectrum clustering matrix can be normalized to obtain a normalized task spectrum clustering matrix.

[0109] For example, the elements in the normalized task spectrum clustering matrix can be represented by the following formula (11): (11) in, This represents the element at the c-th row and d-th column in the normalized task spectrum clustering matrix. This represents the element at the c-th row and d-th column in the task spectrum clustering matrix, where k represents the preset number.

[0110] Here, the task spectrum clustering matrix is The matrix is ​​dimensional, and correspondingly, the normalized task spectrum clustering matrix is ​​still . A matrix of dimension k is subjected to K-means clustering in k dimensions to obtain multiple task clusters.

[0111] For example, see also Figure 6 This is a schematic diagram illustrating a method for determining task clusters, as shown in an exemplary embodiment of this application. In this example, to more clearly demonstrate the clustering effect, the k-dimensional data is mapped to a 2-dimensional representation, such as... Figure 6 As shown in the diagram, a dot represents a subtask, and multiple task clusters can be obtained based on the distribution density of the subtasks.

[0112] After obtaining multiple task clusters, the communication index of each task cluster can be determined based on the target communication volume of each task cluster.

[0113] In some possible implementations, the target traffic includes intra-cluster traffic, which can be understood as indicating all traffic of the corresponding task cluster, including intra-cluster and extra-cluster traffic.

[0114] The step of determining the communication index for each task cluster based on the target communication volume of each task cluster includes: For each task cluster, the communication index of the task cluster is determined based on the intra-cluster communication volume of the task cluster, the target communication volume of the task cluster, and the maximum target communication volume in each task cluster.

[0115] In this step, for each task cluster, the cohesion ratio of the task cluster can be determined based on the intra-cluster communication volume and the target communication volume of the task cluster; the communication index of the task cluster can be determined based on the cohesion ratio of the task cluster, the target communication volume of the task cluster, and the maximum target communication volume in each task cluster.

[0116] Specifically, the cohesion ratio of a task cluster can be determined based on the ratio of the intra-cluster communication volume to the target communication volume of the task cluster.

[0117] For example, the cohesion ratio of a task cluster can be determined by the following formula (12): (12) in, This represents the cohesion ratio of the e-th task cluster. This represents the intra-cluster communication of the e-th task cluster. This represents the target communication volume of the e-th task cluster.

[0118] For example, the communication index of a task cluster can be determined by the following formula (13): (13) in, This represents the communication index of the e-th task cluster. This represents the cohesion ratio of the e-th task cluster. This represents the intra-cluster communication of the e-th task cluster. This represents the maximum target communication volume in each task cluster. Indicates the first communication coefficient. This represents the second communication coefficient.

[0119] The specific values ​​of the first communication coefficient and the second communication coefficient can be set according to the actual task scheduling needs, and there are no restrictions here.

[0120] After determining the communication index of each task cluster, the multiple task clusters can be sorted in descending order according to the communication index to generate the task cluster set.

[0121] For example, the set of task clusters can be represented by the following formula (14): (14) in, This represents a set of task clusters. In this example, the task cluster set includes... A cluster of tasks This represents the first task cluster sorted in descending order of communication index. This represents the second task cluster sorted in descending order of communication index. This indicates the last task cluster sorted in descending order of communication index.

[0122] In this way, using the inter-task communication matrix as input, spectral clustering is first used to obtain a task spectral clustering matrix. Then, row normalization is performed on the task spectral clustering matrix to eliminate dimensional differences. Next, K-means clustering is used to divide the subtasks into multiple task clusters. Then, based on the target communication volume of each task cluster, a communication index is determined, and the task clusters are arranged in descending order of communication index to form a set of task clusters. Thus, the task clusters with the most frequent communication are prioritized, and can be assigned to appropriate node clusters during subsequent mapping, which helps reduce cross-node traffic, lower end-to-end latency, and improve bandwidth utilization.

[0123] In some possible implementations, the clustering of the plurality of task processing nodes based on the network affinity matrix to obtain a node cluster set includes: Based on the network affinity matrix, spectral clustering is performed on the multiple task processing nodes to obtain a node spectral clustering matrix. The nodes are normalized by normalizing each row of the node spectrum clustering matrix to obtain the normalized node spectrum clustering matrix. K-means clustering is performed on the normalized node spectrum clustering matrix to obtain multiple node clusters; The node clusters are sorted in descending order according to the number of task processing nodes included in the node clusters to generate the node cluster set.

[0124] Here, the specific process of obtaining the node spectral clustering matrix based on the network affinity matrix is ​​similar to the process of obtaining the task spectral clustering matrix based on the inter-task communication matrix. The network affinity matrix is ​​used as the weight matrix for spectral clustering processing. For details, please refer to the description in the above embodiments, which will not be repeated here.

[0125] The specific process of obtaining the normalized node spectrum clustering matrix is ​​similar to that of obtaining the normalized task spectrum clustering matrix. For details, please refer to the description in the above embodiments, which will not be repeated here.

[0126] The specific process of obtaining multiple node clusters is similar to that of obtaining multiple task clusters, and can be referred to the description in the above embodiments, which will not be repeated here.

[0127] After obtaining multiple task clusters, the number of task processing nodes included in each task cluster can be determined. The multiple node clusters are then sorted in descending order according to the number of task processing nodes included in each node cluster to generate the node cluster set.

[0128] For example, the set of node clusters can be represented by the following formula (15): (15) in, This represents a set of node clusters. In this example, the set of node clusters includes... A cluster of nodes, This represents the first cluster of nodes sorted in descending order by the number of task processing nodes. This represents the second cluster of nodes sorted in descending order by the number of task processing nodes. This represents the last cluster of nodes sorted in descending order by the number of task processing nodes.

[0129] In this way, using the network affinity matrix as input, spectral clustering is first used to obtain the node spectral clustering matrix. Then, the node spectral clustering matrix is ​​row-normalized to eliminate dimensional differences. Subsequently, K-means clustering is used to divide the task processing nodes into multiple node clusters, which are then arranged in descending order of size to form a set of node clusters. As a result, the node cluster with the most task processing nodes is sorted first, and can be assigned to suitable task clusters during subsequent mapping, which helps to reduce cross-node traffic, reduce end-to-end latency, and improve bandwidth utilization.

[0130] S104: Based on the resource requirement information of each subtask and the resource supply information of each task processing node, map the task cluster set and the node cluster set to determine the mapping result between the multiple subtasks and the multiple task processing nodes.

[0131] In this step, the task cluster set and the node cluster set can be mapped based on whether the resource supply information of each task processing node can meet the resource requirements of each sub-task, thereby determining the mapping result between the multiple sub-tasks and the multiple task processing nodes.

[0132] The resource demand information and the resource supply information include, but are not limited to, the number of GPUs, video memory, specific accelerator card types, and memory.

[0133] In some possible implementations, the step of mapping the task cluster set and the node cluster set based on the resource requirement information of each subtask and the resource supply information of each task processing node, and determining the mapping result between the plurality of subtasks and the plurality of task processing nodes, includes: Based on the resource requirements of each subtask and the resource supply information of each task processing node, each task cluster and each node cluster is mapped sequentially according to the arrangement order of each task cluster in the task cluster set and the arrangement order of each node cluster in the node cluster set, until the allocation of the multiple subtasks or the allocation of the multiple task processing nodes is completed, thereby obtaining the mapping result between the multiple subtasks and the multiple task processing nodes.

[0134] In the above steps, the task clusters can be sequentially assigned to the node clusters according to the arrangement order of each task cluster in the task cluster set and the arrangement order of each node cluster in the node cluster set. By checking whether the task clusters and node clusters meet the resource constraints, the task clusters can be assigned to the node clusters in sequence until the multiple sub-tasks are assigned or the multiple task processing nodes are assigned, so as to obtain the mapping result between the multiple sub-tasks and the multiple task processing nodes.

[0135] Optionally, if a matching error occurs during the mapping process, a rollback can be triggered, returning to the steps of dividing subtasks and determining task processing nodes. The inter-task communication matrix and network affinity matrix can be reconstructed and clustered to obtain a new set of task clusters and node clusters for remapping. An alarm signal can also be issued for manual verification.

[0136] In this way, prioritizing the allocation of task clusters with higher communication volume to node clusters with higher network affinity and richer resources helps to reduce communication overhead, reduce end-to-end latency, improve resource utilization, and increase the efficiency of the entire deep learning task.

[0137] In some possible implementations, the mapping of task clusters and node clusters includes: If the resource supply of the node cluster can meet the resource requirements of the task cluster, the node cluster and the task cluster are mapped. If the resource supply of the node cluster cannot meet the resource requirements of the task cluster, the task cluster is divided into multiple sub-task clusters according to the intra-cluster communication density. The sub-task clusters are then mapped sequentially according to their resource requirements in descending order.

[0138] For example, for the first task cluster in the task cluster set, it is determined whether the resource supply of the first node cluster in the node cluster set can meet the resource requirements of the first task cluster. If the resource supply of the first node cluster can meet the resource requirements of the first task cluster, the first node cluster and the first task cluster are mapped. Then, for the second task cluster in the task cluster set, it is determined whether the resource supply of the second node cluster in the node cluster set can meet the resource requirements of the second task cluster. For example, if the first task cluster includes 10 sub-tasks and the first node cluster includes 10 task processing nodes, then the first node cluster and the first task cluster can be mapped.

[0139] If the resource supply of the first node cluster cannot meet the resource requirements of the first task cluster, the second task cluster is divided into multiple sub-task clusters according to the intra-cluster communication density of the second task cluster. These sub-task clusters are then sorted in descending order of their resource requirements. The first sub-task cluster and the first node cluster are mapped together. Next, resource constraint checks are performed on the second sub-task cluster and the second node cluster to attempt mapping. For example, if the first task cluster includes 10 sub-tasks and the first node cluster includes 6 task processing nodes, the first node cluster can be split into two sub-task clusters: the first sub-task cluster includes 6 sub-tasks, and the second sub-task cluster includes 4 sub-tasks. The first sub-task cluster is then mapped to the first node cluster, and the second sub-task cluster is mapped to the second node cluster.

[0140] Optionally, during the mapping process, in addition to resource constraint judgment, it is also possible to detect whether the task processing nodes in the node cluster are suitable for executing the tasks in the task cluster, and adjust the mapping accordingly based on the detection results.

[0141] In this way, when the resource supply of the node cluster is sufficient to meet the resource requirements of the entire task cluster, mapping is performed directly. If the resource supply is insufficient, the task cluster is partially split and mapped in sequence according to the resource requirements, giving priority to the allocation of task clusters with larger communication volumes, while preserving local high affinity characteristics.

[0142] For a clearer illustration of the task scheduling process, see [link to relevant documentation]. Figure 7 This is a schematic diagram illustrating a task scheduling process as shown in an exemplary embodiment of this application. Figure 7 As shown, based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, an inter-task communication matrix is ​​constructed. Based on the bandwidth and congestion indicators between every two task processing nodes in the multiple task processing nodes to be scheduled, a network affinity matrix is ​​constructed. The multiple subtasks are clustered based on the inter-task communication matrix to obtain a task cluster set. The task clusters in the task cluster set are sorted in descending order of communication index. The multiple task processing nodes are clustered based on the network affinity matrix to obtain a node cluster set. The node clusters in the node cluster set are sorted in descending order of size. Each task cluster and each node cluster is mapped sequentially according to the order of the task clusters in the task cluster set and the order of the node clusters in the node cluster set, resulting in a mapping between the multiple subtasks and the multiple task processing nodes. Specific steps are described in the aforementioned embodiment and will not be repeated here.

[0143] To more clearly demonstrate the effect of task scheduling, please refer to the example provided. Figures 8-9 , Figure 8 This is a schematic diagram illustrating a random mapping between multiple subtasks and multiple task processing nodes. Figure 9This is a schematic diagram illustrating the mapping between multiple subtasks and multiple task processing nodes, as shown in an exemplary embodiment of this application. Red squares represent subtasks; in this example, 16 tasks (T0-T15) are used. Four different colored dots represent four node clusters: purple dots represent node cluster 0, blue dots represent node cluster 1, green dots represent node cluster 2, and yellow dots represent node cluster 3. In this example, each node cluster includes four task processing nodes; for example, node cluster 3 includes task processing nodes N0, N1, N2, and N3. Dashed lines represent the mapping between subtasks and task processing nodes. Green lines between subtasks represent communication between them. Blue lines between task processing nodes within the same node cluster represent intra-cluster data interaction. Purple lines between task processing nodes in different node clusters represent inter-cluster data interaction. (Comparison) Figure 8 and Figure 9 , Figure 9 The mapping results shown are compared to Figure 8 The mapping results shown are more organized, which helps to reduce end-to-end latency of tasks, reduce the number of task clusters crossing low-bandwidth or high-congestion links, reduce communication bottlenecks, and improve resource utilization.

[0144] The task scheduling method provided in this application constructs an inter-task communication matrix based on the task communication volume between subtasks, constructs a network affinity matrix based on the bandwidth and congestion indicators between task processing nodes, and performs clustering processing based on the inter-task communication matrix and the network affinity matrix to obtain a set of task clusters and a set of node clusters with high network affinity, thereby improving the network affinity of task scheduling. Mapping the set of task clusters and the set of node clusters can map communication-intensive task clusters to node clusters with high network affinity, which helps to reduce communication overhead, reduce end-to-end latency of tasks, improve resource utilization, and improve the efficiency of the entire deep learning task.

[0145] Corresponding to the aforementioned embodiments of the task scheduling method, this application also provides embodiments of a task scheduling apparatus.

[0146] Please see Figure 10 This is a schematic diagram illustrating a task scheduling device according to an exemplary embodiment of this application. Figure 10 As shown in the figure, the task scheduling device 1000 provided in this application embodiment includes: The task matrix construction module 1001 is used to construct an inter-task communication matrix based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled. The network matrix construction module 1002 is used to construct a network affinity matrix based on the bandwidth and congestion indicators between every two task processing nodes in the multiple task processing nodes to be scheduled. The clustering processing module 1003 is used to perform clustering processing on the multiple subtasks based on the inter-task communication matrix to obtain a set of task clusters, and to perform clustering processing on the multiple task processing nodes based on the network affinity matrix to obtain a set of node clusters. Each task cluster in the set of task clusters includes at least one subtask, and each node cluster in the set of node clusters includes at least one task processing node. The set mapping module 1004 is used to map the task cluster set and the node cluster set according to the resource requirement information of each subtask and the resource supply information of each task processing node, and determine the mapping result between the multiple subtasks and the multiple task processing nodes.

[0147] In some possible implementations, the task matrix construction module 1001 obtains the plurality of subtasks through the following steps: Based on the task structure of the deep learning distributed task to be scheduled and the predetermined parallel strategy, the deep learning distributed task is divided into multiple subtasks, and the parallel strategy is used to indicate the task division method.

[0148] In some possible implementations, the parallel strategy includes at least one of data parallelism, pipeline parallelism, and tensor parallelism; the task matrix construction module 1001 determines the task communication volume between two subtasks through the following steps: The execution position of each subtask under each strategy is determined. The execution position of the subtask under the data parallel strategy is used to indicate the data parallel group to which the subtask belongs. The execution position of the subtask under the pipeline parallel strategy is used to indicate the execution stage to which the subtask belongs. The execution position of the subtask under the tensor parallel strategy is used to indicate the execution link to which the subtask belongs. Based on the task execution positions of the two subtasks, the task communication volume between the two subtasks is determined.

[0149] In some possible implementations, the network matrix construction module 1002 determines the bandwidth and congestion metrics between two task processing nodes through the following steps: Determine the connection relationship between two task processing nodes, including direct connection, indirect connection, and no connection; Based on the connection relationship between the two task processing nodes, determine the bandwidth and congestion indicators between the two task processing nodes.

[0150] In some possible implementations, the network matrix construction module 1002, when determining the bandwidth and congestion indicators between two task processing nodes based on the connection relationship between them, is specifically used for: When the connection relationship is a direct connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate between the two task processing nodes are normalized. The normalized original bandwidth is determined as the bandwidth between the two task processing nodes; Based on the normalized retransmission rate, the normalized network interface queue length, and the normalized packet loss rate, the congestion index between the two task processing nodes is determined.

[0151] In some possible implementations, the network matrix construction module 1002, when determining the bandwidth and congestion indicators between two task processing nodes based on the connection relationship between them, is specifically used for: In the case where the connection relationship is an indirect connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate corresponding to the connection path between the two task processing nodes are normalized. The normalized original bandwidth corresponding to the shortest connection path between two task processing nodes is determined as the bandwidth between the two task processing nodes. Based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate of the connection path between the two task processing nodes, the congestion index between the two task processing nodes is determined.

[0152] In some possible implementations, the network matrix construction module 1002, when determining the bandwidth and congestion indicators between two task processing nodes based on the connection relationship between them, is specifically used for: If the connection is not established, the bandwidth and congestion metrics between the two task processing nodes are set to 0.

[0153] In some possible implementations, when the clustering processing module 1003 performs clustering processing on the plurality of subtasks based on the inter-task communication matrix to obtain a task cluster set, it is specifically used for: Based on the inter-task communication matrix, spectral clustering is performed on the multiple subtasks to obtain a task spectral clustering matrix. Normalize each row of the task spectrum clustering matrix to obtain a normalized task spectrum clustering matrix; K-means clustering is performed on the normalized task spectrum clustering matrix to obtain multiple task clusters; Based on the target communication volume of each task cluster, determine the communication index of each task cluster; The multiple task clusters are sorted in descending order according to the communication index to generate the task cluster set.

[0154] In some possible implementations, when the clustering processing module 1003 performs spectral clustering processing on the plurality of subtasks based on the inter-task communication matrix to obtain a task spectral clustering matrix, it is specifically used for: The inter-task communication matrix is ​​used as the weight matrix for spectral clustering, and each element in the inter-task communication matrix is ​​used to indicate the weight between every two graph nodes in the undirected connected graph. Based on the inter-task communication matrix, determine the degree matrix of the undirected connected graph; Based on the degree matrix and the inter-task communication matrix, a symmetric normalized Laplace matrix is ​​determined. Calculate multiple eigenvalues ​​of the symmetric normalized Laplace matrix, and select the target eigenvalues ​​with the smallest predetermined number from the multiple eigenvalues; The task spectrum clustering matrix is ​​generated based on the feature vectors corresponding to the target feature values.

[0155] In some possible implementations, the target communication volume includes intra-cluster communication volume; the clustering processing module 1003, when determining the communication index of each task cluster based on the target communication volume of each task cluster, is specifically used for: For each task cluster, the communication index of the task cluster is determined based on the intra-cluster communication volume of the task cluster, the target communication volume of the task cluster, and the maximum target communication volume in each task cluster.

[0156] In some possible implementations, when the clustering processing module 1003 performs clustering processing on the plurality of task processing nodes based on the network affinity matrix to obtain a set of node clusters, it is specifically used for: Based on the network affinity matrix, spectral clustering is performed on the multiple task processing nodes to obtain a node spectral clustering matrix. The nodes are normalized by normalizing each row of the node spectrum clustering matrix to obtain the normalized node spectrum clustering matrix. K-means clustering is performed on the normalized node spectrum clustering matrix to obtain multiple node clusters; The node clusters are sorted in descending order according to the number of task processing nodes included in the node clusters to generate the node cluster set.

[0157] In some possible implementations, the set mapping module 1004 is specifically used for: Based on the resource requirements of each subtask and the resource supply information of each task processing node, each task cluster and each node cluster is mapped sequentially according to the arrangement order of each task cluster in the task cluster set and the arrangement order of each node cluster in the node cluster set, until the allocation of the multiple subtasks or the allocation of the multiple task processing nodes is completed, thereby obtaining the mapping result between the multiple subtasks and the multiple task processing nodes.

[0158] In some possible implementations, the set mapping module 1004, when mapping each task cluster and each node cluster, is specifically used for: If the resource supply of the node cluster can meet the resource requirements of the task cluster, the node cluster and the task cluster are mapped. If the resource supply of the node cluster cannot meet the resource requirements of the task cluster, the task cluster is divided into multiple sub-task clusters according to the intra-cluster communication density. The sub-task clusters are then mapped sequentially according to their resource requirements in descending order.

[0159] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0160] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0161] Based on the same technical concept, this application also provides a computer device 1100, referring to... Figure 11 The diagram shown is a schematic representation of the structure of a computer device according to an exemplary embodiment of this application, comprising: The processor is 1110, the memory is 1120, and the bus is 1130. The memory 1120 is used to store execution instructions and includes main memory 1121 and external memory 1122. The main memory 1121, also known as internal memory, is used to temporarily store the operation data in the processor 1110 and the data exchanged with external memory 1122 such as hard disk. The processor 1110 exchanges data with external memory 1122 through main memory 1121.

[0162] In this embodiment, the memory 1120 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 1110. That is, when the electronic device 1100 is running, the processor 1110 communicates with the memory 1120 through the bus 1130, or the processor 1110 communicates with the memory 1120 through other means, so that the processor 1110 executes the application code stored in the memory 1120, and then executes the steps of the task scheduling method described in any of the foregoing embodiments.

[0163] The memory 1120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0164] Processor 1110 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0165] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1100. In other embodiments of this application, the electronic device 1100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0166] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the task scheduling method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0167] This disclosure also provides a computer program product, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the task scheduling method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0168] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0169] Furthermore, embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0170] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0171] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0172] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0173] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0174] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0175] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0176] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A task scheduling method, characterized in that, The method includes: Based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, an inter-task communication matrix is ​​constructed, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled. Based on the bandwidth and congestion metrics between every two task processing nodes in a plurality of task processing nodes to be scheduled, a network affinity matrix is ​​constructed. Based on the inter-task communication matrix, the multiple subtasks are clustered to obtain a set of task clusters, and based on the network affinity matrix, the multiple task processing nodes are clustered to obtain a set of node clusters. Each task cluster in the set of task clusters includes at least one subtask, and each node cluster in the set of node clusters includes at least one task processing node. Based on the resource requirement information of each subtask and the resource supply information of each task processing node, the task cluster set and the node cluster set are mapped to determine the mapping result between the multiple subtasks and the multiple task processing nodes.

2. The method according to claim 1, characterized in that, The multiple sub-tasks are obtained through the following steps: Based on the task structure of the deep learning distributed task to be scheduled and the predetermined parallel strategy, the deep learning distributed task is divided into multiple subtasks, and the parallel strategy is used to indicate the task division method.

3. The method according to claim 2, characterized in that, The parallel strategy includes at least one of data parallelism, pipelined parallelism, and tensor parallelism; the task communication volume between two subtasks is determined through the following steps: The execution position of each subtask under each strategy is determined. The execution position of the subtask under the data parallel strategy is used to indicate the data parallel group to which the subtask belongs. The execution position of the subtask under the pipeline parallel strategy is used to indicate the execution stage to which the subtask belongs. The execution position of the subtask under the tensor parallel strategy is used to indicate the execution link to which the subtask belongs. Based on the task execution positions of the two subtasks, the task communication volume between the two subtasks is determined.

4. The method according to claim 1, characterized in that, The following steps determine the bandwidth and congestion metrics between two task processing nodes: Determine the connection relationship between two task processing nodes, including direct connection, indirect connection, and no connection; Based on the connection relationship between the two task processing nodes, determine the bandwidth and congestion indicators between the two task processing nodes.

5. The method according to claim 4, characterized in that, The step of determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: When the connection relationship is a direct connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate between the two task processing nodes are normalized. The normalized original bandwidth is determined as the bandwidth between the two task processing nodes; Based on the normalized retransmission rate, the normalized network interface queue length, and the normalized packet loss rate, the congestion index between the two task processing nodes is determined.

6. The method according to claim 4, characterized in that, The step of determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: In the case where the connection relationship is an indirect connection, the original bandwidth, retransmission rate, network interface queue length and packet loss rate corresponding to the connection path between the two task processing nodes are normalized. The normalized original bandwidth corresponding to the shortest connection path between two task processing nodes is determined as the bandwidth between the two task processing nodes. Based on the normalized retransmission rate, normalized network interface queue length, and normalized packet loss rate of the connection path between the two task processing nodes, the congestion index between the two task processing nodes is determined.

7. The method according to claim 4, characterized in that, The step of determining the bandwidth and congestion metrics between the two task processing nodes based on the connection relationship between them includes: If the connection is not established, the bandwidth and congestion metrics between the two task processing nodes are set to 0.

8. The method according to claim 1, characterized in that, The clustering process based on the inter-task communication matrix to obtain a task cluster set includes: Based on the inter-task communication matrix, spectral clustering is performed on the multiple subtasks to obtain a task spectral clustering matrix. Normalize each row of the task spectrum clustering matrix to obtain a normalized task spectrum clustering matrix; K-means clustering is performed on the normalized task spectrum clustering matrix to obtain multiple task clusters; Based on the target communication volume of each task cluster, determine the communication index of each task cluster; The multiple task clusters are sorted in descending order according to the communication index to generate the task cluster set.

9. The method according to claim 8, characterized in that, The step of performing spectral clustering on the multiple subtasks based on the inter-task communication matrix to obtain a task spectral clustering matrix includes: The inter-task communication matrix is ​​used as the weight matrix for spectral clustering, and each element in the inter-task communication matrix is ​​used to indicate the weight between every two graph nodes in the undirected connected graph. Based on the inter-task communication matrix, determine the degree matrix of the undirected connected graph; Based on the degree matrix and the inter-task communication matrix, a symmetric normalized Laplace matrix is ​​determined. Calculate multiple eigenvalues ​​of the symmetric normalized Laplace matrix, and select the target eigenvalues ​​with the smallest predetermined number from the multiple eigenvalues; The task spectrum clustering matrix is ​​generated based on the feature vectors corresponding to the target feature values.

10. The method according to claim 8, characterized in that, The target communication volume includes intra-cluster communication volume; determining the communication index of each task cluster based on the target communication volume of each task cluster includes: For each task cluster, the communication index of the task cluster is determined based on the intra-cluster communication volume of the task cluster, the target communication volume of the task cluster, and the maximum target communication volume in each task cluster.

11. The method according to claim 1, characterized in that, The clustering of the multiple task processing nodes based on the network affinity matrix yields a set of node clusters, including: Based on the network affinity matrix, spectral clustering is performed on the multiple task processing nodes to obtain a node spectral clustering matrix. The nodes are normalized by normalizing each row of the node spectrum clustering matrix to obtain the normalized node spectrum clustering matrix. K-means clustering is performed on the normalized node spectrum clustering matrix to obtain multiple node clusters; The node clusters are sorted in descending order according to the number of task processing nodes included in the node clusters to generate the node cluster set.

12. The method according to claim 1, characterized in that, The step of mapping the task cluster set and the node cluster set based on the resource requirement information of each subtask and the resource supply information of each task processing node, and determining the mapping result between the multiple subtasks and the multiple task processing nodes, includes: Based on the resource requirements of each subtask and the resource supply information of each task processing node, each task cluster and each node cluster is mapped sequentially according to the arrangement order of each task cluster in the task cluster set and the arrangement order of each node cluster in the node cluster set, until the allocation of the multiple subtasks or the allocation of the multiple task processing nodes is completed, thereby obtaining the mapping result between the multiple subtasks and the multiple task processing nodes.

13. The method according to claim 12, characterized in that, The mapping of each task cluster and each node cluster includes: If the resource supply of the node cluster can meet the resource requirements of the task cluster, the node cluster and the task cluster are mapped. If the resource supply of the node cluster cannot meet the resource requirements of the task cluster, the task cluster is divided into multiple sub-task clusters according to the intra-cluster communication density. The sub-task clusters are then mapped sequentially according to their resource requirements in descending order.

14. A task scheduling device, characterized in that, The device includes: The task matrix construction module is used to construct an inter-task communication matrix based on the task communication volume between every two subtasks in the multiple subtasks to be scheduled, wherein the multiple subtasks to be scheduled are obtained by dividing the deep learning distributed tasks to be scheduled. The network matrix construction module is used to construct a network affinity matrix based on the bandwidth and congestion indicators between every two task processing nodes in a plurality of task processing nodes to be scheduled. The clustering module is used to cluster the multiple subtasks based on the inter-task communication matrix to obtain a set of task clusters, and to cluster the multiple task processing nodes based on the network affinity matrix to obtain a set of node clusters. Each task cluster in the set of task clusters includes at least one subtask, and each node cluster in the set of node clusters includes at least one task processing node. The set mapping module is used to map the task cluster set and the node cluster set according to the resource requirement information of each subtask and the resource supply information of each task processing node, and determine the mapping result between the multiple subtasks and the multiple task processing nodes.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the task scheduling method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the task scheduling method according to any one of claims 1 to 13.

Images