Node deployment method and scheduler

Abstract

The invention provides a node deployment method and a scheduler, and the method comprises the steps: responding to a received large model training task, and determining schedulable nodes meeting a preset condition from a communication node set; acquiring a communication frequency between the schedulable node and at least one other node; based on the communication frequency and performance parameters of the schedulable nodes, determining a target node set, the target node set comprising at least one target deployment node; and deploying at least one task process in the large model training task on the target deployment node to perform the large model training task.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a node deployment method and a scheduler. Background Art

[0002] Currently, large-scale model training is usually carried out in a communication cluster, which includes multiple communication nodes. In the node deployment scheme for training tasks, it is often impossible to achieve an optimal match between nodes and training tasks, resulting in low training efficiency and waste of node resources. Summary of the invention

[0003] In view of this, the present disclosure provides a node deployment method and a scheduler.

[0004] One aspect of the present disclosure provides a node deployment method, including: in response to receiving a large model training task, determining a schedulable node that meets preset conditions from a communication node set; obtaining the communication frequency between the schedulable node and at least one other node; based on the communication frequency and the performance parameters of the schedulable node itself, determining a target node set, the target node set including at least one target deployment node; and deploying at least one task process in the large model training task on the target deployment node to perform the large model training task.

[0005] According to an embodiment of the present disclosure, the preset conditions include at least one of the following: the remaining resource amount of the communication nodes in the communication node set meets the task process requirements; the communication node is a non-faulty node; the number of communication nodes is greater than or equal to the number of task processes.

[0006] According to an embodiment of the present disclosure, non-faulty nodes are determined in the following manner: in response to a communication failure in a communication node set, the communication nodes in the communication node set are randomly grouped and a communication test is performed; based on the result of the communication test, a faulty node set and at least one non-faulty node are determined, the non-faulty nodes communicate normally with each other, and the nodes in the faulty node set are removed from the communication cluster.

[0007] According to an embodiment of the present disclosure, determining a target node set based on the communication frequency and the performance parameters of the schedulable node itself includes: calculating a first weight value of the communication frequency and a second weight value corresponding to the performance parameter; determining a communication weight value of the schedulable node based on the first weight value and the second weight value; and determining a target node set based on the communication weight value of the schedulable node.

[0008] According to an embodiment of the present disclosure, calculating the first weight value of the communication frequency includes: determining the total communication volume according to the communication frequency and the single communication volume of the schedulable node; and determining the first weight value according to the size of the total communication volume.

[0009] According to an embodiment of the present disclosure, obtaining the communication frequency between a schedulable node and at least one other node includes: determining a preset number of adjacent nodes connected to the schedulable node; obtaining the communication frequency between the schedulable node and the other nodes based on at least one communication method, the communication methods including data parallelism, pipeline parallelism, and tensor parallelism.

[0010] According to an embodiment of the present disclosure, calculating the second weight value corresponding to the performance parameter includes: obtaining operation information and fault information of the schedulable node, the operation information includes at least one of operation duration, data calculation speed, and service quality, and the fault information includes at least one of packet loss rate and fault rate; determining the performance parameters of the schedulable node itself based on the operation information and fault information; determining the second weight value according to the performance parameters and the bandwidth value between the schedulable node and the connected nodes.

[0011] According to an embodiment of the present disclosure, deploying at least one task process in a large model training task on a target deployment node includes: in response to receiving a large model training task, obtaining the transmission data volume of at least one task process; and determining a target deployment node that matches each task process based on the transmission data volume.

[0012] According to an embodiment of the present disclosure, determining the target node set includes: in response to the existence of multiple communication node sets, determining at least one deployment node set based on the multiple communication node sets, each deployment node set including different nodes; and determining the target node set based on the total communication bandwidth of the deployment node sets.

[0013] Another aspect of the present disclosure also provides a scheduler, including: one or more processors; a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the following method: in response to receiving a large model training task, determine a schedulable node that meets preset conditions from a communication node set; obtain the communication frequency between the schedulable node and at least one other node; based on the communication frequency and the performance parameters of the schedulable node itself, determine a target node set, the target node set including at least one target deployment node; deploy at least one task process in the large model training task on the target deployment node to perform the large model training task.

[0014] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the above method.

[0015] Another aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario of the node deployment method applicable to the embodiment of the present disclosure is schematically shown;

[0019] Figure 2 A flowchart of a node deployment method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A flowchart of a node deployment method according to another embodiment of the present disclosure is schematically shown;

[0021] Figure 4 A flowchart for determining a non-faulty node according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 5 A schematic diagram of determining a non-faulty node according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 6 A schematic diagram schematically illustrates a node deployment method according to an embodiment of the present disclosure; and

[0024] Figure 7 A schematic block diagram of a scheduler that can be used to implement the node deployment method of an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0027] Figure 1The application scenario of the node deployment method applicable to the embodiment of the present disclosure is schematically illustrated.

[0028] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a communication cluster 110, and the communication cluster 110 includes multiple communication nodes (Node1, Node 2, ..., Noden). Each communication node in the communication cluster 110 may be an electronic device having at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor, and a neural network processing unit (NPU), and the communication node can perform model training tasks.

[0029] With the rapid growth of language model parameters and the amount of training data required, the limited resources on a single device cannot meet the requirements of large language model training. Through distributed training, the model training task can be divided into multiple task processes, and the task processes can be distributed to the above multiple communication nodes to execute the training tasks in parallel, thereby completing the large model training task faster.

[0030] The user can perform large model training tasks through at least one communication node in the communication cluster 110. Since each communication node in the communication cluster 110 has different communication quality, data processing capability and other performance, when deploying the task process, a communication node with better performance (such as Node1, Node 2 and Node5) can be selected to deploy the task process, and the node with better performance is used to execute the task with larger data processing volume.

[0031] It should be understood that Figure 1 The number of communication clusters and communication nodes in the embodiment is only for illustration. Any number of communication clusters and communication nodes may be provided according to implementation requirements.

[0032] Figure 2 The flowchart of the node deployment method according to the embodiment of the present disclosure is schematically shown.

[0033] like Figure 2 As shown, the node deployment method of this embodiment includes operations S210 to S240.

[0034] In operation S210, in response to receiving a large model training task, a schedulable node that meets a preset condition is determined from a communication node set.

[0035] In an embodiment of the present disclosure, a large model training task may refer to a machine learning or deep learning model training task based on distributed training. The large model training task may be divided into multiple subtasks, each of which can be trained in parallel on a communication node as an independent task process. A communication node set refers to a set consisting of nodes currently set in a communication cluster. Meeting a preset condition may mean that the performance of a node meets the requirements of a large model training task, or that communication between multiple nodes is normal. A schedulable node refers to a node that has the ability to execute a large model training task.

[0036] For example, communication cluster A includes a communication node set, which has 20 nodes. After screening, 8 nodes that meet the preset conditions can be obtained. These 8 nodes are schedulable nodes and have the ability to perform large model training tasks.

[0037] In operation S220, a communication frequency between the schedulable node and at least one other node is obtained.

[0038] In the embodiments of the present disclosure, the communication frequency refers to the number of successful communications between nodes in a unit time, and the communication frequency can characterize the communication efficiency and stability between two nodes. In a communication cluster, each node has at least one communication link and can communicate with other connected nodes based on the communication link.

[0039] For example, the schedulable node A has communication links with three other nodes, and the communication frequency between the schedulable node and n other nodes can be obtained, where n is an integer less than or equal to 3.

[0040] In operation S230, a target node set is determined based on the communication frequency and the performance parameters of the schedulable node itself, where the target node set includes at least one target deployment node.

[0041] In the embodiments of the present disclosure, the performance parameters can be used to measure the data processing capability, resource allocation and other indicators of the schedulable nodes themselves. The target node set refers to the node set used to deploy the large model training task, which is composed of at least one target deployment node. The target deployment node refers to the node selected from the schedulable nodes.

[0042] In operation S240, at least one task process in the large model training task is deployed on the target deployment node to perform the large model training task.

[0043] For example, a large model training task includes four task processes. According to the performance parameters and communication frequency of the schedulable nodes, the target node set is determined from the eight schedulable nodes. The target node set includes four target deployment nodes. The task processes are deployed on the target deployment nodes respectively to complete the training task of the large model.

[0044] According to the embodiments of the present disclosure, nodes capable of running training tasks are determined from the communication cluster, and further screening is performed based on the performance and communication capabilities of the nodes to obtain deployment nodes. The present disclosure can provide users with a scheduling and deployment solution with better communication performance, while improving training efficiency and maximizing the utilization of node resources.

[0045] Figure 3 A flowchart of a node deployment method according to another embodiment of the present disclosure is schematically shown.

[0046] like Figure 3 As shown, the node deployment method of this embodiment includes operations S301 to S311.

[0047] In operation S301, in response to receiving a large model training task, a schedulable node that meets a preset condition is determined from a communication node set.

[0048] In an embodiment of the present disclosure, a target deployment node for deploying a large model training task can be determined based on a first weight value and a second weight value of the schedulable node, wherein the first weight value of the schedulable node is obtained based on operations S302-S305, and the second weight value of the schedulable node is obtained based on operations S306-S308.

[0049] In operation S302, a preset number of neighboring nodes connected to the schedulable node is determined.

[0050] In the embodiments of the present disclosure, an adjacent node refers to a node that has a communication link with a schedulable node. Since there are communication links between schedulable nodes, each schedulable node has many adjacent nodes. In a large model training task, the communication objects of the node after the task process is deployed are limited, and data will not be transmitted with all other schedulable nodes. Therefore, the number of adjacent nodes of the schedulable node can be preset, and nodes with a higher number of communications with the schedulable node and a longer running time are preferentially selected.

[0051] For example, there are 5 schedulable nodes, and the preset number of adjacent nodes is 3. For one of the schedulable nodes A, the nodes with longer running time among the other schedulable nodes are schedulable node B, schedulable node E and schedulable node F, and these three nodes are taken as adjacent nodes of schedulable node A.

[0052] In operation S303, the communication frequency between the schedulable node and other nodes based on at least one communication mode is obtained, and the communication modes include data parallelism, pipeline parallelism, and tensor parallelism.

[0053] In the embodiments of the present disclosure, large model training tasks can be divided into data parallel (DP), pipeline parallel (PP), tensor parallel (TP) and other training methods. Correspondingly, when executing task processes in different training methods on nodes, the communication methods of nodes can also include data parallelism, pipeline parallelism, tensor parallelism, etc. Based on different communication methods for data communication, the communication frequency between nodes is also different.

[0054] In operation S304, the total communication volume is determined according to the communication frequency and the single communication volume of the schedulable nodes.

[0055] In the embodiments of the present disclosure, the single communication volume of the schedulable node refers to the maximum amount of data that can be transmitted in one communication between the schedulable nodes. The total communication volume may refer to the total amount of data that can be transmitted between the schedulable nodes in a unit time. The higher the total communication volume determined based on the communication frequency and the single communication volume, the stronger the communication capability of the schedulable node.

[0056] According to an embodiment of the present disclosure, the total communication volume of the schedulable nodes can be obtained by formula (1):

[0057] (1)

[0059] Among them, Wij represents the total communication volume between node i and node j, freq(TP,i,j), freq(DP,i,j), and freq(PP,i,j) represent the communication frequencies between node i and node j based on TP, DP, and PP respectively, and Data(TP,i,j), Data(DP,i,j), and Data(PP,i,j) represent the single communication volume when communicating based on different communication modes respectively.

[0060] In operation S305, a first weight value is determined according to the size of the total communication volume.

[0061] In the embodiment of the present disclosure, the first weight value represents the proportion of the communication capability in evaluating the schedulable node. A first weight corresponding to the total communication volume can be preset, and the total communication volume is multiplied by the first weight to obtain the first weight value of the schedulable node.

[0062] In operation S306, operation information and fault information of the schedulable node are obtained, where the operation information includes at least one of operation duration, data calculation speed, and service quality, and the fault information includes at least one of packet loss rate and fault rate.

[0063] In an embodiment of the present disclosure, the operation information includes information related to the node operation quality, for example, it may include operation time, data calculation speed, and service quality. The operation time refers to the continuous operation time of the schedulable node in the communication cluster, the data calculation speed may indicate the speed of data calculation inside the schedulable node, and the service quality refers to the service capability provided by the schedulable node for network communication. The fault information includes information related to the node communication quality, for example, it may include packet loss rate and failure rate. The packet loss rate refers to the ratio of the data packets lost by the schedulable node to the data packets sent in the communication test. The failure rate refers to the ratio of the number of failures in communication to the total number of communications.

[0064] In operation S307, performance parameters of the schedulable node itself are determined based on the operation information and the fault information.

[0065] In the embodiments of the present disclosure, the performance parameters characterize the performance of the schedulable nodes. The performance parameters of the schedulable nodes can be determined by weight calculation, scoring, and the like.

[0066] For example, at least one operating information and / or at least one fault information of a schedulable node can be scored, and a score threshold can be preset. When the corresponding score is greater than or equal to the score threshold, one point is obtained, and the performance parameters are determined based on the scores of each information of the schedulable node.

[0067] For example, at least one operation information and / or at least one fault information of the schedulable node may be scored, and weights of different information may be preset, and performance parameters of the schedulable node may be obtained based on the weight calculation.

[0068] In operation S308, a second weight value is determined according to the performance parameter and the bandwidth values ​​of the schedulable node and the connected node.

[0069] In the embodiment of the present disclosure, the second weight value represents the proportion of the data processing capability of the node in the evaluation of the schedulable node. The second weight corresponding to the performance parameter can be preset, and the performance parameter, the bandwidth value of the connected node and the second weight are multiplied to obtain the second weight value of the schedulable node.

[0070] In operation S309 , a communication weight value of the schedulable node is determined based on the first weight value and the second weight value.

[0071] According to the embodiments of the present disclosure, the communication weight value is used to characterize the communication capability of the schedulable node, and the communication capability of the node can be quantified through the communication weight value. Through the first weight value and the second weight value, the communication capability of the node can be evaluated from multiple dimensions such as transmission capability and computing capability, so as to select a better node to deploy the training task and maximize the utilization of resources.

[0072] In operation S310, a target node set is determined according to the communication weight values ​​of the schedulable nodes, and the target node set includes at least one target deployment node.

[0073] In operation S311, at least one task process in the large model training task is deployed on the target deployment node to perform the large model training task.

[0074] According to an embodiment of the present disclosure, the preset conditions include at least one of the following: the remaining resource amount of the communication nodes in the communication node set meets the task process requirements; the communication node is a non-faulty node; the number of communication nodes is greater than or equal to the number of task processes.

[0075] In one example, the preset condition may be set as the remaining amount of resources of the communication nodes in the communication node set satisfying the task process requirement.

[0076] In an embodiment of the present disclosure, the communication node set is composed of a plurality of communication nodes. The remaining resource amount may refer to the number of CPU cores, GPU cards, and memory remaining on the communication node. Meeting the task process requirements means that the remaining resource amount of the communication node is greater than the resource amount required by the task process.

[0077] For example, satisfying the task process requirement may mean that the number of CPU cores required for the task process is less than the number of CPU cores remaining on the communication node.

[0078] For example, satisfying the task process requirement may also mean that the number of GPU cards required for the task process is less than the number of GPU cards remaining on the communication node.

[0079] For example, satisfying the task process demand may also mean that the amount of memory required by the task process is less than the amount of memory remaining on the communication node.

[0080] In an embodiment of the present disclosure, determining the communication node that meets the task process requirements can be implemented by formula (2):

[0081]

[0082] (2)

[0083] Among them, TASK_CAPACITY_PER_NODE refers to the number of task processes that can be run on each communication node. CPU_PER_TASK, GPU_PER_TASK, and MEMORY_PER_TASK respectively represent the number of CPU cores, number of GPUs, and amount of memory required for each task process. CPU_PER_NODE, GPU_PER_NODE, and MEMORY_PER_NODE respectively represent the number of CPU cores, number of GPU cards, and amount of memory remaining on the node. When TASK_CAPACITY_PER_NODE is greater than or equal to 1, it means that the number of CPU cores, number of GPU cards, and amount of memory remaining on the communication node calculated by the formula meet the task process requirements, and the communication node meets the preset conditions.

[0084] In another example, the preset condition may be set as the number of communication nodes being greater than or equal to the number of task processes.

[0085] For example, a large model training task includes 5 task processes and the number of communication nodes is 4, which means that the communication nodes in the communication node set cannot meet the preset conditions.

[0086] For example, a large model training task includes three task processes and the number of communication nodes is four. Then, a schedulable node that meets the preset conditions can be determined from the communication nodes in the communication node set.

[0087] According to the embodiments of the present disclosure, based on the training method of the training task and the resource requirements of each task process, it is determined which nodes in the communication node set can be used to deploy the task process, thereby further screening the communication nodes and obtaining a better communication deployment plan.

[0088] In another example, the preset condition may be set as the communication node being a non-faulty node.

[0089] In the embodiment of the present disclosure, a non-faulty node may refer to a node that can perform data processing normally or has a low failure rate, or may refer to a node whose communication link with other nodes has no abnormalities.

[0090] Next, combine Figure 4-Figure 5 The determination of non-faulty nodes is further explained.

[0091] Figure 4 A flowchart for determining a non-faulty node according to an embodiment of the present disclosure is schematically shown.

[0092] like Figure 4 As shown, the node deployment method of this embodiment includes operations S410 to S440.

[0093] In operation S410, in response to a communication failure in a communication node set, the communication nodes in the communication node set are randomly grouped and a communication test is performed.

[0094] In the embodiments of the present disclosure, communication failures may include node failures, network failures, communication link failures, etc. Among them, node failures and communication link failures will directly affect the communication between nodes, while network failures can restore the communication between nodes after replacing the network. Therefore, when there is a communication failure in the communication node set, it is necessary to detect and remove the nodes with self-failures and communication link failures, and retain the nodes with normal communication and the nodes whose communication is affected by the network failure. Communication testing refers to a test that can detect whether each node can communicate normally.

[0095] In operation S420, a set of faulty nodes and at least one non-faulty node are determined based on the result of the communication test, and the non-faulty nodes communicate normally with each other.

[0096] In an embodiment of the present disclosure, the communication nodes in the communication node set can be numbered and randomly divided into two groups. The all-gather set communication test is performed on the two groups of communication nodes respectively. If the communication nodes in the group can complete the communication test, it means that the group is a healthy node set. If the communication test of the communication node in the group fails, it means that there is at least one faulty node in the group, and the nodes in the group are further randomly divided into two groups for testing until only one node is left in the group, and the communication node is determined to be a potential faulty node.

[0097] In the embodiment of the present disclosure, multiple healthy sets may be merged, and after the merging process, the nodes in the healthy set with the largest number of nodes are used as non-faulty nodes. The non-faulty nodes themselves and the communication links between the nodes are normal.

[0098] For example, a communication node is extracted from each of the two healthy sets, and a gather communication test is performed between the two communication nodes. If the communication test can be completed, the two healthy sets can be merged. If the communication test fails, it means that the two healthy sets cannot communicate and cannot be merged.

[0099] In the embodiments of the present disclosure, a potential faulty node may be affected by a network fault but the node itself has no problem. In this case, a gather communication test may be performed between the potential faulty node and the non-faulty node. If the communication test can be completed, the potential faulty node may be treated as a non-faulty node. If the communication test fails, the node is a faulty node. After the test is completed, all faulty nodes are added to the faulty node set.

[0100] In operation S430, non-faulty nodes are regarded as schedulable nodes that meet preset conditions.

[0101] In operation S440 , the nodes in the set of failed nodes are removed from the communication cluster.

[0102] According to the embodiments of the present disclosure, by performing communication tests on communication nodes in groups, the type and scope of communication anomalies can be accurately located. Since the composition of communication nodes in each round of the set to be tested is random, communication anomalies caused by communication links or nodes themselves can be effectively detected, and the impact of network failures on communication nodes can be eliminated, avoiding the inability to resume training due to communication anomalies during redeployment after troubleshooting.

[0103] Figure 5 A schematic diagram of determining a non-faulty node according to an embodiment of the present disclosure is schematically shown.

[0104] like Figure 5 As shown, when a communication failure occurs in the communication node set, the communication node set is taken as the potential fault set 1 P1, and the communication nodes in P1 are randomly grouped and a communication test is performed. The communication test of one group fails, and the communication test of the other group succeeds. The group with the failed communication test result is taken as the potential fault set 2 P2, and the grouping is continued and the communication test is performed, and the group with the successful communication test result is taken as the healthy node set 1 H1.

[0105] Next, the communication nodes in P2 are randomly grouped and tested for communication. The communication test results of both groups are failures, and potential fault set 3 P3 and potential fault set 4 P4 are obtained respectively. The communication nodes in P3 and P4 are randomly grouped and tested for communication, and potential fault set 5 P5, potential fault set 6 P6, potential fault set 7 P7 and healthy node set 2 H2 are obtained. The communication nodes in P5, P6, and P7 are randomly grouped and tested for communication, and potential fault nodes E1, E2, E3, E4, E5 and healthy node set 3 H3 are obtained respectively.

[0106] The healthy node set 1 H1, the healthy node set 2 H2 and the healthy node set 3 H3 are merged, and the nodes in the healthy node set with the largest number of nodes are regarded as non-faulty nodes. A node is selected from the non-faulty nodes to perform a communication test with the potential faulty node, and the potential faulty node with a successful communication test result is regarded as a non-faulty node, and the potential faulty node with a failed communication test result is regarded as a faulty node and removed from the communication cluster.

[0107] According to an embodiment of the present disclosure, deploying at least one task process in a large model training task on a target deployment node includes: in response to receiving a large model training task, obtaining the transmission data volume of at least one task process; and determining a target deployment node that matches each task process based on the transmission data volume.

[0108] In the embodiments of the present disclosure, the amount of data transmission required for each task process in the large model training task is different, and the amount of data transmission depends on factors such as the structure of the training model, training parameters, and training strategies. The amount of data transmission may refer to the sum of the amount of data communication between the target deployment node and other nodes when the task process is deployed on the target deployment node. The target deployment node matched by each task process may be determined based on the amount of data transmission.

[0109] For example, a large model training task includes task process A and task process B. According to the amount of data transmitted, task process A with the largest amount of data transmitted is deployed on the target deployment node with the highest communication weight value, and task process B is deployed on the target deployment node with the second highest communication weight value.

[0110] Next, combine Figure 6 The node deployment method of this embodiment is further explained.

[0111] Figure 6 A schematic diagram of a node deployment method according to an embodiment of the present disclosure is schematically shown.

[0112] like Figure 6 As shown, the large model training task includes four task processes, namely task process 511, task process 512, task process 513 and task process 514. The target deployment nodes are node 501, node 502, node 503, node 504 and node 505 respectively.

[0113] By calculating the data transmission volume of each task process, it is determined that the order of data transmission volume from large to small is task process 511>task process 513>task process 512>task process 514. The order of communication weight values ​​from large to small is node 501>node 503>node 505>node 502>node 504. Therefore, task process 511 is deployed on node 501, task process 512 is deployed on node 505, task process 513 is deployed on node 503, and task process 514 is deployed on node 502.

[0114] According to an embodiment of the present disclosure, determining the target node set includes: in response to the existence of multiple communication node sets, determining at least one deployment node set based on the multiple communication node sets, each deployment node set including different nodes; and determining the target node set based on the total communication bandwidth of the deployment node sets.

[0115] In an embodiment of the present disclosure, there may be multiple communication node sets in a communication cluster. When there are schedulable nodes that meet preset conditions in the communication node set, a deployment node set may be determined from the schedulable nodes. The total communication bandwidth refers to the sum of the link bandwidths of all nodes in the deployment node set.

[0116] For example, the deployment node set includes node A, node B and node C. Node A is connected to node B and node C respectively, the bandwidth value between AB is 5, the bandwidth value between AC is 4, node B is connected to node C, the bandwidth value between BC is 7, and the total communication bandwidth of the deployment node set can be determined to be 16.

[0117] In the embodiment of the present disclosure, the larger the total communication bandwidth, the stronger the communication capability of the deployment node set. When there is a deployment node set, a bandwidth threshold can be set. When the total communication bandwidth of the deployment node set is greater than the bandwidth threshold, the deployment node set is used as the target node set to deploy the task process. When there are multiple deployment node sets, the total communication bandwidth of each deployment node set is calculated respectively, and the deployment node set with the largest total communication bandwidth is used as the target node set to deploy the task process.

[0118] According to the embodiments of the present disclosure, a task deployment scheme is determined from a plurality of communication node sets, and the best scheme is selected for deployment, which can improve the training efficiency of the model and maximize the utilization of node resources.

[0119] Figure 7 The schematic block diagram of the scheduler that can be used to implement the node deployment method of the embodiment of the present disclosure is schematically shown. Figure 7 As shown, the scheduler 700 according to the embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an on-board memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0120] In RAM 703, various programs and data required for the operation of the scheduler 700 are stored. The processor 701, ROM 702 and RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. The processor 701 can also execute the following method by executing the program stored in the one or more memories: in response to receiving a large model training task, determine a schedulable node that meets the preset conditions from a communication node set; obtain the communication frequency between the schedulable node and at least one other node; based on the communication frequency and the performance parameters of the schedulable node itself, determine the target node set, the target node set includes at least one target deployment node; deploy at least one task process in the large model training task on the target deployment node to perform the large model training task.

[0121] According to an embodiment of the present disclosure, the scheduler 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The scheduler 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that the computer program read therefrom is installed into the storage portion 708 as needed.

[0122] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0123] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0124] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0125] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0126] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0127] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0128] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0129] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0130] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or couplings fall within the scope of the present disclosure.

[0132] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A node deployment method, comprising: In response to receiving the large model training task, determining a schedulable node that meets a preset condition from a communication node set; Acquire a communication frequency between the schedulable node and at least one other node; Determine a target node set based on the communication frequency and the performance parameters of the schedulable node itself, wherein the target node set includes at least one target deployment node; At least one task process in the large model training task is deployed on the target deployment node to perform the large model training task.

2. According to the method of claim 1, the preset condition includes at least one of the following: The remaining resources of the communication nodes in the communication node set meet the task process requirements; The communication node is a non-faulty node; The number of the communication nodes is greater than or equal to the number of the task processes.

3. The method according to claim 2, wherein the non-faulty node is determined by: In response to a communication failure in the communication node set, randomly grouping the communication nodes in the communication node set and performing a communication test; A set of faulty nodes and at least one non-faulty node are determined based on the result of the communication test, the non-faulty nodes communicate normally with each other, and the nodes in the set of faulty nodes are removed from the communication cluster.

4. According to the method of claim 1, the determining the target node set based on the communication frequency and the performance parameters of the schedulable node itself comprises: Calculating a first weight value of the communication frequency and a second weight value corresponding to the performance parameter; Determine a communication weight value of the schedulable node based on the first weight value and the second weight value; The target node set is determined according to the communication weight values ​​of the schedulable nodes.

5. According to the method of claim 4, the calculating the first weight value of the communication frequency comprises: Determine the total communication volume according to the communication frequency and the single communication volume of the schedulable node; The first weight value is determined according to the size of the total communication volume.

6. According to the method of claim 5, the obtaining of the communication frequency between the schedulable node and at least one other node comprises: Determining a preset number of adjacent nodes connected to the schedulable node; The communication frequency between the schedulable node and other nodes based on at least one communication mode is obtained, and the communication mode includes data parallelism, pipeline parallelism, and tensor parallelism.

7. The method according to claim 4, wherein the calculating the second weight value corresponding to the performance parameter comprises: Acquire operation information and fault information of the schedulable node, wherein the operation information includes at least one of operation duration, data calculation speed, and service quality, and the fault information includes at least one of packet loss rate and fault rate; Determine the performance parameters of the schedulable node itself based on the operation information and the fault information; The second weight value is determined according to the performance parameter and the bandwidth value of the schedulable node and the connected node.

8. According to the method of claim 1, the deploying at least one task process in the large model training task on the target deployment node comprises: In response to receiving the large model training task, obtaining the amount of data transmitted by the at least one task process; A target deployment node matching each task process is determined according to the amount of transmitted data.

9. The method according to claim 1, wherein determining the target node set comprises: In response to the existence of a plurality of communication node sets, determining at least one deployment node set according to the plurality of communication node sets, each deployment node set including different nodes; The target node set is determined according to the total communication bandwidth of the deployment node set.

10. A scheduler, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the following method: In response to receiving the large model training task, determining a schedulable node that meets a preset condition from a communication node set; Acquire a communication frequency between the schedulable node and at least one other node; Determine a target node set based on the communication frequency and the performance parameters of the schedulable node itself, wherein the target node set includes at least one target deployment node; At least one task process in the large model training task is deployed on the target deployment node to perform the large model training task.

Images