Distributed computing system and method for distributed computing system

Abstract

The invention provides a distributed computing system and a method for the distributed computing system.The system comprises a task manager, a resource scheduler and a computing framework, and the task manager sends parallel computing task information to the resource scheduler; the resource scheduler allocates computing equipment in the available computing nodes to each computing task in the parallel computing tasks according to the parallel computing task information and the topological information of the available computing nodes, and sends the information of the allocated computing equipment to the task manager; the task manager sends the corresponding relation between each calculation task and the calculation equipment to the calculation framework; and the computing framework distributes the computing tasks to the corresponding computing devices. Therefore, the calculation tasks of dense communication in the parallel calculation tasks can be distributed to the calculation equipment with the closer topological distance, the time delay of flow communication is reduced, and the calculation performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing, and in particular to a distributed computing system and a method for the distributed computing system. BACKGROUND

[0002] The computing process of machine learning is usually large in model volume, and the storage space of a single computing device is limited, so the model cannot be placed in a computing device to complete the computation. Meanwhile, in order to shorten the computation time, multiple computing nodes can be used for parallel distributed computation.

[0003] The computing node resources are usually uniformly managed by a public cloud or a private cloud platform. Before submitting a computing task for execution, a user submits a resource application to the cloud platform, and the cloud platform allocates computing node resources to the user according to the resource requirements of the application and the idle resources in the cloud, by an internal scheduler component. The user starts the computing task on the allocated computing node.

[0004] During the computation, each node works according to multiple computing modes such as tensor parallelism, pipeline parallelism, and data parallelism. After each node completes a part of the computation, it needs to communicate with a group of nodes defined by the computing mode, until the computation is completed.

[0005] The scheduler of the existing cloud platform can support multiple scheduling strategies, network topology affinity / anti-affinity scheduling, and cabinet affinity / anti-affinity scheduling. The selection of the scheduling strategy is usually based on performance, reliability, and other factors.

[0006] It should be noted that the above introduction to the technical background is only for the convenience of clearly and completely describing the technical solutions of the present application, and for the convenience of understanding by those skilled in the art. The above technical solutions cannot be considered as known to those skilled in the art just because they are described in the background section of the present application. SUMMARY

[0007] The existing distributed computing framework numbers each computing device (for example, GPU) according to the list of computing node addresses provided by the user, and allocates tasks to each computing device according to the corresponding parallel computing mode of the machine learning model. Taking a large-scale language model computing framework (hereinafter referred to as the computing framework) as an example, for computing resources of 8 machines and 16 cards (for example, 8 servers including 16 computing devices, and the server is also referred to as a computing node or a node, and the computing device is also referred to as an acceleration card or a card), the 16 computing devices of nodes H0-H7 are numbered in turn as:

[0008] [g0,g1],[g2,g3],[g4,g5],[g6,g7],[g8,g9],[g10,g11],[g12,g13],[g14,g15]

[0009] Figure 1 is an illustration of resource allocation of distributed parallel computing. As shown, when the computation mode of the computation task is set to tensor parallelism with a degree of 2 and pipeline parallelism with a degree of 4, the computation framework allocates the following to each computation device: Figure 1

[0010] Tensor Parallism (TP): Each layer of the model is split horizontally or vertically, and the split tensors are allocated to different computation devices. The computation devices are divided into 8 TPs:

[0011] [g0, g1], [g2, g3], [g4, g5], [g6, g7], [g8, g9], [g10, g11], [g12, g13], [g14, g15]

[0012] Pipeline Parallism (PP): One or more layers of the model are allocated to different computation devices. The computation devices are divided into 4 PPs:

[0013] [g0, g4, g8, g12], [g1, g5, g9, g13], [g2, g6, g10, g14], [g3, g7, g11, g15]

[0014] Data Parallism (DP): After the above division, the computation devices that maintain the same model part are divided into 8 DPs:

[0015] [g0, g2], [g1, g3], [g4, g6], [g5, g7], [g8, g10], [g9, g11], [g12, g14], [g13, g15]

[0016] All Reduce collective communication is required between members in the tensor parallel group (e.g., computation devices g0 and g1) and between members in the data parallel group (e.g., computation devices g0 and g2). Point-to-point communication is required between members in the pipeline parallel group (e.g., computation devices g0 and g2). The communication volume in these parallel groups, from large to small, is usually tensor parallelism, data parallelism, and pipeline parallelism.

[0017] ​The communication among the members of the tensor parallel group can usually be completed in the machine, for example, the computing devices g0 and g1 are distributed in the same node H0, and the All Reduce set communication between the two is performed in the node H0. The communication among the members of the data parallel group and the pipeline parallel group usually needs to be forwarded across nodes through the network device (the backbone switch Spine0-access switches Leaf0, Leaf1 and Leaf2), for example, the computing devices g0 and g2 are distributed in different nodes H0 and H1 under the same access switch Leaf0, and the All Reduce set communication or point-to-point communication between the two needs to exchange data through the access switch Leaf0.

[0018] The inventors of the present application find that the existing computing framework receives a series of computing node addresses input by a user as available computing resources, numbers and task divides the available computing devices according to the parallel computing mode set by the user. However, the network connection topology of each computing device / computing node used by the task is not considered. Within a given range of computing nodes, nodes with intensive communication needs may need to communicate across multiple switches, resulting in increased latency and reduced computing performance.

[0019] Figure 2 is another schematic diagram of resource allocation of distributed parallel computing. As shown in Figure 2 Leaf0, Leaf1 and Leaf2, and the computing devices under the servers H0-H23 are numbered as g0-g48. When performing an 8-machine 16-card computing task, if the allocated 8 servers are [H0-H3, H8-H11], the data parallel groups are divided into [g0, g2], [g1, g3], [g4, g6], [g5, g7], [g16, g18], [g17, g19], [g20, g22], [g21, g23], so that the computing tasks in the same data parallel group are allocated to the servers connected to the same access switch Leaf0 or Leaf1; if the allocated 8 servers are [H0-H2, H5-H6, H9-H11], the data parallel groups are divided into [g0, g2], [g1, g3], [g4, g10], [g5, g11], [g12, g18], [g13, g19], [g20, g22], [g21, g23], resulting in the computing tasks in the data parallel groups [g12, g18], [g13, g19] being allocated to the servers H6 and H9 connected to different access switches Leaf0 and Leaf1, which needs to communicate across nodes, i.e., across two access switches Leaf0, Leaf1 and one backbone switch Spine0. It can be seen that the former is more friendly to the cross-node communication of computing tasks, and can reduce the amount of data forwarding through the backbone switch Spine0.

[0020] In addition, for the allocation of computing devices in a data parallel group, taking the allocated 8 servers [H0~H3, H8~H11] as an example, compared with the allocation [g0, g16] in a data parallel group, the configuration [g0, g2] can effectively reduce the number of switches through which data passes, because the configuration [g0, g16] needs to cross two access switches Leaf0, Leaf1 and one backbone switch Spine0, while the configuration [g0, g2] only needs to cross one access switch Leaf0.

[0021] However, the existing cloud resource scheduler and computing framework do not consider the parallel computing mode and communication demand of the computing task, and lack the function of allocating computing devices based on the network topology of the computing resources, so that the allocated computing nodes may be scattered under various switches of the network cluster, and in the data traffic generated by the distributed computing task, the proportion of data forwarding across multiple switches is high, resulting in increased communication delay and reduced computing performance.

[0022] To solve at least one of the above problems or other similar problems, embodiments of the present application provide a distributed computing system and a method for the distributed computing system. In the system, a resource scheduler groups parallel computing tasks according to parallel computing task information, and selects corresponding computing resources in combination with topology information of available computing resources, so that the computing tasks with intensive communication in the parallel computing tasks can be allocated to computing devices with close topology, thereby reducing the delay of traffic communication and improving the computing performance.

[0023] According to an aspect of an embodiment of the present application, a distributed computing system is provided, comprising a task manager, a resource scheduler and a computing framework, wherein,

[0024] The task manager sends parallel computing task information to the resource scheduler;

[0025] The resource scheduler allocates computing devices in the available computing nodes to each computing task in the parallel computing tasks according to the parallel computing task information and the topology information of the available computing nodes, and sends the information of the allocated computing devices to the task manager;

[0026] The task manager sends the correspondence between each computing task and the computing device to the computing framework;

[0027] The computing framework distributes each computing task to the corresponding computing device.

[0028] According to another aspect of an embodiment of the present application, a method for a distributed computing system is provided, the distributed computing system comprising a task manager, a resource scheduler and a computing framework,

[0029] The method comprises:

[0030] The task manager sends parallel computing task information to the resource scheduler;

[0031] The resource scheduler allocates computing devices in the available computing nodes to each computing task in the parallel computing task according to the parallel computing task information and topology information of the available computing nodes, and sends information of the allocated computing devices to the task manager;

[0032] The task manager sends the correspondence between each computing task and the computing device to the computing framework; and

[0033] The computing framework distributes each computing task to the corresponding computing device.

[0034] One of the beneficial effects of the embodiments of the present application includes: through the cooperative work of the resource scheduler and the computing framework, the corresponding computing resources are selected based on the available resource topology relationship of the cluster and the parallel computing splitting mode, the computing tasks with intensive communication in the parallel computing task can be allocated to the computing devices with closer topology distance, the time delay of traffic communication is reduced, and the computing performance is improved.

[0035] Specific embodiments of the present application are disclosed in detail in the following description and accompanying drawings, indicating the ways in which the principles of the present application can be employed. It should be understood that the embodiments of the present application are not limited in scope in terms of the appended claims and their equivalents. Embodiments of the present application include many alterations, modifications and equivalents within the scope of the appended claims and their equivalents.

[0036] Features described and / or illustrated with respect to one embodiment can be used in the same or similar way in one or more other embodiments, in combination with or in place of features in other embodiments.

[0037] It should be emphasized that the term "comprises / comprising" as used herein indicates the presence of the stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description, the drawings included are used to provide further understanding of the embodiments of the present application, and constitute a part of the specification, used to explain the embodiments of the present application, and together with the text description to explain the principles of the present application. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0039] Figure 1 is a schematic diagram of resource allocation of distributed parallel computing;

[0040] Figure 2 is another schematic diagram of resource allocation of distributed parallel computing;

[0041] Figure 3 is a schematic diagram of a distributed computing system of the embodiments of the present application;

[0042] Figure 4 is a schematic diagram of a computing device topology inside a computing node;

[0043] Figure 5 is a schematic diagram of a network topology between computing nodes;

[0044] Figure 6 is a schematic diagram of a method for a distributed computing system of the embodiments of the present application. DETAILED DESCRIPTION

[0045] The foregoing and other features of the present application will become apparent to those skilled in the art from the following description of specific embodiments of the application, with reference to the accompanying drawings. In the description of embodiments and drawings, specific details are disclosed to provide a thorough understanding of the principles of the present application. It will be apparent to those skilled in the art, however, that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the understanding of this description. Various embodiments of the present application are described below with reference to the accompanying drawings. These embodiments are merely exemplary and not limiting of the present application.

[0046] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements from the point of view, but do not represent the spatial arrangement or time sequence of the elements, and the elements should not be limited by these terms. The term "and / or" includes any one and all combinations of the associated listed terms. The terms "include", "have" and the like mean the existence of the stated features, elements, elements or components, but do not exclude the existence or addition of one or more other features, elements, elements or components.

[0047] In the embodiments of the present application, the singular form "a", "an", "the" and the like include the plural form, should be broadly understood as "one" or "a kind of" and not limited to the meaning of "one"; in addition, the term "the" should be understood as including both the singular form and the plural form, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least partially according to", unless the context clearly indicates otherwise.

[0048] Embodiments of the first aspect

[0049] The embodiments of the first aspect of the present application provide a distributed computing system.

[0050] Figure 3 is a schematic diagram of the distributed computing system of the embodiments of the present application. As shown in Figure 3 The distributed computing system 100 includes a task manager 101, a resource scheduler 102 and a computing framework 103, wherein:

[0051] The task manager 101 sends parallel computing task information to the resource scheduler 102;

[0052] The resource scheduler 102 allocates computing devices in the computing nodes to each computing task in the parallel computing tasks according to the parallel computing task information and the topology information of the available computing nodes, and sends the information of the allocated computing devices to the task manager 101;

[0053] The task manager 101 sends the corresponding relationship between each computing task and the computing device to the computing framework 103;

[0054] The computing framework 103 distributes each computing task to the corresponding computing device.

[0055] In the present application, the resource scheduler 102 groups the parallel computing tasks according to the parallel computing task information, and selects the corresponding computing resources in combination with the topology information of the available computing resources, which realizes the allocation of the computing tasks with intensive communication in the parallel computing tasks on the computing devices with relatively short distance in topology, reduces the time delay of traffic communication, and improves the computing performance.

[0056] In some embodiments, the parallel computing task information includes at least one of computing resource requirement, parallel computing split parameter and intra-group communication volume, wherein the computing resource requirement includes the number of computing devices required by the parallel computing task, the parallel computing split parameter includes the parallel computing mode adopted by the parallel computing task and the parallel degree of each parallel computing mode, and the intra-group communication volume includes the intra-group communication volume of each parallel computing mode.

[0057] In some embodiments, the parallel computing task can be at least one of a distributed training task, a parallel training task and a distributed inference task.

[0058] In some embodiments, the task manager 102 also sends the trained model information corresponding to the training task to the computing framework.

[0059] In some embodiments, the parallel computing mode includes one or more of tensor parallelism, data parallelism, and pipeline parallelism.

[0060] In some embodiments, the topology information of the available computing nodes (i.e., idle computing nodes) includes computing device topology information inside the available computing nodes, and / or network topology information among the available computing nodes.

[0061] Figure 4 is a schematic diagram of computing device topology inside a computing node. As shown in Figure 4 , a computing node inside includes computing devices G0, G1, G2, G3, G4, G5, G6, and G7. The computing device topology inside the computing node can be: full interconnection between each pair of computing devices G0, G1, G2, G3, full interconnection between each pair of computing devices G4, G5, G6, G7, which can use a high-speed interconnection communication protocol, such as the NVLink protocol. Computing devices G0, G2 are connected to the same interconnection interface P0 with network card N0, having affinity, computing devices G1, G3 are connected to the same interconnection interface P1, such as a PCIe interface, with network card N1, having affinity, the interconnection interfaces P0, P1 are interconnected, so computing devices G0, G1, G2, G3 and network cards N0, N1 all have affinity, and do not need to communicate through the interconnection path Q between central processors C0 and C1. Similarly, computing devices G4, G5, G6, G7 and network cards N2, N3 have affinity, and do not need to communicate through the interconnection path Q between central processors C0 and C1. Computing devices G0 and computing devices G4, G6, G7 are connected to central processors C0 and C1 respectively, and need to communicate through the interconnection path Q between central processors C0 and C1, where the transmission rates from high to low are high-speed interconnection communication protocol, interconnection interfaces P0 and P1, and interconnection path Q.

[0062] Figure 5 is a schematic diagram of network topology among computing nodes. As shown in Figure 5As shown, the network interconnection architecture between the computing nodes in the cluster can include a backbone switch Spine0 and access switches Leaf0, Leaf1 and Leaf2. Each switch is connected to at least two servers (or computers), and each of the servers (or computers) H0, H1, H2, H3, H4 and H5 is provided with a high-speed Internet card R0, R1, R2, R3, R4 and R5, such as a Remote Direct Memory Access (RDMA) card, to achieve full interconnection between each pair of servers, and the high-speed Internet cards R0, R1, R2, R3, R4 and R5 are interconnected with the access switches Leaf0, Leaf1 and Leaf2, respectively. For example, the computing devices G00 and G01 are located in the same server H0, and communication between them does not need to cross a switch; the computing devices G00 and G10 are distributed in the servers H0 and H1 under the same access switch Leaf0, and communication between them is through a high-speed Internet card, i.e., across one switch; the computing devices G00 and G20 are distributed in the servers H0 and H2 under the same backbone switch Spine0 and different access switches Leaf0 and Leaf1, and communication between them needs to be cross-node, i.e., across three switches. Among them, the transmission rates from high to low are, in turn, communication between computing devices in the same server, communication between computing devices across one switch, and communication between computing devices across three switches.

[0063] In some embodiments, the resource scheduler 102 allocates computing devices for each computing task. For example:

[0064] The resource scheduler 102 groups the required computing devices according to the parallel computing mode (i.e., logical grouping, for example, computing device A and computing device B are in a group, etc.); and then, the resource scheduler 102 allocates computing devices for each computing task according to the grouping information and the topology information of the available computing nodes (i.e., mapping the logical grouping to the real computing devices, for example: which specific computing device does computing device A correspond to, what is the address information of the computing device, etc.; which specific computing device does computing device B correspond to, what is the address information of the computing device, etc.).

[0065] In some embodiments, the resource scheduler 102 allocates computing devices for each computing task in order to minimize the communication delay generated by the computing devices for completing the parallel computing task or to minimize the amount of data exchanged by the switches for completing the parallel computing task.

[0066] For example, the computing resource requirement of the parallel computing task is, for example, 16 computing devices, and the parallel computing split parameter is, for example, a tensor parallelism of 2, a pipeline parallelism of 4, and a data parallelism of 2. The available computing devices include G00, G01, G06, G07, G10, G11, G16, G20, G22, G24, G26, G31, G35, G43, G45, G46, G47, and G51.

[0067] According to the parallel computing mode, the required computing devices are logically grouped (i.e., logical grouping), as described above, for example, the computing devices are divided into 8 tensor parallel groups:

[0068] [g0,g1],[g2,g3],[g4,g5],[g6,g7],[g8,g9],[g10,g11],[g12,g13],[g14,g15]

[0069] 4 pipeline parallel groups:

[0070] [g0,g4,g8,g12],[g1,g5,g9,g13],[g2,g6,g10,g14],[g3,g7,g11,g15]

[0071] 8 data parallel groups:

[0072] [g0,g2],[g1,g3],[g4,g6],[g5,g7],[g8,g10],[g9,g11],[g12,g14],[g13,g15]

[0073] Taking the above parallel computing task as an example, the above logical level grouping can be assigned to specific computing devices according to the topology information of the available computing devices, which can be converted into a quadratic assignment problem (QAP) to find the optimal solution.

[0074] For example, the total number of computing devices and computing tasks is set to n, and the topology relationship between the computing devices is represented by D[n][n], where D[x][y] represents the topology distance between computing device x and computing device y. The goal is to determine the serial number of the computing device running for each computing task, represented by loc[*], so that the following cost function f is minimized, where loc[i] represents the serial number of the computing device allocated to the i-th computing task, and loc[j] represents the serial number of the computing device allocated to the j-th computing task.

[0075]

[0076] For example, the computing tasks of 8 tensor parallel groups are respectively allocated real computing devices [G00, G01], [G06, G07], [G10, G11], [G20, G22], [G24, G26], [G31, G35], [G43, G44], [G46, G47], wherein each real computing device has a corresponding IP address, so that the real computing devices corresponding to the computing tasks in each tensor parallel group are located in the same server, and data communication can be performed through a high-speed interconnection communication protocol or an interconnection interface, without cross-node communication, thereby reducing communication latency.

[0077] For example, the computing tasks of 8 tensor parallel groups are respectively allocated real computing devices [G00, G01], [G06, G07], [G10, G11], [G20, G22], [G24, G26], [G31, G35], [G43, G44], [G46, G47], wherein each real computing device has a corresponding IP address, so that the real computing devices corresponding to the computing tasks in each tensor parallel group are located in the same server, and data communication can be performed through a high-speed interconnection communication protocol or an interconnection interface, without cross-node communication, thereby reducing communication latency.

[0078] For example, the computing tasks of 8 tensor parallel groups are respectively allocated real computing devices [G00, G01], [G06, G07], [G10, G11], [G20, G22], [G24, G26], [G31, G35], [G43, G44], [G46, G47], wherein each real computing device has a corresponding IP address, so that the real computing devices corresponding to the computing tasks in each tensor parallel group are located in the same server, and data communication can be performed through a high-speed interconnection communication protocol or an interconnection interface, without cross-node communication, thereby reducing communication latency.

[0079] In some embodiments, each parallel computing mode has a weight value corresponding to the communication amount in the group, and the weight value is used to allocate computing devices to the computing tasks.

[0080] In some examples, the weight value w t is set for the tensor parallel group, the weight value w d is set for the data parallel group, and the weight value w p is set for the pipeline parallel group, wherein w t > w d > wp .

[0081] In some embodiments, the resource scheduler 102 further considers the intra-group communication of each parallel computing mode when allocating computing devices for each computing task. For example, the resource scheduler 102 allocates computing devices for each computing task, including: the resource scheduler 102 groups the required computing devices according to the parallel computing mode; and then, the resource scheduler 102 allocates computing devices for each computing task according to the grouped information, the network topology information between available computing nodes, and the intra-group communication of each parallel computing mode.

[0082] In some embodiments, the resource scheduler 102 allocates computing devices for each computing task in order to minimize the communication delay generated by the computing devices for completing the parallel computing task, for example, in order to minimize the data communication generated by the computing task for completing the parallel computing task.

[0083] For the above parallel computing task, the logical level grouping can be allocated with specific computing devices according to the topology information of available computing devices, the grouping information of each computing task, and the intra-group communication after grouping, which can be converted into a quadratic assignment problem (QAP) to find the optimal solution.

[0084] For example, the total number of computing devices and computing tasks is set to n, and the topology relationship between the computing devices is represented by D[n][n], where D[x][y] represents the topology distance between computing device x and computing device y; and the communication between computing tasks in the kth parallel mode is represented by W k [*][*], where W k [x][y] represents the communication between computing device x and computing device y in the kth parallel mode grouping. The goal is to determine the serial number of the computing device running for each computing task, represented by loc[n], where loc[i] represents the serial number of the computing device allocated for the ith computing task, and loc[j] represents the serial number of the computing device allocated for the jth computing task, so that the following cost function fw is minimized.

[0085]

[0086] Embodiments of the second aspect

[0087] Embodiments of the second aspect of the present application provide a method for a distributed computing system.

[0088] Figure 6 is a schematic diagram of the method for a distributed computing system according to an embodiment of the present application. As shown in Figure 3 and Figure 6As shown, the distributed computing system 100 comprises a task manager 101, a resource scheduler 102 and a computing framework 103, and the method for the distributed computing system comprises:

[0089] Operation 601, the task manager 101 sends parallel computing task information to the resource scheduler 102;

[0090] Operation 602, the resource scheduler 102 allocates computing devices in the computing nodes to each computing task in the parallel computing task according to the parallel computing task information and the topology information of the available computing nodes, and sends the information of the allocated computing devices to the task manager 101;

[0091] Operation 603, the task manager 101 sends the corresponding relationship between each computing task and the computing device to the computing framework 103;

[0092] Operation 604, the computing framework 103 distributes each computing task to the corresponding computing device.

[0093] The distributed computing system 100 and the method for the distributed computing system of the present application are applicable to various large model computing frameworks.

[0094] In the embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components can or can not be physically separated, and the components displayed as units can be one physical unit or multiple physical units, that is, they can be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0097] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to perform all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0098] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0099] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0100] These computer program instructions can also be stored in a computer-readable storage medium that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for performing the functions specified in one or more flows and / or blocks.

[0101] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0102] The above-described embodiments are merely specific implementations of the present application, and are not intended to limit the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above-described embodiments are merely specific implementations of the present application, and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0103] The above content is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A distributed computing system, characterized in that: The system includes a task manager, a resource scheduler and a computing framework. in, The task manager sends the parallel computing task information to the resource scheduler; The resource scheduler allocates a computing device in the available computing nodes to each computing task in the parallel computing task according to the parallel computing task information and the topology information of the available computing nodes, and sends information of the allocated computing device to the task manager; The task manager sends the correspondence between each computing task and the computing device to the computing framework; The computing framework distributes each of the computing tasks to the corresponding computing devices.

2. The system according to claim 1, wherein The parallel computing task information includes at least one of computing resource requirements, parallel computing partitioning parameters, and intra-parallel computing group communication volume. in, The computing resource requirements include the number of computing devices required by the parallel computing task, The parallel computing partitioning parameters include the parallel computing mode used by the parallel computing task and the parallelism of each parallel computing mode. The intra-group communication volume of the parallel computing group includes the intra-group communication volume of each parallel computing mode.

3. The system according to claim 2, wherein: The parallel computing mode includes at least one of tensor parallelism, data parallelism and pipeline parallelism.

4. The system according to claim 1, wherein: The topology information of the available computing nodes includes: Computing device topology information within the available computing nodes, and / or network topology information between the available computing nodes.

5. The system according to claim 1, wherein: The resource scheduler allocates the computing device to each computing task, including: The resource scheduler groups required computing devices according to the parallel computing mode; and The resource scheduler allocates the computing device to each computing task according to the grouping information and the topology information of the available computing nodes.

6. The system according to claim 1, wherein: The resource scheduler allocates the computing device to each computing task, including: The resource scheduler groups required computing devices according to the parallel computing mode; and The resource scheduler allocates the computing device to each computing task according to the grouping information, the topology information of the available computing nodes, and the intra-group communication volume of each parallel computing mode.

7. The system according to claim 6, wherein: Each of the parallel computing modes has a weight value corresponding to the intra-group communication volume, and the weight value is used to allocate the computing device to each computing task.

8. The system according to claim 5 or 6, characterized in that The resource scheduler allocates the computing device to each computing task with the goal of minimizing the communication delay generated by the computing device in completing the parallel computing task.

9. The system according to claim 1, wherein: The computing task includes a training task or an inference task.

10. The system according to claim 9, wherein: The task manager also sends the training model information corresponding to the training task to the computing framework.

11. A method for a distributed computing system, characterized in that: The distributed computing system includes a task manager, a resource scheduler and a computing framework. The method comprises: The task manager sends the parallel computing task information to the resource scheduler; The resource scheduler allocates a computing device in the available computing nodes to each computing task in the parallel computing task according to the parallel computing task information and the topology information of the available computing nodes, and sends information of the allocated computing device to the task manager; The task manager sends the correspondence between each computing task and the computing device to the computing framework; and The computing framework distributes each of the computing tasks to the corresponding computing devices.

Images