Large-model capacity orchestration method and system based on GPU network topology, and medium

Abstract

Disclosed in the present application are a large-model capacity orchestration method and system based on a GPU network topology, and a medium. The method comprises: detecting network bandwidth information between a plurality of devices, wherein the devices comprise GPUs and a first device; on the basis of the network bandwidth information, constructing a device network topology by taking the GPUs and the first device as nodes, taking as an initial link a channel connecting two devices, and taking the network bandwidth information and the type of each initial link as edges; acquiring a training request, and on the basis of the device network topology, selecting a training network topology corresponding to the training request; creating a large-model container on the basis of the training network topology; and by means of configuring the large-model container, executing a training task corresponding to the training request. In the present application, the network bandwidth between the GPUs can be discovered by means of the device network topology, so that the orchestration of a container can meet the bandwidth requirements between the GPUs, thereby improving the training performance and training efficiency of large-model training and the utilization rate of training resources. The present application is applied to the technical field of large-model training.

Description

Large model capacity orchestration method, system and medium based on GPU network topology

[0001] This application claims priority to a Chinese patent application filed with the Patent Office of China on October 25, 2023, with application number 2023113927306 and entitled “Large Model Capacity Orchestration Method, System and Medium Based on GPU Network Topology,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of large model training technology, and in particular to a large model capacity orchestration method, system, and medium based on GPU network topology. Background Art

[0003] In large-scale deep learning training, the Graphics Processing Unit (GPU) plays a key role, responsible for processing huge amounts of computing and data. Common configuration methods for GPUs include single-machine multi-card mode and multi-machine multi-card mode. As the amount of data in the model increases, the bandwidth requirements between single-machine multi-cards and multi-machine multi-cards also increase. Related technologies use communication methods such as PCIe interconnection, NVLink, and NVSwitch to meet the bandwidth requirements between graphics processors. In addition, in large-scale deep learning training, related technologies usually deploy training nodes for large models in the form of containerized deployment. Containerized deployment refers to running multiple virtual machines on a physical machine. Multiple virtual machines share an operating system, and each virtual machine is an independent environment. Containerized deployment prevents multiple environments from affecting each other while improving the security of environmental operation.

[0004] However, the communication method adopted by the relevant technology causes real production clusters to produce complex graphics processor network topology and hierarchy. In this case, the relevant technology uses physical machines such as Ethernet switches as the lowest granularity to build the network topology of physical nodes. This network topology cannot discover the bandwidth between graphics processors. This will result in the orchestration of containers being unable to meet the bandwidth requirements between graphics processors when containerized deployment is performed according to the network topology, thereby reducing the training performance of the training nodes of large models and causing waste of network bandwidth between graphics processors.

[0005] The above problems need to be solved urgently.

[0006] Summary of the Invention

[0007] The purpose of this application is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0008] To this end, the purpose of this application is to provide a large model capacity orchestration method, system and medium based on GPU network topology.

[0009] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0010] On the one hand, embodiments of the present application provide a large model capacity orchestration method based on GPU network topology, including:

[0011] detecting network bandwidth information between a plurality of devices, the devices including a graphics processor and a first device;

[0012] Constructing a device network topology based on the network bandwidth information, using the graphics processor and the first device as nodes, the channel connecting the two devices as an initial link, and the network bandwidth information and type of each initial link as an edge;

[0013] Obtaining a training request, and determining a training network topology corresponding to the training request based on the device network topology;

[0014] Creating a large model container according to the training network topology;

[0015] The large model container is configured to execute a training task corresponding to the training request.

[0016] The large model capacity orchestration method based on GPU network topology according to the above embodiment of the present application may also have the following additional technical features:

[0017] Furthermore, in one embodiment of the present application, detecting network bandwidth information between multiple devices includes:

[0018] The network bandwidth information of each initial link is traversed to obtain the network bandwidth information between multiple devices.

[0019] Furthermore, in one embodiment of the present application, the type of the initial link includes one or more of a PCIe interconnect link, an NVLink link, an NVSwitch link, an IB network link, and an Ethernet link.

[0020] Furthermore, in one embodiment of the present application, determining a training network topology corresponding to the training request according to the device network topology includes:

[0021] Constructing multiple sub-network topologies according to the device network topology;

[0022] Get the node resources of the GPU in each sub-network topology;

[0023] The network flow information of each sub-network topology is calculated based on the node resources of the graphics processor in each sub-network topology, and the sub-network topology with the largest network flow information is selected as the training network topology corresponding to the training request.

[0024] Furthermore, in one embodiment of the present application, in the sub-network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0025] Furthermore, in one embodiment of the present application, the node resources of the graphics processor include the memory quantity, memory usage, core usage, type of the graphics processor and the network bandwidth information and type of the initial link connected to the graphics processor in the device network topology.

[0026] Furthermore, in one embodiment of the present application, calculating network flow information of each sub-network topology based on node resources of the graphics processor in each sub-network topology includes:

[0027] According to the node resources of the graphics processor in each sub-network topology, the maximum flow of each sub-network topology is calculated using the maximum flow algorithm, and the network maximum flow of each sub-network topology is used as the network flow information of each sub-network topology.

[0028] Furthermore, in one embodiment of the present application, the maximum flow algorithm is the Ford-Fulkerson algorithm.

[0029] Furthermore, in one embodiment of the present application, creating a large model container according to the training network topology includes:

[0030] A training device where a graphics processor in the training network topology is located is obtained, a large model container is created on the training device, and the graphics processor in the training network topology is deployed in the corresponding large model container.

[0031] Furthermore, in one embodiment of the present application, creating a large model container on the training device includes:

[0032] A large model container is created on the training device based on Kubernetes.

[0033] Furthermore, in one embodiment of the present application, configuring the large model container to execute the training task corresponding to the training request includes:

[0034] Select any one large model container from the plurality of large model containers as a master node container, and use the other large model containers except the master node container as slave node containers;

[0035] The master node container manages the plurality of slave node containers, and the plurality of slave node containers execute the training tasks corresponding to the training request.

[0036] Furthermore, in one embodiment of the present application, the method further comprises the following steps:

[0037] When a stop request is detected, stopping execution of the training task corresponding to the training request;

[0038] Updating the device network topology, and updating the training network topology according to the updated device network topology;

[0039] Re-execute the training task corresponding to the training request according to the updated training network topology.

[0040] Furthermore, in one embodiment of the present application, updating the device network topology and updating the training network topology according to the updated device network topology include:

[0041] Eliminating an initial link and device corresponding to the stop request from the device network topology, and updating network bandwidth information between multiple devices in the device network topology to generate an updated device network topology;

[0042] Constructing multiple sub-updated network topologies according to the updated device network topology;

[0043] The network flow information of each sub-updated network topology is obtained, and the sub-updated network topology with the largest network flow information is selected as the updated training network topology.

[0044] Furthermore, in one embodiment of the present application, in the sub-update network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0045] Furthermore, in one embodiment of the present application, obtaining network flow information of each sub-updated network topology includes:

[0046] When multiple sub-update network topologies are different from multiple sub-network topologies, the update node resources of the graphics processor in each sub-update network topology are obtained, and the network flow information of each sub-update network topology is calculated based on the update node resources of the graphics processor in each sub-update network topology.

[0047] Furthermore, in one embodiment of the present application, the updated node resources of the graphics processor include, in the updated device network topology, the memory quantity, memory usage, core usage, type of the graphics processor, and network bandwidth information and type of the initial link connected to the graphics processor.

[0048] Furthermore, in one embodiment of the present application, calculating network flow information of each sub-update network topology according to update node resources of the graphics processor in each sub-update network topology includes:

[0049] According to the update node resources of the graphics processor in each sub-update network topology, the maximum flow of each sub-update network topology is calculated using the maximum flow algorithm, and the network maximum flow of each sub-update network topology is used as the network flow information of each sub-update network topology.

[0050] Furthermore, in one embodiment of the present application, the maximum flow algorithm is the Ford-Fulkerson algorithm.

[0051] Furthermore, in one embodiment of the present application, obtaining network flow information of each sub-updated network topology includes:

[0052] When the multiple sub-update network topologies are the same as the multiple sub-network topologies, the network flow information of the multiple sub-network topologies is used as the network flow information of the multiple sub-update network topologies.

[0053] Furthermore, in one embodiment of the present application, re-executing the training task corresponding to the training request according to the updated training network topology includes:

[0054] Rearranging the large model container according to the updated training network topology;

[0055] When the number of graphics processors in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, the training task corresponding to the training request is re-executed by configuring the large model container.

[0056] Furthermore, in one embodiment of the present application, the method further comprises the following steps:

[0057] When a stop request is detected, a ckpt file corresponding to the training task is stored.

[0058] Furthermore, in one embodiment of the present application, re-executing the training task corresponding to the training request includes:

[0059] Call the ckpt file corresponding to the training task and re-execute the training task corresponding to the training request.

[0060] On the other hand, an embodiment of the present application provides a large model capacity orchestration system based on GPU network topology, including:

[0061] a detection module, configured to detect network bandwidth information between a plurality of devices, the devices including a graphics processor and a first device;

[0062] a topology processing module, configured to construct a device network topology based on the network bandwidth information, using the graphics processor and the first device as nodes, a channel connecting the two devices as an initial link, and network bandwidth information and type of each initial link as an edge;

[0063] Acquisition module, used to obtain training requests;

[0064] a node processing module, configured to determine a training network topology corresponding to the training request according to the device network topology;

[0065] A container processing module, configured to create a large model container based on the training network topology;

[0066] The training processing module is used to execute the training task corresponding to the training request by configuring the large model container.

[0067] On the other hand, an embodiment of the present application provides a storage medium storing a program executable by a processor, which, when executed by the processor, is used to implement the aforementioned large model capacity orchestration method based on GPU network topology.

[0068] The beneficial effects of the present application are: providing a large model capacity orchestration method, system and medium based on GPU network topology, the method comprising first detecting the network bandwidth information between multiple devices, the devices comprising a graphics processor and a first device, then constructing a device network topology based on the network bandwidth information, then obtaining a training request, selecting a training network topology corresponding to the training request based on the device network topology, and creating a large model container based on the training network topology, and finally executing the training task corresponding to the training request by configuring the large model container. The present application uses the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as an edge to construct a device network topology, discover the network bandwidth between the graphics processors through the device network topology, so that the orchestration of the container can meet the bandwidth requirements between the graphics processors, while improving the training performance and training efficiency of large model training, it also improves the utilization rate of training resources.

[0069] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] FIG1 is a schematic diagram of a network topology with host nodes as the granularity of the related art;

[0071] FIG2 is a flow chart of a large model container orchestration method based on GPU network topology provided by the present application;

[0072] FIG3 is a schematic diagram of a device network topology provided in this application;

[0073] FIG4 is a schematic diagram of an initial link provided by this application;

[0074] FIG5 is a flowchart of detecting link bandwidth information provided by the present application;

[0075] FIG6 is a flowchart of selecting a training network topology provided by the present application;

[0076] FIG7A is a schematic diagram of a sub-network topology provided by the present application;

[0077] FIG7B is another schematic diagram of a sub-network topology provided by the present application;

[0078] FIG8 is another flow chart of the large model container orchestration method based on GPU network topology provided by the present application;

[0079] FIG9 is a flowchart of re-executing a training task provided by the present application;

[0080] FIG10 is a structural diagram of a large-model container orchestration system based on GPU network topology provided by this application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0082] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0083] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0085] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0086] 1) A GPU (Graphics Processing Unit), also known as a display core, visual processor, or display chip, is a hardware device specifically designed for processing and rendering graphics data. It is a high-performance parallel computing engine used to accelerate graphics, video, and other compute-intensive tasks, and is essential for training large models.

[0087] 2) PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard. It provides high-speed serial point-to-point dual-channel, high-bandwidth transmission. Devices connected via PCIe are allocated exclusive channel bandwidth and do not share bus bandwidth. PCIe offers high data transfer rates and high data reliability.

[0088] 3) NVLink is a high-speed, point-to-point interface that uses a point-to-point architecture and serial transmission. It is used to connect the CPU (Central Processing Unit) and GPU, and can also be used to connect multiple GPUs. NVLink technology is designed to provide a high-bandwidth, low-latency communication channel, allowing GPUs to access remote GPU memory without going through the PCIe bus.

[0089] 4) NVSwitch is a many-to-many GPU communication method that integrates multiple NVLinks to enable many-to-many GPU communication within a single node at the high speed of NVLink, further improving interconnect performance. The combination of NVLink and NVSwitch efficiently scales AI performance to multiple GPUs. Each GPU has multiple NVLink links connected to NVSwitch, enabling high-speed many-to-many communication.

[0090] 5) Infini Band (IB) networking is a low-latency computer network communication standard for high-performance computing. It offers extremely high throughput and low latency, primarily used for computer-to-computer data interconnection. IB networking is also used for direct or switched interconnection between servers and storage systems, as well as for interconnection between storage systems.

[0091] 6) Checkpoint refers to saving the intermediate state of the model during the training process. Checkpoint files are created during the training process to periodically save the model parameters and optimizer status, so that training can be resumed and the model can be evaluated or used when needed.

[0092] 7) Large models are those with a large number of parameters, deep layers, and high complexity. These models typically require significant computing resources and storage space for training and inference, and can achieve better performance and results in certain tasks. Large models have more degrees of freedom and expressive power, enabling them to better fit and capture complex data patterns and regularities.

[0093] 8) Network flows, a problem-solving method analogous to water flow, are closely related to linear programming. A directed graph is defined as a network flow graph if there is a unique source and sink, and every edge in the graph has a non-negative capacity.

[0094] In large-scale deep learning training, GPUs play a key role, responsible for processing huge amounts of computing and data. Common GPU configuration methods include single-machine multi-card configuration and multi-machine multi-card configuration. As the amount of data in the model increases, the bandwidth requirements between single-machine multi-cards and multi-machine multi-card configuration also increase. Related technologies meet the bandwidth requirements between GPUs through communication methods such as PCIe interconnection, NVLink, NVSwitch, GPUDirect RDMA, etc., among which GPUDirect RDMA refers to a communication method that realizes multi-card communication through the IB network. In addition, in large-scale deep learning training, related technologies usually deploy training nodes for large models in the form of containerized deployment. Containerized deployment refers to running multiple virtual machines on a physical machine. Multiple virtual machines share an operating system, and each virtual machine is an independent environment. Containerized deployment prevents multiple environments from affecting each other while improving the security of environmental operation.

[0095] However, the communication method adopted by the related art causes the real production cluster to produce a complex GPU network topology and hierarchy. In this case, the related art uses physical machines such as Ethernet switches as the lowest granularity to construct the network topology of physical nodes. For example, referring to Figure 1, Figure 1 is a schematic diagram of the network topology of the related art with host nodes as the granularity, integrating multiple GPUs into one host, using multiple hosts as nodes, and using the bandwidth between multiple hosts as edges to form a network topology. This network topology cannot discover the bandwidth between GPUs, which will result in the orchestration of containers being unable to meet the bandwidth requirements between GPUs when containerized deployment is performed according to the network topology, thereby reducing the training performance of the training nodes of large models and causing a waste of network bandwidth between GPUs.

[0096] In addition, when a network link or GPU fails, large model training will report an error. Related technologies usually use manual intervention to detect the fault and restore communication and large model training. It cannot self-troubleshoot and adjust the faulty training node, nor can it self-restore training, which increases the maintenance cost of the equipment and reduces the success rate and training efficiency of large model training.

[0097] In response to the problems and defects in the related art, the embodiments of the present application provide a large model capacity orchestration method, system and medium based on GPU network topology. First, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of the initial link are used as edges to construct a device network topology as the basis for the orchestration and deployment of the large model container. Then, when a large model training request is obtained, the GPU node located on the initial link of the maximum network bandwidth information can be dynamically found based on the device network topology. Finally, the large model container is reasonably deployed and orchestrated on these GPU nodes, thereby realizing the training of the large model. In addition, when a network failure occurs or a link changes, the embodiments of the present application suspend the current training task by responding to a stop request, re-find the link with the maximum network bandwidth information and update the GPU node used for training. When the re-found link and GPU node meet the training requirements, the training of the large model is restarted to realize the automatic recovery of the large model training.

[0098] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0099] First, an implementation step of the large model container orchestration method based on GPU network topology proposed in an embodiment of the present application will be described in detail below.

[0100] The large-model container orchestration method based on GPU network topology provided in the embodiments of the present application can be applied to terminals, servers, or software running on terminals or servers. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0101] Referring to FIG. 2 , FIG. 2 is a flow chart of a large model container orchestration method based on a GPU network topology provided by the present application. The large model container orchestration method based on a GPU network topology provided by the present application may include but is not limited to the following steps:

[0102] S101: Detect network bandwidth information between multiple devices.

[0103] It should be noted that the device includes a GPU and a first device.

[0104] In some embodiments, the first device is at least one, and the first device may include, but is not limited to, an Ethernet switch, a CPU, an NVLink bridge, etc. It will be understood by those skilled in the art that other devices such as an IB switch, an NVSwitch bridge, and terminals using PCIe interconnection are also applicable and only require corresponding adjustments. The terminals using PCIe interconnection may be terminals using PCIe 3.0 interconnection or terminals using PCIe 4.0 interconnection, and this embodiment of the application does not specifically limit this.

[0105] In this step, network bandwidth information between multiple GPUs, network bandwidth information between multiple first devices, and network bandwidth information between a GPU and a first device are obtained as network bandwidth information between multiple devices. Network bandwidth information refers to the amount of data that can be transmitted per unit time. A larger network bandwidth information indicates a stronger communication capability.

[0106] S102: Construct a device network topology based on the network bandwidth information.

[0107] Related technologies use physical machines such as Ethernet switches as the lowest granularity to construct the network topology of physical nodes, which is unable to discover the bandwidth between GPUs. This will result in the orchestration of containers being unable to meet the bandwidth requirements between GPUs when containerized deployment is performed based on the network topology. In response to this problem and defect, this step proposes a device network topology based on GPU granularity, which can discover the bandwidth between multiple GPUs and provide a bandwidth basis for subsequent containerized deployment. Specifically, this step uses the GPU and the first device as nodes of the network topology, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as the edge of the network topology, thereby constructing a global device network topology.

[0108] It should be noted that the channel connecting two devices includes a channel connecting two GPUs, a channel connecting a GPU and a first device, and a channel connecting two first devices.

[0109] For example, referring to FIG3, FIG3 is a schematic diagram of the device network topology provided by the present application, wherein the device includes eight GPUs and four first devices, wherein the eight GPUs are GPU1 to GPU8, and the four first devices are NVSwitch bridge, PCIE4.0Switch first terminal, PCIE4.0Switch second terminal and IB switch, respectively. The implementation process of constructing the device network topology is as follows: First, GPU1 to GPU8 are determined as nodes of the device network topology. Then, the initial link is determined. Specifically, referring to FIG4, FIG4 is a schematic diagram of the initial link provided by the present application, wherein the channels connecting multiple GPUs and the channels connecting the GPU and the first device are used as initial links, thereby obtaining twenty-four initial links. Afterwards, the network bandwidth information and type of each initial link are used as edges. For example, if the network bandwidth information of initial link 1 is 600G and the type of initial link 1 is NVLink link, 600G and NVLink are used as edges corresponding to initial link 1. Finally, based on the determined nodes and edges, the device network topology shown in FIG3 is constructed.

[0110] S103: Obtain a training request, and select a training network topology corresponding to the training request according to the device network topology.

[0111] It should be noted that the training request is configured with a corresponding training task.

[0112] It is understandable that the training task can be a training task for large models such as PaLM2, GPT-3, LLaMA, etc. Large models can be applied to technical fields such as natural language processing, computer vision, and autonomous driving. Those skilled in the art can select training tasks according to actual conditions. The embodiments of this application do not make specific limitations on this.

[0113] In this step, the device network topology is used as the basis and basis for network performance. The initial link of the maximum network bandwidth information between multiple GPUs is found from the device network topology. Based on the initial link of the maximum network bandwidth information, multiple GPUs are selected as training processing nodes corresponding to the training request. These training processing nodes and the initial link of the maximum network bandwidth information together constitute the training network topology.

[0114] S104: Create a large model container based on the training network topology.

[0115] In this step, based on the training network topology obtained in the previous steps, multiple large model containers are created and each large model container is specified to use an appropriate network link for communication.

[0116] S105: Execute the training task corresponding to the training request by configuring the large model container.

[0117] In some embodiments of the present application, in step S101, the step of detecting network bandwidth information between multiple devices may include but is not limited to:

[0118] S1011 , traverse the network bandwidth information of each initial link, and then obtain the network bandwidth information between multiple devices.

[0119] It should be noted that the network bandwidth information between multiple devices may include but is not limited to network bandwidth information between multiple GPUs, network bandwidth information between multiple first devices, and network bandwidth information between a GPU and a first device.

[0120] In some embodiments, the type of the initial link includes one or more of a PCIe interconnect link, an NVLink link, an NVSwitch link, an IB network link, and an Ethernet link. Those skilled in the art may make appropriate selections based on actual conditions, and this application does not impose specific limitations on this.

[0121] In this step, the network bandwidth information of each initial link is traversed. The initial link is a channel connecting two devices, and then the network bandwidth information between multiple GPUs, the network bandwidth information between the GPU and the first device, and the network bandwidth information between multiple first devices are obtained.

[0122] In some embodiments, the detection of the network bandwidth information of each initial link may be performed in parallel or in series, which is not specifically limited in the embodiments of the present application.

[0123] For example, taking the parallel detection of the network bandwidth information of each initial link as an example, referring to FIG5 , FIG5 is a flowchart of detecting link bandwidth information provided by the present application, the step of detecting the network bandwidth information of each initial link includes at least one of the following:

[0124] When two devices communicate through a PCIe interconnect link, detect the network bandwidth information of the end-to-end PCIe interconnect link;

[0125] When two devices communicate through an NVLink link, detect the network bandwidth information of the end-to-end NVLink link;

[0126] When two devices communicate through an NVSwitch link, detect the network bandwidth information of the end-to-end NVSwitch link;

[0127] When two devices communicate through an IB network link, the network bandwidth information of the end-to-end IB network link is detected;

[0128] When two devices communicate through an Ethernet link, the network bandwidth information of the end-to-end Ethernet link is detected.

[0129] In some embodiments of the present application, referring to FIG. 6 , which is a flowchart of selecting a training network topology provided by the present application, in step S103 , the steps of determining a training network topology corresponding to a training request based on a device network topology may include but are not limited to:

[0130] S1031: Construct multiple sub-network topologies according to the device network topology.

[0131] It should be noted that in the sub-network topology, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0132] In this step, the training task corresponding to the training request is first determined, and the number of GPUs required for the training task is determined based on the training task. Then, multiple subgraphs are constructed based on the device network topology, where the number of subgraphs is equal to the number of GPUs required for the training task, and the subgraphs are used as the subnetwork topology.

[0133] For example, referring to Figures 3, 4, 7A and 7B, Figure 7A is a schematic diagram of a sub-network topology provided by this application, and Figure 7B is another schematic diagram of a sub-network topology provided by this application. When the number of GPUs required for the training task is four, multiple sub-network topologies can be constructed based on the device network topology shown in Figure 3. The sub-network topologies shown in Figures 7A and 7B are two of the multiple sub-network topologies. In the sub-network topology shown in Figure 7A, GPU1 to GPU4 and the first terminal of PCIE4.0Switch are used as nodes of the sub-network topology, initial links 1, 3, 4, 6, 7, 9, and 11 are used as initial links of the sub-network topology, and the network bandwidth information and type of initial links 1, 3, 4, 6, 7, 9, and 11 are used as edges of the sub-network topology. In the sub-network topology shown in Figure 7B, GPU1, GPU2, GPU5, GPU6, PCIE4.0Switch first terminal and PCIE4.0Switch second terminal are used as nodes of the sub-network topology, initial links 1, 3, 6, 12, 14, 17, 23, 24 are used as initial links of the sub-network topology, and the network bandwidth information and types of initial links 1, 3, 6, 12, 14, 17, 23, 24 are used as edges of the network topology.

[0134] S1032: Obtain node resources of the graphics processor in each sub-network topology.

[0135] In some embodiments, the node resources of the GPU can be obtained through a GPU metric resource detection process. The node resources of the GPU can include, but are not limited to, the amount of memory, memory usage, core usage, type, etc. of the GPU in the sub-network topology, and the network bandwidth information and type of the initial link connected to the GPU.

[0136] S1033: Calculate and obtain network flow information of each sub-network topology according to the node resources of the graphics processor in each sub-network topology.

[0137] In this step, based on the node resources of the GPU in each sub-network topology, the maximum flow algorithm is used to calculate the network maximum flow of each sub-network topology, and the network maximum flow of each sub-network topology is used as the network flow information of each sub-network topology.

[0138] It should be noted that the maximum network flow refers to the feasible flow with maximum flow that satisfies the edge flow constraint and flow balance conditions in the subnetwork topology. The edge flow constraint condition is defined as the condition that the edge weight c(u,v) of the edge u→v in the subnetwork topology is greater than or equal to the set of all edge flows F=f(u,v), and the set of all edge flows F=f(u,v) is greater than or equal to 0. The flow balance condition is defined as the condition that the flow into a node in the subnetwork topology is equal to the flow out of the node.

[0139] In some embodiments, the maximum flow algorithm can be the Ford-Fulkerson algorithm, or other maximum flow algorithms such as the EK algorithm, the Dinic algorithm, etc. Those skilled in the art can select a suitable maximum flow algorithm according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0140] For example, referring to Figures 7A and 7B, the sub-network topology shown in Figure 7A is calculated by the Ford-Fulkerson algorithm, and the calculated maximum network flow of the sub-network topology is 600G. The sub-network topology shown in Figure 7B is calculated by the Ford-Fulkerson algorithm, and the calculated maximum network flow of the sub-network topology is 32G.

[0141] S1034: Select the sub-network topology with the largest network flow information as the training network topology corresponding to the training request.

[0142] In this step, after obtaining the maximum network flow for all sub-network topologies, the sub-network topology with the highest network maximum flow is selected as the training network topology corresponding to the training request. The training network topology has the initial link with the highest network bandwidth information. This initial link with the highest network bandwidth information is defined as the training link, and all GPUs located on the training link are training processing nodes. This indicates that the training link represents a closed loop with the highest network traffic. The training network topology will serve as the foundation and basis for the orchestration and deployment of large model containers.

[0143] For example, referring to Figures 7A and 7B, the maximum network flow of the sub-network topology shown in Figure 7A is 600G, which means that the maximum network bandwidth of the sub-network topology is 600G, while the maximum network flow of the sub-network topology shown in Figure 7B is 32G, which means that the maximum network bandwidth of the sub-network topology is 32G. Obviously, the sub-network topology shown in Figure 7A is better for training tasks than the sub-network topology shown in Figure 7B, so the sub-network topology shown in Figure 7A is selected as the training network topology.

[0144] In some embodiments of the present application, in step S104, the steps of creating a large model container according to the training network topology may include but are not limited to:

[0145] S1041, obtaining a training device where a graphics processor in the training network topology is located, creating a large model container on the training device, and deploying the graphics processor in the training network topology in the corresponding large model container.

[0146] In this step, the training network topology obtained in the previous step is obtained. The training network topology consists of a training link and a training processing node located on the training link. At least one large model container is deployed on at least one training device according to the training network topology. Each large model container has a corresponding training processing node, namely a GPU node, so as to achieve the effect of specifying each large model container to use a suitable network link for communication, ensuring that each large model container can communicate with the maximum network bandwidth.

[0147] For example, referring to Figure 7A, when the sub-network topology shown in Figure 7A is used as the training network topology, the four GPUs in the sub-network topology are located on the first terminal of the same host PCIE4.0Switch, then the host is a training device, and four large model containers are created on the host. Each large model container deploys a GPU, and the four large model containers are respectively deployed with GPU1 to GPU4.

[0148] As another example, referring to Figure 7B, when the sub-network topology shown in Figure 7B is used as the training network topology, GPU1 and GPU2 of the sub-network topology are located on the first terminal of the same host PCIE4.0Switch, while GPU5 and GPU6 are located on the second terminal of the same host PCIE4.0Switch. Then, the above-mentioned first terminal and second terminal are used as training devices, and two large model containers are created on the first terminal host, one of which is deployed with GPU1 and the other is deployed with GPU2; two large model containers are created on the second terminal host, one of which is deployed with GPU5 and the other is deployed with GPU6.

[0149] Furthermore, in step S1041, the step of creating a large model container on the training device may include:

[0150] Create large model containers on the training device based on Kubernetes.

[0151] It is understandable that Kubernetes is a distributed architecture based on container technology. Its essence is a group of server clusters. It can run specific programs on each node in the cluster to manage the containers in the nodes and realize the automation of resource management.

[0152] It should be noted that in other embodiments of the present application, in addition to Kubernetes, other container deployment technologies such as Docker are also applicable and only need to be adjusted accordingly.

[0153] In some embodiments of the present application, in step S105, the steps of executing the training task corresponding to the training request by configuring the large model container may include but are not limited to:

[0154] S1051: Select any one large model container from the multiple large model containers as a master node container, and use the other large model containers except the master node container as slave node containers.

[0155] In this step, multiple large model containers are clustered together. One container is randomly selected from the cluster as the master node container, while the remaining containers are slave nodes. The master node container is the cluster control node, responsible for cluster management and control; the slave node container is the workload node, which is used to execute tasks assigned by the master node container.

[0156] S1052: Manage multiple slave node containers through the master node container, and execute training tasks corresponding to the training request through the multiple slave node containers.

[0157] In this step, the master node container manages and distributes training tasks to multiple slave node containers, and multiple slave node containers are responsible for running the training tasks corresponding to the training requests, thereby realizing the training of large models.

[0158] Then, another implementation step of the large model container orchestration method based on GPU network topology proposed in an embodiment of the present application will be described in detail below.

[0159] 8 , which is another flow chart of a large model container orchestration method based on GPU network topology provided by the present application, the method may further include the following steps:

[0160] S201: When a stop request is detected, stop executing the training task corresponding to the training request.

[0161] In this step, when an abnormal situation is detected, a stop request will be automatically generated. The stop request is used to pause the ongoing training task. In response to the stop request, the training task corresponding to the training request is stopped.

[0162] It should be noted that the abnormal situation may be a network failure caused by a disconnected link, or other situations such as a GPU failure, a first device failure, etc. This application does not make specific limitations on this.

[0163] S202: Update the device network topology, and update the training network topology according to the updated device network topology.

[0164] In this step, the device network topology is updated by eliminating erroneous training links and / or training processing nodes, and then the updated device network topology is used to update the training network topology, thereby updating the training links and training processing nodes in the training network topology used to perform training tasks.

[0165] S203: Re-execute the training task corresponding to the training request according to the updated training network topology.

[0166] In this step, the automatic recovery of the large model training task is guided according to the updated training network topology, so that the training task corresponding to the training request is re-executed.

[0167] In some embodiments of the present application, the method may further include the following steps:

[0168] When a stop request is detected, the ckpt file corresponding to the training task is stored.

[0169] It should be noted that the ckpt file is a checkpoint file, which stores the parameters and variables of the model within a certain period of time during the training process.

[0170] In some embodiments of the present application, in step S202, the device network topology is updated, and the steps of updating the training network topology according to the updated device network topology may include but are not limited to:

[0171] S2021: Remove the initial link and device corresponding to the stop request from the device network topology, and update the network bandwidth information between multiple devices in the device network topology to generate an updated device network topology.

[0172] In this step, when an abnormal situation exists, it is necessary to first determine the training links and / or training processing nodes with the abnormal situation, and update the device network topology by removing the training links and / or training processing nodes with the abnormal situation from the original device network topology. At the same time, the network bandwidth information between multiple devices is updated to obtain a new device network topology to avoid reselecting the wrong node or link when reselecting the training links and training processing nodes.

[0173] S2022: Construct multiple sub-updated network topologies based on the updated device network topology.

[0174] In this step, a new device network topology is obtained by removing abnormal training links and / or training processing nodes from the original device network topology. The nodes and edges in the new device network topology will change. At this time, the subgraph of the device network topology must be reconstructed, that is, multiple sub-updated network topologies must be reconstructed.

[0175] It can be understood that in the sub-update network topology, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0176] S2023, obtaining network flow information of each sub-updated network topology, and selecting the sub-updated network topology with the largest network flow information as the updated training network topology.

[0177] In this step, the maximum network flow of each sub-update network topology is used as the network flow information for each sub-update network topology. The sub-update network topology with the largest network maximum flow is selected as the updated training network topology. The updated training network topology also has the initial link with the largest network bandwidth information. The initial link with the largest network bandwidth information is the training link, and all GPUs located on the training link are training processing nodes. This shows that the updated training link still has the largest network traffic. The updated training network topology will serve as the basis for re-orchestrating and re-deploying large model containers.

[0178] Furthermore, in step S2023, the step of obtaining network flow information of each sub-updated network topology may include:

[0179] When the multiple sub-update network topologies are different from the multiple sub-network topologies, obtaining update node resources of the graphics processor in each sub-update network topology, and calculating network flow information of each sub-update network topology based on the update node resources of the graphics processor in each sub-update network topology;

[0180] When the multiple sub-update network topologies are the same as the multiple sub-network topologies, the network flow information of the multiple sub-network topologies is used as the network flow information of the multiple sub-update network topologies.

[0181] In some embodiments, the GPU's update node resources can be obtained through a GPU metric resource detection process. The GPU's update node resources include, but are not limited to, the GPU's memory quantity, memory usage, core usage, type, etc., and the network bandwidth information and type of the initial link connected to the GPU in the sub-update network topology.

[0182] In this step, after removing erroneous GPU nodes or links from the device network topology, some of the structure of the updated device network topology will change, while others will remain unchanged. This may result in the generation of new subgraphs with different structures than previously constructed subgraphs when constructing subgraphs of the updated device network topology, as well as the generation of some old subgraphs with the same structure as previously constructed subgraphs. Therefore, this step obtains network flow information for the subgraphs in different ways, depending on whether they are new or old.

[0183] If multiple sub-update network topologies differ from multiple sub-sub-network topologies, this indicates that a new sub-graph was generated when constructing the sub-graph of the updated device network topology. In this case, the network maximum flow of each sub-graph needs to be recalculated. Specifically, based on the GPU update node resources in each sub-update network topology, a maximum flow algorithm is used to calculate the network maximum flow of each sub-update network topology. The network maximum flow of each sub-update network topology is used as the network flow information for each sub-update network topology.

[0184] In some embodiments, the maximum flow algorithm can be the Ford-Fulkerson algorithm, or other maximum flow algorithms such as the EK algorithm, the Dinic algorithm, etc. Those skilled in the art can select a suitable maximum flow algorithm according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0185] When multiple sub-updated network topologies are the same as multiple sub-network topologies, no new sub-graph is generated when constructing the sub-graph of the updated device network topology. At this time, the network flow information of the multiple sub-network topologies can be directly obtained as the network flow information of the multiple sub-updated network topologies, omitting the step of using the maximum flow algorithm to recalculate the network maximum flow.

[0186] In some embodiments of the present application, referring to FIG. 9 , which is a flowchart of re-executing a training task provided by the present application, in step S203 , the steps of re-executing the training task corresponding to the training request according to the updated training network topology may include but are not limited to:

[0187] S2031: Rearrange the large model container according to the updated training network topology.

[0188] In this step, an updated training network topology is obtained. The updated training network topology consists of new training links and training processing nodes located on the new training links. Based on the new training network topology, at least one large model container is deployed on at least one training device. Each large model container has a corresponding training processing node, namely a GPU. This ensures that each large model container uses the appropriate network link for communication, ensuring that each large model container can communicate with the maximum network bandwidth. The implementation of this step is the same as that of the previous step S1041 and will not be repeated here.

[0189] S2032: Determine whether the updated training network topology meets the training requirements.

[0190] It should be noted that training requirements can be understood as the minimum network bandwidth and number of GPU nodes required for training tasks.

[0191] In this step, the number of training processing nodes and the maximum bandwidth of the training link in the updated training network topology are obtained. By determining whether the number of training processing nodes in the updated training network topology meets the first value and whether the network bandwidth information of the training link in the updated training network topology meets the second value, it is determined whether the updated training network topology meets the training requirements.

[0192] S2033: When the number of graphics processors in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, re-execute the training task corresponding to the training request by configuring the large model container.

[0193] In this step, when the number of graphics processors in the updated training network topology is greater than or equal to the first value, and the network flow information of the updated training network topology is greater than or equal to the second value, it means that the updated training network topology meets the training requirements. At this time, the training task corresponding to the training request is re-executed by configuring the large model container.

[0194] It should be noted that the first value and the second value can be set according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0195] S2034: When the number of graphics processors in the updated training network topology is less than the first value and / or the network flow information of the updated training network topology is less than the second value, it indicates that the updated training network topology does not meet the training requirements, and the training task is terminated.

[0196] Furthermore, in step S2033, the step of re-executing the training task corresponding to the training request by configuring the large model container may include:

[0197] First, any one large model container is selected from the rearranged plurality of large model containers as a master node container, and the other large model containers except the master node container are selected as slave node containers.

[0198] Then, the master node container manages multiple slave node containers, and the multiple slave node containers call the ckpt files corresponding to the training tasks, and re-execute the training tasks corresponding to the training requests.

[0199] In the above steps, the reorganized large model containers are used as a cluster. One container is randomly selected from the cluster as the master node container, while the remaining containers are slave nodes. The master node container is the cluster control node, responsible for cluster management and control; the slave node containers are workload nodes, responsible for executing tasks assigned by the master node container. The master node container then manages and distributes training tasks to the multiple slave node containers. The training tasks corresponding to the training request are then re-executed through the multiple slave node containers, thereby resuming training of the large model.

[0200] The following example illustrates the implementation process of the large model container orchestration method based on GPU network topology provided in an embodiment of the present application. The implementation process is as follows:

[0201] The first step is network bandwidth detection: detecting network bandwidth information between multiple GPUs, network bandwidth information between multiple first devices, and network bandwidth information between a GPU and a first device.

[0202] The second step is to build a device network topology: Based on the network bandwidth information, a global device network topology is constructed. In this device network topology, the GPU and the first device serve as nodes, the channel connecting the two devices serves as the initial link, and the network bandwidth information and type of each initial link serve as the edge of the network topology.

[0203] The third step is to configure the training network graph topology: First, obtain a training request, which is configured with a corresponding training task. Then, when scheduling large model training tasks, use the device network topology as a basis to build multiple sub-network topologies. The number of sub-network topologies is the same as the number of GPUs required for the training task. The maximum flow algorithm is used to calculate the network maximum flow of each sub-network topology, and the sub-network topology with the maximum network maximum flow is selected as the training network topology. The training network topology consists of a training link with the maximum network bandwidth and multiple training processing nodes (i.e., GPU nodes) located on the training link.

[0204] The fourth step is the arrangement and configuration of the large model container: First, obtain the training device where the GPU in the training network topology is located, create a large model container on the training device, and deploy the GPU in the training network topology in the corresponding large model container. In some embodiments, each large model container is deployed with a training processing node to achieve the effect of specifying each large model container to use a suitable network link for communication, ensuring that each large model container can communicate with the maximum network bandwidth. Then, select any one large model container from multiple large model containers as the master node container, and use other large model containers except the master node container as slave node containers. Among them, the master node container is used to manage and control the slave node container; the slave node container is used to execute tasks assigned by the master node container.

[0205] The fifth step is to execute the large model training task: the master node container is used to manage and control multiple slave node containers and allocate training tasks. Multiple slave node containers are responsible for running the training tasks corresponding to the training requests, thereby realizing the training of the large model.

[0206] Step 6: Anomaly Detection: Determine whether a stop request is detected during large model training. Specifically, if an anomaly is detected during large model training, a stop request is automatically generated to pause the ongoing training task. This anomaly can be a network failure caused by a disconnected link, or other issues such as a GPU failure or a primary device failure.

[0207] Step 7: Rearrange the large model container: When a stop request is detected, stop executing the training task corresponding to the training request. Then, update the device network topology by eliminating erroneous training links and / or training processing nodes. Then, based on the updated device network topology, reconstruct multiple sub-update network topologies and select the sub-update network topology with the maximum network flow value as the new training network topology.

[0208] Step 8: Training requirement detection: Determine whether the updated training network topology meets the training requirements.

[0209] Step 9: Resuming the training task: If the number of GPUs in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, the updated training network topology meets the training requirements. At this time, the training task corresponding to the training request is re-executed by configuring the large model container. Otherwise, the training task is stopped.

[0210] From the above, it can be seen that, on the one hand, the embodiment of the present application orchestrates and deploys large model containers from the perspective of GPU granularity. First, the network bandwidth information between multiple GPUs, the network bandwidth information between multiple first devices, and the network bandwidth information between the GPU and the first device are detected, and a device network topology is constructed based on the network bandwidth information. Among them, in the device network topology, the GPU and the first device are used as nodes of the network topology, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as the edge of the network topology. After obtaining the topological information of the GPU network and the link bandwidth between GPUs, when scheduling large model training tasks, the device network topology is used as the basis for network performance, and the sub-network topology with the largest network flow information is selected from the multiple sub-network topologies (subgraphs) of the device network topology. The large model container is reasonably orchestrated and deployed based on the sub-network topology with the largest network flow information, and the training container uses a suitable network link to communicate. In the embodiment of the present application, the GPU and the first device are used as nodes, and the network bandwidth information and type of each initial link are used as edges to construct a device network topology. The network bandwidth between the GPUs is discovered through the device network topology, so that the orchestration of containers can meet the bandwidth requirements between the GPUs, thereby improving the training performance and training efficiency of large model training while also improving the utilization of training resources.

[0211] On the other hand, when a network failure or link change occurs, the embodiment of the present application pauses the current training task by responding to a stop request, updates the device network topology, and searches for a sub-updated network topology of a link with the maximum network bandwidth information in the updated device network topology. When the newly found sub-updated network topology meets the training requirements, the training of the large model is restarted, thereby achieving automatic recovery of the large model training. The embodiment of the present application can automatically diagnose the fault, adjust the faulty training node, and re-arrange the container without manual intervention when a network link or graphics processor fails, thereby achieving automatic recovery of the large model training, reducing the maintenance cost and manpower investment of the equipment while improving the success rate and training efficiency of the large model training.

[0212] In addition, referring to FIG10 , FIG10 is a structural diagram of a large model container orchestration system based on a GPU network topology provided by the present application. The embodiment of the present application further provides a large model container orchestration system based on a GPU network topology, the system comprising:

[0213] The detection module 301 is configured to detect network bandwidth information between multiple devices.

[0214] The topology processing module 302 is configured to construct a device network topology based on the network bandwidth information, using the graphics processor and the first device as nodes, the channel connecting the two devices as an initial link, and the network bandwidth information and type of each initial link as an edge.

[0215] An acquisition module 303 is used to acquire a training request;

[0216] The node processing module 304 is configured to select a training network topology corresponding to the training request according to the device network topology.

[0217] The container processing module 305 is used to create a large model container according to the training network topology.

[0218] The training processing module 306 is used to execute the training task corresponding to the training request by configuring the large model container.

[0219] In some embodiments of the present application, the system may further include:

[0220] The fault handling module is used to stop executing the training task corresponding to the training request when a stop request is detected, update the device network topology, update the training network topology using the updated device network topology, and re-execute the training task corresponding to the training request based on the updated training network topology.

[0221] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0222] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to execute the large model container orchestration method based on GPU network topology described above.

[0223] Similarly, the contents of the above method embodiments are applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0224] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flow chart of the present application are provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0225] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present application as set forth in the claims using ordinary techniques without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0226] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several programs for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0227] The logic and / or steps represented in a flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, a program execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can retrieve and execute a program from a program execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, a program execution system, apparatus, or device.

[0228] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0229] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0230] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0231] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0232] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A large model capacity orchestration method based on GPU network topology, including: Detecting network bandwidth information between a plurality of devices, the devices including a graphics processor and a first device; According to the network bandwidth information, a device network topology is constructed with the graphics processor and the first device as nodes, a channel connecting the two devices as an initial link, and the network bandwidth information and type of each initial link as an edge; Obtaining a training request, and determining a training network topology corresponding to the training request according to the device network topology; Creating a large model container according to the training network topology; The training task corresponding to the training request is executed by configuring the large model container.

2. The large model capacity arrangement method based on GPU network topology according to claim 1, wherein: The detecting of network bandwidth information between multiple devices includes: The network bandwidth information of each initial link is traversed to obtain the network bandwidth information between multiple devices.

3. The large model capacity arrangement method based on GPU network topology according to claim 1, wherein: The determining, according to the device network topology, a training network topology corresponding to the training request includes: According to the device network topology, construct multiple sub-network topologies; Get the node resources of the GPU in each sub-network topology; The network flow information of each sub-network topology is calculated according to the node resources of the graphics processor in each sub-network topology, and the sub-network topology with the largest network flow information is selected as the training network topology corresponding to the training request.

4. The large model capacity arrangement method based on GPU network topology according to claim 3, wherein: The step of creating a large model container according to the training network topology includes: A training device where a graphics processor in the training network topology is located is obtained, a large model container is created on the training device, and the graphics processor in the training network topology is deployed in the corresponding large model container.

5. The large model capacity arrangement method based on GPU network topology according to claim 4, wherein: The configuring the large model container to execute the training task corresponding to the training request includes: Select any one large model container from the plurality of large model containers as a master node container, and use other large model containers except the master node container as slave node containers; The master node container manages the plurality of slave node containers, and the plurality of slave node containers execute the training tasks corresponding to the training request.

6. The large model capacity arrangement method based on GPU network topology according to claim 1, further comprising: When a stop request is detected, stopping execution of the training task corresponding to the training request; Updating the device network topology, and updating the training network topology according to the updated device network topology; Re-execute the training task corresponding to the training request according to the updated training network topology.

7. The large model capacity arrangement method based on GPU network topology according to claim 6, wherein: The updating of the device network topology, and updating the training network topology according to the updated device network topology, includes: Eliminating the initial link and device corresponding to the stop request from the device network topology, and updating the network bandwidth information between the multiple devices in the device network topology to generate an updated device network topology; According to the updated device network topology, multiple sub-updated network topologies are constructed; The network flow information of each sub-updated network topology is obtained, and the sub-updated network topology with the largest network flow information is selected as the updated training network topology.

8. The large model capacity arrangement method based on GPU network topology according to claim 7, wherein: The re-executing the training task corresponding to the training request according to the updated training network topology includes: Rearrange the large model container according to the updated training network topology; When the number of graphics processors in the updated training network topology satisfies the first value and the network flow information of the updated training network topology satisfies the second value, the training task corresponding to the training request is re-executed by configuring the large model container.

9. The large model capacity arrangement method based on GPU network topology according to claim 2, wherein: The type of the initial link includes one or more of a PCIe interconnect link, an NVLink link, an NVSwitch link, an IB network link, and an Ethernet link.

10. The large model capacity arrangement method based on GPU network topology according to claim 3, wherein: In the sub-network topology, the graphics processor and the first device are used as nodes, a channel connecting the two devices is used as an initial link, and network bandwidth information and type of each initial link are used as edges.

11. The large model capacity arrangement method based on GPU network topology according to claim 3, wherein: The node resources of the graphics processor include the memory quantity, memory usage, core usage, type of the graphics processor and network bandwidth information and type of the initial link connected to the graphics processor in the device network topology.

12. The large model capacity arrangement method based on GPU network topology according to claim 3, wherein: The calculating and obtaining the network flow information of each sub-network topology according to the node resources of the graphics processor in each sub-network topology includes: According to the node resources of the graphics processor in each sub-network topology, the maximum flow algorithm is used to calculate the network maximum flow of each sub-network topology, and the network maximum flow of each sub-network topology is used as the network flow information of each sub-network topology.

13. The large model capacity arrangement method based on GPU network topology according to claim 12, wherein: The maximum flow algorithm is the Ford-Fulkerson algorithm.

14. The large model capacity arrangement method based on GPU network topology according to claim 4, wherein: The step of creating a large model container on the training device comprises: A large model container is created on the training device based on Kubernetes.

15. The large model capacity arrangement method based on GPU network topology according to claim 7, wherein: In the sub-update network topology, the graphics processor and the first device are used as nodes, a channel connecting the two devices is used as an initial link, and network bandwidth information and type of each initial link are used as edges.

16. The large model capacity arrangement method based on GPU network topology according to claim 7, wherein: The obtaining of network flow information of each sub-updated network topology includes: When multiple sub-update network topologies are different from multiple sub-network topologies, the update node resources of the graphics processor in each sub-update network topology are obtained, and the network flow information of each sub-update network topology is calculated according to the update node resources of the graphics processor in each sub-update network topology.

17. The large model capacity arrangement method based on GPU network topology according to claim 16, wherein: The updated node resources of the graphics processor include, in the updated device network topology, the memory quantity, memory usage, core usage, type of the graphics processor, and network bandwidth information and type of the initial link connected to the graphics processor.

18. The large model capacity arrangement method based on GPU network topology according to claim 16, wherein: The calculating and obtaining the network flow information of each sub-update network topology according to the update node resources of the graphics processor in each sub-update network topology includes: According to the update node resources of the graphics processor in each sub-update network topology, the maximum flow of each sub-update network topology is calculated by using the maximum flow algorithm, and the network maximum flow of each sub-update network topology is used as the network flow information of each sub-update network topology.

19. The large model capacity arrangement method based on GPU network topology according to claim 7, wherein: The obtaining of network flow information of each sub-updated network topology includes: When the multiple sub-update network topologies are the same as the multiple sub-network topologies, the network flow information of the multiple sub-network topologies is used as the network flow information of the multiple sub-update network topologies.

20. The large model capacity arrangement method based on GPU network topology according to claim 1, further comprising: When a stop request is detected, a ckpt file corresponding to the training task is stored.

21. Large model capacity orchestration system based on GPU network topology, including: include: A detection module, configured to detect network bandwidth information between a plurality of devices, wherein the devices include a graphics processor and a first device; A topology processing module, configured to construct a device network topology according to the network bandwidth information, using the graphics processor and the first device as nodes, a channel connecting the two devices as an initial link, and network bandwidth information and type of each initial link as an edge; An acquisition module is used to obtain training requests; A node processing module, configured to determine a training network topology corresponding to the training request according to the device network topology; A container processing module, used to create a large model container according to the training network topology; The training processing module is used to execute the training task corresponding to the training request by configuring the large model container.

22. A computer-readable storage medium storing a program executable by a processor, wherein: The processor-executable program is used to implement the large model capacity orchestration method based on GPU network topology as described in any one of claims 1 to 20 when executed by the processor.

Images