Method and system for model capacity orchestration based on gpu network topology, and medium

Abstract

The application discloses a GPU network topology-based large model capacity arrangement method and system and a medium. The method comprises the following steps: detecting network bandwidth information among a plurality of devices, wherein the devices comprise GPUs and first devices; constructing a device network topology according to the network bandwidth information, taking the GPUs and the first devices as nodes, taking channels connecting two devices as initial links, and taking network bandwidth information and types of each initial link as edges; obtaining a training request; selecting a training network topology corresponding to the training request according to the device network topology; creating a large model container according to the training network topology; and executing a training task corresponding to the training request by configuring the large model container. The application can discover network bandwidth among the GPUs through the device network topology, so that the arrangement of the container can meet the bandwidth demand among the GPUs, and the training performance and training efficiency of the large model training and the utilization rate of training resources are improved. The application is applied to the technical field of large model training.

Description

Technical Field

[0001] This invention relates to the field of large model training technology, and in particular to a method, system and medium for large model capacity orchestration based on GPU network topology. Background Technology

[0002] In large-scale deep learning training, Graphics Processing Units (GPUs) play a crucial role, handling massive amounts of computation and data. Common GPU configurations include single-machine multi-GPU setups and multi-machine multi-GPU setups. As the amount of data in the model increases, the bandwidth requirements between single-machine multi-GPU setups and multi-machine multi-GPU setups also increase. Related technologies utilize communication methods such as PCIe interconnect, NVLink, and NVSwitch to meet these bandwidth requirements. Furthermore, in large-scale deep learning training, related technologies typically deploy large model training nodes using containerization. Containerization refers to running multiple virtual machines on a single physical machine, with each virtual machine sharing an operating system and operating as an independent environment. Containerization prevents these environments from interfering with each other while improving operational security.

[0003] However, the communication methods employed by these technologies result in complex graphics processor network topologies and hierarchical structures in real-world production clusters. In this context, these technologies construct the network topology of physical nodes using physical machines such as Ethernet switches as the lowest granularity. This network topology cannot detect the bandwidth between graphics processors, which leads to the container orchestration failing to meet the bandwidth requirements between graphics processors when containerizing deployments based on the network topology. Consequently, the training performance of training nodes for large models degrades, while also wasting network bandwidth between graphics processors.

[0004] The above problems urgently need to be solved. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, the purpose of this invention is to provide a method, system, and medium for large model capacity orchestration based on GPU network topology.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a method for orchestrating large model capacity based on GPU network topology, including the following steps: Detecting network bandwidth information among multiple devices, the devices including a graphics processor and a first device; Based on the network bandwidth information, the device network topology is constructed using the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as edges. Obtain a training request, and determine the training network topology corresponding to the training request based on the device network topology; Based on the training network topology, create a large model container; By configuring the large model container, the training task corresponding to the training request is executed.

[0008] The large model capacity orchestration method based on GPU network topology according to the above embodiments of the present invention may also have the following additional technical features: Furthermore, in one embodiment of the present invention, detecting network bandwidth information among multiple devices includes: By iterating through the network bandwidth information of each initial link, the network bandwidth information between multiple devices can be obtained.

[0009] Furthermore, in one embodiment of the present invention, the type of the initial link includes one or more of PCIe interconnect link, NVLink link, NVSwitch link, IB network link and Ethernet link.

[0010] Further, in one embodiment of the present invention, determining the training network topology corresponding to the training request based on the device network topology includes: Based on the device network topology, construct multiple sub-network topologies; Obtain the node resources of the graphics processor in each sub-network topology; Based on the node resources of the graphics processor in each sub-network topology, the network flow information of each sub-network topology is calculated, and the sub-network topology with the largest network flow information is selected as the training network topology corresponding to the training request.

[0011] Furthermore, in one embodiment of the present invention, in the sub-network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0012] Furthermore, in one embodiment of the present invention, the node resources of the graphics processor include, in the device network topology, the amount of memory, memory usage, core usage, type, and network bandwidth information and type of the initial link connected to the graphics processor.

[0013] Further, in one embodiment of the present invention, the step of calculating the network flow information of each sub-network topology based on the node resources of the graphics processor in each sub-network topology includes: Based on the node resources of the graphics processor in each sub-network topology, the maximum flow of each sub-network topology is calculated using the maximum flow algorithm, and the maximum flow of each sub-network topology is used as the network flow information of each sub-network topology.

[0014] Furthermore, in one embodiment of the present invention, the maximum flow algorithm is the Ford-Fulkerson algorithm.

[0015] Furthermore, in one embodiment of the present invention, creating a large model container based on the training network topology includes: Obtain the training device where the graphics processor in the training network topology is located, create a large model container on the training device, and deploy the graphics processor in the training network topology in the corresponding large model container.

[0016] Furthermore, in one embodiment of the present invention, creating a large model container on the training device includes: Large model containers are created on the training device based on Kubernetes.

[0017] Furthermore, in one embodiment of the present invention, the step of configuring the large model container to execute the training task corresponding to the training request includes: Select any one of the large model containers as the master node container, and use the other large model containers besides the master node container as slave node containers; The master node container manages multiple slave node containers, and the multiple slave node containers execute training tasks corresponding to the training request.

[0018] Furthermore, in one embodiment of the present invention, the method further includes the following steps: When a stop request is detected, the training task corresponding to the training request is stopped. Update the device network topology, and update the training network topology based on the updated device network topology; Based on the updated training network topology, the training task corresponding to the training request is re-executed.

[0019] Furthermore, in one embodiment of the present invention, updating the device network topology, and updating the training network topology according to the updated device network topology, includes: Remove the initial link and device corresponding to the stop request from the device network topology, and update the network bandwidth information between multiple devices in the device network topology to generate an updated device network topology; Based on the updated device network topology, construct multiple sub-update network topologies; Obtain the network flow information for each sub-updated network topology, and select the sub-updated network topology with the largest network flow information as the updated training network topology.

[0020] Furthermore, in one embodiment of the present invention, in the sub-update network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0021] Furthermore, in one embodiment of the present invention, obtaining the network flow information for each sub-updated network topology includes: When multiple sub-update network topologies are different from multiple sub-network topologies, the update node resources of the graphics processor in each sub-update network topology are obtained, and the network flow information of each sub-update network topology is calculated based on the update node resources of the graphics processor in each sub-update network topology. Furthermore, in one embodiment of the present invention, the updated node resources of the graphics processor include, in the updated device network topology, the amount of memory, memory usage, core usage, type, and network bandwidth information and type of the initial link connected to the graphics processor.

[0022] Further, in one embodiment of the present invention, the step of calculating the network flow information of each sub-updated network topology based on the update node resources of the graphics processor in each sub-updated network topology includes: Based on the update node resources of the graphics processor in each sub-update network topology, the maximum flow of each sub-update network topology is calculated using the maximum flow algorithm, and the maximum flow of each sub-update network topology is used as the network flow information of each sub-update network topology.

[0023] Furthermore, in one embodiment of the present invention, the maximum flow algorithm is the Ford-Fulkerson algorithm.

[0024] Furthermore, in one embodiment of the present invention, obtaining the network flow information for each sub-updated network topology includes: When multiple sub-updated network topologies are the same as multiple sub-network topologies, the network flow information of the multiple sub-network topologies is used as the network flow information of the multiple sub-updated network topologies.

[0025] Furthermore, in one embodiment of the present invention, the step of re-executing the training task corresponding to the training request based on the updated training network topology includes: The large model container is rearranged based on the updated training network topology; When the number of graphics processors in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, the training task corresponding to the training request is re-executed by configuring the large model container.

[0026] Furthermore, in one embodiment of the present invention, the method further includes the following steps: When a stop request is detected, the ckpt file corresponding to the training task is stored.

[0027] Furthermore, in one embodiment of the present invention, the re-execution of the training task corresponding to the training request includes: Call the ckpt file corresponding to the training task and re-execute the training task corresponding to the training request.

[0028] On the other hand, embodiments of the present invention provide a large model capacity orchestration system based on GPU network topology, including: A detection module is used to detect network bandwidth information between multiple devices, wherein the devices include a graphics processor and a first device; The topology processing module is used to construct a device network topology based on the network bandwidth information, using the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as the edge. The acquisition module is used to acquire training requests; The node processing module is used to determine the training network topology corresponding to the training request based on the device network topology; A container processing module is used to create large model containers based on the training network topology; The training processing module is used to execute the training task corresponding to the training request by configuring the large model container.

[0029] In another aspect, embodiments of the present invention provide a storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the aforementioned large model capacity orchestration method based on GPU network topology.

[0030] The beneficial effects of this invention are: it provides a method, system, and medium for orchestrating large-scale models based on GPU network topology. The method includes first detecting network bandwidth information between multiple devices, including a graphics processor and a first device; then constructing a device network topology based on the network bandwidth information; subsequently obtaining a training request; selecting a training network topology corresponding to the training request based on the device network topology; creating a large model container based on the training network topology; and finally executing the training task corresponding to the training request by configuring the large model container. This invention uses the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as edges to construct the device network topology. By discovering the network bandwidth between graphics processors through the device network topology, the orchestration of containers can meet the bandwidth requirements between graphics processors, thereby improving the training performance and efficiency of large-scale model training while also increasing the utilization rate of training resources.

[0031] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description and the accompanying drawings. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the network topology of related technologies at the host node level; Figure 2 This is a flowchart of a large model container orchestration method based on GPU network topology provided by the present invention; Figure 3 This is a schematic diagram of the device network topology provided by the present invention; Figure 4 This is a schematic diagram of the initial link provided by the present invention; Figure 5 This is a flowchart of the detection link bandwidth information provided by the present invention; Figure 6 This is a flowchart of the selection of training network topology provided by the present invention; Figure 7A This is a schematic diagram of a sub-network topology provided by the present invention; Figure 7B This is another schematic diagram of the sub-network topology provided by the present invention; Figure 8 This is another flowchart of the large model container orchestration method based on GPU network topology provided by the present invention; Figure 9 This is a flowchart of the re-execution of the training task provided by the present invention; Figure 10This is a structural diagram of the large model container orchestration system based on GPU network topology provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0037] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0038] 1) GPU (Graphics Processing Unit), also known as a display core, visual processor, or display chip, is a hardware device specifically designed for processing and rendering graphics data. It is a high-performance parallel computing engine used to accelerate graphics, video, and other computationally intensive tasks, and is an essential hardware device for training large models.

[0039] 2) PCIe (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard. It belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission. Devices connected via PCIe are allocated dedicated channel bandwidth and do not share the bus bandwidth. PCIe features high data transmission rate and high data transmission reliability.

[0040] 3) NVLink is a high-speed point-to-point connection interface that uses a point-to-point structure and serial transmission. It is used for connections between the CPU (Central Processing Unit) and the GPU, and can also be used for interconnection between multiple GPUs. NVLink technology aims to provide a high-bandwidth and low-latency communication channel, allowing the GPU to access remote GPU memory without going through the PCIe bus.

[0041] 4) NVSwitch is a many-to-many GPU communication method that integrates multiple NVLinks to achieve many-to-many GPU communication within a single node at the high speed of NVLink, thereby further improving interconnect performance. The combination of NVLink and NVSwitch efficiently extends AI performance to multiple GPUs, with each GPU having multiple NVLink links connected to the NVSwitch, enabling high-speed many-to-many communication.

[0042] 5) InfiniBand (IB) networks are a computer network communication standard for high-performance computing with low latency. They offer extremely high throughput and very low latency, primarily used for data interconnection between computers. In addition, IB networks are also used for direct or switched interconnection between servers and storage systems, as well as interconnection between storage systems.

[0043] 6) Checkpoint refers to saving the intermediate state of the model during training. By creating checkpoint files during training, the model's parameters and optimizer state are saved periodically so that training can be resumed and the model can be evaluated or used when needed.

[0044] 7) Large models refer to models with a large number of parameters, deep hierarchies, and high complexity. These models typically require significant computational resources and storage space for training and inference, and can achieve better performance and results in certain tasks. Large models have more degrees of freedom and expressive power, enabling them to better fit and capture complex data patterns and regularities.

[0045] 8) Network-Flows is a problem-solving method analogous to water flow, closely related to linear programming. It is defined as follows: for a given directed graph, if there exists a unique source and sink, and each edge in the graph has a non-negative capacity, then the directed graph is called a network-flow graph.

[0046] In large-scale deep learning training, GPUs play a crucial role, handling massive amounts of computation and data. Common GPU configurations include single-machine multi-GPU setups and multi-machine multi-GPU setups. As the amount of data in the model increases, the bandwidth requirements between single-machine multi-GPU setups and multi-machine multi-GPU setups also increase. Related technologies address these bandwidth requirements through communication methods such as PCIe interconnect, NVLink, NVSwitch, and GPUDirect RDMA. GPUDirect RDMA refers to a communication method that enables multi-GPU communication via an IB network. Furthermore, in large-scale deep learning training, related technologies often deploy large model training nodes using containerization. Containerization involves running multiple virtual machines on a single physical machine, with each virtual machine sharing an operating system and operating as an independent environment. Containerization prevents these environments from interfering with each other while improving operational security.

[0047] However, the communication methods employed by these technologies result in complex GPU network topologies and hierarchical structures in real-world production clusters. In such cases, these technologies construct the network topology of physical nodes using physical machines, such as Ethernet switches, as the lowest granularity. For example, see... Figure 1 , Figure 1 This diagram illustrates a network topology based on host nodes, where multiple GPUs are integrated into a single host, with these hosts acting as nodes and the bandwidth between them forming edges. This network topology fails to detect the bandwidth between GPUs. Consequently, when containerizing deployments based on this topology, container orchestration may not meet the bandwidth requirements between GPUs, leading to decreased training performance for large models and wasted network bandwidth between GPUs.

[0048] In addition, when the network link or GPU fails, large model training will fail. Related technologies usually rely on manual intervention to detect the fault and restore communication and large model training. They cannot automatically troubleshoot and adjust the faulty training nodes, nor can they automatically resume training. This increases the maintenance cost of the equipment and reduces the success rate and training efficiency of large model training.

[0049] To address the problems and deficiencies in related technologies, this invention provides a method, system, and medium for orchestrating large-scale models based on GPU network topology. First, using GPUs and a first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of the initial link as edges, a device network topology is constructed as the basis for orchestrating and deploying large-scale model containers. Then, when a large-scale model training request is received, GPU nodes located on the initial link with the maximum network bandwidth information can be dynamically located based on the device network topology. Finally, the large-scale model containers are rationally deployed and orchestrated on these GPU nodes, thereby achieving large-scale model training. Furthermore, when network failures or link changes occur, this invention suspends the current training task by responding to a stop request, re-finds the link with the maximum network bandwidth information, and updates the GPU nodes used for training. When the re-founded link and GPU nodes meet the training requirements, large-scale model training restarts, achieving automatic recovery of large-scale model training.

[0050] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0051] First, the following will describe in detail one implementation step of the large model container orchestration method based on GPU network topology proposed in the embodiments of the present invention.

[0052] The large-scale container orchestration method based on GPU network topology provided in this invention can be applied to terminals, servers, or software running on either terminal or server. Terminals can be tablets, laptops, desktop computers, etc., but are not limited to these. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Furthermore, a server can be a node server in a blockchain network, but is not limited to these. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0053] Reference Figure 2 , Figure 2 This is a flowchart of a large model container orchestration method based on GPU network topology provided by the present invention. The large model container orchestration method based on GPU network topology provided by the present invention may include, but is not limited to, the following steps: S101 detects network bandwidth information between multiple devices.

[0054] It should be noted that the device includes the GPU and the first device.

[0055] Optionally, there may be at least one first device, which may include, but is not limited to, Ethernet switches, CPUs, NVLink bridges, etc. Those skilled in the art will understand that other devices such as IB switches, NVSwitch bridges, and PCIe interconnected terminals are also applicable, requiring only corresponding adjustments. The PCIe interconnected terminal may be a PCIe 3.0 interconnected terminal or a PCIe 4.0 interconnected terminal; this embodiment of the invention does not specifically limit the type of interconnected terminal.

[0056] In this step, network bandwidth information between multiple GPUs, between multiple first devices, and between a GPU and a first device are obtained to serve as the network bandwidth information between multiple devices. Here, network bandwidth refers to the amount of data that can be transmitted per unit time; the larger the network bandwidth, the stronger the communication capability.

[0057] S102, construct the device network topology based on network bandwidth information.

[0058] Existing technologies construct network topologies for physical nodes at the lowest granularity, such as Ethernet switches, failing to discover bandwidth between GPUs. This leads to container orchestration failing to meet the bandwidth requirements between GPUs during containerized deployment based on the network topology. To address this issue and its shortcomings, this step proposes a device network topology from a GPU-level perspective, capable of discovering bandwidth between multiple GPUs and providing bandwidth information for subsequent containerized deployment. Specifically, this step uses GPUs and a first device as nodes in the network topology, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as edges in the network topology, thereby constructing a global device network topology.

[0059] It should be noted that the channels connecting the two devices include channels connecting the two GPUs, channels connecting the GPU and the first device, and channels connecting the two first devices.

[0060] For example, refer to Figure 3 , Figure 3 This is a schematic diagram of the device network topology provided by the present invention. The device includes eight GPUs and four first devices. The eight GPUs are designated as GPU1 to GPU8, and the four first devices are an NVSwitch bridge, a PCIe 4.0 Switch first terminal, a PCIe 4.0 Switch second terminal, and an IB switch. The specific implementation process of constructing the device network topology is as follows: First, GPU1 to GPU8 are determined as nodes in the device network topology. Then, the initial links are determined, specifically, referring to... Figure 4 , Figure 4This is a schematic diagram of the initial links provided by the present invention. Channels connecting multiple GPUs and channels connecting GPUs to a first device are used as initial links, resulting in twenty-four initial links. Then, the network bandwidth information and type of each initial link are used as edges. For example, if the network bandwidth information of initial link 1 is 600G and the type of initial link 1 is an NVLink link, then 600G and NVLink are used as the corresponding edges for initial link 1. Finally, based on the determined nodes and edges, the following is constructed: Figure 3 The device network topology is shown.

[0061] S103, Obtain the training request, and select the training network topology corresponding to the training request based on the device network topology.

[0062] It should be noted that the training request is configured with a corresponding training task.

[0063] It is understood that the training task can be the training task of large models such as PaLM2, GPT-3, LLaMA, etc. Large models can be applied to the fields of natural language processing, computer vision, autonomous driving, etc. Those skilled in the art can select training tasks according to the actual situation, and the embodiments of the present invention do not make specific limitations in this regard.

[0064] In this step, the device network topology is used as the basis and basis for network performance. The initial link with the maximum network bandwidth information between multiple GPUs is found from the device network topology. Based on the initial link with the maximum network bandwidth information, multiple GPUs are selected as training processing nodes corresponding to the training request. These training processing nodes and the initial link with the maximum network bandwidth information together constitute the training network topology.

[0065] S104: Create a large model container based on the training network topology.

[0066] In this step, based on the training network topology obtained in the previous steps, multiple large model containers are created and each large model container is specified to use an appropriate network link for communication.

[0067] S105 executes the training task corresponding to the training request by configuring a large model container.

[0068] In some embodiments of the present invention, the step of detecting network bandwidth information among multiple devices in step S101 may include, but is not limited to: S1011, traverse the network bandwidth information of each initial link, and then obtain the network bandwidth information between multiple devices.

[0069] It should be noted that the network bandwidth information between multiple devices may include, but is not limited to, the network bandwidth information between multiple GPUs, the network bandwidth information between multiple first devices, and the network bandwidth information between a GPU and a first device.

[0070] Optionally, the type of the initial link may include one or more of PCIe interconnect links, NVLink links, NVSwitch links, IB network links, and Ethernet links. Those skilled in the art can make appropriate selections according to actual circumstances, and the present invention does not impose specific limitations on this.

[0071] In this step, the network bandwidth information of each initial link is traversed. The initial link is a channel connecting two devices, thereby obtaining the network bandwidth information between multiple GPUs, the network bandwidth information between the GPU and the first device, and the network bandwidth information between multiple first devices.

[0072] Optionally, the detection of network bandwidth information for each initial link can be performed in parallel or serially, and this embodiment of the invention does not specifically limit this.

[0073] For example, taking the parallel detection of network bandwidth information for each initial link as an example, refer to... Figure 5 , Figure 5 This is a flowchart of the detection of link bandwidth information provided by the present invention. The step of detecting the network bandwidth information of each initial link includes at least one of the following: When two devices are connected to communicate via a PCIe interconnect link, the network bandwidth information of the end-to-end PCIe interconnect link is detected; When two devices are connected to communicate via an NVLink link, the network bandwidth information of the end-to-end NVLink link is detected; When two devices communicate via an NVSwitch link, detect the network bandwidth information of the end-to-end NVSwitch link; When two devices are connected to communicate via an IB network link, the network bandwidth information of the end-to-end IB network link is detected; When two devices are connected to communicate via an Ethernet link, the network bandwidth information of the end-to-end Ethernet link is detected.

[0074] In some embodiments of the present invention, reference is made to... Figure 6 , Figure 6 This is a flowchart of the selection of training network topology provided by the present invention. In step S103, the step of determining the training network topology corresponding to the training request based on the device network topology may include, but is not limited to: S1031, construct multiple sub-network topologies based on the device network topology.

[0075] It should be noted that in the sub-network topology, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0076] In this step, the training task corresponding to the training request is first determined, and the number of GPUs required for the training task is determined based on the training task. Then, multiple subgraphs are constructed based on the device network topology, where the number of subgraphs equals the number of GPUs required for the training task, and these subgraphs serve as the sub-network topology.

[0077] For example, refer to Figure 3 , Figure 4 , Figure 7A and Figure 7B , Figure 7A This is a schematic diagram of a sub-network topology provided by the present invention. Figure 7B This is another schematic diagram of the subnetwork topology provided by the present invention. When the number of GPUs required for the training task is four, as shown in the diagram above... Figure 3 Based on the device network topology shown, multiple sub-network topologies can be constructed, such as... Figure 7A and Figure 7B The subnetwork topology shown is two of multiple subnetwork topologies. Among them, Figure 7A In the sub-network topology shown, GPU1 to GPU4 and the first PCIE4.0 Switch terminal are used as nodes of the sub-network topology, initial links 1, 3, 4, 6, 7, 9, and 11 are used as initial links of the sub-network topology, and the network bandwidth information and type of initial links 1, 3, 4, 6, 7, 9, and 11 are used as edges of the sub-network topology. Figure 7B In the sub-network topology shown, GPU1, GPU2, GPU5, GPU6, PCIE4.0 Switch first terminal, and PCIE4.0 Switch second terminal are used as nodes of the sub-network topology. Initial links 1, 3, 6, 12, 14, 17, 23, and 24 are used as initial links of the sub-network topology. The network bandwidth information and type of initial links 1, 3, 6, 12, 14, 17, 23, and 24 are used as edges of the network topology.

[0078] S1032, obtain the node resources of the graphics processor in each sub-network topology.

[0079] Optionally, GPU node resources can be obtained through the GPU metric resource detection process. GPU node resources may include, but are not limited to, the amount of GPU memory, memory utilization, core utilization, type, etc., as well as the network bandwidth information and type of the initial link connected to the GPU, within the subnetwork topology.

[0080] S1033, calculate the network flow information of each sub-network topology based on the node resources of the graphics processor in each sub-network topology.

[0081] In this step, based on the node resources of the GPUs in each sub-network topology, the maximum flow algorithm is used to calculate the maximum flow of each sub-network topology, and the maximum flow of each sub-network topology is used as the network flow information of each sub-network topology.

[0082] It should be noted that the maximum flow in a network refers to the feasible flow in a subnetwork topology that satisfies both edge flow constraints and flow balance conditions and has the maximum flow. The edge flow constraints are defined as follows: in the subnetwork topology, the edge... The edge weight c(u,v) is greater than or equal to the set of flows on all edges F=f(u,v), and the set of flows on all edges F=f(u,v) is greater than or equal to 0; the flow balance condition is defined as the flow into a node in a sub-network topology equal to the flow out of that node.

[0083] Optionally, the maximum flow algorithm can be the Ford-Fulkerson algorithm, or other maximum flow algorithms such as the EK algorithm, Dinic algorithm, etc. Those skilled in the art can choose a suitable maximum flow algorithm according to the actual situation, and the embodiments of the present invention do not make specific limitations in this regard.

[0084] For example, refer to Figure 7A and Figure 7B Using the Ford-Fulkerson algorithm to perform such... Figure 7A The subnetwork topology shown is calculated to have a maximum flow of 600G. The Ford-Fulkerson algorithm is then used to further analyze this subnetwork topology. Figure 7B The subnetwork topology shown is calculated, and the maximum flow of the subnetwork topology is found to be 32G.

[0085] S1034, Select the sub-network topology with the largest network flow information as the training network topology corresponding to the training request.

[0086] In this step, after obtaining the maximum network flow for all sub-network topologies, the sub-network topology with the maximum network flow is selected as the training network topology corresponding to the training request. The training network topology has an initial link with the maximum network bandwidth information, which is defined as the training link. All GPUs located on the training link are training processing nodes, indicating that the training link is a closed loop with the maximum network traffic. The training network topology will serve as the basis and foundation for the orchestration and deployment of large model containers.

[0087] For example, refer to Figure 7A and Figure 7B ,like Figure 7A The maximum network flow of the subnetwork topology shown is 600G, which means that the maximum network bandwidth of this subnetwork topology is 600G. Figure 7B The maximum flow of the subnetwork topology shown is 32G, which means that the maximum network bandwidth of this subnetwork topology is 32G. Clearly, Figure 7A The subnetwork topology shown is more effective for training tasks than Figure 7B The subnetwork topology shown is better, therefore it is chosen. Figure 7A The subnetwork topology shown is used as the training network topology.

[0088] In some embodiments of the present invention, the step of creating a large model container based on the training network topology in step S104 may include, but is not limited to: S1041, Obtain the training device where the graphics processor in the training network topology is located, create a large model container on the training device, and deploy the graphics processor in the training network topology in the corresponding large model container.

[0089] In this step, the training network topology obtained in the previous steps is obtained. The training network topology consists of training links and training processing nodes located on the training links. Based on the training network topology, at least one large model container is deployed on at least one training device. Each large model container holds a corresponding training processing node, i.e., a GPU node, so as to achieve the effect of specifying that each large model container uses an appropriate network link for communication, and ensuring that each large model container can communicate with the maximum network bandwidth.

[0090] For example, refer to Figure 7A ,when Figure 7A When the sub-network topology shown is used as the training network topology, the four GPUs in the sub-network topology are located on the same host PCIE4.0 Switch first terminal. Then the host is the training device. Four large model containers are created on the host, and each large model container deploys one GPU. Thus, these four large model containers are deployed with GPU1 to GPU4 respectively.

[0091] For another example, refer to Figure 7B ,when Figure 7BWhen the sub-network topology shown is used as the training network topology, GPU1 and GPU2 of this sub-network topology are located on the same host PCIE4.0 Switch first terminal, while GPU5 and GPU6 are located on the same host PCIE4.0 Switch second terminal. Using the aforementioned first terminal and second terminal as training devices, two large model containers are created on the first terminal host, one large model container deploying GPU1 and the other large model container deploying GPU2; two large model containers are created on the second terminal host, one large model container deploying GPU5 and the other large model container deploying GPU6.

[0092] Furthermore, step S1041, the step of creating a large model container on the training device, may include: Create large model containers on the training device based on Kubernetes.

[0093] Understandably, Kubernetes is a distributed architecture based on container technology. Essentially, it is a cluster of servers that can run specific programs on each node of the cluster to manage the containers on the nodes, thereby automating resource management.

[0094] It should be noted that in other embodiments of the present invention, in addition to Kubernetes, other container deployment technologies such as Docker are also applicable, requiring only corresponding adjustments.

[0095] In some embodiments of the present invention, step S105, which involves configuring a large model container to execute the training task corresponding to the training request, may include, but is not limited to: S1051: Select any one of the large model containers as the master node container from multiple large model containers, and use the other large model containers besides the master node container as slave node containers.

[0096] In this step, multiple large model containers are used as a cluster. A container is randomly selected from the cluster as the master node container, while the remaining containers serve as slave nodes. The master node container is the cluster control node, responsible for cluster management; the slave nodes are the workload nodes, used to execute tasks assigned by the master node container.

[0097] S1052 manages multiple slave node containers through the master node container, and executes the training tasks corresponding to the training requests through the multiple slave node containers.

[0098] In this step, the master node container manages and allocates training tasks to multiple slave node containers, and the multiple slave node containers are responsible for running the training tasks corresponding to the training requests, thereby realizing the training of a large model.

[0099] Next, another implementation step of the large model container orchestration method based on GPU network topology proposed in this embodiment of the invention will be described in detail below.

[0100] Reference Figure 8 , Figure 8 This is another flowchart of the large model container orchestration method based on GPU network topology provided by the present invention. The method may further include the following steps: S201, When a stop request is detected, stop executing the training task corresponding to the training request.

[0101] In this step, when an abnormal situation is detected, a stop request will be automatically generated. The stop request is used to pause the ongoing training task. In response to the stop request, the training task corresponding to the training request will stop being executed.

[0102] It should be noted that abnormal situations can be network failures caused by link disconnection, or other situations such as GPU failure, primary device failure, etc. This invention does not specifically limit these situations.

[0103] S202, Update the device network topology, and update the training network topology based on the updated device network topology.

[0104] In this step, the device network topology is updated by removing erroneous training links and / or training processing nodes. Then, the updated device network topology is used to update the training network topology, thereby updating the training links and training processing nodes used to perform training tasks in the training network topology.

[0105] S203, based on the updated training network topology, re-execute the training task corresponding to the training request.

[0106] In this step, the automatic recovery of large model training tasks is guided based on the updated training network topology, so that the training tasks corresponding to the training requests are re-executed.

[0107] In some embodiments of the present invention, the method may further include the following steps: When a stop request is detected, the ckpt file corresponding to the training task is stored.

[0108] It should be noted that the ckpt file is a checkpoint file, which stores the model's parameters and variables for a certain period of time during the training process.

[0109] In some embodiments of the present invention, in step S202, the step of updating the device network topology and updating the training network topology according to the updated device network topology may include, but is not limited to: S2021, remove the initial link and device corresponding to the stop request from the device network topology, update the network bandwidth information between multiple devices in the device network topology, and generate an updated device network topology.

[0110] In this step, when there are abnormal situations, it is necessary to first identify the training links and / or training processing nodes that have abnormal situations. The device network topology is updated by removing the training links and / or training processing nodes that have abnormal situations from the original device network topology. At the same time, the network bandwidth information between multiple devices is updated to obtain a new device network topology, so as to avoid reselecting the wrong nodes or links when reselecting training links and training processing nodes in the future.

[0111] S2022, based on the updated device network topology, construct multiple sub-update network topologies.

[0112] In this step, a new device network topology is obtained by removing training links and / or training processing nodes with abnormal conditions from the original device network topology. The nodes and edges in the new device network topology will change. At this time, the subgraph of the device network topology needs to be reconstructed, that is, multiple sub-update network topologies are rebuilt.

[0113] It is understandable that in the sub-update network topology, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges.

[0114] S2023: Obtain the network flow information for each sub-updated network topology, and select the sub-updated network topology with the largest network flow information as the updated training network topology.

[0115] In this step, the maximum network flow of each sub-updated network topology is used as the network flow information for that sub-updated network topology. The sub-updated network topology with the maximum network flow is selected as the updated training network topology. The updated training network topology also has an initial link with the maximum network bandwidth. This initial link with the maximum network bandwidth is the training link, and all GPUs located on the training link are training processing nodes. This indicates that the updated training link is still a closed loop with the maximum network traffic. The updated training network topology will serve as the basis for re-orchestrending and deploying large model containers.

[0116] Further, in step S2023, the step of obtaining network flow information for each sub-updated network topology may include: When multiple sub-update network topologies are different from multiple sub-network topologies, the update node resources of the graphics processor in each sub-update network topology are obtained, and the network flow information of each sub-update network topology is calculated based on the update node resources of the graphics processor in each sub-update network topology. When multiple sub-updated network topologies are the same as multiple sub-network topologies, the network flow information of the multiple sub-network topologies is used as the network flow information of the multiple sub-updated network topologies.

[0117] Optionally, the update node resources of the GPU can be obtained through the GPU metric resource detection process. The update node resources of the GPU include, but are not limited to, the amount of memory, memory utilization, core utilization, and type of the GPU, as well as the network bandwidth information and type of the initial link connected to the GPU, in the sub-update network topology.

[0118] In this step, after removing erroneous GPU nodes or links from the device network topology, some parts of the updated device network topology will change, while others will remain unchanged. This may result in the creation of new subgraphs with structures different from the previously constructed subgraphs when building the subgraphs of the updated device network topology. Additionally, some older subgraphs with the same structure as the previously constructed subgraphs will also be generated. Therefore, depending on the type of the subgraphs in the updated device network topology, this step obtains the network flow information of the subgraphs using different methods.

[0119] When multiple sub-updated network topologies differ from multiple sub-network topologies, it indicates that a new subgraph was generated during the construction of the updated device network topology. In this case, it is necessary to recalculate the maximum network flow for each subgraph. Specifically, based on the update node resources of the GPUs in each sub-updated network topology, the maximum network flow for each sub-updated network topology is calculated using the maximum flow algorithm, and the maximum network flow for each sub-updated network topology is used as the network flow information for each sub-updated network topology.

[0120] Optionally, the maximum flow algorithm can be the Ford-Fulkerson algorithm, or other maximum flow algorithms such as the EK algorithm, Dinic algorithm, etc. Those skilled in the art can choose a suitable maximum flow algorithm according to the actual situation, and the embodiments of the present invention do not make specific limitations in this regard.

[0121] When multiple sub-updated network topologies are the same as multiple sub-network topologies, no new subgraph is generated when constructing the subgraph of the updated device network topology. In this case, the network flow information of multiple sub-network topologies can be directly obtained as the network flow information of multiple sub-updated network topologies, omitting the step of recalculating the maximum network flow using the maximum flow algorithm.

[0122] In some embodiments of the present invention, reference is made to... Figure 9 , Figure 9 This is a flowchart of the re-execution of the training task provided by the present invention. In step S203, the step of re-executing the training task corresponding to the training request according to the updated training network topology may include, but is not limited to: S2031, rearrange the large model container based on the updated training network topology.

[0123] In this step, the updated training network topology is obtained. The updated training network topology consists of new training links and training processing nodes located on the new training links. Based on the new training network topology, at least one large model container is deployed on at least one training device. Each large model container holds a corresponding training processing node, i.e., a GPU, to ensure that each large model container uses an appropriate network link for communication, guaranteeing that each large model container can communicate with the maximum network bandwidth. The implementation of this step is the same as that of the previous step S1041, and will not be described again here.

[0124] S2032, determine whether the updated training network topology meets the training requirements.

[0125] It should be noted that training requirements can be understood as the minimum network bandwidth and number of GPU nodes required for the training task.

[0126] In this step, the number of training processing nodes and the maximum bandwidth of the training links in the updated training network topology are obtained. By determining whether the number of training processing nodes in the updated training network topology meets the first value and whether the network bandwidth information of the training links in the updated training network topology meets the second value, it is determined whether the updated training network topology meets the training requirements.

[0127] S2033, when the number of graphics processors in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, the training task corresponding to the training request is re-executed by configuring the large model container.

[0128] In this step, when the number of graphics processors in the updated training network topology is greater than or equal to the first value, and the network flow information of the updated training network topology is greater than or equal to the second value, it indicates that the updated training network topology meets the training requirements. At this time, the training task corresponding to the training request is re-executed by configuring the large model container.

[0129] It should be noted that the first and second values ​​can be set according to the actual situation, and the embodiments of the present invention do not impose specific limitations on them.

[0130] S2034, when the number of graphics processors in the updated training network topology is less than the first value and / or the network flow information of the updated training network topology is less than the second value, it indicates that the updated training network topology does not meet the training requirements, and the training task is terminated at this time.

[0131] Furthermore, in step S2033, the step of re-executing the training task corresponding to the training request by configuring the large model container may include: First, select any one of the rearranged large model containers as the master node container, and use the other large model containers as slave node containers.

[0132] Then, the master node container manages multiple slave node containers, and the multiple slave node containers call the ckpt file corresponding to the training task to re-execute the training task corresponding to the training request.

[0133] In the above steps, multiple rearranged large model containers are used as a cluster. One container is randomly selected from the cluster as the master node, and the remaining containers serve as slave nodes. The master node is the cluster control node, responsible for cluster management; the slave nodes are the workload nodes, used to execute tasks assigned by the master node. Then, the master node manages and allocates training tasks to the multiple slave nodes, and the training tasks corresponding to the training requests are re-executed by the multiple slave nodes, thus resuming the training of the large model.

[0134] The following example illustrates the implementation process of the large model container orchestration method based on GPU network topology provided in this embodiment of the invention. The specific implementation process is as follows: The first step is network bandwidth detection: detecting network bandwidth information between multiple GPUs, between multiple first devices, and between a GPU and a first device.

[0135] The second step is to construct the device network topology: A global device network topology is built based on network bandwidth information. In this topology, the GPU and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges in the network topology.

[0136] The third step is configuring the training network graph topology: First, a training request is obtained, which is configured with a corresponding training task. Then, when scheduling large model training tasks, multiple sub-network topologies are constructed based on the device network topology. The number of sub-network topologies is the same as the number of GPUs required for the training task. The maximum flow algorithm is used to calculate the maximum flow of each sub-network topology. The sub-network topology with the maximum maximum flow is selected as the training network topology. The training network topology consists of the training link with the maximum network bandwidth and multiple training processing nodes (i.e., GPU nodes) located on the training link.

[0137] The fourth step is the orchestration and configuration of large model containers: First, obtain the training device where the GPUs in the training network topology reside. Create large model containers on the training device and deploy the GPUs in the training network topology into the corresponding large model containers. Optionally, each large model container can be deployed with one training processing node to ensure that each large model container uses an appropriate network link for communication, guaranteeing that each large model container can communicate with maximum network bandwidth. Then, select any one of the large model containers as the master node container, and designate the other large model containers as slave node containers. The master node container is used to manage the slave node containers; the slave node containers are used to execute the tasks assigned by the master node container.

[0138] The fifth step is the execution of the large model training task: the master node container manages and distributes training tasks to multiple slave node containers, and the multiple slave node containers are responsible for running the training tasks corresponding to the training requests, thereby realizing the training of the large model.

[0139] Step 6, Anomaly Detection: Determine if a stop request is detected during the training of the large model. Specifically, during the training of the large model, when an anomaly is detected, a stop request will be automatically generated to pause the ongoing training task. Anomalies can be network failures caused by a disconnected link, or other issues such as GPU failure, primary device failure, etc.

[0140] Step 7, Rearrangement of the large model container: When a stop request is detected, the training task corresponding to the training request is stopped. Then, the device network topology is updated by removing erroneous training links and / or training processing nodes. After that, based on the updated device network topology, multiple sub-update network topologies are reconstructed, and the sub-update network topology with the maximum network flow value is selected as the new training network topology.

[0141] Step 8, Training Requirements Detection: Determine whether the updated training network topology meets the training requirements.

[0142] Step 9, Resumption of Training Task: When the number of GPUs in the updated training network topology meets the first value, and the network flow information of the updated training network topology meets the second value, it indicates that the updated training network topology meets the training requirements. At this point, the training task corresponding to the training request is re-executed by configuring a large model container. Otherwise, the training task is stopped.

[0143] In summary, this invention, on the one hand, orchestrates and deploys large model containers from a GPU-level granularity perspective. First, it detects network bandwidth information between multiple GPUs, between multiple first devices, and between a GPU and a first device, constructing a device network topology based on this information. In this topology, GPUs and first devices are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth and type of each initial link are used as edges. After obtaining the GPU network topology and the link bandwidth between GPUs, when scheduling large model training tasks, the device network topology is used as the basis for network performance. The sub-network topology with the largest network flow information is selected from multiple sub-network topologies (subgraphs) of the device network topology. Based on this sub-network topology, large model containers are rationally orchestrated and deployed, ensuring that the training containers use appropriate network links for communication. In this embodiment of the invention, GPUs and a first device are used as nodes, and the network bandwidth information and type of each initial link are used as edges to construct a device network topology. The network bandwidth between GPUs is discovered through the device network topology, so that the orchestration of containers can meet the bandwidth requirements between GPUs. This improves the training performance and efficiency of large model training while also increasing the utilization of training resources.

[0144] On the other hand, when network failures or link changes occur, this embodiment of the invention pauses the current training task by responding to a stop request, updates the device network topology, and searches for a sub-updated network topology with the link having the maximum network bandwidth information within the updated device network topology. When the newly found sub-updated network topology meets the training requirements, the training of the large model restarts, achieving automatic recovery of large model training. This embodiment of the invention can automatically diagnose faults, adjust faulty training nodes, and rearrange containers without manual intervention when network links or graphics processors fail, achieving automatic recovery of large model training. This reduces equipment maintenance costs and manpower input while improving the success rate and training efficiency of large model training.

[0145] Additionally, refer to Figure 10 , Figure 10 This is a structural diagram of the large model container orchestration system based on GPU network topology provided by the present invention. The embodiments of the present invention also provide a large model container orchestration system based on GPU network topology, the system comprising: The detection module 301 is used to detect network bandwidth information between multiple devices.

[0146] The topology processing module 302 is used to construct a device network topology based on network bandwidth information, using the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as the edge.

[0147] Module 303 is used to obtain training requests; The node processing module 304 is used to select the training network topology corresponding to the training request based on the device network topology.

[0148] The container processing module 305 is used to create large model containers based on the training network topology.

[0149] The training processing module 306 is used to execute the training task corresponding to the training request by configuring a large model container.

[0150] Optionally, in some embodiments of the present invention, the system may further include: The fault handling module is used to stop the execution of the training task corresponding to the training request when a stop request is detected, update the device network topology, update the training network topology using the updated device network topology, and re-execute the training task corresponding to the training request based on the updated training network topology.

[0151] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0152] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned large model container orchestration method based on GPU network topology.

[0153] Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0154] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0155] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0156] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several programs to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable programs for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, a program execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can retrieve and execute a program from or in conjunction with such a program execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit a program for use by or in conjunction with a program execution system, apparatus, or device.

[0158] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0159] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable program execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0160] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0161] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0162] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A large model capacity orchestration method based on GPU network topology, characterized in that, Includes the following steps: Detecting network bandwidth information among multiple devices, the devices including a graphics processor and a first device; Based on the network bandwidth information, the device network topology is constructed using the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as edges. Obtain a training request, and determine the training network topology corresponding to the training request based on the device network topology; Based on the training network topology, create a large model container; By configuring the large model container, the training task corresponding to the training request is executed; Wherein, determining the training network topology corresponding to the training request based on the device network topology includes: The number of GPUs required for the training task is determined based on the training task. Multiple subgraphs are constructed based on the device network topology, and the subgraphs of the device network topology are used as sub-network topologies. In the sub-network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges. Obtain the node resources of the graphics processor in each sub-network topology; Based on the node resources of the graphics processor in each sub-network topology, the network flow information of each sub-network topology is calculated, and the sub-network topology with the largest network flow information is selected as the training network topology corresponding to the training request.

2. The large model capacity orchestration method based on GPU network topology according to claim 1, characterized in that, The detection of network bandwidth information between multiple devices includes: By iterating through the network bandwidth information of each initial link, the network bandwidth information between multiple devices can be obtained.

3. The large model capacity orchestration method based on GPU network topology according to claim 1, characterized in that, The step of creating a large model container based on the training network topology includes: Obtain the training device where the graphics processor in the training network topology is located, create a large model container on the training device, and deploy the graphics processor in the training network topology in the corresponding large model container.

4. The large model capacity orchestration method based on GPU network topology according to claim 3, characterized in that, The step of configuring the large model container to execute the training task corresponding to the training request includes: Select any one of the large model containers as the master node container, and use the other large model containers besides the master node container as slave node containers; The master node container manages multiple slave node containers, and the multiple slave node containers execute training tasks corresponding to the training request.

5. The large model capacity orchestration method based on GPU network topology according to claim 1, characterized in that, The method further includes the following steps: When a stop request is detected, the training task corresponding to the training request is stopped. Update the device network topology, and update the training network topology based on the updated device network topology; Based on the updated training network topology, the training task corresponding to the training request is re-executed.

6. The large model capacity orchestration method based on GPU network topology according to claim 5, characterized in that, The step of updating the device network topology, and updating the training network topology based on the updated device network topology, includes: Remove the initial link and device corresponding to the stop request from the device network topology, and update the network bandwidth information between multiple devices in the device network topology to generate an updated device network topology; Multiple subgraphs are constructed based on the updated device network topology, and the subgraphs of the updated device network topology are used as sub-updated network topologies. In the sub-updated network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges. Obtain the network flow information for each sub-updated network topology, and select the sub-updated network topology with the largest network flow information as the updated training network topology.

7. The large model capacity orchestration method based on GPU network topology according to claim 6, characterized in that, The step of re-executing the training task corresponding to the training request based on the updated training network topology includes: The large model container is rearranged based on the updated training network topology; When the number of graphics processors in the updated training network topology meets the first value and the network flow information of the updated training network topology meets the second value, the training task corresponding to the training request is re-executed by configuring the large model container.

8. A large-model-capacity orchestration system based on GPU network topology, characterized in that, include: A detection module is used to detect network bandwidth information between multiple devices, wherein the devices include a graphics processor and a first device; The topology processing module is used to construct a device network topology based on the network bandwidth information, using the graphics processor and the first device as nodes, the channel connecting the two devices as the initial link, and the network bandwidth information and type of each initial link as the edge. The acquisition module is used to acquire training requests; The node processing module is used to determine the training network topology corresponding to the training request based on the device network topology; A container processing module is used to create large model containers based on the training network topology; The training processing module is used to execute the training task corresponding to the training request by configuring the large model container; Wherein, determining the training network topology corresponding to the training request based on the device network topology includes: The number of GPUs required for the training task is determined based on the training task. Multiple subgraphs are constructed based on the device network topology, and the subgraphs of the device network topology are used as sub-network topologies. In the sub-network topology, the graphics processor and the first device are used as nodes, the channel connecting the two devices is used as the initial link, and the network bandwidth information and type of each initial link are used as edges. Obtain the node resources of the graphics processor in each sub-network topology; Based on the node resources of the graphics processor in each sub-network topology, the network flow information of each sub-network topology is calculated, and the sub-network topology with the largest network flow information is selected as the training network topology corresponding to the training request.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the large model capacity orchestration method based on GPU network topology as described in any one of claims 1 to 7.

Images