Distributed training scheduling system and method, electronic equipment, storage medium and computer program product
By creating parallel working containers in the distributed training system and asynchronously reading training metadata, using the hardware GPU to start the training process, and implementing a recovery strategy in the event of a failure, the problems of long startup time and difficult fault recovery in large-scale distributed training are solved, and the efficiency and stability of the system are improved.
Patent Information
- Application Number
- CN202510788312.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-09-23
AI Technical Summary
In large-scale distributed training scenarios, the existing distributed training scheduling system takes a long time to start the training process and cannot effectively recover when the training container fails.
By starting the container, multiple working containers are created, and training metadata is generated. The training process is read asynchronously and in parallel from the database and started. The mounted hardware GPU is used for training to start the training process in parallel and implement a fault recovery strategy in case of failure.
It effectively reduces the startup time of distributed training scheduling, improves the debuggability and stability of training, and reduces the difficulty of fault recovery.
Smart Images
Figure CN120687250A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a distributed training scheduling system and method, electronic equipment, storage medium, and computer program product. Background Art
[0002] Distributed training of large models typically involves scheduling training on large-scale graphics processing unit (GPU) clusters. Network interaction between the training process and the control process completes the training process networking, enabling large-scale distributed training. During training, the control process typically monitors the status of distributed training. If any failure occurs, further troubleshooting strategies are implemented to ensure smooth training. Current distributed training scheduling suffers from the problem of long training process startup times. Summary of the Invention
[0003] The present disclosure provides a distributed training scheduling system and method, an electronic device, a storage medium, and a computer program product.
[0004] According to one aspect of the present disclosure, a distributed training scheduling system is provided, which includes a startup container, multiple working containers, and a database; the startup container is used to create the multiple working containers according to the training task custom resource CRD of the target distributed training task, and generate training metadata corresponding to each working container, wherein each working container is mounted with multiple hardware graphics processing devices GPU; the startup container is used to write the training metadata corresponding to each working container into the database; the multiple working containers are used to asynchronously and parallelly read the training metadata from the database; any one of the working containers is used to start a training process in the working container according to the training metadata corresponding to the working container read from the database, using the multiple hardware GPUs mounted on the working container.
[0005] In one possible implementation, the system further includes: a coordinator; the coordinator is configured to create the training task CRD for the target distributed training task, and create the startup container based on the training task CRD.
[0006] In one possible implementation, the startup container is specifically used to: create the multiple working containers by reading the training task CRD, and generate training metadata corresponding to each working container; and write the training metadata corresponding to each working container into the database.
[0007] In one possible implementation, any working container is specifically used to: read the training metadata corresponding to the working container from the database; and start the training process in the working container according to the training process start command included in the training metadata corresponding to the working container.
[0008] In one possible implementation, any working container is specifically used to: determine the training status information of the working container by monitoring the training process within the working container; and write the training status information of the working container into the training metadata corresponding to the working container stored in the database.
[0009] In one possible implementation, the startup container is also used to: periodically poll the training metadata corresponding to each working container stored in the database; detect whether there is an abnormal working container based on the training status information included in the training metadata corresponding to each working container; and when an abnormal working container is detected, determine a training fault recovery strategy based on the training status information corresponding to the abnormal working container.
[0010] In one possible implementation, the training fault recovery strategy is to delete the abnormal working container and create a new working container; the startup container is specifically used to: delete the abnormal working container and create a new working container according to the training fault recovery strategy; determine the training metadata of the newly created working container based on the training metadata of the abnormal working container stored in the database, and write the training metadata of the newly created working container into the database.
[0011] In a possible implementation, the newly created working container is used to read the corresponding training metadata from the database and resume the training process using the read training metadata.
[0012] In one possible implementation, the training fault recovery strategy is to restart the training process; the startup container is used to update the training status information included in the training metadata corresponding to the abnormal working container stored in the database to restart according to the training fault recovery strategy; the abnormal working container is used to read the corresponding training metadata from the database, and restart the training process according to the training status information included in the read training metadata. According to one aspect of the present disclosure, a ** device is provided, including:
[0013] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0014] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.
[0015] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the above method when executed by a processor.
[0016] In an embodiment of the present disclosure, a distributed training scheduling system is involved. A startup container creates multiple working containers based on the training task custom resource (Custom Resource Definition, CRD) of the target distributed training task, and generates training metadata corresponding to each working container, wherein each working container is mounted with multiple hardware GPUs; the startup container can write the training metadata corresponding to each working container into a database, so that multiple working containers can asynchronously and parallelly read the training metadata from the database. Any working container starts a training process in the working container based on the training metadata corresponding to the working container read from the database and using the multiple hardware GPUs mounted on the working container, thereby realizing the parallel startup of the training processes of multiple working containers based on the startup container-database-working container mode, effectively reducing the startup time of distributed training scheduling.
[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.
[0019] Figure 1 A block diagram of a distributed training scheduling system according to an embodiment of the present disclosure is shown.
[0020] Figure 2 A flowchart of a distributed training scheduling method according to an embodiment of the present disclosure is shown.
[0021] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0022] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.
[0023] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0024] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.
[0025] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.
[0026] In the related art, distributed training scheduling can be mainly achieved in two ways: 1. Start the training process by executing ssh remote commands in sequence through the control node, and monitor the status of the training process through the control node to ensure the stable operation of the training; 2. Through containerized orchestration technology, the training script is used as the container startup command when the container is started to start the training process, and then interact with the control process. However, both of the above methods have some technical problems. The problem with starting the training process by remotely executing remote ssh commands in sequence is that in scenarios involving large-scale distributed training, the time to execute ssh commands remotely in sequence will increase linearly. If it involves large-scale model training, the training process startup time will become longer and longer. If the training process fails when starting the training process through the container initialization command, the training container will be directly deleted, and the execution environment when the training container fails cannot be saved, and thus training fault debugging and corresponding fault recovery cannot be performed.
[0027] To address the above technical issues, the present disclosure provides a distributed training scheduling system that can enable parallel startup of training processes for multiple working containers in large-scale distributed training scenarios, effectively reducing the startup time of distributed training scheduling. The following describes the distributed training scheduling system provided by the present disclosure in detail.
[0028] Figure 1 FIG. 1 is a block diagram of a distributed training scheduling system according to an embodiment of the present disclosure. Figure 1 As shown, the distributed training scheduling system includes a startup container, multiple working containers: working container 1 to working container N, and a database.
[0029] Start a container to create multiple working containers based on the training task CRD of the target distributed training task, and generate training metadata corresponding to each working container, where each working container is mounted with multiple hardware GPUs.
[0030] The startup container is responsible for launching multiple training processes in the target distributed training task, while the worker container is responsible for actually running the training processes. Each worker container has multiple hardware GPUs mounted on it, so the worker container can also be called a GPU container.
[0031] The number of working containers and the number of hardware GPUs mounted on each working container can be flexibly set according to the requirements of the target distributed training task, and this disclosure does not make specific restrictions on this.
[0032] For example, according to the training task CRD of the target distributed training task, it can be determined that the target distributed training task requires 1024 hardware GPUs to execute. Therefore, starting the container can create 128 working containers, each of which is mounted with 8 hardware GPUs.
[0033] After the startup container creates multiple worker containers for the target distributed training task, it also generates training metadata for each worker container. The training metadata for each worker container indicates the various data required for the training process to be started on that worker container.
[0034] Start a container to write the training metadata corresponding to each working container into the database.
[0035] After starting the container to generate the training metadata corresponding to each working container, the training metadata corresponding to each working container also needs to be written into the database to prepare for the subsequent asynchronous and parallel startup of the training processes in multiple working containers.
[0036] The database here can be a database corresponding to the target distributed training task. The specific form can be flexibly set according to the actual application scenario, and this disclosure does not make specific limitations on this.
[0037] Multiple working containers are used to asynchronously and parallelly read training metadata from the database; any one of the working containers is used to start a training process in the working container based on the training metadata corresponding to the working container read from the database, using multiple hardware GPUs mounted on the working container.
[0038] Since the training metadata corresponding to each working container has been written to the database, multiple working containers can read the corresponding training metadata from the database asynchronously and in parallel, so that each working container can start the training process in the working container based on the training metadata read from the database and using the multiple hardware GPUs mounted on the working container.
[0039] In large-scale distributed training scenarios, especially those involving thousands or tens of thousands of cards, asynchronous parallel startup of distributed training tasks can optimize the linear increase in startup time caused by sequentially executing ssh remote commands to start multiple training processes in series, thereby reducing the startup time of distributed training tasks from T1+T2+…+T N , optimized to Max(T1, T2, ..., T N ), where T i (i ranges from 1 to N) represents the time required to start the training process on each worker container, thereby accelerating the startup time of large-scale distributed training.
[0040] In an embodiment of the present disclosure, a distributed training scheduling system is involved. A startup container creates multiple working containers according to the training task CRD of the target distributed training task, and generates training metadata corresponding to each working container, wherein each working container is mounted with multiple hardware GPUs; the startup container can write the training metadata corresponding to each working container into a database, so that multiple working containers can asynchronously and in parallel read the training metadata from the database. Any working container starts a training process in the working container based on the training metadata corresponding to the working container read from the database and using the multiple hardware GPUs mounted on the working container, thereby realizing the parallel startup of the training processes of multiple working containers based on the startup container-database-working container mode, effectively reducing the startup time of distributed training scheduling.
[0041] In one possible implementation, the distributed training scheduling system further includes: a coordinator; the coordinator is configured to create a training task CRD for a target distributed training task, and to create a startup container based on the training task CRD.
[0042] The coordinator can act as the application controller (operator) of the Kubernetes platform, responsible for creating the above-mentioned startup containers.
[0043] The coordinator creates a training task CRD (e.g., a Kubernetes job CRD) for the target distributed training task. This CRD describes the target distributed training task so that the startup container responsible for launching the target distributed training task can be created based on the CRD. The specific creation process can be referenced in the related art for creating containers on the Kubernetes platform and will not be detailed here.
[0044] In one possible implementation, a container is started, specifically for: creating multiple working containers by reading a training task CRD, and generating training metadata corresponding to each working container; and writing the training metadata corresponding to each working container into a database.
[0045] Since the above-mentioned training task CRD is used to describe the target distributed training task, the startup container can create multiple working containers for the target distributed training task by reading the training task CRD, and generate training metadata corresponding to each working container.
[0046] In one example, the container is started through a cmd command (for example, launcher.sh), and the first training task startup process is pulled up within the startup container, so that the training task CRD of the target distributed training task can be read using the first training task startup process, and then multiple working containers are created for the target distributed training task based on the training task CRD.
[0047] In one example, the startup container uses the first training task startup process to read the training task CRD of the target distributed training task, thereby generating and writing training metadata corresponding to each working container into the database.
[0048] In one example, after writing the training metadata corresponding to each working container to the database, the startup container can simultaneously issue asynchronous parallel read instructions to each working container, enabling multiple working containers to asynchronously and concurrently read the training metadata from the database. The specific form of the asynchronous parallel read instructions can be flexibly configured based on the actual application scenario and is not specifically limited in this disclosure.
[0049] In one possible implementation, any working container is specifically used to: read the training metadata corresponding to the working container from a database; and start the training process in the working container according to the training process start command included in the training metadata corresponding to the working container.
[0050] The training metadata corresponding to each working container includes the training process start command corresponding to the working container. Therefore, after each working container reads the corresponding training metadata from the database, it can start the training process in the working container according to the training process start command included in the training metadata.
[0051] For any working container, the working container can start the second training task startup process in the working container through the cmd command (for example, worker.sh), so that the working container can use the second training task startup process to read the training metadata corresponding to the working container from the database, and then start the training process in the working container according to the training process startup command (for example, torchrun cmd) included in the training metadata corresponding to the working container.
[0052] Compared with the containerized orchestration technology in the prior art that uses training scripts as container startup commands, the embodiment of the present disclosure stores the training scripts (training metadata) in a database, and starts the training process in the working container through asynchronous reading of the working container, that is, the startup command of the working container and the startup command of the training process are separated. Based on the startup container-database-working container model, when a failure occurs in the training process, the working container will not be directly deleted, and the abnormal working container can be retained, which provides the possibility for subsequent environmental problem troubleshooting and debugging of the abnormal working container. The problem of being unable to locate the root cause of the training abnormality due to the direct deletion of the abnormal working container will not occur, thereby increasing the debuggability of training.
[0053] In one possible implementation, any working container is specifically used to: determine the training status information of the working container by monitoring the training process within the working container; and write the training status information of the working container into the training metadata corresponding to the working container stored in the database.
[0054] After starting the training process, each working container can also determine the training status information of the working container by monitoring the training process, and write the training status information of the working container into the training metadata corresponding to the working container stored in the database.
[0055] In one example, each working container can use the second training task in the working container to start a process, monitor the training process in the working container, and determine the training status information of the working container; then, use the second training task in the working container to start a process and write the training status information of the working container into the training metadata corresponding to the working container in the database.
[0056] In one possible implementation, the container is started and is also used to: periodically poll the training metadata corresponding to each working container stored in the database; detect whether there is an abnormal working container based on the training status information included in the training metadata corresponding to each working container; and when an abnormal working container is detected, determine a training fault recovery strategy based on the training status information corresponding to the abnormal working container.
[0057] Saving the training metadata corresponding to each working container (including: training process startup commands, training status information) in the database can give the startup container greater autonomy. The startup container can use the database to manage and coordinate the startup, fault detection, fault recovery, etc. of each working container, thereby enhancing the stability of large-scale distributed training.
[0058] In one example, the startup container may utilize the first training task to start a process, and periodically poll the training metadata corresponding to each working container stored in the database to determine whether there is an abnormal working container.
[0059] When the startup container detects the existence of an abnormal working container, it determines the corresponding training fault recovery strategy based on the training status information corresponding to the abnormal working container, so that fault recovery can be effectively performed based on the startup container-database-working container model, thereby improving the stability of large-scale distributed training.
[0060] In one example, the startup container can pre-set the corresponding training fault recovery strategy for various training status information corresponding to the abnormal working container. The specific form can be flexibly set according to the actual application scenario, and this disclosure does not make specific limitations on this.
[0061] In one example, when an abnormal working container fails, the real-time environmental information of the abnormal working container can be saved, so that the historical situation of the abnormal working container can be traced back, the cause of the failure can be analyzed, and a fault recovery strategy corresponding to the fault type can be formulated, which effectively increases the debuggability of large-scale distributed training.
[0062] In one possible implementation, the training fault recovery strategy is to delete the abnormal working container and create a new working container; start the container, specifically for: deleting the abnormal working container and creating a new working container according to the training fault recovery strategy; determining the training metadata of the newly created working container based on the training metadata of the abnormal working container stored in the database, and writing the training metadata of the newly created working container into the database.
[0063] In one possible implementation, a new working container is created to read corresponding training metadata from a database and to resume the training process using the read training metadata.
[0064] The training fault recovery strategy may be to delete the abnormal working container and create a new working container. In this case, the startup container starts the process using the first training task, deletes the abnormal working container, and creates a new working container.
[0065] Furthermore, the startup container uses the first training task to start the process, determines the training metadata of the newly created working container based on the training metadata of the abnormal working container stored in the database, and writes the training metadata of the newly created working container into the database, so that the newly created working container can pull up the second training task startup process, and then the second training task startup process reads the corresponding training metadata from the database, and uses the read training metadata to resume the training process.
[0066] In one possible implementation, the training fault recovery strategy is to restart the training process; a startup container is used to update the training status information included in the training metadata corresponding to the abnormal working container stored in the database to restart according to the training fault recovery strategy; the abnormal working container is used to read the corresponding training metadata from the database, and restart the training process according to the training status information included in the read training metadata.
[0067] The training failure recovery strategy can be restarting the training process. In this case, the startup container uses the first training task to start the process and updates the training status information included in the training metadata corresponding to the abnormal working container stored in the database to restart. This allows the abnormal working container to start the process using the second training task to read the corresponding training metadata from the database and restart the training process based on the training status information included in the read training metadata.
[0068] In addition to the two examples above, the training fault recovery strategy may also include other strategies based on actual conditions, and this disclosure does not specifically limit this.
[0069] In an embodiment of the present disclosure, a distributed training scheduling system is involved. A startup container creates multiple working containers according to the training task CRD of the target distributed training task, and generates training metadata corresponding to each working container, wherein each working container is mounted with multiple hardware GPUs; the startup container can write the training metadata corresponding to each working container into a database, so that multiple working containers can asynchronously and in parallel read the training metadata from the database. Any working container starts a training process in the working container based on the training metadata corresponding to the working container read from the database and using the multiple hardware GPUs mounted on the working container, thereby realizing the parallel startup of the training processes of multiple working containers based on the startup container-database-working container mode, effectively reducing the startup time of distributed training scheduling.
[0070] Figure 2 A flow chart of a distributed training scheduling method according to an embodiment of the present disclosure is shown. Figure 1 The distributed training scheduling system shown in Figure 2 As shown, the method includes:
[0071] In step S21, based on the startup container, multiple working containers are created according to the training task CRD of the target distributed training task, and training metadata corresponding to each working container is generated, wherein each working container is mounted with multiple hardware GPUs.
[0072] In step S22, based on the startup container, the training metadata corresponding to each working container is written into the database.
[0073] In step S23, multiple working containers asynchronously and in parallel read the training metadata from the database.
[0074] In step S24, any working container starts a training process in the working container based on the training metadata corresponding to the working container read from the database and using multiple hardware GPUs mounted on the working container.
[0075] In a possible implementation, the method further includes:
[0076] Based on the coordinator, a training task CRD is created for the target distributed training task, and a startup container is created based on the training task CRD.
[0077] In one possible implementation, upon starting the container, the training metadata corresponding to each working container is written to the database, including:
[0078] By reading the training task CRD, creating multiple working containers and generating training metadata corresponding to each working container;
[0079] Write the training metadata corresponding to each working container to the database.
[0080] In a possible implementation, the method further includes:
[0081] For any working container, read the training metadata corresponding to the working container from the database;
[0082] The training process is started in the working container according to the training process start command included in the training metadata corresponding to the working container.
[0083] In a possible implementation, the method further includes:
[0084] For any working container, the training status information of the working container is determined by monitoring the training process in the working container;
[0085] The training status information of the working container is written into the training metadata corresponding to the working container stored in the database.
[0086] In a possible implementation, the method further includes:
[0087] Based on the started container, periodically poll the training metadata corresponding to each working container stored in the database;
[0088] Detect whether there is an abnormal working container based on the training status information included in the training metadata corresponding to each working container;
[0089] When an abnormal working container is detected, a training fault recovery strategy is determined based on the training status information corresponding to the abnormal working container.
[0090] In one possible implementation, the training failure recovery strategy is to delete the abnormal working container and create a new working container;
[0091] The method further includes:
[0092] Based on the startup container, delete the abnormal working container and create a new working container according to the training failure recovery strategy;
[0093] The training metadata of the newly created working container is determined according to the training metadata of the abnormal working container stored in the database, and the training metadata of the newly created working container is written into the database.
[0094] In a possible implementation, the method further includes:
[0095] Based on the newly created working container, the corresponding training metadata is read from the database, and the training process is resumed using the read training metadata.
[0096] In one possible implementation, the training failure recovery strategy is to restart the training process;
[0097] The method further includes:
[0098] Based on the startup container, according to the training fault recovery strategy, the training status information included in the training metadata corresponding to the abnormal working container stored in the database is updated to restart;
[0099] Based on the abnormal working container, the corresponding training metadata is read from the database, and the training process is restarted according to the training status information included in the read training metadata.
[0100] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.
[0101] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.
[0102] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.
[0103] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
[0104] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0105] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0106] The electronic device may be provided as a terminal, a server, or other forms of devices.
[0107] Figure 3 FIG. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 3 , the electronic device 1900 can be provided as a server or a terminal device. Figure 3The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.
[0108] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.
[0109] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0110] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0111] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.
[0112] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0113] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0114] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0115] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0116] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0117] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0118] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0119] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.
[0120] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0121] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0122] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A distributed training scheduling system, characterized in that: The system includes a startup container, multiple working containers, and a database; The startup container is used to create the multiple working containers according to the training task custom resource CRD of the target distributed training task, and generate training metadata corresponding to each working container, wherein each working container is mounted with multiple hardware graphics processing devices GPU; The startup container is used to write the training metadata corresponding to each working container into the database; The multiple working containers are used to asynchronously and parallelly read training metadata from the database; Any working container is used to start a training process in the working container using multiple hardware GPUs mounted on the working container according to the training metadata corresponding to the working container read from the database.
2. The system according to claim 1, wherein: The system further includes: a coordinator; The coordinator is used to create the training task CRD for the target distributed training task, and create the startup container according to the training task CRD.
3. The system according to claim 1, wherein: The startup container is specifically used to: By reading the training task CRD, creating the multiple working containers, and generating training metadata corresponding to each working container; The training metadata corresponding to each working container is written into the database.
4. The system according to claim 1, wherein: Any working container, specifically used for: Reading the training metadata corresponding to the working container from the database; The training process is started in the working container according to the training process start command included in the training metadata corresponding to the working container.
5. The system according to claim 1, wherein: Any working container, specifically used for: Determine the training status information of the working container by monitoring the training process in the working container; The training status information of the working container is written into the training metadata corresponding to the working container stored in the database.
6. The system according to claim 5, characterized in that The startup container is also used to: Periodically polling the training metadata corresponding to each working container stored in the database; Detect whether there is an abnormal working container based on the training status information included in the training metadata corresponding to each working container; When an abnormal working container is detected, a training fault recovery strategy is determined based on the training state information corresponding to the abnormal working container.
7. The system according to claim 6, characterized in that The training fault recovery strategy is to delete the abnormal working container and create a new working container; The startup container is specifically used to: According to the training fault recovery strategy, delete the abnormal working container and create a new working container; The training metadata of a newly created working container is determined according to the training metadata of the abnormal working container stored in the database, and the training metadata of the newly created working container is written into the database.
8. The system according to claim 7, characterized in that The newly created working container is used to read the corresponding training metadata from the database and resume the training process using the read training metadata.
9. The system according to claim 7, wherein: The training fault recovery strategy is to restart the training process; The startup container is used to update the training status information included in the training metadata corresponding to the abnormal working container stored in the database to restart according to the training fault recovery strategy; The abnormal working container is used to read the corresponding training metadata from the database, and restart the training process according to the training status information included in the read training metadata.
10. A distributed training scheduling method, characterized in that: include: Based on the startup container, multiple working containers are created according to the training task CRD of the target distributed training task, and training metadata corresponding to each working container is generated. Each working container is mounted with multiple hardware GPUs. Based on the startup container, writing the training metadata corresponding to each working container into the database; The multiple working containers asynchronously and in parallel read the training metadata from the database; Any working container starts a training process in the working container based on the training metadata corresponding to the working container read from the database and using multiple hardware GPUs mounted on the working container.
11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method of claim 10.
12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to claim 10 is implemented.
13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that: When the computer program is executed by a processor, the method according to claim 10 is implemented.