A distributed training method and platform based on GPU cluster, and electronic equipment

Abstract

The present disclosure relates to a distributed training method and platform based on a GPU cluster, and an electronic device. The method includes: performing fault detection on each training node in the GPU cluster, and when a faulty training node is detected, initiating a GPU snapshot request to the faulty training node, determining a GPU memory snapshot corresponding to the faulty training node, wherein the training node is a node in the GPU cluster that executes training tasks, and the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node; creating a target memory object for the GPU memory snapshot, and storing the target memory object in a distributed memory management system; starting an idle node in the GPU cluster, and copying the target memory object from the distributed memory management system to the idle node; and after performing GPU memory snapshot recovery on the idle node using the target memory object, continuing to execute the training task using the idle node.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a distributed training method and platform based on a GPU cluster, and an electronic device. Background Art

[0002] Checkpoint technology is a technology that can be applied to model training (for example, AI training). When the training time is long, or distributed training is performed on many machines, the possibility of machine failure increases. If a machine fails, the training framework can be restored from the last saved checkpoint without having to start from the beginning. When it is found that convergence cannot be achieved or errors increase during training, it can be stopped and returned to the last saved checkpoint. The dataset may need to be updated during training. In this case, training can also be restarted based on the most recent checkpoint. In related technologies, model training based on checkpoint technology requires writing all checkpoint data to disk and then reading it from disk to restart training. This will result in a large loss of training time and low training efficiency. Summary of the Invention

[0003] This disclosure proposes a technical solution for a distributed training method and platform based on a GPU cluster, as well as electronic equipment.

[0004] According to one aspect of the present disclosure, a distributed training method based on a GPU cluster is provided, comprising: performing fault detection on each training node in the GPU cluster, and when a faulty training node is detected, initiating a GPU snapshot request to the faulty training node, and determining a GPU memory snapshot corresponding to the faulty training node, wherein the training node is a node in the GPU cluster that performs a training task, and the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node; creating a target memory object for the GPU memory snapshot, and storing the target memory object in a distributed memory management system; starting an idle node in the GPU cluster, and copying the target memory object from the distributed memory management system to the idle node; and after performing GPU memory snapshot recovery on the idle node using the target memory object, continuing to execute the training task using the idle node.

[0005] In one possible implementation, the method further includes: controlling a GPU memory monitor component within each node in the GPU cluster to determine GPU memory information corresponding to the node by monitoring the GPU L2 cache within the node; and storing the GPU memory information corresponding to the node in a GPU memory page table component within the node.

[0006] In one possible implementation, when it is determined that there is a faulty training node, a GPU snapshot request is initiated to the faulty training node, and a GPU memory snapshot corresponding to the faulty training node is determined, including: when the existence of the faulty training node is detected, calling the GPU snapshot interface corresponding to the faulty training node, and initiating the GPU snapshot request to the faulty training node; controlling the faulty training node to respond to the GPU snapshot request, and performing GPU snapshot processing based on the current GPU memory information stored in the faulty training node to obtain the GPU memory snapshot.

[0007] In one possible implementation, the distributed memory management system includes: a memory object daemon; creating a target memory object for the GPU memory snapshot includes: copying the GPU memory snapshot corresponding to the faulty training node to the host reserved memory space within the faulty training node; based on the memory object daemon, reading the GPU memory snapshot from the host reserved memory space within the faulty training node, and creating a target memory object for the GPU memory snapshot.

[0008] In one possible implementation, the distributed memory management system includes: a memory object space within each node in the GPU cluster; storing the target memory object in the distributed memory management system includes: storing the target memory object in the memory object space within each node in the GPU cluster based on the memory object daemon.

[0009] In a possible implementation, the method further includes: maintaining a memory object space in each node in the GPU cluster based on transparent huge page technology.

[0010] In one possible implementation, copying the target memory object from the distributed memory management system to the idle node includes: based on the memory object daemon, copying the target memory object from the memory object space within the idle node to the host reserved memory space within the idle node.

[0011] In a possible implementation, the idle node is a reserved node in the GPU cluster, or the space node is an elastically pulled-up node in the GPU cluster.

[0012] In a possible implementation, performing fault detection on each training node in the GPU cluster includes: controlling a fault management component in each training node in the GPU cluster to perform fault detection on the training node.

[0013] According to one aspect of the present disclosure, a distributed training platform based on a GPU cluster is provided, including: a fault management system for performing fault detection on each training node in the GPU cluster, and when a faulty training node is detected, initiating a GPU snapshot request to the faulty training node, and determining a GPU memory snapshot corresponding to the faulty training node, wherein the training node is a node in the GPU cluster that performs training tasks, and the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node; a distributed memory management system for creating a target memory object for the GPU memory snapshot and storing the target memory object; the fault management system is also used to start an idle node in the GPU cluster; the distributed memory management system is also used to copy the target memory object to the idle node; a training system for continuing to execute the training task using the idle node after performing GPU memory snapshot recovery on the idle node using the target memory object.

[0014] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0015] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0016] In an embodiment of the present disclosure, fault detection is performed on each training node that executes a training task in a GPU cluster. When a faulty training node is detected, a GPU snapshot request is initiated to the faulty training node, and the GPU memory snapshot corresponding to the faulty training node is determined, thereby effectively and quickly saving the GPU memory information of the faulty training node; a target memory object is created for the GPU memory snapshot, and the target memory object is stored in a distributed memory management system, thereby achieving unified management of the target memory object and preparing for subsequent cross-node GPU memory snapshot recovery; an idle node in the GPU cluster is started, and the target memory object is copied from the distributed memory management system to the idle node, so that after the target memory object is used to perform GPU memory snapshot recovery on the idle node, the idle node can be used to effectively continue to execute the interrupted training task on the faulty training node, thereby effectively improving the training efficiency of distributed training.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0019] Figure 1 A flowchart of a distributed training method based on a GPU cluster according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A schematic diagram of distributed training based on a GPU cluster according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram illustrating a node in a GPU cluster according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram illustrating a distributed memory management system corresponding to a GPU cluster according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A block diagram of a distributed training platform based on a GPU cluster according to an embodiment of the present disclosure is shown.

[0024] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0026] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0027] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0028] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0029] In related technologies, model training based on checkpointing requires writing all checkpoint data to disk and then re-reading it to restart training. Small models directly use the training framework's checkpoint API to save checkpoint data to local disk. Large-scale models require model splitting, saving each node's checkpoint data to a distributed file system, and then merging them.

[0030] During large-scale training, the amount of model checkpoint data can be very large. For example, the checkpoint data size corresponding to a 130B GPT model is nearly 2TB. The training framework requires that all node checkpoint data be saved to disk before training can continue. This causes training interruption time to increase with increasing model size. In addition, in the scenario of automatic recovery after training interruption, the checkpoint data of all nodes needs to be written to disk and re-read from disk before training can resume. Due to the large amount of checkpoint data, the read and write I / O volume and read and write time will be very large, which will seriously affect model training efficiency.

[0031] To address the above issues, the present disclosure provides a distributed training method based on a GPU cluster, which can improve the efficiency of saving checkpoint data and thus improve training efficiency. The distributed training method provided by the present disclosure is described in detail below.

[0032] Figure 1 The flowchart of a distributed training method based on a GPU cluster according to an embodiment of the present disclosure is shown. The method can be executed by a distributed training platform based on a GPU cluster, which includes a software system, a hardware system, and a control system for executing distributed training, etc. The present disclosure does not make specific limitations on this. Figure 1 As shown, the method includes:

[0033] In step S11, fault detection is performed on each training node in the GPU cluster, and when a faulty training node is detected, a GPU snapshot request is initiated to the faulty training node to determine the GPU memory snapshot corresponding to the faulty training node, where the training node is a node that executes training tasks in the GPU cluster, and the GPU memory snapshot includes the current GPU memory information corresponding to the faulty training node.

[0034] During distributed training, the model is split into multiple parts and distributed across multiple training nodes. The training app on each training node performs a training round based on a sample subset (mini batch), then exchanges data and proceeds to the next training round based on the next mini batch. The communication mode between training nodes can be allgathere, allreduce, or other modes, which are not specifically limited in this disclosure. During a training round, the training tasks on each training node run independently.

[0035] The distributed training method of the embodiment of the present disclosure is applied to a scenario where large-scale training is performed using a GPU cluster. The GPU cluster includes multiple training nodes that execute training tasks. During a certain round of training, the training tasks on a single training node run independently.

[0036] Figure 2 A schematic diagram of a distributed training based on a GPU cluster according to an embodiment of the present disclosure is shown. Figure 2 As shown, nodes 1 to 5 in the GPU cluster are training nodes that execute training tasks.

[0037] Perform fault detection on each training node in the GPU cluster to detect any faulty training nodes. A faulty training node is one that cannot properly execute training tasks. The specific methods for performing fault detection on each training node in the GPU cluster will be described in detail later in conjunction with possible implementations of the present disclosure and are not detailed here.

[0038] In one example, the fault of the faulty training node may be a software fault, such as a process out of memory (OOM), a runtime error, a driver error, etc., which is not specifically limited in this disclosure. The number of faulty training nodes may be one or more, which is not specifically limited in this disclosure.

[0039] When a faulty training node is detected, a GPU snapshot request is initiated to the faulty training node to determine the GPU memory snapshot corresponding to the faulty training node, thereby effectively and quickly saving the current GPU memory information corresponding to the faulty training node. The specific method for initiating a GPU snapshot request to the faulty training node upon detection of a faulty training node and determining the GPU memory snapshot corresponding to the faulty training node will be described in detail later in conjunction with possible implementations of the present disclosure, and is not further elaborated here.

[0040] by Figure 2As shown in the example, a fault is detected in training node 1. At this time, training node 1 is a faulty training node. A GPU snapshot request is initiated to the faulty training node 1 to determine the GPU memory snapshot corresponding to the faulty training node 1, so as to effectively and quickly save the current GPU memory information corresponding to the faulty training node 1.

[0041] In step S12, a target memory object is created for the GPU memory snapshot, and the target memory object is stored in the distributed memory management system.

[0042] After obtaining the GPU memory snapshot corresponding to the failed training node, a target memory object is created for the GPU memory snapshot and stored in the distributed memory management system. This effectively implements unified management of the target memory object and prepares for subsequent cross-node GPU memory snapshot recovery. The specific methods for creating the target memory object for the GPU memory snapshot and storing the target memory object in the distributed memory management system will be described in detail later in conjunction with possible implementations of the present disclosure and are not detailed here.

[0043] by Figure 2 As shown in the figure, for example, a target memory object is created for the GPU memory snapshot corresponding to the faulty training node 1, and the target memory object is stored in the distributed memory management system.

[0044] In step S13 , an idle node in the GPU cluster is started, and the target memory object is copied from the distributed memory management system to the idle node.

[0045] Start an idle node in the GPU cluster, where an idle node refers to a node that is not executing training tasks. Copy the target memory object corresponding to the failed training node from the distributed memory management system to the idle node, so that GPU memory snapshot recovery can be performed based on the target memory object on the idle node. The specific method of copying the target memory object from the distributed memory management system to the idle node will be described in detail later in conjunction with possible implementations of the present disclosure and is not detailed here.

[0046] like Figure 2 As shown, node 6 in the GPU cluster is an idle node that is not executing any training tasks. The idle node 6 in the GPU cluster is started, and the target memory object corresponding to the faulty training node 1 is copied from the distributed memory management system to the idle node 6.

[0047] In step S14, after the GPU memory snapshot is restored on the idle node using the target memory object, the training task is continued using the idle node.

[0048] After using the target memory object to perform GPU memory snapshot recovery on the idle node, the GPU memory information corresponding to the idle node will be consistent with the memory information corresponding to the faulty training node when the GPU snapshot was executed. At this time, the idle node will serve as a new training node to continue executing the interrupted training task on the faulty training node, thereby effectively improving the efficiency of training recovery.

[0049] by Figure 2 Taking the example shown above, after performing GPU memory snapshot recovery on idle node 6 using the target memory object corresponding to the faulty training node 1, the GPU memory information corresponding to idle node 6 will be consistent with the memory information corresponding to when the GPU snapshot was executed on the faulty training node 1. At this time, idle node 6 will serve as a new training node 6 to continue executing the interrupted training task on the faulty training node 1.

[0050] According to an embodiment of the present disclosure, fault detection is performed on each training node that executes a training task in a GPU cluster. When a faulty training node is detected, a GPU snapshot request is initiated to the faulty training node, and the GPU memory snapshot corresponding to the faulty training node is determined, thereby effectively and quickly saving the GPU memory information of the faulty training node; a target memory object is created for the GPU memory snapshot, and the target memory object is stored in a distributed memory management system, thereby achieving unified management of the target memory object and preparing for subsequent cross-node GPU memory snapshot recovery; an idle node in the GPU cluster is started, and the target memory object is copied from the distributed memory management system to the idle node, so that after the target memory object is used to perform GPU memory snapshot recovery on the idle node, the idle node can be used to effectively continue to execute the interrupted training task on the faulty training node, thereby effectively improving the training efficiency of distributed training.

[0051] In one possible implementation, the method further includes: controlling a GPU memory monitor component within each node in the GPU cluster to determine GPU memory information corresponding to the node by monitoring the GPU L2 cache within the node; and storing the GPU memory information corresponding to the node in a GPU memory page table component within the node.

[0052] For any node, the GPU memory monitor component in the node can effectively determine the GPU memory information corresponding to the node by monitoring the GPU L2 cache in the node, and then effectively store the GPU memory information corresponding to the node in the GPU memory page table component in the node.

[0053] In one example, each node may include one, two, or more GPUs, and the GPU memory monitor component within the node may monitor the GPU L2 cache of each GPU within the node. The specific number of GPUs included in each node can be flexibly set based on actual circumstances and is not specifically limited in this disclosure.

[0054] Figure 3 A schematic diagram of a node in a GPU cluster according to an embodiment of the present disclosure is shown. Figure 3 As shown, the node includes two GPUs: GPU0 and GPU1. The GPU memory monitor component in the node monitors the GPU L2 cache of GPU0 and the GPU L2 cache of GPU1, obtains the GPU memory information corresponding to the node, and stores it in the GPU memory page table component in the node.

[0055] In a possible implementation, performing fault detection on each training node in the GPU cluster includes controlling a fault management component in each training node in the GPU cluster to perform fault detection on the training node.

[0056] The GPU cluster executes the checkpoint business process during the distributed training process, controls the fault management components in each training node in the GPU cluster, and performs fault detection on the training node, thereby effectively determining whether the training node has a fault.

[0057] For any training node, the fault management component within the training node performs fault detection on the training node according to a preset inspection period to determine whether the training node has a fault. The specific value of the preset inspection period can be flexibly set according to actual conditions and is not specifically limited in this disclosure.

[0058] In one possible implementation, when it is determined that there is a faulty training node, a GPU snapshot request is initiated to the faulty training node, and a GPU memory snapshot corresponding to the faulty training node is determined, including: when a faulty training node is detected, calling the GPU snapshot interface corresponding to the faulty training node, and initiating a GPU snapshot request to the faulty training node; controlling the faulty training node to respond to the GPU snapshot request, performing GPU snapshot processing based on the current GPU memory information stored in the faulty training node, and obtaining a GPU memory snapshot.

[0059] When a faulty training node is detected in the GPU cluster, the GPU snapshot interface corresponding to the faulty training node is called, and a GPU snapshot request is initiated to the faulty training node, so that GPU snapshot processing can be performed based on the current GPU memory information stored in the GPU memory page table component in the faulty training node, and the GPU memory snapshot corresponding to the faulty training node is obtained, that is, the latest checkpoint data corresponding to the faulty training node.

[0060] In a possible implementation, the method further includes: copying a GPU memory snapshot corresponding to the faulty training node to a host reserved memory space within the faulty training node.

[0061] For any node in the GPU cluster, the GPU memory page table component within that node can be accessed by the host within that node. The host reserves memory space and copies the GPU memory snapshot corresponding to the faulty training node to the host's reserved memory space within the faulty training node, thus resolving the issue of excessive latency associated with direct access to the distributed memory management system.

[0062] like Figure 3 As shown in the figure, the node includes the host, and the host includes the host reserved memory space. Figure 3 When the node shown is a faulty training node, the faulty training node responds to the GPU snapshot request, performs GPU snapshot processing based on the current GPU memory information stored in the GPU memory page table component in the faulty training node, obtains the GPU memory snapshot corresponding to the faulty training node, and copies the GPU memory snapshot to the host reserved memory space in the faulty training node.

[0063] In one possible implementation, the distributed memory management system includes: a memory object daemon; creating a target memory object for a GPU memory snapshot, including: reading the GPU memory snapshot from a host reserved memory space in a faulty training node based on the memory object daemon, and creating a target memory object for the GPU memory snapshot.

[0064] The memory object daemon in the distributed memory management system is used to initialize the memory object space in each node in the GPU cluster, maintain memory objects, and transfer memory objects between different nodes.

[0065] After the faulty training node copies the corresponding GPU memory snapshot to the host reserved memory space, the GPU memory snapshot can be read from the host reserved memory space in the faulty training node based on the memory object daemon, and a target memory object can be created for the GPU memory snapshot.

[0066] In one possible implementation, the distributed memory management system includes: a memory object space within each node in the GPU cluster; storing the target memory object in the distributed memory management system includes: storing the target memory object in the memory object space within each node in the GPU cluster based on a memory object daemon process.

[0067] Based on the memory object daemon, the target memory object corresponding to the faulty training node is stored in the memory object space of each node in the GPU cluster, effectively enabling each node in the GPU cluster to share the target memory object corresponding to the faulty training node.

[0068] In a possible implementation, the method further includes: maintaining the memory object space in each node in the GPU cluster based on transparent huge page technology.

[0069] In order to reduce the memory access latency caused by paging, the memory object space in each node in the GPU cluster is maintained based on transparent huge page technology, thereby effectively improving the memory access efficiency of the distributed memory management system.

[0070] Figure 4 Schematic diagram showing a distributed memory management system corresponding to a GPU cluster according to an embodiment of the present disclosure. Figure 4 As shown, the GPU cluster includes nodes 1 through N, each of which contains host-reserved memory space. After node 1 copies the corresponding GPU memory snapshot to the host-reserved memory space, the memory daemon process included in the distributed memory management system reads the GPU memory snapshot from node 1's host-reserved memory space and creates a target memory object for the GPU memory snapshot. The memory daemon then stores the target memory object in the memory object spaces of nodes 1 through N.

[0071] To continue training tasks that were interrupted on the faulty training node, start idle nodes in the GPU cluster.

[0072] In a possible implementation, the idle node is a reserved node in the GPU cluster, or the space node is a node elastically pulled up in the GPU cluster.

[0073] In a possible implementation, copying the target memory object from the distributed memory management system to the idle node includes: based on the memory object daemon process, copying the target memory object from the memory object space in the idle node to the host reserved memory space in the idle node.

[0074] Based on the memory object daemon included in the distributed memory management system, the target memory object corresponding to the faulty training node is copied from the memory object space in the idle node in the distributed memory management system to the host reserved memory space in the idle node. The user is unaware of the copying process, which improves the user experience.

[0075] The idle node performs GPU memory snapshot recovery based on the target memory object corresponding to the faulty training node in the host's reserved memory space, so that the GPU memory information corresponding to the idle node is consistent with the memory information corresponding to the faulty training node when the GPU snapshot is executed. At this time, the idle node will serve as a new training node to continue executing the interrupted training task on the faulty training node, thereby effectively improving the efficiency of training recovery.

[0076] In an embodiment of the present disclosure, fault detection is performed on each training node that executes a training task in a GPU cluster. When a faulty training node is detected, a GPU snapshot request is initiated to the faulty training node, and the GPU memory snapshot corresponding to the faulty training node is determined, thereby effectively and quickly saving the GPU memory information of the faulty training node; a target memory object is created for the GPU memory snapshot, and the target memory object is stored in a distributed memory management system, thereby achieving unified management of the target memory object, and preparing for subsequent cross-node GPU memory snapshot recovery; an idle node in the GPU cluster is started, and the target memory object is copied from the distributed memory management system to the idle node, so that after the target memory object is used to perform GPU memory snapshot recovery on the idle node, the idle node can be used to effectively continue to execute the interrupted training task on the faulty training node, effectively achieving user-imperceptible automatic training task recovery after interruption, effectively improving the training efficiency of distributed training, and improving the user experience.

[0077] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0078] In addition, the present disclosure also provides a distributed training platform based on a GPU cluster, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the distributed training methods based on a GPU cluster provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0079] Figure 5 FIG. 1 is a block diagram of a distributed training platform based on a GPU cluster according to an embodiment of the present disclosure. Figure 5 As shown in the figure, the distributed training platform based on GPU cluster includes:

[0080] A fault management system is configured to perform fault detection on each training node in the GPU cluster and, upon detecting a faulty training node, initiate a GPU snapshot request to the faulty training node and determine a GPU memory snapshot corresponding to the faulty training node, where the training node is a node in the GPU cluster that performs training tasks, and the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node;

[0081] Distributed memory management system, used to create target memory objects for GPU memory snapshots and store target memory objects;

[0082] The fault management system is also used to start idle nodes in the GPU cluster;

[0083] The distributed memory management system is also used to copy the target memory object to the idle node;

[0084] The training system is used to restore the GPU memory snapshot on the idle node using the target memory object and then continue to execute the training task using the idle node.

[0085] In one possible implementation, the distributed training platform based on a GPU cluster further includes a GPU memory monitor component in each node of the GPU cluster, configured to:

[0086] Determine the GPU memory information corresponding to the node by monitoring the GPU L2 cache in the node;

[0087] The GPU memory information corresponding to the node is stored in the GPU memory page table component within the node.

[0088] In one possible implementation, the fault management system is specifically configured to:

[0089] When a faulty training node is detected, the GPU snapshot interface corresponding to the faulty training node is called to initiate a GPU snapshot request to the faulty training node.

[0090] The faulty training node is controlled to respond to the GPU snapshot request, and GPU snapshot processing is performed based on the current GPU memory information stored in the faulty training node to obtain a GPU memory snapshot.

[0091] In one possible implementation, the distributed memory management system includes: a memory object daemon process;

[0092] Fault management system, specifically for:

[0093] Copy the GPU memory snapshot corresponding to the faulty training node to the host reserved memory space within the faulty training node;

[0094] Distributed memory management system, specifically used for:

[0095] Based on the memory object daemon, the GPU memory snapshot is read from the host reserved memory space in the faulty training node, and a target memory object is created for the GPU memory snapshot.

[0096] In one possible implementation, the distributed memory management system includes: a memory object space within each node in the GPU cluster;

[0097] Distributed memory management system, specifically used for:

[0098] Based on the memory object daemon, the target memory object is stored in the memory object space of each node in the GPU cluster.

[0099] In one possible implementation, the distributed memory management system is specifically configured to:

[0100] Based on transparent huge page technology, it maintains the memory object space within each node in the GPU cluster.

[0101] In one possible implementation, the distributed memory management system is specifically configured to:

[0102] Based on the memory object daemon, the target memory object is copied from the memory object space in the idle node to the host reserved memory space in the idle node.

[0103] In a possible implementation, the idle node is a reserved node in the GPU cluster, or the space node is a node elastically pulled up in the GPU cluster.

[0104] In one possible implementation, the fault management system is specifically configured to:

[0105] Control the fault management component within each training node in the GPU cluster to perform fault detection on the training node.

[0106] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0108] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0109] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0110] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0111] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0112] Figure 6 FIG. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 , the electronic device 1900 can be provided as a server or a terminal device. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0113] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0114] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0115] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0116] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0117] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0118] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0119] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0121] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0123] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0124] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0125] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0126] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0127] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A distributed training method based on GPU cluster, characterized in that: include: Performing fault detection on each training node in the GPU cluster, and when a faulty training node is detected, initiating a GPU snapshot request to the faulty training node, determining a GPU memory snapshot corresponding to the faulty training node, wherein the training node is a node in the GPU cluster that performs training tasks, the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node, and the faulty training node has a software fault; Creating a target memory object for the GPU memory snapshot, and storing the target memory object in a distributed memory management system; Starting an idle node in the GPU cluster, and copying the target memory object from the distributed memory management system to the idle node; After performing a GPU memory snapshot recovery on the idle node using the target memory object, continuing to execute the training task using the idle node; The distributed memory management system includes: a memory object daemon process and a memory object space within each node in the GPU cluster; creating a target memory object for the GPU memory snapshot and storing the target memory object in the distributed memory management system include: creating a target memory object for the GPU memory snapshot based on the memory object daemon process, and storing the target memory object in the memory object space within each node in the GPU cluster.

2. The method according to claim 1, characterized in that The method further comprises: Controlling a GPU memory monitor component in each node of the GPU cluster to determine GPU memory information corresponding to the node by monitoring the GPU L2 cache in the node; The GPU memory information corresponding to the node is stored in the GPU memory page table component within the node.

3. The method according to claim 1, characterized in that When a faulty training node is detected, initiating a GPU snapshot request to the faulty training node and determining a GPU memory snapshot corresponding to the faulty training node includes: When the faulty training node is detected, calling the GPU snapshot interface corresponding to the faulty training node and initiating the GPU snapshot request to the faulty training node; The faulty training node is controlled to respond to the GPU snapshot request, and GPU snapshot processing is performed based on current GPU memory information stored in the faulty training node to obtain the GPU memory snapshot.

4. The method according to claim 1, wherein The step of creating a target memory object for the GPU memory snapshot includes: Copying the GPU memory snapshot corresponding to the faulty training node to the host reserved memory space within the faulty training node; Based on the memory object daemon, the GPU memory snapshot is read from the host reserved memory space in the faulty training node, and a target memory object is created for the GPU memory snapshot.

5. The method according to claim 1, characterized in that The method further comprises: Based on transparent huge page technology, the memory object space in each node in the GPU cluster is maintained.

6. The method according to claim 1, characterized in that The copying of the target memory object from the distributed memory management system to the idle node comprises: Based on the memory object daemon process, the target memory object is copied from the memory object space in the idle node to the host reserved memory space in the idle node.

7. The method according to claim 1, characterized in that The idle node is a reserved node in the GPU cluster, or the idle node is an elastically pulled-up node in the GPU cluster.

8. The method according to claim 1, characterized in that The fault detection for each training node in the GPU cluster includes: Control the fault management component in each training node in the GPU cluster to perform fault detection on the training node.

9. A distributed training platform based on GPU cluster, characterized by: include: A fault management system is configured to perform fault detection on each training node in the GPU cluster, and when a faulty training node is detected, initiate a GPU snapshot request to the faulty training node, and determine a GPU memory snapshot corresponding to the faulty training node, wherein the training node is a node in the GPU cluster that performs training tasks, the GPU memory snapshot includes current GPU memory information corresponding to the faulty training node, and the faulty training node has a software fault; A distributed memory management system, configured to create a target memory object for the GPU memory snapshot and store the target memory object; The fault management system is further used to start idle nodes in the GPU cluster; The distributed memory management system is further configured to copy the target memory object to the idle node; A training system, configured to, after performing a GPU memory snapshot restore on the idle node using the target memory object, continue to execute the training task using the idle node; The distributed memory management system includes: a memory object daemon process and a memory object space in each node in the GPU cluster; The memory object daemon is used to create a target memory object for the GPU memory snapshot and store the target memory object in the memory object space of each node in the GPU cluster.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images