Cluster-based training method, device, electronic device and storage medium

Abstract

To provide: a cluster-based training method and device for shortening a fault recovery time and improving a model training efficiency; electronic equipment; and a storage medium.SOLUTION: In a method, in response to the presence of a hardware fault in a training node, a target standby node is selected from multiple standby nodes, and a target training snapshot of a model training task in the training node is obtained. The target training snapshot comprises training state data of the model training task. In the method, based on a container image and the training state data of a model training procedure in the training node, the target standby node is initialized so that the target standby node replaces the training node to continuously execute the model training task.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of data processing technology, particularly to fields such as artificial intelligence and deep learning, and in particular to cluster-based training methods, devices, electronic devices, and storage media. [Background technology]

[0002] As artificial intelligence (AI) advances, the need for model training is also increasing. For training scenarios with high computing power demands, such as large-scale model training scenarios, relevant researchers have proposed using clusters to train large-scale models in order to improve the training efficiency of large-scale models. However, when using clusters to train large-scale models, the clusters provide high computing power and improve model training efficiency to a certain extent. However, if a single node in the cluster fails, the cluster will stop training operations, which will affect the model training efficiency. Therefore, when using clusters to train models, it is necessary to further improve the model training efficiency. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a cluster-based training method, apparatus, device, and storage medium. [Means for solving the problem]

[0004] According to a first aspect of the present disclosure, there is provided a cluster-based training method, the cluster including: a training node for performing a model training task and a plurality of standby nodes; In response to a hardware failure in a training node, selecting a target standby node from a plurality of standby nodes and obtaining a target training snapshot of a model training task at the training node, the target training snapshot including training state data of the model training task; and initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node.

[0005] According to a second aspect of the present disclosure, there is provided a cluster-based training apparatus, the cluster including: a training node for performing a model training task and a plurality of standby nodes; a first response module for selecting a target standby node from a plurality of standby nodes in response to a hardware failure in a training node and obtaining a target training snapshot of a model training task at the training node, the target training snapshot including training state data of the model training task; an initialization module for initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continuously performs the model training task in place of the training node.

[0006] According to a third aspect of the present disclosure, there is provided an electronic device, at least one processor; a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the cluster-based training method described in the first aspect.

[0007] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions causing the computer to perform the cluster-based training method according to the first aspect.

[0008] According to a fifth aspect of the present disclosure, there is provided a computer program, the computer instructions of which, when executed by a processor, implement the steps of the cluster-based training method according to the first aspect.

[0009] The cluster-based training method, apparatus, and electronic device provided by the present disclosure have the following beneficial effects: in response to a hardware failure in a training node, the present disclosure selects a target standby node from a plurality of standby nodes, obtains a target training snapshot of a model training task in the training node, the target training snapshot includes training state data of the model training task, and initializes the target standby node based on a container image of the model training procedure in the training node and the training state data, so that the target standby node replaces the training node and continuously executes the model training task. Thus, when a hardware failure occurs in the training node, the present disclosure can quickly initialize the target standby node based on the container image and the target training snapshot, so that the target standby node quickly replaces the training node and continuously executes the model training task, thereby shortening the failure recovery time and improving model training efficiency.

[0010] It should be understood that the contents set forth herein are not intended to identify key or important features of the embodiments of the present disclosure, nor should they be used to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following description. [Brief explanation of the drawings]

[0011] The drawings are used for a better understanding of the present solution and are not intended to limit the present disclosure. [Figure 1] 1 is a flowchart of a cluster-based training method provided by one embodiment of the present disclosure. [Figure 2] 1 is a flowchart of a cluster-based training method provided by another embodiment of the present disclosure. [Figure 3] 10 is a flowchart of a cluster-based training method provided by yet another embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic block diagram of a cluster-based training device provided according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram of an electronic device for implementing the cluster-based training method of an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings. Various details of the embodiments of the present disclosure are included here for ease of understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and simplicity, the following description will omit descriptions of known functions and structures.

[0013] Hereinafter, a cluster-based training method, apparatus, and electronic device according to embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be noted that the entity that executes the cluster-based training method of this embodiment is a cluster-based training device, which can be realized in the form of software and / or hardware and can be arranged in an electronic device.

[0014] FIG. 1 is a flowchart of a cluster-based training method provided by one embodiment of the present disclosure.

[0015] A cluster includes a training node and multiple standby nodes for performing model training tasks. The number of training nodes included in a cluster may be one or more. When the number of training nodes in a cluster is multiple, one model can be trained by multiple training nodes, and each training node is responsible for a part of the training task of the model.

[0016] The training node and the standby node may refer to nodes that have undergone pressure measurement, such as nodes that have undergone communication unit pressure measurement, calculation unit pressure measurement, and model end-to-end pressure measurement. The period for pressure measurement of the node may be determined by the user according to actual needs.

[0017] As shown in FIG. 1, the cluster-based training method includes the following steps: Step 101: In response to a hardware failure occurring in a training node, select a target standby node from a plurality of standby nodes, and obtain a target training snapshot of a model training task in the training node, where the target training snapshot includes training state data of the model training task.

[0018] Alternatively, a hardware failure may refer to a failure occurring in hardware in a training node, such as, for example and without limitation, a graphics processing unit (GPU), a central processing unit (CPU), a network card, etc.

[0019] Optionally, selecting a target standby node from the plurality of standby nodes may include randomly selecting one standby node from the plurality of standby nodes as the target standby node.

[0020] To increase node utilization and fully utilize node resources, standby nodes are generally not idle, i.e., they generally perform tasks other than model training tasks.

[0021] Optionally, the step of selecting a target standby node from the plurality of standby nodes includes the steps of obtaining a priority of the standby node, where the priority indicates the importance of other tasks, and selecting a target standby node from the plurality of standby nodes based on the priority.

[0022] Selecting a target standby node based on priority can, on the one hand, prevent other more important tasks from being terminated while they are running on the standby node, and, on the other hand, ensure high availability of the cluster and improve node utilization.

[0023] It is assumed that the higher the priority, the more important other tasks the standby node is to perform. When selecting a target standby node based on the current priority, the target standby node can be selected in order from lowest to highest priority. If there is no low-priority standby node or the low-priority standby node is unavailable, the high-priority standby node is selected as the target standby node.

[0024] By way of example and not limitation, assume that multiple standby nodes have corresponding priorities A and B, with A being higher than B. When selecting a target standby node, the standby node with priority B can be preferentially selected as the target standby node; if there is no standby node with priority B or the standby node with priority B is unavailable, the standby node with priority A is selected as the target standby node. Meanwhile, the number of standby nodes with priority A may be greater than the number of standby nodes with priority B.

[0025] Regarding GPU hardware, if a GPU fails, there is a high probability that it will fail again during subsequent use. In order to reduce the frequency of failures across the entire cluster and improve training efficiency, the present disclosure proposes that previously failed GPUs can be concentrated on one or more nodes, and such nodes can be used as low-priority standby nodes and prioritized for scheduling.

[0026] Optionally, the present disclosure further includes obtaining a predicted availability of the cluster and a predicted number of failed nodes per unit time, and determining the number of standby nodes based on the predicted availability and the predicted number.

[0027] In an alternative implementation, the number of standby nodes can be obtained by Poisson fitting based on the predicted availability and the predicted number.

[0028] By way of example and not limitation, the number of standby nodes may be determined based on the following formula:

[0029]

number

[0030] Wherein, α is the predicted availability, μ is the predicted number, and x is the number of standby nodes to be solved. The number of standby nodes x is obtained by solving the above equation.

[0031] The predicted number of failed nodes per unit time can be obtained by statistically analyzing the historical data, and the predicted number of failed nodes per unit time can also be obtained by converting the predicted failure frequency of the cluster.

[0032] By determining the number of standby nodes based on the predicted availability and the predicted number, the standby nodes can cover for failed nodes, thereby achieving high cluster availability.

[0033] Alternatively, the target training snapshot can refer to a training snapshot at the time when a hardware failure occurs on a training node in a model training task, or can refer to a training snapshot at a time before a hardware failure occurs on a training node in a model training task.

[0034] The training state data of the model training task included in the target training snapshot may refer to data such as training data state, model parameters, and optimizer state, where the training data state is used to indicate whether the training data has already been used, and based on the training data state, the target standby node can directly train the model based on unused training data, thereby avoiding repeatedly training the model based on the same training data.

[0035] Step 102: Initialize a target standby node based on the container image of the model training procedure and the training state data in the training node, so that the target standby node continuously performs the model training task in place of the training node.

[0036] The container image of the model training procedure can be created in advance, and after determining the target standby node, the container image can be deployed to the target standby node, thereby improving the deployment speed of the model training procedure.

[0037] After the container image is deployed, data placement can be performed on the target standby node based on the training state data.

[0038] The process of initializing the target standby node can be understood as preempting the resources of the target standby node to perform model training tasks.

[0039] By initializing the target standby node based on the container image and training state data, hardware failure recovery can be performed automatically, thereby shortening the hardware failure recovery time and improving the hardware failure recovery efficiency.

[0040] In an embodiment of the present disclosure, in response to a hardware failure of a training node, a target standby node is selected from a plurality of standby nodes, a target training snapshot of a model training task on the training node is obtained, the target training snapshot includes training state data of the model training task, and the target standby node is initialized based on a container image and the training state data of the model training procedure on the training node so that the target standby node replaces the training node and continuously executes the model training task. Thus, when a hardware failure occurs on the training node, the present disclosure can quickly initialize the target standby node based on the container image and the target training snapshot, so that the target standby node quickly replaces the training node and continuously executes the model training task, thereby shortening the failure recovery time and improving the model training efficiency.

[0041] FIG. 2 is a flowchart of a cluster-based training method provided by another embodiment of the present disclosure.

[0042] As shown in FIG. 2 , the cluster-based training method includes the following steps: Step 201: In response to a hardware failure occurring in a training node, select a target standby node from a plurality of standby nodes, and obtain at least one training snapshot of a model training task in the training node, the training snapshot being obtained from the training node based on a target periodicity, and select a target training snapshot from the at least one training snapshot.

[0043] After the training node starts to execute the model training task, the training snapshot of the model training task is taken at regular intervals based on the target period, thereby avoiding the problems of data loss and invalid training caused by training node hardware failure.

[0044] The cluster further includes a storage node, and optionally, the step of obtaining at least one training snapshot of the model training task in the training node includes the steps of: controlling the training node to store the training snapshot in a memory of the training node based on a first process based on the target period; controlling the training node to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node based on a second process; and controlling the training node to read the training snapshot from the solid-state disk based on a third process and send the training snapshot to the storage node.

[0045] If the training node sequentially stores the training snapshot in memory, on a solid-state disk, and in a storage node through only one process, the process can only return to executing other instructions after storing the training snapshot in the storage node, resulting in low overall program execution efficiency.

[0046] By having multiple processes asynchronously acquire or store training snapshots, blocking during program execution can be avoided, further improving the concurrent performance of the program.

[0047] Alternatively, the target period can be determined empirically by the user.

[0048] Alternatively, the target period may be determined by the following method: A single predicted storage overhead of a training snapshot of the model training task is obtained, a total storage overhead function with a target period as a variable is constructed based on the set time length and the single predicted storage overhead, an invalid training overhead function with the target period as a variable is constructed, a target overhead function is determined based on the total storage overhead function and the invalid training overhead function, and a solving process is performed on the target overhead function to determine the target period.

[0049] By way of example and not limitation, a single predicted storage overhead may refer to the storage overhead associated with controlling a training node to store training snapshots in a storage node using three processes, or may refer to the overhead associated with storing training snapshots using other storage methods.

[0050] By way of example and not limitation, the set time length may refer to the average length of time from when the training node starts to execute the model training task until the model training task is interrupted, and may be a length of time determined by a user based on experience.

[0051] By way of example and not limitation, the target period may be determined by the following formula:

[0052]

number

[0053] Where H(t) is the target overhead function,

[0054]

number

[0055] is the total storage overhead function, T is the set time length, c is the predicted storage overhead for one time, t is a variable, i.e., the target period, t is greater than 0,

[0056]

number

[0057] is the invalid training overhead function, where invalid training can refer to training between the time the last training snapshot was taken before the hardware fault appeared and the time the hardware fault appeared. Note that the coefficient of the variable t in H2(t) can be determined according to the actual situation.

[0058] The target period can be obtained by solving the minimum value of H(t), where the total storage overhead and the unnecessary training overhead mentioned above can refer to the time overhead.

[0059] The target period obtained by solving the above formula is a good period, which can balance the total training snapshot storage overhead and the invalid training overhead, thereby minimizing the overhead during the model training process.

[0060] By way of example and not limitation, the target period may further be determined by the following formula:

[0061]

number

[0062] Among them, f failure is the predicted fault frequency during which the training node performs the model training task.

[0063] Optionally, selecting a target training snapshot from the at least one training snapshot includes selecting a target training snapshot from the at least one training snapshot based on a corresponding training effect, wherein the training effect can be determined based on a loss value corresponding to the training snapshot.

[0064] By selecting the training snapshot with the best corresponding training effect as the target training snapshot, the target standby node can continuously perform model training tasks based on the best model training result before the hardware failure occurs, thereby improving model training efficiency and reducing model training costs.

[0065] The training snapshots correspond to acquisition times, and optionally, from the at least one training snapshot, selecting a target training snapshot includes obtaining a failure time corresponding to a hardware failure of the training node, and selecting a training snapshot having a corresponding acquisition time and a shortest failure time interval as the target training snapshot.

[0066] By selecting the last training snapshot taken before the hardware failure occurs as the target training snapshot, the target standby node can base its model training results on the latest model training results and continuously execute model training tasks, ensuring continuity of model training and reducing unnecessary training overhead.

[0067] Step 202: Perform communication bandwidth verification and / or silent data error verification on the target standby node.

[0068] Optionally, performing communication bandwidth verification on the target standby node means verifying whether the communication bandwidth corresponding to the target standby node is equal to or greater than a bandwidth threshold, and if the communication bandwidth is equal to or greater than the bandwidth threshold, the communication bandwidth verification is passed; if the communication bandwidth is less than the bandwidth threshold, the communication bandwidth verification is not passed, where the communication bandwidth may refer to an aggregate communication bandwidth.

[0069] Alternatively, performing silent data error verification on the target standby node refers to the target standby node verifying whether a silent data error exists, and if no silent data error exists, the silent data error verification is passed, and if a silent data error exists, the silent data error verification is not passed.

[0070] By way of example and not limitation, a training node may perform a particular operation multiple times and compare the results of the operations; if the results are the same, then there is no silent data error in the training node; and if there are at least two different results in the results of the operations, then there is a silent data error in the training node.

[0071] By performing communication bandwidth verification and / or silent data error verification on the target standby node, risk monitoring can be performed on the performance of the target standby node, avoiding the introduction of poorly performing or faulty nodes and reducing the probability of subsequent failure of the target standby node.

[0072] Step 203: In response to passing the communication bandwidth verification and / or passing the silent data error verification, initialize a target standby node based on the container image and training state data of the model training procedure in the training node.

[0073] The relevant content of step 203 can refer to the relevant description of step 102, and will not be described in detail here.

[0074] In an embodiment of the present disclosure, in response to a hardware failure in the training node, a target standby node is selected from a plurality of standby nodes, at least one training snapshot of a model training task in the training node is acquired, the training snapshot being acquired from the training node based on a target periodicity, a target training snapshot is selected from the at least one training snapshot, a communication bandwidth verification and / or a silent data error verification is performed on the target standby node, and in response to the communication bandwidth verification and / or the silent data error verification being passed, the target standby node is initialized based on the container image of the model training procedure and the training state data in the training node. Thus, by acquiring the at least one training snapshot, the present disclosure can avoid data loss and invalid training issues caused by a training node hardware failure, and by performing the communication bandwidth verification and / or the silent data error verification on the target standby node, risk monitoring can be performed on the performance of the target standby node, and the introduction of a poorly performing or faulty node can be avoided.

[0075] FIG. 3 is a flowchart of a cluster-based training method provided according to another embodiment of the present disclosure.

[0076] As shown in FIG. 3 , the cluster-based training method includes the following steps: Step 301: fault monitoring is performed on the training node.

[0077] Optionally, the model training procedure at the training node can be monitored, i.e., whether the model training procedure terminates with a failure.

[0078] Step 302: In response to obtaining the failure code of the training node, determine the failure type of the training node based on a mapping relationship between the failure code and the failure type, where the failure type includes a hardware failure or a software failure.

[0079] If it is monitored that the model training procedure has ended with a failure, obtain the corresponding failure code, and search for the corresponding failure type based on the mapping relationship failure code to determine whether the training node has a hardware failure or a software failure.

[0080] The present disclosure realizes self-diagnosis of integrated hardware and software faults, and regardless of whether it is a hardware fault or a software fault, it is embodied in the model training procedure as a program fault termination, so that rapid detection of the fault can be realized based on the fault code.

[0081] Step 303, in response to a hardware failure existing in the training node, select a target standby node from the plurality of standby nodes, and obtain a target training snapshot of the model training task in the training node.

[0082] Compared with a fault monitoring method that scans a hardware error log at a fixed interval, the hardware fault monitoring method that performs hardware fault monitoring based on fault codes is more time-consuming. Furthermore, since there is a mapping relationship between the fault code and the hardware fault, the hardware in which the fault exists can be located based on the fault code.

[0083] Step 304, in response to the existence of a software fault in the training node, determines an abnormal process in the training node and restarts the abnormal process.

[0084] There is also a mapping relationship between the fault code and the software fault, so that the abnormal process can be determined based on the fault code, and further the abnormal process can be restarted.

[0085] The present disclosure can select corresponding failure recovery schemes in response to different failure types, shorten the failure recovery time, and ensure the training stability of the cluster. Alternatively, the failure types may be mapped as failure levels, and each failure level may correspond to a failure recovery scheme. For example and without limitation, the failure level corresponding to a software failure may be M, and the failure level corresponding to a hardware failure may be N, where the failure degree of the software failure is lower than that of the hardware failure, and therefore the failure level M may be lower than the failure level N. Wherein, the failure recovery scheme corresponding to the failure level M is to restart the abnormal process, and the failure recovery scheme corresponding to the failure level N is to select a target standby node and initialize the target standby node.

[0086] Specifically, the failure level can be determined based on the failure code, and a corresponding failure recovery method can be selected based on the failure level.

[0087] Alternatively, if the target standby node fails to initialize or fails to restart the abnormal process, the cluster can be scheduled, i.e., another cluster replaces the current cluster to continue training the model.

[0088] Based on the cluster-based training method of the present disclosure, the effective training rate for the cluster model can be effectively improved, and the effective training rate can be used to indicate the time occupancy that truly contributes to model parameter update within a desired model training time period. Optionally, the effective training rate of the cluster can be predicted by the following formula:

[0089]

number

[0090] where ρ is the effective training rate, α is the predicted availability of the cluster, and F failure is the predicted failure frequency of the cluster, and C recovery is the predicted single failure recovery time occupancy, and C recovery can be understood as the ratio of the time to recover from a single failure to the time to predict the training of the model, and F checkpoint is the expected frequency of saving training snapshots, and c is the expected overhead of one save.

[0091] After the training node completes the model training task, the actual values ​​of the parameters in the above formula can be obtained to determine the actual effective training rate of the cluster.

[0092] Based on the effective training rate and the training time, the effective training time can be determined, and based on the effective training time and the throughput data amount within a unit time, the total throughput data amount when the cluster executes the model training task can be determined.

[0093] In addition to including a training node and a standby node, the cluster also includes training service nodes, such as a scheduling node for selecting a target standby node and a monitoring node for fault monitoring of the training node, and fault monitoring of the training service node can be performed based on the communication status between the training service nodes.

[0094] In an embodiment of the present disclosure, based on the steps of performing fault monitoring on the training node; determining a fault type of the training node based on a mapping relationship between the fault code and the fault type in response to obtaining a fault code of the training node; selecting a target standby node from a plurality of standby nodes in response to a hardware fault existing in the training node; obtaining a target training snapshot of the model training task in the training node; and determining an abnormal process in the training node and resuming the abnormal process in response to a software fault existing in the training node, the present disclosure can select corresponding fault recovery schemes in response to different fault types and shorten the fault recovery time.

[0095] 4 is a schematic diagram of a cluster-based training apparatus provided according to an embodiment of the present disclosure. The cluster includes a training node and multiple standby nodes for performing model training tasks.

[0096] As shown in FIG. 4, the cluster-based training apparatus 400 includes a first response module 401 and an initialization module 402; The first response module 401 is used to select a target standby node from the plurality of standby nodes in response to a hardware failure existing in the training node, and obtain a target training snapshot of the model training task in the training node, wherein the target training snapshot includes training state data of the model training task; The initialization module 402 is used to initialize the target standby node based on the container image of the model training procedure and the training state data in the training node, so that the target standby node continuously performs the model training task in place of the training node.

[0097] Optionally, the first response module 401: Acquire at least one training snapshot of the model training task in the training node, the training snapshot being acquired from the training node based on the target periodicity; Used to select a target training snapshot from the at least one training snapshot.

[0098] Optionally, the apparatus further includes an overhead acquisition module, a first construction module, a second construction module, a function determination module, and a solving module; The overhead acquisition module is used to acquire the prediction storage overhead of a single training snapshot of the model training task. The first construction module is used to construct a total storage overhead function with a target period as a variable based on the set time length and a single predicted storage overhead; The second construction module is used to construct an invalid training overhead function with the target period as a variable, The function determination module is used to determine a target overhead function based on the total saved overhead function and the invalid training overhead function; The solving module is used to perform a solving process on the target overhead function and determine the target period.

[0099] Optionally, the cluster includes a storage node, and the first response module 401 is Controlling the training node based on the target period to store the training snapshot in the memory of the training node based on the first process; Controlling the training node based on a second process to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node; A third process is used to control the training node to read the training snapshot from the solid-state disk and send the training snapshot to the storage node.

[0100] Optionally, the training snapshot corresponds to a time of acquisition, and the first response module 401: Obtain the failure time corresponding to the hardware failure of the training node, The training snapshot with the shortest corresponding acquisition time and fault time interval is used to select as the target training snapshot.

[0101] Optionally, the standby node is used to perform other tasks than the model training task, and the first response module 401: Obtain the priority of the standby node, which indicates the importance of other tasks. It is used to select a target standby node from multiple standby nodes based on priority.

[0102] Optionally, the initialization module 402: Performing communication bandwidth verification and / or silent data error verification on the target standby node; Used to initialize a target standby node in response to passing a communication bandwidth verification and / or passing a silent data error verification.

[0103] Optionally, the apparatus comprises: It further includes a second response module for determining an abnormal process at the training node and restarting the abnormal process in response to the software fault being present at the training node.

[0104] Optionally, the device further includes a fault monitoring module and a type determination module; The failure monitoring module is used to monitor the failure of the training nodes. The type determination module is used to determine a failure type of the training node based on a mapping relationship between the failure code and the failure type in response to obtaining the failure code of the training node, where the failure type includes a hardware failure or a software failure.

[0105] Optionally, the apparatus further includes a data acquisition module and a number determination module; The data acquisition module is used to acquire the predicted availability of the cluster and the predicted number of failed nodes per unit time; The number determination module is used to determine the number of standby nodes based on the predicted availability and the predicted number.

[0106] The above description of the cluster-based training method also applies to the cluster-based training apparatus of this embodiment, and will not be described in detail here. Based on the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0107] According to an embodiment of the present disclosure, a computer program is provided, the instructions of which, when executed, perform the cluster-based training method.

[0108] 5 is a schematic block diagram of an exemplary electronic device 500 for implementing embodiments of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the description herein and / or the practice of the present disclosure as claimed.

[0109] 5, the electronic device 500 includes a computing unit 501 that performs various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 502 or loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data necessary for the operation of the electronic device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to one another via a bus 1104. An input / output (I / O) interface 505 is also connected to the path.

[0110] The components of the electronic device 500 are connected to an I / O interface 505, which includes an input unit 506 such as a keyboard, a mouse, etc., an output unit 507 such as various types of displays and speakers, a storage unit 508 such as a magnetic disk, an optical disk, etc., and a communication unit 509 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 enables the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 501 may be various general-purpose and / or specialized processing components having processing and computational capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various machine driving learning model algorithm computing units, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes the cluster-based training method. For example, in some embodiments, the cluster-based training method may be implemented as a computer software program tangibly embodied in a machine-readable medium such as the storage unit 508. In some embodiments, some or all of the computer program may be loaded and / or installed into the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the cluster-based training method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured in any other suitable manner (eg, via firmware) to perform the cluster-based training method described above.

[0112] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being embodied in one or more computer programs that can be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and that can receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0113] Program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the functions / acts specified in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or can store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a suitable combination of any of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or a suitable combination of any of the above.

[0115] To provide for user interaction, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can also provide for user interaction; for example, the feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and can receive input from the user in any form (including acoustic, speech, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), a computing system including a middleware component (e.g., an application server), a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0117] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a host product in a cloud computing service system that solves the problems of traditional physical hosts and VPS services (also known as Virtual Private Servers, or VPSs) such as difficulty in management and poor business scalability. The server may be a server in a distributed system or a server combined with a blockchain.

[0118] Artificial intelligence is a field that studies how computers can simulate certain human thought processes and intelligent behaviors (learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, etc. AI software technologies mainly include several major areas such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0119] It should be understood that steps can be rearranged, added, or deleted using the various types of flows shown above. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, but this specification is not limited thereto as long as the technical solution disclosed in the present disclosure can achieve the desired results.

[0120] The above specific implementation methods do not limit the scope of protection of the present disclosure. It should be understood that those skilled in the art can make various modifications, combinations, subcombinations, and substitutions according to design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. A cluster-based training method executed by a training device, comprising: the cluster includes a training node for executing a model training task and a plurality of standby nodes; In response to a hardware failure in a training node, selecting a target standby node from a plurality of standby nodes and obtaining a target training snapshot of a model training task at the training node, the target training snapshot including training state data of the model training task; initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node; the training snapshot is taken from the training node based on a target period; The method comprises: obtaining a one-time prediction storage overhead of a training snapshot of the model training task; constructing a total storage overhead function with the target period as a variable based on the set time length and the single predicted storage overhead; constructing an invalid training overhead function that takes the target period as a variable; determining a target overhead function based on the total storage overhead function and the invalid training overhead function; performing a solving process on the target overhead function to determine the target period; A cluster-based training method characterized by:

2. The step of obtaining a target training snapshot of a model training task at the training node includes: obtaining at least one training snapshot of the model training task at the training node, the training snapshot being obtained from the training node based on the target periodicity; selecting the target training snapshot from at least one training snapshot; The cluster-based training method of claim 1 .

3. The cluster includes a storage node, and the step of obtaining at least one training snapshot of the model training task at the training node includes: controlling the training node to store the training snapshot in a memory of the training node based on a first process based on the target period; controlling the training node to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node based on a second process; controlling the training node to read the training snapshot from the solid state disk and send the training snapshot to the storage node based on a third process; The cluster-based training method of claim 2 .

4. The training snapshots correspond to acquisition times, and selecting the target training snapshot from the at least one training snapshot includes: obtaining a failure time corresponding to the hardware failure of the training node; selecting as the target training snapshot a training snapshot whose corresponding capture time has the shortest interval between the impairment time. The cluster-based training method of claim 2 .

5. The standby node is used to perform tasks other than a model training task, and selecting a target standby node from the plurality of standby nodes includes: obtaining a priority of the standby node, the priority indicating an importance of the other task; selecting the target standby node from the plurality of standby nodes based on the priority; The cluster-based training method of claim 1 .

6. The step of initializing the target standby node includes: performing a communication bandwidth verification and / or a silent data error verification on the target standby node; initializing the target standby node in response to passing a communication bandwidth verification and / or passing a silent data error verification; The cluster-based training method of claim 1 .

7. The method comprises: and, in response to a software fault being present in the training node, determining an abnormal process in the training node and restarting the abnormal process. The cluster-based training method of claim 1 .

8. The method comprises: monitoring the training node for faults; and determining a fault type of the training node based on a mapping relationship between the fault code and the fault type in response to the fault code of the training node being obtained; The failure type includes a hardware failure or a software failure. The cluster-based training method of claim 1 .

9. The method comprises: obtaining a predicted availability of the cluster and a predicted number of failed nodes per unit time; determining the number of standby nodes based on the predicted availability and the predicted number; The cluster-based training method of claim 1 .

10. 1. A cluster-based training apparatus, comprising: the cluster includes a training node for executing a model training task and a plurality of standby nodes; a first response module for selecting a target standby node from a plurality of standby nodes in response to a hardware failure in a training node and obtaining a target training snapshot of a model training task at the training node, the target training snapshot including training state data of the model training task; and an initialization module for initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node; the training snapshot is taken from the training node based on a target period; The device comprises: an overhead acquisition module for acquiring a one-time prediction storage overhead of a training snapshot of the model training task; a first construction module for constructing a total storage overhead function with the target period as a variable based on a set time length and the single predicted storage overhead; a second construction module for constructing an invalid training overhead function that takes the target period as a variable; a function determination module for determining a target overhead function based on the total saved overhead function and the invalid training overhead function; a solving module for performing a solving process on the target overhead function to determine the target period.

1. A cluster-based training apparatus comprising:

11. The first response module includes: obtaining at least one training snapshot of the model training task at the training node, the training snapshot being obtained from the training node based on the target periodicity; used to select the target training snapshot from at least one training snapshot.

11. The cluster-based training apparatus of claim 10.

12. The cluster includes a storage node, and the first response module is Controlling the training node based on the target period to store the training snapshot in a memory of the training node based on a first process; controlling the training node to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node according to a second process; a third process for controlling the training node to read the training snapshot from the solid-state disk and send the training snapshot to the storage node; 12. The cluster-based training apparatus of claim 11.

13. The training snapshot corresponds to a capture time, and the first response module: obtain a failure time corresponding to the hardware failure of the training node; a training snapshot having the shortest interval between its corresponding acquisition time and the failure time is used to select the target training snapshot; 12. The cluster-based training apparatus of claim 11.

14. The standby node is used to perform tasks other than a model training task, and the first response module: obtaining a priority of the standby node, the priority indicating an importance of the other task; used to select the target standby node from the plurality of standby nodes based on the priority.

11. The cluster-based training apparatus of claim 10.

15. The initialization module: performing a communication bandwidth verification and / or a silent data error verification on the target standby node; used to initialize the target standby node in response to passing a communication bandwidth verification and / or passing a silent data error verification; 11. The cluster-based training apparatus of claim 10.

16. The device comprises: a second response module for determining an abnormal process at the training node and restarting the abnormal process in response to a software fault being present at the training node; 11. The cluster-based training apparatus of claim 10.

17. The device comprises: a fault monitoring module for monitoring the training node for faults; a type determination module for determining a failure type of the training node based on a mapping relationship between a failure code and a failure type in response to the failure code of the training node being obtained, the failure type including a hardware failure or a software failure; 11. The cluster-based training apparatus of claim 10.

18. The device comprises: a data acquisition module for acquiring a predicted availability of the cluster and a predicted number of failed nodes per unit time; a number determination module for determining the number of standby nodes based on the predicted availability and the predicted number.

11. The cluster-based training apparatus of claim 10.

19. An electronic device, at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1 to 9. An electronic device characterized by:

20. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause a computer to carry out the method of any one of claims 1 to 9. A non-transitory computer-readable storage medium comprising:

21. A computer program comprising: The computer program, when executed by a processor, implements the method according to any one of claims 1 to 9. A computer program characterized by:

Images