Cluster-based training method and device, electronic equipment, and storage medium

Abstract

To provide: a cluster-based training method and device for shortening a fault recovery time and improving a model training efficiency; electronic equipment; and a storage medium.SOLUTION: In a method, in response to the presence of a hardware fault in a training node, a target standby node is selected from multiple standby nodes, and a target training snapshot of a model training task in the training node is obtained. The target training snapshot comprises training state data of the model training task. In the method, based on a container image and the training state data of a model training procedure in the training node, the target standby node is initialized so that the target standby node replaces the training node to continuously execute the model training task.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of data processing technology, particularly to fields such as artificial intelligence and deep learning, and in particular to a cluster-based training method, device, electronic device, and storage medium. [Background technology]

[0002] As artificial intelligence (AI) advances, the need for model training is also growing. For training scenes with high computing power demands, such as large-scale model training scenes, in order to improve the training efficiency of large-scale models, relevant researchers propose using clusters to train large-scale models. However, when using clusters to train large-scale models, the clusters provide high computing power and improve the model training efficiency to a certain extent. However, when a single node in the cluster fails, the cluster will stop training work, which will affect the training efficiency of the model. Therefore, when using clusters to train models, the training efficiency of the model needs to be further improved. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a cluster-based training method, apparatus, device, and storage medium. [Means for solving the problem]

[0004] According to a first aspect of the present disclosure, there is provided a cluster-based training method, the cluster including a training node and a plurality of standby nodes for performing a model training task; responsive to a hardware failure existing in a training node, selecting a target standby node from a plurality of standby nodes and obtaining a target training snapshot of a model training task in the training node, the target training snapshot including training state data of the model training task; and initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node.

[0005] According to a second aspect of the present disclosure, there is provided a cluster-based training apparatus, the cluster including: a training node for performing a model training task and a plurality of standby nodes; a first response module for selecting a target standby node from a plurality of standby nodes in response to a hardware failure existing in a training node, and obtaining a target training snapshot of a model training task in the training node, the target training snapshot including training state data of the model training task; and an initialization module for initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continuously performs the model training task in place of the training node.

[0006] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: At least one processor; a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the cluster-based training method described in the first aspect.

[0007] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing the computer to perform the cluster-based training method according to the first aspect.

[0008] According to a fifth aspect of the present disclosure, there is provided a computer program comprising computer instructions which, when executed by a processor, implement the steps of the cluster-based training method according to the first aspect.

[0009] The cluster-based training method, apparatus and electronic device provided by the present disclosure have the following beneficial effects: in response to a hardware failure in a training node, the present disclosure selects a target standby node from a plurality of standby nodes, obtains a target training snapshot of a model training task in the training node, the target training snapshot includes training state data of the model training task, and initializes the target standby node based on a container image and the training state data of the model training procedure in the training node, so that the target standby node replaces the training node and continuously executes the model training task. Thus, when a hardware failure exists in the training node, the present disclosure can quickly initialize the target standby node based on the container image and the target training snapshot, so that the target standby node quickly replaces the training node and continuously executes the model training task, thereby shortening the failure recovery time and improving the model training efficiency.

[0010] It should be understood that the contents described herein are not intended to identify key or important features of the embodiments of the present disclosure, nor should they be used to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief description of the drawings]

[0011] The drawings are used for a better understanding of the present solution and are not intended to limit the present disclosure. [Figure 1] 1 is a flowchart of a cluster-based training method provided by one embodiment of the present disclosure. [Diagram 2] 4 is a flowchart of a cluster-based training method provided by another embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of a cluster-based training method provided according to yet another embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a cluster-based training apparatus provided according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an electronic device for implementing a cluster-based training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, exemplary embodiments of the present disclosure will be described in combination with the drawings. Various details of the embodiments of the present disclosure are included here for ease of understanding, and they should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and simplicity, the following description omits the description of known functions and structures.

[0013] Hereinafter, a cluster-based training method, apparatus, and electronic device according to embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the entity that executes the cluster-based training method of this embodiment is a cluster-based training device, and the cluster-based training device can be realized in the form of software and / or hardware and can be arranged in an electronic device.

[0014] FIG. 1 is a flowchart of a cluster-based training method provided by one embodiment of the present disclosure.

[0015] A cluster includes a training node and multiple standby nodes for performing model training tasks. The number of training nodes included in a cluster may be one or multiple. When the number of training nodes in a cluster is multiple, a model can be trained by multiple training nodes, and each training node is responsible for a part of the training task of the model.

[0016] The training node and the standby node may refer to nodes that have undergone pressure measurement, such as nodes that have undergone communication unit pressure measurement, calculation unit pressure measurement, and model end-to-end pressure measurement. The period of node pressure measurement may be determined by the user according to actual needs.

[0017] As shown in FIG. 1, the cluster-based training method includes the following steps: Step 101: in response to a hardware failure existing in a training node, select a target standby node from a plurality of standby nodes, and obtain a target training snapshot of a model training task in the training node, where the target training snapshot includes training state data of the model training task.

[0018] Alternatively, the hardware failure may refer to a failure occurring in hardware in the training node. By way of example and not limitation, the hardware may refer to a graphics processing unit (GPU), a central processing unit (CPU), a network card, etc.

[0019] Optionally, the step of selecting a target standby node from the plurality of standby nodes may include the step of randomly selecting one standby node from the plurality of standby nodes as the target standby node.

[0020] In order to increase node utilization and fully utilize node resources, the standby nodes are generally not idle, i.e., the standby nodes generally perform other tasks besides the model training task.

[0021] Optionally, the step of selecting a target standby node from the multiple standby nodes includes the steps of obtaining a priority of the standby node, where the priority indicates an importance of other tasks, and selecting a target standby node from the multiple standby nodes based on the priority.

[0022] Selecting a target standby node based on priority can, on the one hand, avoid other more important tasks being terminated while they are running on the standby node, and, on the other hand, ensure high availability of the cluster and improve node utilization.

[0023] It is assumed that the higher the priority, the more important other tasks the standby node is to perform. When selecting a target standby node based on the current priority, the target standby node can be selected in order from lowest to highest priority. If there is no low-priority standby node or the low-priority standby node is unavailable, a high-priority standby node is selected as the target standby node.

[0024] By way of example and not limitation, assume that multiple standby nodes have corresponding priorities A and B, with A being higher than B, and when selecting a target standby node, the standby node with priority B can be preferentially selected as the target standby node, and if there is no standby node with priority B or the standby node with priority B is unavailable, the standby node with priority A is selected as the target standby node. Wherein, the number of standby nodes with priority A may be greater than the number of standby nodes with priority B.

[0025] Regarding GPU hardware, if a GPU fails, there is a high probability that it will fail again during subsequent use. In order to reduce the frequency of failures across the entire cluster and improve training efficiency, the present disclosure proposes that previously failed GPUs can be concentrated on one or more nodes, and such nodes can be used as low-priority standby nodes and scheduled preferentially.

[0026] Optionally, the present disclosure further includes obtaining a predicted availability of the cluster and a predicted number of failed nodes per unit time, and determining a number of standby nodes based on the predicted availability and the predicted number.

[0027] In an alternative implementation, the number of standby nodes can be obtained by Poisson fitting based on the predicted availability and the predicted number.

[0028] By way of example and not limitation, the number of standby nodes may be determined based on the following formula:

[0029]

number

[0030] Wherein, α is the predicted availability, μ is the predicted number, and x is the number of standby nodes to be solved. The above equation is solved to obtain the number x of standby nodes.

[0031] The predicted number of failed nodes per unit time can be obtained by statistically analyzing the historical data, and the predicted number of failed nodes per unit time can also be obtained by converting the predicted failure frequency of the cluster.

[0032] By determining the number of standby nodes based on the predicted availability and the predicted number, the standby nodes can cover for failed nodes, thereby achieving high cluster availability.

[0033] Alternatively, the target training snapshot can refer to a training snapshot at a time when a hardware failure appears on a training node in a model training task, or can refer to a training snapshot at a time before a hardware failure appears on a training node in a model training task.

[0034] The training state data of the model training task included in the target training snapshot may refer to data such as training data state, model parameters, and optimizer state, etc., where the training data state is used to indicate whether the training data has already been used, and based on the training data state, the target standby node can directly train the model based on the training data that has not been used, and can avoid repeatedly training the model based on the same training data.

[0035] Step 102: Initialize a target standby node according to the container image and training state data of the model training procedure in the training node, so that the target standby node continuously performs the model training task on behalf of the training node.

[0036] The container image of the model training procedure can be created in advance, and after determining the target standby node, the container image can be deployed to the target standby node, which can improve the deployment speed of the model training procedure.

[0037] After the container image is deployed, data placement can be performed on the target standby node based on the training state data.

[0038] The process of initializing the target standby node can be understood as preempting resources of the target standby node to perform model training tasks.

[0039] By initializing the target standby node based on the container image and training state data, hardware failure recovery can be performed automatically, thereby shortening the hardware failure recovery time and improving the hardware failure recovery efficiency.

[0040] In an embodiment of the present disclosure, in response to a hardware failure in the training node, a target standby node is selected from a plurality of standby nodes, a target training snapshot of a model training task in the training node is obtained, the target training snapshot includes training state data of the model training task, and the target standby node is initialized based on a container image and the training state data of the model training procedure in the training node, so that the target standby node replaces the training node and continuously executes the model training task. Thus, the present disclosure can quickly initialize the target standby node based on the container image and the target training snapshot when a hardware failure exists in the training node, so that the target standby node quickly replaces the training node and continuously executes the model training task, thereby shortening the failure recovery time and improving the model training efficiency.

[0041] FIG. 2 is a flowchart of a cluster-based training method provided by another embodiment of the present disclosure.

[0042] As shown in FIG. 2 , the cluster-based training method includes the following steps: Step 201: in response to a hardware failure existing in a training node, select a target standby node from a plurality of standby nodes, and obtain at least one training snapshot of a model training task in the training node, the training snapshot being obtained from the training node based on a target periodicity, and select a target training snapshot from the at least one training snapshot.

[0043] After a training node starts to execute a model training task, a training snapshot of the model training task is taken at regular intervals based on a target period, thereby avoiding problems of data loss and invalid training caused by training node hardware failure.

[0044] The cluster further includes a storage node, and optionally, the step of obtaining at least one training snapshot of the model training task in the training node includes the steps of controlling the training node to store the training snapshot in a memory of the training node based on a first process based on the target periodicity, controlling the training node to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node based on a second process, and controlling the training node to read the training snapshot from the solid-state disk based on a third process and send the training snapshot to the storage node.

[0045] If the training node sequentially stores the training snapshots in memory, solid-state disk, and storage node through only one process, the process can only return to executing other instructions after storing the training snapshot in the storage node, resulting in low overall execution efficiency of the program.

[0046] By having multiple processes asynchronously acquire or store training snapshots, blocking during program execution can be avoided, further improving the concurrent performance of the program.

[0047] Alternatively, the target period may be determined empirically by the user.

[0048] Alternatively, the target period may be determined in the following manner: A single predicted storage overhead of a training snapshot of a model training task is obtained, a total storage overhead function with a target period as a variable is constructed based on a set time length and the single predicted storage overhead, an invalid training overhead function with the target period as a variable is constructed, a target overhead function is determined based on the total storage overhead function and the invalid training overhead function, and a solving process is performed on the target overhead function to determine the target period.

[0049] By way of example and not limitation, a single predicted storage overhead may refer to the corresponding storage overhead when controlling a training node to store a training snapshot in a storage node by three processes, or may refer to the overhead of storing a training snapshot by other storage methods.

[0050] By way of example and not limitation, the set time length may refer to an average length of time from when a training node starts to execute a model training task to when the model training task is interrupted, and may be a length of time determined by a user based on experience.

[0051] By way of example and not limitation, the target period may be determined by the following formula:

[0052]

number

[0053] Where, H(t) is the target overhead function,

[0054]

number

[0055] is the total storage overhead function, T is the set time length, c is the expected storage overhead for one time, t is a variable, i.e., the target period, t is greater than 0,

[0056]

number

[0057] is an invalid training overhead function, where invalid training may refer to training between the time when the last training snapshot was taken before the hardware failure appeared and the time when the hardware failure appeared. Note that the coefficient of the variable t in H2(t) may be determined according to the actual situation.

[0058] The target period can be obtained by solving the minimum value of H(t), where the total storage overhead and the unnecessary training overhead mentioned above can refer to the time overhead.

[0059] The target period obtained by solving the above formula is a good period, which can balance the total storage overhead of training snapshots and the invalid training overhead, and minimize the overhead during the model training process.

[0060] By way of example and not limitation, the target period may be further determined by the following formula:

[0061]

number

[0062] Among them, f failure is the predicted fault frequency for the period during which the training node performs the model training task.

[0063] Optionally, selecting a target training snapshot from the at least one training snapshot includes selecting a target training snapshot from the at least one training snapshot based on a corresponding training effect, where the training effect can be determined based on a loss value corresponding to the training snapshot.

[0064] By selecting the training snapshot with the best corresponding training effect as the target training snapshot, the target standby node can continuously perform model training tasks based on the best model training result before the hardware failure occurs, thereby improving model training efficiency and reducing model training costs.

[0065] The training snapshots correspond to acquisition times, and optionally, from the at least one training snapshot, selecting a target training snapshot includes obtaining a failure time corresponding to a hardware failure of the training node, and selecting a training snapshot having a corresponding acquisition time and a shortest failure time interval as the target training snapshot.

[0066] By selecting the last training snapshot taken before a hardware failure occurs as the target training snapshot, the target standby node can base its model training results on the latest model training results and continuously execute model training tasks, so that model training is continuous and invalid training overhead is reduced.

[0067] Step 202, performing communication bandwidth verification and / or silent data error verification on the target standby node.

[0068] Optionally, performing communication bandwidth verification on the target standby node refers to verifying whether the communication bandwidth corresponding to the target standby node is equal to or greater than a bandwidth threshold, and if the communication bandwidth is equal to or greater than the bandwidth threshold, the communication bandwidth verification is passed, and if the communication bandwidth is less than the bandwidth threshold, the communication bandwidth verification is not passed, where the communication bandwidth can refer to an aggregate communication bandwidth.

[0069] Alternatively, performing silent data error verification on a target standby node refers to the target standby node verifying whether a silent data error exists, and if a silent data error does not exist, the silent data error verification is passed, and if a silent data error exists, the silent data error verification is not passed.

[0070] By way of example and not limitation, a training node can perform a particular operation multiple times and compare the multiple operation results; if the multiple operation results are the same, then a silent data error does not exist in the training node; and if there are at least two different operation results in the multiple operation results, then a silent data error exists in the training node.

[0071] By performing communication bandwidth verification and / or silent data error verification on the target standby node, risk monitoring can be performed on the performance of the target standby node, avoiding the introduction of poorly performing or failed nodes and reducing the probability of subsequent failure of the target standby node.

[0072] Step 203: In response to passing the communication bandwidth verification and / or passing the silent data error verification, initialize a target standby node based on the container image and training state data of the model training procedure in the training node.

[0073] The relevant content of step 203 may refer to the relevant description of step 102, and will not be described in detail here.

[0074] In an embodiment of the present disclosure, in response to a hardware failure in the training node, select a target standby node from a plurality of standby nodes, obtain at least one training snapshot of a model training task in the training node, the training snapshot being obtained from the training node based on a target periodicity, select a target training snapshot from the at least one training snapshot, perform communication bandwidth verification and / or silent data error verification on the target standby node, and in response to the communication bandwidth verification being passed and / or the silent data error verification being passed, initialize the target standby node based on the container image and training state data of the model training procedure in the training node. Thereby, the present disclosure can avoid data loss and invalid training problems caused by the training node hardware failure by obtaining at least one training snapshot, and can perform risk monitoring on the performance of the target standby node by performing communication bandwidth verification and / or silent data error verification on the target standby node, and avoid introducing a node with poor performance or a node with a failure.

[0075] FIG. 3 is a flowchart of a cluster-based training method provided according to another embodiment of the present disclosure.

[0076] As shown in FIG. 3, the cluster-based training method includes the following steps: Step 301: Fault monitoring is performed on the training node.

[0077] Optionally, the model training procedure at the training node can be monitored, i.e., whether the model training procedure terminates with a failure.

[0078] Step 302, in response to obtaining the failure code of the training node, determine a failure type of the training node based on a mapping relationship between the failure code and the failure type, where the failure type includes a hardware failure or a software failure.

[0079] If it is monitored that the model training procedure ends with a failure, obtain the corresponding failure code, and look up the corresponding failure type based on the mapping relationship failure code to determine whether the training node has a hardware failure or a software failure.

[0080] The present disclosure realizes self-diagnosis of integrated hardware and software faults, and regardless of whether it is a hardware fault or a software fault, it is embodied in the model training procedure as a program fault termination, so that rapid detection of the fault can be realized based on the fault code.

[0081] Step 303, in response to a hardware failure existing in the training node, select a target standby node from the multiple standby nodes, and obtain a target training snapshot of the model training task in the training node.

[0082] Compared with a fault monitoring method that scans a hardware error log at a fixed interval, the time-consuming nature of hardware fault monitoring based on fault codes is high. There is also a mapping relationship between the fault code and the hardware fault, so that the hardware in which the fault exists can be located based on the fault code.

[0083] Step 304, in response to the software fault existing in the training node, determines an abnormal process in the training node and restarts the abnormal process.

[0084] There is also a mapping relationship between the fault code and the software fault, so that the abnormal process can be determined based on the fault code, and further, the abnormal process can be resumed.

[0085] The present disclosure can select corresponding failure recovery schemes in response to different failure types, shorten the failure recovery time, and ensure the training stability of the cluster. Alternatively, the fault types can be mapped as fault levels, and each fault level can correspond to a fault recovery scheme. By way of example and not limitation, the fault level corresponding to a software fault can be M, and the fault level corresponding to a hardware fault can be N, where the fault degree of the software fault is lower than that of the hardware fault, and thus the fault level M can be lower than the fault level N. Wherein, the fault recovery scheme corresponding to the fault level M is to restart the abnormal process, and the fault recovery scheme corresponding to the fault level N is to select a target standby node and initialize the target standby node.

[0086] Specifically, a failure level can be determined based on the failure code, and a corresponding failure recovery method can be selected based on the failure level.

[0087] Alternatively, if the target standby node fails to initialize or fails to restart the abnormal process, the cluster can be scheduled, i.e., another cluster replaces the current cluster to continue training the model.

[0088] Based on the cluster-based training method of the present disclosure, the effective training rate for the model of the cluster can be effectively improved, and the effective training rate can be used to indicate the time occupancy that really contributes to the model parameter update within a desired model training time period. Optionally, the effective training rate of the cluster can be predicted by the following formula:

[0089]

number

[0090] Where, ρ is the effective training rate, α is the predicted availability of the cluster, and F failure is the predicted failure frequency of the cluster, and C recovery is the predicted single failure recovery time occupancy, and C recovery can be understood as the ratio of one failure recovery time to the prediction training time of the model training task, F checkpoint is the expected frequency of saving training snapshots, and c is the expected overhead of one save.

[0091] After the training node completes the model training task, the actual values ​​of the parameters in the above formula can be obtained to determine the actual effective training rate of the cluster.

[0092] Based on the effective training rate and the training time, the effective training time can be determined, and based on the effective training time and the amount of throughput data within a unit time, the total amount of throughput data when the cluster executes the model training task can be determined.

[0093] In addition to including a training node and a standby node, the cluster further includes a training service node, such as a scheduling node for selecting a target standby node and a monitoring node for fault monitoring of the training node, and fault monitoring of the training service node can be performed based on the communication status between the training service nodes.

[0094] In an embodiment of the present disclosure, based on the steps of performing fault monitoring on the training node; determining a fault type of the training node based on a mapping relationship between the fault code and the fault type in response to obtaining a fault code of the training node; selecting a target standby node from multiple standby nodes in response to a hardware fault existing in the training node; obtaining a target training snapshot of the model training task in the training node; and determining an abnormal process in the training node and resuming the abnormal process in response to a software fault existing in the training node, the present disclosure can select corresponding fault recovery schemes in response to different fault types and shorten the fault recovery time.

[0095] 4 is a schematic diagram of a cluster-based training apparatus provided according to an embodiment of the present disclosure. The cluster includes a training node and multiple standby nodes for performing model training tasks.

[0096] As shown in FIG. 4, the cluster-based training apparatus 400 includes a first response module 401 and an initialization module 402; The first response module 401 is used to select a target standby node from the multiple standby nodes in response to a hardware failure existing in the training node, and obtain a target training snapshot of the model training task in the training node, where the target training snapshot includes training state data of the model training task; The initialization module 402 is used to initialize the target standby node based on the container image and training state data of the model training procedure in the training node, so that the target standby node continuously performs model training tasks in place of the training node.

[0097] Optionally, the first response module 401 is Obtain at least one training snapshot of the model training task in the training node, the training snapshot being obtained from the training node based on the target periodicity; Used to select a target training snapshot from the at least one training snapshot.

[0098] Optionally, the apparatus further includes an overhead acquisition module, a first construction module, a second construction module, a function determination module, and a solving module; The overhead acquisition module is used to obtain the one-time prediction storage overhead of the training snapshot of the model training task, The first construction module is used to construct a total overhead saving function with a target period as a variable based on the set time length and a single predicted overhead saving; The second construction module is used to construct an invalid training overhead function with the target period as a variable, The function determination module is used to determine a target overhead function based on the total saved overhead function and the invalid training overhead function; The solving module is used to perform a solving process on the target overhead function and determine the target period.

[0099] Optionally, the cluster includes a storage node, and the first response module 401 is: Control the training node according to the target period to store the training snapshot in the memory of the training node according to the first process; Controlling the training node based on the second process to read the training snapshot from the memory and store the training snapshot on a solid-state disk of the training node; A third process is used to control the training node to read the training snapshot from the solid-state disk and send the training snapshot to the storage node.

[0100] Optionally, the training snapshot corresponds to a time of acquisition, and the first response module 401: Obtain the failure time corresponding to the hardware failure of the training node, The training snapshot with the shortest corresponding acquisition time and failure time interval is used to select as the target training snapshot.

[0101] Optionally, the standby node is used to perform other tasks than the model training task, and the first response module 401: Get the priority of the standby node, which indicates the importance of other tasks; It is used to select a target standby node from multiple standby nodes based on priority.

[0102] Optionally, the initialization module 402 may: Performing communication bandwidth verification and / or silent data error verification on the target standby node; Used to initialize a target standby node in response to passing a communication bandwidth verification and / or passing a silent data error verification.

[0103] Optionally, the apparatus comprises: It further includes a second response module for determining an abnormal process at the training node and restarting the abnormal process in response to the software fault being present at the training node.

[0104] Optionally, the apparatus further includes a fault monitoring module and a type determination module; The failure monitoring module is used to monitor failures on the training nodes. The type determination module is used to determine a failure type of the training node based on a mapping relationship between the failure code and the failure type in response to obtaining the failure code of the training node, where the failure type includes a hardware failure or a software failure.

[0105] Optionally, the apparatus further includes a data acquisition module and a number determination module; The data acquisition module is used to obtain a predicted availability of the cluster and a predicted number of failed nodes per unit time; The number determination module is used to determine the number of standby nodes based on the predicted availability and the predicted number.

[0106] The above description of the cluster-based training method also applies to the cluster-based training apparatus of the present embodiment, and will not be described in detail here. Based on the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0107] According to an embodiment of the present disclosure, a computer program is provided, the instructions of which, when executed, perform the cluster-based training method.

[0108] 5 is a schematic block diagram of an exemplary electronic device 500 for implementing embodiments of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital processors, mobile phones, smart phones, wearable devices, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure as described herein and / or claimed herein.

[0109] 5, the electronic device 500 includes a computing unit 501 that performs various appropriate operations and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or loaded from a storage unit 508 into a RAM (Random Access Memory) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 1104. An I / O (Input / Output) interface 505 is also connected to the path.

[0110] The components of the electronic device 500 are connected to an I / O interface 505, including an input unit 506 such as a keyboard, a mouse, etc., an output unit 507 such as various types of displays, speakers, etc., a storage unit 508 such as a magnetic disk, an optical disk, etc., and a communication unit 509 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 enables the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 501 may be various general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various machine driving learning model algorithm computing units, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes the cluster-based training method. For example, in some embodiments, the cluster-based training method can be realized as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the cluster-based training method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured in any other suitable manner (eg, via firmware) to perform the cluster-based training method described above.

[0112] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and that may receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0113] The program codes for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in combination with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage, magnetic storage, or any suitable combination of the above.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can provide interaction with a user, for example, the feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and can receive input from the user in any form (including acoustic, speech, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0117] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other through a communication network. The relationship between the client and the server is generated by a computer program that runs on a corresponding computer and has a client-server relationship with each other. The server may be a cloud server, also called a cloud computing server or cloud host, which is a host product in a cloud computing service system and solves the defects of the traditional physical host and VPS service (also called Virtual Private Server, or VPS for short) such as difficult management and weak business scalability. The server may be a server of a distributed system or a server combined with a blockchain.

[0118] Artificial intelligence is a field that studies simulating certain human thought processes and intelligent behaviors (learning, reasoning, thinking, planning, etc.) in computers, and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and other technologies. Artificial intelligence software technologies mainly include several major directions, such as computer vision technology, voice recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0119] It should be understood that steps can be rearranged, added, or deleted using various types of flows shown above. For example, each step described in the present disclosure may be performed in parallel, sequentially, or in a different order, but is not limited herein as long as the technical solution disclosed in the present disclosure can achieve the desired results.

[0120] The above specific implementation methods do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. A cluster-based training method comprising the steps of: The cluster includes a training node for executing a model training task and a plurality of standby nodes; In response to a hardware failure existing in a training node, selecting a target standby node from a plurality of standby nodes and obtaining a target training snapshot of a model training task in the training node, the target training snapshot including training state data of the model training task; Initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node; 2. A cluster-based training method comprising:

2. The step of obtaining a target training snapshot of a model training task at the training node includes: obtaining at least one training snapshot of the model training task at the training node, the training snapshot being obtained from the training node based on a target periodicity; selecting the target training snapshot from at least one training snapshot. The cluster-based training method of claim 1.

3. The method comprises: obtaining a one-time prediction storage overhead of a training snapshot of the model training task; constructing a total overhead storage function with the target period as a variable based on the set time length and the one-time predicted overhead storage; constructing an invalid training overhead function that takes the target period as a variable; determining a target overhead function based on the total storage overhead function and the invalid training overhead function; performing a solving process on the target overhead function to determine the target period. The cluster-based training method of claim 2.

4. The cluster includes a storage node, and the step of obtaining at least one training snapshot of the model training task on the training node includes: controlling the training node to store the training snapshot in a memory of the training node based on a first process according to the target period; controlling the training node to read the training snapshot from the memory based on a second process and store the training snapshot on a solid-state disk of the training node; controlling the training node to read the training snapshot from the solid state disk according to a third process and to send the training snapshot to the storage node; The cluster-based training method of claim 2.

5. The training snapshots correspond to acquisition times, and selecting the target training snapshot from the at least one training snapshot comprises: obtaining a failure time corresponding to the hardware failure of the training node; selecting as the target training snapshot a training snapshot whose corresponding capture time and the impairment time are closest in time. The cluster-based training method of claim 2.

6. The standby node is used to execute tasks other than a model training task, and the step of selecting a target standby node from the plurality of standby nodes includes: obtaining a priority of the standby node, the priority indicating an importance of the other task; selecting the target standby node from the plurality of standby nodes based on the priority; The cluster-based training method of claim 1.

7. The step of initializing the target standby node includes: performing a communication bandwidth verification and / or a silent data error verification on the target standby node; and in response to passing a communication bandwidth verification and / or passing a silent data error verification, initializing the target standby node. The cluster-based training method of claim 1.

8. The method comprises: and determining, in response to a software fault being present in the training node, an abnormal process at the training node and restarting the abnormal process. The cluster-based training method of claim 1.

9. The method comprises: monitoring the training node for a fault; In response to obtaining the fault code of the training node, determining a fault type of the training node based on a mapping relationship between a fault code and a fault type; The failure type includes a hardware failure or a software failure. The cluster-based training method of claim 1.

10. The method comprises: obtaining a predicted availability of the cluster and a predicted number of failed nodes per unit time; determining the number of standby nodes based on the predicted availability and the predicted number. The cluster-based training method of claim 1.

11. 1. A cluster-based training apparatus comprising: The cluster includes a training node for executing a model training task and a plurality of standby nodes; a first response module for selecting a target standby node from a plurality of standby nodes in response to a hardware failure existing in a training node, and obtaining a target training snapshot of a model training task in the training node, the target training snapshot including training state data of the model training task; and an initialization module for initializing the target standby node based on a container image of a model training procedure in the training node and the training state data, so that the target standby node continues to perform the model training task in place of the training node; 13. A cluster-based training apparatus comprising:

12. The first response module includes: obtaining at least one training snapshot of the model training task at the training node, the training snapshot being obtained from the training node based on a target periodicity; used to select the target training snapshot from at least one training snapshot. The cluster-based training apparatus of claim 11.

13. The apparatus comprises: an overhead acquisition module for acquiring a one-time prediction storage overhead of a training snapshot of the model training task; a first construction module for constructing a total overhead storage function with the target period as a variable based on a set time length and the one-time predicted overhead storage; a second construction module for constructing an invalid training overhead function that has the target period as a variable; a function determination module for determining a target overhead function based on the total saved overhead function and the invalid training overhead function; and a solving module for performing a solving process on the target overhead function to determine the target period.

13. The cluster-based training apparatus of claim 12.

14. The cluster includes a storage node, and the first response module is Controlling the training node based on the target period to store the training snapshot in a memory of the training node based on a first process; Controlling the training node to read the training snapshot from the memory based on a second process and store the training snapshot on a solid-state disk of the training node; A third process is used to control the training node to read the training snapshot from the solid-state disk and send the training snapshot to the storage node according to the third process.

13. The cluster-based training apparatus of claim 12.

15. The training snapshot corresponds to a capture time, and the first response module: obtain a failure time corresponding to the hardware failure of the training node; a training snapshot having a shortest interval between its corresponding acquisition time and the impairment time is used to select the target training snapshot; 13. The cluster-based training apparatus of claim 12.

16. The standby node is used to perform tasks other than a model training task, and the first response module: obtaining a priority of the standby node, the priority indicating an importance of the other task; used to select the target standby node from the plurality of standby nodes based on the priority. The cluster-based training apparatus of claim 11.

17. The initialization module includes: performing a communication bandwidth verification and / or a silent data error verification on the target standby node; used to initialize the target standby node in response to passing a communication bandwidth verification and / or passing a silent data error verification; The cluster-based training apparatus of claim 11.

18. The apparatus comprises: a second response module for determining an abnormal process at the training node and restarting the abnormal process in response to a software fault being present at the training node. The cluster-based training apparatus of claim 11.

19. The apparatus comprises: a fault monitoring module for monitoring the training node for faults; and a type determination module for determining a failure type of the training node based on a mapping relationship between a failure code and a failure type in response to the failure code of the training node being obtained, the failure type including a hardware failure or a software failure. The cluster-based training apparatus of claim 11.

20. The apparatus comprises: a data acquisition module for acquiring a predicted availability of the cluster and a predicted number of failed nodes per unit time; and a number determination module for determining the number of standby nodes based on the predicted availability and the predicted number. The cluster-based training apparatus of claim 11.

21. An electronic device, at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor performs a method according to any one of claims 1 to 10.

1. An electronic device comprising:

22. A non-transitory computer-readable storage medium having computer instructions stored thereon, The computer instructions cause a computer to carry out a method according to any one of claims 1 to 10. A non-transitory computer-readable storage medium comprising:

23. A computer program comprising: The computer program, when executed by a processor, implements the method according to any one of claims 1 to 10. A computer program comprising:

Images