Systems and methods for error recovery

Abstract

Embodiments of the present disclosure include an error recovery method comprising detecting a computing error, restarting a first artificial intelligence processor of a plurality of artificial intelligence processors processing a data set, and loading a model in the artificial intelligence processor, wherein the model corresponds to a same model processed by the plurality of artificial intelligence processors during a previous processing iteration of data from the data set by the plurality of artificial intelligence processors.

Description

Background Technology

[0001] This disclosure relates to a computation. More specifically, this disclosure relates to techniques for error recovery in artificial intelligence processing.

[0002] Artificial intelligence (AI) processing typically involves loading some or all of an AI model (e.g., a neural network model) onto one or more processors. A dataset is applied as input to the AI ​​model, and an output is generated. For inference, the output may correspond to a classification or recognition of specific features of the input dataset. For training, the output is compared to known outputs on the input data, and errors are backpropagated through the model, and the model's parameters are tuned. For large models and datasets, processing can be partitioned across multiple processors to obtain results more quickly.

[0003] One problem with such systems is when a node in a multiprocessor system encounters an error. In many cases, restarting the computation may require recalculating large amounts of data. Attached Figure Description

[0004] Various embodiments of this disclosure are illustrated by way of example rather than limitation in the accompanying drawings.

[0005] Figure 1 The illustration depicts error recovery in a multiprocessor computing environment according to an embodiment.

[0006] Figure 2 The illustration depicts a method for recovering from processor errors in a multiprocessor computing environment according to an embodiment.

[0007] Figure 3 The illustration shows the reloading of a model during training in a multiprocessor computing environment according to an embodiment.

[0008] Figure 4 The illustration depicts a multiprocessor computing architecture according to an embodiment.

[0009] Figure 5 The illustrations depict synchronization and global checkpoints during each iteration according to various embodiments.

[0010] Figure 6 The illustration shows a return to the global checkpoint according to various embodiments.

[0011] Figure 7 The illustration shows the reloading of a model from a previous iteration according to various embodiments.

[0012] Figure 8 The illustration shows the operation of a controller and processing device according to an example embodiment.

[0013] Figure 9An example architecture for error recovery according to another embodiment is illustrated.

[0014] Figure 10 The illustration shows recovery when an error occurs during the result aggregation phase, according to an example embodiment.

[0015] Figure 11 The illustration shows an example result generated according to an embodiment.

[0016] Figure 12 An example of result aggregation according to an embodiment is illustrated.

[0017] Figure 13 The illustration depicts error recovery in a multiprocessor computing environment according to an embodiment.

[0018] Figure 14 The illustration shows the computation of distributing faulty processors across multiple processors according to an embodiment. Detailed Implementation

[0019] In the following description, numerous examples and specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. Such examples and details should not be construed as unduly limiting the elements of the claims or the subject matter claimed in whole. Based on the language of the different claims, it will be apparent to those skilled in the art that the claimed subject matter may include some and all of the features in these examples, individually or in combination, and may also include modifications and equivalents of the features and techniques described herein.

[0020] Artificial intelligence (AI) processing systems typically require handling large amounts of data. Distributed processing improves processing speed. For example, using synchronous or hybrid data parallelism for distributed training in deep learning is an effective method for converging models across many AI processors with high throughput and accuracy.

[0021] One example technique used in AI networks (e.g., for training) is called data parallelism. Data parallelism breaks down the training dataset into pieces, and AI processors are loaded with models designed to process the data in parallel. For example, in one embodiment of data parallelism, the training data can be divided into pieces (also called shards), and each shard can be distributed across multiple AI processors (also called worker or target processors) for processing. The shards are further divided into minibatches, which are iteratively processed by multiple AI processors in successive iterations. During each iteration, the AI ​​processors receive a minibatch (e.g., a minibatch of training data) and determine the changes in the model parameters (also called “gradients” or “increments”). At the end of each iteration, the AI ​​processors can combine and synchronize their model parameters and update the model with the new parameter values.

[0022] Features and advantages of this disclosure include a process for recovering from a fault. Figure 1 The illustration depicts multiple AI processors configured to process data in parallel using model M. In this example, the AI ​​processors are configured in multiple N worker groups 101-103, where N is an integer. For example, a worker group may include one or more AI processors. AI processors may include graphics processing units (GPUs), AI accelerators, or other digital processors optimized for AI operations (e.g., matrix multiplication and von Neumann architecture processors such as x86 processors). For example, the example AI processor may include a GPU (e.g., an NVIDIA GPU with 800 cores and 64 multi-accumulators). ) or Tensor Processing Unit (TPU) (e.g., 4 cores with 16k parallel operations).

[0023] This example illustrates an iteration where each worker group receives input data (e.g., mini-batches) and processes the data using models 105-107. In this example, the iteration begins at 110, where each worker group starts with substantially the same model. For example, as part of a previous iteration, models 105-107 in each worker group may have synchronized their parameters (e.g., by performing an all-reduce). For example, in one embodiment, one or more copies of the model may be saved as model 104 between each iteration cycle. At 111, each worker group processes different input data, such as mini-batches from the same training dataset. However, in this example, one of the worker groups 102 experiences an error (e.g., a hardware or software failure). Advantageously, at 112, the saved model 104 used at the start of the iteration can be loaded into worker group 102 and worker group 102 can quickly restart processing to produce results. At 112, for example, the results of all worker groups 101-103 can be combined to produce an updated model, and the resulting model can be saved again for the next iteration. For example, in various embodiments described in more detail below, a worker group experiencing an error can receive a new model 104 from a controller (shown below), another worker group, or from the worker group's local memory.

[0024] Figure 2The diagram illustrates an error recovery method according to an embodiment. At 201, a computational error is detected. For example, a computational error could be a software or hardware error in an AI processor among multiple AI processors processing a dataset. At 202, the AI ​​processor can eliminate the error. For example, in some embodiments, some or all components of the AI ​​processor (e.g., hardware or software components) can be restarted. As illustrated below in various example embodiments, for example, the AI ​​processor can be restarted by a controller coupled to the AI ​​processor. At 203, a model is loaded into the AI ​​processor, wherein the model corresponds to the same model processed by the multiple AI processors during previous processing iterations of data from the dataset by the multiple AI processors.

[0025] Features and advantages of this disclosure include the ability of the worker group to access the model used at the beginning of each processing iteration for rapid restart. Traditionally, AI systems undergo multiple iterations before reaching a global checkpoint, during which the system's state information is saved. Errors require some systems to return to the global checkpoint across multiple iterations, which is time-consuming. Advantageously, the AI ​​processor experiencing the error can return to the beginning of the current iteration, while other processors can wait while they finish generating the results of the current iteration. Once the faulty AI processor is reset and the error is cleared, it can reload the current iteration model and recover. As described herein, the model can be stored in multiple different locations accessible to the AI ​​processor experiencing the error. An example AI model is a combination of AI parameters, such as weights or biases, for a specific AI topology. Processing the model may include generating gradients during each iteration. Gradients may include deviations (increments) from the current parameter values ​​(e.g., increments in the values ​​of specific weights in a neural network). Gradients are generated by each AI processor as a processing result and can be combined (e.g., aggregated via averages, mean values, etc.) and then applied to the model's values ​​at the beginning of the iteration. For example, the average increment of all weights in a neural network model can be calculated, and this average increment can be applied to generate a subsequent model for the next iteration.

[0026] Figure 3The illustration depicts error recovery in a computer processing system performing training according to an example embodiment. In this example, training data 301 is used to train parameters of an AI model, such as the weights of a neural network. The training dataset 301 can be divided into segments (referred to herein as slices or “parts”) 302a-N. These slices are then forwarded to different worker groups for processing. Each slice can be further divided into smaller segments 303a-N (referred to herein as “mini-batches” or sometimes simply “batches”). Mini-batches 303a-N of each slice are sequentially coupled to worker groups 304a-N, one at a time. A worker group can receive the mini-batches of training data and perform AI training on model 320, producing training results. The training results from each worker group can then be combined at 350 to produce an updated model 321, which, for example, can be loaded into each worker group for processing the next mini-batch. As used in this paper, an "epoch" occurs when each worker group processes all its shards and a complete training dataset 301 has been processed once. For example, the training dataset 301 may be processed in multiple epochs to obtain the final set of trained model parameters. It should be understood that other ways of splitting and processing the dataset 301 across multiple worker groups can utilize the error recovery techniques described in this paper.

[0027] In this example, the iteration includes receiving mini-batches by worker group 304a-N, processing the mini-batches to produce results, and combining the results to produce an updated model. The iteration also includes loading the updated model into the worker group (e.g., at the beginning or end of the iteration). Figure 3 The diagram illustrates the i-th iteration, where the N-th minibatch (minibatch_i) is loaded into N worker groups 304a-N for processing (e.g., where N is an integer). The i-th model M_i320 generated in the previous (i-1)-th iteration is loaded into each of the worker groups 304a-N. The results from processing each minibatch_i are combined at 350 to produce the subsequent (or, next) model model_i+1 321, which is then loaded into each of the worker groups 304a-N to process the (i+1)-th minibatch in the next (i+1)-th iteration. As described above, if one of the worker groups encounters an error (e.g., a hard or soft failure), each of the worker groups can access the model for each iteration. Therefore, for example, if worker group 304b fails during processing minibatch_i, it can reload the i-th model (M_i) and complete its processing. Other systems can detect that worker group 304b is experiencing an error and wait. When worker group 304b clears the error and produces a result, the results from the worker group are combined and the calculation continues.

[0028] Figure 4 The diagram illustrates a computational architecture for processing AI data. In this example, multiple worker groups are coupled to a controller, and the controller may be coupled to a network. For example, worker groups 410a-N are coupled to controller 401 and worker groups 411a-N are coupled to controller 402. Controllers 401 and 402 may be coupled together via network 450 (e.g., via an Ethernet connection and one or more network switches (not shown)). Worker groups may also be coupled together via communication links (e.g., PCIe) such as links 451 and 452. For example, multiple such controller / worker groups may be used to process AI data in parallel, as described above, for use as training data. In various embodiments, combinations of the above processing results (e.g., delta_parameters) may be performed by the controller between worker groups (e.g., via a full reduction) or using combinations thereof.

[0029] As described above, for example, each worker group may include one or more workers, and each worker may be one or more GPUs, TPUs, or another AI processor optimized for performing multiplication and addition (multiplication-accumulation "MAC"), matrix multiplication ("MatMul"), and other operations. The controller is sometimes referred to as a host or gateway. For example, the controller may be a traditional CPU, FPGA, system-on-a-chip (SoC), application-specific integrated circuit (ASIC), or embedded ARM controller, or another processor that can run software and communicate with the worker group based on software instructions. The system may include drivers that allow software to organize and control the tasks that need to be performed on the target device.

[0030] Figure 5 The image illustrates a high-level representation of a typical synchronous data parallelism process. In this example, each iteration ends with model synchronization across a group of workers (WG). Traditionally, global checkpoints are performed periodically to recover from any errors or failures in the group of workers. Based on some previous systems, frequent checkpointing can severely degrade throughput, so global checkpointing is often distributed (e.g., once per hour). However, a potential problem is that recovery from such errors is also slow, such as… Figure 6 As shown. Due to Figure 6 The failure shown causes all worker groups to be interrupted and quickly returned to a global checkpoint. If errors or failures are frequent enough (such as poisoning in a large cluster), this can have a severe impact on performance.

[0031] The features and advantages of this disclosure include the ability to recover from errors and certain failures occurring within a large cluster by accessing models from previous iterations, enabling faster recovery (e.g., within seconds rather than hours) without having to restore the entire group to a global checkpoint. Figure 7 As illustrated and as described above, errors occurring in one or more worker groups can be resolved during the current iteration, where a local recalculation is performed based on the model used at the start of the iteration. Therefore, for example, a worker group experiencing an error can recover quickly, and all worker groups can continue with subsequent iterations without having to reprocess data from multiple previous iterations.

[0032] The exemplary embodiments of this disclosure can leverage the observation that a state (e.g., the model) can be recalculated from a previous state as long as a fast and redundant copy available for recovery exists. Therefore, in one embodiment, a “master copy” of the current model (e.g., parameters such as neural network weights used by the worker group at the start of an iteration) can be stored in a location accessible to each worker group (e.g., on the controller). Note that the master copy may only need to be the minimum state information required for recalculation, so a copy of the model from the current iteration may not have some recalculated state information (e.g., activations). Alternatively, if a worker group encounters an error, the master copy may also reside directly on the worker group (e.g., in local memory protected by error-correcting codes (ECC)) for local access by a specific worker group. In other embodiments, each worker group maintains an additional copy of the model for the current iteration, which is not updated during processing, so it can be used by other worker groups that may encounter error conditions. Advantageously, for example, if the model used for the current iteration is maintained by each worker group, different parts of the model (different subsets of the whole model) can be sent to the failed worker group simultaneously by multiple different worker groups, which in some architectures can be much faster than sending the model from the controller to the failed worker group.

[0033] In one embodiment, redundant copies of the model can be distributed across worker groups, such that each worker group receives two distinct portions of the two copies (e.g., if it carries the same portions of both copies, a failure in the worker group would result in irreparable loss). The primary copy may be updated frequently at the end of each iteration. It can also be updated more frequently with some form of data parallelism that allows for local updates. Finally, in some example embodiments, the controller can be notified of any unrecoverable errors (such as parity errors) or whether a local timeout is set by a worker in the worker group; for example, the local timeout may be much smaller than the global timeout minus the estimated time recovery time, but sufficient to identify the error. In addition to timeouts, workers can send heartbeats to the controller so that the controller can determine when a worker has encountered an error.

[0034] In various embodiments, the recovery method can depend on the failure condition. For parity errors (poisoning): the controller can reset the worker group to rerun the model from the primary copy using the same small batch of data. For local timeouts (or heartbeat misses), the controller can force the failed worker to reset (e.g., via a sideband operation). If this operation succeeds, recovery continues as in the parity error or poisoning cases described above. If unsuccessful after repeated attempts, the controller can recompile the less efficient model on the same worker group, or it can employ a dedicated backup worker group. If these options do not work or are unavailable, the controller can fail on its own.

[0035] In the event of a controller failure, all controllers can have the same primary copy of the model at the end of each iteration. Therefore, a controller failure that causes a global timeout may not necessarily require recovery to a global checkpoint. For example, after the software resizes the operational worker groups and data shards for a new cluster size, the controller can continue from the current iteration point.

[0036] In various embodiments, multiple methods can be used to recover redundant copies from the end of a previous iteration. In one embodiment, the controller provides copies from its own copy in memory. In another embodiment, the failed worker group may have a primary copy in local memory (e.g., in directly attached ECC-protected memory). In yet another embodiment, the failed worker group collects copies from one or more operable worker groups (e.g., in parallel).

[0037] Figure 8 The illustration depicts an example error recovery method where the controller interacts with a target device in a group of workers. Here, "device" 891 refers to a worker or group of workers (e.g., a group of devices sharing a copy of a model). In this example, for simplicity, only one device is shown, but the controller 890 may have processes for each device 891. For example, arrows 850-857 illustrate the flow of data and / or control information between the controller and the AI ​​processor. Figure 8The example method illustrated in the diagram shows multiple iterations. At 801, controller 890 can initialize device 891. Thus, for example, at 820, device 891 can perform a soft reset. At 802, the model is initialized, and devices 891 can each receive a copy of the initial model. At 803 and 804, each controller 890 and associated device 891 performs initial synchronization. Iteration begins at 804, which illustrates the first iteration (e.g., iter = 1). At 805, controller 890 causes a small batch of data to be sent to each device 891 (e.g., DevId is the device identifier). Each device 891 receives the data at 823 and runs the data against the model at 824. At 825, device 891 may or may not detect an error. Again, controller 890 can poll the devices during processing to look for errors. For example, if the device does not detect an error at 825 or the controller does not detect an error at 806, the controller and device synchronize at 809 and 829, and controller 890 can initiate the next iteration. However, if device 891 detects an error at 825, the device with the error can wait for controller 890 to reset it at 826. In this example, controller 890 can detect the device ID (“devID”) of the device with the error at 806 and perform a soft reset of the device at 807. At 808, the controller can send a copy of the model used during the current iteration to the device with the error. At 827, the device performs a soft reset, and at 828, the device receives and loads the model. For example, the “RecoverModel” box can correspond to one of the embodiments described above for the recovery technique. The device can then reload the data at 823 and run the data against the reloaded model at 824. Other devices that have not experienced the error can enter a waiting state and recover after the device that experienced the error has completed its iteration processing. In some embodiments described herein, for example, other devices can receive portions of the model and portions of the data, and the loading and reprocessing of the data of the device that experienced the error can be performed by multiple devices to reduce recovery time.

[0038] exist Figure 9In one embodiment of the fault (controller recovery) mechanism, the controller may have a primary copy. In this example, “n” controllers 901-905 may be coupled to “k” worker groups (e.g., “k” groups of one or more AI processors) and memories 911-914, respectively. For example, at each iteration, the primary copy of the model may be stored in memories 911-914 coupled to the controllers 901-905. For example, the memories may be protected by error-correcting codes (ECC) to ensure the integrity of the stored models. Recovery may be initiated by the controller by detecting a local timeout (or, a lost heartbeat) or poisoning. In either case, it is assumed that the faulty worker is recoverable. If the worker itself is not working, the controller may simply signal the error, which can only be corrected by returning to a global checkpoint and rebalancing the cluster. In another case where the faulty worker is not working, the controller may rebalance the same mini-batches across the remaining workers (only where possible). Figure 9 The example situation shown is fully recoverable, although the worker may only report a detectable soft bug poisoning.

[0039] As described above, in the second embodiment (self-recovery), ECC-protected memory is attached to each worker. When a worker detects poisoning, it will attempt self-recovery. It will retry the same small batch by rebooting and loading the model / graphics / data from the attached ECC memory. Poisoning can be further segmented by category to speed up recovery. For example, the worker specifies the location where poisoning occurred (by address range), and the recovery code then uses that location to repair only that segment before rebooting. In the case of self-recovery, a soft-suspended worker can still recover if the worker incorporates a watchdog timer interrupt (self-heartbeat) (which is possible if there is a core dedicated to this purpose).

[0040] In the third embodiment (neighbor recovery), a group of k workers (e.g., T1 to Tk) with or without a controller can recover even in the event of a hard failure by regrouping into smaller groups that still operate on the same mini-batches. To achieve this, the group can incorporate model redundancy. This is particularly possible for model partitioning (model parallelism), where the worker group splits the model across multiple workers (e.g., different workers process different parts of the model). In this partitioning, a portion of each worker's memory carries a redundant copy of the model state of another worker in a mutually exclusive manner (e.g., only the minimum model state required for recovery). For example, whenever worker T1 updates its segment Seg(1), it also updates the redundant state in worker Tk. This can be performed as a hardware-assisted mirror write, a software write, or during a model update after a full reduction, for example...

[0041]

[0042] Table 1 – Redundancy in model segmentation across groups.

[0043] Therefore, in various embodiments, redundant replica distribution is used, where two or more replicas can be distributed across mutually exclusive partitions (i.e., the same target does not store the same segment in different replicas) so that any new (or restarted) target can collect complete replicas from other members. Having two replicas ensures recovery from one failure, having three replicas ensures recovery from two failures, and so on. However, even for large clusters, two replicas can be used to recover from soft errors or restarts.

[0044] Therefore, in various embodiments, local recovery can be performed using a primary copy of the current iterative model, which is stored in the controller, locally on the worker group, or, for multiple workers in the worker group, the primary copy can be exclusively partitioned across multiple workers in the same worker group (e.g., partitioned specifically to the original copy so that no worker has overlapping parts of the model).

[0045] Therefore, when there are multiple workers in a worker group, the primary replica can be exclusively partitioned from the running replica across the same worker group. One example alternative is that two or more replicas can be exclusively partitioned across workers, so that any failure can be recovered by collecting one of the full replicas into a restarted or replaced target. In another embodiment, the replicas can be redundant replicas from a specific target.

[0046] Figure 10 The illustration depicts recovery from an error occurring during the result aggregation phase according to an example embodiment. In some embodiments, recovery from errors occurring during the result aggregation phase of each iteration may be advantageous. For example, as... Figure 10 As illustrated, the iteration can include synchronizing the model across all worker groups at step 1001. At step 1002, mini-batches are received and applied to the model to produce results. At step 1003, post-data synchronization can occur, marking the beginning of the results aggregation phase. In some cases, errors may occur in one of the AI ​​processors after data has been applied to the model. Typically, for example, each worker group can generate a unique incremental value vector (e.g., gradient) that indicates the changes in model parameters after each mini-batch of data has been processed.

[0047] Figure 11The illustration depicts an example result generation according to an embodiment. Here, N worker groups WG0-WGN (N is an integer) produce N vectors of length M (e.g., where M is an integer and equal to the number of neural network weights in the model). For example, the increment value Δij in each vector can be a floating-point number. When the system is working (e.g., without errors), the vectors produced by each worker group are passed to all other worker groups, and each worker group aggregates a subset of fields from each vector. For N worker groups, there can be N partitions, and each worker group aggregates the results of specific fields of the vectors received from other worker groups. For example, worker group WG0 can receive vectors from other worker groups and aggregate Δ1j-Δij fields to produce, for example, a result array R0. For example, the aggregation can include the average of the weights or other functions known to those skilled in the art of AI processing. However, if one of the worker groups encounters an error during result processing, that worker group can send an invalid result indicator to the other worker groups. In this example, WG0 sends a result vector of length M, which includes garbage bits (represented here as "xxxx"). During result processing, if another worker group receives an invalid result indicator from any other worker group, that worker group can be triggered into a waiting state. Therefore, a worker group can wait while the worker group experiencing the error corrects the error and processes a valid result.

[0048] Figure 12 The illustration depicts an example of result aggregation according to an embodiment. In some embodiments, worker groups can be configured in a circular manner, where worker groups can pass gradient vectors (as described above) and then pass the results (e.g., aggregated gradients) to other worker groups. In this example, each worker group can receive an array of aggregated gradient results. When all worker groups have all the results from all the other worker groups, each worker group will have a complete set of aggregated results that they can use to modify their model version. In this example, since all worker groups start with the same model, each update to the model will result in the model remaining substantially the same (e.g., the AI ​​parameters change together, so each worker group has substantially the same model across all iterations). Similarly, if a worker group encounters an error, that worker group can output an invalid result indicator, while other worker groups can wait until the worker group that encountered the error recovers and sends a valid result.

[0049] Figure 13The illustration depicts error recovery in a multiprocessor computing environment according to an embodiment. In this example, worker group 1301 encounters an error and outputs an invalid result indicator x 1311. Other worker groups (e.g., 1300, 1302) can generate valid gradient vectors Δ (e.g., 1310, 1312). In this example, each of the other worker groups can wait until worker group 1301 corrects its error and generates a valid result. The system can then pass the valid gradient vectors, compute the aggregated results, and forward the results to other worker groups during the result aggregation phase, e.g., so that each worker group has an updated model.

[0050] Figure 14 The illustration depicts the distribution of computation for faulty processors across multiple processors according to an embodiment. In some embodiments, different parts of the model can be loaded across worker groups, including those that experienced the error, as worker groups detect and eliminate errors. Therefore, the time spent recalculating results for worker groups that experienced errors during a particular iteration can be reduced. Figure 14 As illustrated, for example, worker groups 1410-1413 could use the same model to process mini-batches. Here, worker group 1412 encounters an error. However, in this example, the model used for the current iteration is split into multiple worker groups, including worker group 1412. (Reference) Figure 14 When worker group 1412 eliminates errors, worker group 1412 can trigger the loading of model 1450 across worker groups 1410-1413. Thus, for example, a portion of the training data intended to be processed by worker group 1412 in the current iteration can be processed across multiple worker groups 1410-1413 to reduce recovery time.

[0051] Other example embodiments

[0052] In various embodiments, this disclosure includes an error recovery method. This method may be embodied in a non-transitory computer-readable storage medium on which program code executable by a computer system is stored, causing the computer system to perform the techniques described herein. In some embodiments, the computer system may include a plurality of artificial intelligence processors and one or more controllers. For example, the non-transitory computer-readable storage medium may be a memory that can be coupled to, for example, one or more controllers or one or more artificial intelligence processors.

[0053] The following techniques may be embodied individually or in different combinations, and may be further embodied together with other techniques described herein.

[0054] For example, in one embodiment, this disclosure includes a method comprising: detecting computational errors in a first artificial intelligence processor among a plurality of artificial intelligence processors during a first processing iteration of data from a dataset; eliminating the errors from the first artificial intelligence processor; and loading a model in one or more artificial intelligence processors, including the first artificial intelligence processor, wherein the model corresponds to the same model processed by the plurality of artificial intelligence processors during the first processing iteration of the data from the dataset.

[0055] In one embodiment, the plurality of AI processors other than the first AI processor wait while the first AI processor eliminates the error, and wherein the plurality of processors simultaneously process data from the dataset in the next processing iteration using a second identical model generated from the same model used in the first processing iteration.

[0056] In one embodiment, the computational error is detected during the result aggregation phase of the first processing iteration, and at least a portion of the plurality of AI processors wait for the first AI processor to produce a valid result during the aggregation phase before completing the result aggregation phase.

[0057] In one embodiment, the first AI processor sends an invalid result indicator to at least a portion of the plurality of AI processors to trigger the wait.

[0058] In one embodiment, the result aggregation phase is a full specification.

[0059] In one embodiment, loading the model includes loading different portions of the model by one or more AI processors, including the first AI processor, in the AI ​​processor, and the method further includes processing a first portion of the data received by the first AI processor in the first processing iteration in the one or more AI processors, including the first AI processor, in the AI ​​processor.

[0060] In one embodiment, loading the model includes loading the model in the first artificial intelligence processor, and the method further includes further processing in the first artificial intelligence processor a first portion of the data received by the first artificial intelligence processor in the first processing iteration.

[0061] In one embodiment, the model is received from the controller in the first artificial intelligence processor.

[0062] In one embodiment, the model is received in the first artificial intelligence processor from one or more other processors among the plurality of artificial intelligence processors.

[0063] In one embodiment, the model is received in the first artificial intelligence processor from the local memory of the first artificial intelligence processor.

[0064] In one embodiment, the model includes artificial intelligence parameters.

[0065] In one embodiment, the model includes neural network weights.

[0066] In one embodiment, the dataset is a training dataset.

[0067] The foregoing description illustrates various embodiments of this disclosure and examples of aspects that may implement particular embodiments. The examples above should not be considered as the only embodiments and are presented to illustrate the flexibility and advantages of particular embodiments as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents may be employed based on the foregoing disclosure and the appended claims without departing from the scope of this disclosure as defined by the claims.

Claims

1. A method of error recovery, comprising: detecting a computing error in a first artificial intelligence processor of a plurality of artificial intelligence processors during a first time period of a first processing iteration of data from a data set, the computing error comprising a software error and a hardware error; remedying the computing error from the first artificial intelligence processor; and loading a model in the first artificial intelligence processor during a second time period of the first processing iteration, wherein the model is a reloaded model corresponding to a same model processed successfully at least in part by a second artificial intelligence processor of the plurality of artificial intelligence processors during the first time period, and the second artificial intelligence processor waits after processing the same model until the first artificial intelligence processor completes processing in the second time period.

2. The method of claim 1, wherein the plurality of artificial intelligence processors other than the first artificial intelligence processor wait while the first artificial intelligence processor remedies the computing error, and wherein the plurality of artificial intelligence processors simultaneously process data from the data set in a next processing iteration using a second same model generated from the same model used in the first processing iteration.

3. The method of claim 1, wherein the computing error is detected during a result aggregation phase of the first processing iteration, and wherein at least a portion of the plurality of artificial intelligence processors wait for the first artificial intelligence processor to produce valid results during the result aggregation phase before completing the result aggregation phase.

4. The method of claim 3, wherein the first artificial intelligence processor sends an invalid result indicator to at least a portion of the plurality of artificial intelligence processors to trigger the waiting.

5. The method of claim 3, wherein the result aggregation phase is a full reduction.

6. The method of claim 1, wherein the loading a model comprises loading different portions of the model in one or more artificial intelligence processors of the plurality of artificial intelligence processors including the first artificial intelligence processor, the method further comprising processing a first portion of the data received by the first artificial intelligence processor in the first processing iteration in the one or more artificial intelligence processors of the plurality of artificial intelligence processors including the first artificial intelligence processor.

7. The method of claim 1, wherein the loading a model comprises loading the model in the first artificial intelligence processor, the method further comprising processing a first portion of the data received by the first artificial intelligence processor in the first processing iteration in the first artificial intelligence processor.

8. The method of claim 1, wherein the model is received in the first artificial intelligence processor from a controller.

9. The method of claim 1, wherein the model is received in the first artificial intelligence processor from one or more other processors of the plurality of artificial intelligence processors. ​ 10. The method of claim 1, wherein the model is received in the first artificial intelligence processor from a local memory of the first artificial intelligence processor.

11. The method of claim 1, wherein the model comprises artificial intelligence parameters.

12. The method of claim 1, wherein the model comprises neural network weights.

13. The method of claim 1, wherein the data set is a training data set.

14. A non-transitory computer-readable storage medium having stored thereon program code executable by a computer system, the program code causing the computer system to: detect a computing error in a first artificial intelligence processor of a plurality of artificial intelligence processors during a first time period of a first processing iteration of data from a data set, the computing error comprising a software error and a hardware error; eliminate the computing error from the first artificial intelligence processor; and load a model in one or more artificial intelligence processors of the artificial intelligence processors including the first artificial intelligence processor during a second time period of the first processing iteration, wherein the model is a reloaded model corresponding to a same model successfully processed at least in part by a second artificial intelligence processor of the plurality of artificial intelligence processors during the first time period during the first processing iteration of the data from the data set, and the second artificial intelligence processor waits after processing the same model until the first artificial intelligence processor completes processing in the second time period.

15. The non-transitory computer-readable storage medium of claim 14, wherein the plurality of artificial intelligence processors other than the first artificial intelligence processor wait while the first artificial intelligence processor eliminates the computing error, and wherein the plurality of artificial intelligence processors simultaneously process data from the data set in a next processing iteration using a second same model generated from the same model used in the first processing iteration.

16. The non-transitory computer-readable storage medium of claim 14, wherein the computing error is detected during a result aggregation phase of the first processing iteration, and wherein at least a portion of the plurality of artificial intelligence processors wait for the first artificial intelligence processor to produce a valid result during the result aggregation phase before completing the result aggregation phase.

17. The non-transitory computer-readable storage medium of claim 16, wherein the first artificial intelligence processor sends an invalid result indicator to at least a portion of the plurality of artificial intelligence processors to trigger the wait.

18. A system comprising: a plurality of artificial intelligence processors; one or more controllers; and a memory having stored thereon program code executable by the one or more controllers and the plurality of artificial intelligence processors, the program code causing the system to: ​ detecting a computing error in a first artificial intelligence processor of a plurality of artificial intelligence processors during a first time period of a first processing iteration of data from a data set, the computing error comprising a software error and a hardware error; removing the computing error from the first artificial intelligence processor; and loading a model in the first artificial intelligence processor during a second time period of the first processing iteration, wherein the model is a reloaded model corresponding to a same model processed successfully at least in part by a second artificial intelligence processor of the plurality of artificial intelligence processors during the first time period, and the second artificial intelligence processor waits after processing the same model until the first artificial intelligence processor completes processing in the second time period.

19. The system of claim 18, wherein the plurality of artificial intelligence processors other than the first artificial intelligence processor wait while the first artificial intelligence processor removes the computing error, and wherein the plurality of artificial intelligence processors simultaneously process data from the data set in a next processing iteration using a second same model generated from the same model used in the first processing iteration.

20. The system of claim 18, wherein the computing error is detected during a results aggregation phase of the first processing iteration, and wherein at least a portion of the plurality of artificial intelligence processors wait for the first artificial intelligence processor to produce valid results during the results aggregation phase before completing the results aggregation phase.

Images