Data processing method, data processing apparatus, electronic device, storage medium

Abstract

A data processing method, a data processing device, an electronic equipment and a storage medium are provided. The method is applied to the field of data processing. The method comprises the following steps: in response to starting a checkpoint storage operation, screening and determining whether there is at least one first tensor; in response to there being no at least one first tensor, performing a persistent storage operation on all tensors; in response to there being at least one first tensor: performing a data consistency check on the first tensor to confirm whether the content of the data copy of each first tensor on multiple dies is completely consistent; in response to the content of the data copy of any first tensor on multiple dies being not completely consistent, not performing the persistent storage operation on all tensors; in response to the content of the data copy of each first tensor in the first tensors on multiple dies being completely consistent, performing the persistent storage operation on all tensors. The method can intercept a fault at an early stage of a data fork fault.

Description

Technical Field

[0001] The embodiments of this disclosure are applied to the field of electrical data processing, specifically relating to a data processing method, a data processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] Large-scale distributed training of large models requires extremely high computing power and GPU memory, which is difficult to meet with a single chip. Therefore, multi-die packaged chips are typically used in a network for parallel training. These long-term training tasks are prone to interruption due to hardware failures or node crashes. Therefore, it is necessary to record and persistently store training information such as model parameters and optimizer states through checkpoints to enable resume training from where it was interrupted.

[0003] Global logical tensors are assigned tensor distribution labels to define the distributed storage mapping rules for global logical tensors across multiple dies. For example, Split-Broadcast-Partial semantics (SBP semantic labels) can be used. In a multi-die architecture, model parameters are distributed across various dies. For tensors with Broadcast-Broadcast semantics (BB semantic labels), it is required that the dies maintain completely consistent data copies. However, traditional checkpoint protection methods cannot guarantee that the data copies between dies are completely consistent before persistent storage. This results in the training states before and after loading checkpoints being unreproducible, and the fault cannot be traced. Summary of the Invention

[0004] This disclosure provides at least one embodiment of a data processing method for a hardware architecture including multiple die chips, the multiple die chips including multiple dies, the hardware architecture being used for distributed data processing through the multiple dies, the data processing method including: in response to initiating a checkpoint storage operation, filtering and determining whether at least one first tensor exists from all tensors that need to be persistently stored at the current checkpoint, wherein the first tensor is defined as having completely consistent content among data copies on the multiple dies; in response to the absence of the at least one first tensor, performing a persistent storage operation on all tensors; in response to the presence of the at least one first tensor: performing a data consistency check on the at least one first tensor, wherein the data consistency check includes confirming whether the content of the data copies of the first tensor on the multiple dies is completely consistent; in response to the content of any of the at least one first tensor's data copies on the multiple dies not being completely consistent, not performing the persistent storage operation on all tensors; in response to each of the at least one first tensor having completely consistent content among the data copies of the first tensor on the multiple dies, performing the persistent storage operation on all tensors.

[0005] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which the existence of at least one first tensor is screened and determined from all tensors that need to be persistently saved at the current checkpoint, including: traversing the tensor distribution labels of all tensors, and screening tensors with tensor distribution labels as first distribution labels as the first tensor, wherein the first distribution label requires that the content of the tensor is completely consistent among the data copies on the plurality of bare dies.

[0006] For example, in at least one embodiment of this disclosure, a data processing method is provided, which involves traversing the tensor distribution labels of all tensors and selecting tensors whose tensor distribution labels are the first distribution labels as the first tensor. This includes: constructing a state dictionary corresponding to the current checkpoint, wherein the state dictionary includes the tensor distribution labels of all tensors; traversing the state dictionary, and selecting tensors whose tensor distribution labels are the first distribution labels as the first tensor based on the tensor distribution labels of all tensors.

[0007] For example, in at least one embodiment of this disclosure, a data processing method is provided, wherein the tensor distribution label is a fragmented-broadcast-local reduction tensor parallel semantic label, and the first distribution label is a broadcast-broadcast semantic label.

[0008] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which a data consistency check is performed on the at least one first tensor, including: traversing the at least one first tensor and performing a data consistency check on each first tensor to confirm whether the contents of the data copies of each first tensor on the plurality of bare dies are completely consistent.

[0009] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which data consistency checks are performed on the at least one first tensor, including: performing data consistency checks on the at least one first tensor sequentially, and stopping the data consistency checks on subsequent first tensors in response to the fact that the contents of the data copies of any first tensor on the plurality of bare dies are not completely consistent.

[0010] For example, in at least one embodiment of this disclosure, a data processing method is provided, wherein the data consistency check includes: reading multiple data copies of the first tensor from the multiple dies; performing a consistency check operation on the multiple data copies to confirm whether the multiple data copies are completely consistent at the bit level; in response to the multiple data copies being completely consistent at the bit level, determining that the content of the data copies of the first tensor on the multiple dies is completely consistent; and in response to the multiple data copies not satisfying the requirement of being completely consistent at the bit level, determining that the content of the data copies of the first tensor on the multiple dies is not completely consistent.

[0011] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which a consistency check operation is performed on the plurality of data copies to confirm whether the plurality of data copies are completely consistent at the bit level, including: performing fingerprint calculation on the plurality of data copies respectively to obtain a data fingerprint corresponding to each data copy; comparing the data fingerprints of the plurality of data copies; and confirming that the plurality of data copies are completely consistent at the bit level in response to the data fingerprints of the plurality of data copies being completely consistent, otherwise confirming that the plurality of data copies are not completely consistent at the bit level.

[0012] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which a consistency check operation is performed on the plurality of data copies to confirm whether the plurality of data copies are completely consistent at the bit level, including: performing a full byte comparison on the plurality of data copies; in response to the result of the full byte comparison indicating that the contents of the plurality of data copies are completely consistent, confirming that the plurality of data copies are completely consistent at the bit level, otherwise confirming that the plurality of data copies are not completely consistent at the bit level.

[0013] For example, in at least one embodiment of this disclosure, a data processing method is provided, in which a consistency check operation is performed on the plurality of data replicas to confirm whether the plurality of data replicas are completely consistent at the bit level. This includes: dividing each of the plurality of data replicas into a plurality of data blocks of a fixed size, and performing fingerprint calculation on each data block to obtain a data fingerprint for each data block; comparing the data fingerprints of data blocks at corresponding positions in the plurality of data replicas; determining that the plurality of data replicas are not completely consistent at the bit level in response to any group of data blocks having inconsistent data fingerprints, wherein the data blocks at corresponding positions in the plurality of data replicas are considered as a group of data blocks; and determining that the plurality of data replicas are completely consistent at the bit level in response to all data fingerprints of the data blocks at corresponding positions in the plurality of data replicas being completely consistent.

[0014] For example, at least one embodiment of this disclosure provides a data processing method that further includes: in response to the fact that the contents of any one of the at least one first tensor on the plurality of bare dies are not completely consistent, generating a difference location report for the any one first tensor, wherein the difference location report describes the location information of the data intervals or data elements that differ among the plurality of data copies of the any one first tensor.

[0015] For example, at least one embodiment of this disclosure provides a data processing method that further includes: in response to the fact that the contents of any one of the at least one first tensor on the data copies of the plurality of bare dies are not completely consistent, providing a fault indication, wherein the fault indication includes indication information of the any one of the first tensors.

[0016] For example, in at least one embodiment of this disclosure, a data processing method is provided, wherein the multi-die chip includes an artificial intelligence accelerator or an artificial intelligence chip, and the distributed data processing includes deep learning distributed training.

[0017] This disclosure provides at least one embodiment of a data processing apparatus for a hardware architecture including multiple die chips, the hardware architecture being used for distributed data processing. The data processing apparatus includes: a filtering module configured to, in response to initiating a checkpoint storage operation, filter and determine whether at least one first tensor exists from all tensors that need to be persistently stored at the current checkpoint, wherein the first tensor is defined as having completely identical content among data copies on the multiple dies; and a processing module configured to, in response to the absence of the at least one first tensor, perform a persistent storage operation on all tensors; In response to the existence of at least one first tensor: a data consistency check is performed on the at least one first tensor, wherein the data consistency check includes confirming whether the contents of the data copies of the first tensor on the plurality of dies are completely consistent; in response to the contents of the data copies of any one of the at least one first tensor on the plurality of dies not being completely consistent, the persistent storage operation is not performed on all tensors; in response to the contents of the respective data copies of the first tensor on the plurality of dies being completely consistent for each of the at least one first tensor, the persistent storage operation is performed on all tensors.

[0018] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement a data processing method according to any embodiment of this disclosure.

[0019] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any embodiment of this disclosure.

[0020] In the data processing method provided in at least one embodiment, after initiating the checkpoint storage operation and before performing the persistent storage operation, a data consistency check is performed on the tensor that requires complete consistency of content among multiple data copies of the same die on the checkpoint storage path. If any tensor of this type is found to be inconsistent across dies, subsequent persistent storage operations are refused. This allows for the timely detection of potential data fork failures between dies before data is stored on disk, preventing faulty data from being stored on disk and causing silent errors to propagate to downstream training or inference. Faults are intercepted early in their occurrence. For large model training, the cost of retraining after a single fault is extremely high. The data processing method provided in at least one embodiment of this disclosure can effectively avoid the high retraining overhead caused by continuing training based on faulty checkpoints, thus promoting the standardization of reliability design for distributed data processing in multi-die architectures. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure, and are not intended to limit this disclosure.

[0022] Figure 1 This is a schematic diagram of a heterogeneous computing cluster. Figure 2 This is a schematic diagram of the architecture of a general-purpose graphics processing unit (GPGPU). Figure 3 A schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure; Figure 4 A schematic diagram of the processing flow of a data processing method provided in at least one embodiment of this disclosure; Figure 5 A schematic block diagram of a data processing apparatus provided in at least one embodiment of this disclosure; Figure 6 A schematic block diagram of an electronic device provided in one embodiment of this disclosure; Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0024] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0025] In distributed deep learning training, model checkpoints are a crucial mechanism for saving intermediate training states, enabling fault recovery, and facilitating training continuation. Checkpoints typically store model parameters (such as network weights, biases, batch normalization statistics, etc.) and optimizer states on disk as serialized files, allowing for model state recovery in case of training interruption or transfer errors. As model size increases and multi-chip distributed training becomes the norm, the reliable saving and loading of checkpoints directly impacts training reproducibility and the final model quality.

[0026] The hardware architecture of multi-die distributed training typically uses multiple dies within a package as the basic computing unit. The dies communicate with each other via high-speed inter-die interconnects to achieve intra-die data exchange and tensor communication. A single chip can realize the fragmented storage and parallel computation of weights, gradients, and intermediate features. A large number of multi-die chips form a cluster by relying on board-level interconnects, high-speed buses within servers, and cross-node network links.

[0027] For example, the hardware architecture may include one or more multi-die chips, and multiple dies may be set in a single multi-die chip, such as two dies. For example, the chip may be a graphics processor or a general-purpose graphics processor chip, and each die can be regarded as a small GPU unit.

[0028] For example, the hardware architecture used for multi-die distributed training is typically a heterogeneous computing cluster, which also includes a central processing unit.

[0029] Figure 1 This is a schematic diagram of a heterogeneous computing cluster.

[0030] exist Figure 1The example shows four fully connected nodes (nodes 0-3), meaning that every two nodes have a direct communication connection. Nodes can communicate across nodes via high-speed interconnect networks, such as Remote Direct Memory Access (RDMA). However, it should be noted that... Figure 2 The number of nodes and connection relationships in the example are merely illustrative, and this disclosure does not impose any specific limitations on them.

[0031] exist Figure 1 In the example, node 2 is used to illustrate the specific structure of a single node. The structures of other nodes are similar and will not be described in detail here.

[0032] like Figure 1 As shown, node 2 illustrates two central processing units (CPUs), each electrically connected to multiple parallel processors via an electrical switch. For example, in one example, a node includes two CPUs, each electrically connected to four parallel processors via an electrical switch; for example, the parallel processors could be... Figure 1 The architecture of the graphics processing unit (GPU) is as follows: The central processing unit (CPU) is used to schedule, coordinate, and control multiple parallel processors connected to the CPU via electrical switches. Each electrical switch is used to forward data flows between the multiple parallel processors connected to the CPU.

[0033] like Figure 1 As shown, there is also an electrical connection between the two central processing units, thereby enabling communication between parallel processors within the node.

[0034] Figure 1 The specific connection relationship between parallel processors is not shown. For example, parallel processors can be fully interconnected within a group and paired interconnected between groups, or they can be fully interconnected in a topology. This disclosure does not impose any specific restrictions on this.

[0035] Of course, the number of central processing units and parallel processors in a node is not specifically limited in this disclosure, and those skilled in the art can set their specific numbers and topological relationships as needed.

[0036] For example, a parallel processor can be a graphics processing unit chip that includes multiple dies. Of course, a parallel processor can also be other multi-die chip architectures, such as a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), a data processing unit (DPU), a neural processing unit (NPU), etc.

[0037] Figure 2 This is a schematic diagram of the structure of a general-purpose graphics processor.

[0038] like Figure 2 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 2 The diagram shows multiple streaming processor clusters, all with identical structures. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0039] like Figure 2 As shown, a streaming processor cluster comprises multiple compute units (CUs), all with identical structures. Each CU performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A CU contains multiple cores, each including an arithmetic logic unit (ALU), a floating-point unit, etc., which execute specific computational tasks. Furthermore, a CU also includes registers (e.g.,...). Figure 2 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0040] like Figure 2 As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0041] like Figure 2As shown, each streaming processor cluster also provides a buffer for data caching of multiple computing units within the streaming processor cluster.

[0042] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 2 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0043] In each computing unit, the thread beam scheduling / distribution module ( Figure 2 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 2 High Bandwidth Memory (HBM) is used for read and write operations.

[0044] For example, this hardware architecture containing multiple bare chips can be a multi-card (multi-GPU) system. This system integrates multiple bare chips in the form of GPU cards. The multiple GPU cards work together to complete large-scale parallel computing tasks, and are commonly used in applications such as artificial intelligence training, scientific computing, and high-performance graphics rendering. This hardware architecture containing multiple bare chips supports multi-level parallel scheduling across bare chips, across chips, and across servers, thereby breaking through the limitations of single-die computing power and storage, and completing ultra-large-scale distributed training tasks for large models.

[0045] Furthermore, this hardware architecture, which includes multiple bare chips, can also be widely used in scenarios requiring large-scale data processing, high-throughput computing, or low-latency parallel tasks. Examples include deep learning and large model training, high-performance computing, large-scale data processing and AI inference, real-time rendering and visualization, and it can be applied to fields such as large model training, scientific computing, high-concurrency AI services, and real-time rendering.

[0046] For hardware architectures containing multiple bare chips, the physical characteristics are that a single physical card contains multiple independent bare chips, which communicate through on-chip interconnects. Tensors carry tensor distribution labels at the framework abstraction layer, such as SBP semantic labels, where the BB semantic label explicitly requires that the bare chips hold completely consistent copies of data. However, due to compiler defects, communication primitive anomalies, incorrect SBP transformation paths, hardware soft errors (such as bit flips or uncorrectable errors in ECC (Error-Correcting Code)), or oversights in the implementation of custom operators, the data copies between bare chips may quietly fork during training.

[0047] Currently, the mainstream model checkpoint protection solutions in the industry mainly include the following three categories: (1) Direct serialization scheme: The model parameters are written directly to the disk without any runtime data verification. This scheme only guarantees that the byte-level writing is completed and cannot detect whether the data has been corrupted or forked before it is saved.

[0048] (2) File-level hash verification scheme: Calculate the file hash value (such as SHA256) after saving, and recalculate and compare it during loading. This scheme can detect storage media level damage such as storage media damage, file transmission truncation or bit flipping, but it cannot detect inconsistencies in the internal data that already existed in the model parameters before saving.

[0049] (3) Shard integrity verification scheme: For distributed sharded checkpoints, the scheme verifies whether the data ranges stored by each computing node are complete and non-overlapping. This scheme only verifies the correctness of the sharded metadata and the range allocation rules, and does not involve the consistency verification of the data content itself.

[0050] The common limitation of the three types of solutions mentioned above is that they all assume the data to be saved is correct before saving, and their focus is on preventing data corruption after saving. However, in a multi-die architecture, data corruption may occur during the computation and communication stages. Moreover, the data within a single die may be intact, correctly shaped, and have a seemingly reasonable numerical range. Furthermore, the training loss may continue to decrease in the short term. Existing training frameworks and checkpointing mechanisms lack the ability to detect this and are difficult to detect in a timely manner through conventional monitoring methods.

[0051] Specifically, taking a single chip including die 0 and die 1 as an example, currently for the tensor of BB semantic tags, during the storage phase, the framework is based on the assumption of "data consistency across dies" and usually only persists the copy of die 0 to reduce storage space; during the loading phase, after the single copy of data in the disk is loaded into die 0, it is then copied to other dies through cross-die broadcast to reconstruct the multi-copy state required by semantics.

[0052] This mechanism is a reasonable performance optimization when the data consistency assumption holds. However, if data forks between wafers before saving, the saving phase will unilaterally adopt the view of one wafer and discard the views of the others, and the loading phase will broadcast that view to cover all wafers. This will result in the permanent loss of forked data and the erasure of fault traces; unexplained discontinuities in training states (such as loss values ​​and gradient norms) before and after checkpoints, compromising training reproducibility; and wafer forks that might have manifested as numerical anomalies or accuracy degradation in subsequent training may be repaired into superficially consistent erroneous versions by a single save / load operation, making it difficult to trace the root cause of the problem. This irreversible loss of state will cause fault checkpoints to be saved intact, resulting in sudden divergence of loss and a sharp drop in accuracy during subsequent training or inference, with extremely high localization costs.

[0053] This disclosure provides at least one embodiment of a data processing method, a data processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. The data processing method is based on a hardware architecture including multiple die chips, the hardware architecture being used for distributed data processing across multiple dies. The data processing method includes: in response to initiating a checkpoint storage operation, filtering and determining whether at least one first tensor exists from all tensors that need to be persistently stored at the current checkpoint, wherein the first tensor is defined as having completely consistent content among data copies on the multiple dies; in response to the absence of at least one first tensor, performing a persistent storage operation on all tensors; in response to the existence of at least one first tensor: performing a data consistency check on the at least one first tensor to confirm whether the content of each first tensor's data copies on the multiple dies is completely consistent; in response to any of the at least one first tensor having completely inconsistent content among its data copies on the multiple dies, not performing a persistent storage operation on all tensors; in response to each of the at least one first tensor having completely consistent content among its data copies on the multiple dies, performing a persistent storage operation on all tensors.

[0054] In the data processing method provided in at least one embodiment of this disclosure, after initiating the checkpoint storage operation and before performing the persistent storage operation for that checkpoint, a data consistency check is performed on the first tensor, which requires complete consistency of content among multiple data copies of the bare chips, along the checkpoint storage path. If any inconsistency is found between the bare chips in any of the first tensors, subsequent persistent storage operations are rejected. This allows for the timely detection of potential data fork failures between bare chips before data is stored on disk, preventing faulty data from being stored on disk and causing silent errors to propagate to downstream training or inference, thus intercepting the fault at an early stage.

[0055] For large model training, the cost of retraining after a single failure is extremely high. The data processing method provided in at least one embodiment of this disclosure can effectively avoid the high retraining overhead caused by continuing training based on error checkpoints and subsequent failures, thus promoting the standardization of reliability design for distributed data processing in multi-die architectures.

[0056] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0057] The data processing method provided in at least one embodiment of this disclosure is used in a hardware architecture including multiple die chips. For example, the hardware architecture may include a central processing unit and one or more multiple die chips. The multiple die chips may include an artificial intelligence accelerator or an artificial intelligence chip. The multiple dies may implement the functions of a graphics processing unit or a general-purpose graphics processing unit. For example, the artificial intelligence accelerator or artificial intelligence chip may be scheduled, coordinated, and controlled by the central processing unit.

[0058] For example, the data processing method provided in at least one embodiment of this disclosure can be applied to a central processing unit in a hardware architecture, which is used to schedule, coordinate and control multiple dies to perform distributed data processing using the multiple dies.

[0059] For example, distributed data processing can include distributed training of deep learning. For example, the data processing method provided in at least one embodiment of this disclosure can be used in distributed data processing scenarios such as large language models and hybrid expert models that require long-term training and frequent saving of intermediate states.

[0060] Of course, this disclosure is not limited thereto. The data processing method provided in at least one embodiment of this disclosure can also be applied to distributed data processing of network models with similar functions and structures.

[0061] Figure 3 This is a schematic flowchart illustrating a data processing method provided in at least one embodiment of the present disclosure.

[0062] like Figure 3 As shown, the data processing method includes steps S10-S30. For example, steps S10-S30 of the data processing method provided in at least one embodiment of this disclosure can be specifically executed by a central processing unit in a hardware architecture.

[0063] like Figure 3 As shown, in step S10, in response to initiating the checkpoint storage operation, from all tensors that need to be persistently saved at the current checkpoint, at least one first tensor is filtered and determined.

[0064] The first tensor is defined as ensuring that the contents of the data copies across multiple bare wafers are completely identical.

[0065] This disclosure does not specifically limit the shape and size of the first tensor. The first tensor can be a one-dimensional scalar, a two-dimensional matrix, or a tensor of higher dimensions such as three-dimensional or four-dimensional.

[0066] Here, "being specified as" means, for example, that the semantic requirement of the first tensor is that the content of the data copies across multiple bare chips must be completely consistent, but this is not a semantic constraint that is necessarily true in practice. As mentioned earlier, during training, due to compiler defects, communication primitive anomalies, SBP conversion path errors, hardware soft errors (such as bit flips or uncorrectable ECC errors), or oversights in the implementation of custom operators, the data copies between bare chips may quietly fork. The hardware framework treats the complete consistency of the content between data copies as an established fact at the programming model layer, without runtime verification, but physically it is merely an invariant maintained by communication primitives, a semantic constraint that should be true but is not actually true.

[0067] Complete consistency between data copies means that the tensor dimensions, element values, and data arrangement of the data copies of the first tensor on different bare chips are completely identical.

[0068] For example, when a tensor carries a tensor distribution label at the framework abstraction layer, such as carrying an SBP semantic label (fragment-broadcast-local reduction tensor parallel semantic label), the first tensor can be a tensor carrying a BB semantic label (broadcast-broadcast semantic label).

[0069] Checkpoint storage operations can be save-checkpoint operations. For example, a checkpoint storage operation can be initiated when the main training loop completes N iterations or batches, the training round ends, the task exits, or a save command is received. Checkpoint storage freezes the current global training state, blocks temporary writes, locks all global logical tensors (such as weights, biases, activations, gradients, optimizer states, etc.), and locks context information such as iteration counts, learning rate scheduler states, random number seeds, and data loading cursors.

[0070] All tensors that need to be persistently saved at the current checkpoint can include all global logical tensors locked at the current checkpoint. These global logical tensors need to be written to disk, and the normal training computation pipeline can be directly restored through the data saved at the checkpoint during training continuation.

[0071] For example, in some embodiments, step S10 may include: traversing all tensor distribution labels and selecting tensors with tensor distribution labels as first distribution labels as first tensors, wherein the first distribution label requires that the content of the tensor is completely consistent among the data copies on multiple bare dies.

[0072] For example, the tensor distribution label is a fragmented-broadcast-local reduction tensor parallel semantic label, and the first distribution label is a broadcast-broadcast semantic label.

[0073] For example, all tensors can be traversed, and tensors whose SBP semantic labels are BB semantic labels can be identified as the first tensor. For example, the first tensors can be combined into a tensor set for subsequent processing in step S30.

[0074] For example, traversing all tensor distribution labels and selecting tensors whose tensor distribution label is the first distribution label as the first tensor can include: constructing a state dictionary corresponding to the current checkpoint, wherein the state dictionary includes the tensor distribution labels of all tensors; traversing the state dictionary and, based on the tensor distribution labels of all tensors, selecting tensors whose tensor distribution label is the first distribution label as the first tensor.

[0075] For example, after initiating the checkpoint storage operation, the state dictionary (state_dict) is constructed first. The state dictionary is a structured container that uses a key-value dictionary structure to uniformly store all recoverable state data from deep learning training. It is used to organize the scattered model parameters, optimizer variables, and training runtime information in memory into a data set that can be serialized, persisted, and reloaded at breakpoints.

[0076] The state dictionary includes the storage addresses of all tensors, as well as their attributes such as dimensions, data types, SBP semantic labels, etc. In addition, the state dictionary also includes optimizer states, and metadata information such as random number seeds and iteration steps.

[0077] After constructing the state dictionary, iterate through the state dictionary. If a tensor has a tensor distribution label and the tensor distribution label is the first distribution label, add it as the first tensor to the first tensor set.

[0078] In step S20, in response to the absence of at least one first tensor, all tensors are persistently stored.

[0079] If, among all the tensors that the checkpoint currently needs to persist, there is no first tensor whose content is completely identical among the data copies on multiple raw chips, the subsequent persistence storage operation is performed directly, and the tensor is passed directly to the serializer for subsequent persistence storage operations without introducing additional overhead.

[0080] In this disclosure, a data consistency check is performed on the first tensor because the cost of checking the first tensor is independent of the tensor parallelism scale. The first tensor has only one data copy on each bare chip, and checking only requires reading multiple data copies from the bare chip for comparison, resulting in low complexity and decoupling from the tensor parallelism scale. However, for tensors with other semantic labels, such as split-broadcast semantics (SB semantic labels) tensors, performing an all-gather operation to restore the global view is too costly. Therefore, this disclosure checks the first tensor to avoid expensive global alignment for other types of tensors, minimizing the high retraining overhead.

[0081] For example, in some embodiments, persistent storage operations on all tensors may include: serializing all tensors and related storage information of the current checkpoint; and writing the serialization result obtained from the serialization operation to disk.

[0082] The serializer is the core execution unit for storing distributed tensor training snapshots. For example, in one embodiment, during persistent storage operations, the frozen tensors, along with related storage information such as iteration counts, random number seeds, and optimizer states, are extracted from multiple raw shards into host memory; the tensors are preprocessed based on SBP semantic tags, such as sharding or global aggregation; the tensor values ​​are encoded into binary data streams, and the tensor attributes, SBP semantic tags, optimizer states, and other information are serialized, converting the memory object into a storable byte stream; the serialization result is then written to disk, completing the persistent storage of the training state.

[0083] In step S30, in response to the existence of at least one first tensor: a data consistency check is performed on at least one first tensor; in response to the fact that the contents of the data copies of any of the at least one first tensor on the multiple dies are not completely consistent, no persistent storage operation is performed on all tensors; in response to the fact that for each of the at least one first tensor, the contents of the various data copies of the first tensor on the multiple dies are completely consistent, a persistent storage operation is performed on all tensors.

[0084] After the first tensor is selected, a data consistency check is performed on it. The data consistency check can confirm whether the data copies of the first tensor on multiple bare wafers are completely identical.

[0085] If any of the first tensors fails the data consistency check, meaning that the contents of any of the first tensors are not completely consistent across multiple raw dies, then a fault is considered to have occurred, and subsequent persistent storage operations are not performed to prevent the fault from being written directly to disk.

[0086] Assuming that all first tensors pass the data consistency check, that is, for each first tensor, the contents of each data copy of the first tensor on multiple raw dies are completely consistent, then all tensors are persisted. For details of the persistence operation, please refer to the relevant description of step S20 above, which will not be repeated here.

[0087] In at least one embodiment of this disclosure, after initiating the checkpoint storage operation and before executing the checkpoint persistence storage operation, the system filters and determines whether at least one first tensor exists from all tensors that need to be persistently saved at the current checkpoint, and then performs data consistency verification on the first tensor. That is, in at least one embodiment of this disclosure, the verification is performed before the persistence storage action, rather than as a post-hoc remedy during loading. As mentioned earlier, the saving and loading of checkpoints in a multi-die architecture are asymmetrical: saving only retrieves a data copy from die 0, while loading broadcasts the data copy from die 0 back to die 1. This means that once the data copy of the first tensor has forked during saving, the actual data on die 1 will be unconditionally discarded and unrecoverable the instant it is written to disk. No matter how rigorous the checks performed by the loading end, it will be unable to detect this fork; it can only observe the late-arriving, indirect phenomenon of loss before and after the checkpoint and discontinuity in gradient normalization. Therefore, data consistency verification must be performed before the data copy of die 0 is written out, i.e., within the window after the checkpoint storage operation has been initiated but before the serializer has been triggered.

[0088] For example, in some embodiments, step S30 may include: traversing at least one first tensor and performing a data consistency check on each first tensor to confirm whether the contents of the data copies of each first tensor on the multiple dies are completely consistent.

[0089] For example, in this embodiment, assuming the first tensors constitute a first tensor set, when the first tensor set is not empty, a data consistency check is performed on each first tensor in the first tensor set to confirm whether the contents of the data copies of each first tensor on multiple bare dies are completely consistent. Since the first tensors may be independent of each other, the first tensors with problems may not be related to each other. This method can identify all first tensors with data fork problems through a single traversal, facilitating subsequent location and inspection.

[0090] For example, in some other embodiments, step S30 may include: sequentially performing data consistency checks on at least one first tensor, and stopping data consistency checks on subsequent first tensors in response to any inconsistency in the content of data copies of any first tensor across multiple dies. In this embodiment, since persistent storage operations are stopped when there are inconsistencies in the content of data copies of a first tensor across multiple dies, data consistency checks on the first tensors can be performed sequentially. When a first tensor with inconsistencies in the content of data copies across multiple dies is found, data consistency checks on subsequent first tensors are no longer performed, reducing the overhead of data consistency checks.

[0091] For example, in some embodiments, a data consistency check for a first tensor may include: reading multiple data copies of the first tensor from multiple dies; performing a consistency check operation on the multiple data copies to confirm whether the multiple data copies are completely consistent at the bit level; in response to the multiple data copies being completely consistent at the bit level, determining that the content of the data copies of the first tensor on the multiple dies is completely consistent; and in response to the multiple data copies not satisfying the requirement of being completely consistent at the bit level, determining that the content of the data copies of the first tensor on the multiple dies is not completely consistent.

[0092] For example, in some embodiments, performing a consistency check operation on multiple data copies to confirm whether the multiple data copies are completely consistent at the bit level may include: calculating fingerprints for each of the multiple data copies to obtain a data fingerprint corresponding to each data copy; comparing the data fingerprints of the multiple data copies; and confirming that the multiple data copies are completely consistent at the bit level in response to the data fingerprints of the multiple data copies being completely consistent, otherwise confirming that the multiple data copies are not completely consistent at the bit level.

[0093] For example, fingerprint calculation can be implemented using bitwise hashing, cyclic redundancy check (CRC), bitwise XOR accumulation, etc. For instance, bitwise hashing can be a non-encrypted hash algorithm that uses binary bits as the smallest unit of operation, performing bitwise operations such as shifting, XORing, modulo division, and addition on the first tensor bit by bit, ultimately compressing it into a fixed-length hash value. In this method, even if there is a 1-bit change between the data copies of the first tensor, the output hash value will be completely different. Because the central processing unit natively supports single-cycle instructions such as shifting, XORing, and addition, the bitwise hashing pipeline is almost uninterrupted and very fast. Of course, those skilled in the art can also use other feasible methods for fingerprint calculation, and this disclosure does not impose specific limitations on them.

[0094] For example, for a certain first tensor, data copies of the first tensor are first read from multiple raw dies and cached in the host cache. Then, the data fingerprints of each comparable data copy are calculated. By comparing the data fingerprints, it can be quickly determined whether the data of the data copies of the first tensor are completely identical.

[0095] For example, in other embodiments, performing a consistency check operation on multiple data copies to confirm whether the multiple data copies are completely consistent at the bit level may include: performing a full byte comparison on the multiple data copies; in response to the result of the full byte comparison indicating that the contents of the multiple data copies are completely consistent, confirming that the multiple data copies are completely consistent at the bit level, otherwise confirming that the multiple data copies are not completely consistent at the bit level.

[0096] For example, for some small first tensors, after reading data copies of the first tensor from multiple raw dies and caching them in the host cache, a full byte comparison can be performed directly, and the result of the full byte comparison can be used to determine whether the contents of the data copies are completely consistent.

[0097] For example, in other embodiments, performing a consistency check operation on multiple data replicas to confirm whether the multiple data replicas are completely consistent at the bit level may include: dividing each of the multiple data replicas into multiple data blocks of a fixed size, and calculating a fingerprint for each data block to obtain a data fingerprint for each data block; comparing the data fingerprints of data blocks at corresponding positions in the multiple data replicas; determining that the multiple data replicas are not completely consistent at the bit level in response to any inconsistency in the data fingerprints of any group of data blocks, wherein the data blocks at corresponding positions in the multiple data replicas are considered as a group of data blocks; and determining that the multiple data replicas are completely consistent at the bit level in response to all data fingerprints of data blocks at corresponding positions in the multiple data replicas being completely consistent.

[0098] For example, when some of the first tensors are large, the data copies of the first tensor can be divided into multiple data blocks of fixed size. A fingerprint is calculated for each data block to obtain its data fingerprint. Then, the data fingerprints of corresponding data blocks in the data copies are compared. If the data fingerprints of any group of data blocks are inconsistent, the multiple data copies are considered not to be completely identical at the bit level. If the data fingerprints of all groups of data blocks are consistent, the multiple data copies are considered to be completely identical at the bit level.

[0099] For example, in some embodiments, the data processing method provided in at least one embodiment of this disclosure further includes: in response to the fact that the contents of any one of the first tensors on multiple dies are not completely consistent, generating a difference location report for any one of the first tensors, wherein the difference location report describes the location information of the data range or data element that differs among the multiple data copies of the one tensor.

[0100] For example, in this disclosure, when it is found that the contents of data copies of a first tensor on multiple bare wafers are not completely consistent, a difference location report is generated. The difference location report describes the location information of the data range or data element that differs among the multiple data copies of the first tensor.

[0101] For example, binary segmentation or linear scan block methods can be used to determine the first interval with a difference. For instance, the first tensor can be divided into multiple data blocks, and the data fingerprint of each data block can be calculated. The data fingerprints of the data blocks are compared sequentially using either traversal or binary segmentation. The location information of the first data block with a difference is described in the difference location report and provided to the user, allowing the user to quickly locate the faulty interval.

[0102] The difference location report can also provide information such as the name, shape, layout, tensor distribution label, memory block (rank), and die number of the first tensor, so that users can obtain more detailed information about the first tensor.

[0103] For example, in some embodiments, the data processing method provided in at least one embodiment of this disclosure further includes: in response to the fact that the contents of any one of the first tensors on multiple dies are not completely consistent, a fault indication is given, wherein the fault indication includes indication information of any one of the first tensors.

[0104] For example, a fault message can alert the user that the contents of the first tensors are not completely consistent across multiple dies. For instance, the fault message could include information such as the name, shape, layout, tensor distribution label, memory block (rank), and die number of these first tensors.

[0105] Figure 4 This is a schematic diagram of the processing flow of a data processing method provided in at least one embodiment of the present disclosure.

[0106] The following is combined with Figure 4 This fully describes the overall process of the data processing method provided in at least one embodiment of the present disclosure.

[0107] like Figure 4 As shown, in response to the start checkpoint storage operation, a state dictionary is constructed. Specifically, the data processing method provided in at least one embodiment of this disclosure is executed after the return of the state dictionary (generate_state_dict) within the start checkpoint storage operation and before the persistent storage operation (dist_checkpointing.save / torch.save) is called.

[0108] After constructing the state dictionary, iterate through the state dictionary, scan all tensor distribution labels, and select tensors whose tensor distribution labels are the first distribution labels to form a tensor set. For example, tensor distribution labels are SBP semantic labels, and the first distribution label is, for example, BB semantic labels.

[0109] This disclosure intercepts the control flow at a critical moment when the state dictionary is constructed, performs tensor distribution label filtering first, and only includes the first tensor in the verification set, thus avoiding expensive global alignment of tensors with other tensor distribution labels.

[0110] If the tensor set is empty, it means that the first tensor is not among all the tensors that need to be saved at the current checkpoint. In this case, the persistent storage operation is performed directly on all tensors, which includes first serializing them and then writing the serialization result to disk to complete the checkpoint storage operation. For details, please refer to the relevant description of parameter step S20, which will not be repeated here.

[0111] If the tensor set is not empty, it means that the first tensor exists among all the tensors that need to be saved at the current checkpoint. In this case, the tensor set is traversed and the data consistency check is performed on the first tensor one by one.

[0112] Specifically, for a given first tensor t, multiple data copies of the first tensor t are first collected or read from multiple raw dies into the host memory. Then, a consistency check operation is performed on the multiple data copies. For example, the consistency check operation may include using fingerprint calculation, full byte comparison, or other methods to confirm whether the contents of the multiple data copies are completely consistent, i.e., achieving bit-level complete consistency. The specific process of the consistency check operation can be referred to the relevant description in step S30 above, and will not be repeated here.

[0113] For example, if the consistency check operation of the first tensor t shows that multiple data copies of the first tensor t are completely consistent at the bit level, then the first tensor t passes the data consistency check. Next, it is determined whether the first tensor t is the last first tensor in the tensor set. If yes, it means that all first tensors have passed the data consistency check, and then the original serialization path is allowed to proceed, performing persistent storage operations on all tensors. If not, the above process continues for the next first tensor.

[0114] This method guarantees that at any checkpoint persisted to disk, all of its first sheets are bitwise consistent across multiple raw sheets.

[0115] Furthermore, if the consistency check operation of the first tensor t shows that multiple data copies of the first tensor t are not completely consistent at the bit level, then the first tensor t fails the data consistency check. At this time, a difference location report is generated, and subsequent persistent storage operations are refused, indicating that there is a fault or anomaly, leaving no possibility of writing faulty data directly to the disk.

[0116] For information on how to generate differential localization reports, please refer to the aforementioned content; it will not be repeated here.

[0117] Of course, it should be noted that when the consistency check operation of a first tensor t shows that multiple data copies of the first tensor t are not completely consistent at the bit level, if the tensor set has not been traversed, the above operation can be performed on subsequent first tensors to generate a difference location report containing all first tensors with data fork faults.

[0118] The data processing method provided in at least one embodiment of this disclosure, when all tensors to be saved at the current checkpoint do not include the first tensor, or when the data consistency check of all first tensors passes, only requires one additional filtering and several consistency check operations. The input and output time is negligible compared to a complete checkpoint storage operation, resulting in low overhead. This data processing method only verifies the first tensor, avoiding a full collection operation on, for example, SB semantic tensors, thus decoupling the verification complexity from the parallel scale of tensors.

[0119] The data processing method provided in at least one embodiment of this disclosure introduces "the content of data copies on multiple dies is completely consistent" as an online verifiable semantic constraint into the checkpoint saving path for the first time under a multi-die architecture. This compensates for the blind spots of upstream deep learning frameworks, deep learning framework generation libraries, or dedicated frameworks in multi-die scenarios. Data crossover failures are detected before the first write to disk, avoiding the propagation of silent errors to downstream training or inference. This effectively avoids the high retraining overhead caused by continuing training based on erroneous checkpoints, thus promoting the standardization of reliability design for distributed data processing in multi-die architectures.

[0120] The data processing method provided in at least one embodiment of this disclosure also includes a difference location report. This report describes the location and name information of the first tensor where a data crossover fault exists, enabling direct location of the fault point. Furthermore, this data processing method is inserted after constructing the state dictionary and before serialization, eliminating the need to modify the downstream serializer. It is applicable to all multi-die chip hardware architectures, reducing development costs.

[0121] The data processing method provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 6The illustrated electronic device 300 can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server, etc. For example, the hardware architecture of the electronic device 300 may include a graphics processor. The data processing method provided in at least one embodiment of this disclosure can be applied to scenarios involving CPU, high-performance computing (HPC), and artificial intelligence (AI) in the electronic device 300.

[0122] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 5 A schematic block diagram of a data processing apparatus provided for at least one embodiment of the present disclosure.

[0123] The data processing device is used in a hardware architecture that includes multiple bare chips for distributed data processing.

[0124] For example, the hardware architecture can consist of a network of computing nodes comprising multiple hardware units integrating a central processing unit (CPU) and a graphics processing unit (GPU). The overall physical form is a rack-mount server, supplemented by various computing terminals adapted to different deployment environments. It leverages an internal interconnect architecture to achieve collaborative scheduling of heterogeneous computing power. This hardware architecture can comprehensively utilize the general-purpose computing capabilities of the CPU and the parallel acceleration capabilities of the GPU, adapting to the needs of various high-performance computing applications such as artificial intelligence computing, large-scale numerical simulations, and big data processing.

[0125] For example, the hardware architecture may include a central processing unit and an artificial intelligence chip containing multiple dies.

[0126] For details regarding the hardware architecture of the multi-die chip and the specifics of distributed data processing, please refer to the relevant descriptions in the aforementioned data processing method embodiments, which will not be repeated here.

[0127] like Figure 5 As shown, the data processing device 200 includes a filtering module 201 and a processing module 202.

[0128] The filtering module 201 is configured to, in response to the start of the checkpoint storage operation, filter and determine whether there is at least one first tensor from all tensors that need to be persistently saved at the current checkpoint, wherein the first tensor is defined as having completely identical content among the data copies on multiple bare dies.

[0129] Processing module 202 is configured to: in response to the absence of at least one first tensor, perform persistent storage operation on all tensors; in response to the presence of at least one first tensor, perform a data consistency check on at least one first tensor, wherein the data consistency check includes confirming whether the contents of the data copies of the first tensor on multiple dies are completely consistent; in response to the inconsistency of the contents of the data copies of any one of the at least one first tensor on multiple dies, not perform persistent storage operation on all tensors; in response to the complete consistency of the contents of the data copies of each of the at least one first tensor on multiple dies, perform persistent storage operation on all tensors.

[0130] For example, the filtering module 201 and the processing module 202 include code and programs stored in memory. The filtering module 201 and the processing module 202 are implemented, for example, as a central processing unit (CPU) or other processing units with data processing capabilities and / or instruction execution capabilities. The processing unit can be a general-purpose processor, and can also be a microcontroller, microprocessor, digital signal processor, dedicated image processing chip, or field-programmable logic array, etc. The filtering module 201 and the processing module 202 execute the code and programs to implement some or all of the functions of the filtering module 201 and the processing module 202 as described above. For example, the filtering module 201 and the processing module 202 can be a circuit board or a combination of multiple circuit boards for implementing the functions described above. In the embodiments of this disclosure, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0131] It should be noted that the filtering module 201 can be used to implement Figure 3 The step S10 shown; the processing module 202 can be used to implement Figure 3 Steps S20-S30 are shown. For a detailed description of the functions that the filtering module 201 can perform, please refer to the relevant description of step S10 in the embodiments of the data processing method described above. For a detailed description of the functions that the processing module 202 can perform, please refer to the relevant description of steps S20-S30 in the embodiments of the data processing method described above. Repeated descriptions will not be repeated here. Furthermore, the data processing device 200 can achieve similar technical effects to the aforementioned data processing method, and will not be described further here.

[0132] It should be noted that in at least one embodiment of this disclosure, the data processing device 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0133] For example, the data processing device 200 may be implemented in hardware, software, or a combination of hardware and software, and this disclosure does not impose any specific limitations on it.

[0134] In some embodiments, the data processing apparatus provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). A SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces, etc.

[0135] The heterogeneous computing cluster provided in at least one embodiment of this disclosure can achieve similar technical effects to the aforementioned data processing methods, and the repetitions will not be repeated.

[0136] Figure 6 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. For example... Figure 6 As shown, the electronic device 300 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. It should be noted that... Figure 6 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0137] like Figure 6 As shown, the electronic device 300 may include a processing device 301 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0138] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the data processing method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the data processing method can be found in the relevant descriptions in the embodiments of the above data processing method.

[0139] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0140] For example, the processing device 301, the read-only memory (ROM) 302, and the random access memory (RAM) 303 are interconnected via a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0141] Typically, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, a processing device 301 can control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0142] Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 7As shown, storage medium 400 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 401 can be stored non-transitory. For example, when the computer-readable instructions 401 are executed by a processor, one or more steps in the data processing method described above can be performed.

[0143] For example, the storage medium 400 can be used in an electronic device 300, such as the storage medium 400 including the storage device 308 in the electronic device 300.

[0144] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0145] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0148] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0149] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0150] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0151] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0152] The following points should be noted regarding this disclosure: (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0153] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0154] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, A hardware architecture comprising multiple dies is used for distributed data processing across the multiple dies. The data processing method includes: In response to initiating a checkpoint storage operation, from all tensors that need to be persistently saved at the current checkpoint, filter and determine whether there is at least one first tensor, wherein the first tensor is defined as having completely identical content among the data copies on the plurality of bare dies; In response to the absence of at least one first tensor, a persistent storage operation is performed on all tensors; In response to the existence of at least one of the first tensors: Perform a data consistency check on the at least one first tensor, wherein the data consistency check includes confirming whether the contents of the data copies of the first tensor on the plurality of bare dies are completely consistent; In response to the fact that the contents of the data copies of any one of the at least one first tensors on the plurality of raw dies are not completely consistent, the persistent storage operation on all tensors is not performed; In response to the fact that for each of the at least one first tensor, the contents of the respective data copies of the first tensor on the plurality of raw dies are completely identical, the persistent storage operation is performed on all the tensors.

2. The data processing method according to claim 1, characterized in that, From all tensors that need to be persisted at the current checkpoint, filter and determine if there exists at least one first tensor, including: Iterate through the tensor distribution labels of all tensors, and select the tensor with the tensor distribution label as the first distribution label as the first tensor, wherein the first distribution label requires that the content of the tensor is completely consistent among the data copies of the tensor on the multiple bare dies.

3. The data processing method according to claim 2, characterized in that, Iterate through all the tensor distribution labels of the tensors, and select the tensor whose tensor distribution label is the first distribution label as the first tensor, including: Construct a state dictionary corresponding to the current checkpoint, wherein the state dictionary includes tensor distribution labels for all tensors; Traverse the state dictionary, and based on the tensor distribution labels of all tensors, select the tensors whose tensor distribution labels are the first distribution labels as the first tensor.

4. The data processing method according to claim 2, characterized in that, The tensor distribution label is a fragmented-broadcast-local reduction tensor parallel semantic label, and the first distribution label is a broadcast-broadcast semantic label.

5. The data processing method according to claim 1, characterized in that, Performing a data consistency check on the at least one first tensor includes: The at least one first tensor is traversed, and the data consistency check is performed on each first tensor to confirm whether the contents of the data copies of each first tensor on the plurality of bare dies are completely consistent.

6. The data processing method according to claim 1, characterized in that, Performing a data consistency check on the at least one first tensor includes: The data consistency check is performed sequentially on the at least one first tensor. In response to any first tensor having inconsistencies in the content of its data copies across the plurality of bare dies, the data consistency check on subsequent first tensors is stopped.

7. The data processing method according to claim 1, characterized in that, The data consistency check includes: Read multiple data copies of the first tensor from the plurality of bare dies; A consistency check operation is performed on the multiple data replicas to confirm whether the multiple data replicas are completely consistent at the bit level; In response to the fact that the plurality of data copies are completely identical at the bit level, it is determined that the contents of the data copies of the first tensor on the plurality of bare dies are completely identical. In response to the fact that the plurality of data copies do not satisfy complete consistency at the bit level, it is determined that the contents of the data copies of the first tensor on the plurality of bare dies are not completely consistent.

8. The data processing method according to claim 7, characterized in that, Perform a consistency check operation on the plurality of data replicas to confirm whether the plurality of data replicas are completely consistent at the bit level, including: Fingerprint calculation is performed on each of the multiple data replicas to obtain the data fingerprint corresponding to each data replica; Compare the data fingerprints of the multiple data copies; If the data fingerprints of the plurality of data replicas are completely consistent, it is confirmed that the plurality of data replicas are completely consistent at the bit level; otherwise, it is confirmed that the plurality of data replicas are not completely consistent at the bit level.

9. The data processing method according to claim 7, characterized in that, Perform a consistency check operation on the plurality of data replicas to confirm whether the plurality of data replicas are completely consistent at the bit level, including: Perform a full byte comparison on the multiple data copies; In response to the result of the full byte comparison indicating that the contents of the plurality of data copies are completely consistent, it is confirmed that the plurality of data copies are completely consistent at the bit level; otherwise, it is confirmed that the plurality of data copies are not completely consistent at the bit level.

10. The data processing method according to claim 7, characterized in that, Perform a consistency check operation on the plurality of data replicas to confirm whether the plurality of data replicas are completely consistent at the bit level, including: Each of the multiple data replicas is divided into multiple data blocks of fixed size, and a fingerprint is calculated for each data block to obtain the data fingerprint of each data block; Compare the data fingerprints of corresponding data blocks in the multiple data replicas; In response to the inconsistency of the data fingerprints of any group of data blocks, it is determined that the multiple data replicas are not completely consistent at the bit level, wherein the data blocks at corresponding positions in the multiple data replicas are considered as a group of data blocks; If the data fingerprints of the corresponding data blocks in the plurality of data replicas are completely identical, it is determined that the plurality of data replicas are completely identical at the bit level.

11. The data processing method according to claim 1, characterized in that, Also includes: In response to the fact that the contents of any one of the at least one first tensors on the plurality of bare dies are not completely consistent, a difference location report for the first tensor is generated, wherein the difference location report describes the location information of the data range or data element that differs among the plurality of data copies of the first tensor.

12. The data processing method according to claim 1, characterized in that, Also includes: In response to the fact that the contents of any one of the at least one first tensors are not completely consistent across the data copies of the plurality of bare dies, a fault indication is issued, wherein the fault indication includes indication information of the any one of the first tensors.

13. The data processing method according to claim 1, characterized in that, The multi-die chip includes an artificial intelligence accelerator or an artificial intelligence chip, and the distributed data processing includes deep learning distributed training.

14. A data processing apparatus, characterized in that, A hardware architecture comprising multiple dies is used for distributed data processing. The data processing device includes: The filtering module is configured to, in response to the start checkpoint storage operation, filter and determine whether there is at least one first tensor from all tensors that need to be persistently saved at the current checkpoint, wherein the first tensor is defined as having completely identical content among the data copies on the plurality of bare dies; The processing module is configured as follows: In response to the absence of at least one first tensor, a persistent storage operation is performed on all tensors; In response to the existence of at least one of the first tensors: Perform a data consistency check on the at least one first tensor, wherein the data consistency check includes confirming whether the contents of the data copies of the first tensor on the plurality of bare dies are completely consistent; In response to the fact that the contents of the data copies of any one of the at least one first tensors on the plurality of raw dies are not completely consistent, the persistent storage operation on all tensors is not performed; In response to the fact that for each of the at least one first tensor, the contents of the respective data copies of the first tensor on the plurality of raw dies are completely identical, the persistent storage operation is performed on all the tensors.

15. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to any one of claims 1-13.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they implement the data processing method according to any one of claims 1-13.

Images