Fault handling method for deep learning training and related products

Abstract

The present disclosure provides a deep learning training fault processing method and related products, which relates to the technical field of computer, wherein the deep learning training fault processing method is executed by an electronic device, and the method comprises: in the case that a first process in a first communication group fails, configuration information of a second communication group is determined; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that do not fail; based on the configuration information and information of the first process, output data corresponding to a local process running in the electronic device is determined; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for a neural network to perform deep learning training. The method can avoid interruption of the training task.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a fault handling method and related products for deep learning training. Background Technology

[0002] During deep learning training, when one or more processes in the communication group malfunction, it can easily lead to the failure of all-to-all communication, which in turn forces the entire training task to be interrupted, making it impossible to guarantee the continuity of training. Summary of the Invention

[0003] This disclosure presents a fault handling method and related products for deep learning training.

[0004] In a first aspect, embodiments of this disclosure propose a fault handling method for deep learning training. The method is executed by an electronic device and includes: determining configuration information of a second communication group when a first process in a first communication group fails; wherein the configuration information indicates that the second communication group includes processes in the first communication group that have not failed; determining output data corresponding to a local process running in the electronic device based on the configuration information and information of the first process; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network.

[0005] Secondly, embodiments of this disclosure propose a fault handling device, comprising: an information determination module configured to determine configuration information of a second communication group when a first process in a first communication group fails; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that have not failed; and a data acquisition module configured to determine output data corresponding to a local process running in the fault handling device based on the configuration information and information of the first process; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of a neural network.

[0006] Thirdly, embodiments of this disclosure propose a neural network training system, including: a first electronic device; and a second electronic device configured to carry a first process; wherein the first electronic device is configured to: determine configuration information of a second communication group in the event of a failure of the first process in a first communication group; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that have not failed; and determine output data corresponding to a local process running in the first electronic device based on the configuration information and information of the first process; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network.

[0007] Fourthly, embodiments of this disclosure provide an electronic device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to, when executing instructions stored in the memory, implement a fault location method as described in any implementation of the first aspect.

[0008] Fifthly, embodiments of this disclosure provide a chip including a processor capable of executing a fault location method as described in any implementation of the first aspect.

[0009] In a sixth aspect, embodiments of this disclosure provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, can implement the fault location method as described in any implementation of the first aspect.

[0010] In a seventh aspect, embodiments of this disclosure provide a computer program product including a computer program, which, when executed by a processor, can implement the fault location method as described in any implementation of the first aspect.

[0011] The technical solution provided in this disclosure allows each normal process to determine the configuration information of the second communication group when the first process fails. Based on the configuration information and the information of the first process, it determines the output data that does not contain data blocks from the first process. This effectively avoids the impact of data blocks from the failed process on subsequent communication and training, ensuring that the normal processes in the second communication group can continue to perform communication tasks based on valid data. This, in turn, ensures that the remaining normal processes can continue to advance deep learning training and avoids interruption of the training task.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0013] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0014] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart of a fault handling method provided in an embodiment of this disclosure; Figure 3 A schematic diagram of obtaining a reconstructed communication group provided in an embodiment of this disclosure; Figure 4 A schematic diagram illustrating a fault-free all-to-all communication process provided in an embodiment of this disclosure; Figure 5 A schematic diagram illustrating a full-to-full communication process when a process fails, provided as an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of a fault handling device provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of a neural network training system provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for performing a fault handling method, provided as an embodiment of the present disclosure. Detailed Implementation

[0015] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0016] It should be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0017] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this application are not limited to the order in which they are described, but can be performed in any order or in parallel. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] Figure 1 An exemplary system architecture 100 is shown, illustrating embodiments of fault handling methods, apparatuses, chips, electronic devices, computer-readable storage media, and computer program products for which deep learning training of the present disclosure can be applied.

[0020] like Figure 1 As shown, system architecture 100 may include a first device 101 and a second device 102, which can be connected via a network. The network serves as the medium for providing a communication link between the first device 101 and the second device 102. The network may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0021] The first device 101 and the second device 102 can be either hardware or software. When the first device 101 and the second device 102 are hardware, the first device 101 can be various electronic devices, components on those devices, etc., and the second device 102 can be various electronic devices, components on those devices, etc. When the first device 101 and the second device 102 are software, they can be installed in electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitations are made here.

[0022] For example, a process deployed on the first device 101 can be called a local process, and a process deployed on the second device 102 can be called a first process.

[0023] It should be understood that Figure 1 The number of first and second devices shown is merely illustrative. Any number of first and second devices can be used depending on implementation requirements.

[0024] For example, the implementation of all-to-all communication operators mainly relies on underlying communication libraries, such as Message Passing Interface (MPI) and set communication libraries.

[0025] In some embodiments, the typical flow of all-to-all communication includes: an initialization phase, a data preparation phase, a communication execution phase, and a result aggregation phase.

[0026] Initialization phase: Based on the number of processes preset in the training task, a fixed communication group is built. All processes complete the registration within the group through the communication library interface, and the unique identifier (rank ID) and communication address information of each process are specified.

[0027] Data preparation phase: Each process divides the local input data according to the total number of processes in the communication group and the division rules, and different parts need to be distributed to different cards.

[0028] Communication execution phase: Each process sends the divided data blocks to the corresponding target process through the communication library interface provided by the communication library, and at the same time receives data blocks sent to it by all other processes.

[0029] Result aggregation stage: Each process concatenates all received data blocks in a preset order to form communication output data, which is then transmitted to subsequent local computing stages, such as expert network computing and feature fusion.

[0030] The typical process of all-to-all communication mentioned above may have problems such as fixed communication groups, lack of dynamic adaptation capabilities, high fault recovery costs, and low training efficiency.

[0031] For example, all-to-all communication relies on a fixed communication group built during initialization. Once established, the number of processes and the identities of its members cannot be dynamically adjusted. When any process in the communication group exits due to hardware failure (such as GPU damage or memory overflow) or software exception (such as program crash or communication timeout), the entire communication group becomes unavailable, causing the all-to-all communication task to fail directly.

[0032] For example, after a process failure, the entire training task needs to be terminated, checkpoint data reloaded, and all processes restarted. For large-scale deep learning training tasks, such as training a large language model with billions of parameters (the checkpoint file is huge, usually tens or even hundreds of GB), the loading process is time-consuming, and after restarting, the training steps that were completed before the failure need to be repeated, resulting in a significant reduction in training efficiency and a serious waste of computing resources and time.

[0033] Please refer to Figure 2 , Figure 2 The flowchart of a fault handling method for deep learning training provided in this embodiment of the disclosure is shown. The execution subject is an electronic device, which may be a first device 101 or a first electronic device. The process 200 includes the following steps.

[0034] Step 201: In the event of a failure in the first process of the first communication group, determine the configuration information of the second communication group.

[0035] For example, the first communication group refers to a set of processes pre-built to realize data interaction and collaborative computing between processes during deep learning training, such as distributed deep learning training, such as Mixture of Experts (MoE) training. It contains multiple processes that communicate with each other. Each process performs communication tasks through this communication group, such as all-to-all communication, and jointly completes the relevant processes of deep learning training, such as data fragmentation and transmission, and result aggregation.

[0036] For example, the first process refers to one of the processes included in the first communication group, which is the process that fails and is unable to participate in communication and computing tasks normally in this step. The process failure includes, but is not limited to, situations that prevent the process from executing the preset communication and computing tasks, such as process crash, communication link interruption, abnormal computing power, and no response.

[0037] For example, the second communication group refers to the communication group formed by the remaining normal processes that have not failed after the first process that failed is removed from the first communication group. After the first process fails, it replaces the first communication group to continue to realize the collaborative communication between normal processes and ensure that the deep learning training is not interrupted.

[0038] For example, configuration information is used to indicate that the second communication group includes processes in the first communication group that have not experienced failures.

[0039] Optionally, the configuration information can be used to define a set of parameters related to the attributes of the second communication group, such as the composition, size, process identifier, and other key information of the second communication group.

[0040] In this step, when the first process in the first communication group fails, causing the first communication group to be unable to perform the collaborative communication task normally, each normal process participating in the deep learning training can actively determine the configuration information of the second communication group.

[0041] For example, the configuration information is used to indicate that the second communication group includes the processes in the first communication group that have not failed. That is, the second communication group is a subset of the first communication group. It retains only all the processes in the first communication group that can perform communication and computing tasks normally, completely removes the first processes that have failed, and ensures that all processes in the second communication group are valid processes that can participate normally in subsequent deep learning training-related operations.

[0042] Step 202: Based on the configuration information and the information of the first process, determine the output data corresponding to the local process running in the electronic device.

[0043] For example, the local process belongs to the second communication group, and the output data does not include data blocks from the first process.

[0044] For example, a local process may include a process running in the first device 101 or the first electronic device.

[0045] For example, a local process refers to the process currently executing this fault handling method. This process belongs to the second communication group. That is, the local process is one of the normal processes in the first communication group that has not experienced a fault. It can receive data, perform computing tasks, and communicate collaboratively with other normal processes in the second communication group.

[0046] For example, the information of the first process may refer to a set of parameters used to uniquely identify the first process and / or associate data related to the first process. For instance, the information of the first process may include the identifier of the first process, the communication address of the first process, and the identifier of the data block that the first process is responsible for transmitting in the first communication group. Its core function is to help the local process identify and distinguish data blocks from the first process. The identifier of the first process may be, for example, a process ID, i.e., a process number, used to distinguish different processes within the communication group.

[0047] For example, the output data can be used to perform deep learning training on a neural network.

[0048] For example, output data can refer to the effective data that the local process has filtered and used to input into the subsequent neural network to perform deep learning training. It is the data that ensures the accuracy and continuity of deep learning training, and it is necessary to ensure that no invalid data (such as data blocks from faulty processes) is mixed in.

[0049] For example, neural networks can be various network models used in distributed deep learning training, such as Mixture of Experts (MoE) models, Transformer models, etc.

[0050] In this step, after the local process obtains the configuration information of the second communication group and the information of the first process that has failed, it analyzes and filters these two types of information to determine the output data that meets the requirements of the local process, ensuring that the output data is valid and can support the normal progress of subsequent deep learning training.

[0051] For example, when determining the output data, the local process, based on the information from the first process, identifies all data blocks from the first process that have been received by the local process, and removes all such data blocks. The remaining data blocks then constitute the output data corresponding to the local process. If the first process has failed, the data blocks it transmitted may be incomplete, erroneous, or invalid. If such data is mixed into the output data, it will severely affect the accuracy of subsequent deep learning training and may even cause abnormal training interruptions.

[0052] The method provided in this disclosure, when the first process fails, each normal process determines the configuration information of the second communication group. Based on the configuration information and the information of the first process, it determines the output data that does not contain data blocks from the first process. This can effectively avoid the impact of data blocks from the failed process on subsequent communication and training, ensuring that the normal processes in the second communication group can continue to perform communication tasks based on valid data, thereby ensuring that the remaining normal processes can continue to advance deep learning training and avoid interruption of training tasks.

[0053] In some embodiments, the method provided in this disclosure may further include: step A1.

[0054] Step A1: If the first process meets the first condition, determine that the first process has failed.

[0055] Optionally, the first condition may include at least one of the following: no heartbeat packet from the first process was received during the first period; and the received heartbeat packet from the first process included error information.

[0056] The first condition mentioned above, which includes the case that no heartbeat packet from the first process is received within the first period and the received heartbeat packet from the first process contains error information, can mean that no heartbeat packet from the first process is received within the first period, but the heartbeat packet from the first process is obtained during the critical period of the first period (e.g., the period before the start of the first period), and the obtained heartbeat packet carries error information.

[0057] For example, a heartbeat packet is a signal data used by processes within a communication group to exchange their running status. It is sent periodically by each process according to preset rules and contains the core running information of the process. It is the core carrier for detecting the running status of the process.

[0058] For example, the first cycle is a preset heartbeat packet sending cycle, which can be flexibly configured according to the training scenario.

[0059] For example, if the preset first period is 50ms, a normal process acts as the monitor and starts timing from the last time it receives a heartbeat packet sent by the first process. If the timing reaches 50ms and no heartbeat packet is received from the first process again, it indicates that the first process may be running abnormally.

[0060] For example, error messages include information that indicates the first process is running abnormally, such as resource overflow, program exception, and communication link interruption. If the monitoring party receives a heartbeat packet sent by the first process and finds after parsing that it contains error messages such as "GPU utilization is 100% and lasts for 10ms (resource overflow)" and "program execution error (software exception)," it means that the first process has run abnormally.

[0061] For example, a periodic heartbeat interaction mechanism can be established among all processes in the communication group to achieve real-time synchronization of the running status of each process, with each process sending heartbeat packets to all other processes at a preset period.

[0062] Optionally, the heartbeat packet may include the process ID, current communication stage, and resource usage status.

[0063] For example, a process ID is used to identify the process that sends a heartbeat packet. A process ID can also be called a temporary process identifier.

[0064] For example, the current communication phase can be used to indicate the current full-to-full communication stage of the process sending the heartbeat packet. For instance, the current communication phase can be divided into a communication parameter configuration phase, a data transmission phase, a data reception phase, and a communication result aggregation phase.

[0065] For example, resource usage status can be used to indicate the current hardware resource usage of the process sending the heartbeat packet. Examples include Central Processing Unit (CPU) utilization, Graphics Processing Unit (GPU) utilization, and memory usage. This information can be used to determine whether the process is experiencing operational failure due to abnormal resource usage, such as memory overflow or GPU overload.

[0066] Optionally, each process maintains a heartbeat reception timeout counter to track the time interval between receiving heartbeat packets from each target process in real time. Through timeout checks and error message identification, the faulty process is determined. Upon receiving a heartbeat packet from a target process, each process immediately resets the heartbeat reception timeout counter corresponding to that target process and restarts the timing.

[0067] For example, in addition to timeout checks, each process, upon receiving a heartbeat packet from the target process, parses the information within the packet. If the parsing reveals errors, such as resource overflow, program exception, or communication link interruption, the target process is directly identified as a faulty process, enabling rapid fault identification. After detecting a faulty process, the fault information needs to be promptly synchronized to all normal processes within the communication group. This ensures that all normal processes have a consistent understanding of the current communication group's status, providing a unified basis for subsequent fault handling (such as communication group reconstruction and data filtering).

[0068] Optionally, the aforementioned executing entity may collect relevant fault information of the faulty process after detecting and determining that the process is faulty. The fault information includes the process ID of the faulty process, the fault type, and / or the time of occurrence of the fault.

[0069] Optionally, the information for the first process may include fault information.

[0070] For example, fault types can be categorized into hardware faults (such as GPU damage or CPU failure), software crashes (such as program errors and exits or process termination), and resource overflows (such as insufficient memory or GPU utilization consistently at 100%). The fault occurrence time is the specific moment when the detection node detects the faulty process.

[0071] For example, after collecting the fault information, it can be broadcast to all normal processes in the communication group. Broadcasting ensures that all normal processes receive the fault information simultaneously, avoiding processing chaos caused by information synchronization delays. After receiving the fault information, each normal process can update the locally stored communication group status information, mark the faulty process as unavailable, and cease communication and data interaction with the faulty process, thus preparing for subsequent communication group reconstruction and training tasks.

[0072] The fault detection method in this embodiment can promptly and accurately detect faulty processes within the all-to-all communication group, avoiding problems such as communication blockage, data transmission errors, and training task interruption caused by the failure to detect faulty processes in a timely manner. At the same time, it provides accurate fault information for subsequent fault handling, reduces fault recovery costs, and improves the stability and reliability of distributed deep learning training.

[0073] Optionally, the configuration information can be obtained by: generating the configuration information of the second communication group by the temporary main process in the training system (if the temporary main process has not failed) and broadcasting it to all processes in the first communication group that have not failed; or, after each process that has not failed communicates with each other to confirm the information of the failed process (first process) in the first communication group, they jointly negotiate to generate the configuration information of the second communication group to ensure that the configuration information obtained by each normal process is consistent and accurate.

[0074] In some embodiments, determining the configuration information of the second communication group in step 201 may include receiving the configuration information of the second communication group.

[0075] For example, the configuration information of the second communication group can be determined by a device other than the first device 101. The device other than the first device 101 sends the configuration information of the second communication group to the first device 101, and correspondingly, the first device 101 receives the configuration information of the second communication group from the device other than the first device 101.

[0076] In some embodiments, determining the configuration information of the second communication group in step 201 may include steps B1-B2.

[0077] Step B1: Based on the information of the first process and the processes included in the first communication group, create a second communication group.

[0078] Step B2: Assign new temporary identifiers to each process in the second communication group to obtain the configuration information of the second communication group.

[0079] Optionally, the configuration information may include: the new temporary identifier and the original temporary identifier of each process in the second communication group.

[0080] Optionally, the new temporary identifier is the identifier of the process within the second communication group.

[0081] Optionally, the original temporary identifier is the identifier of the process within the first communication group.

[0082] For example, after determining the fault of the first process, the relevant operations for rebuilding the communication group are performed. From the processes included in the first communication group, the first process in the faulty state is removed, and all remaining normal processes are retained. The retained normal processes are then used to create a second communication group. After the second communication group is established, a unique identity is assigned to each normal process within the group.

[0083] For example, the new temporary identifier is a unique number used to distinguish different processes within the second communication group, adapting to the communication scheduling requirements of the rebuilt communication group.

[0084] For example, the original temporary identifier is the original identity number assigned to each process within the first communication group, used to retain the process's identity association information within the original communication group.

[0085] In this way, the communication group can be dynamically rebuilt and the process identifiers can be adapted and allocated after a process failure. Configuration information containing the new temporary identifiers and the original temporary identifiers of each process can be generated, ensuring that normal processes in the second communication group can work together to carry out subsequent communication and training tasks, and avoiding training interruption.

[0086] In some embodiments, step B1, based on the information of the first process and the processes included in the first communication group, creates a second communication group, which may include steps B11-B13.

[0087] Step B11: Determine the process set based on the information of the first process and the processes included in the first communication group.

[0088] Step B12: Obtain the communication information of each process in the process set.

[0089] Step B13: Based on the communication information of each process in the process set, create a second communication group through the communication library interface.

[0090] Optionally, the process set includes processes that have not failed in the first communication group.

[0091] Optionally, the communication information may include at least one of the following: communication address and process identifier; For example, the process set refers to the set of processes selected from the first communication group that are in normal operation, used to rebuild the second communication group. The number of processes in the set is M, which is equal to the original number of processes N in the first communication group minus the number of faulty processes K.

[0092] For example, a communication address can be used to locate the hardware node or network location where a process resides. For instance, a communication address may include an IP address, port number, etc.

[0093] For example, a process identifier is an identifier used to uniquely distinguish different processes, and it does not have to be a temporary identifier.

[0094] For example, all processes that have not experienced a failure in the first communication group are selected to form a process set.

[0095] For example, the process set contains M processes, where M equals the original number of processes N in the first communication group minus the number of faulty processes K, combined with... Figure 3 If a communication group originally contains 4 processes, one of which is the faulty first process, then the process set contains 3 normal processes, M=3.

[0096] Combination Figure 3 The original communication group contained four normal processes: process 0, process 1, process 2, and process 3. Process 1 failed, so the failed process was removed, and only the normal processes 0, 2, and 3 were retained to form the rebuilt communication group.

[0097] In this embodiment of the disclosure, the communication group may also be referred to as a process group.

[0098] For example, steps B11-B13 can be executed by a temporary main process, or all normal processes can independently complete the filtering operation to ensure that the process sets obtained by each normal process are completely consistent.

[0099] For example, the acquisition of communication information is performed by a temporary master process, which is a normal process selected from the process set. The selection rules can be preset, for example, selecting the process with the smallest process ID in the process set as the temporary master process.

[0100] For example, the temporary master process sends a message retrieval request to all other normal processes in the process set. After receiving the request, each normal process sends its own communication information to the temporary master process.

[0101] Optionally, after the temporary master process has collected the communication information of all normal processes in the process set, it organizes and verifies this communication information to ensure that the communication address and process identifier of each normal process are accurate and to avoid communication group creation failure due to information errors.

[0102] Optionally, the temporary main process calls a preset communication library interface, which is a standardized interface provided by the distributed communication library. For example, the communication library interface may include MPI communication domain creation function (MPI_Comm_create), communication group rank initialization function, etc., to implement the initialization and creation of communication groups.

[0103] Optionally, when calling the communication library interface, the temporary main process passes the organized communication information of all normal processes to the communication library interface. Based on this communication information, the communication library interface associates all normal processes and creates a brand new communication group, namely the second communication group.

[0104] Optionally, while creating the second communication group, the temporary master process assigns a new temporary identifier to each normal process in the second communication group. The allocation rule for the new temporary identifier can be preset. For example, it can be incremented sequentially from 0 to M-1, forming a one-to-one mapping relationship with the original temporary identifier of each normal process, which facilitates subsequent process identity matching and data interaction.

[0105] In this way, by filtering normal processes based on the fault information of the first process to form a process set, obtaining the communication information of each process, and creating a second communication group through the communication library interface, the communication group can be dynamically rebuilt in the fault scenario, which can ensure the continuous progress of subsequent communication and training tasks.

[0106] In some embodiments, the method provided in this disclosure may further include: step C1.

[0107] Step C1: Send configuration information to the device corresponding to the process other than the local process in the second communication group.

[0108] For example, the local process is a temporary main process that completes the creation of the second communication group, and this process belongs to the second communication group.

[0109] For example, processes other than local processes in the second communication group can be referred to as other processes.

[0110] For example, the temporary main process organizes and generates the configuration information corresponding to the second communication group.

[0111] Optionally, the configuration information includes, but is not limited to, one or more of the following: the original temporary identifier of the normal process, the new temporary identifier, the communication group size M, etc.

[0112] For example, the local process sends the configuration information of the second communication group to the devices corresponding to all normal processes in the second communication group except itself. After receiving the configuration information through their corresponding devices, each normal process in the second communication group updates its local communication group configuration, thus completing the establishment of the new communication group.

[0113] In this way, the unified communication group configuration can ensure the orderly execution of subsequent all-to-all communication tasks and ensure that the communication scheduling of each normal process remains consistent.

[0114] In some embodiments, step 202, based on configuration information and information of the first process, determines the output data corresponding to the local process running in the electronic device, and may include steps D1-D2.

[0115] Step D1: Based on the information from the first process, delete the data blocks from the first process in the received data blocks to obtain the data blocks that have not been deleted.

[0116] Step D2: Based on the new temporary identifiers of each process in the configuration information, concatenate the data blocks that have not been deleted to determine the output data.

[0117] For example, the received data block is the data block that the local process completed receiving during the communication execution phase. The data block from the first process is the data unit sent by the first process in a fault state before the communication was interrupted.

[0118] For example, information from the first process can be used to distinguish data blocks belonging to the first process among the received data blocks.

[0119] For example, each normal process, based on the information from the first process, traverses the received data blocks and removes the data blocks belonging to the first process. After the removal operation is completed, the remaining data blocks are the ones that were not deleted. In this way, precise trimming of the data corresponding to the faulty process can be achieved, removing invalid data.

[0120] For example, each normal process performs a concatenation operation on the data blocks that have not been deleted, according to the order of the new temporary identifiers. The complete data formed after concatenation is the output data, which can be directly adapted to the communication requirements of the reconstructed second communication group. This data processing method can reduce system resource overhead, avoid resource waste caused by redundant backups, and the processed output data can be directly used in subsequent training processes, improving the efficiency and scalability of distributed training.

[0121] In this way, invalid data associated with faulty processes can be removed, and the processed data can be adapted to the rebuilt communication group, thus ensuring the integrity and validity of communication data.

[0122] In some embodiments, step 202, based on configuration information and information of the first process, determines the output data corresponding to the local process running in the electronic device, and may include steps E1-E4.

[0123] Step E1: Determine the new input data based on the information from the first process and the original input data from the local process.

[0124] Step E2: Based on the configuration information, send the second data block from the new input data to the target process of the second data block respectively.

[0125] Step E3: Receive data blocks from processes other than the local process in the second communication group.

[0126] Step E4: Based on the new temporary identifiers of each process in the configuration information, the obtained data blocks are concatenated to determine the output data.

[0127] For example, the new input data does not include: the first data block whose target process is the first process.

[0128] For example, the target process of the second data block belongs to the second communication group.

[0129] For example, the original input data is the data to be sent that the local process initially acquires when performing a communication task in the first communication group.

[0130] For example, the new input data is valid data that the local process has filtered to perform all-to-all communication within the second communication group.

[0131] For example, the local process combines the information of the first process to filter the original input data. After the filtering is completed, the remaining data to be sent is the new input data. The new input data does not include the first data block whose target process is the first process.

[0132] For example, the second data block is a data unit that has been split from new input data and distributed to different target processes. The target process is the process that receives the second data block, and the target process belongs to the second communication group.

[0133] Optionally, the local process first configures all-to-all (all-to-all) communication parameters based on the size M of the second communication group. For example, communication parameters may include, but are not limited to: data block size, send buffer address, receive buffer address, and / or communication timeout. The data block size can be determined by dividing the length of the new input data by M, and the communication timeout can be the first cycle. Optionally, the configuration information may also include communication parameters.

[0134] For example, the local process splits the new input data into multiple second data blocks, and then determines the target process corresponding to each second data block based on the configuration information. Some second data blocks are retained locally. The local process adapts the all-to-all interface of the new communication group through the communication library and sends each second data block to the corresponding target process.

[0135] For example, while sending the second data block, the local process initiates a data receiving process, and the local process receives data blocks sent by all other normal processes in the second communication group through the same all-to-all interface.

[0136] Optionally, the local process can temporarily store the received data blocks in the receive buffer to ensure that the data blocks are intact and not lost.

[0137] Optionally, during the receiving process, the local process monitors the communication timeout period to ensure the stable execution of the receiving process.

[0138] For example, the obtained data blocks include all data blocks received by the local process from other normal processes.

[0139] For example, the local process extracts new temporary identifiers for each process from the configuration information, determines the concatenation order, arranges all data blocks from other normal processes in order according to the new temporary identifiers, performs a concatenation operation on all the obtained data blocks, and the complete data formed after concatenation is the output data. This output data is adapted to the input requirements of subsequent local expert calculations, providing effective data support for subsequent training steps.

[0140] Optionally, steps E1-E4 can be applied to scenarios where the fault occurs before communication begins, such as when the first process has been identified as a faulty process before full-to-full communication has started. This can ensure that the rebuilt second communication group can complete full-to-full communication normally.

[0141] In some embodiments, steps E1-E4 and steps D1-D2 can be used in combination or individually.

[0142] For example, the combined use of steps D1-D2 and E1-E4 can be applied to scenarios where the failure occurs during the execution of all-to-all communication, i.e., when all-to-all communication has started and the local process has received some data blocks from other processes, but not yet all data transmission and reception are complete, and the first process fails. The core purpose of this combined solution is to comprehensively process the data in the failure scenario, both by removing invalid received data and standardizing the transmission and reception of unsent data, ensuring that the final output data is complete and valid, adapting to the communication and training needs of the second communication group, and ensuring the continuous progress of distributed deep learning training.

[0143] For example, steps D1-D2 process the data blocks already received by the local process, removing invalid data from the faulty process. Steps E1-E4 process the original input data that the local process has not yet sent, filtering it before sending it to the normal process and receiving feedback data. Finally, the two parts of processed valid data are integrated to obtain output data suitable for subsequent training. These two processes complement each other, comprehensively covering the full data processing needs during communication execution, avoiding the mixing of invalid data, and ensuring no data is missed.

[0144] Step 1: Execute steps D1-D2 to process the received invalid data.

[0145] For example, when a fault occurs, all-to-all communication has been initiated, and the local process has received data blocks from processes outside the second communication group (including some normal processes and the first process). The local process, combining information from the first process, iterates through all received data blocks, identifies those belonging to the first process, and deletes them, retaining only data blocks from normal processes within the second communication group—the data blocks that were not deleted. This step quickly removes invalid received data, preventing data blocks from faulty processes from affecting subsequent data processing and training accuracy.

[0146] For example, the local process extracts new temporary identifiers for each process in the second communication group from the configuration information. Following the ordered arrangement of these new temporary identifiers, the undeleted data blocks obtained in step D1 are temporarily concatenated to form temporary valid data. This temporary valid data is the valid portion of the received data and is temporarily stored locally for later integration with the data processed in steps E1-E4.

[0147] Step 2: Execute steps E1-E4 to process the unsent original input data and complete the communication.

[0148] For example, the local process still has unsent original input data, which initially contains data blocks destined for all processes (including the first process) within the first communication group. The local process, combining information from the first process, filters the original input data, removing the first data block whose target process is the first process. The remaining data blocks are then integrated to form new input data. The new input data only contains data destined for normal processes within the second communication group, ensuring that all unsent data is valid.

[0149] Based on the configuration information of the second communication group and its size M, the local process configures the all-to-all communication parameters. The local process splits the new input data into multiple second data blocks, determines the target process corresponding to each second data block according to the configuration information, and ensures that all target processes belong to the second communication group. Subsequently, the local process, by adapting to the all-to-all communication interface of the second communication group, sends the second data blocks to the corresponding target processes.

[0150] While sending the second data block, the local process initiates a data reception process, receiving data blocks sent by all other normal processes within the second communication group through the same all-to-all communication interface. The local process temporarily stores the received data blocks in a receive buffer to ensure that each received data block is complete, not lost, and originates from a normal process within the second communication group, with no invalid data mixed in.

[0151] The local process extracts new temporary identifiers for each process from the configuration information. Following the ordered arrangement of these new temporary identifiers, it performs a concatenation operation on all data blocks received in step E3, resulting in valid communication data. This valid communication data is the effective portion of the unsent data after transmission and reception, and together with the temporary valid data obtained in step D2, constitutes the full set of valid data.

[0152] Step 3: Integrate the two sets of valid data to determine the final output data.

[0153] For example, the local process extracts new temporary identifiers for each process from the configuration information and determines a unified concatenation order. The temporary valid data obtained in step D2 (the received valid data) and the valid data obtained in step E4 are integrated and concatenated according to the order of the new temporary identifiers to form the final output data.

[0154] For example, the final output data includes both the normal data received before the failure occurred and the normal data that has been filtered, sent and received after the failure occurred, and there is no invalid data from the first process. It is fully adapted to the scale and communication requirements of the second communication group and can be directly input into the local expert network to perform subsequent deep learning training computation tasks.

[0155] In some embodiments, the configuration information may also include a new output buffer; the method provided in this disclosure may also include: creating a new output buffer based on the total number of processes in the configuration information.

[0156] Optionally, the size of the new output buffer can be matched to the total number of processes.

[0157] For example, an output buffer can also be called a receive buffer.

[0158] For example, the new output buffer is a dedicated storage space for storing the output data after subsequent processing. It is specifically adapted to the communication and data processing needs of the second communication group and is used to temporarily store or save output_new data and various types of valid output data.

[0159] For example, the total number of processes refers to the number of all normal processes within the second communication group.

[0160] For example, the total number of processes contained in the configuration information is obtained, and a new output buffer is created based on the obtained total number of processes. The size of the buffer is set according to the total number of processes, which can just fit the data storage needs of all normal processes sending or receiving data. It will not cause data overflow due to the buffer being too small, nor will it cause memory waste due to the buffer being too large.

[0161] When a fault occurs before communication begins, that is, before all-to-all communication has started and before any normal processes have started sending or receiving any data blocks, the output buffer initialization operation is performed. This can also be called the data structure for output data. For example, the output buffer is initialized based on the total number of processes, without reserving any storage space related to the first process or loading any data related to the first process.

[0162] This initialization method can directly bypass the configuration of faulty processes, avoid invalid data occupying memory resources, and ensure that the data structure of the output data after initialization can accurately adapt to the scale of the second communication group.

[0163] Optionally, after data pruning is complete, data consistency verification can be performed. Data pruning here includes various data processing operations such as initializing the output data structure, deleting and concatenating received data blocks.

[0164] For example, each normal process exchanges pruned verification information with each other through a new communication group. The exchanged verification information includes, but is not limited to, the number of data blocks, data length, and checksum. Each normal process sends its own pruned verification information to all or some of the other normal processes in the second communication group, while simultaneously receiving verification information from other normal processes. Each normal process compares its own verification information with the received verification information from other normal processes. This comparison confirms that the data structures of the new input and output data of all normal processes are completely consistent, and that parameters such as the number of data blocks, data length, and checksum are all the same. This ensures that data integrity meets the requirements of full-to-full communication and avoids communication failures or training errors caused by abnormal data from individual processes.

[0165] If the verification information is found to be inconsistent, each normal process can re-execute the data pruning or initialization operation until the verification information of all normal processes is consistent, thus ensuring the stability of subsequent communication and training.

[0166] In some embodiments, step E1, based on the information of the first process and the original input data of the local process, determines new input data, which may include steps F1-F2.

[0167] Step F1: Based on the information from the first process, delete the first data block from the original input data to obtain the retained data block.

[0168] Step F2: Based on the new temporary identifiers of each process in the configuration information, concatenate the reserved data blocks to determine the new input data.

[0169] For example, the original input data includes N data blocks, which correspond to N processes in the first communication group, with each data block corresponding to one process in the original communication group.

[0170] For example, the retained data blocks are all the data blocks remaining after the original input data has been pruned and the first data block has been removed. The target processes of the retained data blocks are all normal processes that have not experienced any failures.

[0171] For example, the local process processes the original input data, where N data blocks correspond to N processes within the original communication group, including the first data block corresponding to the first process. Based on the information of the first process, the local process identifies the first data block corresponding to the first process from the N data blocks of the original input data, deletes the identified first data block, and retains only the data blocks in the original input data corresponding to the processes that have not experienced failures.

[0172] After the deletion operation is completed, the remaining data blocks are the reserved data blocks, and the number of reserved data blocks is consistent with the size M of the second communication group. Based on the new temporary identifiers of each process in the configuration information, the reserved data blocks are concatenated to determine the new input data. The data formed after concatenation is the new input data, and the length of the new input data is M / N times the length of the original input data.

[0173] For example, the new input data can be split into multiple second data blocks, each second data block corresponds to a target process, and all target processes of the second data blocks are fault-free and belong to the second communication group.

[0174] Optionally, each normal process's corresponding device executes the same steps F1 to F2 to ensure that the new input data obtained by all normal processes has a consistent structure and is complete, meeting the requirements of full-to-full communication.

[0175] Thus, this embodiment can remove only the data blocks of the corresponding faulty processes without requiring redundant backups of the full dataset. This significantly reduces system resource overhead while ensuring the fault tolerance of distributed training, preventing invalid data from consuming memory and computing resources, and effectively improving the efficiency and scalability of distributed training.

[0176] In some embodiments, the method provided in this disclosure may further include step G1.

[0177] Step G1: Perform deep learning training based on the output data using a neural network.

[0178] For example, after the second communication group completes full-to-full communication, each normal process receives its corresponding output data. The output data does not contain any invalid data from the faulty process, and the data structure and length meet the subsequent calculation requirements of full-to-full communication, providing a data foundation for seamless integration.

[0179] For example, after acquiring the output data, each normal process directly transmits the output data to its corresponding local expert network. There is no need to restart the local expert network, reinitialize the computation parameters, or reload the training data, achieving seamless integration of communication and computation, uninterrupting the training process, and ensuring the continuity of training after fault recovery. The neural network can be called a local expert network or an expert network.

[0180] Optionally, after receiving the output data, the local expert network synchronously executes preset computational tasks, including feature transformation and gradient calculation.

[0181] For example, during the computation, the operating parameters of the local expert network remain consistent with those before the failure, ensuring the continuity of the computational logic and not affecting the accuracy and convergence of model training.

[0182] For example, throughout the entire process, the training flow remains uninterrupted, the computational logic remains unchanged, and the model parameters are not lost. After the fault is recovered, the training progress of each normal process remains consistent with that before the fault occurred. There is no need for redundant backups of computation results, which reduces system resource overhead and avoids resource waste while ensuring fault tolerance.

[0183] In some embodiments, step G1 may include steps G11-G14, in which deep learning training is performed on the neural network based on the output data.

[0184] Step G11: Perform learning-type calculations on the output data using a neural network to obtain the third data.

[0185] Step G12: Based on the third data, perform full-to-full communication with all processes in the second communication group other than the local process through the communication library interface to obtain the data after communication.

[0186] Step G13: Concatenate the first data block from the original input data of the local process with the data after communication to generate the fifth data.

[0187] Step G14: Perform deep learning training iteratively using the neural network based on the fifth data.

[0188] Optionally, the first data block can be a data block whose target process is the first process.

[0189] For example, learning-based computation is used to extract patterns from output data and generate computational results, which may include feature transformation, gradient calculation, etc.

[0190] For example, the local process performs learning-type calculations on the output data through a neural network to obtain third data, namely the sharded results generated by local expert calculations, which are used for subsequent cross-process aggregation.

[0191] For example, iterative deep learning training refers to the neural network continuously performing learning-related calculations, model parameter updates, loss calculations, or the next round of communication tasks based on the fifth data, gradually improving model performance and realizing a complete closed loop of "communication-computation-aggregation".

[0192] For example, a local expert network performs learning-type computations on the output data, which may specifically perform feature transformations and optionally gradient computations.

[0193] For example, when performing feature transformation, the local expert network performs in-depth processing on each feature in the output data, transforming shallow features into more expressive deep features and uncovering patterns in the data.

[0194] For example, if gradient calculation is performed, the local expert network calculates the gradient information of the model parameters based on the result of feature transformation, in order to prepare for subsequent global parameter updates.

[0195] Based on the third data, full-to-full communication is established with all processes in the second communication group (excluding the local process) via the communication library interface to obtain the communicated data, enabling cross-process aggregation of the third data from each process. The first data block from the original input data of the local process is concatenated with the communicated data to generate the fifth data, which integrates unprocessed valid data, avoids data loss, and ensures training accuracy. The local process then feeds the fifth data into the neural network, initiating the iterative process of deep learning training.

[0196] In this way, the iterative training process continues to run, and each normal process continues to complete collaborative communication and aggregation of calculation results through the second communication group. The model parameters are gradually optimized to ensure the continuity of the training process after the fault is recovered, without affecting the accuracy and convergence of the model training.

[0197] In some embodiments, step G12, based on third data, performs full-to-full communication with other processes in the second communication group (excluding the local process) through the communication library interface to obtain the communicated data, and may include steps G121-G122.

[0198] Step G121: Send the computed data blocks in the third data to the source process of the computed data blocks respectively to obtain the retained computed data blocks.

[0199] Step G122: Receive computation data blocks from other processes.

[0200] For example, other processes are those in the second communication group other than the local process.

[0201] Optionally, the source process can be a process in the second communication group.

[0202] Optionally, the communicated data may include: retained computational data blocks and computational data blocks from other processes.

[0203] For example, a computational data block is a data block obtained by splitting third-party data, corresponding to a slice of the local expert's computation result. For instance, by performing learning-type computations on data block 1 in the output data through a neural network, computational data block 1 is obtained. If data block 1 comes from process a, then the source process of the computational data block is process a.

[0204] For example, the reserved computational data block is the remaining unsent computational data block 0 after the local process sends computational data blocks 1 to 3 to the corresponding source processes.

[0205] In this way, the local process, by adapting to the communication library interface of the second communication group, sends each computational data block to the corresponding source process, and receives computational data blocks sent by all other normal processes in the second communication group through the same communication library interface. The data after communication includes the computational data blocks retained by the local process and the computational data blocks received from other processes, which can provide complete and effective data for subsequent training processes.

[0206] In some embodiments, step G13 concatenates the first data block in the original input data of the local process with the data after communication to generate the fifth data, which may include steps G131-G132.

[0207] Step G131: Combine the reserved computational data blocks with computational data blocks from other processes to generate the fourth data.

[0208] Step G132: Integrate the first data block from the original input data of the local process into the fourth data in the initial order to generate the fifth data.

[0209] For example, the retained computational data blocks and computational data blocks from other processes can be concatenated according to preset aggregation rules. These aggregation rules may include feature dimension concatenation, gradient-weighted summation, and output result fusion. The data obtained after concatenation is the fourth data, which is the intermediate data after the local computation results of each process are fragmented and aggregated.

[0210] For example, the first data block is the data source block corresponding to the first process in the original input data. This data block is valid data that has not been processed. If it is discarded directly, a large amount of token data will be lost, which will affect the training accuracy.

[0211] For example, the local process extracts the first data block from the original input data and integrates it into the fourth data block according to its initial order in the original input data. The data generated after integration is the fifth data block. The fifth data block contains both the calculation results of the aggregation of each process and the valid data that has not been processed, which can ensure the integrity of the training data and can be directly used for subsequent iterative training to improve training accuracy.

[0212] The following is combined Figure 4 Describe the process of full-to-full communication without any failures.

[0213] For example, during the initialization phase, a fixed communication group is constructed based on a preset number of processes. All processes register within the group, specifying their own unique process ID and the communication addresses of other processes. For example, Figure 4 It contains four processes: process 0, process 1, process 2, and process 3. Each process has its own process ID and communication address with other processes.

[0214] After initialization, the data preparation phase begins. Each process divides its local input data into N data blocks according to the total number of processes in the communication group (N) using the MOE routing mechanism, with each block corresponding to a target process. For example, Figure 4 The local token of each process is split into corresponding data blocks according to the target process. For example, the local token of process 0 is split into four data blocks corresponding to process 0, process 1, process 2 and process 3, and each data block corresponds to a target process.

[0215] After data preparation is complete, the communication execution phase begins. Through the communication library interface, each process sends its split data blocks to its corresponding target process, while simultaneously receiving data blocks sent to it by all other processes. For example, Figure 4 In the process, through all-to-all dispatch, each process sends the split data blocks to the corresponding target process, while receiving data blocks corresponding to itself from other processes. For example, process 1 receives corresponding data blocks from process 0, process 2, and process 3.

[0216] After communication is complete, the local computation phase begins. Each process passes the received data block to the local expert network (MLP) for computation, resulting in a processed data block. For example, Figure 4 Each process in the process will pass the received data block into the local MLP, perform feature transformation and other calculations, and generate processed data blocks. For example, the data block received by process 0 will be processed by the MLP to get 0-0', 1-0', 2-0', and 3-0'.

[0217] After local computation is complete, the results aggregation stage begins. Through an all-to-all combine operation, each process concatenates the processed data blocks in a preset order, restoring the original word order to form output data, which is then fed into the subsequent training process. For example, Figure 4 Each process performs a full-to-full aggregation operation to concatenate the processed data blocks in the original word order, restoring the initial word arrangement. For example, process 0 concatenates 0-0', 1-0', 2-0', and 3-0' in the original order to form output data suitable for subsequent training.

[0218] The following is combined Figure 5 Describe the all-to-all communication process when a process fails.

[0219] Combination Figure 5 Just as communication was about to begin, process 1 crashed due to a GPU malfunction and stopped sending heartbeat packets. The other three processes (process 0, process 2, and process 3) did not receive a heartbeat packet from process 1 after a 50ms timeout, and process 1 was determined to be a faulty process.

[0220] For example, in combination Figure 5 Process 1 is marked with a cross as a faulty process, and the corresponding term data block will not participate in subsequent communication and calculation.

[0221] All normal processes (process 0, process 2, process 3) form a set of normal processes S, with a set size M=3.

[0222] Process 0 acts as a temporary main process, collecting the IP addresses and port numbers of all processes in S, calling the communication library interface to create a new communication group, and assigning new temporary process IDs to each normal process: original process 0 → new process 0, original process 2 → new process 1, original process 3 → new process 2.

[0223] Process 0 synchronizes the member mapping of the new communication group to all normal processes, and each process updates its local communication configuration.

[0224] For example, in combination Figure 5 Subsequent communication and computation are all performed within a new communication group consisting of process 0, process 2, and process 3, and no longer include faulty processes, such as process 1.

[0225] For example, each normal process removes the data block corresponding to the faulty process 1 from the original input data, retains 3 data blocks (corresponding to normal processes), and reorders them according to the new temporary process ID to form new input data.

[0226] For example, the input data of the original process 0 includes 4 blocks that are sent to processes 0 to 3. After trimming, 3 blocks are retained that are sent to the new processes 0 to 2 (i.e., the original process 0, process 2, and process 3).

[0227] For example, Figure 5 The new input data is the new input data formed after trimming each normal process. For example, the new input data of process 0 removes the data block corresponding to process 1 and only retains the three data blocks corresponding to process 0, process 2 and process 3.

[0228] For example, since the fault occurs before the communication begins, each process directly initializes the data structure of the new output data based on M=3, and sets the length of the output buffer to the length of the actual received data.

[0229] Each process exchanges the data length and checksum of the new input data to confirm that the data structure of the new input data of all processes is consistent and that there is no missing or redundant data.

[0230] For example, each process configures all communication parameters of the communication library, sets the send buffer to associate with new input data, and the receive buffer to associate with new output data.

[0231] The communication is executed by calling the all-to-all interface. Each process sends three data blocks of the new input data to the three normal processes in the new communication group, and at the same time receives the data blocks sent by these three processes.

[0232] After communication is completed, each process will concatenate the received data blocks according to the new temporary process ID to form the communication result.

[0233] Each process feeds new output data into the local expert network, which then performs calculations on the new output data.

[0234] For example, Figure 5 Each process in the process will pass the new output data into the local MLP (expert network). For example, the expert in process 0 will complete the calculation of the data blocks passed from process 0, process 2 and process 3, and generate the processed data blocks.

[0235] After the local expert has processed all the data blocks sent by all other processes, it needs to return the processed data that should belong to other processes to the corresponding process.

[0236] Each process originally has 4 blocks of data. The data block assigned to the abnormal process 1 cannot be directly discarded. Instead, the data block corresponding to the original process 1 is inserted into the new output data and returned as the final result, without affecting any subsequent calculation process.

[0237] For example, such as Figure 5 As shown, each process re-inserts the data block corresponding to process 1 in the original input data (such as the 0-1 data block of process 1 in process 0) into the new output data, restoring the final result containing 4 data blocks, ensuring that no data is lost and that the subsequent calculation process is not affected.

[0238] The method provided in this disclosure dynamically reconstructs communication groups based on the remaining normal processes, without terminating the training task. It achieves seamless reconstruction of communication groups by filtering normal processes, reallocating process IDs, and initializing new communication groups.

[0239] The method provided in this disclosure removes input data blocks and / or output data blocks related to faulty processes based on a faulty process list, ensuring that the pruned data is adapted to the new communication group size and guaranteeing the validity of communication data.

[0240] The method provided in this disclosure allows the all-to-all communication results of a new communication group to be directly transmitted to a local expert network. By dynamically adjusting the calculation parameters, the continuity of the training process after a fault can be achieved.

[0241] The method provided in this disclosure does not directly discard data blocks sent to abnormal processes in each process, but integrates them into the final output data according to their original positions, which can minimize the impact on training.

[0242] The method provided in this disclosure does not require terminating the training task when a process failure occurs. By dynamically rebuilding the communication group, pruning data, and re-executing communication, seamless recovery after a failure is achieved, ensuring that the training process continues. This solves the problem that existing checkpoint mechanisms require restarting training.

[0243] The method provided in this disclosure does not require loading a large checkpoint file, nor does it require repeating the training steps before the failure. The failure recovery time is only the time spent on communication group reconstruction and data pruning, which is usually in the millisecond range. This greatly reduces the recovery cost and improves the efficiency of large-scale deep learning training.

[0244] The method provided in this disclosure achieves fault tolerance by dynamically adapting to normal processes. It does not require setting backup nodes for processes, thus avoiding the waste of CPU, GPU, memory and other resources caused by redundant backups. It can improve the utilization of system resources and is applicable to large-scale distributed training scenarios.

[0245] The method provided in this disclosure ensures the integrity and validity of communication data by removing the input and output data corresponding to the faulty process. Furthermore, the local expert calculation module can dynamically adjust the calculation parameters to avoid data deviation caused by faults, thus ensuring the accuracy and convergence of model training.

[0246] The method provided in this disclosure does not rely on a specific communication library or deep learning framework. It can be adapted to mainstream communication libraries such as MPI, ensemble communication library, and Gloo, and supports common deep learning frameworks such as TensorFlow and PyTorch. It can also handle scenarios with single or multiple process failures. As the number of processes increases, its fault tolerance overhead grows slowly, and its scalability is excellent.

[0247] The method provided in this disclosure can detect process failures in real time during all-to-all communication and dynamically rebuild the communication group, ensuring that the remaining normal processes can continue to complete the communication task and avoiding the overall interruption of the training task.

[0248] The method provided in this disclosure can solve the problems of communication failure, training interruption, high recovery cost, and large resource overhead that exist in the all-to-all communication operator when process failure occurs.

[0249] The method provided in this disclosure, in large-scale distributed training involving all-to-all communication operators, when one or more processes fail and exit training, the all-to-all communication operator will not report an error and exit, but will ignore the data of the erroneous process and continue training without stopping.

[0250] The method provided in this disclosure does not terminate the training task when a process failure is detected within the communication group. Instead, it dynamically reconstructs the all-to-all communication group based on the remaining normal processes. Simultaneously, it trims and splices the input and output data, removing the data source and target data portions corresponding to the failed process. After completing all-to-all communication on the reconstructed communication group, the communication results are transmitted to a local expert network for calculation, achieving seamless recovery after the failure. Finally, the processed data is added back to the original data.

[0251] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a fault handling device, which corresponds to the above method embodiment and can be specifically applied to various electronic devices.

[0252] like Figure 6 As shown, the fault handling device 600 of this embodiment may include an information determination module 601 and a data acquisition module 602. The information determination module 601 is configured to determine configuration information of a second communication group when a first process in the first communication group fails; wherein the configuration information indicates that the second communication group includes processes in the first communication group that have not failed. The data acquisition module 602 is configured to determine the output data corresponding to a local process running in the fault handling device 600 based on the configuration information and the information of the first process; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network.

[0253] In this embodiment, the specific processing of the information determination module 601 and the data acquisition module 602 in the fault handling device 600 and the resulting technical effects can be referred to the relevant descriptions of the above method embodiments, and will not be repeated here.

[0254] In some optional implementations of the embodiments of this disclosure, the fault handling device 600 further includes a process fault detection module. The process fault detection module is configured to: determine that the first process has failed if the first process meets a first condition; wherein the first condition includes at least one of the following: no heartbeat packet from the first process is received within a first period; and the received heartbeat packet from the first process includes error information.

[0255] In some optional implementations of the embodiments of this disclosure, the information determination module 601 includes a communication group dynamic reconstruction module. The communication group dynamic reconstruction module is configured to: create a second communication group based on the information of the first process and the processes included in the first communication group; assign new temporary identifiers to each process in the second communication group to obtain configuration information of the second communication group; wherein the configuration information includes: new temporary identifiers and original temporary identifiers for each process in the second communication group. The new temporary identifier is the identifier of the process in the second communication group, and the original temporary identifier is the identifier of the process in the first communication group.

[0256] In some optional implementations of the embodiments of this disclosure, the communication group dynamic reconstruction module is further configured to: determine a process set based on the information of the first process and the processes included in the first communication group; wherein the process set includes processes in the first communication group that have not experienced a failure; obtain the communication information of each process in the process set; wherein the communication information includes at least one of the following: communication address and process identifier; and create a second communication group through the communication library interface based on the communication information of each process in the process set.

[0257] In some optional implementations of the embodiments of this disclosure, the fault handling device 600 further includes: a sending module; the sending module is configured to send configuration information to the device corresponding to a process other than the local process in the second communication group.

[0258] In some optional implementations of the embodiments of this disclosure, the data acquisition module 602 includes: an input / output data trimming module; the input / output data trimming module is configured to: delete data blocks from the first process in the received data blocks based on the information of the first process to obtain data blocks that have not been deleted; and splice the data blocks that have not been deleted based on the new temporary identifiers of each process in the configuration information to determine the output data.

[0259] In some optional implementations of the embodiments of this disclosure, the data acquisition module 602 includes an input / output data trimming module and a communication module; the input / output data trimming module is configured to: determine new input data based on the information of the first process and the original input data of the local process; wherein the new input data does not include: a first data block whose target process is the first process; the communication module is configured to: send the second data block in the new input data to the target process of the second data block respectively based on configuration information; wherein the target process of the second data block belongs to the second communication group; receive data blocks from other processes in the second communication group other than the local process; and concatenate the obtained data blocks based on the new temporary identifier of each process in the configuration information to determine the output data.

[0260] In some optional implementations of the embodiments of this disclosure, the configuration information further includes a new output buffer; the input / output data pruning module is further configured to: create a new output buffer based on the total number of processes in the configuration information; wherein the size of the new output buffer matches the total number of processes.

[0261] In some optional implementations of the embodiments of this disclosure, the input / output data trimming module is further configured to: delete the first data block in the original input data based on the information of the first process to obtain the retained data block; wherein the target process of the second data block has not failed; and splice the retained data block based on the new temporary identifier of each process in the configuration information to determine the new input data.

[0262] In some optional implementations of the embodiments of this disclosure, the fault handling device 600 further includes: a local expert computing module; the local expert computing module is configured to: perform deep learning training based on output data through a neural network.

[0263] In some optional implementations of the embodiments of this disclosure, the local expert computing module is further configured to: perform learning-type computation on the output data through a neural network to obtain third data; based on the third data, perform full-to-full communication with other processes in the second communication group (excluding the local process) through a communication library interface to obtain communicated data; concatenate the first data block in the original input data of the local process with the communicated data to generate fifth data; and iteratively perform deep learning training based on the fifth data through a neural network. Here, the first data block is a data block whose target process is the first process.

[0264] In some optional implementations of the embodiments of this disclosure, the local expert computing module is further configured to: send the computing data blocks in the third data to the source process of the computing data blocks respectively to obtain the retained computing data blocks; the source process is a process in the second communication group; receive computing data blocks from other processes; wherein, the data after communication includes: the retained computing data blocks and computing data blocks from other processes.

[0265] In some optional implementations of the embodiments of this disclosure, the local expert computing module is further configured to: concatenate the reserved computing data block and the computing data block from other processes to generate fourth data; and integrate the first data block from the original input data of the local process into the fourth data in the initial order to generate fifth data.

[0266] This embodiment exists as a device embodiment corresponding to the above method embodiment. The technical effects brought about by the fault handling device 600 provided in this embodiment can be referred to the relevant descriptions in the above method embodiment, and will not be repeated here.

[0267] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a neural network training system, which corresponds to the above method embodiments and can be applied to various electronic devices.

[0268] like Figure 7 As shown, the neural network training system 700 of this embodiment may include: a first electronic device 701 and a second electronic device 702. The first electronic device 701 is configured to: determine configuration information of a second communication group in the event of a failure of a first process in a first communication group; wherein the configuration information indicates that the second communication group includes processes in the first communication group that have not failed; and determine the output data corresponding to a local process running in the first electronic device 701 based on the configuration information and information of the first process; wherein the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network. The second electronic device 702 is configured to carry the first process.

[0269] In this embodiment, the specific processing of the first electronic device 701 and the second electronic device 702 in the neural network training system 700 and the resulting technical effects can be referred to the relevant descriptions of the above method embodiments, and will not be repeated here.

[0270] This disclosure also provides a chip including a processor that can be used to implement any of the possible implementations of the methods provided in this disclosure.

[0271] This disclosure also proposes an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing instructions stored in the memory.

[0272] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the described method. The computer-readable storage medium can be volatile or non-volatile.

[0273] This disclosure also provides a computer program product that, when executed by a processor, can implement the methods described in any of the above embodiments.

[0274] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0275] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 8 As shown, the electronic device 800 of this embodiment includes a processor 801 and may further include a memory 802; wherein, the memory 802 is used to store computer execution instructions; the processor 801 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. The electronic device 800 may be a first electronic device or a second electronic device.

[0276] Alternatively, the memory 802 can be either standalone or integrated with the processor 801.

[0277] When the memory 802 is set up independently, the electronic device also includes a bus 803 for connecting the memory 802 and the processor 801.

[0278] It should be understood that the processor 801 described above can be a Central Processing Unit (CPU), or other general-purpose processors, GPUs, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0279] The memory 802 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0280] Bus 803 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0281] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0282] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0283] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0284] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0285] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0286] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0287] According to the technical solution of this disclosure embodiment, when the first process fails, each normal process determines the configuration information of the second communication group. Based on the configuration information and the information of the first process, it determines the output data that does not contain data blocks from the first process. This can effectively avoid the impact of the data blocks of the failed process on subsequent communication and training, ensuring that the normal processes in the second communication group can continue to perform communication tasks based on valid data, thereby ensuring that the remaining normal processes can continue to advance deep learning training and avoid interruption of training tasks.

[0288] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0289] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A fault handling method for deep learning training, the method being executed by an electronic device, the method comprising: In the event of a failure in the first process of the first communication group, configuration information for the second communication group is determined; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that have not failed. Based on the configuration information and the information of the first process, the output data corresponding to the local process running in the electronic device is determined; wherein, the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network; the data for deep learning training of the neural network includes fifth data; the fifth data is obtained by concatenating the data blocks of the first process in the original input data of the local process with the cross-process aggregated data of each process in the second communication group obtained according to the output data.

2. The method according to claim 1, wherein, The method further includes: If the first process satisfies a first condition, it is determined that the first process has failed; wherein the first condition includes at least one of the following: No heartbeat packet was received from the first process during the first cycle; and The heartbeat packet received by the first process includes error information.

3. The method according to claim 1, wherein, Determine the configuration information for the second communication group, including: Based on the information of the first process and the processes included in the first communication group, a second communication group is created; The configuration information of the second communication group is obtained by allocating configuration information to each process in the second communication group; wherein, the configuration information includes: a new temporary identifier and an original temporary identifier for each process in the second communication group, wherein the new temporary identifier is the identifier of the process in the second communication group, and the original temporary identifier is the identifier of the process in the first communication group.

4. The method according to claim 3, wherein, Creating the second communication group based on the information of the first process and the processes included in the first communication group includes: Based on the information of the first process and the processes included in the first communication group, a process set is determined; wherein, the process set includes processes in the first communication group that have not experienced failures; Obtain the communication information of each process in the process set; wherein the communication information includes at least one of the following: communication address and process identifier; Based on the communication information of each process in the process set, the second communication group is created through the communication library interface.

5. The method according to claim 3 or 4, wherein, The method further includes: The configuration information is sent to the device corresponding to the process other than the local process in the second communication group.

6. The method according to any one of claims 1-3, wherein, The step of determining the output data corresponding to the local process running in the electronic device based on the configuration information and the information of the first process includes: Based on the information from the first process, delete the data blocks from the first process in the received data blocks to obtain the data blocks that have not been deleted; Based on the new temporary identifiers of each process in the configuration information, the data blocks that have not been deleted are concatenated to determine the output data.

7. The method according to any one of claims 1-3, wherein, The step of determining the output data corresponding to the local process running in the electronic device based on the configuration information and the information of the first process includes: Based on the information of the first process and the original input data of the local process, new input data is determined; wherein, the new input data does not include: the first data block whose target process is the first process; Based on the configuration information, the second data block in the new input data is sent to the target process of the second data block respectively; wherein, the target process of the second data block belongs to the second communication group; Receive data blocks from processes other than the local process in the second communication group; Based on the new temporary identifiers of each process in the configuration information, the obtained data blocks are concatenated to determine the output data.

8. The method according to claim 7, wherein, The configuration information also includes a new output buffer; the method further includes: Based on the total number of processes in the configuration information, a new output buffer is created; wherein the size of the new output buffer matches the total number of processes.

9. The method according to claim 7, wherein, The step of determining new input data based on the information of the first process and the original input data of the local process includes: Based on the information from the first process, the first data block in the original input data is deleted to obtain the retained data block; Based on the new temporary identifiers of each process in the configuration information, the reserved data blocks are concatenated to determine the new input data.

10. The method according to any one of claims 1-3, wherein, The method further includes: The neural network performs deep learning training based on the output data.

11. The method according to claim 10, wherein, The step of performing deep learning training based on the output data through the neural network includes: The third data is obtained by performing learning-type calculations on the output data through the neural network. Based on the third data, full-to-full communication is performed with other processes in the second communication group, excluding the local process, through the communication library interface to obtain the data after communication. The first data block in the original input data of the local process and the data after communication are concatenated to generate the fifth data; wherein, the first data block is a data block whose target process is the first process; The neural network is used to iteratively perform deep learning training based on the fifth data.

12. The method according to claim 11, wherein, Based on the third data, full-to-full communication is performed with all processes in the second communication group other than the local process through the communication library interface to obtain the communicated data, including: The computed data blocks in the third data are sent to the source process of the computed data blocks respectively to obtain the retained computed data blocks; the source process is a process in the second communication group; Receive computational data blocks from the other processes; wherein the data after communication includes: the reserved computational data blocks and the computational data blocks from the other processes.

13. The method according to claim 12, wherein, The step of concatenating the first data block from the original input data of the local process with the data after communication to generate the fifth data includes: The reserved computational data block and the computational data block from the other processes are concatenated to generate the fourth data; The first data block from the original input data of the local process is integrated into the fourth data in the initial order to generate the fifth data.

14. A fault handling device, comprising: The information determination module is configured to determine the configuration information of a second communication group in the event that a first process in a first communication group fails; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that have not failed. The data acquisition module is configured to determine the output data corresponding to the local process running in the fault handling device based on the configuration information and the information of the first process; wherein, the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network; the data for deep learning training of the neural network includes fifth data; the fifth data is obtained by concatenating the data blocks of the first process in the original input data of the local process with the cross-process aggregated data of each process in the second communication group obtained according to the output data.

15. A neural network training system, comprising: First electronic device; as well as The second electronic device is configured to carry the first process; The first electronic device is configured as follows: In the event of a failure in the first process of the first communication group, configuration information for the second communication group is determined; wherein the configuration information is used to indicate that the second communication group includes processes in the first communication group that have not failed. Based on the configuration information and the information of the first process, the output data corresponding to the local process running in the first electronic device is determined; wherein, the local process belongs to the second communication group, the output data does not include data blocks from the first process, and the output data is used for deep learning training of the neural network; the data for deep learning training of the neural network includes fifth data; the fifth data is obtained by concatenating the data blocks of the first process in the original input data of the local process with the cross-process aggregated data of each process in the second communication group obtained according to the output data.

16. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 13 when executing instructions stored in the memory.

17. A chip comprising a processor for performing the method of any one of claims 1 to 13.

18. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 13.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images