Abnormality processing method, direct memory access node (HDMA) and DCU cluster system

Abstract

The application relates to an exception processing method, a direct memory access node HDMA and a DCU cluster system. The method comprises the following steps: performing exception detection on an in-network computing task according to response information obtained in each execution stage of the in-network computing task; determining an error type identifier corresponding to an exception event in a target execution stage of the in-network computing task; generating an exception interrupt according to the error type identifier, a node identifier corresponding to the direct memory access node HDMA and a task identifier corresponding to the in-network computing task; and reporting the exception interrupt to a predefined target depth calculator DCU, so that the target depth calculator DCU processes the exception event according to the exception interrupt, the DCU can perform directional repair or reconfiguration based on the exception interrupt, the whole system does not need to be restarted, and the error processing cost is significantly reduced.

Description

Technical Field

[0001] This application relates to the field of on-network computing technology, and in particular to an exception handling method, a direct memory access node (HDMA), and a DCU cluster system. Background Technology

[0002] With the continuous expansion of high-performance computing and distributed training in artificial intelligence, the communication overhead between nodes within a cluster has become a key bottleneck restricting the linear speedup of the system. In-network computing technology has emerged to address this need, aiming to utilize network devices (such as network interface cards and switches) to perform online computation on data during data transmission, thereby reducing communication latency and improving overall computing efficiency.

[0003] In related technologies, error handling for on-net computing tasks typically relies on a Deep Computing Unit (DCU) for centralized management, which means that when an anomaly is detected in an on-net computing task, the DCU system is reset.

[0004] However, in large-scale DCU cluster systems, this method of handling errors by overall reset requires a lot of computing resources, and the cost of error handling is high. Summary of the Invention

[0005] Therefore, it is necessary to provide an exception handling method, a Direct Memory Access Node (HDMA), and a DCU cluster system to address the aforementioned technical problems.

[0006] Firstly, this application provides an exception handling method applied to a Direct Memory Access Node (HDMA), comprising:

[0007] Based on the response information obtained from each execution stage of the online computing task, anomaly detection is performed on the online computing task.

[0008] If an abnormal event is detected during the target execution phase of an on-network computing task, the error type identifier corresponding to the abnormal event is determined; the target execution phase can be any phase among the various execution phases.

[0009] An exception interrupt is generated based on the error type identifier, the node identifier corresponding to the HDMA direct memory access node, and the task identifier corresponding to the on-network computing task.

[0010] An abnormal interrupt is reported to a predefined target depth calculator (DCU) so that the DCU can handle the abnormal event based on the abnormal interrupt.

[0011] In one embodiment, the target execution phase is a task splitting phase; the method further includes, in the event that an abnormal event is detected in the target execution phase of the on-network computing task:

[0012] Once the response information for all subtasks in the task splitting phase is obtained, the task flow of the online computing tasks is drained.

[0013] In one embodiment, an exception interrupt is generated based on the error type identifier, the node identifier corresponding to the Direct Memory Access Node (HDMA), and the task identifier corresponding to the on-network computing task, including:

[0014] Determine the target error type identifier based on the priority of the error type identifiers corresponding to each subtask;

[0015] An exception interrupt is generated based on the target error type identifier, node identifier, and task identifier.

[0016] In one embodiment, the target error type identifier is determined based on the priority of the error type identifiers corresponding to each subtask, including:

[0017] The error type identifier corresponding to the highest priority is determined as the target error type identifier.

[0018] In one embodiment, the abnormal event includes reading the abnormal event; and performing anomaly detection on the online computing task based on the response information obtained from each execution stage corresponding to the online computing task, including:

[0019] If the response information obtained during the circular buffer read phase is an error response, it is determined that a read exception event has occurred during the circular buffer read phase.

[0020] In one embodiment, the abnormal event includes responding to the abnormal event; based on the response information obtained from each execution stage corresponding to the online computing task, anomaly detection is performed on the online computing task, including:

[0021] If the response information obtained during the dependency check phase is an erroneous response, it is determined that an abnormal response event occurred during the dependency check phase.

[0022] In one embodiment, the abnormal events include comparison failure events; based on the response information obtained from each execution stage corresponding to the online computing task, anomaly detection is performed on the online computing task, including:

[0023] If the response information obtained during the dependency check phase is correct and the dependency check times out, it is determined that a comparison failure event occurred during the dependency check phase.

[0024] In one embodiment, the abnormal events include fatal abnormal events; based on the response information obtained from each execution stage corresponding to the online computing task, anomaly detection is performed on the online computing task, including:

[0025] If the response information obtained during the task splitting phase is a fatal error response, it is determined that a fatal exception event occurred during the task splitting phase.

[0026] In one embodiment, when there are multiple links between the Direct Memory Access Node (HDMA) and the target depth calculator (DCU), the method further includes:

[0027] When the target detects an abnormal event during the target execution phase of the on-net computing task, the target execution phase is re-executed through other links between the Direct Memory Access Node (HDMA) and the Target Depth Calculator (DCU); other links are those other than the link where the abnormal event occurred.

[0028] Secondly, this application also provides a Direct Memory Access Node (HDMA) for performing the steps of any of the methods in the first aspect.

[0029] Thirdly, this application also provides a DCU cluster system, including the Direct Memory Access Node (HDMA), the Target Depth Calculator (DCU), the first agent node, and the second agent node as described in the second aspect.

[0030] The aforementioned anomaly handling method, Direct Memory Access Node (HDMA), and DCU cluster system perform anomaly detection on the online computing task based on the response information obtained from each execution stage. When an anomaly is detected in the target execution stage of the online computing task, the error type identifier corresponding to the anomaly is determined. The target execution stage can be any stage among the execution stages. An anomaly interrupt is generated based on the error type identifier, the node identifier corresponding to the HDMA, and the task identifier corresponding to the online computing task. The anomaly interrupt is reported to a predefined target depth calculator (DCU), which then processes the anomaly event based on the interrupt. This allows the HDMA to autonomously perform anomaly detection and error type identification, and report the error information to the designated DCU in the form of an interrupt. This eliminates the need for the DCU to participate in the error discovery and initial processing, avoiding large-scale computing resource waste and service interruption caused by a system reset. Furthermore, because the HDMA can identify specific error types and carry node and task identifiers, the DCU can perform targeted repair or reconfiguration based on refined anomaly information without restarting the entire system, significantly reducing the cost of error handling. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a diagram illustrating the application environment of an exception handling method in one embodiment.

[0033] Figure 2 This is a flowchart illustrating an exception handling method in one embodiment;

[0034] Figure 3 This is a flowchart illustrating an exception interruption method in one embodiment;

[0035] Figure 4 This is a schematic diagram of the structure of a multi-level Switch interconnection system in one embodiment;

[0036] Figure 5 This is a schematic diagram of the HDMA workflow in one embodiment;

[0037] Figure 6 This is a schematic diagram of an error handling process in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0040] With the continuous expansion of high-performance computing and distributed training in artificial intelligence, the communication overhead between nodes within a cluster has become a key bottleneck restricting the linear speedup of the system. In-network computing technology has emerged to address this need, aiming to utilize network devices (such as network interface cards and switches) to perform online computation on data during data transmission, thereby reducing communication latency and improving overall computing efficiency.

[0041] In related technologies, error handling for on-net computing tasks typically relies on a Deep Computing Unit (DCU) for centralized management, which means that when an anomaly is detected in an on-net computing task, the DCU system is reset.

[0042] However, in large-scale DCU cluster systems, this method of handling errors by overall reset requires a lot of computing resources, and the cost of error handling is high.

[0043] The exception handling method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown. Figure 1 This is a schematic diagram of the structure of a DCU cluster system provided in an embodiment of this application. Figure 1 As shown, the DCU cluster system includes Hyswitch Direct Memory Access (HDMA) nodes, a Depth Computer (DCU), a first proxy node, a second proxy node, on-network compute nodes (SA), and a full-switching network (CrossBar, XBAR). The first proxy node can be a remote proxy (RP), and the second proxy node can be a local proxy (LP). XBAR is responsible for data exchange and routing between various ports; HDMA is the only interface between the Hyswitch chip and the software, responsible for distributing large-scale and ultra-large-scale on-net computing tasks and handling errors; RP receives requests from HDMA, executes specific on-net computing content according to the request type, and returns responses to HDMA. If multicast reads or writes are involved, RP merges multiple responses for the subtask and returns the highest priority error type to HDMA for further processing; LP sends the received on-net computing tasks to the corresponding DCU of the port, and returns the responses to the RP that received the responses; SA handles requests involving broadcast writes in on-net computing and initiates write requests to its own DCU according to the communication domain division.

[0044] For example, the workflow of a DCU cluster system is as follows: The HDMA registers are configured in software. Based on the software register configuration, the HDMA initiates a RingBuf read operation to its near-end DCU. The RP forwards the RingBuf read request initiated by the HDMA to the LP, which then stores it in the near-end DCU. The near-end DCU returns the response corresponding to the RingBuf read request to the LP, which then forwards it back to the RP. The RP passes the response corresponding to the RingBuf read request to the HDMA, which performs packet format parsing to obtain the specific task content.

[0045] HDMA determines whether to initiate a local dependency check based on the task configuration information. RP forwards the dependency check request to LP, which then writes it to the local DCU. The last operation performed by the software in the previous task execution writes a predefined value (e.g., 1) to the address read in the dependency check. The local DCU returns the written address value to LP, which then forwards it to RP. RP returns the response to the dependency check to HDMA. HDMA compares the data in the response with a pre-defined reference value. If the comparison is successful, it considers the local address space required by the current task to meet the task execution requirements and prepares to begin task execution.

[0046] HDMA performs task splitting, sends an All Reduce sub-request to the RP, and informs the RP of the current task's communication domain division. The RP replicates the All Reduce request and sends it to the LPs corresponding to all DCUs within that communication domain, according to the communication domain division. Each DCU returns a response to its LP, which then returns it to the source RP. The RP uses these responses to perform an on-network computation and merges the Status carried in the Response according to the error priority agreed upon with the software. If an error exists in the on-network computation, the RP also fills in the final Response Status.

[0047] HDMA receives and records the final execution result of each subtask. If execution is successful, it continues; otherwise, it stops and enters the business Ras processing flow.

[0048] In one exemplary embodiment, such as Figure 2 As shown, an exception handling method is provided, which can be applied to... Figure 1 Taking HDMA as an example, the explanation includes the following steps:

[0049] S101, based on the response information obtained from each execution stage of the online computing task, perform anomaly detection on the online computing task.

[0050] For example, an on-network computing task refers to a task that performs online data computation simultaneously during data transmission through network devices such as switches, such as an All-Reduce task, a Reduce task, or a Multicast task. The execution phases include a RingBuf reading phase, a Dependency Check phase, and a Task Splitting phase.

[0051] In this embodiment, the response information refers to the response data packets received by HDMA from the DCU or agent node at each stage of performing on-network computing tasks. The response data packets carry response status information to indicate whether the current operation has been successfully executed or whether an abnormality has occurred.

[0052] Optionally, when performing on-network computing tasks, HDMA proceeds with the task flow according to preset execution stages. In each execution stage, HDMA sends a request to the corresponding DCU or agent node and receives the returned response information. HDMA analyzes each response information in real time to determine whether there are any abnormal events in the current execution stage.

[0053] S102, if an abnormal event is detected in the target execution phase of the online computing task, determine the error type identifier corresponding to the abnormal event and drain the task flow of the online computing task.

[0054] The target execution phase is any execution phase in which an abnormal event occurs.

[0055] In this embodiment, the abnormal events include at least one of the following: a read abnormal event in the RingBuf read phase, a response abnormal event in the Dependency Check phase, a comparison failure event in the Dependency Check phase, and a fatal abnormal event in the task splitting phase.

[0056] For example, the error type identifier is used to uniquely indicate the specific error type of an abnormal event. HDMA can pre-agree with the software on the encoding corresponding to various error types and use the error type encoding as the error type identifier.

[0057] In this embodiment, task flow emptying refers to HDMA stopping the continued delivery of currently online computing tasks, reclaiming all responses corresponding to requests that have been sent downstream but are still in transit, and clearing all cached resources and status information related to the current task within HDMA, so as to avoid residual data from abnormal tasks interfering with subsequent normal tasks.

[0058] Optionally, HDMA can look up the corresponding error type identifier in a preset error type mapping table based on the type of the detected abnormal event. The error type mapping table stores the correspondence between abnormal event types and error type identifiers.

[0059] S103 generates an exception interrupt based on the error type identifier, the node identifier corresponding to the direct memory access node (HDMA), and the task identifier corresponding to the on-network computing task.

[0060] In this embodiment, the node identifier is the encoded information used to uniquely identify the current HDMA in the DCU cluster system, and the task identifier is the encoded information used to uniquely identify the current on-network computing task within the HDMA.

[0061] Optionally, HDMA will assemble the determined error type identifier, its own node identifier in the DCU cluster system, and the task identifier of the current on-network computing task into a packet according to a preset interrupt message format to generate an abnormal interrupt message.

[0062] S104, report the abnormal interruption to the predefined target depth calculator DCU, so that the target depth calculator DCU can handle the abnormal event according to the abnormal interruption.

[0063] In this embodiment, the target DCU is a pre-designated DCU in the DCU cluster system used to uniformly handle abnormal events of on-network computing tasks. After receiving an abnormal interruption, the target DCU parses the error type identifier, node identifier, and task identifier carried in the abnormal interruption, locates the specific HDMA and specific task where the abnormality occurred based on this information, and performs the corresponding abnormality repair operation.

[0064] Optionally, the HDMA sends the generated abnormal interrupt to the target DCU based on the target DCU address information pre-configured in the software. Upon receiving the abnormal interrupt, the target DCU reads the fields in the interrupt, determines the specific error type based on the error type identifier, locates the HDMA node where the abnormality occurred based on the node identifier, and locates the on-network computing task where the abnormality occurred based on the task identifier. For non-fatal errors, the target DCU can record the error information and process it uniformly after the task execution is completed, or simply notify maintenance personnel for investigation, without affecting the continued execution of the current task. For link-related errors, the target DCU can configure the HDMA to initiate an automatic retransmission mechanism. For fatal errors, the target DCU can report complete error information to higher-level system management software, which then locates the specific fault source and performs targeted repair or reconfiguration of the HDMA.

[0065] It should be noted that non-fatal errors usually occur during the task splitting phase, while errors occurring during the RingBuf reading phase and the Dependency Check phase are all fatal errors. Therefore, it is only necessary to perform a fatality check on the abnormal events that occur during the task splitting phase, that is, to determine whether the abnormal events that occur during the task splitting phase are fatal errors.

[0066] In this embodiment, anomaly detection is performed on the online computing task based on the response information obtained from each execution stage corresponding to the online computing task. If an anomaly event is detected in the target execution stage of the online computing task, the error type identifier corresponding to the anomaly event is determined. The target execution stage can be any stage among the execution stages. An abnormal interrupt is generated based on the error type identifier, the node identifier corresponding to the Direct Memory Access Node (HDMA), and the task identifier corresponding to the online computing task. The abnormal interrupt is reported to a predefined target depth calculator (DCU), which processes the abnormal event based on the abnormal interrupt. This allows the HDMA to autonomously complete anomaly detection and error type identification, and report the error information to the designated DCU in the form of an interrupt. This eliminates the need for the DCU to participate in error discovery and initial processing, avoiding large-scale waste of computing resources and service interruption caused by a system reset. Furthermore, since the HDMA can identify specific error types and carry node and task identifiers, the DCU can perform targeted repair or reconfiguration based on refined anomaly information without restarting the entire system, thus significantly reducing the cost of error handling.

[0067] Based on the above embodiments, if the target execution stage is a task splitting stage, and an abnormal event is detected in the target execution stage of the on-network computing task, the method further includes the following steps:

[0068] Once the response information for all subtasks in the task splitting phase is obtained, the task flow of the online computing tasks is drained.

[0069] In this embodiment, a subtask refers to multiple independent computing requests generated by HDMA after splitting the on-net computing task according to the task configuration information. Each subtask corresponds to the operation of one DCU or a group of DCUs in the communication domain. The task splitting phase refers to the stage where HDMA, after completing the ring buffer read and dependency check, splits the complete on-net computing task into multiple subtasks and distributes them to each proxy node. The response information refers to the execution result response returned by each DCU to HDMA through the proxy node after each subtask is executed.

[0070] Optionally, after HDMA sends multiple subtask requests to each agent node during the task splitting phase, it can continue to monitor the response return status of each subtask. When HDMA detects an abnormal event in the target execution phase, it does not immediately stop receiving responses, but continues to receive responses until it obtains the response information corresponding to all sent subtasks.

[0071] For example, HDMA can use a counter to record the total number of subtasks sent and the number of subtasks that have returned responses. When the number of returned responses equals the total number of subtasks sent, it is confirmed that responses to all subtasks have been received. Only after confirming that responses to all subtasks have been received does HDMA begin to perform a task flow drain operation, clearing all cached resources and state information related to the current task.

[0072] In this embodiment of the application, after obtaining the response information corresponding to all subtasks in the task splitting stage, the task flow of the network computing task is drained so that the execution results of the subtasks in transit will not be lost due to early draining. HDMA can obtain the complete task execution status, thereby providing a complete information basis for subsequent error type merging.

[0073] like Figure 3 As shown, in one embodiment, if the target execution stage is a task splitting stage, step S103 above may include the following steps:

[0074] S201, determine the target error type identifier based on the priority of the error type identifiers corresponding to each subtask.

[0075] In this embodiment, after collecting the response information corresponding to all subtasks, HDMA parses the error type identifier carried by each subtask from the response information. Since different subtasks may return different error types, HDMA merges the error type identifiers according to a preset priority rule and determines a final target error type identifier from multiple error type identifiers.

[0076] Optionally, the error type identifier with the highest priority among multiple error type identifiers can be determined as the target error type identifier.

[0077] For example, assuming DCU0 returns an Okay Response, DCU1 returns the currently defined High Error Response, DCU2 returns the currently defined Med Error Response, and DCU3 returns the currently defined Low Error Response, then the error type identifier corresponding to the High Error Response will be determined as the target error type identifier.

[0078] S202, generate an exception interruption based on the target error type identifier, node identifier, and task identifier.

[0079] In this embodiment, HDMA will assemble the target error type identifier determined from multiple error type identifiers, its own node identifier, and the current task identifier into a packet according to the interrupt message format to generate an abnormal interrupt.

[0080] In this embodiment, the target error type identifier is determined according to the priority of the error type identifier corresponding to each subtask; an abnormal interrupt is generated based on the target error type identifier, node identifier, and task identifier. This enables HDMA to select the highest priority error type as the target error type identifier according to a preset priority rule after collecting responses from multiple subtasks during the task splitting phase, and generate an abnormal interrupt to report to the target DCU. This avoids reporting redundant or low-priority error information to the DCU, reduces the interference of invalid interrupts to the target DCU, and lowers the processing burden and bandwidth usage of the DCU.

[0081] Based on the above embodiments, step S101 can be implemented in the following ways:

[0082] If the response information obtained during the circular buffer read phase is an error response, it is determined that a read exception event has occurred during the circular buffer read phase.

[0083] Optionally, the software pre-configures a RingBuf in the DCU's memory space and writes the information of the on-net computing task to be executed into this RingBuf. The HDMA monitors the pointer changes of the RingBuf based on the register information configured by the software. When new task information is detected, the HDMA initiates a RingBuf read request to the near-end DCU to read the detailed information of the on-net computing task. The HDMA receives the read response returned by the near-end DCU and parses the status information carried in the response. If the parsed status information is an error response, it is determined that a read exception event has occurred during the RingBuf read phase. Internally, the HDMA drains this task flow and determines the error type identifier corresponding to the read exception event according to the error type encoding agreed upon with the software. Then, it reports an interrupt, carrying link error information and the current HDMA and task encoding in the entire system.

[0084] If the response information obtained during the dependency check phase is an erroneous response, it is determined that an abnormal response event occurred during the dependency check phase.

[0085] Optionally, HDMA determines whether the current task needs to enable Dependency Check based on the task configuration information read from RingBuf. If it does, HDMA initiates a Dependency Check request to the near-end DCU, which reads data values ​​from a pre-agreed address space in the near-end DCU. In the last operation of the previous task, the software writes the agreed-upon data value, such as the value 1, into this address space. The near-end DCU returns the read data value as a response to HDMA. Upon receiving this response, HDMA first checks whether the status information carried in the response is an error response. If the status information is an error response, HDMA determines that an abnormal response event has occurred during the dependency check phase. Internally, HDMA drains this task flow and determines the error type identifier corresponding to the abnormal response event according to the error type encoding agreed upon with the software, then reports an interrupt.

[0086] If the response information obtained during the dependency check phase is correct and the dependency check times out, it is determined that a comparison failure event occurred during the dependency check phase.

[0087] Optionally, after HDMA initiates a dependency check request to the near-end DCU, if the status information carried in the received task response is a correct response (Okay Response), it indicates that the address space of the near-end DCU can be read normally. HDMA further compares the data value carried in the response information with a pre-agreed reference value. If the comparison result is consistent, HDMA confirms that the near-end address space on which the current task depends has met the task execution requirements and prepares to start task execution. If the comparison result is inconsistent, that is, the read data value does not match the reference value, HDMA continues to wait for the response of the next dependency check cycle and performs a re-comparison. If the dependency check consistently receives correct responses but the content consistently fails to match the reference value, the software's timeout mechanism will monitor this. After the software detects that the comparison operation has timed out, it issues a forced memory reclamation command. Upon receiving the forced memory reclamation command, HDMA forcibly stops the current operation and performs internal memory drain. After the drain is completed, HDMA reports an interrupt and pulls the drain status register high to inform the software, which then investigates why the corresponding source and destination address spaces cannot be released. At this point, HDMA determines that a comparison failure event has occurred during the dependency check phase and identifies the error type identifier corresponding to the comparison failure event.

[0088] If the response information obtained during the task splitting phase is a fatal error response, it is determined that a fatal exception event occurred during the task splitting phase.

[0089] In this embodiment, a fatal error response refers to a response in which the error type carried in the response information returned by each subtask is a fatal error. A fatal error indicates that the current anomaly cannot be recovered by a simple retransmission or retry mechanism, and it is necessary to stop the current task and conduct a systematic investigation.

[0090] Optionally, after completing dependency checks, HDMA begins task splitting, dividing the on-net computation task into multiple subtasks and issuing them to the corresponding remote proxy nodes (RPs). Each RP, upon receiving a subtask request from HDMA, executes the specific on-net computation content according to the request type and forwards the subtask to the corresponding near proxy node (LP), which then executes it to the corresponding DCU. After each DCU completes its subtask, it returns a response to the LP, which then forwards it back to the source RP. The RP merges the responses from all DCUs within the communication domain and returns them to HDMA. HDMA receives the final execution result of each subtask and parses the error type identifier carried in the response. If the error type identifier in the response returned by a subtask is a fatal error identifier, HDMA determines that a fatal exception has occurred during the task splitting phase. HDMA immediately stops issuing new subtasks and collects all responses already in transit. After confirming that all responses in transit have been returned, it drains all internal resources. If the returned in-transit response contains multiple error types of different priorities, HDMA merges these error types and uses the highest priority error type as the final reported error type. Finally, HDMA reports an interrupt carrying detailed error type and HDMA information to the target DCU specified by the software, where it is processed uniformly.

[0091] Based on the above embodiments, if there are multiple links between the Direct Memory Access Node (HDMA) and the target depth calculator (DCU), the method may further include the following steps:

[0092] When the target detects an abnormal event during the target execution phase of the on-net computing task, the target execution phase is re-executed through other links between the Direct Memory Access Node (HDMA) and the Target Depth Calculator (DCU); other links are those other than the link where the abnormal event occurred.

[0093] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of a multi-level Switch interconnection system provided in an embodiment of this application. For example, as shown... Figure 4As shown, if the current network topology supports multiple physical or logical links to a target DCU, the software can pre-configure HDMA to allow automatic retransmission. When HDMA detects an abnormal event during the target execution phase of an on-network computing task, it parses the error type carried in the abnormal event. If the error type is a link error, HDMA determines that the current link is abnormal. HDMA retransmits the current task or subtask, selecting other links from the multiple links except the one where the abnormal event occurred for transmission during retransmission.

[0094] For example, the RP (Redirect Resource) performs round-robin arbitration on multiple paths to the same destination. During retransmission, it avoids the path that previously reported an error response and selects another available path for task retransmission. If the HDMA (Host DMA) retransmission is successful and receives the correct task response information, it continues to execute the subsequent task process and finally reports to the software to complete the task normally. This process does not require software intervention, and the upper-layer user is unaware of any link problems with the current switch. If the retransmission fails after reaching a preset upper limit threshold, the HDMA finally reports an error interruption information to notify the software for troubleshooting.

[0095] In this embodiment, when the target detects an abnormal event in the target execution phase of the on-network computing task, the target execution phase is re-executed through other links between the Direct Memory Access Node (HDMA) and the Target Depth Calculator (DCU). These other links are those other than the link where the abnormal event occurred. This allows the HDMA to actively select other available links to re-execute the current execution phase without software intervention after detecting the abnormal event, significantly reducing software involvement and system processing overhead.

[0096] The following will use a complete embodiment to illustrate... Figure 5-6 The overall implementation logic and beneficial effects of the performance data acquisition method proposed in this application are explained below:

[0097] In this embodiment, taking the All-Reduce on-network computing task as an example, the HDMA register is first configured by software. Based on the configuration information, the HDMA initiates a RingBuf read operation to the near-end DCU. The RP forwards the HDMA request to the LP, which then stores the read request to the near-end DCU. The near-end DCU returns a RingBuf read response to the LP, which forwards it back to the RP. The RP then passes the RingBuf read response to the HDMA. The HDMA parses the response to obtain the specific task content, including task type, communication domain division, source and destination address spaces, etc. Based on the parsed task configuration information, the HDMA determines whether dependency checking needs to be enabled. If so, the HDMA initiates a dependency check request to the near-end DCU, which the RP forwards to the LP, and the LP stores the request to the near-end DCU. In the last operation of the previous task execution, the software writes a pre-defined value (e.g., the number 1) to the address read by the dependency check. The DCU returns this value to the LP, which then forwards it to the RP. The RP returns the dependency check response to the HDMA. The HDMA compares the data content carried in the response with a pre-defined reference value. If the comparison is successful, it confirms that the near-end address space on which the current task depends meets the task execution requirements, and prepares to begin task execution.

[0098] HDMA splits tasks according to the task configuration information, sends All-Reduce sub-requests to RP, and informs RP of the communication domain division of the current task. RP replicates the All-Reduce request and sends it to the LP corresponding to all DCUs in that communication domain according to the communication domain division. After each DCU executes the sub-task, it returns the response to the LP, which then returns it to the source RP. RP uses these responses for on-network computation and merges the status information carried in the responses according to the error priority agreed upon with the software. For example, if DCU0 returns a normal response, DCU1 returns a high-level error response, and DCU2 returns a medium-level error response, then RP will use the high-level error as the final merged result. During the task splitting phase, HDMA monitors the final execution result of each sub-task in real time. If an abnormal event is detected, HDMA first identifies the error type and determines the corresponding error type identifier. For fatal error responses, HDMA stops sending new sub-tasks, collects all in-transit sub-task responses, and after confirming that all responses have been returned, drains the task flow, clearing all cached resources and status information related to the current task. HDMA selects the highest-priority error type from multiple error types as the target error type identifier based on the priority of the error type identifiers returned by each subtask. Then, it generates an exception interrupt based on the target error type identifier, the HDMA's own node identifier, and the current task's task identifier, and reports this exception interrupt to a predefined target DCU. Upon receiving the exception interrupt, the target DCU accurately locates the specific HDMA and task where the exception occurred based on the error type identifier, node identifier, and task identifier carried in the interrupt, and performs corresponding exception repair operations. For example, for non-fatal errors, error information is recorded and processed uniformly after the task ends; for link errors, HDMA is configured to initiate automatic retransmission; and for fatal errors, complete error information is reported to higher-level system management software to locate the fault source and perform targeted repair.

[0099] In this embodiment, anomaly detection and preliminary processing are both autonomously performed by HDMA on the switch side, without the need for DCU involvement in error discovery and flushing decisions. This avoids the large-scale waste of computing resources caused by overall reset in traditional solutions. Furthermore, HDMA only flushes after receiving responses from all subtasks, preventing data loss in transit and downstream resource residue issues, ensuring the integrity of error recovery. The anomaly interrupts reported by HDMA carry precise error types, enabling the target DCU to quickly locate faults and take targeted remedial measures. This significantly reduces the error handling cost of on-network computing tasks and improves the overall stability and operational efficiency of the DCU cluster system.

[0100] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0101] Based on the same inventive concept, this application also provides a Direct Memory Access Node (HDMA) for implementing the exception handling method described above. The solution provided by this HDMA is similar to the implementation described in the above method; therefore, the specific limitations in one or more HDMA embodiments provided below can be found in the limitations of the exception handling method described above, and will not be repeated here.

[0102] In an exemplary embodiment, a Direct Memory Access Node (HDMA) is provided. The HDMA is used to perform anomaly detection on the online computing task based on response information obtained from each execution stage corresponding to the online computing task; when an anomaly event is detected in the target execution stage of the online computing task, it determines the error type identifier corresponding to the anomaly event; the target execution stage is any stage among the execution stages; an anomaly interrupt is generated based on the error type identifier, the node identifier corresponding to the HDMA, and the task identifier corresponding to the online computing task; and the anomaly interrupt is reported to a predefined target depth calculator (DCU) so that the target depth calculator (DCU) can process the anomaly event based on the anomaly interrupt.

[0103] In an exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to drain the task flow of on-network computing tasks after obtaining the response information corresponding to all subtasks in the task splitting phase.

[0104] In an exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine the target error type identifier based on the priority of the error type identifiers corresponding to each subtask; and to generate an exception interrupt based on the target error type identifier, the node identifier, and the task identifier.

[0105] In one exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine the error type identifier corresponding to the highest priority as the target error type identifier.

[0106] In one exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine that a read exception event has occurred during the ring buffer read phase if the response information obtained during the ring buffer read phase is an error response.

[0107] In one exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine that an abnormal response event has occurred during the dependency check phase if the response information obtained during the dependency check phase is an erroneous response.

[0108] In an exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine that a comparison failure event has occurred in the dependency check phase when the response information obtained in the dependency check phase is a correct response and the dependency check times out.

[0109] In one exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to determine that a fatal exception event has occurred during the task splitting phase when the response information obtained during the task splitting phase is a fatal error response.

[0110] In an exemplary embodiment, the Direct Memory Access Node (HDMA) is specifically used to re-execute the target execution phase through other links between the HDMA and the target depth calculator (DCU) when the target detects an abnormal event during the target execution phase of the on-net computing task; the other links are links other than the link where the abnormal event occurred among multiple links.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0112] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0113] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An exception handling method, characterized in that, Applied to Direct Memory Access Node (HDMA), the method includes: Based on the response information obtained from each execution stage corresponding to the online computing task, anomaly detection is performed on the online computing task; If an abnormal event is detected during the target execution phase of the on-network computing task, an error type identifier corresponding to the abnormal event is determined; the target execution phase is any one of the execution phases. An abnormal interrupt is generated based on the error type identifier, the node identifier corresponding to the Direct Memory Access Node (HDMA), and the task identifier corresponding to the on-network computing task. The abnormal interruption is reported to a predefined target depth calculator (DCU) so that the target depth calculator (DCU) can process the abnormal event based on the abnormal interruption.

2. The method according to claim 1, characterized in that, The target execution phase is the task breakdown phase; In the event that an abnormal event occurs during the target execution phase of the on-network computing task, the method further includes: Once the response information corresponding to all subtasks in the task splitting phase is obtained, the task flow of the on-network computing task is drained.

3. The method according to claim 2, characterized in that, The step of generating an abnormal interrupt based on the error type identifier, the node identifier corresponding to the Direct Memory Access Node (HDMA), and the task identifier corresponding to the on-network computing task includes: The target error type identifier is determined based on the priority of the error type identifiers corresponding to each subtask; The abnormal interruption is generated based on the target error type identifier, the node identifier, and the task identifier.

4. The method according to claim 3, characterized in that, The step of determining the target error type identifier based on the priority of the error type identifiers corresponding to each subtask includes: The error type identifier corresponding to the highest priority is determined as the target error type identifier.

5. The method according to claim 1, characterized in that, The abnormal events include read abnormal events; The step of performing anomaly detection on the network computing task based on the response information obtained from each execution stage corresponding to the network computing task includes: If the response information obtained during the circular buffer read phase is an error response, it is determined that a read exception event has occurred during the circular buffer read phase.

6. The method according to claim 1, characterized in that, The abnormal events include response abnormal events; the step of performing anomaly detection on the online computing task based on the response information obtained from each execution stage corresponding to the online computing task includes: If the response information obtained during the dependency check phase is an erroneous response, it is determined that an abnormal response event has occurred during the dependency check phase.

7. The method according to claim 1, characterized in that, The abnormal events include comparison failure events; The step of performing anomaly detection on the online computing task based on the response information obtained from each execution stage corresponding to the online computing task includes: If the response information obtained during the dependency check phase is a correct response and the dependency check times out, it is determined that the comparison failure event has occurred during the dependency check phase.

8. The method according to claim 1, characterized in that, The abnormal events include fatal abnormal events; the abnormal detection of the online computing task based on the response information obtained from each execution stage corresponding to the online computing task includes: If the response information obtained during the task splitting phase is a fatal error response, it is determined that the fatal exception event has occurred during the task splitting phase.

9. The method according to any one of claims 1-8, characterized in that, When multiple links exist between the Direct Memory Access Node (HDMA) and the target depth calculator (DCU), the method further includes: When the target detects an abnormal event during the target execution phase of the on-network computing task, the target execution phase is re-executed through other links between the Direct Memory Access Node (HDMA) and the target depth calculator (DCU); the other links are those other than the link where the abnormal event occurred.

10. A Direct Memory Access Node (HDMA), characterized in that, The Direct Memory Access Node (HDMA) is used to perform the steps of the method according to any one of claims 1 to 9.

11. A DCU cluster system, characterized in that, The DCU cluster system includes the Direct Memory Access Node (HDMA) as described in claim 10, the Target Depth Calculator (DCU), the first proxy node, and the second proxy node.

Images