Switching device, multiprocessor system, exception handling method, and electronic device

Abstract

The invention relates to switching equipment, a multiprocessor system, an exception handling method and electronic equipment, and belongs to the technical field of communication. The switching equipment comprises N transmission links and N exception handling modules. The N transmission links are used for connecting N processors, and N is a positive integer greater than or equal to 2; each transmission link is used for transmitting information sent by the processor; the N exception handling modules are in one-to-one correspondence with the N transmission links; a target exception handling module in the N exception handling modules is used for receiving and responding to an emptying instruction and emptying information transmitted by a target transmission link corresponding to the target exception handling module in the first direction; wherein the emptying instruction is sent when the processor connected with the target transmission link is abnormal; the first direction is a direction leading to a processor connected with the target transmission link. According to the invention, the fault-tolerant capability of the switching equipment or the multiprocessor system in the case of abnormal conditions can be enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and particularly relates to a switching device, a multi-processor system, an exception processing method and an electronic device. BACKGROUND

[0002] In high-performance computing and artificial intelligence applications (such as deep learning and large models), efficient communication between processors (such as GPUs (Graphics Processing Unit)) is the key to achieving powerful computing capabilities. With the continuous increase in model complexity and data volume, a single GPU has been difficult to meet the demand, and multiple GPUs working together has become the norm. Switches as a kind of high-performance GPU bridging device, through providing cross-networks, realize high-speed interconnection between GPUs, greatly improving the efficiency of data transmission.

[0003] Since the Switch system (i.e. multi-processor system) will involve frequent data interaction, the existing Switch system is prone to cause system downtime or data loss when facing link exceptions, timeouts and other abnormal situations, which will affect the continuity of other GPU computing tasks and the accuracy of the results, especially in data transmission intensive applications. SUMMARY

[0004] In view of this, the purpose of the present application is to provide a switching device, a multi-processor system, an exception processing method and an electronic device to enhance the fault tolerance of the switching device or the multi-processor system when facing abnormal situations.

[0005] Embodiments of the present application are implemented as follows: In a first aspect, the embodiments of the present application provide a switching device, comprising: N transmission links and N exception processing modules; the N transmission links are used to connect N processors, and N is a positive integer greater than or equal to 2; each transmission link is used to transmit information sent by a processor; the N exception processing modules correspond one-to-one to the N transmission links; a target exception processing module in the N exception processing modules is used to receive and respond to an emptying instruction, and perform an emptying operation on information transmitted by a target transmission link in a first direction; wherein the target transmission link is a transmission link corresponding to the target exception processing module; the emptying instruction is sent by a processor connected to the target transmission link when the processor is abnormal; and the first direction is a direction leading to the processor connected to the target transmission link.

[0006] In the above embodiments, when the processor connected to the target transmission link in the switching device malfunctions, the information transmitted on the target transmission link in the first direction (the direction leading to the processor connected to the target transmission link) is drained to enhance the fault tolerance of the switching device or multiprocessor system and prevent problems such as shutdown or information loss caused by message blocking. For example, when processor 0 malfunctions, assuming the router in the switching device can only temporarily store 4 messages, all of which are sent to processor 0, a draining process is required due to the limited buffering capacity of the transmission link connected to processor 0. Otherwise, the 4 messages temporarily stored by the router cannot be sent to the transmission link connected to processor 0. Only when the number of messages on the transmission link connected to processor 0 decreases can the router send its messages to the transmission link connected to processor 0. Similarly, other processors can also send information to the router, so that the information exchanged by other processors through the router can "flow". Otherwise, message blocking will cause problems such as shutdown or information loss of the entire switching device or multiprocessor system.

[0007] In one possible implementation of the first aspect embodiment, the target anomaly handling module is further configured to respond to the emptying command, control the target transmission link to stop receiving new information in the second direction, perform a completion operation and add anomaly markers to the data not received in the second direction, and control the target transmission link to send the completed data with added anomaly markers, so that the source end releases the resources sent to the processor connected to the target transmission link; wherein, the second direction is opposite to the first direction.

[0008] In the above embodiment, by performing a completion operation on the data not collected in the second direction and adding an anomaly mark, the completed data with the anomaly mark is then sent to the source end, so that the source end can release the resources of the processor connected to the target transmission link, thereby improving the performance of the source end processor.

[0009] In one possible implementation of the first aspect embodiment, the switching device further includes: N tracking queues, each corresponding to one of the N transmission links and the N exception handling modules; each tracking queue is used to record basic information of each request transmitted on the corresponding transmission link; the target exception handling module is further used to generate a target response marked as an error state based on the basic information of the target request recorded in the tracking queue when the target transmission link times out and does not receive a response corresponding to the target request, and send the target response through the target transmission link so that the source end sending the target request releases the resources sent to the processor connected to the target transmission link.

[0010] In the above embodiments, a tracking queue is set up to record the basic information of each request transmitted through the corresponding transmission link. When no response is received for the target request after a timeout, a target response marked as an error is generated based on the basic information of the target request recorded in the tracking queue, and the target response is sent. This causes the source end that sent the target request to release the resources of the processor connected to the target transmission link, thereby improving the performance of the source end processor.

[0011] In one possible implementation of the first aspect embodiment, the target anomaly handling module is further configured to clear the basic information of the corresponding request recorded in the tracking queue when the target transmission link receives the response corresponding to the request.

[0012] In the above embodiments, after receiving the response to the request, the basic information of the corresponding request recorded in the tracking queue is cleared in a timely manner. By reducing the amount of data, it is beneficial to promptly detect when the response to the target request has not been received within a timeout period.

[0013] In one possible implementation of the first aspect embodiment, each transmission module in the transmission link integrates a tracking queue.

[0014] In the above embodiments, integrating the tracking queue into the transmission link helps reduce latency and improve data processing speed. At the same time, integrating the tracking queue into the transmission module of the transmission link, compared to integrating it into the communication interface of the transmission link, allows the transmission module to be located closer to the source end, which is beneficial for early request monitoring and saving downstream resources.

[0015] In one possible implementation of the first aspect embodiment, the switching device further includes: N anomaly detection modules, each corresponding to one of the N transmission links; each anomaly detection module is used to detect whether the processor connected to the corresponding transmission link is abnormal, and to report anomaly information when an anomaly is detected.

[0016] In the above embodiments, by integrating an advanced anomaly detection mechanism into the switching equipment, various abnormal events such as link anomalies and timeouts can be monitored in real time. Once an anomaly is detected, the multiprocessor system will quickly trigger the corresponding processing flow.

[0017] In one possible implementation of the first aspect embodiment, each transmission link includes: a communication interface and a transmission module; the communication interface is used to connect to a processor; the transmission module is connected to the communication interface and is used to transmit messages; wherein the communication interface integrates an anomaly detection module.

[0018] In the above embodiments, by integrating the anomaly detection module into the communication interface, each communication interface can independently perform anomaly detection and report the detected link or timeout anomalies via the transmission module.

[0019] In one possible implementation of the first aspect embodiment, the switching device further includes N exception handling modules; each communication interface and transmission module in each transmission link integrates an exception handling module.

[0020] In the above embodiments, by integrating an exception handling module into the communication interface and the transmission module in the transmission link, both the communication interface and the transmission module in the transmission link have exception handling functions, which helps to improve the exception handling capability and enhance the fault tolerance of the system.

[0021] Secondly, embodiments of this application also provide a multiprocessor system, including: N processors and a switching device as provided in any possible implementation of the first aspect embodiment and / or in combination with the first aspect embodiment, wherein the switching device is connected to the N processors respectively.

[0022] In one possible implementation of the second aspect embodiment, the multiprocessor system further includes: a central processing unit (CPU), the CPU being connected to the switching device; the CPU being configured to determine that a processor connected to a target transmission link in the switching device is faulty, and to send a drain instruction to a target fault handling module corresponding to the target transmission link in the switching device.

[0023] In one possible implementation of the second aspect embodiment, the central processing unit is further configured to control other processors connected to the switching device to suspend sending information to the processor connected to the target transmission link, and to control the processor connected to the target transmission link to go offline. In the above embodiment, when the processor connected to the target transmission link malfunctions, in addition to controlling the switching device to perform a drain operation on the information transmitted in the first direction of the target transmission link, it also controls other processors connected to the switching device to suspend sending information to the processor connected to the target transmission link. This allows for faster draining of the information transmitted in the first direction of the target transmission link, while also reducing the resource overhead of other processors. Furthermore, it controls the processor connected to the target transmission link to go offline and be isolated to ensure that other normal processors continue to operate stably and maintain the high availability of the multiprocessor system.

[0024] Thirdly, embodiments of this application also provide an anomaly handling method, including: determining that the processor connected to the target transmission link in the switching device is abnormal; and performing a drain operation on the information transmitted by the target transmission link in a first direction, wherein the first direction is the direction leading to the processor connected to the target transmission link.

[0025] In one possible implementation of the third aspect embodiment, the method further includes: controlling the target transmission link to stop receiving new information in a second direction; wherein the second direction is opposite to the first direction; performing a completion operation and adding an anomaly mark to the data not received in the second direction; and controlling the target transmission link to send the completion data with the anomaly mark added, so that the source end releases the resources sent to the processor connected to the target transmission link.

[0026] Fourthly, embodiments of this application also provide an electronic device, including: a multiprocessor system provided as described in the first aspect embodiments and / or any possible implementation in conjunction with the first aspect embodiments.

[0027] Other features and advantages of this application will be set forth in the following description. The objectives and other advantages of this application can be realized and obtained through the structures specifically pointed out in the written description and the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings. The above and other objects, features, and advantages of this application will become clearer through the accompanying drawings.

[0029] Figure 1a A schematic diagram of the structure of a first multiprocessor system provided in an embodiment of this application is shown.

[0030] Figure 1b A schematic diagram of the structure of a second multiprocessor system provided in an embodiment of this application is shown.

[0031] Figure 2 This illustration shows a schematic diagram of the connection between a first type of switching device and a processor, as provided in an embodiment of this application.

[0032] Figure 3 A schematic diagram of the connection between a second type of switching device and a processor, as provided in an embodiment of this application, is shown.

[0033] Figure 4 A schematic diagram of the structure of a third multiprocessor system provided in an embodiment of this application is shown.

[0034] Figure 5 A schematic diagram of the structure of an MCCI interface provided in an embodiment of this application is shown.

[0035] Figure 6A schematic diagram of the structure of an FM module provided in an embodiment of this application is shown.

[0036] Figure 7 A schematic diagram of a multiprocessor system provided in an embodiment of this application is shown.

[0037] Figure 8 A flowchart illustrating an exception handling method provided in an embodiment of this application is shown.

[0038] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The following embodiments are provided as examples to more clearly illustrate the technical solutions of this application, and should not be used to limit the scope of protection of this application. Those skilled in the art will understand that, without conflict, the following embodiments and features can be combined with each other.

[0040] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, relational terms such as "first," "second," etc., in the description of this application are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0041] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] In the description of the embodiments of this application, unless otherwise expressly specified and limited, the technical term "connection" can be a direct connection or an indirect connection through an intermediate medium.

[0043] To improve the fault tolerance of multiprocessor systems (switch systems) in the face of abnormal situations such as link failures and timeouts, and to ensure that the system can quickly recover and continue to work normally after anomalies occur, thereby providing a more stable and reliable processor interconnect solution for high-performance computing and artificial intelligence applications, this application provides a switching device, a multiprocessor system, an anomaly handling method, and an electronic device. When the processor connected to the target transmission link in the switching device fails, the information transmitted on the target transmission link in the first direction (the direction leading to the processor connected to the target transmission link) is drained to enhance the fault tolerance of the switching device or multiprocessor system and prevent problems such as downtime or information loss caused by message blocking.

[0044] The following is combined Figure 1a The multiprocessor system provided in the embodiments of this application will be described, such as... Figure 1a As shown, the multiprocessor system includes a switching device and N processors, with the switching device connected to each of the N processors, where N is a positive integer greater than or equal to 2. In some possible implementations, the multiprocessor system also includes a Central Processing Unit (CPU), which is connected to the switching device. It is understood that... Figure 1a Only the case involving a central processing unit, switching devices, and N processors is shown.

[0045] In some possible implementations, such as Figure 1b As shown, there can be multiple switching devices in a multiprocessor system, for example, in Figure 1b The example includes two switching devices, each connected to N graphics processing units (GPUs). Understandably, the N processors connected to different switching devices can be different.

[0046] The N processors can be of the same type, such as all being GPUs, or they can be of different types, such as some being GPUs and others being other types of processors, such as accelerated processing units (EPUs). The following examples will primarily use GPUs as the example.

[0047] The processor shown in this application can be an integrated circuit chip with signal processing capabilities. This includes CPUs, network processors (NPs), GPUs, accelerator processors, multimedia application processors (MAPs), microprocessors, etc.; it can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Alternatively, the processor can be any conventional processor.

[0048] The switching device shown in this application includes, but is not limited to, switches and switching chips. The switching device includes N transmission links and N exception handling modules. Furthermore, the switching device may also include components such as routers and interrupt management units (IMUs).

[0049] N transmission links are used to connect N processors, as shown in Figure 1. The first end of each transmission link is connected to a processor, and the second end of each transmission link is connected to a router. The N transmission links communicate with each other through the router.

[0050] In addition, the router is connected to the IMU, which is connected to the central processing unit, which is connected to each of the N processors (not shown in Figure 1).

[0051] A transmission link is a physical or logical channel in a communication system responsible for the reliable and efficient transmission of data between two or more points. Each transmission link is used to transmit information sent by the processor. In addition, each transmission link can also be used to transmit information sent by the exception handling module. The information transmitted by the transmission link includes at least one of the following: request, response, and data.

[0052] There are N exception handling modules, each corresponding to one of N transmission links. Each transmission link has its own exception handling module. In some possible implementations, these N exception handling modules can be merged into a single exception handling module. Each exception handling module is connected to its corresponding transmission link, and each exception handling module can communicate with the router through its corresponding transmission link. To increase the reliability of the multiprocessor system, each exception handling module can also be directly connected to the router (not shown in Figure 1).

[0053] Upon receiving a drain command, the target exception handling module among the N exception handling modules responds by draining the information transmitted in the first direction on the target transmission link corresponding to the target exception handling module. During the drain operation, the information in the first direction of the target transmission link can be cleared directly, or the information in the first direction of the target transmission link can be continuously sent to the exception handler until it is drained. For example, if processor 0 malfunctions, assuming the router can only temporarily store 4 messages, all of which are sent to processor 0, draining is necessary due to the limited buffering capacity of the transmission link connected to processor 0. Otherwise, the 4 messages temporarily stored by the router cannot be sent to the transmission link connected to processor 0. Only when the number of messages on the transmission link connected to processor 0 decreases can the router send its messages to the transmission link connected to processor 0. Similarly, other processors can also send information to the router, allowing information exchanged by other processors through the router to "flow." Otherwise, message blocking could cause the entire switching device or multi-processor system to shut down or result in information loss.

[0054] The processor malfunctions described in this application include: malfunctions of the processor itself and / or malfunctions of the transmission link and the connection link between the processor and the transmission link. For example, when the processor 0 itself malfunctions and / or the link link 0 connecting the processor 0 and the switching device malfunctions, it can be considered that the processor 0 is malfunctioning.

[0055] The drain command is sent when the processor connected to the target transmission link malfunctions. For example, when the central processing unit (CPU) learns that the processor connected to the target transmission link in the switching equipment is malfunctioning, it can send a drain command to the target fault handling module corresponding to the target transmission link in the switching equipment. The drain command may also originate from a terminal, not the CPU. For instance, when a user learns that the processor connected to the target transmission link is malfunctioning, they can send a drain command to the target fault handling module corresponding to the target transmission link in the switching equipment through their terminal.

[0056] The target anomaly handling module is the anomaly handling module corresponding to the target transmission link. The target transmission link can be any of the N transmission links mentioned above. For example, when processor 0 in Figure 1 is abnormal, the transmission link connected to processor 0 at this time is the target transmission link. Similarly, if processor 2 in Figure 1 is abnormal, the transmission link connected to processor 2 at this time is the target transmission link. Other cases will not be exemplified.

[0057] In some possible implementations, the target anomaly handling module is also used to respond to a drain command, control the target transmission link to stop receiving new information in the second direction, perform padding operations and add anomaly flags to the unreceived data in the second direction, and control the target transmission link to send the anomaly-marked padding data so that the source end releases the resources of the processor connected to the target transmission link. For example, if the response data for a request is 256 bytes, and only 128 bytes are received, the remaining 128 bytes are padded and an anomaly flag is added, and then the anomaly-marked padding data is sent to the source end.

[0058] In addition to emptying the information transmitted in the first direction, the emptying command can also be used to trigger the completion operation of incomplete data in the second direction. In some possible implementations, a command or instruction different from the emptying command can also be sent separately to trigger the completion operation of incomplete data in the second direction.

[0059] The second direction is the opposite of the first direction. For example, assuming processor 0 in Figure 1 malfunctions, the first direction could be from the router to processor 0, and the second direction could be from processor 0 to the router. Another example: if the first direction is the TX (transmit) direction, then the second direction is the RX (receive) direction.

[0060] In one possible implementation, each exception handling module may include an error controller (ErrorControl) for responding to a drain command and performing information draining and data completion operations. In some implementations, the error controller may also be used to collect timeout information in the TX and RX directions.

[0061] One possible implementation, such as Figure 2As shown, each transmission link includes a communication interface and a transmission module (TM). The communication interface is used to connect to the processor and can be an MCCI (Multi-Chip Communication Interface) interface. The transmission module is connected to the communication interface and is used to transmit messages. The transmission module is located between the MCCI interface and the router. During information emptying, the information in the first direction needs to be emptyed by both the communication interface and the transmission module. During data completion, any missing data in the second direction can be completed by the communication interface and / or the transmission module.

[0062] The fault handling module can be separate from the transmission link. In one possible implementation, the fault handling module can be integrated into the transmission link, for example, into the communication interface and the TM module. When the fault handling module is integrated into the transmission link, the switching equipment also includes N fault handling modules, meaning the switching equipment contains a total of 2N fault handling modules. Each communication interface and transmission module in each transmission link integrates one fault handling module. For example, each communication interface and transmission module in each transmission link integrates one fault controller. For ease of distinction, the TM module integrating the fault handling module can be called the FM (Fault Management) module.

[0063] In one possible implementation, such as Figure 3 As shown, the switching device also includes: N TrackQueues, each corresponding to one of the N transmission links and N exception handling modules. One TrackQueue is connected to one exception handling module and one transmission link. Each TrackQueue records basic information (such as request ID, request type, request content, context information, etc.) for each request transmitted through the corresponding transmission link. The TrackQueue can be either the transmission link itself or the corresponding exception handling module, using it to record the basic information of each request transmitted through the corresponding transmission link.

[0064] The target error handling module is also used to generate a target response marked as an error state based on the basic information of the target request recorded in the tracking queue when the target transmission link times out and no response is received for the target request, and to send the target response so that the source end that sent the target request releases the resources of the processor connected to the target transmission link.

[0065] The target anomaly handling module is also used to clear the basic information of the corresponding request recorded in the tracking queue when the target transmission link receives a response to the request. In some possible implementations, clearing the basic information of the request recorded in the tracking queue can also be implemented by the target transmission link itself.

[0066] Figure 3 The N tracking queues can be set up separately from the N transmission links, or they can be integrated. For example, each transmission module in a transmission link can integrate a tracking queue; the tracking queue can be integrated into the transmission module of the transmission link. In one implementation, the tracking queue is integrated into the transmission link and set up separately from the exception handling module. In other implementations, both the tracking queue and the exception handling module are integrated into the transmission link.

[0067] In some possible implementations, in addition to integrating an anomaly handling module into the switching equipment, an advanced anomaly detection mechanism can also be integrated into the switching equipment to ensure that link anomalies or timeout anomalies can be detected in a timely manner. This mechanism can monitor various anomalies such as link anomalies and timeouts in real time. Once an anomaly is detected, the multiprocessor system will quickly trigger the corresponding processing flow.

[0068] In addition to link anomalies and timeout detection, the anomaly detection mechanism can also include other types of anomaly detection, such as data integrity verification (e.g., CRC (Cyclic Redundancy Check) verification), memory anomaly detection (e.g., ECC (Error Correcting Code) verification), instruction execution anomaly detection, etc. By integrating multiple detection methods, it can more comprehensively cover the types of anomalies that may occur and improve the system's anomaly detection capabilities.

[0069] In one possible implementation, the switching device further includes N anomaly detection modules (not shown in the figure), each corresponding to one of the N transmission links. Each anomaly detection module is used to detect whether the processor connected to the corresponding transmission link is abnormal, and reports the anomaly information when an anomaly is detected. The reported anomaly information is then reported to the central processing unit via the router and the IMU. For example, each anomaly detection module can be used to detect whether the connection link between the transmission link and the processor is abnormal, such as detecting whether the request or response in the transmission link has timed out. When a processor is abnormal, requests transmitted to the abnormal processor through the transmission link cannot receive a response, triggering a timeout anomaly. After receiving the anomaly information reported by the router, the IMU will report the anomaly information to the central processing unit. After the abnormal processor completes the offline process, the IMU will clear the anomaly information of the corresponding link to avoid the anomaly information from continuously interfering with the multiprocessor system and to ensure that other hardware modules can work normally.

[0070] Each anomaly detection module can include multiple anomaly detection units, with different units detecting different anomaly types, such as link anomalies, timeouts, CRC, ECC, and instruction execution anomaly detection. Alternatively, an anomaly detection module can be viewed as an anomaly detection set, where anomaly detection units can be configured according to detection needs.

[0071] N anomaly detection modules can be set up separately from N transmission links. In one possible implementation, the anomaly detection modules can be integrated into the transmission links. For example, the communication interface can integrate an anomaly detection module.

[0072] In some possible implementations, the multiprocessor system shown in this application can be as follows: Figure 4 As shown, Figure 4 The MCCI interface and FM module in the Switch integrate exception handling modules. Furthermore, FM can integrate a tracking queue, and the MCCI interface can integrate an exception detection module. Integrating the exception detection and handling modules onto each MCCI interface of the Switch allows each interface to independently perform error detection and preliminary processing. For example, in scenario one: if link0 fails, the MCCI interface will initiate a reconnection operation. If the reconnection fails, the MCCI interface will report the detected link exception, indicating an error in the connected GPU, and the software will trigger the GPU0 shutdown process. In scenario two: if GPU0 malfunctions, it may cause the MCCI interface to fail to collect all data or the FM module to fail to receive a response, resulting in a timeout. The MCCI interface and FM module will report the timeout exception, and the software will also trigger the GPU0 shutdown process.

[0073] To better illustrate this, the following will be combined with... Figure 5 The structure of the MCCI interface with integrated exception handling module is described below. This MCCI interface includes: a transmit controller (TX Control), a transmit multiplexer (MUX for Multiplexer), a TX module, a receive controller (RX Control), a receive multiplexer (MUX), an RX module, and an error controller (ErrorControl). An exception detection module can also be integrated into the MCCI interface. Figure 5 (Not shown in the image).

[0074] The sending controller includes a request output queue, a response output queue, and a data output buffer. Requests from the request output queue, responses from the response output queue, and data from the data output buffer are sent to the TX module via the MUX. Similarly, the receiving controller includes a request input queue, a response input queue, and a data input buffer. Requests, responses, and data received by the RX module are stored in the request input queue, response input queue, and data input buffer respectively via the MUX.

[0075] The MCCI interface detects link or timeout anomalies. Error Control generates an exception message and sends it to FM. FM then forwards the exception message to IMU via Router. IMU reports the exception to the CPU. Upon receiving the exception, the CPU issues a drain command. Error Control, upon receiving the drain command, performs a drain operation in the TX direction, draining requests in the request output queue, responses in the response output queue, and data in the data output buffer. In the RX direction, it stops receiving new information, completes any incomplete data, adds an exception flag, and sends the completed data to FM. FM then forwards the data to the source GPU via Router to release resources from the source GPU destined for the faulty GPU.

[0076] To better illustrate this, the following will be combined with... Figure 6 right Figure 4 The structure of the integrated exception handling module and the FM module with a tracking queue is described below. The FM module includes a request output queue, a response output queue and a data output buffer, a request input queue, a response input queue and a data input buffer, an error control system, and a tracking queue. Each time the FM module sends a request, it records the basic information of that request in the tracking queue. When a response is received, the basic information of the request in the tracking queue is cleared. If the FM module does not receive a response, it generates an exception message, which is forwarded to the IMU via the Router. The IMU then reports the exception message to the central processing unit (CPU), triggering the CPU to issue a drain command.

[0077] It should be noted that, for the same transmission link, although the central processing unit (CPU) may receive abnormal information reported by the IMU multiple times, it will only issue a drain command once within a certain period of time, and will not repeatedly send drain commands. The CPU can also block redundant abnormal information sent by abnormal GPUs or switching devices.

[0078] Upon receiving an exclusion command, Error Control will perform a drain operation on the information sent to the MCCI, specifically draining the requests in the request output queue, the responses in the response output queue, and the data in the data output buffer. In addition to draining, Error Control will also generate a target response marked as error based on the basic information of timed-out requests recorded in the trace queue, along with corresponding error data (optional). The target response and error data will then be sent to the source GPU via the Router to release resources that were being redirected from the source GPU to the malfunctioning GPU.

[0079] The central processing unit (CPU) is used to determine if the processor connected to the target transmission link in the switching equipment is malfunctioning, and sends a drain command to the target malfunction handling module corresponding to the target transmission link in the switching equipment. In some possible implementations, the CPU can issue a drain command upon receiving malfunction information reported by the switching equipment indicating that the processor connected to the target transmission link is malfunctioning. In this implementation, the switching equipment integrates an malfunction detection mechanism that can monitor various malfunction events such as link malfunctions and timeouts in real time. Furthermore, malfunction information can also be obtained by the malfunctioning processor reporting it to the CPU.

[0080] The central processing unit (CPU) is also used to control other processors to suspend sending information to the processor connected to the target transmission link after determining that the processor connected to the target transmission link is abnormal, and to take the processor connected to the target transmission link offline. For example, when processor 0 itself is abnormal or the link between processor 0 and the switching equipment (link0) is abnormal, the CPU will control other processors, such as processor 1, processor 2... processor n, to suspend sending information to the processor connected to the target transmission link to prevent the abnormality from spreading further or affecting other normally operating services. At the same time, in order to reduce the impact of the abnormal processor on the overall multiprocessor system, it will also take the abnormal processor offline. The offline process can quickly isolate the abnormal processor and its related links from the multiprocessor system, while ensuring that other normal processors continue to operate stably and maintaining the high availability of the multiprocessor system.

[0081] In one possible implementation, before taking the abnormal processor offline, the central processing unit first controls other processors to suspend sending information to the processor connected to the target transmission link, and ensures that the information transmitted in the first direction of the target transmission link has been cleared, and then takes the abnormal processor offline.

[0082] The central processing unit (CPU) is also used to dynamically allocate and schedule processor resources based on the real-time operating status and task requirements of the multiprocessor system after a malfunctioning processor goes offline. For example, when a processor malfunctions (such as performance degradation), tasks originally assigned to that processor can be dynamically migrated to other working processors, while resource allocation is adjusted to ensure that the overall performance of the multiprocessor system is not affected.

[0083] The central processing unit (CPU) is also used to perform the processor's online operation when the abnormal processor has been repaired or the processor has switched to cold standby mode. This ensures that the repaired or standby processor can be reintegrated into the multiprocessor system and resume working in conjunction with other processors, thereby minimizing downtime and performance loss in the multiprocessor system caused by the abnormality.

[0084] The go-live time can be determined by the software in the central processing unit. During go-live, it must be ensured that there are no blocked requests on the disconnected path. The go-live process is as follows: Step 1: Hardware initialization, loading the processor driver, configuring the processor's hardware parameters, and checking the processor's hardware status.

[0085] Step 2: Software configuration. Add the online processor to the computing resource pool of the multiprocessor system and load the necessary computing tasks and data.

[0086] Step 3: Task takeover. After the standby processor completes the online operation, it needs to take over the tasks of the main processor, reallocate computing tasks, ensure task continuity, and notify other processors in the multiprocessor system that the standby processor is ready.

[0087] To mitigate the impact of anomalies on multiprocessor systems, the multiprocessor system described in this application supports a service rollback mechanism. This mechanism allows the multiprocessor system to restore computational tasks to their state before the anomaly occurred, thereby preventing the entire task from failing due to a single processor failure. In this way, the impact of anomalies on the overall service of the multiprocessor system can be minimized, and resource utilization can be improved. Through the service rollback function, other processors can roll back to their state before the anomaly occurred and continue executing unfinished computational tasks, further enhancing the fault tolerance and availability of the multiprocessor system.

[0088] Business rollback refers to the mechanism of restoring the business state to a previous normal point in time when a system failure or anomaly occurs. In a multiprocessor system, the system periodically saves the computing state of each processor. When a processor goes offline abnormally, the multiprocessor system can detect this anomaly in a timely manner. The system rolls back the other related processors to the most recently saved normal state point, discards the computing results that may have been affected by the anomaly, and restarts the calculation from the normal state point to ensure the integrity and correctness of the computing task.

[0089] In one possible implementation, the exception handling process of the multiprocessor system in this application can be as follows: Figure 7 As shown. When a processor experiences an anomaly such as a link failure or timeout, the central processing unit (CPU) will take the faulty processor offline. Simultaneously, when a processor malfunctions, tasks originally assigned to that processor can be dynamically migrated to other working processors. Other working processors performing the same task as the faulty processor can restore their computational tasks to their pre-anomaly state through service rollback and restart computation from that point. Once the backup or faulty processor is repaired, it can be reintegrated into the multiprocessor system and resume collaborative work with other processors.

[0090] The multiprocessor system provided in this application embodiment can be a distributed system. For example, multiple switching devices in a multiprocessor system can be distributed in different locations, and the multiple switching devices can be connected through a network. And / or, processors such as GPUs connected to the same switching device can be distributed on different devices.

[0091] In some possible implementations, the multiprocessor system provided in this application embodiment may be a system within an electronic device. Based on this, this application embodiment also provides an electronic device that may include the aforementioned multiprocessor system. This electronic device may be a server, a computing center, a data center, or a supercomputer, etc.

[0092] Based on the same inventive concept, this application also provides a switching device, the structure of which is the same as that of the switching device in the multiprocessor system described above. Further details will not be provided here.

[0093] Based on the same inventive concept, this application also provides an exception handling method, which can be applied to the above-described multiprocessor system and also to the electronic devices described below. The following is in conjunction with... Figure 8 The exception handling method is described, which includes S1 and S2.

[0094] S1: Determines that the processor connected to the target transmission link in the switching device is faulty.

[0095] In one possible implementation, a processor, such as a central processing unit (CPU), can determine that the processor connected to the target transmission link in the switching device is abnormal. For example, when the processor is abnormal, it can actively report the abnormal information to the CPU. Alternatively, an abnormality detection mechanism can be integrated into the switching device to report the abnormality information to the CPU when the abnormality of the processor connected to the target transmission link is detected. In this way, the CPU can determine that the processor connected to the target transmission link in the switching device is abnormal.

[0096] S2: Perform a draining operation on the information transmitted in the first direction of the target transmission link.

[0097] After determining that the processor connected to the target transmission link in the switching equipment is malfunctioning, the central processing unit (CPU) can perform a drain operation on the information transmitted on the target transmission link in the first direction. In one possible implementation, during step S2, the CPU may issue a drain command to the switching equipment to control the switching equipment to drain the information transmitted on the target transmission link in the first direction. Here, the first direction is the direction leading to the processor connected to the target transmission link.

[0098] In some possible implementations, the above-described anomaly handling method further includes: controlling the target transmission link to stop receiving new information in the second direction (opposite to the first direction); performing a completion operation and adding an anomaly mark to the data not received in the second direction; and controlling the target transmission link to send the completed data with the anomaly mark added, so that the source end releases the resources sent to the processor connected to the target transmission link.

[0099] For example, the central processing unit (CPU) can execute the above process by controlling the switching equipment. For instance, the CPU can issue a drain command or other commands. After receiving the command, the switching equipment can control the target transmission link to stop receiving new information in the second direction (opposite to the first direction). It can also perform a fill operation on the data that is not received in the second direction and add an anomaly mark. The CPU can then control the target transmission link to send the fill data with the anomaly mark so that the source end can release the resources sent to the processor connected to the target transmission link.

[0100] In some possible implementations, the above-described exception handling method further includes: when the target transmission link times out and does not receive a response corresponding to the target request, generating a target response marked as an error state based on the basic information of the target request recorded in the tracking queue, and sending the target response via the target transmission link, so that the source end that sent the target request releases the resources sent to the processor connected to the target transmission link. The tracking queue is used to record the basic information of each request transmitted by the target transmission link.

[0101] The exception handling method provided in this application embodiment has the same implementation principle and technical effect as the aforementioned system embodiment. For the sake of brevity, any part not mentioned in the method embodiment can be referred to the corresponding content in the aforementioned system embodiment.

[0102] This application also provides an electronic device for executing the above-described exception handling method, such as... Figure 9 As shown, the electronic device 200 includes: a transceiver 210, a memory 220, a communication bus 230, and a processor 240. The transceiver 210, memory 220, and processor 240 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected through one or more communication buses 230 or signal lines. The transceiver 210 is used to send and receive data. The memory 220 is used to store computer programs. The computer program includes at least one software functional module that can be stored in the memory 220 in the form of software or firmware or embedded in the operating system (OS) of the electronic device 200. The processor 240 is used to execute the software functional modules or computer programs stored in the memory 220. For example, the processor 240 is used to execute the above-described exception handling method.

[0103] The memory 220 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0104] Processor 240 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), Network Processor (NP), Graphics Processing Unit (GPU), Accelerated Processing Unit (ACCU), Multimedia Application Processor (MAP), microprocessor, etc.; it can also be a Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. Alternatively, processor 240 can also be any conventional processor.

[0105] Among them, the aforementioned electronic devices 200 include, but are not limited to, mobile phones, tablets, computers, etc.

[0106] This application embodiment also provides a non-volatile computer-readable storage medium (hereinafter referred to as the storage medium) storing a computer program, which, when run by a computer such as the electronic device 200 described above, executes the exception handling method described above.

[0107] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, laptop, server, or electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A switching device, characterized in that, include: N transmission links are used to connect N processors, where N is a positive integer greater than or equal to 2; Each transmission link is used to transmit information sent by the processor; There are N exception handling modules, each corresponding to one of the N transmission links; the target exception handling module among the N exception handling modules is used to receive and respond to the drain command, and to drain the information transmitted by the target transmission link in the first direction. Wherein, the target transmission link is the transmission link corresponding to the target anomaly handling module; the drain command is sent when the processor connected to the target transmission link is abnormal; the first direction is the direction leading to the processor connected to the target transmission link.

2. The switching device according to claim 1, characterized in that, The target anomaly handling module is also used to respond to the emptying command, control the target transmission link to stop receiving new information in the second direction, perform a completion operation and add anomaly markers to the data that was not received in the second direction, and control the target transmission link to send the completed data with added anomaly markers so that the source end releases the resources sent to the processor connected to the target transmission link. The second direction is opposite to the first direction.

3. The switching device according to claim 1, characterized in that, The switching equipment also includes: There are N tracking queues, each corresponding to one of the N transmission links and the N exception handling modules; each tracking queue is used to record the basic information of each request transmitted through the corresponding transmission link. The target anomaly handling module is further configured to generate a target response marked as an error state based on the basic information of the target request recorded in the tracking queue when the target transmission link times out and does not receive a response corresponding to the target request, and send the target response through the target transmission link so that the source end that sent the target request releases the resources sent to the processor connected to the target transmission link.

4. The switching device according to claim 1, characterized in that, The switching equipment also includes: There are N anomaly detection modules, each corresponding to one of the N transmission links. Each anomaly detection module is used to detect whether the processor connected to the corresponding transmission link is abnormal, and to report the anomaly information when an anomaly is detected.

5. The switching device according to claim 4, characterized in that, Each transmission link includes: A communication interface used to connect to the processor; A transmission module, connected to the communication interface, is used to transmit messages; The communication interface integrates an anomaly detection module.

6. The switching equipment according to any one of claims 1 to 5, characterized in that, The switching equipment also includes N exception handling modules; each communication interface and transmission module in each transmission link integrates an exception handling module.

7. A multiprocessor system, characterized in that, include: N processors and a switching device as described in any one of claims 1 to 6, wherein the switching device is connected to each of the N processors.

8. The multiprocessor system according to claim 7, characterized in that, The multiprocessor system further includes a central processing unit, which is connected to the switching device; The central processing unit is used to determine that the processor connected to the target transmission link in the switching device is abnormal, and to send a drain command to the target abnormality handling module corresponding to the target transmission link in the switching device.

9. The multiprocessor system according to claim 8, characterized in that, The central processing unit is also used to control other processors connected to the switching equipment to suspend sending information to the processor connected to the target transmission link, and to control the processor connected to the target transmission link to go offline.

10. An exception handling method, characterized in that, include: The processor connected to the target transmission link in the switching equipment is found to be faulty. The information transmitted on the target transmission link in a first direction is drained, wherein the first direction is the direction leading to the processor to which the target transmission link is connected.

11. The method according to claim 10, characterized in that, The method further includes: The target transmission link is controlled to stop receiving new information in a second direction; wherein the second direction is opposite to the first direction; Perform data completion and add anomaly markers for any missing data in the second direction; Control the target transmission link to send supplementary data with added anomaly markers, so that the source end releases the resources sent to the processor connected to the target transmission link.

12. An electronic device, characterized in that, include: The multiprocessor system as described in any one of claims 7 to 9.

Images