Method for handling link failure, monitor, and server

Abstract

The application discloses a link fault processing method, a monitor and a server, and relates to the technical field of servers.The method comprises the following steps: at a triggering moment of a fault detection operation, target performance data of a target link transmission signal and a target count value of a target parameter are acquired, and the target parameter is used for recording the number of consecutive failures of a target link transmission test data packet.Both the target performance data and the number of consecutive failures can reflect the fault type.Therefore, the fault type can be determined according to the target performance data and the target count value.Furthermore, in the case that the target link cannot transmit data according to the fault type, resource usage information of other graphic processors can be acquired, and the data flow from the central processor to the target graphic processor is switched to other routing paths, so that the data transmission between the central processor and the target graphic processor can be completed in the case of link interruption, and the stability is relatively high.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to methods for handling link failures, monitors, and servers. Background Technology

[0002] In server technology, the Central Processing Unit (CPU) and Graphics Processing Unit (GPU) communicate via a high-speed Peripheral Component Interconnect Express (PCIE) link to meet the extremely high data processing speed requirements of applications such as artificial intelligence computing, big data analysis, and graphics rendering. However, in actual operation, the PCIE link may be affected by various factors and fail, which can severely impact the server's computing performance and efficiency.

[0003] Generally, related technologies employ retransmission mechanisms; however, in the event of a link interruption, data transmission remains impossible. Furthermore, retransmission mechanisms consume significant resources, further increasing the server's burden. This can lead to unstable service. Summary of the Invention

[0004] This application provides a method, apparatus, monitor, server, storage medium, and program product for handling link failures, in order to solve the problem of service instability caused by link failures in related technologies.

[0005] This application provides a method for handling link failures, including: At the trigger time of the fault detection operation, the target count value of the target parameter and the target performance data of the signal transmitted by the target link are obtained. The target parameter is used to record the number of consecutive failures of the target link transmission test data packet. The target link is the link between the central processing unit and the target graphics processor. Based on the target count value and target performance data, determine the fault type of the target link; Based on the type of fault, determine whether to continue using the target link to transmit data; If it is determined that the target link will not be used to transmit data, the resource usage information of the other graphics processors (excluding the target graphics processor) among the multiple graphics processors is obtained. Based on the resource usage information of other graphics processors, a routing switch operation is performed on the data stream corresponding to the target graphics processor.

[0006] This application also provides a link failure handling apparatus, including: The acquisition module is used to acquire the target count value of the target parameter and the target performance data of the signal transmitted by the target link at the time of triggering the fault detection operation. The target parameter is used to record the number of consecutive failures of the target link transmission test data packet. The target link is the link between the central processing unit and the target graphics processor. The determination module is used to determine the fault type of the target link based on the target count value and target performance data; and based on the fault type, to determine whether to continue using the target link to transmit data. The acquisition module is also used to acquire resource usage information for each of the multiple graphics processors other than the target graphics processor when it is determined that the target link will not be used to transmit data. The routing switching module is used to perform routing switching operations on the data stream corresponding to the target graphics processor based on the resource usage information of other graphics processors.

[0007] This application also provides a monitor for implementing the steps of any of the above-described link failure handling methods when executing a computer program.

[0008] This application also provides a server, including: Central processing unit, monitor, switch, and multiple graphics processors; The central processing unit is connected to the monitor, the monitor is connected to the switch, the switch is connected to multiple graphics processors, and any two graphics processors are interconnected. The steps of the monitoring device in implementing any of the above-mentioned link failure handling methods when executing a computer program.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described link failure handling methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described link failure handling methods.

[0011] This application, firstly, at the trigger moment of the fault detection operation, can acquire target performance data of the target link's transmission signal and target count values ​​of target parameters. Target parameters are used to record the number of consecutive failures in transmitting test data packets on the target link. Both target performance data and the number of consecutive failures reflect whether the target link is faulty and the specific fault type under fault conditions. There are various fault types. In some cases, the graphics processor can recover erroneous data based on its own data recovery mechanism; in other cases, after a link failure, data transmission is completely impossible. Therefore, this solution, upon determining that the target link cannot transmit data based on the fault type, can acquire resource usage information of other graphics processors and switch the data flow from the central processing unit (CPU) to the target GPU to other routing paths. This allows data transmission between the CPU and the target GPU to continue even when the link is interrupted, without consuming excessive resources and ensuring service stability. Furthermore, this solution eliminates the need to set up backup physical links for each link, saving hardware costs. Furthermore, the process of switching to a backup physical link may result in temporary data loss or service interruption, which cannot meet the needs of business scenarios with extremely high real-time requirements. However, this solution can switch the data stream route and continue transmitting data from the point of interruption, thus solving this technical problem. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of a server architecture provided for an embodiment of this application; Figure 2 A schematic diagram of the architecture of a monitor provided in an embodiment of this application; Figure 3 A schematic diagram illustrating a link failure handling process provided in an embodiment of this application; Figure 4 A schematic diagram of another server architecture provided for an embodiment of this application; Figure 5 A schematic diagram illustrating another link failure handling process provided in this application embodiment; Figure 6 This is a schematic diagram of a link failure processing device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] In server technology, the Central Processing Unit (CPU) and Graphics Processing Unit (GPU) communicate via a high-speed Peripheral Component Interconnect Express (PCIe) link to meet the extremely high data processing speed requirements of applications such as artificial intelligence computing, big data analysis, and graphics rendering. However, the PCIe link can be affected by various factors during actual operation, leading to failures. For example, signal attenuation, electromagnetic interference, hardware aging, and physical damage can all cause link transmission errors, interruptions, or performance degradation. When the PCIe link between the CPU and GPU malfunctions, especially after a link interruption, it severely impacts the server's computing performance and efficiency. For instance, in artificial intelligence training tasks, the GPU needs to receive large amounts of data and instructions from the CPU in real time. If the PCIe link fails, data transmission will be delayed or even interrupted, causing the training process to stall and consuming significant time and computing resources.

[0018] In related technologies, the communication link between the central processing unit (CPU) and the graphics processing unit (GPU) is singular. When a serious failure occurs in this link, normal communication between the CPU and GPU is impossible, leading to data loss in the GPU and impacting server operation. Specifically, when recoverable errors occur in the PCIe link, such as data packet timeouts or symbol errors, the PCIe mechanism attempts to recover data at the software level through error correction and retransmission mechanisms; these errors do not interrupt transmission. However, when serious errors occur, such as protocol errors or link interruptions, the PCIe mechanism cannot correct them automatically, resulting in data transmission errors on the link and causing GPU loss. In summary, although the PCIe mechanism has error correction and retransmission functions, it only operates at the software level, lacking hardware-level error correction, and the link cannot be recovered when serious errors occur.

[0019] To address the aforementioned technical problems, this application provides a method for handling link failures, which can be implemented by a server, such as... Figure 1 As shown, a server may include a central processing unit (CPU), a monitor, a switch, and multiple graphics processors (GPUs). The CPU is connected to the monitor, the monitor is connected to the switch, and the switch is connected to multiple GPUs. Specifically, the monitor may be connected to the switch's upport, and a GPU may be connected to a switch's downport, which may be a display port.

[0020] The CPU (Central Processing Unit) sends instructions and data to the GPU (Graphics Processing Unit). A monitor detects link failures between the CPU and multiple GPUs and handles any such failures. A switch connects multiple GPUs. Each GPU receives instructions and data from the CPU and performs related tasks based on these instructions and data. Multiple GPUs can be interconnected.

[0021] Specifically, such as Figure 2 As shown, the monitor can be designed in hardware as an integrated unit of "controller + sampler + signal conditioning unit + comparator + buffer unit".

[0022] The controller, which can be used to control the fault detection process, can be either a Field-Programmable Gate Array (FPGA) or a Microcontroller Unit (MCU). The controller generates a sampling clock signal of a preset duration (e.g., 10 nanoseconds) and transmits it to the sampler. Upon receiving the sampling clock signal, the sampler performs a sampling operation to obtain the performance data of the link. The performance data is then transmitted to a signal conditioning unit, which in turn transmits it to a comparator. The signal conditioning unit's role is to protect the acquired data from interference from the transmission link. The comparator compares the performance data with a preset threshold range to determine if the performance data falls within that range. If the performance data is determined to be outside the preset threshold range, an anomaly flag is output. Upon receiving the anomaly flag, the buffer unit stores the time of the anomaly and the corresponding performance data.

[0023] To address the aforementioned technical problems, embodiments of this application provide a method for handling link failures, which can be executed by a monitor, such as... Figure 3 As shown, the specific steps of the link failure handling method may include: Step S301: At the trigger time of the fault detection operation, acquire the target count value of the target parameter and the target performance data of the signal transmitted by the target link.

[0024] The target parameter can be used to record the number of consecutive failures in transmitting test data packets on the target link. The target link is the link between the central processing unit (CPU) and the target graphics processor (GPU), and can be a PCIe link. The target GPU can be any one of multiple GPUs connected to the switch. The test data packet can be a PCIe Transaction Layer Packet (TLP).

[0025] Specifically, the monitor can monitor the signals transmitted on the target link, periodically collect signal performance index values, and generate performance data based on these values—that is, hardware-level performance monitoring data. The period duration can be 10 nanoseconds, and the performance index values ​​can be signal strength, operating voltage, transmission current, etc. Additionally, the monitor can periodically (e.g., a period duration of 50 milliseconds) send test data packets to each graphics processor and record the number of consecutive failed test data packet transmissions, providing monitoring data for the PCIe mechanism. The test period duration can be an integer multiple of the acquisition period.

[0026] If the count value of the target parameter is found to be equal to a preset threshold, a fault detection operation is determined, i.e., the current moment is determined as the trigger moment for the fault detection operation. Therefore, the count value of the target parameter can be determined as the target count value. Simultaneously, the monitor can determine the performance data collected in the most recent sampling period as the target performance data.

[0027] Alternatively, the monitor can trigger a fault detection operation when it detects that any performance metric value in the performance data of the most recent sampling period is outside the corresponding threshold range (i.e., the trigger time of the fault detection operation). Simultaneously, the monitor can determine the latest count value of the target parameter as the target count value and the performance data collected in the most recent sampling period as the target performance data.

[0028] Alternatively, the fault detection operation can be triggered periodically. At the trigger time of the periodic fault detection operation, the latest count value of the target parameter can be obtained and determined as the target count value. Simultaneously, the monitor can determine the latest count value of the target parameter as the target count value and determine the target performance data based on the performance data collected in the most recent sampling period.

[0029] The above three methods can be used individually to trigger fault handling operations, or any two or three methods can be combined to trigger fault handling operations. This ensures that fault detection operations are triggered promptly under different fault conditions.

[0030] Step S302: Determine the fault type of the target link based on the target count value and target performance data.

[0031] Among them, the fault type can be link interruption, link signal attenuation, signal error, etc. Link signal attenuation can be specifically divided into severe link signal attenuation (data packet error rate is greater than the preset error rate threshold) and slight link signal attenuation (data packet error rate is less than the preset error rate threshold). The data packet error rate can be the ratio of the number of data packets with errors to the total number of data packets transmitted, which is calculated by the target graphics processor and sent to the monitor.

[0032] Specifically, after each sampling period, the monitor can analyze the target performance data and target count value to determine the fault type. For example, the collected performance data and the latest count value of the target parameter can be input into a pre-built fault diagnosis model to obtain the fault type output by the fault diagnosis model. The fault diagnosis model can be a machine learning model.

[0033] Step S303: Determine whether to continue using the target link to transmit data based on the fault type.

[0034] Specifically, the monitor can store at least one fault handling method corresponding to each fault type, and the fault handling method can be used to indicate whether the link can continue to transmit data. Accordingly, after determining the fault type of the target link, a fault handling method matching the fault type of the target link can be determined from the fault handling methods corresponding to each of the at least one fault type.

[0035] Step S304: If it is determined that the target link will not be used to transmit data, the resource usage information of the other graphics processors besides the target graphics processor is obtained.

[0036] Resource usage information can be used to indicate the communication strength between the graphics processor and other components; for example, it can be the resource utilization rate of the graphics processor.

[0037] Specifically, if the monitor determines that the fault handling method indicates that data transmission should not continue using the target link, it can switch the routing path. To reduce the impact on data transmission on other links, it can first obtain the resource usage information of other graphics processors. Specifically, the monitor can send a resource information retrieval command to other graphics processors. After receiving the resource information retrieval command, the graphics processor will send its own resource usage information to the monitor. Alternatively, it can send a resource information retrieval command to the central processing unit (CPU), which will automatically invoke a query command to obtain the resource usage information of each graphics processor and send it to the monitor.

[0038] Step S305: Based on the resource usage information of other graphics processors, perform a routing switch operation on the data stream corresponding to the target graphics processor.

[0039] Specifically, the monitor may include at least one routing path information for the target graphics processor (GPU). Then, based on resource usage information of other GPUs, it selects one of the at least one routing path information. Finally, based on the selected routing path information, it performs a routing switch operation on the data stream corresponding to the target GPU in the switch, enabling the switch to send the received data to other GPUs corresponding to the selected routing path information, which then forward the data to the target GPU.

[0040] The link failure handling method of this application, in its embodiments, firstly, at the trigger time of the fault detection operation, acquires target performance data of the target link's transmission signal and target count values ​​of target parameters. Target parameters are used to record the number of consecutive failures of the target link in transmitting test data packets. Both target performance data and the number of consecutive failures reflect whether the target link is faulty and the specific fault type under fault conditions. There are various fault types. In some cases, the graphics processor can recover erroneous data based on its own data recovery mechanism; in other cases, after a link failure, data transmission is completely impossible. Therefore, this solution, upon determining that the target link cannot transmit data based on the fault type, can acquire resource usage information of other graphics processors and switch the data flow from the central processing unit (CPU) to the target GPU to other routing paths. This allows data transmission between the CPU and the target GPU to continue even when the link is interrupted, without consuming excessive resources and ensuring service stability. Furthermore, this solution eliminates the need to set up backup physical links for each link, saving hardware costs. Furthermore, the process of switching to a backup physical link may result in temporary data loss or service interruption, which cannot meet the needs of business scenarios with extremely high real-time requirements. However, this solution can switch the data stream route and continue transmitting data from the point of interruption, thus solving this technical problem.

[0041] In some optional implementations, in step S301 above, the monitor may obtain the target count value of the target parameter by the following specific steps: Step 1: Periodically send test data packets to the target graphics processor.

[0042] Step 2: If no valid response packet corresponding to the test data is received within a preset time period after sending the test data packet, the count value of the target parameter is increased by a first preset value.

[0043] Alternatively, in step three, after receiving a valid response packet within a preset time after sending the test data packet, initialize the count value of the target parameter to the second preset value.

[0044] Step 4: At the moment the fault detection operation is triggered, the count value of the target parameter is determined as the target count value.

[0045] Specifically, monitoring software can be installed in the monitor, running on the monitoring thread of the controller's Linux kernel. The monitor can then periodically send test data packets to each graphics processor (GPU). Taking a target GPU as an example, the monitor can be configured with a timer corresponding to that GPU. After sending a test data packet to the target GPU, this timer can be started. If no valid response packet is received after the timer has elapsed for a preset time threshold (including no response packet received, or an erroneous response packet received), the count value of the target parameter of the target GPU can be incremented by a first preset value, for example, by 1. If a valid response packet is received before the timer has elapsed for the preset time threshold, the count value of the target parameter of the target GPU can be directly initialized to a second preset value, for example, to 0.

[0046] In addition, if no corresponding valid response packet is received for two consecutive cycles, the target graphics processor's state can be marked as abnormal. When the corresponding valid response packet is received again, the target graphics processor's state can be restored to normal.

[0047] In this way, proactively testing the target link through software and combining it with performance data detected by the hardware to jointly diagnose the type of fault in the target link is more accurate and comprehensive, solving the problem of false detections or missed detections caused by a single dimension. Specifically, detecting an anomaly in any dimension can trigger fault diagnosis, which can improve the timeliness of fault diagnosis.

[0048] In some alternative implementations, if the signal strength is detected to be equal to a first preset threshold during any sampling period (i.e., in the event of a link interruption), the monitor can stop sending test data packets. This saves resources.

[0049] In some optional implementations, in step S302 above, the monitor may use the following specific steps to determine the fault type of the target link based on the target performance data and the target count value: Step 1: If the signal strength is equal to the first preset threshold, the link interruption is determined as the fault type of the target link.

[0050] Alternatively, in step two, if the signal strength is greater than the first preset threshold, less than the second preset threshold, and the target count value is greater than or equal to the third preset threshold, the link signal attenuation is determined as the fault type of the target link.

[0051] Alternatively, if the signal strength is greater than the first preset threshold, less than the second preset threshold, and the target count value is less than the third preset threshold, the signal error is determined as a fault type of the target link.

[0052] Specifically, the monitor can first compare the signal strength with a first preset threshold and a second preset threshold to determine the relationship between the signal strength and the first and second preset thresholds. It can also compare the target count value with a third preset threshold to determine the relationship between the target count value and the third preset threshold. Then, based on the determined comparison results, it determines the fault type corresponding to the comparison result in a pre-built mapping table of comparison results and fault types. If the comparison result cannot be matched in the mapping table, it can be determined that the target link is fault-free and no fault handling is required. The second preset threshold is the minimum signal strength within the normal signal strength range.

[0053] Specifically, if the signal strength is equal to a first preset threshold (e.g., 0), it indicates that there is currently no signal transmission on the target link, and a link interruption can be determined. In this case, the link interruption can be directly identified as the fault type of the target link, which is an unrecoverable (i.e., uncorrectable) error type. Additionally, the target link can be marked to prepare for subsequent redirection routes, thus avoiding selecting the faulty target link again to transmit signals.

[0054] If the signal strength is greater than the first preset threshold, less than the second preset threshold, and the target count value is less than the third preset threshold, it indicates that there is signal transmission in the target link and the error rate is low, which is convenient for error correction. Therefore, the signal error can be identified as the fault type of the target link, and the signal error is a self-recoverable error type.

[0055] If the signal strength is greater than the first preset threshold, less than the second preset threshold, and the target count value is greater than or equal to the third preset threshold, it indicates that the target link is currently transmitting a signal, but the error rate is high and the error correction is difficult. Therefore, the link signal attenuation can be identified as the fault type of the target link, and the link signal attenuation is an error type that cannot be recovered.

[0056] In this way, by specifically identifying the fault type, appropriate fault handling methods can be selected accordingly, making the process more flexible, saving resources, and improving fault handling efficiency. For example, in the case of a link interruption, alternative paths can be directly selected to transmit data without retransmission; instead, the data stream can be switched directly to continue interrupted data transmission. Furthermore, judging whether a link is faulty using data from two dimensions is more accurate and comprehensive.

[0057] In some optional implementations, the performance data may include at least one performance metric value, and the monitor may set a threshold range for each performance metric value. For example, the period may be 10 minutes, and the specific update method is as follows: For each performance metric, samples are taken from fault-free links a preset number of times (e.g., 1000 times), obtaining multiple performance metric values ​​equal to the preset number of times. Then, the mean and standard deviation are calculated based on these multiple performance metric values. Based on the mean and standard deviation, and the threshold calculation method corresponding to that performance metric, the threshold range for that performance metric is determined. For example, for signal strength, the threshold range is the mean minus twice the standard deviation (i.e., the third preset threshold mentioned above); for transmission current, the threshold range is the mean ± 10% variance; and for operating voltage, the threshold range is the mean ± 0.2%. This avoids misjudgment of fault types caused by changes in ambient temperature and humidity.

[0058] In some alternative implementations, in step S304 above, determining whether to continue using the target link to transmit data based on the fault type may include the following two cases: In the first scenario, the fault type of the target link is link interruption or link signal attenuation.

[0059] Option 1: Directly decide not to continue using the target link to transmit data.

[0060] Method 2: Perform a link recovery operation on the target link and obtain the first link recovery result. If the first link recovery result indicates successful link recovery, determine to continue using the target link for data transmission. Alternatively, if the first link recovery result indicates failed link recovery, then determine not to continue using the target link for data transmission.

[0061] Specifically, in Method 1, since a link interruption or signal attenuation indicates that the target link cannot be used further, in order to minimize the impact on the target graphics processor's task execution, it can be directly determined that data transmission through the target link will not continue. This improves fault handling efficiency and reduces the impact of link failure on task efficiency.

[0062] In Method Two, since link interruption and signal attenuation may be caused by physical connection loss, hardware damage, or deliberate disabling by device firmware or software, the former may not be resolved after a link recovery operation, while the latter may be resolved. Therefore, a link recovery operation can be performed on the target link first, and the first link recovery result can be obtained. If the first link recovery result indicates successful link recovery, it can be determined to continue using the target link for data transmission. Alternatively, if the first link recovery result indicates failed link recovery, it can be determined not to continue using the target link for data transmission. The link recovery operation can specifically include link retraining or resetting the target graphics processor. By performing the link recovery operation first, and if the link recovery is successful, other transmission paths do not need to be occupied, reducing the impact on other graphics processors' task execution.

[0063] Scenario 2: The fault type of the target link is signal error.

[0064] Method 1: Obtain at least one error correction result corresponding to the transmitted data from the target graphics processor. If any error correction result indicates successful error correction, determine to continue using the target link to transmit data. Alternatively, if all error correction results in at least one error correction result indicate error correction failure, determine not to continue using the target link to transmit data.

[0065] Method 2: Obtain at least one error correction result corresponding to the transmitted data from the target graphics processor. If any error correction result indicates successful error correction, determine to continue using the target link to transmit data. Alternatively, if all error correction results in at least one error correction result indicate error correction failure, perform a link recovery operation on the target link and obtain a second link recovery result. If the second link recovery result indicates successful link recovery, determine to continue using the target link to transmit data. Alternatively, if the second link recovery result indicates failed link recovery, determine not to continue using the target link to transmit data.

[0066] Specifically, if the fault type of the target link is determined to be a signal error, at least one error correction result corresponding to the transmitted data can be directly obtained from the target graphics processor.

[0067] Taking the target graphics processing unit (GPU) as an example, after receiving data transmitted by the central processing unit (CPU), it can determine the current data frame through the frame synchronization word (i.e., the start of the data frame) in the frame header. Then, it extracts the check bit and data bits from the current data frame and calculates a check value for the data bits. If the check value matches the value of the check bit, the data bits are considered valid. At this point, the valid data can be deinterleaved, and the original sub-block arrangement can be restored in reverse order of the interleaving. Then, Low-Density Parity-Check (LDPC) is used to correct the original sub-block arrangement, obtaining the error correction result, which is sent to the monitor. The number of LDPC decoding iterations can be set to 20. The error correction result includes the number of error bits. Correspondingly, when the monitor determines that the number of error bits is less than a preset error bit threshold (which can be 32), it can determine that the error correction result indicates successful error correction; when it determines that the number of error bits is greater than or equal to the preset error bit threshold, it can determine that the error correction result indicates failure.

[0068] If the target graphics processor determines that the number of error bits exceeds a preset error bit threshold, it can add the identification information of the current data frame to the retransmission request and send it to the central processing unit (CPU). Upon receiving the retransmission request, the CPU can retrieve the corresponding data from its own buffer based on the identification information of the current data frame and retransmit it to the target graphics processor. The target graphics processor then performs the aforementioned error correction operation to obtain the corrected result again.

[0069] Similarly, if the target graphics processor fails to correct the error after performing a preset number of retransmission operations (e.g., 3 times), it determines that the current error is of an unrecoverable type and can stop performing retransmission operations, and start subsequent redirection routing operations.

[0070] Accordingly, the monitor can obtain one or more error correction results from the target graphics processor.

[0071] If the monitor determines that any error correction result indicates successful error correction, it means that the target graphics processor can successfully recover the data and can continue to use the target link to transmit data.

[0072] In Method 1, if the monitor determines that all error correction results obtained indicate error correction failure, it means the target graphics processor cannot recover the data, and it is determined that the target link should not be used to transmit data. In this way, in the event of error correction failure, the routing path can be switched directly, improving fault handling efficiency.

[0073] In Method 2, if the monitor determines that all error correction results in at least one error correction result indicate error correction failure, the error is relatively serious, but can generally be recovered through link recovery operations. Therefore, a link recovery operation can be performed on the target link first, and a second link recovery result can be obtained. If the second link recovery result indicates successful link recovery, it can be determined to continue using the target link for data transmission. Alternatively, if the second link recovery result indicates link recovery failure, it can be determined not to continue using the target link for data transmission.

[0074] In this way, by performing the link recovery operation first, and if the link recovery is successful, there is no need to occupy other transmission paths, which can reduce the impact on other graphics processors performing tasks.

[0075] In some optional implementations, in step S305 above, the monitor may perform the following specific steps to perform a routing switch operation on the data stream corresponding to the target graphics processor based on the resource usage information corresponding to other graphics processors: Step 1: Based on the resource usage information of the other graphics processors, select the intermediate graphics processor and obtain its routing information.

[0076] Step two: Determine the routing register corresponding to the target graphics processor in the switch.

[0077] Step 3: Write the routing information into the routing register so that the data stream corresponding to the target graphics processor can be switched to the intermediate graphics processor.

[0078] Specifically, the monitor can select the graphics processor with the lowest resource utilization rate from among the other graphics processors based on the resource usage information of each of the other graphics processors, and then determine the selected graphics processor as the target graphics processor.

[0079] The monitor can store routing information corresponding to each graphics processor (GPU), or the switch can store routing information (e.g., communication address, port, etc.) corresponding to each GPU. For the former, the monitor can directly select the routing information corresponding to the intermediate GPU based on its identification information. For the latter, the monitor can send the intermediate GPU's identification information to the switch, allowing the switch to determine the corresponding routing information based on the intermediate GPU's identification information and send it back to the monitor. In this way, the monitor can obtain the routing information of the intermediate GPU.

[0080] Then, the monitor can determine the routing register corresponding to the target graphics processor from the routing registers of the multiple graphics processors included in the switch, based on the target graphics processor's identification information. It then writes the routing information from the intermediate routing register into the routing register corresponding to the target graphics processor, thus modifying the routing table for that processor. In this way, the switch can continue sending buffered, incomplete data to the intermediate graphics processor; that is, it can switch the data stream corresponding to the target graphics processor to the intermediate graphics processor. The intermediate graphics processor can then forward the received data to the target graphics processor based on the destination address carried in the data (i.e., the target graphics processor's communication address).

[0081] In this way, when a target link failure is detected, there is no need to restart the entire application or server. The data stream can be seamlessly switched to a healthy transmission path in a short time, resuming data transmission from the point of interruption, thereby ensuring the business continuity of upper-layer computing tasks (such as training tasks and graphics rendering). Furthermore, by directing the data stream to a less loaded graphics processor, the utilization of the entire graphics processor pool can be maximized, reducing the impact on computing tasks.

[0082] In some alternative implementations, the monitor can also periodically (e.g., every 50 nanoseconds) determine whether the bandwidth of the backup path (i.e., the link between the switch and the intermediate processor) is sufficient for signal transmission. If not, an intermediate graphics processor can be reselected based on the resource usage information of each graphics processor; if so, the intermediate graphics processor can continue to be used to forward data. In this way, the real-time performance of server tasks can be ensured through real-time optimization.

[0083] In some alternative implementations, the monitor can send a redirection signal to the switch to instruct the switch to perform the aforementioned redirection operation. For example, the operations in steps one to three of step S305 can be performed by the switch, and correspondingly, the operation of reselecting the intermediate graphics processor can also be performed by the switch. For example, GPU2 can be selected as the intermediate graphics processor to enable communication between the switch and GPU1.

[0084] In some alternative implementations, if the fault type is determined to be link signal attenuation, the monitor can also adjust the data transmission rate and bandwidth of the target link, which may include the following specific steps: Step 1: Obtain the current raw data transmission rate and raw bandwidth of the target link.

[0085] Step 2: If the original data transmission rate is not the smallest among the preset multiple data transmission rates, select a data transmission rate that is smaller than the original data transmission rate from among the multiple data transmission rates as the target data transmission rate.

[0086] Step 3: Set the data transmission rate of the target link to the target data transmission rate.

[0087] And, in step four, determine the target bandwidth based on the original bandwidth and the target data transmission rate.

[0088] Step 5: Set the bandwidth of the target link to the target bandwidth.

[0089] Specifically, the switch can record the data transmission rate and bandwidth of the link. The monitor can obtain the current raw data transmission rate and bandwidth of the target link from the switch. If the raw data transmission rate is not the minimum among a set of preset data transmission rates, it indicates that there is room for adjustment. Therefore, based on the raw data transmission rate, a data transmission rate lower than the raw data transmission rate can be selected as the target data transmission rate from a set of preset rates (for example, the largest or smallest rate among those lower than the raw rate can be selected as the target data transmission rate). This allows the data transmission rate of the target link to be set as the target data transmission rate. Alternatively, the product of the raw bandwidth and the raw data transmission rate can be calculated first, and the ratio of this product to the target data transmission rate can be determined as the target bandwidth. Finally, the bandwidth of the target link can be set as the target data transmission rate.

[0090] In some alternative implementations, upon detecting that the target link has returned to normal, the data transmission rate of the target link is restored from the target data transmission rate to the original data transmission rate, and the bandwidth of the target link is restored from the target bandwidth to the original bandwidth.

[0091] In some optional implementations, in step S303 above, if the resource usage information is the resource utilization rate, the monitor can also first obtain the resource utilization rate of other graphics processors and determine whether there are any idle graphics processors (e.g., graphics processors with zero resource utilization). If so, if the fault type is determined to be a link interruption, the monitor can directly use method two to determine whether to continue using the target link to transmit data. If not, if the fault type is determined to be a link interruption, the monitor can directly use method one to determine whether to continue using the target link to transmit data.

[0092] In this way, if there are idle GPUs, switching paths will not affect the tasks. Therefore, there is no need to perform link recovery operations; we can directly determine whether to continue using the target link for data transmission and directly execute the route switching operation to ensure service real-time performance. This also ensures load balancing among GPUs. Since the target link may recover after performing the link recovery operation, there is no need to transmit data through other paths. Therefore, if there are no idle GPUs, to avoid affecting other tasks, we can perform the link recovery operation first, and then determine whether to continue using the target link for data transmission, thus reducing the impact of link failures on tasks.

[0093] In some alternative implementations, such as Figure 4 As shown, the controller in the monitor can run monitoring software, which may include a fault diagnosis and classification module, an error correction and recovery module, a routing and redirection module, and an optimization module. When the monitor detects an anomaly in either hardware or software, it can trigger the fault diagnosis and classification module to perform relevant diagnostic and classification operations. The fault diagnosis and classification module is mainly used to determine the fault type based on target performance data and target count values. The error correction and recovery module is mainly used to perform link recovery operations and determine whether to continue using the target link for data transmission. The routing and redirection module is mainly used to select intermediate graphics processors and modify the routing table corresponding to the target graphics processor. The optimization module is mainly used to adjust the data transmission rate and bandwidth of the target link, as well as adjust the error correction strategies of each graphics processor. For example, if the fault type is determined to be link signal attenuation, the target graphics processor is notified to enable the LPDC strategy for error correction; if the fault type is determined to be link interruption, the target graphics processor is notified not to enable any error correction strategy. This saves resources.

[0094] Under the above structure, reference Figure 5 During data transmission, link performance data and target parameters can be monitored in real time. Based on the counts of these data, it can be determined whether a link failure has occurred. If not, the original link can continue to be used for data transmission. If a failure does occur, the fault type can be further determined, and based on the fault type, it can be determined whether error correction is feasible. If error correction is feasible, it is performed, and the success of the correction is determined. If the correction is successful, the original link can continue to be used for data transmission. If error correction is not feasible or fails, the routing and redirection module can be activated to find an alternative path, which is the process of determining the intermediate graphics processor mentioned above. Additionally, optimization operations can be performed during this process; that is, if the performance of the alternative path is found to be insufficient for data transmission requirements, an alternative path can be selected.

[0095] In this way, link fault tolerance is achieved through software algorithms and intelligent control, eliminating the need for a large number of additional hardware devices, thus reducing hardware costs and server design complexity. At the same time, rapid fault handling and performance optimization mechanisms improve server efficiency and reduce waste of computing resources.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0097] Embodiments of this application also provide a link failure handling apparatus, such as... Figure 6 As shown, it includes: The acquisition module 610 is used to acquire the target count value of the target parameter and the target performance data of the signal transmitted by the target link at the trigger time of the fault detection operation. The target parameter is used to record the number of consecutive failures of the target link transmission test data packet. The target link is the link between the central processing unit and the target graphics processor. The determination module 620 is used to determine the fault type of the target link based on the target count value and target performance data; and based on the fault type, to determine whether to continue using the target link to transmit data. The acquisition module 610 is also used to acquire resource usage information for each of the multiple graphics processors other than the target graphics processor when it is determined that the target link will not be used to transmit data. The routing switching module 630 is used to perform a routing switching operation on the data stream corresponding to the target graphics processor based on the resource usage information of other graphics processors.

[0098] In some optional implementations, the target performance data includes signal strength; the determination module 620 is specifically used for: Once the signal strength is determined to be equal to the first preset threshold, the link interruption is identified as the fault type of the target link; Alternatively, if the signal strength is determined to be greater than the first preset threshold, less than the second preset threshold, and the target count value is greater than or equal to the third preset threshold, the link signal attenuation is determined as the fault type of the target link. Alternatively, if the signal strength is greater than the first preset threshold, less than the second preset threshold, and the target count value is less than the third preset threshold, the signal error is determined as a fault type of the target link.

[0099] In some alternative implementations, the determining module 620 is specifically used for: If the fault type is determined to be link interruption or link signal attenuation, it is directly determined not to continue using the target link to transmit data; Alternatively, if the fault type is determined to be a signal error, at least one error correction result corresponding to the transmitted data is obtained from the target graphics processor; If any error correction result is determined to indicate successful error correction, it is determined to continue using the target link to transmit data. Alternatively, if it is determined that all error correction results in at least one error correction result are used to indicate error correction failure, it is determined that data transmission should not continue using the target link.

[0100] In some alternative implementations, the determining module 620 is specifically used for: If the fault type is determined to be link interruption or link signal attenuation, perform a link recovery operation on the target link and obtain the first link recovery result; If the result of the first link recovery is determined to indicate that the link recovery was successful, it is determined to continue using the target link to transmit data. Alternatively, if the result of the first link recovery is determined to indicate that the link recovery has failed, then it can be determined not to continue using the target link to transmit data.

[0101] In some alternative implementations, the determining module 620 is specifically used for: If all error correction results in at least one error correction result are used to indicate error correction failure, perform a link recovery operation on the target link and obtain a second link recovery result; If the result of the second link recovery is determined to indicate that the link recovery was successful, it is determined to continue using the target link to transmit data. Alternatively, if the result of the second link recovery is determined to indicate that the link recovery has failed, then it can be determined not to continue using the target link to transmit data.

[0102] In some alternative implementations, the routing switching module 630 is specifically used for: Based on the resource usage information of the other graphics processors, select the intermediate graphics processor from among the other graphics processors and obtain the routing information of the intermediate graphics processor; The routing register corresponding to the target graphics processor is determined in the switch; The routing information is written to the routing register so that the data stream corresponding to the target graphics processor can be switched to the intermediate graphics processor.

[0103] In some optional implementations, the acquisition module 610 is specifically used for: Periodically send test data packets to the target graphics processor; If no valid response packet corresponding to the test data is received within a preset time period after the test data packet is sent, the count value of the target parameter is increased by a first preset value. Alternatively, after a preset time period following the sending of the test data packet, upon receiving a valid response packet, the count value of the target parameter is initialized to a second preset value. At the moment the fault detection operation is triggered, the latest count value of the target parameter is determined as the target count value.

[0104] In some alternative embodiments, the device further includes a setting module 640 for: If the fault type is determined to be link signal attenuation, obtain the current original data transmission rate and original bandwidth of the target link; If the original data transmission rate is not the smallest among a variety of preset data transmission rates, then a data transmission rate smaller than the original data transmission rate is selected as the target data transmission rate from among a variety of data transmission rates, based on the original data transmission rate. Set the data transmission rate of the target link to the target data transmission rate; Furthermore, the target bandwidth is determined based on the original data transmission rate, the original bandwidth, and the target data transmission rate; Set the target link bandwidth to the target bandwidth.

[0105] For a description of the features in the embodiment corresponding to the link failure handling device, please refer to the relevant description in the embodiment corresponding to the link failure handling method, which will not be repeated here.

[0106] The aforementioned acquisition module 610, determination module 620, route switching module 630, and setting module 640 can be modules that have been reclassified and renamed from the aforementioned fault diagnosis classification module, error correction and recovery module, routing and redirection module, and optimization module.

[0107] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described methods for handling link failures.

[0108] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0109] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for handling link failures.

[0110] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described methods for handling link failures.

[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] The foregoing has provided a detailed description of the link failure handling method, apparatus, monitor, server, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for handling link failures, characterized in that, include: At the trigger time of the fault detection operation, the target count value of the target parameter and the target performance data of the signal transmitted by the target link are obtained. The target parameter is used to record the number of consecutive failures of the target link in transmitting test data packets. The target link is the link between the central processing unit and the target graphics processor. Based on the target count value and the target performance data, the fault type of the target link is determined, wherein the fault type is any one of link interruption, link signal attenuation, and signal error. If the fault type is determined to be link interruption or link signal attenuation, it is directly determined that the target link should not be used to transmit data. Alternatively, if the fault type is determined to be the signal error, at least one error correction result corresponding to the transmitted data is obtained from the target graphics processor; If any of the error correction results is determined to indicate successful error correction, it is determined to continue using the target link to transmit data. Alternatively, if it is determined that all of the error correction results in at least one of the error correction results are used to indicate error correction failure, it is determined that the target link should not be used to transmit data. If it is determined that the target link will not be used to transmit data, the resource usage information of the other graphics processors among the multiple graphics processors, excluding the target graphics processor, is obtained; Based on the resource usage information corresponding to the other graphics processors, a routing switch operation is performed on the data stream corresponding to the target graphics processor.

2. The link failure handling method according to claim 1, characterized in that, The target performance data includes signal strength; determining the fault type of the target link based on the target count value and the target performance data includes: If the signal strength is determined to be equal to a first preset threshold, the link interruption is identified as a fault type of the target link; Alternatively, if it is determined that the signal strength is greater than the first preset threshold and less than the second preset threshold, and the target count value is greater than or equal to the third preset threshold, the link signal attenuation is determined as the fault type of the target link; Alternatively, if the signal strength is determined to be greater than the first preset threshold, less than the second preset threshold, and the target count value is less than the third preset threshold, the signal error can be determined as a fault type of the target link.

3. The link failure handling method according to claim 1, characterized in that, The method further includes: If the fault type is determined to be link interruption or link signal attenuation, a link recovery operation is performed on the target link, and a first link recovery result is obtained; If the result of the first link recovery is determined to indicate that the link recovery was successful, it is determined to continue using the target link to transmit data. Alternatively, if the result of the first link recovery is determined to indicate that the link recovery has failed, it can be further determined not to continue using the target link to transmit data.

4. The link failure handling method according to claim 1, characterized in that, The method further includes: If all error correction results in at least one of the error correction results are used to indicate error correction failure, a link recovery operation is performed on the target link, and a second link recovery result is obtained; If the second link recovery result is determined to indicate successful link recovery, it is determined to continue using the target link for data transmission; Alternatively, if the second link recovery result is determined to indicate link recovery failure, then it is determined not to continue using the target link to transmit data.

5. The method for handling link failures according to any one of claims 1 to 3, characterized in that, The step of performing a routing switch operation on the data stream corresponding to the target graphics processor based on the resource usage information corresponding to the other graphics processors includes: Based on the resource usage information corresponding to the other graphics processors, an intermediate graphics processor is selected from the other graphics processors, and the routing information of the intermediate graphics processor is obtained. The routing register corresponding to the target graphics processor is determined in the switch; The routing information is written into the routing register so that the data stream corresponding to the target graphics processor is switched to the intermediate graphics processor.

6. The method for handling link failures according to any one of claims 1 to 3, characterized in that, The acquisition of the target count value of the target parameter includes: Test data packets are periodically sent to the target graphics processor; If no valid response packet corresponding to the test data is received within a preset time period after the test data is sent, the count value of the target parameter is increased by a first preset value. Alternatively, after receiving the valid response packet within a preset time period following the sending of the test data packet, the count value of the target parameter is initialized to a second preset value. At the trigger time of the fault detection operation, the latest count value of the target parameter is determined as the target count value.

7. The link failure handling method according to claim 3, characterized in that, The method further includes: If the fault type is determined to be link signal attenuation, the current original data transmission rate and original bandwidth of the target link are obtained; If it is determined that the original data transmission rate is not the smallest among a variety of preset data transmission rates, then a data transmission rate smaller than the original data transmission rate is selected as the target data transmission rate from among the various data transmission rates, based on the original data transmission rate. Set the data transmission rate of the target link to the target data transmission rate; And, based on the original data transmission rate, the original bandwidth, and the target data transmission rate, the target bandwidth is determined; Set the bandwidth of the target link to the target bandwidth.

8. A monitor, characterized in that, The monitor is used to perform the steps of the link failure handling method as described in any one of claims 1 to 7.

9. A server, characterized in that, The server includes a central processing unit, a monitor, a switch, and multiple graphics processors; The central processing unit is connected to the monitor, the monitor is connected to the switch, the switch is connected to multiple graphics processors, and any two graphics processors are interconnected. The monitor is used to perform the steps of the link failure handling method as described in any one of claims 1 to 7.

Images