Troubleshooting methods and computing equipment
By sending heartbeat detection requests to PCIe devices to read the quadruple information, the timeliness problem of PCIe device fault detection is solved, and timely isolation of faulty devices is achieved, ensuring the stable operation of the system.
Patent Information
- Application Number
- CN202410140337.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-31
AI Technical Summary
When a PCIe device malfunctions, existing technologies cannot detect and isolate the faulty device in a timely manner, leading to system failure.
By sending a heartbeat detection request to the PCIe device, the device's quadruple information is read, and if the result is abnormal, the faulty device is identified and isolated.
This improved the timeliness and accuracy of fault detection, prevented system crashes, and ensured the normal operation of the system.
Smart Images

Figure CN118051366B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to fault handling methods and computing devices. Background Technology
[0002] With the widespread use of peripheral component interconnect express (PCIe) devices on computing devices, a failure in a PCIe link can cause other PCIe devices to fail in addition to the PCIe device itself.
[0003] Currently, when a PCIe device malfunctions, it reports a downstream port containment (DPC) interrupt. After the computing device receives the interrupt, it can process the PCIe device that sent the interrupt. This leads to a situation where the malfunctioning PCIe device has already affected the system operation before it can be processed, thus causing the system to fail. Summary of the Invention
[0004] This application provides a fault handling method and computing device, which improves the timeliness of identifying faulty PCIe devices, thereby ensuring the normal operation of the system.
[0005] In a first aspect, this application provides a fault handling method, which includes: sending a heartbeat detection request to a target PCIe device; the heartbeat detection request is used to request the target PCIe device to read specified information of the target PCIe device; the specified information is one or more of four-tuple information; receiving a request response returned by the target PCIe device; the request response includes the result of the target PCIe device reading the specified information; if the result of the target PCIe device reading the specified information indicates an abnormality, determining that the target PCIe device has failed, and isolating the target PCIe device.
[0006] Understandably, by actively sending heartbeat detection requests to the target PCIe device, the target PCIe device is prompted to read specified information from the four-tuple according to the heartbeat detection request, and the result of reading the specified information is reported. This allows for timely identification of a faulty target PCIe device when abnormal information is read, and ensures that the faulty target PCIe device can be isolated, preventing system crashes due to target PCIe device failure, thereby improving system performance.
[0007] In one possible implementation, before sending a heartbeat detection request to the target PCIe device, the method further includes: obtaining the class identification identifier of each PCIe device; wherein, different types of PCIe devices correspond to different class identification identifiers; different types of PCIe devices include PCIe Switch devices or PCIe Terminal EP devices; and determining the target PCIe device from among the various PCIe devices based on the class identification identifier.
[0008] Understandably, by querying the class identification identifier of each PCIe device, the type of each PCIe device can be determined, that is, whether the PCIe device is a PCIe Switch device or a PCIe EP device. Then, according to the type of PCIe device that needs to be fault detected, the target PCIe device can be identified from the various PCIe devices to realize the subsequent heartbeat detection process of the target PCIe device.
[0009] In one possible implementation, if the target PCIe device includes multiple PCIe switch devices, sending a heartbeat detection request to the target PCIe device includes: sending heartbeat detection requests to multiple PCIe switch devices in parallel via multi-threading.
[0010] Understandably, if the target PCIe device includes multiple PCIe switch devices, heartbeat detection can be performed on multiple PCIe switch devices simultaneously to identify the faulty PCIe switch device among multiple PCIe switch devices in parallel, thereby improving the efficiency of identifying faults in the PCIe link.
[0011] In one possible implementation, sending a heartbeat detection request to the target PCIe device includes: sending a heartbeat detection request to the target PCIe device at a specified period; if the result of the target PCIe device reading the specified information indicates an abnormality, determining that the target PCIe device has failed includes: if the number of consecutive abnormal results of the same target PCIe device reading the specified information exceeds a specified threshold, determining that the target PCIe device has failed.
[0012] Understandably, by periodically sending heartbeat detection requests to the target PCIe device, it is possible to receive the response to each subsequent heartbeat detection request. If the number of times the results of continuously reading the specified information indicate an anomaly exceeds a specified threshold, it is possible to accurately determine that the target PCIe device has malfunctioned, thus improving the accuracy of fault determination.
[0013] In one possible implementation, if the target PCIe device includes a PCIe Switch device, the target PCIe device is isolated, including: batch isolating the target PCIe device and each PCIe device connected to the downstream port of the target PCIe device.
[0014] Understandably, since PCIe Switch devices have downstream ports that are connected to other PCIe devices, once a PCIe Switch device is confirmed to be faulty, the PCIe Switch device and all the PCIe devices connected to its downstream ports can be isolated in batches to ensure that the system can continue to operate normally.
[0015] In one possible implementation, the target PCIe device and the various PCIe devices connected to its downstream port are subjected to batch isolation processing, including: determining the various PCIe devices connected to the downstream port of the target PCIe device according to the correspondence; the correspondence is used to indicate the PCIe devices connected to the downstream port of each PCIe switch device respectively; and performing batch isolation processing on the target PCIe device and the various PCIe devices connected to its downstream port.
[0016] Understandably, by establishing a correspondence, the downstream port of each PCIe Switch can be identified as being connected to a specific PCIe device. This allows for the rapid and accurate identification of the PCIe device connected to the downstream port after identifying the faulty PCIe Switch, thereby improving the effectiveness of isolation procedures.
[0017] In one possible implementation, before determining the PCIe devices connected to the downlink port of the target PCIe device according to the correspondence, the method further includes: determining the downlink port of each PCIe switch device; determining the PCIe devices connected to each downlink port; generating a correspondence based on the PCIe devices connected to the downlink port of each PCIe switch device; and writing the correspondence into memory.
[0018] Understandably, before performing heartbeat detection, the downstream ports of each PCIe Switch device can be obtained in advance, and the PCIe devices connected to each downstream port can be determined. This generates a correspondence between each PCIe Switch device and the PCIe devices connected to its downstream port, and the correspondence is written into memory. This allows for accurate and rapid identification of other PCIe devices that need to be isolated, in addition to the PCIe Switch device, based on the correspondence if a PCIe Switch device is found to be faulty.
[0019] In one possible implementation, if the target PCIe device also includes a PCIe terminal EP device; sending a heartbeat detection request to the target PCIe device includes: sending a heartbeat detection request to a first PCIe EP device; the first PCIe EP device is a PCIe EP device other than the PCIe EP device connected to the downlink port of the PCIe Switch device.
[0020] Understandably, in addition to sending heartbeat requests to the PCIe Switch device to determine if the PCIe Switch device is faulty, it is also possible to send heartbeat requests to other PCIe EP devices besides those connected to the PCIe Switch device's downlink port. This ensures the completeness of fault detection on the PCIe link and avoids repeatedly performing fault detection on the same PCIe device.
[0021] In one possible implementation, the quadruple information includes device identity, manufacturer identity, subsystem identifier, or subsystem manufacturer identifier.
[0022] It is understandable that, since the quadruple information is an identifier used by each PCIe device to indicate its identity, a PCIe device can read its own quadruple information when it is running normally. Therefore, if a PCIe device encounters an error when reading one or more of the quadruple information, it can be determined that the PCIe device has malfunctioned.
[0023] Secondly, embodiments of this application provide a fault handling apparatus for executing any of the fault handling methods provided in the first aspect above.
[0024] In one possible implementation, this application can divide the fault handling device into functional modules according to the method provided in the first aspect above. For example, each function can be divided into a separate functional module, or two or more functions can be inherited into a single processing module. For example, this application can divide the fault handling device into a sending module, an acquisition module, and a processing module according to their functions. The descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the first aspect above or its corresponding possible implementations, and will not be repeated here.
[0025] Thirdly, embodiments of this application provide a computing device including a processor and a memory for storing computer program instructions, which are loaded and executed by the processor to enable the computing device to implement the fault handling method as described above.
[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing at least one computer program instruction, which is loaded and executed by a processor to implement the fault handling method described above.
[0027] Fifthly, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform the fault handling methods provided in the various optional implementations of the above aspects.
[0028] For a detailed description of the second to fifth aspects and their various implementations in this application, please refer to the detailed description in the first aspect and its various implementations; and for a detailed analysis of the beneficial effects of the second to fifth aspects and their various implementations in the first aspect and its various implementations, please refer to the beneficial effect analysis in the first aspect and its various implementations, which will not be repeated here.
[0029] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description
[0030] Figure 1 This is a topology diagram of a PCIe bus according to an exemplary embodiment;
[0031] Figure 2 yes Figure 1 A schematic diagram of a PCIE tree structure involved in the illustrated embodiment;
[0032] Figure 3 yes Figure 1 The illustrated embodiment is a schematic diagram of the structure of a fault handling system.
[0033] Figure 4 This is a flowchart illustrating a fault handling method according to an exemplary embodiment;
[0034] Figure 5 yes Figure 4 The illustrated embodiment is a schematic diagram of a PCIe link fault handling process.
[0035] Figure 6 yes Figure 4 The illustrated embodiment presents a schematic diagram of a PCIe link failure scenario.
[0036] Figure 7 This is a schematic diagram of the structure of a fault handling device provided in an exemplary embodiment of this application;
[0037] Figure 8 This is a schematic diagram of the structure of a computing device provided in an exemplary embodiment of this application. Detailed Implementation
[0038] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0039] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0040] First, some concepts involved in the fault handling method and computing device provided in the embodiments of this application will be explained:
[0041] PCIe devices are hardware devices connected via slots on the PCIe bus. PCIe devices can include the following three types: root complex (RC) devices, PCIe switch devices, and PCIe endpont (PCIe EP) devices.
[0042] The RC device is used to parse and generate PCIe packets. The RC device receives IO instructions from the CPU to generate corresponding PCIe packets, or receives PCIe TLP packets from the PCIe device, parses them, and transmits the data to the CPU or memory.
[0043] PCIe switch devices can be used to expand the PCIe bus. Since the PCIe bus adopts an end-to-end connection method, only one device can be connected to each end of each PCIe link. If more PCIe devices need to be connected, a PCIe switch device can be used. The PCIe switch device can have one upstream port and multiple downstream ports.
[0044] PCIe endpoint devices can be leaf nodes in a PCIe tree structure. PCIe endpoint devices can include network interface cards (NICs), graphics processing units (GPUs), sound cards, graphics cards, smart NICs with independent operating systems, smart GPUs, etc.
[0045] Next, the application scenarios of the embodiments of this application will be described by way of example.
[0046] Figure 1 This diagram illustrates a PCIe bus topology according to an exemplary embodiment of this application. Figure 1 As shown, the computing device 10 includes a CPU and various types of PCIe devices. The CPU can be connected to an RC device, the RC device can be directly connected to a PCIe EP device or a PCIe switch device, and the PCIe switch device can connect to multiple PCIe EP devices or other PCIe switch devices.
[0047] For example, Figure 2 This is a schematic diagram of a PCIE tree structure according to an embodiment of this application. Figure 2 As shown, a PCIe switch device can include uplink ports and downlink ports. For example, PCIe switch device 1 connects to the downlink ports of other PCIe devices via its uplink port and uplink link; PCIe switch device 1 connects to the uplink port of PCIe switch device 2 via downlink port 1 and downlink link 1; PCIe switch device 1 connects to the uplink port of PCIe EP device 3 via downlink port 2 and downlink link 2. The uplink link of PCIe switch device 2 and the downlink link 1 of PCIe switch device 1 are the same PCIe link. PCIe switch device 2 can connect to both PCIe EP device 1 and PCIe EP device 2 via its downlink port, thus forming a PCIe tree structure.
[0048] When a PCIe link in a PCIe tree structure fails, it is possible that in addition to the PCIe devices in the failed PCIe link malfunctioning, it may also cause PCIe devices in other PCIe links to fail. This could not only affect the performance of services, but may even lead to system failure.
[0049] In some implementations, when a PCIe device in a computing device fails, it can send a DPC interrupt to the CPU. The DPC mechanism allows the operating system or DPC driver to handle the DPC interrupt, thereby preventing the failure caused by that PCIe device from propagating and affecting other PCIe devices.
[0050] However, the aforementioned implementation of the CPU to identify faulty PCIe devices requires both the PCIe device and the operating system to support the DPC driver. Therefore, on the one hand, since DPC is a mechanism for detecting and recovering from faults in the downstream ports of PCIe devices, if there are PCIe devices in the environment with abnormal upstream ports, DPC alone cannot detect the fault and achieve the goal of batch isolation of faulty PCIe devices. On the other hand, because by the time a PCIe device reports a DPC interrupt to the CPU, the PCIe device has already failed, or even been failing for some time, the passive uploading of PCIe status has a certain lag. In this case, due to the lag in fault detection and isolation operations, the operating system may already be experiencing runtime errors.
[0051] In view of this, the following embodiments of this application also provide a fault handling method. A computing device can actively send a heartbeat detection request to a PCIe device via the CPU, causing the PCIe device to read one or more of its own four-tuple information according to the received heartbeat detection request. The PCIe device then feeds back the reading result to the CPU. If the result indicates an anomaly, it can be determined that the PCIe device has failed, and the PCIe device can be isolated. Since the CPU can actively send a heartbeat detection request to the PCIe device, causing the PCIe device to read the four-tuple information, if the PCIe device is malfunctioning, the result of the PCIe device reading the four-tuple information according to the heartbeat detection request will be abnormal. This achieves the process of the CPU actively detecting whether the PCIe device is faulty, avoiding the problem of delay in fault determination caused by waiting for the PCIe device to generate and report an interrupt after a fault occurs. This ensures the efficiency of fault handling, timely isolation of the faulty PCIe device and related PCIe devices, and reduces the risk of PCIe device failure affecting the normal operation of the operating system.
[0052] Figure 3 This is a schematic diagram of the structure of a fault handling system according to an embodiment of this application. Figure 3As shown, this fault handling system can run on computing device 10, which may include a CPU, memory, and different types of PCIe devices, including RC devices, PCIe switch devices, and PCIe EP devices. Each type of PCIe device includes registers for storing the PCIe configuration space.
[0053] The PCIe configuration space can be divided into two types: type0 and type1. The PCIe configuration space in the registers of a PCIe EP device can be of type0; the PCIe configuration space in the registers of an RC device and a PCIe switch device can be of type1.
[0054] The PCIe configuration space of type 0 includes Device ID, Version ID, Subsystem Vendor ID, and Subsystem ID; the PCIe configuration space of type 1 may include Device ID and Version ID.
[0055] When a PCIe device is running normally, each PCIe device can obtain the quadruple information contained in its own PCIe configuration space. The quadruple information can include Device ID, Version ID, Subsystem Vendor ID, and Subsystem ID.
[0056] The PCIe device's four-tuple information can be used to identify the PCIe device. Device ID is a code that identifies the device itself, distinguishing different models of the same type of device. Version ID is an identifier representing the technology vendor, a unique vendor identifier. Subsystem Vendor ID is an identifier for the secondary manufacturer, also a unique vendor identifier. The Subsystem ID is a secondary code for the licensed manufactured device; it is not the original technology vendor's code, but rather the code for the secondary manufacturer's contracted device.
[0057] Since the quadruple tuple information is used to identify a PCIe device, the device can only function properly if this quadruple tuple information can be correctly recognized. In other words, by recognizing the quadruple tuple information, the CPU can guide the operating system to detect the PCIe device's status, discover the device, and install different versions of drivers to ensure the PCIe device operates normally and securely within the system platform. Therefore, if the PCIe device malfunctions, the CPU cannot read the correct quadruple tuple information.
[0058] Optionally, the CPU can send heartbeat detection requests to each PCIe device. Upon receiving a heartbeat detection request, the PCIe device retrieves specified information from the PCIe configuration space according to the heartbeat detection request. This specified information can be one or more of DeviceID, Version ID, Subsystem Vendor ID, and Subsystem ID. The retrieved result is then returned to the CPU. If the result indicates an anomaly, it can be determined that the corresponding PCIe device has failed, thereby isolating the PCIe device.
[0059] Figure 4 A schematic flowchart of a fault handling method provided in an exemplary embodiment of this application is shown. This fault handling method can be applied to a computing device; for example, the computing device may be... Figure 1 The computing device 10 shown. The fault handling method includes the following steps:
[0060] S101, send a heartbeat detection request to the target PCIe device.
[0061] In this embodiment of the application, the computing device sends a heartbeat detection request to the target PCIe device via the CPU.
[0062] The heartbeat detection request can be used to request the target PCIe device to read specified information from the target PCIe device. This execution information can be one or more of the four-tuple information.
[0063] Optionally, the quadruple information may include Device ID, Version ID, Subsystem Vendor ID, or Subsystem ID.
[0064] The quadruple information of a PCIe device is stored in the PCIe configuration space of the PCIe device, and the quadruple information stored in the PCIe configuration space can be different for different types of PCIe devices. For example, the PCIe configuration space of RC devices and PCIe switch devices includes Device ID and Version ID; the PCIe configuration space of PCIe EP devices includes Device ID, Version ID, Subsystem Vendor ID, and Subsystem ID.
[0065] In other words, if the target PCIe device includes an RC device or a PCIe switch device, the specified information may include one or more of the Device ID and Version ID; if the target PCIe device includes a PCIe EP device, the specified information may include one or more of the Device ID, Version ID, Subsystem Vendor ID, and Subsystem ID.
[0066] In one possible implementation, the computing device can send heartbeat detection requests to the target PCIe device via the CPU at specified intervals.
[0067] One method involves setting a timer task in the computing device to send heartbeat detection requests to the target PCIe device at specified intervals.
[0068] In other words, the computing device can periodically send heartbeat detection requests to the target PCIe device according to a timer task, so as to achieve the purpose of periodic fault detection of the target PCIe device.
[0069] For example, a timer task can instruct the CPU to send a heartbeat request to the target PCIe device every 0.5 seconds, which means that every 0.5 seconds the target PCIe device reads the specified information from the quadruple information stored in its own PCIe configuration space.
[0070] In one possible scenario, the target PCIe device may include an RC device. Since all the PCIe switch devices and PCIe EP devices in the computing device are connected to the downlink of the RC device, in order to save time and cost in fault detection, the RC device can be prioritized for fault detection. That is, the computing device can send a heartbeat detection request to the RC device through the CPU.
[0071] For example, such as Figure 4As shown, if the computing device sends a heartbeat request to the RC device via the CPU, it can determine whether the RC device has failed. If the RC device is determined to be faulty, the various PCIe devices connected to the downlink of the RC device can be directly isolated. Since the various PCIe devices connected to the downlink of the RC device (PCIe switch device 1, PCIe switch device 2, PCIe EP device 3 connected to the downlink port of PCIe switch device 1, PCIe EP device 1 connected to the downlink port of PCIe switch device 2, PCIe EP device 2 connected to the downlink port of PCIe switch device 1, and PCIe EP device 2 and PCIe EP device 4 connected to the downlink port of the RC device) are all the PCIe devices in the computing device, no further fault detection is needed. If the RC device is determined not to be faulty, the computing device can continue to send heartbeat requests to each PCIe switch device to complete the fault detection of each PCIe switch device.
[0072] In one possible scenario, the target PCIe device may include a PCIe switch device. Since a PCIe switch device has one uplink port and multiple downlink ports, after detecting and determining that the PCIe switch device is faulty, it can be determined that the uplink port link of the PCIe switch device is faulty, and the downlink port link of the PCIe switch device is also faulty.
[0073] If the target PCIe device includes a PCIe switch device, and the computing device contains multiple PCIe switch devices, the computing device can use multi-threaded CPU to send heartbeat detection requests to multiple PCIe switch devices in parallel. This enables parallel fault detection of each PCIe switch device, saving fault detection time.
[0074] For example, such as Figure 4As shown, if the computing device determines that the RC device is not faulty, it sends heartbeat requests to each PCIe switch device (PCIe switch device 1 and PCIe switch device 2) to determine whether each PCIe switch device is faulty. If PCIe switch device 1 is determined to be faulty, it can then directly isolate PCIe switch device 1 and the PCIe devices connected to its downlink. This is because the PCIe devices connected to the downlink of PCIe switch device 1 include PCIe switch device 2 connected to its downlink port, PCIe EP devices 1 and 2 connected to its downlink port, and PCIe EP device 3 connected to its downlink port. After batch isolating PCIe switch device 1 and the PCIe devices connected to its downlink, the computing device can continue to send heartbeat requests to the undetected and unisolated PCIe EP device 4 to complete the fault detection of each PCIe device in the computing device. In another possible scenario, the target PCIe device may also include a PCIe EP device. This PCIe EP device can be directly connected to the RC device, or it can be a PCIe EP device connected to the downstream port of a PCIe switch. Since a PCIe EP device has one upstream port but no downstream port, if it fails in the PCIe link, devices without a downstream link will be affected. Furthermore, since the PCIe EP device connected to the downstream port of the failed PCIe switch has already been processed during the PCIe switch fault heartbeat detection process, after detecting the PCIe switch fault, if a PCIe EP device is identified as not being isolated, this can include PCIe EP devices not detected connected to the downstream port of the failed PCIe switch, or PCIe EP devices directly connected to the RC device. A heartbeat request is then sent to these PCIe EP devices.
[0075] In other words, the computing device first uses CPU multithreading to send heartbeat requests to multiple PCIe switch devices in parallel. After determining whether each PCIe switch device has failed based on the heartbeat requests, since the downstream ports of the PCIe switch devices may be connected to PCIe EP devices, the computing device can perform batch isolation of the determined failed PCIe switch devices and the various PCIe EP devices connected to their downstream ports. Then, the computing device can use CPU multithreading to send heartbeat requests to multiple non-isolated PCIe EP devices in parallel to determine whether any of the non-isolated PCIe EP devices are faulty, thereby isolating the faulty PCIe EP devices.
[0076] For example, such as Figure 4 As shown, the computing device first sends heartbeat requests in parallel to PCIe switch device 1 and PCIe switch device 2. Based on these requests, it determines whether PCIe switch device 1 and PCIe switch device 2 are faulty. If PCIe switch device 1 is not faulty, but PCIe switch device 2 is, the computing device can perform batch isolation of PCIe switch device 2, PCIe EP device 1 connected to the downstream port of PCIe switch device 2, and PCIe EP device 2. Then, the computing device sends heartbeat requests to the unisolated PCIe EP devices, namely PCIe EP device 3 and PCIe EP device 4, to determine whether they are faulty. Similarly, if PCIe EP device 3 is determined to be faulty, it is isolated; if PCIe EP device 4 is determined to be faulty, it is isolated. If the target PCIe device may also include a PCIe EP device, a heartbeat detection request is sent to the first PCIe EP device. The first PCIe EP device may be any other PCIe EP device besides the PCIe EP device that has been marked as isolated.
[0077] For example, such as Figure 4As shown, if the computing device first sends heartbeat requests to each PCIe switch device and RC device, and determines that PCIe switch device 1 has failed, the computing device can isolate PCIe switch device 1 and the PCIe devices connected to its downstream port. That is, the computing device can isolate PCIe switch device 1, PCIe switch device 2, PCIe EP device 1, PCIe EP device 2, and PCIe EP device 3. Since the computing device has not yet detected whether a PCIe EP device has failed, it can then send a heartbeat request to the first unisolated PCIe EP device (e.g., PCIe EP device 4) to complete fault detection for all PCIe devices in the computing device. Similarly, if the target PCIe device includes PCIe EP devices, and the computing device contains multiple unisolated PCIe EP devices, the computing device can use CPU multi-threading to send heartbeat requests to multiple PCIe EP devices in parallel. This allows for parallel fault detection of each PCIe EP device, saving fault detection time.
[0078] S102, Receive the request response returned by the target PCIe device.
[0079] In this embodiment of the application, the computing device can receive request responses returned by each target PCIe device through the CPU.
[0080] The response to the request may include the result of the target PCIe device reading the specified information.
[0081] For example, if the target PCIe device includes a PCIe switch device or an RC device, the specified information may include one or more of Device ID and Version ID. The request response returned by the PCIe switch device or RC device may be a specific value of the specified information, or it may be a null value or an abnormal value. If the target PCIe device includes a PCIe EP device, the specified information may include one or more of Device ID, Version ID, Subsystem Vendor ID, and Subsystem ID. The request response returned by the PCIe EP device may be a specific value of the specified information, or it may be a null value or an abnormal value.
[0082] Specifically, when the PCIe switch or RC device returns a response containing the specified value, it indicates that the PCIe switch or RC device successfully read the specified information, meaning that the PCIe switch or RC device is not malfunctioning. When the PCIe switch or RC device returns a null or abnormal value, it indicates that the PCIe switch or RC device failed to read the specified information, meaning that the PCIe switch or RC device's heartbeat detection indication is abnormal, indicating that the PCIe switch or RC device may be malfunctioning. Similarly, when the PCIe EP device returns a response containing the specified value, it indicates that the PCIe EP device successfully read the specified information, meaning that the PCIe EP device is not malfunctioning. When the PCIe EP device returns a null or abnormal value, it indicates that the PCIe EP device failed to read the specified information, meaning that the PCIe EP device's heartbeat detection indication is abnormal, indicating that the PCIe EP device may be malfunctioning.
[0083] For example, when a computing device receives a 0xffff response from a PCIe switch device via its CPU, it can be determined that the PCIe switch device's heartbeat detection indication is abnormal, indicating that the PCIe switch device may be malfunctioning.
[0084] In one possible implementation, if the computing device sends a heartbeat detection request to the target PCIe device via the CPU at a specified cycle, and if the number of consecutive times the result of reading specified information by the same target PCIe device indicates an anomaly exceeds a specified threshold, it is determined that the target PCIe device has failed.
[0085] In other words, if the computing device sends a heartbeat request to the PCIe switch device every 0.5 seconds via the CPU, and the PCIe switch device receives the first heartbeat request and reads the specified information according to the result of the first heartbeat request, then the PCIe switch device can continue to receive a second heartbeat request and read the specified information according to the result of the second heartbeat request. If the result is still abnormal, then the PCIe switch device can continue to receive a third heartbeat request and read the specified information according to the result of the third heartbeat request. If the result is still abnormal, then it can be determined that the PCIe switch device has failed.
[0086] S103, if the result of the target PCIe device reading the specified information is abnormal, determine that the target PCIe device has failed, and isolate the target PCIe device.
[0087] In this embodiment of the application, if the result of the target PCIe device reading the specified information obtained by the computing device through the CPU indicates an abnormality, under certain circumstances, it can be determined that the target PCIe device has failed and the target PCIe device can be isolated.
[0088] In one possible implementation, if the number of times the result of reading specified information from the same target PCIe device continuously indicates an anomaly exceeds a specified threshold, it can be determined that the target PCIe device has malfunctioned.
[0089] If the target PCIe device includes a PCIe switch device or an RC device, the computing device can use the CPU to perform batch isolation of the target PCIe device and the various PCIe devices connected to its downstream ports. If the target PCIe device includes a PCIe EP device, the computing device can use the CPU to isolate the target PCIe device.
[0090] In one possible scenario, the computing device can obtain the correspondence through the CPU, and determine the various PCIe devices connected to the downlink port of the target PCIe device according to the correspondence. This enables the process of determining the various PCIe devices connected to the downlink port of the PCIe switch device after identifying the faulty PCIe switch device, thus facilitating the process of batch isolation.
[0091] The aforementioned mapping can be used to indicate the PCIe devices connected to the downstream ports of each PCIe switch device. This mapping can be established after the computing device powers on by acquiring the class identification identifiers of each PCIe device; different types of PCIe devices correspond to different class identification identifiers; different types of PCIe devices can include PCIe switch devices or PCIe terminal block (EP) devices. Then, the computing device, through its CPU, can determine the target PCIe device from among the various PCIe devices based on the class identification identifiers. That is, after determining the PCIe switch device from among the various PCIe devices, the downstream port of each PCIe switch device can be determined, the PCIe devices connected to each downstream port can be determined, and then a mapping can be generated based on the PCIe devices connected to the downstream ports of each PCIe switch device. This mapping is then written into memory for the CPU to access.
[0092] For example, Figure 5This is a schematic diagram of a PCIe link fault handling process according to an embodiment of this application. Figure 5 As shown, the steps for a computing device to handle PCIe link faults are as follows.
[0093] S11, The computing device determines the PCIe switch device.
[0094] After the computing device is powered on and starts running, it can obtain the class identifier (ClassID) of each PCIe device through the CPU. For example, the ClassID of each PCIe switch device is 0x0604. Then, the type of each PCIe device can be determined based on the ClassID, thereby identifying the PCIe switch device.
[0095] S12, The computing device creates an uplink port object.
[0096] The computing device determines the type of PCIe device as a PCIe switch device according to the ClassID indication, obtains the uplink port of each PCIe switch device, and creates an object for each uplink port, that is, assigns a value to each uplink port.
[0097] For example, with Figure 4 For example, the computing device can determine that it includes PCIe switch device 1 and PCIe switch device 2. The uplink port of PCIe switch device 1 can be assigned the value 'a', and the uplink port of PCIe switch device 2 can be assigned the value 'b'.
[0098] S13, The computing device creates a list of downlink port objects.
[0099] The computing device can traverse the downlink ports of each PCIe switch device and create an object list.
[0100] Since a PCIe switch device may have multiple downstream ports and multiple PCIe devices connected to them, it is necessary to form a dictionary with one-to-one correspondence between the upstream port objects and the downstream port object lists, forming a one-to-many correspondence. That is, an upstream port of a PCIe switch device corresponds to multiple PCIe devices connected to its downstream ports.
[0101] For example, with Figure 4For example, the computing device can iterate through the downstream ports of PCIe switch device 1 and PCIe switch device 2, assigning values m and n to the downstream ports of PCIe switch device 1, and assigning values x and y to the downstream ports of PCIe switch device 2. The correspondence can include upstream port a corresponding to downstream ports m and n, and upstream port b corresponding to downstream ports x and y.
[0102] S14, The computing device starts the heartbeat detection timer task.
[0103] The computing device can start a timer task to poll and run a detection task at regular intervals, which means sending heartbeat detection requests to each PCIe switch device.
[0104] For example, with Figure 4 For example, the computing device can send a heartbeat detection request to the uplink port a of PCIe switch device 1 and the uplink port b of PCIe switch device 2 every 0.5 seconds according to the timer task.
[0105] S15, the computing device determines whether the heartbeat of each uplink port is abnormal.
[0106] One method the computing device can use to detect whether the uplink port receiving the heartbeat detection request has an abnormal heartbeat is to check whether the uplink port is abnormal by reading the vendorID / DeviceID in the configuration space of the PCIe switch device where the uplink port is located (for example, an abnormal value of vendorID / DeviceID is 0xffff). If the vendorID / DeviceID read by the PCIe switch device in the configuration space is an abnormal value, the cumulative count of abnormal heartbeats for the uplink port of that PCIe switch device is incremented by 1. If the cumulative number of abnormal heartbeats is consecutive and exceeds 3, the uplink port link can be marked as faulty. Otherwise, it is determined that the uplink port link has not failed.
[0107] S16, the computing device isolates the devices connected to the downlink ports corresponding to all uplink ports marked as faulty.
[0108] The computing device can obtain all downlink ports corresponding to the uplink port marked as faulty and the PCIe devices connected to each downlink port according to the correspondence recorded in S13. By marking all the connected PCIe devices, it can perform batch isolation operations on the PCIe devices connected to all downlink ports corresponding to the uplink port marked as faulty.
[0109] S17, the computing device determines whether the heartbeat of each downlink port is abnormal.
[0110] The computing device can traverse the list of downlink ports to detect whether the vendorID / DeviceID of each PCIE EP device connected to the PCIe switch is abnormal (for example, the abnormal value of vendorID / DeviceID is 0xffff). If the vendorID / DeviceID read by the PCIE EP device in the configuration space is abnormal, the cumulative count of abnormal heartbeats is incremented by 1. If the cumulative number of abnormal heartbeats is consecutive and exceeds 3 times, the downlink port link of the PCIe switch device can be marked as faulty.
[0111] S18, Compute device isolates all devices connected to downlink ports marked as faulty.
[0112] In this case, isolating all devices connected to downlink ports marked as faulty can be achieved by removing all PCIe EP devices connected to the downlink ports marked as faulty.
[0113] Then, the computing device can also determine the first PCIE EP device, which is the PCIE EP device directly connected to the downlink port of the RC device, according to the ClassID indication and whether it is in the corresponding relationship. The link status of the first PCIE EP device can be determined directly by heartbeat detection, which will not be elaborated here.
[0114] The above-described process for PCIe link fault detection can include the following three fault types and isolation handling scenarios. In the first scenario, the computing device can proceed as follows: Figure 5 Steps S11 to S18, as shown, detect whether the uplink or downlink link of the PCIe switch device is abnormal by sending a heartbeat request to the uplink or downlink port. When an uplink link abnormality is detected, the PCIe switch device and all PCIe devices in the downlink link are batch isolated. For example, ... Figure 6As shown, the computing device performs heartbeat detection on PCIe switch device 2. If an abnormal heartbeat is detected and the cumulative number of abnormal heartbeats is continuous and exceeds a specified threshold, it can be determined that PCIe switch device 2 has failed, i.e., the uplink port link is abnormal. Batch isolation can then be performed on PCIe switch device 2 and all PCIe devices connected to PCIe switch device 2 (e.g., FPGA, GPU, and other devices). Alternatively, the computing device performs heartbeat detection on all PCIe devices connected to PCIe switch device 1. If an abnormal FPGA heartbeat is detected and the cumulative number of abnormal heartbeats is continuous and exceeds a specified threshold, it can be determined that the FPGA has failed, i.e., the downlink port link is abnormal. The FPGA can then be isolated.
[0115] In the second scenario, the computing device can directly send a heartbeat request to the PCIe EP device to detect any anomalies in the link's end device, i.e., the PCIe EP device link. If an anomaly is detected, the PCIe EP device is isolated. For example, ... Figure 6 As shown, the computing device can perform heartbeat detection on the RAID card, that is, send a heartbeat detection request to the RAID card. If it is determined that the RAID card heartbeat is abnormal and the cumulative abnormal heartbeats are continuous and exceed the specified threshold, it can be determined that the RAID card has failed. The RAID card can be removed directly or isolated.
[0116] In the third scenario, when a PCIe device in the computing device fails, it can actively send a DPC interrupt to the CPU. By monitoring DPC interrupts in real time and utilizing the DPC mechanism, the operating system or DPC driver can handle these interrupts, thus preventing the fault caused by that PCIe device from propagating and affecting other PCIe devices. This fault detection method, which involves loading a DPC driver and monitoring PCIe device DPC faults in real time, can serve as a fallback solution. After the computing device has completed PCIe link fault detection according to the first and second scenarios, to prevent omissions or errors in fault detection, the computing device can monitor whether the PCIe device is responding to a DPC interrupt in real time according to the third scenario. This ensures that the computing device can handle PCIe link faults promptly and avoid affecting the normal operation of the system.
[0117] In summary, the computing device can proactively send heartbeat detection requests to the PCIe device via the CPU, enabling the PCIe device to read the quadruple information. If the PCIe device malfunctions, the result of reading the quadruple information according to the heartbeat detection request will be abnormal. This allows the CPU to proactively detect whether the PCIe device is faulty, avoiding the delay in fault determination caused by waiting for the PCIe device to generate and report an interrupt after a fault occurs. This ensures efficient fault handling, timely isolation of faulty PCIe devices and related PCIe devices, and reduces the risk of PCIe device failures affecting the normal operation of the operating system.
[0118] The foregoing mainly describes the solutions of the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the fault handling device includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0119] This application embodiment can divide the fault handling device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0120] For example, Figure 7 A schematic diagram of a fault handling apparatus 200 provided in an exemplary embodiment of this application is shown. The fault handling apparatus 200 is applied in a computing device and includes:
[0121] The sending module 410 is used to send a heartbeat detection request to the target PCIe device; the heartbeat detection request is used to request the target PCIe device to read specified information of the target PCIe device; the specified information is one or more of the four-tuple information.
[0122] The receiving module 420 is configured to receive a request response returned by the target PCIe device; the request response includes the result of the target PCIe device reading the specified information;
[0123] The processing module 430 is configured to determine that the target PCIe device has malfunctioned if the result of the target PCIe device reading the specified information indicates an abnormality, and to isolate the target PCIe device.
[0124] In one possible implementation, the apparatus further includes: an acquisition module, configured to acquire a class identification identifier for each PCIe device before sending a heartbeat detection request to the target PCIe device; wherein different types of PCIe devices correspond to different class identification identifiers; the different types of PCIe devices include PCIe Switch devices or PCIe Terminal EP devices; and to determine the target PCIe device from among the various PCIe devices based on the class identification identifier.
[0125] In one possible implementation, if the target PCIe device includes multiple PCIe switch devices, the sending module 410 is further configured to send the heartbeat detection request to the multiple PCIe switch devices in parallel via multi-threading.
[0126] In one possible implementation, the sending module 410 is further configured to send the heartbeat detection request to the target PCIe device at a specified period.
[0127] The processing module 430 is further configured to determine that the target PCIe device has malfunctioned if the number of consecutive abnormal results of reading the specified information from the same target PCIe device exceeds a specified threshold.
[0128] In one possible implementation, if the target PCIe device includes a PCIe Switch device, the processing module 430 is further configured to perform batch isolation processing on the target PCIe device and each PCIe device connected to the downstream port of the target PCIe device.
[0129] In one possible implementation, the processing module 430 is further configured to determine, according to a correspondence, each PCIe device connected to the downlink port of the target PCIe device; the correspondence is used to indicate the PCIe devices connected to the downlink port of each PCIe switch device respectively.
[0130] The target PCIe device and the various PCIe devices connected to its downstream port are batch isolated.
[0131] In one possible implementation, the apparatus further includes: a generation module, configured to: determine the downlink port of each of the PCIESwitch devices; determine the PCIE devices connected to each of the downlink ports; generate the correspondence based on the PCIE devices connected to the downlink ports of each of the PCIESwitch devices; and write the correspondence into memory.
[0132] In one possible implementation, if the target PCIe device further includes a PCIe terminal EP device, the sending module 410 is further configured to send a heartbeat detection request to the first PCIe EP device; the first PCIe EP device is a PCIe EP device other than the PCIe EP device connected to the downlink port of the PCIe Switch device.
[0133] In one possible implementation, the quadruple information includes a device identity identifier, a manufacturer identity identifier, a subsystem identifier, or a subsystem manufacturer identifier.
[0134] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the fault handling devices provided above and the description of their beneficial effects can be found in the corresponding method embodiments described above, and will not be repeated here.
[0135] As an example, combined Figure 3 The functions implemented by some or all of the transmitting module 410, receiving module 420, and processing module 430 in the fault handling device can be achieved through... Figure 3 The CPU executes the program code.
[0136] Figure 8 This illustration shows a schematic diagram of a computing device 1100 provided in an exemplary embodiment of this application. The computing device 1100 can be an electronic device such as a server, smartphone, tablet computer, e-book reader, portable personal computer, or smart wearable device. The computing device 1100 in this application may include one or more of the following components: a processor 1110 and a memory 1120.
[0137] Processor 1110 may include one or more processing cores. Processor 1110 connects to various parts of the terminal using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1120, and by calling data stored in memory 1120. Optionally, processor 1110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1110 may integrate one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 1110 and may be implemented separately using a communication chip.
[0138] The memory 1120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1120 may include a non-transitory computer-readable storage medium. The memory 1120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc. The operating system may be an Android system (including systems deeply developed based on the Android system), an iOS system developed by Apple Inc. (including systems deeply developed based on the iOS system), or other systems. The data storage area may also store data created by the terminal during use (such as phonebook data, audio and video data, chat history data, etc.).
[0139] In addition, those skilled in the art will understand that the structure of the computing device 1100 shown in the above figures does not constitute a limitation on the computing device 1100. The computing device 1100 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, imaging components, sensors, audio circuits, wireless fidelity (WiFi) components, power supplies, Bluetooth components, etc., which will not be described in detail here.
[0140] This application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the PCIe device management method described in the above embodiments. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.
[0141] This application also provides a chip. The chip integrates a control circuit for implementing the functions of the aforementioned fault handling device and one or more ports. Optionally, the functions supported by the chip can be referred to above, and will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, random access memory, etc. The aforementioned processing unit or processor can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
[0142] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or may include one or more data storage devices such as servers or data centers that can be integrated with the medium. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.
[0143] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.
[0144] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A fault handling method, characterized in that, The method includes: A heartbeat detection request is sent to the target PCIe device; the heartbeat detection request is used to request the target PCIe device to read specified information of the target PCIe device; the specified information is one or more of the four-tuple information; the four-tuple information includes: device identity identifier, manufacturer identity identifier, subsystem identification code, and subsystem manufacturer identification code; the target PCIe device includes multiple PCIe adapter switch devices; Receive a request response returned by the target PCIe device; the request response includes the result of the target PCIe device reading the specified information; If the target PCIe device reads the specified information and the result indicates an abnormality, it is determined that the target PCIe device has malfunctioned. According to the correspondence, the downstream port of the target PCIe device is determined to be connected to each PCIe device; the correspondence is used to indicate the PCIe devices connected to the downstream port of each PCIe switch device respectively; wherein, generating the correspondence includes: determining the downstream port of each PCIe switch device; determining the PCIe devices connected to each downstream port; generating the correspondence based on the PCIe devices connected to the downstream port of each PCIe switch device; and writing the correspondence into memory; The target PCIe device and the various PCIe devices connected to its downstream port are batch isolated.
2. The method according to claim 1, characterized in that, Before sending the heartbeat detection request to the target PCIe device, the method further includes: Obtain the class identification identifier for each PCIe device; wherein, different types of PCIe devices correspond to different class identification identifiers; the different types of PCIe devices include PCIe Switch devices or PCIe Terminal EP devices; The target PCIe device is determined from the various PCIe devices based on the class identification identifier.
3. The method according to claim 1, characterized in that, Sending a heartbeat detection request to the target PCIe device includes: The heartbeat detection request is sent to the multiple PCIe Switch devices in parallel using multiple threads.
4. The method according to any one of claims 1 to 3, characterized in that, Sending a heartbeat detection request to the target PCIe device includes: The heartbeat detection request is sent to the target PCIe device at a specified period. If the result of the target PCIe device reading the specified information indicates an abnormality, determining that the target PCIe device has malfunctioned includes: If the number of consecutive abnormal results obtained from reading the specified information from the same target PCIe device exceeds a specified threshold, it is determined that the target PCIe device has malfunctioned.
5. The method according to claim 4, characterized in that, If the target PCIe device further includes a PCIe terminal device (EP device); the step of sending a heartbeat detection request to the target PCIe device includes: Send a heartbeat detection request to the first PCIe EP device; the first PCIe EP device is any PCIe EP device other than the PCIe EP device connected to the downlink port of the PCIe switch device.
6. A computing device, characterized in that, The computing device includes: a processor and a memory for storing processor-executable instructions; the processor is configured to execute the instructions, causing the computing device to perform the fault handling method as described in any one of claims 1-5.
Citation Information
Patent Citations
Task processing method, apparatus, and system
US20190324797A1
Failover method, device and system
WO2018076972A1
