Methods for Constructing Evidence Chains and Locating Faults in Cross-Layer Testing of Artificial Intelligence Computing Centers
By constructing a cross-layer resource dependency topology and a globally unified identifier, a cross-layer test evidence chain is generated, which solves the problems of fragmented test data and fault diagnosis in AI computing centers, and achieves efficient and accurate fault location and full-process management, reducing the impact on normal training tasks.
Patent Information
- Application Number
- CN202611134926.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-29
- Publication Date
- 2026-11-17
Smart Images

Figure CN123114808A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of operation and maintenance and fault diagnosis technology of artificial intelligence computing centers, and in particular to a method for constructing a cross-layer test evidence chain and locating faults in artificial intelligence computing centers. Background Technology
[0002] With the rapid development of large language models, generative AI, and high-performance computing technologies, AI computing centers have become the core infrastructure for large-scale model training, inference services, and scientific research computing. A typical AI computing center's technology stack encompasses multiple layers, including GPU hardware, CPU and memory, PCIe / NVLink interconnects, high-speed network cards and switching networks, driver firmware, integrated communication libraries, distributed training frameworks, and job scheduling systems. The number of devices is enormous, the relationships between them are complex, and the communication paths are long, resulting in fault propagation exhibiting significant characteristics of being cross-layered and multi-faceted.
[0003] Currently, AI computing centers are equipped with a large number of testing and diagnostic tools from different vendors, including GPU health monitoring tools, PCIe link status monitoring tools, network bandwidth and latency testing tools, aggregated communication performance testing tools, and end-to-end training testing tools. However, the existing testing and fault diagnosis system has the following core shortcomings;
[0004] The test data is fragmented, the object identification is inconsistent, different tools use their own device numbers and data formats, and the same GPU may correspond to multiple identifications in the hardware management system, driver layer, communication library and operating system. Test results cannot be automatically correlated, and troubleshooting requires manual cross-system log comparison, which is extremely inefficient and prone to errors.
[0005] Resource topology does not substantially participate in fault diagnosis. The existing platform's resource topology is only used for visualization and display. It cannot trace the upstream fault source back based on abnormal phenomena, nor can it deduce the scope of fault impact. The generation of candidate fault causes depends on the experience of operation and maintenance personnel, which is prone to omissions or redundancy.
[0006] There is a lack of scientific mechanisms to handle conflicting evidence from multiple sources. Different testing tools may give contradictory conclusions. For example, a network card may be normal in point-to-point testing but its aggregate communication performance may be reduced. Existing solutions often rely on human experience or simple voting, which cannot quantify the weight of evidence, handle conflicting and missing evidence, and make the root cause judgments inaccurate.
[0007] Blindly selecting verification tests has a significant impact on business operations. Fault verification often involves executing the entire set of tests in a fixed order without considering the ability of different tests to distinguish candidate causes, resulting in a large number of invalid and duplicate tests.
[0008] Furthermore, the failure to incorporate test duration, resource consumption, and business impact into the selection criteria could easily disrupt normal training tasks, resulting in a loss of computing power.
[0009] The repair loop is incomplete and has poor traceability. After the fault is repaired, only the initially failed single test is usually retested, without verifying the related impact and end-to-end business recovery. This can easily lead to repeated faults.
[0010] Furthermore, the repair operation, retest results, and failure event do not form a complete chain of evidence, which is not conducive to auditing, review, and knowledge accumulation.
[0011] Therefore, there is an urgent need for a fault location method for AI computing centers that can achieve cross-layer test data association, structured evidence reasoning, intelligent verification optimization, and closed-loop management of the entire process. Summary of the Invention
[0012] The purpose of this invention is to at least solve one of the technical problems existing in the prior art, and to provide a method for constructing a cross-layer test evidence chain and locating faults in artificial intelligence computing power centers, which can solve the problems in the background art mentioned above.
[0013] To achieve the above objectives, the present invention provides the following technical solution: a method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing power center, comprising the following steps:
[0014] S1 collects resource object data, resource connection relationships, software dependency relationships, training job information, and multi-source test task information within the artificial intelligence computing center;
[0015] S2 assigns a globally unified identifier to all resource objects, test tasks, test execution instances, test metrics, log events, and exception events, and establishes a mapping relationship between the internal identifiers of each test tool and the globally unified identifier;
[0016] S3 constructs a cross-layer resource dependency topology covering the hardware layer to the operational layer based on the operation, invocation, dependency, connection, binding, passage, sharing and mapping relationships between resources;
[0017] S4 integrates test result data from multi-source testing tools, extracting test metrics, test logs, runtime status, and test environment information.
[0018] S5 detects abnormal test results and generates abnormal events through threshold comparison, baseline deviation analysis, or log event identification;
[0019] S6. Based on a globally unified identifier, the abnormal event is associated with the corresponding resource object, and the resource object is identified as the abnormal anchor point.
[0020] S7, starting from the abnormal anchor point, searches upstream dependent resources in reverse along the cross-layer resource dependency topology, generates multiple candidate fault causes by combining the preset fault mode library, and determines the scope of the abnormal impact by searching forward.
[0021] S8: For each candidate fault cause, extract the associated test results, log events and device status data as evidence, and construct a cross-layer test evidence chain that includes abnormal phenomena, resource paths, candidate causes, evidence sets and verification nodes.
[0022] S9 performs directional classification and conflict detection on the evidence, and calculates the credibility of each candidate cause of failure based on the evidence attributes.
[0023] S10. If the candidate fault cause meets the preset root cause determination conditions, the target fault cause is directly determined; if not, the verification test set is selected based on the candidate cause discrimination and execution cost, the verification test is executed and the evidence and credibility are iteratively updated until the root cause determination conditions are met.
[0024] S11 records the repair operation information corresponding to the cause of the target fault. After the repair is completed, multi-dimensional retesting is performed. When all retest indicators meet the fault closure conditions, the abnormal event is closed and a complete fault evidence chain report is output.
[0025] Preferably, the resource nodes of the cross-layer resource-dependent topology include training job nodes, training framework instance nodes, aggregated communication instance nodes, GPU nodes, CPU and NUMA nodes, PCIe link nodes, PCIe switch chip nodes, NVLink link nodes, high-speed network card nodes, network card port nodes, switch nodes, switch port nodes, network path nodes, and driver firmware nodes.
[0026] The relationships between resource nodes include those that run on, call, depend on, connect, bind, pass through, share, map to, and belong to, forming a top-down, full-level dependency chain.
[0027] Preferably, the fault mode library in step S7 pre-stores at least one of the following fault modes: GPU hardware anomaly, GPU memory anomaly, GPU power consumption / temperature limitation, PCIe link speed reduction, PCIe link retransmission anomaly, NVLink link error, network card port anomaly, switch port congestion, network path packet loss, RDMA configuration error, aggregated communication topology / parameter error, driver version incompatibility, and training framework synchronization anomaly.
[0028] The scope of the anomaly includes all computing nodes, communication instances, test tasks, and training jobs that share the same faulty resource, and is used to determine the scope of evidence collection and retesting.
[0029] Preferably, the evidence direction classification in step S9 includes four categories: supporting evidence, opposing evidence, indirect evidence, and missing evidence;
[0030] The rules for determining evidence conflict detection are as follows: when two or more pieces of evidence point to the same resource object, describe the same resource status, have overlapping collection time windows and comparable test conditions, but their test conclusions are mutually exclusive, they are determined to be evidence conflict.
[0031] Conflicting evidence is labeled and penalized independently of the evidence orientation classification.
[0032] Preferably, the confidence calculation step for the candidate fault cause in step S9 includes:
[0033] S91: Calculate the evidence weight for each piece of evidence. The weight is obtained by weighting the evidence type basic weight, the source reliability coefficient, the time validity coefficient, the execution quality coefficient, and the topological relevance coefficient.
[0034] The topological correlation coefficient is negatively correlated with the number of topological hops between the evidence object and the candidate fault object, while the time validity coefficient decreases as the interval between the evidence generation time and the anomaly occurrence time increases.
[0035] S92: Based on the evidence direction, the weight of supporting evidence is accumulated, the weight of opposing evidence is deducted, the contribution of indirect evidence is added, and the penalty value of conflicting evidence and the penalty value of missing evidence are deducted to obtain the original score of the candidate fault cause.
[0036] S93: Normalize the original scores of all candidate fault causes to obtain the confidence value of each candidate fault cause.
[0037] Preferably, the screening step of the verification test set in step S10 includes:
[0038] S101: Calculate the discrimination ability of each candidate validation test against all candidate cause pairs. The discrimination ability is the proportion of the number of candidate cause pairs that the test can distinguish to the total number of candidate cause pairs.
[0039] S102: Calculate the overall execution cost of each candidate verification test. The overall cost is calculated by weighting the test execution time, resource consumption, business impact level, execution risk, and historical validity.
[0040] S103: Under the constraint of covering all candidate cause pairs to be distinguished, select the test subset with the lowest overall execution cost as the verification test set; the constraints also include the total test duration limit, the available resource quantity limit, and the high-priority service interruption limit.
[0041] Preferably, the verification tests are performed in the following order: priority is given to uninterrupted service, priority is given to short execution time, and priority is given to high unit cost differentiation.
[0042] After each verification test is completed, the test results are immediately added to the evidence set, and the credibility of all candidate failure causes is recalculated.
[0043] When the candidate cause with the highest credibility meets the absolute credibility threshold, the difference between it and the candidate cause with the second highest credibility meets the difference threshold, and there is no unresolved high-weight conflict evidence or key missing evidence, the execution of the remaining verification tests shall be stopped immediately.
[0044] Preferably, the method further includes an abnormal fingerprint matching step:
[0045] A structured anomaly fingerprint is constructed based on the anomaly resource type, anomaly indicator combination, log event characteristics, topology location, anomaly occurrence sequence, impact scope, and software and hardware version information.
[0046] The current abnormal fingerprint is matched with the fingerprints in the historical fault database to obtain the root causes of historical faults, effective verification tests and repair solutions, which are used to assist in the sorting of candidate causes and the adjustment of verification test priorities.
[0047] Historical matching results are for reference only and do not replace the current evidence calculation and verification process.
[0048] Preferably, the multi-dimensional retesting after the repair in step S11 includes root cause-specific verification testing, original abnormal index retesting, correlation impact range testing, and end-to-end training performance testing.
[0049] The fault closure conditions include: passing the root cause test, the original abnormal indicators returning to the normal baseline range, no new abnormalities in the correlation test, end-to-end performance meeting the standard, no new evidence of high-weight conflicts, and no reproduction of the same type of abnormality within the stable observation window.
[0050] A system for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center includes:
[0051] The resource information acquisition module is used to collect information on resource objects, connection relationships, software dependencies, and training jobs within the computing center.
[0052] The identifier management module is used to allocate globally unified resource identifiers and maintain the identifier mapping relationship between multiple systems;
[0053] The topology management module is used to build and update cross-layer resource dependency topologies, and supports reverse fault search and positive impact analysis.
[0054] The multi-source test access module is used to access test tool data from different vendors and levels and convert it into a unified format;
[0055] The anomaly detection module is used to detect test anomalies and generate abnormal events using multi-dimensional rules;
[0056] The candidate cause generation module is used to generate candidate fault causes and determine the scope of impact based on anomaly anchor points and resource topology;
[0057] The evidence chain construction module is used to organize anomalies, resources, candidate causes, evidence, verification, repair and retesting data to form a structured cross-layer evidence chain;
[0058] The evidence evaluation module is used to perform evidence direction classification, conflict detection, and confidence calculation of candidate fault causes;
[0059] The verification test optimization module is used to evaluate the test discrimination ability, calculate the execution cost, and select the optimal set of verification tests.
[0060] The closed-loop management module is used to record repair operations, schedule retesting tasks, and determine fault closure conditions.
[0061] The fault report output module is used to generate a fault diagnosis report containing a complete chain of evidence.
[0062] Compared with the prior art, the beneficial effects of the present invention are:
[0063] 1. The method for constructing a cross-layer test evidence chain and locating faults in this artificial intelligence computing center provides a closed-loop coverage of the entire process, from data access, anomaly discovery, fault location, verification testing to repair and shutdown. It replaces existing scattered tools and manual processes, realizes full lifecycle management of fault handling, and achieves strong cross-layer data correlation. Through unified resource identification and dependency topology, it breaks down data barriers between different levels and different vendors' testing tools, automatically correlates cross-layer test results, and significantly reduces the workload of manual comparison.
[0064] 2. The cross-layer test evidence chain construction and fault location method of this artificial intelligence computing power center is scientific and precise in evidence processing. It establishes an evidence system with four categories of support / opposition / indirectness / missing information and independent conflict detection. It combines multi-factor weight quantification to calculate credibility, avoids misjudgment of single test items and empiricism, significantly improves the accuracy of root cause judgment, and verifies that the test is efficient and low-interference. It selects effective tests based on the ability to distinguish, generates the optimal test set based on the execution cost, supports item-by-item iteration and early stopping, greatly reduces invalid tests, and prioritizes tests without business interruption to reduce the impact on normal computing power services.
[0065] 3. The cross-layer test evidence chain construction and fault location method of this artificial intelligence computing power center is closed-loop, reliable and traceable. Through multi-dimensional retesting and multi-condition shutdown judgment, it ensures that the fault is completely repaired. The complete evidence chain records the information of the whole process, supports auditing, review and knowledge accumulation, and realizes the reuse of historical experience by combining anomaly fingerprints. It has strong scalability and adopts a modular design, which can flexibly expand new resource types, test tools, fault modes and verification methods, and adapt to different computing power center scenarios such as domestic computing power and heterogeneous computing. Attached Figure Description
[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0067] Figure 1 This is a diagram illustrating the overall architecture of the cross-layer testing evidence chain construction and fault location system for the artificial intelligence computing power center of this invention.
[0068] Figure 2 This is a schematic diagram of the cross-layer resource dependency topology and test evidence chain of the artificial intelligence computing center of this invention.
[0069] Figure 3 This is a flowchart of the process for calculating the credibility of candidate fault causes and selecting the verification test set in this invention. Detailed Implementation
[0070] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.
[0071] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0072] In the description of this invention, terms such as greater than, less than, and exceeding are understood to exclude the stated number, while terms such as above, below, and within are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0073] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0074] Please see Figure 1-3 This invention provides a technical solution: a method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing power center, comprising the following steps:
[0075] S1. Multi-source information collection: Collect full information on hardware resources, network topology, software components, training jobs and testing tasks within the computing center to provide a data foundation for subsequent modeling;
[0076] S2. Construction of a unified identification system: Assign globally unique identifiers to all resource objects, test entities and events, establish identifier mapping relationships between multiple systems and tools, and solve the problem of data fragmentation;
[0077] S3. Cross-layer resource dependency topology construction: Based on the relationships between resources such as operation, invocation, connection, and binding, a directed dependency topology is constructed from the job layer to the hardware layer to support fault reverse tracing and impact forward analysis.
[0078] S4. Multi-source test data access: Adapts to test tools from different vendors and at different levels, converts heterogeneous test data into a unified structured format, and enables centralized management of test results;
[0079] S5. Anomaly Identification and Anchoring: Anomalies are identified through multi-dimensional rules such as fixed thresholds, historical baselines, baselines of similar resources, and log error patterns, and anomalies are accurately associated with specific resource objects to form anomaly anchors.
[0080] S6. Candidate Fault Cause Generation and Impact Analysis: Starting from the abnormal anchor point, perform a reverse search along the topology and generate candidate fault causes by combining the fault mode library.
[0081] Simultaneously, forward search determines the scope of the anomaly's impact and delineates the boundaries for evidence collection and retesting;
[0082] S7. Construction of Cross-Layer Test Evidence Chain: Organize abnormal events, resource paths, candidate causes, various types of evidence, verification tests, repair operations and retest results into a structured evidence chain graph to achieve full-link traceability of faults;
[0083] S8. Evidence Classification and Conflict Detection: Evidence is classified into four categories: supporting, opposing, indirect, and missing.
[0084] At the same time, we independently examine the conflicting relationships between pieces of evidence to avoid contradictory evidence interfering with the determination of root causes.
[0085] S9. Quantitative calculation of the credibility of candidate causes: The evidence weight is calculated by comprehensively considering the evidence type, source reliability, time validity, execution quality and topological relevance. Combined with the accumulation of supporting / opposing evidence, conflict and missing evidence penalties, the credibility score of each candidate cause is obtained and ranked.
[0086] S10. Root cause determination and verification test optimization: If the candidate cause with the highest confidence meets the absolute threshold and relative difference conditions, it is directly determined as the root cause;
[0087] Otherwise, the optimal set of verification tests is selected based on the ability to distinguish and the overall cost, and the tests are executed one by one with the credibility updated iteratively. The test is stopped early when the conditions are met.
[0088] S11, Repair and Retest Closed-Loop Management: Record complete repair operation information, perform root cause verification, original anomaly retest, related impact test and end-to-end performance test after repair, close the fault after all closure conditions are met, and output a complete evidence chain report.
[0089] Furthermore, the present invention also includes an anomaly fingerprint matching mechanism, which constructs a structured anomaly fingerprint and matches it with a historical fault database, reuses historical diagnostic experience, and accelerates the sorting of candidate causes and the selection of verification tests.
[0090] Correspondingly, the present invention also provides a cross-layer test evidence chain construction and fault location system for artificial intelligence computing power centers, including a resource information acquisition module, an identification management module, a topology management module, a multi-source test access module, an anomaly identification module, a candidate cause generation module, an evidence chain construction module, an evidence evaluation module, a verification test optimization module, a repair closed-loop management module, and a fault report output module. Each module works together to implement the above method flow.
[0091] Example 1:
[0092] This embodiment provides a method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center. It is deployed in a GPU computing center containing 64 computing nodes. Each node is configured with 8 GPUs and 2 high-speed IB network cards. NVLink is used to realize the interconnection of GPUs within the node, and InfiniBand network is used to realize the interconnection between nodes. The NCCL collection communication library and distributed training framework are running.
[0093] like Figure 3 As shown, the specific execution flow of this method is as follows:
[0094] Step 1: Basic Information Collection and Modeling
[0095] The system collects full resource information, including GPU serial number and PCIe address, CPU and NUMA node information, PCIe switch chip topology, NVLink connection relationship, network card port and switch port correspondence, driver and firmware version, training job resource allocation information, etc., by calling device management interface, reading operating system device enumeration information, parsing switch forwarding table, and connecting to job scheduling system.
[0096] Step 2: Establishing a unified identifier and mapping
[0097] Assign a globally unique identifier to each type of entity: resource objects use RID, test tasks use TID, test execution instances use TRID, test metrics use MID, abnormal events use AID, candidate fault causes use CID, and repair operations use FID;
[0098] Taking GPUs as an example, the system establishes a mapping relationship between hardware serial numbers, PCIe bus addresses, CUDA device numbers, NCCLrank numbers, job scheduling system card numbers, and global RIDs to ensure that test data generated by different tools can be accurately associated with the same physical GPU.
[0099] Step 3: Cross-layer resource dependency topology construction
[0100] Construct a cross-layer resource dependency topology G=(V,E) with a directed graph structure, where the node set V contains 14 types of nodes, including training jobs, training framework instances, set communication domains, GPUs, CPUs / NUMA, PCIe links, PCIe switching chips, NVLink links, network cards, network card ports, switches, switch ports, network paths, and driver firmware.
[0101] The edge set E contains nine types of relationships: running at, calling, depending on, connecting, binding, passing through, sharing, mapping, and belonging to.
[0102] For example, a typical dependency path is: Training job J01 → Training framework instance F01 → NCCL communication domain C01 → GPU G03 → PCIe link P03 → Network card port NIC1-P1 → Switch port SP12 → Network path NP05.
[0103] Step 4: Multi-source test data access
[0104] The system integrates test results from various tools, including GPU health checks, memory tests, PCIe link status checks, NVLink bandwidth tests, IB network card self-tests, RDMA bandwidth and latency tests, network packet loss and retransmission detection, NCCL aggregated communication tests, and end-to-end training performance tests, through API integration, log collection, file parsing, and message queues. These results are then uniformly converted into structured data containing test tasks, tested resources, execution time, metric values, running status, log content, and environmental information.
[0105] Step 5: Abnormal Event Identification
[0106] The system employs multi-dimensional anomaly detection rules:
[0107] Fixed threshold method: If the PCIe link width is less than x16 or the network card retransmission rate is higher than 0.1%, it is directly judged as abnormal;
[0108] Historical baseline method: Calculate the deviation rate of the test index relative to the normal baseline of the same period in history. The formula is Deviation(m)=|Value(m)-Baseline(m)| / Baseline(m). When the deviation rate exceeds the preset threshold, it is judged as abnormal.
[0109] Same-baseline method: When the resource indicators of the same model and configuration deviate from the cluster average by more than 3 times the standard deviation, it is considered abnormal;
[0110] Log event method: An exception is determined when a preset error code or timeout event is matched in the test log;
[0111] In this embodiment, the routine NCCLAllReduce test shows that the bus bandwidth of node A is 18GB / s, which is 60% lower than the historical baseline of 45GB / s. The deviation rate far exceeds the threshold, and the system generates an abnormal event A001.
[0112] Step 6: Locating Abnormal Anchor Points
[0113] Based on a unified identifier, the system associates the abnormal event A001 with the NCCL communication domain C01 of node A, corresponding to 8 GPUs and 2 network cards, and sets the communication domain C01 and node A as the initial abnormal anchor point;
[0114] Step 7: Candidate Fault Cause Generation and Impact Analysis
[0115] Starting from the anomaly anchor point, the upstream dependent resources are searched in reverse along the resource dependency topology, and four candidate fault causes are generated by combining the fault mode library:
[0116] C1: GPUG03 hardware malfunction;
[0117] C2: PCIe link P03 corresponding to GPUG03 is slowed down;
[0118] C3: Network card port NIC1-P1 hardware malfunction;
[0119] C4: NCCL communication configuration error;
[0120] Simultaneously, a forward search was conducted to determine the scope of the anomaly's impact, identifying two training jobs and three periodic test tasks related to this communication domain as the objects for subsequent retesting and business verification.
[0121] Step 8: Evidence Extraction and Chain of Evidence Construction
[0122] The system automatically extracts evidence related to each candidate cause from existing test data to construct an initial chain of evidence:
[0123] E1: PCIe link status detection shows that the P03 link width is x8, which should be x16 normally;
[0124] E2: Network card port statistics show that there are a small number of retransmissions in NIC1-P1;
[0125] E3: GPUG03 memory test passed in full, no hardware errors;
[0126] E4: NCCL test logs show that the performance bottleneck is concentrated in the communication phase involving GPUG03;
[0127] E5: Inter-node RDMA point-to-point bandwidth test shows that the bandwidth of NIC1-P1 is normal;
[0128] E6: Performance is normal when other nodes use the same NCCL configuration;
[0129] The chain of evidence starts with an unusual event, connects resource paths, candidate causes, and corresponding evidence, and forms a traceable graph structure.
[0130] Step 9: Evidence Classification, Conflict Detection, and Credibility Calculation
[0131] First, categorize each piece of evidence by direction:
[0132] Evidence supporting C2: E1, E4;
[0133] Evidence against C1: E3;
[0134] Evidence against C3: E5;
[0135] Evidence against C4: E6;
[0136] Indirect evidence: E2;
[0137] Subsequently, conflict detection was performed. In this embodiment, all evidence had mutually exclusive conclusions with no time overlap, and there was no conflicting evidence.
[0138] Next, we calculate the weight of evidence:
[0139] Evidence type weighting: Direct test evidence has the highest weight, followed by indirect evidence;
[0140] Source reliability: PCIe status read from hardware registers is more reliable than statistical logs;
[0141] Time validity: The closer to the time of the anomaly, the higher the time coefficient, using the exponential decay formula Rtime=exp(-Δt / τ);
[0142] Topological correlation: The fewer the topological hops between the evidence object and the candidate fault object, the higher the correlation coefficient, and the formula is Rtopo=1 / (1+d);
[0143] Execution quality: Tests with complete test parameters and stable execution environments have higher quality coefficients.
[0144] Finally, calculate the original scores for the candidate reasons:
[0145] RawScore=ΣW_support-λ1×ΣW_oppose+λ2×ΣW_indirect-λ3×W_conflict-λ4×W_miss;
[0146] The confidence level is obtained by softmax normalization. In this embodiment, C2 has the highest confidence level, followed by C3. However, the difference between the two does not reach the judgment threshold and needs further verification.
[0147] Step 10: Verification Test Selection and Execution
[0148] The system first calculates the discrimination of each candidate validation test:
[0149] PCIe point-to-point bandwidth test: can distinguish between C2 and C3, and C2 and C4, with high differentiation;
[0150] Local loopback test of network card: Can distinguish between C3 and C2, and C3 and C4, with moderate discrimination;
[0151] The overall execution cost was then calculated, and both tests showed that they did not require business interruption, had short execution time, and low risk.
[0152] The system prioritized performing PCIe point-to-point bandwidth tests. The test results showed that the bandwidth of P03 was only at the level corresponding to Gen4x8, which was consistent with the expectation of C2.
[0153] After adding the result to the evidence set and recalculating the credibility, the credibility of C2 was greatly improved, meeting the absolute threshold and relative difference conditions. The system stopped the test early and determined that the PCIe link P03 speed reduction was the root cause of the fault.
[0154] Step 11: Repair and retest the closed loop
[0155] When maintenance personnel perform GPU reconnection, the system records the repair time, the person who performed the operation, the operation content, and the state before the repair.
[0156] After the repair is completed, perform a Level 4 retest:
[0157] Root cause verification: PCIe link state detection has been restored to x16, and point-to-point bandwidth is normal;
[0158] Retest of the original anomaly: NCCLAllReduce bandwidth has recovered to 44GB / s, which is within the normal baseline range;
[0159] Related impact test: The communication performance of other GPUs under the same PCIe switching chip is normal, and there are no abnormal retransmissions of the network card;
[0160] End-to-end testing: The single-step time of the affected training jobs has returned to normal levels;
[0161] After 24 hours of stable observation without the recurrence of similar anomalies, the system determines that all shutdown conditions are met, shuts down the fault event, generates a diagnostic report containing a complete chain of evidence, and stores it in the historical fault database.
[0162] Example 2:
[0163] This embodiment adds an anomaly fingerprint matching mechanism based on embodiment 1. When an anomaly event is generated, the system constructs an anomaly fingerprint FP based on the anomaly resource type, indicator deviation characteristics, log keywords, topology location, occurrence time pattern, impact range and driver version.
[0164] The system calculates the similarity between the current fingerprint and the historical fault database, and obtains a comprehensive similarity by weighting it from seven dimensions, including resource features, indicator features, log features, and topology features.
[0165] In this embodiment, three similar historical faults were matched, all of which were caused by PCIe link slowdown. In the verification test, the PCIe point-to-point test had a 100% effectiveness rate.
[0166] The system uses historical matching results as prior weights to fine-tune the initial sorting of candidate causes and the priority of verification tests, further shortening the diagnosis time.
[0167] Historical matching results are only used as a supplementary reference. The final root cause determination is still based on real-time evidence calculation to avoid empirical misjudgment.
[0168] System Implementation Examples
[0169] This invention also provides a system for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center, the architecture of which is as follows: Figure 1 As shown, it includes:
[0170] Resource information acquisition module: Supports resource information acquisition using multiple protocols and interfaces, covering all dimensions of hardware, network, software, and operations;
[0171] Identifier Management Module: Maintains a globally unified identifier system and mapping relationships across multiple systems, and provides identifier query and conversion services;
[0172] Topology Management Module: Responsible for constructing, updating, and querying cross-layer resource dependency topologies, and supports graph traversal algorithms for reverse tracing and positive impact analysis;
[0173] Multi-source test access module: Provides adaptation plugins for various test tools to achieve standardized access and storage of heterogeneous data;
[0174] Anomaly detection module: integrates multi-dimensional anomaly detection algorithms and rule engine, supporting multi-mode anomaly detection based on threshold, baseline, and logs;
[0175] Candidate cause generation module: Combines topology reverse search and fault mode library to automatically generate candidate fault causes and their impact range;
[0176] Evidence chain construction module: Stores evidence chain graphs based on graph databases, supporting full-link visual tracing;
[0177] Evidence evaluation module: Implements the core logic of evidence orientation classification, conflict detection, and credibility quantification calculation;
[0178] Verification test optimization module: includes test discrimination evaluation, cost calculation and optimal test set selection algorithm;
[0179] Repair closed-loop management module: manages repair operation records, retest task scheduling, and fault closure condition determination;
[0180] Fault report output module: Automatically generates structured fault diagnosis reports, supporting export and archiving.
[0181] The system backend can be developed using Java or Go. Resource and business data are stored in a relational database, topology and evidence chain data are stored in a graph database, and test metric data are stored in a time-series database, providing good performance and scalability.
[0182] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center, characterized in that, Includes the following steps: S1 collects resource object data, resource connection relationships, software dependency relationships, training job information, and multi-source test task information within the artificial intelligence computing center; S2 assigns a globally unified identifier to all resource objects, test tasks, test execution instances, test metrics, log events, and exception events, and establishes a mapping relationship between the internal identifiers of each test tool and the globally unified identifier; S3 constructs a cross-layer resource dependency topology covering the hardware layer to the operational layer based on the operation, invocation, dependency, connection, binding, passage, sharing and mapping relationships between resources; S4 integrates test result data from multi-source testing tools, extracting test metrics, test logs, runtime status, and test environment information. S5 detects abnormal test results and generates abnormal events through threshold comparison, baseline deviation analysis, or log event identification; S6. Based on a globally unified identifier, the abnormal event is associated with the corresponding resource object, and the resource object is identified as the abnormal anchor point. S7, starting from the abnormal anchor point, searches upstream dependent resources in reverse along the cross-layer resource dependency topology, generates multiple candidate fault causes by combining the preset fault mode library, and determines the scope of the abnormal impact by searching forward. S8: For each candidate fault cause, extract the associated test results, log events and device status data as evidence, and construct a cross-layer test evidence chain that includes abnormal phenomena, resource paths, candidate causes, evidence sets and verification nodes. S9 performs directional classification and conflict detection on the evidence, and calculates the credibility of each candidate cause of failure based on the evidence attributes. S10. If the candidate fault cause meets the preset root cause determination conditions, the target fault cause is directly determined; if not, the verification test set is selected based on the candidate cause discrimination and execution cost, the verification test is executed and the evidence and credibility are iteratively updated until the root cause determination conditions are met. S11 records the repair operation information corresponding to the cause of the target fault. After the repair is completed, multi-dimensional retesting is performed. When all retest indicators meet the fault closure conditions, the abnormal event is closed and a complete fault evidence chain report is output.
2. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The resource nodes of the cross-layer resource-dependent topology include training job nodes, training framework instance nodes, aggregated communication instance nodes, GPU nodes, CPU and NUMA nodes, PCIe link nodes, PCIe switch chip nodes, NVLink link nodes, high-speed network card nodes, network card port nodes, switch nodes, switch port nodes, network path nodes, and driver firmware nodes. The relationships between resource nodes include those that run on, call, depend on, connect, bind, pass through, share, map to, and belong to, forming a top-down, full-level dependency chain.
3. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The fault mode library described in step S7 pre-stores at least one of the following fault modes: GPU hardware anomaly, GPU memory anomaly, GPU power consumption / temperature limitation, PCIe link speed reduction, PCIe link retransmission anomaly, NVLink link error, network card port anomaly, switch port congestion, network path packet loss, RDMA configuration error, aggregated communication topology / parameter error, driver version incompatibility, and training framework synchronization anomaly. The scope of the anomaly includes all computing nodes, communication instances, test tasks, and training jobs that share the same faulty resource, and is used to determine the scope of evidence collection and retesting.
4. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The evidence classification described in step S9 includes four categories: supporting evidence, opposing evidence, indirect evidence, and missing evidence. The rules for determining evidence conflict detection are as follows: when two or more pieces of evidence point to the same resource object, describe the same resource status, have overlapping collection time windows and comparable test conditions, but their test conclusions are mutually exclusive, they are determined to be evidence conflict. Conflicting evidence is labeled and penalized independently of the evidence orientation classification.
5. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The confidence calculation step for the candidate fault cause in step S9 includes: S91: Calculate the evidence weight for each piece of evidence. The weight is obtained by weighting the evidence type basic weight, the source reliability coefficient, the time validity coefficient, the execution quality coefficient, and the topological relevance coefficient. The topological correlation coefficient is negatively correlated with the number of topological hops between the evidence object and the candidate fault object, while the time validity coefficient decreases as the interval between the evidence generation time and the anomaly occurrence time increases. S92: Based on the evidence direction, the weight of supporting evidence is accumulated, the weight of opposing evidence is deducted, the contribution of indirect evidence is added, and the penalty value of conflicting evidence and the penalty value of missing evidence are deducted to obtain the original score of the candidate fault cause. S93: Normalize the original scores of all candidate fault causes to obtain the confidence value of each candidate fault cause.
6. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The screening step for the verification test set in step S10 includes: S101: Calculate the discrimination ability of each candidate validation test against all candidate cause pairs. The discrimination ability is the proportion of the number of candidate cause pairs that the test can distinguish to the total number of candidate cause pairs. S102: Calculate the overall execution cost of each candidate verification test. The overall cost is calculated by weighting the test execution time, resource consumption, business impact level, execution risk, and historical validity. S103: Under the constraint of covering all candidate cause pairs to be distinguished, select the test subset with the lowest overall execution cost as the verification test set; the constraints also include the total test duration limit, the available resource quantity limit, and the high-priority service interruption limit.
7. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The verification tests are performed in the following order: priority is given to uninterrupted service, shortest execution time, and highest unit cost differentiation. After each verification test is completed, the test results are immediately added to the evidence set, and the credibility of all candidate failure causes is recalculated. When the candidate cause with the highest credibility meets the absolute credibility threshold, the difference between it and the candidate cause with the second highest credibility meets the difference threshold, and there is no unresolved high-weight conflict evidence or key missing evidence, the execution of the remaining verification tests shall be stopped immediately.
8. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The method also includes an abnormal fingerprint matching step: A structured anomaly fingerprint is constructed based on the anomaly resource type, anomaly indicator combination, log event characteristics, topology location, anomaly occurrence sequence, impact scope, and software and hardware version information. The current abnormal fingerprint is matched with the fingerprints in the historical fault database to obtain the root causes of historical faults, effective verification tests and repair solutions, which are used to assist in the sorting of candidate causes and the adjustment of verification test priorities. Historical matching results are for reference only and do not replace the current evidence calculation and verification process.
9. The method for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center according to claim 1, characterized in that: The multi-dimensional retesting after the repair in step S11 includes root cause-specific verification testing, original anomaly index retesting, correlation impact range testing, and end-to-end training performance testing. The fault closure conditions include: passing the root cause test, the original abnormal indicators returning to the normal baseline range, no new abnormalities in the correlation test, end-to-end performance meeting the standard, no new evidence of high-weight conflicts, and no reproduction of the same type of abnormality within the stable observation window.
10. A system for constructing a cross-layer test evidence chain and locating faults in an artificial intelligence computing center, characterized in that, include: The resource information acquisition module is used to collect information on resource objects, connection relationships, software dependencies, and training jobs within the computing center. The identifier management module is used to allocate globally unified resource identifiers and maintain the identifier mapping relationship between multiple systems; The topology management module is used to build and update cross-layer resource dependency topologies, and supports reverse fault search and positive impact analysis. The multi-source test access module is used to access test tool data from different vendors and levels and convert it into a unified format; The anomaly detection module is used to detect test anomalies and generate abnormal events using multi-dimensional rules; The candidate cause generation module is used to generate candidate fault causes and determine the scope of impact based on anomaly anchor points and resource topology; The evidence chain construction module is used to organize anomalies, resources, candidate causes, evidence, verification, repair and retesting data to form a structured cross-layer evidence chain; The evidence evaluation module is used to perform evidence direction classification, conflict detection, and confidence calculation of candidate fault causes; The verification test optimization module is used to evaluate the test discrimination ability, calculate the execution cost, and select the optimal set of verification tests. The closed-loop management module is used to record repair operations, schedule retesting tasks, and determine fault closure conditions. The fault report output module is used to generate a fault diagnosis report containing a complete chain of evidence.
