A hierarchical decoupling-based intelligent system cross-level defect positioning method

Abstract

The application discloses a kind of based on hierarchical decoupling intelligent system cross-level defect positioning method, belong to intelligent system test and reliability guarantee technical field.This method constructs chip hardware layer, computing software stack layer and model application layer three-layer architecture, utilizes data flow, runtime and hardware state probe in combination with PTP precision clock, to register, instruction stream, operator execution state and precision drift etc.Information is synchronously collected;When detecting exception, save full-stack snapshot, and determine the level of defect by the hierarchical decoupling process of freezing hardware, switching model and rolling back data;Further based on defect correlation graph, root cause positioning is carried out, and traceable report and reproducible test case containing trigger condition and reproduction step are generated, to realize the positioning and reproduction of intelligent system cross-level defect.

Description

Technical Field

[0001] This invention belongs to the field of intelligent system testing and reliability assurance technology, specifically relating to a cross-level defect localization method for a full-stack intelligent system composed of domestically produced AI chips, inference frameworks, and model applications. This method is based on a three-layer probe approach (data flow, runtime, and hardware state) and a bisection-based hierarchical decoupling. The invention aims to solve the problems of difficult fault localization and slow root cause analysis caused by hierarchical coupling in intelligent systems, achieving precise defect tracing at the chip register level and framework operator level. It is applicable to full-stack testing, defect diagnosis, and reliability verification of high-end intelligent equipment in harsh environments such as industrial control, aerospace, and deep sea. Background Technology

[0002] Against the backdrop of the comprehensive rise of the domestic artificial intelligence ecosystem and the large-scale deployment of edge intelligent equipment, intelligent systems have deeply penetrated key fields such as industrial control, autonomous driving, aerospace, deep-sea exploration, and national defense equipment, becoming the core carrier for the high-quality development of the national digital economy and high-end manufacturing. Currently, intelligent systems exhibit distinct technical characteristics such as chip heterogeneity, framework diversification, model complexity, and extreme application scenarios. Their operational correctness, inference accuracy, real-time performance, environmental reliability, and long-term robustness directly determine the operational safety of high-end equipment, industrial economic benefits, and the success or failure of major national strategic tasks. However, intelligent systems are essentially composed of deeply coupled chip hardware, computing software stacks, model applications, data flow, and environmental conditions. Their hierarchical interactions are tight, fault propagation is hidden, and anomaly mechanisms are complex. Traditional testing and verification methods are no longer adequate for the full-stack quality assurance requirements, leading to increasingly prominent problems such as missed defect detection, ambiguous localization, difficulty in reproduction, and unclear root causes.

[0003] Currently, the field of intelligent system testing still largely uses traditional verification models that are discrete, single-level, and offline, resulting in a series of insurmountable technical bottlenecks. Testing layers are severely fragmented, with chip, framework, and model testing operating independently, making it impossible to build an integrated collaborative observation system. Many layer-coupling defects cannot be effectively identified. Domestic adaptation capabilities are severely insufficient; mainstream testing equipment lacks low-level instruction-level, operator-level, and register-level adaptation capabilities for domestic AI chips and inference frameworks, leading to interface incompatibility and unobservable states. Environmental stress and computational performance are completely disconnected; traditional environmental testing cannot be linked with real-time computing, inference accuracy, and operator states, making it difficult to meet the high-reliability verification requirements of harsh environments. Defect localization methods are extremely crude, only able to determine whether the system has failed, unable to achieve fine-grained root cause localization at the register-level, operator-level, and model-level, resulting in low fault diagnosis efficiency and high difficulty in reproduction.

[0004] Furthermore, existing testing systems suffer from common shortcomings such as asynchronous multi-source data, untimely anomaly triggering, and lack of support for defect reproduction. Due to the lack of a unified nanosecond-level clock synchronization and collaborative acquisition architecture, hardware, software, model, and environmental data cannot achieve precise time-series alignment, making it difficult to trace the complete causal chain when anomalies occur. Simultaneously, the lack of standardized anomaly judgment, on-site data preservation, hierarchical isolation, and defect reproduction mechanisms makes it difficult to quickly determine the defect's level when faced with typical anomalies such as accuracy degradation, performance drops, and inference crashes, resulting in slow defect localization, difficult analysis, and low reproducibility. In summary, a unified, standardized, and fully stackable method for intelligent system defect localization has not yet been established domestically or internationally. Developing a cross-level defect localization method that is fully observable, deeply compatible with domestic technology, precisely time-series aligned, rapidly decoupled at different levels, and reproducible has become an urgent need and a key technological bottleneck for ensuring the security of the domestic artificial intelligence ecosystem and supporting the high-quality development of high-end intelligent equipment. Summary of the Invention

[0005] This invention provides a precise defect localization technology for intelligent systems that utilizes cross-level collaborative observation, bisection-based hierarchical decoupling, and nanosecond-level timing alignment. This technology effectively addresses current industry challenges such as fragmented testing levels, insufficient domestic adaptation, disconnect between environment and computation, coarse defect localization, and difficulty in reproducing faults. Furthermore, it enables integrated verification across the entire chip-framework-model chain, helping testers quickly locate root causes and complete reliability verification in scenarios where the domestic AI ecosystem is coupled with harsh environments.

[0006] To achieve the above objectives, this invention proposes a method for cross-layer defect localization in intelligent systems based on hierarchical decoupling. The method is characterized by dividing the intelligent system defect localization into three layers according to the full-stack testing and verification logic: the chip hardware layer, the computing software stack layer, and the model application layer. Then, by deploying dedicated probes in three dimensions—data flow, runtime, and hardware state—synchronous multi-dimensional data acquisition across the entire stack is achieved. When anomalies such as accuracy degradation, performance drop, or inference crash occur, a binary localization process of "freezing hardware - switching model - rolling back data" is automatically executed to quickly determine the defect attribution. Relying on the PTP precision clock, nanosecond-level alignment of the three layers of data is achieved, constructing a cross-layer defect correlation map to realize accurate root cause localization at the chip register level and the framework operator level. This method includes the following steps:

[0007] 1) Deployment of three-layer probes and synchronous data acquisition

[0008] To achieve non-intrusive end-to-end observation and organize data into a unified time-series structure, the collected information is divided into three layers: hardware state layer, runtime layer, and data flow layer.

[0009] The hardware state layer includes: chip registers, instruction stream, power consumption waveform, I / O timing, signal integrity, power ripple, single-event response, and bit-level fault injection feedback.

[0010] The runtime layer includes: operator execution status, scheduling latency, thread preemption, memory access, cache hit rate, driver interaction, compilation optimization logs, and operator execution cycle.

[0011] The data flow layer includes: model input / output, tensor read / write, data preprocessing, post-processing chain, precision drift, output distortion, quantization loss, and robustness anomalies.

[0012] This method employs a unified clock synchronization mechanism to achieve three types of data acquisition: non-intrusive hardware acquisition via JTAG / SWD / PCIe, full lifecycle acquisition during runtime via probe instrumentation, and end-to-end data stream acquisition via model parsing; all data is uniformly timestamped at the nanosecond level to ensure complete timing consistency.

[0013] 2) Anomaly monitoring and bisection-based hierarchical decoupling localization

[0014] This method utilizes real-time acquired hardware, software, and model data to determine anomalies in inference accuracy, latency, power consumption, and throughput. Using the timing synchronization data acquired in step 1), preset anomaly thresholds, and defect judgment rules as input, the three layers of data are first analyzed and metrics calculated in real-time. The method sequentially checks for accuracy drift, performance drops sharply, inference crashes, output distortion, operator execution failures, and signal integrity degradation. If the current metrics are normal, monitoring continues; if an anomaly is triggered, a full-stack snapshot is immediately saved, including register context, instruction stream, operator call stack, intermediate model results, input data, environmental conditions, and fault injection records. Subsequently, a binary hierarchical decoupling process is initiated: hardware is frozen level by level, the baseline model is switched, input data is rolled back, variables are isolated, and cross-validation is performed to quickly classify defects as hardware defects, software stack defects, model defects, or cross-level coupling defects.

[0015] 3) Fine-grained root cause localization and test case generation

[0016] Based on the hierarchical decoupling results, the root cause of defects is located, outputting chip register-level, framework operator-level, or model application-level location results. A traceable defect location report is generated, including defect location, triggering conditions, reproduction steps, and optimization suggestions. Located defects are categorized into a defect set, classified and archived according to defect level, impact scope, and reproduction difficulty, providing a complete timeline, hardware snapshot, software call stack, and model execution trajectory. One-click reproduction, full-process traceability, and auditability are supported. Finally, defect information and test configurations are encapsulated into reproducible test cases for subsequent verification, regression, and iterative optimization. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the layered full-stack testing and verification principle and three-layer architecture of the present invention;

[0018] Figure 2 This is a schematic diagram illustrating the deployment and acquisition of three types of probes: data flow, runtime, and hardware status, according to the present invention.

[0019] Figure 3 This is a flowchart of the "freeze hardware - switch model - roll back data" binary decoupling and positioning process of the present invention;

[0020] Figure 4 This is a schematic diagram illustrating the overall architecture and defect tracing of the "three physical units + one logic plane" of this invention;

[0021] Figure 5 This is a flowchart of the full-stack defect location, report output, and one-click reproduction process of the present invention. Detailed Implementation

[0022] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0023] Figure 1 This is a schematic diagram illustrating the layered full-stack testing and verification principle and three-layer architecture of the present invention; Figure 2 This is a schematic diagram illustrating the deployment and acquisition of three types of probes: data flow, runtime, and hardware status, according to the present invention. Figure 3 This is a flowchart of the "freeze hardware - switch model - roll back data" binary decoupling and positioning process of the present invention; Figure 4 This is a schematic diagram illustrating the overall architecture and defect tracing of the "three physical units + one logic plane" of this invention; Figure 5 This is a flowchart of the full-stack defect location, report output, and one-click reproduction process of the present invention.

[0024] A method for cross-level defect localization in intelligent systems based on hierarchical decoupling, characterized by the following steps:

[0025] ① Deployment of three-layer probes and synchronous data acquisition

[0026] like Figure 1 , Figure 2 As shown, three types of probes are deployed simultaneously at the chip hardware layer, computing software stack layer, and model application layer of the intelligent system to establish a nanosecond-level synchronous acquisition link:

[0027] 1) Hardware status probe deployment: The probe is connected to the domestic AI chip under test through the JTAG / SWD / PCIe debugging interface, and non-intrusive data collection is performed on register status, instruction stream, real-time power consumption, IO timing, signal integrity, power ripple and single-event response.

[0028] 2) Runtime probe deployment: Instrument nodes in operator execution, scheduling logic, memory access, and driver interaction to collect operator execution cycle, scheduling latency, thread preemption, cache hit rate, and compilation optimization logs;

[0029] 3) Data stream probe deployment: In the model input / output, tensor reading / writing, data preprocessing and postprocessing links, it collects data to detect accuracy drift, output distortion, quantization loss, and robustness anomalies.

[0030] 4) Enable PTP precision clock and FPGA synchronous triggering to make the timestamp alignment accuracy of the three types of data ≤100ns, forming a unified timing full-stack dataset.

[0031] ② Anomaly monitoring and bisection-based hierarchical decoupling localization

[0032] like Figure 3 , Figure 4 As shown, taking time-series synchronized data, anomaly thresholds, and defect judgment rules as inputs, the hierarchical decoupling process is monitored and triggered in real time:

[0033] 1) Real-time analysis of inference accuracy, latency, power consumption, throughput, and signal integrity metrics to determine whether accuracy degradation, performance drop, inference crash, output distortion, or operator execution failure has occurred;

[0034] 2) If an exception is triggered, immediately save a full stack snapshot: register context, instruction stream, operator call stack, intermediate model results, input data, environmental conditions and fault injection records;

[0035] 3) Execute the binary search decoupling process:

[0036] ① Freeze the hardware operating state, keep the chip power supply, clock and load unchanged, and lock the hardware status;

[0037] ② Switch to the standardized benchmark model, keeping the framework and hardware unchanged, and differentiate the defect domains;

[0038] ③ Roll back the input data to a clean baseline sample to eliminate data-related anomalies;

[0039] 4) By isolating and cross-validating at each level, quickly distinguish the attribution of defects: hardware defects, software stack defects, model defects, or cross-level coupling defects.

[0040] ③ Fine-grained root cause localization and defect archiving

[0041] like Figure 4 , Figure 5 As shown, accurate root cause localization is achieved based on the decoupling results, outputting register-level / operator-level localization results:

[0042] 1) Hardware defects are located to specific registers, memory units, I / O pins, and instruction execution units within the chip;

[0043] 2) Software stack defects are located to specific operators, scheduling nodes, compilation optimization items, driver interfaces, and memory access segments;

[0044] 3) Model defects were located in the network layer, weight parameters, quantization process, preprocessing logic, and postprocessing module;

[0045] 4) Generate a defect location report, including defect type, attribution level, location information, triggering conditions, reproduction steps, scope of impact, optimization suggestions and avoidance strategies;

[0046] 5) Archive defects by severity, scope of impact, and difficulty of reproduction, providing complete time-series data, hardware snapshots, software call stacks, and model execution trajectories, supporting one-click reproduction, full-process traceability, and auditability;

[0047] 6) Encapsulate defect information and test configuration into reproducible test cases for subsequent regression verification and system iteration optimization.

[0048] ④ Environment-Computation Joint Validation and Full-Stack Report Output

[0049] Combination Figure 1 Full-stack architecture and Figure 5 The output process completes the harsh environment coupling verification:

[0050] 1) Simultaneously apply environmental stresses such as temperature, vibration, and radiation, aligning environmental parameters with the calculation performance timeline;

[0051] 2) Quantify the coupled impact of the environment on accuracy, latency, power consumption, and robustness to form conclusions on the environment-computation correlation analysis;

[0052] 3) Output a complete full-stack verification report, covering five dimensions of indicators: correctness, accuracy, performance, reliability, and robustness;

[0053] 4) Save test configurations, timing data, and fault scenarios, supporting one-click reproduction, full-process traceability, and reproduction verification.

[0054] Through the above implementation steps and specific technologies, this invention realizes a layered full-stack testing and verification method for intelligent systems, which solves the problems of fragmented testing layers, insufficient localization adaptation, disconnect between environment and computing, and inaccurate defect location in traditional testing. It achieves full-coverage testing from chip packaging and testing to end-to-end applications, significantly improving the integrity of full-stack verification, the accuracy of defect location, and engineering practicality. It provides reliable technical support for the adaptation of domestic artificial intelligence ecosystem, the research and development of intelligent equipment for harsh environments, and the quality assurance of the entire life cycle.

Claims

1. A method for cross-level defect localization in intelligent systems based on hierarchical decoupling, characterized in that, By deploying three types of probes at the data flow layer, runtime layer, and hardware state layer, full-stack multi-dimensional data synchronous acquisition is achieved, breaking through the limitation of traditional testing that can only locate faults at the module level. A binary defect localization process of "freezing hardware - switching models - rolling back data" is constructed, enabling precise root cause localization at the chip register level and framework operator level. This solves the problems of difficult fault localization and slow root cause analysis caused by cross-level coupling in intelligent systems. The method consists of the following steps: 1) Deploy dedicated probes synchronously in the data flow layer, runtime layer, and hardware state layer of the intelligent system to establish a multi-dimensional, nanosecond-level synchronous data acquisition link; 2) Real-time acquisition of chip hardware instructions, register status, power consumption and timing data, as well as data on the execution of computing software stack operators, scheduling logic, memory access and framework operation status, while also acquiring model inference accuracy, latency, throughput and output anomaly data. 3) When inference accuracy degradation, performance drop, power consumption abnormality, inference crash, or output distortion abnormality is detected, the binary search hierarchical decoupling and localization process is automatically started. 4) Execute the hierarchical isolation operations of "freeze hardware running status - switch baseline model - roll back input data" in sequence to distinguish the defect attribution level step by step; 5) Based on the decoupling results, complete the root cause localization of defects, output the localization results at the chip register level, framework operator level, or model application level, and generate a reproducible and traceable defect localization report.

2. The method for synchronously deploying dedicated probes at the data flow layer, runtime layer, and hardware state layer as described in claim 1, characterized in that: 1) Data flow layer probes are used to collect data from the entire chain of model input and output, tensor reading and writing, data preprocessing and postprocessing, and record data distribution, outliers and precision drift; runtime layer probes are used to collect information on deep learning framework operator execution status, scheduling latency, thread preemption, cache hit rate, memory access and driver interaction. Hardware status layer probes achieve non-intrusive data acquisition through JTAG / SWD / PCIe interfaces, obtaining chip register values, instruction pointers, power consumption waveforms, I / O timing and signal integrity data; 2) All three types of probes use a unified PTP precision clock for synchronization, with timestamp alignment accuracy ≤100ns, ensuring complete consistency in timing of multi-source data.

3. The real-time acquisition of chip hardware, software stack, and model application data across all dimensions as described in claim 1, characterized in that: 1) The chip hardware data acquisition includes chip electrical characteristics, computing load, power consumption waveform, interface communication, register status and fault injection response; the software stack data acquisition includes operator execution cycle, compilation optimization status, memory out-of-bounds, data dependency conflicts, driver compatibility and scheduling anomalies. The data collected for model application includes model inference accuracy, confidence, end-to-end latency, throughput, robustness, output distortion, and quantization loss. 2) All data is stored in a unified time-series structure to form a full-stack test dataset that is associative, traceable, and reproducible.

4. The method described in claim 1, which detects abnormalities such as deterioration in detection accuracy and sharp drop in performance and initiates bisection hierarchical decoupling, is characterized in that: 1) Establish an anomaly detection rule base to monitor in real time for accuracy drift, sudden increase in latency, power consumption jump, inference crash, illegal output, and operator execution failure; 2) Once an exception is triggered, immediately save all stack data, including register snapshots, operator call stacks, intermediate model results, and input datasets; 3) Automatically initiate the binary decoupling process, isolating and locating layers step by step in the order of "hardware layer → software stack layer → model application layer"; 4) Abnormal scenarios are fully preserved, supporting fault reproduction, root cause analysis, and cross-level correlation analysis.

5. The binary search positioning process of "freezing hardware - switching models - rolling back data" as described in claim 1, characterized in that: 1) Hardware Freeze Steps: Keep the operating state and workload of the chip under test unchanged, shield environmental disturbances and external command interference, and lock the hardware operating environment; 2) Model switching steps: Replace the model under test with a standardized baseline model, keep the framework and hardware unchanged, and determine whether the defect comes from the model or the data; 3) Data rollback step: Restore the input data to the benchmark test set and clean samples to determine whether the defect is caused by data anomalies, noise or distribution offset; 4) Separate and cross-validate at each level to quickly distinguish defects as hardware defects, software stack defects, model defects, or cross-level coupling defects.

6. The method for completing defect root cause localization and outputting register-level / operator-level results and generating a reproducible and traceable defect localization report according to claim 1, characterized in that: 1) Hardware defects are located to specific registers, memory units, I / O pins or instruction execution units of the chip; software stack defects are located to specific operators, scheduling nodes, compilation optimization items, driver interfaces or memory access segments; model defects are located to network layers, weight parameters, quantization stages, preprocessing logic or postprocessing modules. 2) Output a defect location report, including defect type, attribution level, location information, triggering conditions, reproduction steps, scope of impact, optimization suggestions and avoidance strategies; 3) Archive defect information by defect level, scope of impact, and difficulty of reproduction, and provide complete time-series links, hardware snapshots, software call stacks and model execution trajectories simultaneously; 4) Supports one-click reproduction function, which can completely reproduce the defect scene on the same test platform, providing a basis for system debugging, defect repair and reliability improvement.

Images