Tranform model hierarchical reasoning framework based on hardware perception
By adopting a hardware-aware hierarchical inference framework, the problems of idle computing resources and communication congestion in the Transformer model in a multi-device environment are solved, achieving dynamic optimization and efficient utilization of resources, thereby improving overall inference efficiency and system reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-08
- Publication Date
- 2026-04-14
AI Technical Summary
Existing Transformer models suffer from issues such as idle computing resources, memory bottlenecks, communication congestion, inability of static layering strategies to adapt to real-time load fluctuations, lack of dynamic optimization capabilities, and insufficient cross-layer dependency management during inference in multi-device environments, leading to a decrease in overall throughput and an increase in latency.
A hardware-aware hierarchical reasoning framework is adopted, including a framework adaptation layer, a scheduling layer, a communication adaptation layer, and a hardware support layer. Through hardware abstraction, model adaptation, hierarchical combination, dynamic optimization, and pre-execution verification, intelligent perception and dynamic scheduling of hardware resources are achieved, optimizing the overlapping execution of computation and communication.
It significantly improves the efficiency and controllability of collaborative inference across large models and multiple devices, reduces latency, improves hardware utilization and system robustness, supports unified perception and abstract modeling of various types of heterogeneous hardware, and achieves seamless cross-platform execution and adaptive load balancing.
Smart Images
Figure CN121860056A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a reasoning framework, specifically a hardware-aware Transformer model hierarchical reasoning framework, belonging to the fields of artificial intelligence and large model technology. Background Technology
[0002] As the scale of deep learning models continues to expand, the Transformer architecture has demonstrated outstanding performance in natural language processing, computer vision, and multimodal tasks. However, Transformer models typically contain hundreds to thousands of layers and billions or even tens of billions of parameters, placing extremely high demands on hardware in terms of inference computation and storage requirements. In practical deployments, a single device is often insufficient to handle the inference task of the entire model, necessitating the collaborative execution of multiple devices.
[0003] Furthermore, the complex inter-layer dependencies of the Transformer model and the attention mechanism introduce numerous cross-layer interactions, making the inference process not only computationally demanding but also sensitive to inter-device communication latency and bandwidth. When large models are executed in multi-device environments, without a reasonable hierarchical partitioning and scheduling strategy, idle computing resources or memory bottlenecks and communication congestion can easily occur, leading to a decrease in overall throughput and an increase in latency.
[0004] Meanwhile, most existing inference frameworks employ a static mapping strategy, which assigns the model layer to specific hardware before deployment. This makes them unable to adapt to real-time load fluctuations or hardware state changes, and lacks dynamic optimization capabilities. While mixed precision or quantization strategies can reduce memory usage, they can usually only be configured offline and cannot adaptively adjust according to runtime load.
[0005] Therefore, for efficient inference of large-scale Transformer models in heterogeneous hardware environments, there is an urgent need for a hierarchical inference framework that can uniformly perceive hardware capabilities, dynamically optimize hierarchical mapping, achieve a balance between accuracy and performance, and make full use of cross-layer communication and computational overlap to improve overall inference efficiency, scalability and system reliability.
[0006] The existing technology has the following drawbacks:
[0007] Hardware compatibility is fragmented: Traditional frameworks rely heavily on manual configuration or specific interfaces to support different types of hardware (CPU, GPU, NPU, FPGA, etc.), lacking a unified abstraction layer, which makes cross-platform migration difficult.
[0008] Static layering and allocation strategy: Model layer mapping is usually determined before deployment, ignoring real-time hardware state changes and task load fluctuations, which can easily cause some devices to be overloaded, video memory to be fully occupied, or communication bottlenecks.
[0009] Lack of dynamic optimization capabilities: Existing frameworks lack pre-execution verification and dynamic adjustment mechanisms for inference plans, and cannot dynamically optimize performance based on hardware utilization, communication latency, or batch size during inference.
[0010] It is difficult to balance accuracy and performance: Large model inference has high requirements for video memory and computing resources. Mixed precision and quantization strategies can often only be configured offline and lack online adaptive adjustment capabilities.
[0011] Insufficient cross-layer dependency management: The dependencies and communication relationships between complex model layers are not fully considered, and the optimization effects of pipeline parallelism and computation-communication overlap are limited. Therefore, a new solution is urgently needed to address this technical problem. Summary of the Invention
[0012] This invention addresses the technical problems existing in the prior art by providing a hardware-aware Transformer model hierarchical inference framework, aiming to overcome the shortcomings of the prior art. The framework includes a framework adaptation layer, a scheduling layer, a communication adaptation layer, and a hardware support layer.
[0013] To achieve the above objectives, the technical solution of the present invention is as follows: a hardware-aware Transformer model hierarchical inference framework, the framework including a framework adaptation layer, a scheduling layer, a communication adaptation layer, and a hardware support layer, and the specific implementation method includes the following steps:
[0014] Step 1: Perform hardware abstraction.
[0015] Step 2: Model adapter initialization.
[0016] Step 3: Based on hardware capabilities and model structure, perform hierarchical combination of models.
[0017] Step 4: Dynamic optimization and pre-execution verification of the hierarchical inference plan.
[0018] Step 5: Intermediate results generated by each device are collected uniformly at the scheduling layer.
[0019] Step 1, hardware abstraction, is as follows:
[0020] 1.1 Hardware topology discovery and system-level resource scanning
[0021] The framework scans global computing resources through a hardware detection subsystem to identify the topology and connectivity of various computing units, including but not limited to: the number of GPUs / NPUs, their generations, interconnect topology, storage hierarchy, network communication link bandwidth and latency, and inter-node topology.
[0022] A computational node graph is constructed using topological traversal, and the hardware graph is represented using a weighted adjacency matrix, enabling graph computation and path selection during subsequent hierarchical reasoning.
[0023] 1.2 Quantification of computing power and modeling of heterogeneous execution characteristics,
[0024] In addition to hardware testing, it is also necessary to evaluate the hardware's actual execution efficiency curve, construct a hardware capability feature vector, conduct basic computing power assessment, obtain the hardware's instruction set performance curve, and obtain a dynamic execution characteristic model by dynamically evaluating factors such as throughput and latency, thus forming a hardware capability embedding.
[0025] 1.3 Storage capacity modeling and tensor scheduling cost modeling,
[0026] Transformer models are often constrained by memory-level bandwidth, therefore a cost function from the memory level to the operator needs to be established, focusing on evaluating: read / write bandwidth, probability of multi-stream concurrent access conflicts, and tensor rearrangement cost. A memory-level cost function is constructed as follows:
[0027] Cs = α * access_latency(shape) + β * bandwidth_cost(shape) + γ * re-layout_cost
[0028] 1.4 Construct a unified hardware capability description.
[0029] All the data collected above is eventually aggregated into a unified data structure that supports multiple granularities from the operator level to the node level.
[0030] New hardware features can be dynamically loaded and directly used in graph algorithms or optimization searches.
[0031] Step 2, model adapter initialization, involves parsing and reconstructing the original Transformer model structure into a unified intermediate representation (UTIR) that can be recognized, segmented, and scheduled by the hierarchical inference framework. The specific details are as follows:
[0032] 2.1 Model structure analysis and semantic layering,
[0033] The model adapter first performs structural parsing on the input Transformer model. The adapter uses a semantic layering algorithm.
[0034] Transform the model into structured layers (attention layers, feedforward layers, etc.) with explicit semantic labels.
[0035] In a Transformer model, the layers are denoted as L1, L2, ..., Ln, the input is represented by X, and the output is represented by Y. Semantic layering can be represented by the mapping function fsem, which indicates the mapping from each layer to the layer group.
[0036] S=fsem(L)
[0037] Where L={L1,L2,…,Ln} and S={S1,S2,…,Sm}, it represents mapping n layers to m semantic hierarchical groups.
[0038] 2.2 Operator pattern normalization and equivalent transformation,
[0039] The operators in Transformer models differ across frameworks (PyTorch, TensorFlow, and custom inference frameworks). This step executes an operator pattern normalization algorithm to identify normalizable operator patterns, replaces them using standard operator templates specified by UTIR, and merges operator chains with equivalent mathematical semantics. Operator fusion and defusion are supported.
[0040] 2.3 Parameters and Tensor Projection
[0041] After the model's semantic structure stabilizes, the parameters and tensors are bound to the nodes of the UTIR using a parametric projection algorithm, including:
[0042] Capabilities in parameters such as weight quantization features, accuracy requirements, and dynamic range;
[0043] The tensor size and lifetime required for KV Cache;
[0044] Intermediate activation storage and scheduling strategies;
[0045] The parametric projection algorithm mainly implements the following functions:
[0046] The placement of parameters is estimated based on hardware capabilities;
[0047] Perform preliminary mixing precision marking on the parameters;
[0048] Model the lifecycle of the KV cache and generate KV lifecycle descriptors;
[0049] This step, which explicitly establishes the initial association between parameters and hardware, is a core prerequisite for hierarchical inference to perform precision switching and storage optimization.
[0050] 2.4 Execution graph construction and cross-layer dependency graph generation,
[0051] Based on the UTIR representation of the semantic layer, a directed acyclic graph representation is used, and the execution sequence is automatically generated based on dependency topology sorting. The adapter constructs two types of graph structures:
[0052] (1) Execution Graph (EG):
[0053] Describe the execution order and parallelization possibilities of all operators.
[0054] The nodes in EG include high-granularity execution units such as Attention, FFN, and LayerNorm.
[0055] (2) Inter-layer Dependency Graph (IDG):
[0056] Cross-layer dependency graphs are used to characterize: residual connections, cross-layer references of LayerNorm, KV Cache write / read dependencies, and dependencies between encoders and decoders;
[0057] This layer makes the complex cross-layer dependencies of Transformer explicit, providing a foundation for subsequent layered composition and hardware-aware scheduling.
[0058] It facilitates the segmentation and mapping of dependency chains, reducing the cost of cross-hardware communication.
[0059] 2.5 Model schedulability analysis and hierarchical capability assessment: The model adapter ultimately generates a model schedulability description based on UTIR, EG, and IDG.
[0060] The content includes: computational density of each layer, communication requirements of each layer, feasibility of mixed precision switching, and topological compatibility of the layered structure.
[0061] The smallest unit of interlayer reorganization.
[0062] Step 3 involves layering and combining models based on hardware capabilities and model structure.
[0063] The specific implementation method includes the following sub-steps:
[0064] 3.1 Hardware performance modeling,
[0065] The system performs performance abstraction on various hardware components and extracts computing power. The metrics include memory capacity (Mh), bandwidth (Bh), and latency (Lh). A unified resource matrix is constructed, which provides the basis for subsequent model hierarchical mapping and can be dynamically updated to adapt to changes in hardware status.
[0066] 3.2 Semantic grouping at the model layer
[0067] The Transformer model layers L1, L2, ..., Ln are divided into several semantic hierarchical groups S1, S2, ..., Sm using the semantic hierarchical algorithm S = fsem(L). The algorithm considers features such as inter-layer dependencies, attention patterns, computational cost, and communication overhead.
[0068] The following metrics are calculated for each layer group: computational load, GPU memory requirements, and cross-layer communication overhead.
[0069] 3.3 Hardware-Model Matching
[0070] Calculate the matching degree Pi,h based on the resource requirements Wi, Mi, Ti of each hierarchical group and the hardware resource matrix H:
[0071]
[0072] in , where is an adjustable weighting coefficient. The hardware node with the highest matching degree is selected to execute this hierarchical group, thereby achieving the optimal mapping between hardware and model.
[0073] 3.4 Layered pipeline combination: optimize the layer allocation order according to the hardware topology, minimize cross-device communication, and use same-node allocation or low-latency link transmission for closely dependent layer groups to form a layered pipeline and realize the overlap of computing and communication.
[0074] 3.5 Hybrid precision and dynamic adjustment strategy: For computationally intensive or communication-intensive layers, a hybrid precision strategy is adopted to reduce memory usage and computation latency. During inference, the hardware load is monitored in real time to achieve dynamic layer mapping adjustment and adaptive load balancing, ensuring the overall inference performance is optimized.
[0075] Step 4, dynamic optimization and pre-execution verification of the hierarchical inference plan, involves performing consistency verification and local or global optimization on the plan based on real-time hardware status and preset constraints before issuing the preliminary hierarchical inference plan for execution. This generates a final executable plan or a safe rollback scheme to ensure that the inference meets the target requirements in terms of performance, resources, and accuracy. The details are as follows:
[0076] 4.1 Fast consistency check: Detects Plan0 with minimal latency whether it violates the most basic resource / topology constraints.
[0077] Check memory constraints: Verify memory compliance for each device.
[0078] Check link bandwidth: For each link e, verify the concurrent transmission capacity utilization.
[0079] Check topology connectivity: Confirm that all cross-device paths are reachable in UHCD.
[0080] If successful, proceed directly to 4.4 (perform pre-simulation); otherwise, proceed to 4.2.
[0081] 4.2 Local conflict location and repair: Prioritize the repair of conflicts identified in 4.1 using the least cost-effective methods.
[0082] Local reduction in precision: Temporarily change non-critical layers from FP16 to FP8 or further quantize them;
[0083] Reduce batch size: decrease the size of each batch to reduce memory usage spikes;
[0084] Layer merging: Combining tightly coupled layers into the same device to eliminate cross-device communication;
[0085] Proximity migration: migrate conflicting layers to topologically adjacent and idle devices.
[0086] Technical points: The repair attempt uses a heuristic cost evaluation function, applying changes only when acceptable; 4.1 is executed after each local repair until the conflict is resolved or the attempt limit is reached;
[0087] 4.3 Approximate Global Optimization: When local strategies fail to satisfy constraints, an approximate global remapping is performed to obtain a feasible solution.
[0088] The semantic layer diagram is partitioned to generate layer group units;
[0089] Perform a constrained greedy matching on the partitioned layers: prioritize mapping high-demand partitions to nodes with strong computing power, and place low-demand partitions closer to nodes with abundant cache / bandwidth.
[0090] 4.4 Perform pre-simulation and risk assessment, using lightweight simulation or fast models to verify PlanN's performance and resource consumption under the current UHCD, and identify high-risk points:
[0091] A model-based delay estimator was used to calculate the P95 / P99 estimate;
[0092] Perform short-term stress tests on critical links and critical layers to verify bandwidth and throughput assumptions;
[0093] If the risk is too high, a rollback strategy will be generated.
[0094] 4.5 Generate the final plan and prepare for distribution.
[0095] PlanN is embedded into a delivery structure, which includes device-side execution metadata, monitoring anchor points, and rollback trigger conditions.
[0096] Before deployment, write the snapshot point and fault tolerance checkpoint policies, and configure monitoring and alarm thresholds.
[0097] In step 5, the intermediate results generated by each device are collected uniformly at the scheduling layer. After each device completes its hierarchical inference subtask in parallel, the scheduling layer is responsible for uniformly collecting, verifying, dequantizing, and reconstructing the intermediate results. The results are aggregated and post-processed according to the model hierarchy and semantic relationships, and finally outputting a complete inference result that meets the accuracy and timing constraints. When necessary, partial recalculation or rollback strategies are triggered to ensure result consistency and system robustness, as detailed below:
[0098] 5.1 Intermediate result retrieval and acceptance confirmation: Reliably and sequentially receive and register intermediate results from each device, and establish a reception tracking table to support subsequent consistency checks and retries.
[0099] 5.2 Intermediate result format standardization and dequantization: Incremental dequantization and local buffering are used to avoid instantaneous memory spikes caused by overall expansion. Intermediate results from different devices, with different precisions and memory layouts are unified into a standard representation within the scheduling layer, facilitating semantic aggregation and post-processing.
[0100] Layout Reconstruction: Convert common layouts into a unified internal layout, rearranging only necessary dimensions to reduce memory copying overhead.
[0101] Sparse / Compressed Recovery: Performs a recovery algorithm on sparsely coded or bit-compressed data, prioritizing incremental recovery to reduce peak memory usage.
[0102] 5.3 Result Aggregation and Semantic Reconstruction: Standardized intermediate results are merged, concatenated, and reconstructed according to model topology and semantic dependencies to generate the input tensor for the next stage or final output. Streaming aggregation is employed to support pipelined result arrival, avoiding waiting for all intermediate results to complete before aggregation begins.
[0103] Inter-layer mapping table: Using the layer group-to-result mapping relationship provided by PlanN, the merge order and aggregation strategy are determined.
[0104] splicing / merging operations:
[0105] Perform semantic concatenation on the tensors to be concatenated; maintain index consistency and temporal alignment during concatenation;
[0106] Perform local merging on fusionable operators to reduce subsequent computation;
[0107] Consistency check: After aggregation, range checks and normalization verifications are performed on key statistics.
[0108] 5.4 Post-processing and accuracy correction: Based on the aggregated results, dequantization, semantic alignment, output scaling, and necessary accuracy correction are performed to ensure that the final output meets the quality indicators.
[0109] Dequantization and scaling: Perform a unified scaling mapping on paths with mixed precision, and correct system biases using least squares or calibration tables.
[0110] Semantic concatenation: Performs semantic joining, duplication removal, and boundary processing on text / sequence results generated across fragments.
[0111] Eventual consistency verification: Quality checks are performed using task-related metrics; if the deviation exceeds the threshold, a partial rollback is executed and a partial recalculation is triggered, recalculating only the affected layers to save time.
[0112] 5.5 Output deployment and runtime data records, return the final results to the client or subsequent processing modules, and fully record runtime logs, performance metrics, and reusable intermediate caches to support acceleration of subsequent requests.
[0113] Output encapsulation: The final result is packaged into an output unit and sent down or written back to shared storage via the control plane.
[0114] Result caching and reuse: Intermediate results with reuse value are written to UIRP and indexed, supporting cross-request sharing and cache hit strategies.
[0115] Auditing and Metric Recording: Recording key metrics and parameters during the process provides data for the scheduler's adaptive learning and offline analysis.
[0116] Rollback output: If some recalculations fail or the risk is too high, return a conservative output and mark the alternative and quality impact in the metadata.
[0117] The above technical solutions enable intelligent perception and dynamic scheduling of heterogeneous hardware resources, maximize the utilization of the overall computing power of the system, significantly improve the efficiency and controllability of collaborative inference of large models across multiple devices, effectively reduce latency, and improve inference quality and overall service experience.
[0118] Compared with existing technologies, this invention has the following advantages: 1) It achieves unified perception and abstract modeling of multiple types of heterogeneous hardware. This invention automatically identifies various hardware device types such as CPU, GPU, NPU, and FPGA through a resource perception module, and comprehensively models their performance indicators such as computing power, memory, bandwidth, latency, and energy consumption; forming a unified resource abstract description model, enabling upper-layer scheduling strategies to be executed seamlessly across hardware platforms; solving the problems of fragmented support for heterogeneous devices and complex manual configuration in traditional inference frameworks, thus improving the system's versatility and scalability. The scheduling layer of the topology-based intelligent scheduling and load balancing framework adopts a dynamic monitoring and feedback mechanism, adjusting the computation graph partitioning and task allocation in real time according to the task execution status; combined with hardware topology (such as the NVLink interconnection relationship between GPUs), it achieves optimal path scheduling based on topology perception; effectively reducing cross-device communication latency and improving the overall parallel efficiency and load balancing of tasks; in large-scale multi-device scenarios, it implements intelligent inference scheduling strategies. It achieves time overlap and performance optimization between communication and computation.
[0119] During the inference phase, the framework coordinates communication and computation tasks through a communication adaptation layer, achieving pipelined overlapping of communication and computation. Through techniques such as mixed-precision computation, tensor partitioning, and asynchronous data transmission, it effectively reduces latency and improves hardware utilization. Compared to traditional serial inference, this invention significantly improves system throughput and inference speed while reducing average latency. It possesses adaptive resource dynamic adjustment capabilities, sensing device load, power consumption, and temperature in real time during inference. When a bottleneck occurs in a certain type of device, it can automatically migrate tasks or adjust precision strategies. It supports resource reuse and task isolation in multi-task scenarios, ensuring priority execution of critical inference tasks. This mechanism significantly improves the system's robustness and reliability in complex environments. The framework adaptation layer is compatible with multiple mainstream AI frameworks and inference engines, including PyTorch, TensorFlow, MindSpore, and inference engines such as LLAMA.cpp and vLLM.
[0120] Users can seamlessly switch between different models and engines without modifying the core code, achieving a highly compatible architecture that is model-independent and platform-independent; it significantly reduces the threshold for model deployment and improves cross-platform inference efficiency.
[0121] The modular system architecture design improves system maintainability and scalability, enabling independent upgrades and expansions of the scheduling layer, communication layer, framework layer, and hardware layer; it supports rapid access to new hardware (such as AI chips and edge inference cards) and has the ability to continuously evolve and iterate technologies; and it provides a general basic support framework for future multimodal models, distributed training, edge-cloud collaboration, and other scenarios. Attached Figure Description
[0122] Figure 1 This is a system module relationship diagram.
[0123] Figure 2 This is a sequence diagram of the inference execution process;
[0124] Figure 3 Diagram of heterogeneous resource perception and modeling mechanism;
[0125] Figure 4 A flowchart for dynamic task scheduling and load balancing;
[0126] Figure 5 Communication optimization and computational overlap execution structure diagram. Detailed Implementation
[0127] To enhance understanding of the present invention, the embodiments will be described in detail below with reference to the accompanying drawings.
[0128] Example 1: See Figure 1 , Figure 2 A hardware-aware Transformer model hierarchical inference framework is proposed, comprising a framework adaptation layer, a scheduling layer, a communication adaptation layer, and a hardware support layer. The specific implementation method includes the following steps:
[0129] Step 1: Perform hardware abstraction.
[0130] Step 2: Model adapter initialization.
[0131] Step 3: Based on hardware capabilities and model structure, perform hierarchical combination of models.
[0132] Step 4: Dynamic optimization and pre-execution verification of the hierarchical inference plan.
[0133] Step 5: Intermediate results generated by each device are collected uniformly at the scheduling layer.
[0134] Step 1, hardware abstraction, is as follows:
[0135] 1.1 Hardware topology discovery and system-level resource scanning
[0136] The framework scans global computing resources through a hardware detection subsystem to identify the topology and connectivity of various computing units, including but not limited to: the number of GPUs / NPUs, their generations, interconnect topology, storage hierarchy, network communication link bandwidth and latency, and inter-node topology.
[0137] A computational node graph is constructed using topological traversal, and the hardware graph is represented using a weighted adjacency matrix, enabling graph computation and path selection during subsequent hierarchical reasoning.
[0138] 1.2 Quantification of computing power and modeling of heterogeneous execution characteristics,
[0139] In addition to hardware testing, it is also necessary to evaluate the hardware's actual execution efficiency curve, construct a hardware capability feature vector, conduct basic computing power assessment, obtain the hardware's instruction set performance curve, and obtain a dynamic execution characteristic model by dynamically evaluating factors such as throughput and latency, thus forming a hardware capability embedding.
[0140] 1.3 Storage capacity modeling and tensor scheduling cost modeling,
[0141] Transformer models are often constrained by memory-level bandwidth, therefore a cost function from the memory level to the operator needs to be established, focusing on evaluating: read / write bandwidth, probability of multi-stream concurrent access conflicts, and tensor rearrangement cost. A memory-level cost function is constructed as follows:
[0142] Cs = α * access_latency(shape) + β * bandwidth_cost(shape) + γ * re-layout_cost
[0143] 1.4 Construct a unified hardware capability description.
[0144] All the data collected above is eventually aggregated into a unified data structure that supports multiple granularities from the operator level to the node level.
[0145] New hardware features can be dynamically loaded and directly used in graph algorithms or optimization searches.
[0146] Step 2, model adapter initialization, involves parsing and reconstructing the original Transformer model structure into a unified intermediate representation (UTIR) that can be recognized, segmented, and scheduled by the hierarchical inference framework. The specific details are as follows:
[0147] 2.1 Model structure analysis and semantic layering,
[0148] The model adapter first performs structural parsing on the input Transformer model. The adapter uses a semantic layering algorithm.
[0149] Transform the model into structured layers (attention layers, feedforward layers, etc.) with explicit semantic labels.
[0150] In a Transformer model, the layers are denoted as L1, L2, ..., Ln, the input is represented by X, and the output is represented by Y. Semantic layering can be represented by the mapping function fsem, which indicates the mapping from each layer to the layer group.
[0151] S=fsem(L)
[0152] Where L={L1,L2,…,Ln} and S={S1,S2,…,Sm}, it represents mapping n layers to m semantic hierarchical groups.
[0153] 2.2 Operator pattern normalization and equivalent transformation,
[0154] The operators in Transformer models differ across frameworks (PyTorch, TensorFlow, and custom inference frameworks). This step executes an operator pattern normalization algorithm to identify normalizable operator patterns, replaces them using standard operator templates specified by UTIR, and merges operator chains with equivalent mathematical semantics. Operator fusion and defusion are supported.
[0155] 2.3 Parameters and Tensor Projection
[0156] After the model's semantic structure stabilizes, the parameters and tensors are bound to the nodes of the UTIR using a parametric projection algorithm, including:
[0157] Capabilities in parameters such as weight quantization features, accuracy requirements, and dynamic range;
[0158] The tensor size and lifetime required for KV Cache;
[0159] Intermediate activation storage and scheduling strategies;
[0160] The parametric projection algorithm mainly implements the following functions:
[0161] The placement of parameters is estimated based on hardware capabilities;
[0162] Perform preliminary mixing precision marking on the parameters;
[0163] Model the lifecycle of the KV cache and generate KV lifecycle descriptors;
[0164] This step, which explicitly establishes the initial association between parameters and hardware, is a core prerequisite for hierarchical inference to perform precision switching and storage optimization.
[0165] 2.4 Execution graph construction and cross-layer dependency graph generation,
[0166] Based on the UTIR representation of the semantic layer, a directed acyclic graph representation is used, and the execution sequence is automatically generated based on dependency topology sorting. The adapter constructs two types of graph structures:
[0167] (1) Execution Graph (EG):
[0168] Describe the execution order and parallelization possibilities of all operators.
[0169] The nodes in EG include high-granularity execution units such as Attention, FFN, and LayerNorm.
[0170] (2) Inter-layer Dependency Graph (IDG):
[0171] Cross-layer dependency graphs are used to characterize: residual connections, cross-layer references of LayerNorm, KV Cache write / read dependencies, and dependencies between encoders and decoders;
[0172] This layer makes the complex cross-layer dependencies of Transformer explicit, providing a foundation for subsequent layered composition and hardware-aware scheduling.
[0173] It facilitates the segmentation and mapping of dependency chains, reducing the cost of cross-hardware communication.
[0174] 2.5 Model schedulability analysis and hierarchical capability assessment: The model adapter ultimately generates a model schedulability description based on UTIR, EG, and IDG.
[0175] The content includes: computational density of each layer, communication requirements of each layer, feasibility of mixed precision switching, and topological compatibility of the layered structure.
[0176] The smallest unit of interlayer reorganization.
[0177] Step 3 involves layering and combining models based on hardware capabilities and model structure.
[0178] The specific implementation method includes the following sub-steps:
[0179] 3.1 Hardware performance modeling,
[0180] The system performs performance abstraction on various hardware components and extracts computing power. The metrics include memory capacity (Mh), bandwidth (Bh), and latency (Lh). A unified resource matrix is constructed, which provides the basis for subsequent model hierarchical mapping and can be dynamically updated to adapt to changes in hardware status.
[0181] 3.2 Semantic grouping at the model layer
[0182] The Transformer model layers L1, L2, ..., Ln are divided into several semantic hierarchical groups S1, S2, ..., Sm using the semantic hierarchical algorithm S = fsem(L). The algorithm considers features such as inter-layer dependencies, attention patterns, computational cost, and communication overhead.
[0183] The following metrics are calculated for each layer group: computational load, GPU memory requirements, and cross-layer communication overhead.
[0184] 3.3 Hardware-Model Matching
[0185] Calculate the matching degree Pi,h based on the resource requirements Wi, Mi, Ti of each hierarchical group and the hardware resource matrix H:
[0186]
[0187] in , where is an adjustable weighting coefficient. The hardware node with the highest matching degree is selected to execute this hierarchical group, thereby achieving the optimal mapping between hardware and model.
[0188] 3.4 Layered pipeline combination: optimize the layer allocation order according to the hardware topology, minimize cross-device communication, and use same-node allocation or low-latency link transmission for closely dependent layer groups to form a layered pipeline and realize the overlap of computing and communication.
[0189] 3.5 Hybrid precision and dynamic adjustment strategy: For computationally intensive or communication-intensive layers, a hybrid precision strategy is adopted to reduce memory usage and computation latency. During inference, the hardware load is monitored in real time to achieve dynamic layer mapping adjustment and adaptive load balancing, ensuring the overall inference performance is optimized.
[0190] Step 4, dynamic optimization and pre-execution verification of the hierarchical inference plan, involves performing consistency verification and local or global optimization on the plan based on real-time hardware status and preset constraints before issuing the preliminary hierarchical inference plan for execution. This generates a final executable plan or a safe rollback scheme to ensure that the inference meets the target requirements in terms of performance, resources, and accuracy. The details are as follows:
[0191] 4.1 Fast consistency check: Detects Plan0 with minimal latency whether it violates the most basic resource / topology constraints.
[0192] Check memory constraints: Verify memory compliance for each device.
[0193] Check link bandwidth: For each link e, verify the concurrent transmission capacity utilization.
[0194] Check topology connectivity: Confirm that all cross-device paths are reachable in UHCD.
[0195] If successful, proceed directly to 4.4 (perform pre-simulation); otherwise, proceed to 4.2.
[0196] 4.2 Local conflict location and repair: Prioritize the repair of conflicts identified in 4.1 using the least cost-effective methods.
[0197] Local reduction in precision: Temporarily change non-critical layers from FP16 to FP8 or further quantize them;
[0198] Reduce batch size: decrease the size of each batch to reduce memory usage spikes;
[0199] Layer merging: Combining tightly coupled layers into the same device to eliminate cross-device communication;
[0200] Proximity migration: migrate conflicting layers to topologically adjacent and idle devices.
[0201] Technical points: The repair attempt uses a heuristic cost evaluation function, applying changes only when acceptable; 4.1 is executed after each local repair until the conflict is resolved or the attempt limit is reached;
[0202] 4.3 Approximate Global Optimization: When local strategies fail to satisfy constraints, an approximate global remapping is performed to obtain a feasible solution.
[0203] The semantic layer diagram is partitioned to generate layer group units;
[0204] Perform a constrained greedy matching on the partitioned layers: prioritize mapping high-demand partitions to nodes with strong computing power, and place low-demand partitions closer to nodes with abundant cache / bandwidth.
[0205] 4.4 Perform pre-simulation and risk assessment, using lightweight simulation or fast models to verify PlanN's performance and resource consumption under the current UHCD, and identify high-risk points:
[0206] A model-based delay estimator was used to calculate the P95 / P99 estimate;
[0207] Perform short-term stress tests on critical links and critical layers to verify bandwidth and throughput assumptions;
[0208] If the risk is too high, a rollback strategy will be generated.
[0209] 4.5 Generate the final plan and prepare for distribution.
[0210] PlanN is embedded into a delivery structure, which includes device-side execution metadata, monitoring anchor points, and rollback trigger conditions.
[0211] Before deployment, write the snapshot point and fault tolerance checkpoint policies, and configure monitoring and alarm thresholds.
[0212] In step 5, the intermediate results generated by each device are collected uniformly at the scheduling layer. After each device completes its hierarchical inference subtask in parallel, the scheduling layer is responsible for uniformly collecting, verifying, dequantizing, and reconstructing the intermediate results. The results are aggregated and post-processed according to the model hierarchy and semantic relationships, and finally outputting a complete inference result that meets the accuracy and timing constraints. When necessary, partial recalculation or rollback strategies are triggered to ensure result consistency and system robustness, as detailed below:
[0213] 5.1 Intermediate result retrieval and acceptance confirmation: Reliably and sequentially receive and register intermediate results from each device, and establish a reception tracking table to support subsequent consistency checks and retries.
[0214] 5.2 Intermediate result format standardization and dequantization: Incremental dequantization and local buffering are used to avoid instantaneous memory spikes caused by overall expansion. Intermediate results from different devices, with different precisions and memory layouts are unified into a standard representation within the scheduling layer, facilitating semantic aggregation and post-processing.
[0215] Layout Reconstruction: Convert common layouts into a unified internal layout, rearranging only necessary dimensions to reduce memory copying overhead.
[0216] Sparse / Compressed Recovery: Performs a recovery algorithm on sparsely coded or bit-compressed data, prioritizing incremental recovery to reduce peak memory usage.
[0217] 5.3 Result Aggregation and Semantic Reconstruction: Standardized intermediate results are merged, concatenated, and reconstructed according to model topology and semantic dependencies to generate the input tensor for the next stage or final output. Streaming aggregation is employed to support pipelined result arrival, avoiding waiting for all intermediate results to complete before aggregation begins.
[0218] Inter-layer mapping table: Using the layer group-to-result mapping relationship provided by PlanN, the merge order and aggregation strategy are determined.
[0219] splicing / merging operations:
[0220] Perform semantic concatenation on the tensors to be concatenated; maintain index consistency and temporal alignment during concatenation;
[0221] Perform local merging on fusionable operators to reduce subsequent computation;
[0222] Consistency check: After aggregation, range checks and normalization verifications are performed on key statistics.
[0223] 5.4 Post-processing and accuracy correction: Based on the aggregated results, dequantization, semantic alignment, output scaling, and necessary accuracy correction are performed to ensure that the final output meets the quality indicators.
[0224] Dequantization and scaling: Perform a unified scaling mapping on paths with mixed precision, and correct system biases using least squares or calibration tables.
[0225] Semantic concatenation: Performs semantic joining, duplication removal, and boundary processing on text / sequence results generated across fragments.
[0226] Eventual consistency verification: Quality checks are performed using task-related metrics; if the deviation exceeds the threshold, a partial rollback is executed and a partial recalculation is triggered, recalculating only the affected layers to save time.
[0227] 5.5 Output deployment and runtime data records, return the final results to the client or subsequent processing modules, and fully record runtime logs, performance metrics, and reusable intermediate caches to support acceleration of subsequent requests.
[0228] Output encapsulation: The final result is packaged into an output unit and sent down or written back to shared storage via the control plane.
[0229] Result caching and reuse: Intermediate results with reuse value are written to UIRP and indexed, supporting cross-request sharing and cache hit strategies.
[0230] Auditing and Metric Recording: Recording key metrics and parameters during the process provides data for the scheduler's adaptive learning and offline analysis.
[0231] Rollback output: If some recalculations fail or the risk is too high, return a conservative output and mark the alternative and quality impact in the metadata.
[0232] Figure 3 Diagram of heterogeneous resource perception and modeling mechanism:
[0233] This diagram reflects the system's resource modeling process: constructing a unified resource representation from multiple dimensions such as performance testing, energy consumption estimation, and bandwidth testing of heterogeneous devices, thereby achieving unified scheduling and performance awareness for different types of devices;
[0234] Figure 4 Dynamic task scheduling and load balancing flowchart:
[0235] This diagram illustrates the system's core "intelligent scheduling loop mechanism":
[0236] First, generate an initial allocation plan;
[0237] Continuously monitor latency, bandwidth, and utilization during execution;
[0238] When performance deviates from a threshold, a reallocation, migration, or mixed-precision strategy adjustment is triggered.
[0239] It achieves task-level dynamic optimization and system self-adaptation capabilities;
[0240] Figure 5 Communication optimization and computational overlap execution structure diagram:
[0241] As shown in the figure, communication and computation overlap during system inference execution:
[0242] While device A is performing the first stage of calculations, device B begins data reception preparation in advance.
[0243] Data transmission and computation are performed alternately to achieve pipelined parallel inference;
[0244] The communication adaptation layer reduces waiting time and improves overall throughput through topology awareness and communication path optimization algorithms.
[0245] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.
Claims
1. A hardware-aware, hierarchical inference framework for Transformer models, characterized in that, The framework includes a framework adaptation layer, a scheduling layer, a communication adaptation layer, and a hardware support layer. The specific implementation method includes the following steps: Step 1: Perform hardware abstraction. Step 2: Model adapter initialization. Step 3: Based on hardware capabilities and model structure, perform hierarchical combination of models. Step 4: Dynamic optimization and pre-execution verification of the hierarchical inference plan. Step 5: Intermediate results generated by each device are collected uniformly at the scheduling layer.
2. The hardware-aware Transformer model hierarchical inference framework according to claim 1, characterized in that, Step 1: Perform hardware abstraction, as follows: 1.1 Hardware topology discovery and system-level resource scanning The framework scans global computing resources through a hardware detection subsystem to identify the topology and connectivity of various computing units, including but not limited to: the number of GPUs / NPUs, their generations, interconnect topology, storage hierarchy, network communication link bandwidth and latency, and inter-node topology. A computational node graph is constructed using topological traversal, and the hardware graph is represented using a weighted adjacency matrix, enabling graph computation and path selection during subsequent hierarchical inference. 1.2 Quantification of computing power and modeling of heterogeneous execution characteristics, In addition to hardware testing, it is also necessary to evaluate the hardware's actual execution efficiency curve, construct a hardware capability feature vector, conduct basic computing power assessment, obtain the hardware's instruction set performance curve, and obtain a dynamic execution characteristic model by dynamically evaluating factors such as throughput and latency, thus forming a hardware capability embedding. 1.3 Storage capacity modeling and tensor scheduling cost modeling, Transformer models are often constrained by memory-level bandwidth, therefore a cost function from the memory level to the operator needs to be established, focusing on evaluating: read / write bandwidth, probability of multi-stream concurrent access conflicts, and tensor rearrangement cost. A memory-level cost function is constructed as follows: Cs = α * access_latency(shape) + β * bandwidth_cost(shape) + γ * re-layout_cost 1.4 Construct a unified hardware capability description. All the data collected above is eventually aggregated into a unified data structure that supports multiple granularities from the operator level to the node level. New hardware features can be dynamically loaded and can be directly used in graph algorithms or optimization searches.
3. The hardware-aware Transformer model hierarchical inference framework according to claim 1, characterized in that, Step 2: Model adapter initialization. The original Transformer model structure is parsed and reconstructed into a unified intermediate representation (UTIR) that can be recognized, segmented, and scheduled by the hierarchical inference framework. Specifically: 2.1 Model structure analysis and semantic layering, The model adapter first performs structural parsing on the input Transformer model. The adapter uses a semantic layering algorithm. Transform the model into a structured layer with explicit semantic labels. In a Transformer model, the layers are denoted as L1, L2, ..., Ln, the input is represented by X, and the output is represented by Y. Semantic layering can be represented by a mapping function fsem, which indicates the mapping from each layer to the layer group. S=fsem(L) Where L={L1,L2,…,Ln} and S={S1,S2,…,Sm}, it represents mapping n layers to m semantic hierarchical groups. 2.2 Operator pattern normalization and equivalent transformation, The operators of the Transformer model differ across different frameworks. This step executes an operator pattern normalization algorithm to identify normalizable operator patterns, replaces them using standard operator templates specified by UTIR, and merges operator chains with equivalent mathematical semantics, supporting operator fusion and defusion. 2.3 Parameters and Tensor Projection After the model's semantic structure stabilizes, the parameters and tensors are bound to the nodes of the UTIR using a parametric projection algorithm, including: Weight quantization features, accuracy requirements, and dynamic range parameter capabilities; The tensor size and lifetime required for KV Cache; Intermediate activation storage and scheduling strategies; The parametric projection algorithm mainly implements the following functions: The placement of parameters is estimated based on hardware capabilities; Perform preliminary mixing precision marking on the parameters; Model the lifecycle of the KV cache and generate KV lifecycle descriptors; 2.4 Execution graph construction and cross-layer dependency graph generation, Based on the UTIR representation of the semantic layer, a directed acyclic graph representation is used, and the execution sequence is automatically generated based on dependency topology sorting. The adapter constructs two types of graph structures: (1) Execution Graph (EG): Describe the execution order and parallelization possibilities of all operators. The nodes in EG include Attention, FFN, and LayerNorm high-granularity execution units. (2) Inter-layer Dependency Graph (IDG): Cross-layer dependency graphs are used to characterize: residual connections, cross-layer references of LayerNorm, KV Cache write / read dependencies, and dependencies between encoders and decoders; This layer makes the complex cross-layer dependencies of Transformer explicit, providing a foundation for subsequent layered composition and hardware-aware scheduling. It facilitates the segmentation and mapping of dependency chains, reducing the cost of cross-hardware communication. 2.5 Model schedulability analysis and hierarchical capability assessment: The model adapter ultimately generates a model schedulability description based on UTIR, EG, and IDG. The content includes: computational density of each layer, communication requirements of each layer, feasibility of mixed precision switching, and topologyability of the layered structure. The smallest unit of interlayer reorganization.
4. The hardware-aware Transformer model hierarchical inference framework according to claim 2, characterized in that, Step 3: Based on hardware capabilities and model structure, perform layered combination of models. The specific implementation method includes the following sub-steps: 3.1 Hardware performance modeling, The system performs performance abstraction on various hardware components and extracts computing power. The system uses memory capacity (Mh), bandwidth (Bh), and latency (Lh) to construct a unified resource matrix. This matrix provides the basis for subsequent hierarchical mapping of the model and can be dynamically updated to adapt to changes in hardware status. 3.2 Semantic grouping at the model layer The Transformer model layers L1, L2, ..., Ln are divided into several semantic hierarchical groups S1, S2, ..., Sm using the semantic hierarchical algorithm S = fsem(L). The algorithm considers inter-layer dependencies, attention patterns, computational cost, and communication overhead. The following metrics are calculated for each layer group: computational load, GPU memory requirements, and cross-layer communication overhead. 3.3 Hardware-Model Matching Calculate the matching degree Pi,h based on the resource requirements Wi, Mi, Ti of each hierarchical group and the hardware resource matrix H: in , where is an adjustable weighting coefficient. The hardware node with the highest matching degree is selected to execute this hierarchical group, thereby achieving the optimal mapping between hardware and model. 3.4 Layered pipeline assembly: Based on the hardware topology optimization, the layer allocation order is optimized to minimize cross-device communication. For closely dependent layer groups, intra-node allocation or low-latency link transmission is used to form a layered pipeline, achieving overlap between computation and communication. 3.5 Hybrid precision and dynamic adjustment strategy: For computationally intensive or communication-intensive layers, a hybrid precision strategy is adopted to reduce memory usage and computation latency. During inference operation, dynamic layer mapping adjustment and adaptive load balancing are achieved by monitoring hardware load in real time to ensure the optimization of overall inference performance.
5. The hardware-aware Transformer model hierarchical inference framework according to claim 3, characterized in that, Step 4: Dynamic optimization and pre-execution verification of the hierarchical inference plan. Before the preliminary hierarchical inference plan generated in Step 3 is deployed for execution, consistency verification and local or global optimization are performed on the plan based on real-time hardware status and preset constraints. A final executable plan or safe rollback scheme is then generated to ensure that the inference meets the target requirements in terms of performance, resources, and accuracy. Specifically, as follows: 4.1 Fast consistency check: Detects Plan0 with minimal latency whether it violates the most basic resource / topology constraints. Check memory constraints: Verify memory compliance for each device. Check link bandwidth: For each link e, verify the concurrent transmission capacity utilization. Check topology connectivity: Confirm that all cross-device paths are reachable in UHCD. If successful, proceed directly to 4.4 (perform pre-simulation); otherwise, proceed to 4.
2. 4.2 Local conflict location and repair: Prioritize the repair of conflicts identified in 4.1 using the least cost-effective methods. Local reduction in precision: Temporarily change non-critical layers from FP16 to FP8 or further quantize them; Reduce batch size: decrease the size of each batch to reduce memory usage spikes; Layer merging: Combining tightly coupled layers into the same device to eliminate cross-device communication; Proximity migration: Migrate conflicting layers to topologically adjacent and idle devices. Technical points: The repair attempt uses a heuristic cost evaluation function, applying changes only when acceptable; 4.1 is executed after each local repair until the conflict is resolved or the attempt limit is reached; 4.3 Approximate Global Optimization: When local strategies fail to satisfy constraints, an approximate global remapping is performed to obtain a feasible solution. The semantic layer diagram is partitioned to generate layer group units; Perform a constrained greedy matching on the partitioned layers: prioritize mapping high-demand partitions to nodes with strong computing power, and place low-demand partitions closer to nodes with abundant cache / bandwidth. 4.4 Perform pre-simulation and risk assessment, using lightweight simulation or fast models to verify PlanN's performance and resource consumption under the current UHCD, and identify high-risk points: A model-based delay estimator was used to calculate the P95 / P99 estimate; Perform short-term stress tests on critical links and critical layers to verify bandwidth and throughput assumptions; If the risk is too high, a rollback strategy will be generated. 4.5 Generate the final plan and prepare for distribution. PlanN is embedded into a delivery structure, which includes device-side execution metadata, monitoring anchor points, and rollback trigger conditions. Before deployment, write the snapshot point and fault tolerance checkpoint policies, and configure monitoring and alarm thresholds.
6. The hardware-aware Transformer model hierarchical inference framework according to claim 3, characterized in that, Step 5: Intermediate results generated by each device are collected uniformly at the scheduling layer. After each device completes its hierarchical inference subtask in parallel, the scheduling layer is responsible for uniformly collecting, verifying, dequantizing, and reconstructing the intermediate results. The results are aggregated and post-processed according to the model hierarchy and semantic relationships, ultimately outputting a complete inference result that meets accuracy and timing constraints. When necessary, partial recalculation or rollback strategies are triggered to ensure result consistency and system robustness, as detailed below: 5.1 Intermediate result retrieval and acceptance confirmation: Reliably and sequentially receive and register intermediate results from each device, and establish a reception tracking table to support subsequent consistency checks and retries. 5.2 Intermediate result format standardization and dequantization: Incremental dequantization and local buffering are used to avoid instantaneous memory spikes caused by overall expansion. This unifies intermediate results from different devices, with different precisions and memory layouts into a standard representation within the scheduling layer, facilitating semantic aggregation and post-processing. Layout Reconstruction: Convert common layouts into a unified internal layout, rearranging only necessary dimensions to reduce memory copying overhead. Sparse / Compressed Recovery: Performs a recovery algorithm on sparsely coded or bit-compressed data, prioritizing incremental recovery to reduce peak memory usage. 5.3 Result Aggregation and Semantic Reconstruction: Standardized intermediate results are merged, concatenated, and reconstructed according to model topology and semantic dependencies to generate the input tensor for the next stage or final output. Streaming aggregation is used to support pipelined result arrival, avoiding waiting for all intermediate results to complete before aggregation begins. Inter-layer mapping table: Using the layer group-to-result mapping relationship provided by PlanN, the merge order and aggregation strategy are determined. splicing / merging operations: Perform semantic concatenation on the tensors to be concatenated; maintain index consistency and temporal alignment during concatenation; Perform local merging on fusionable operators to reduce subsequent computation; Consistency check: After aggregation, range checks and normalization verifications are performed on key statistics. 5.4 Post-processing and accuracy correction: Based on the aggregated results, dequantization, semantic alignment, output scaling, and necessary accuracy correction are performed to ensure that the final output meets the quality indicators. Dequantization and scaling: Perform a unified scaling mapping on paths with mixed precision, and correct system biases using least squares or calibration tables. Semantic concatenation: Performs semantic joining, duplication removal, and boundary processing on text / sequence results generated across fragments. Eventual consistency verification: Quality checks are performed using task-related metrics; if the deviation exceeds the threshold, a partial rollback is executed and a partial recalculation is triggered, recalculating only the affected layers to save time. 5.5 Output deployment and runtime data records, return the final results to the client or subsequent processing modules, and fully record runtime logs, performance metrics, and reusable intermediate caches to support acceleration of subsequent requests. Output encapsulation: The final result is packaged into an output unit and sent down or written back to shared storage via the control plane. Result caching and reuse: Intermediate results with reuse value are written to UIRP and indexed, supporting cross-request sharing and cache hit strategies. Auditing and Metric Recording: Recording key metrics and parameters during the process provides data for the scheduler's adaptive learning and offline analysis. Rollback output: If some recalculations fail or the risk is too high, return a conservative output and mark the alternative and quality impact in the metadata.