Accelerator adaptation method for ai algorithm inference

Abstract

This invention relates to an accelerator adaptation method for AI algorithm inference. It involves structural analysis of the AI ​​inference model to be deployed, identifying computational paths that influence the inference results based on the target inference output, and pruning or reorganizing the model structure to generate a set of subtasks representing the main inference behaviors. Based on the basic computational unit structure of the target accelerator, the subtask set is modularized, ensuring that the computational granularity of each module matches the execution units supported by the accelerator, and adjusting the operation order between modules. For the execution paths of the modular tasks, the resource mapping relationships on the target accelerator are analyzed, and the execution order of computational paths located in resource bottleneck areas is reorganized. Based on the current execution state of the accelerator, including thread activity, cache usage, and resource consumption, a dynamic adaptation instruction sequence is generated and optimized, achieving multi-level adaptive optimization from the model structure level to instruction-level scheduling.

Description

Technical Field

[0001] This invention relates to the field of algorithm acceleration technology, specifically to an accelerator adaptation method for AI algorithm inference. Background Technology

[0002] Existing technologies, such as Chinese patent CN114691148A (Model Inference Acceleration Method, Device, Electronic Device, and Storage Medium), primarily focus on converting deep learning models trained on dynamic languages ​​into models based on predetermined static language descriptions. This allows the model to execute within a deep learning inference optimizer, thereby improving compatibility and performance. However, while this method possesses some universality and engineering value, it still suffers from numerous shortcomings and structural flaws. Firstly, this patent relies heavily on language conversion as the entry point for model optimization, converting dynamic languages ​​(such as dynamic graph models in Python and PyTorch) into static language descriptions (such as TensorFlow Graph or intermediate representations IR). However, this strategy does not directly address the core bottleneck of inference performance—the deep coupling between hardware architecture and model execution paths. The optimization focus of this invention is concentrated on the static conversion of syntax and structure, failing to consider runtime microarchitectural characteristics such as thread usage, cache hits, and bandwidth bottlenecks in specific AI chip architectures. The key to AI inference accelerator adaptation methods lies in the dynamic collaborative optimization of inference paths and hardware resource scheduling, rather than simply replacing language-level semantic expressions. Therefore, this technical approach lacks responsiveness when dealing with complex scenarios such as extreme input distribution changes, dynamic variations in model structure, and unbalanced loads in hardware multi-processor clusters, and is at risk of performance degradation.

[0003] Secondly, the patent lacks an adaptive scheduling mechanism with a closed-loop feedback loop for execution status. Its optimization logic is a one-time mapping, meaning all operations are completed during the model conversion phase, without supporting dynamic adjustment of path selection or instruction scheduling strategies during inference. Real-world AI accelerators face challenges such as fluctuating thread activity, changes in the sparsity of batch processing input tensors, and the intervention of device thermal management strategies during actual inference. Without mechanisms for runtime instruction insertion, rollback, and replacement, dynamic responses to performance fluctuations are impossible, leading to unstable inference latency, decreased throughput, and low energy efficiency. Thirdly, the patent does not propose a module-level path granularity partitioning and computation graph weight analysis method. Its optimization method assumes the model structure remains consistent during conversion, only changing semantic-level object descriptions, without evaluating, ranking, pruning, or serializing the importance of paths within the model. Fourthly, the patent lacks a path rollback mechanism, meaning that when an optimization strategy experiences performance anomalies during actual execution, it cannot quickly revert to a stable basic path. This patent focuses only on the static mapping correctness during the model conversion phase, neglecting the importance of runtime performance evaluation and adjustment. Fifth, in terms of scheduling granularity, existing technologies are coarse-grained model-level conversions, which are essentially model format adaptations for deployment toolchains rather than microstructure optimizations for actual inference execution efficiency.

[0004] In summary, while existing technologies provide an engineering approach for static language conversion of models, they still have significant technical shortcomings in areas such as resource adaptation, path scheduling, runtime feedback control, and multi-level granularity partitioning for AI inference accelerators. Summary of the Invention

[0005] The purpose of this invention is to provide an accelerator adaptation method for AI algorithm inference, thereby addressing some of the drawbacks and shortcomings pointed out in the background art.

[0006] The present invention addresses the aforementioned technical problems by employing the following technical solution: an accelerator adaptation method for AI algorithm inference, comprising: performing structural analysis on the AI ​​inference model to be deployed; identifying computational paths that influence the inference results based on the target inference output; and trimming or reorganizing the model structure to generate a set of subtasks representing the main inference behaviors; and modularly dividing the subtask set according to the basic computational unit structure of the target accelerator, so that the computational granularity of each module matches the execution units supported by the accelerator, and adjusting the operation order between modules;

[0007] For the execution path of the modular task, the resource mapping relationship on the target accelerator is analyzed, and the execution order of the computing path located in the resource bottleneck area is reorganized; based on the current execution status of the accelerator, including thread activity, cache usage and resource consumption, the adaptation instruction sequence is dynamically generated and optimized.

[0008] Furthermore, the identification of the computational path includes calculating the inverse gradients of multiple input samples on different feature channels and establishing a channel importance ranking to filter out paths with fluctuating gradient contributions; the pruning operation includes dynamically constructing a recoverable structure, wherein the pruned path can be reactivated based on the input complexity at runtime to optimize the inference path of the input distribution; during the construction of the subtask set, parallel paths are converted into equivalent serial modules.

[0009] Furthermore, during the modular partitioning process, a dependency graph is established for each module and topological weights are labeled for the nodes, so that the substructures with larger weights are retained when the modules are merged in the future. The larger substructures are those substructures among the candidate substructures whose nodes have an out-degree greater than a preset threshold in the computation graph and whose frequency of occurrence in the most recent model inferences is higher than the average. The operation sequence between modules includes spatial remapping of operations to balance the distributed load among the multi-processing clusters of the accelerator.

[0010] Furthermore, the resource bottleneck judgment is based on the multi-path scheduling estimation generated by the simulated execution graph builder, and the reordered paths are selected to avoid peak memory bandwidth conflict areas; the path execution order reorganization includes inserting virtual dependency nodes to delay the computation startup on the bottleneck path; the path adjustment is achieved by introducing an interruptible backup path as a backoff mechanism when the main path fails.

[0011] Furthermore, the dynamically generated instruction sequence supports a hybrid static and dynamic scheduling mode, where the invariant part is pre-compiled and the variable part is inserted by the runtime scheduler according to the accelerator state; the instruction generation logic supports hardware primitive acceleration replacement of path instruction blocks in the model, including replacement with low-latency shared memory operations or atomic operation instructions.

[0012] Furthermore, during the static pre-compilation stage, a set of semantic tag sequences is embedded into the model instruction stream for runtime triggering of dynamic instruction block insertion position alignment. Combined with a cascaded dynamic activation mechanism, the inserted instruction block triggers and performs conditional judgments on adjacent regions to be inserted. In addition, during the pre-compilation stage, execution paths under various hardware configurations are modeled, and a minimum conditional jump path tree is generated to reduce branch judgment overhead during scheduling. To achieve effective control of the above scheduling behavior, this invention constructs a path activation response function based on execution feedback state to measure whether a certain tag position should activate the corresponding dynamic instruction block under the current hardware state. The response function is as follows:

[0013]

[0014] in:

[0015] Indicates path location Given hardware status parameters With status feedback The activation willingness value below; Representing a path Current hardware status parameters The expected delay prediction function is obtained by offline training of the prior performance model. It represents the sensitivity of the delay function to changes in hardware state parameters in the current state, and is used to measure the degree to which the path depends on environmental changes; Representing a path For state feedback variables The trigger compliance scoring function, the larger the value, the more suitable the path is for the current state; Representing a path The jump cost weighting factor is determined by the structural depth of the path in the minimum path tree and the number of historical backtracks.

[0016] when At that time, path location Activated, triggering the insertion of a dynamic instruction block at that location; simultaneously triggering... Adjacent candidate path regions enter the activation check state to achieve chained loading; otherwise, static path execution is maintained. Through the control mechanism of the path response function, the system can realize dynamic instruction scheduling logic with structure-oriented, state-adjustable, trigger-predictable, and cost-reversible characteristics.

[0017] In dynamic instruction insertion scheduling mechanisms, the system faces two core issues: first, whether a certain instruction needs to be inserted; and second, at which path positions is the insertion optimal. To achieve unified measurement and control over these two issues, a path response function is designed. as path location The activation driving force indicator reflects whether the path is worth dynamically inserting under the current hardware state; the first step in constructing this function is to define the path location. Delayed prediction function It represents the current hardware state parameter vector. (For example, thread activity, shared memory utilization, SM occupancy) the expected inference latency introduced by executing this path; not directly using The activation decision is not based on the value of the variable itself, but rather on its response trend to state changes; therefore, its gradient term is introduced. This represents the latency sensitivity of the path under different hardware states. It is a local dynamics estimate used to determine whether the path is highly adaptable to the current state.

[0018] However, the gradient alone is insufficient to determine whether a path should be inserted; it is also necessary to determine whether the path conforms to the state feedback characteristics of the current task execution context, such as whether it is a hotspot region, whether it has a high input hit rate, or whether there is a recent execution cache. Therefore, a second scoring function is introduced. It is a path location and runtime feedback variables The state fit function (such as current load, backoff frequency, hit rate) is used as input to measure the activation potential or fit of the current path in a given state. The gradient term and the fit term are combined and obtained by dot product to ensure that the path has a high response value only when it is sensitive enough to the state (necessary to adjust) and fits well enough (adjustment is favorable).

[0019] The product result described above is essentially a positive activation trend. However, to achieve control system stability, a reverse braking term must be included to measure the substitution cost or jump risk of this path; this is why a reverse braking term is introduced. The reason for the item is that it is a path. The corresponding jump cost factor consists of multiple parts, including its depth in the pre-built minimum path tree (the deeper the jump, the longer the jump chain and the greater the overhead), the frequency of failures or backtracking in historical scheduling, etc., as a kind of positive penalty term; finally, the structural cost of the path is subtracted by the joint positive activation driver to form the complete path response function.

[0020] Furthermore, the runtime scheduler performs a latency residual assessment on the target region before inserting dynamic instructions. If the potential latency increment after insertion exceeds a threshold, it switches to a conservative execution path. A shared counter is established between the static and dynamic paths to monitor the hit ratio of the two paths during the inference process and dynamically adjust the path strategy weight for the next step based on the ratio.

[0021] Furthermore, after the instruction block is replaced with a shared memory operation, the data residency time and the frequency of access conflicts in the same cycle are used as replacement evaluation indicators. If the indicators are not met, a rollback is triggered. The primitive acceleration replacement mechanism prioritizes matching short circular dependency chains in the path block so that the path can be mapped to a single instruction loop structure. When the primitive acceleration replacement is performed, a candidate replacement set is constructed. After the replacement, the execution indicators of different alternative schemes are recorded in each round of inference and used for the next round of candidate sorting.

[0022] Furthermore, after performing the shared memory replacement, a replacement efficiency score is jointly calculated based on the average waiting time and cache line hit rate within the actual execution cycle. If the score is lower than the dynamically calculated reference threshold, the replacement is revoked and the original instruction path is restored. The rollback is performed after the replacement to perform a micro bypass simulation to predict the stability distribution of the replacement region under different batches of input data.

[0023] Furthermore, the length of the circular dependency chain is calculated by weighting the results after topological sorting and combining them with the execution time of each dependent node; when there are multiple candidate circular paths with the same length, the path containing atomic accumulation operation is selected as the primitive replacement target.

[0024] The beneficial effects of this invention are as follows: By performing structural analysis on the inference model and extracting critical paths affecting the inference results, this invention can effectively eliminate redundant computations and generate a set of semantically representative subtasks. These subtasks are divided into modules matching the computational granularity of the target accelerator, and topology optimization is performed by combining path dependency weights, achieving a balance between execution efficiency and structural representation in module partitioning. Simultaneously, bottleneck path analysis is performed using simulated execution graphs, and fallback backup execution channels are constructed, significantly enhancing the robustness of the inference path under dynamic resource loads. Furthermore, this invention employs a static-dynamic hybrid scheduling mechanism, statically compiling stable path instructions while delaying uncertain parts to runtime for on-demand activation. Through marker-guided and cascading triggering methods, low-latency response and chained coordination of instruction insertion are achieved, reducing control branch overhead.

[0025] Regarding primitive-level instruction replacement, this invention further introduces shared memory and atomic operation replacement strategies, while constructing a replacement candidate set and dynamically sorting the candidate paths based on the actual execution performance data recorded during the inference process; and by introducing a micro-simulation after replacement and a joint efficiency scoring mechanism, the quality of replacement is controlled and safe rollback is supported, thereby obtaining a better execution path selection without destroying the original logic. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the adaptation process for the AI ​​inference model accelerator of this invention.

[0027] Figure 2 This is a diagram showing the core process and key functional relationships for adapting the AI ​​inference accelerator of this invention.

[0028] Figure 3 This is a simplified flowchart of the dynamic path activation and scheduling optimization of the present invention.

[0029] Figure 4 This is the main flowchart for accelerating the adaptation of the medical image model in Embodiment 1 of the present invention.

[0030] Figure 5 This is a schematic diagram of the dynamic instruction scheduling and primitive optimization process of the medical model in Embodiment 2 of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] Combined with appendix Figure 1 This invention provides an accelerator adaptation method for AI algorithm inference. It performs structural analysis on the AI ​​inference model to be deployed. Based on the model's inference task objectives and the decision sensitivity of its output results, it quantifies and identifies the contribution of each path in the model's internal computation graph, extracting computational paths that significantly influence the inference results. While preserving their key semantics, it prunes and reorganizes the model structure through methods such as channel pruning, inter-layer pruning, or operator-level structural transformation to generate a set of subtasks representing the main inference behaviors. This subtask set not only retains the core inference capabilities of the original model structure but also possesses a smaller computation graph size and clearer semantic mapping boundaries. After obtaining the subtask set, it considers the microstructural characteristics of the target accelerator, including but not limited to supported execution unit types, parallelism, vector width, and sharing. Storage capacity, etc., are used to divide the subtask set into multiple modular task units using structural mapping rules. Each module is consistent with the instruction scheduling logic and data channel configuration of the accelerator hardware in terms of computational dimension, memory read / write strategy, and operator merging method, ensuring that the divided modules can be efficiently mapped to the underlying computing units during deployment. In addition, to further improve the pipeline efficiency of the inference stage, the operation order between the divided modules is topologically reordered. The execution graph is structurally optimized by adjusting the dependency edges between modules, identifying reorderable subgraphs, and inserting scheduling delay buffer nodes. In particular, reordering is performed in areas with operation hotspots, low parallel redundancy, or overlapping data paths to ensure that the load between processing clusters in the accelerator can be kept as balanced as possible in the spatial and temporal dimensions, maximizing the release of hardware potential and reducing execution blocking.

[0033] Resource mapping relationship analysis is performed on the execution paths of the modular tasks to obtain the resource consumption characteristics distribution of each computing path in the target accelerator hardware. This analysis is based on the mapping relationship between the constructed task execution graph and the accelerator hardware topology. By simulating the scheduling distribution of paths on different computing units (such as SM, NPUCore, TensorCluster), the analysis identifies the key resource types that the computing paths occupy during operation, including but not limited to shared memory bandwidth, on-chip cache levels, and execution unit usage conflicts. If certain computing paths are found to have concentrated resources or high conflict areas in the hardware structure, it is determined that the path is located in a resource bottleneck area. Furthermore, the resource adaptability of the path in the accelerator is optimized by adjusting the execution order in the path. This order reordering process includes delaying or advancing computing nodes in the path, reordering branches, and operations. By integrating or inserting buffer logic and other methods, bottleneck resources can be staggered in time dimension to avoid resource contention between critical computing regions. Based on path reorganization, the execution status of each task module under specific states is collected and fed back based on real-time indicators such as thread activity, cache hit rate, shared memory usage, and resource saturation of each execution unit, according to the execution status of the accelerator. This feedback is used as a dynamic scheduling control condition to generate an adapted instruction sequence for the current state. This instruction sequence contains a statically compiled stable part and a dynamically generated variable part. The dynamic part is inserted or adjusted as needed by the runtime scheduler in combination with the real-time resource status. By constructing a state-aware scheduling strategy model, priority weights, candidate replacement instructions, and fallback schemes are set for different path nodes to achieve precise scheduling at the instruction level and intelligent resource reuse.

[0034] Combined with appendix Figure 2This invention identifies the impact of each path in the model on the final inference result. Multiple representative input samples are selected, and backpropagation is performed on the static graph of the model to calculate the gradient response value of each feature channel at the target output. The stability of the gradient contribution of each channel under different samples is statistically analyzed to establish a channel-level gradient importance ranking. This ranking is used to determine which computational paths are high-fluctuation channels in the inference process—paths whose contributions change drastically under different samples, leading to decreased inference stability. These paths are then structurally filtered out to improve the stability and simplicity of the overall inference path. In path pruning, this invention does not use traditional irreversible pruning methods but introduces a dynamic and recoverable structure design mechanism, namely, introducing virtual nodes and soft connections in the model graph. The system records the upstream and downstream relationships of the pruned paths and dynamically determines whether to reactivate the relevant path based on the structural complexity of the input samples (such as feature entropy, channel density, and input dimension changes) during inference runtime. This enables adaptive structural recovery based on input distribution, optimizing the model's generalization ability to heterogeneous data inputs and inference path coverage. Furthermore, in the process of constructing a set of subtasks from the remaining main inference paths after pruning, the parallel paths in the original computation graph are further analyzed to determine their equivalence at the semantic dependency and operator execution levels. Branch paths with equivalent output structures are converted into serial execution modules through operator fusion, scheduling order encoding, or control flow rewriting, thereby reducing runtime concurrent scheduling pressure, reducing on-chip resource contention, and achieving more efficient execution path linearization and adapter resource mapping.

[0035] During the modularization of the deployment model, a dependency graph of the internal operation nodes is established for each module. This dependency graph is based on the model computation graph, preserving the execution dependencies and data flow paths between each operator node. A complete node topology is obtained through static analysis and path unrolling techniques. Furthermore, topological weights are assigned to each node on this dependency graph. These weights quantify the structural importance of each node in the global inference process. The weight calculation is based not only on the node's connectivity (e.g., in-degree, out-degree) and path level, but also comprehensively considers its access frequency, operator latency, and computational intensity during model execution. This ensures that, during subsequent module merging or pruning, core substructures with high connectivity and high execution activity in the dependency graph are prioritized for retention. Substructures with higher weights are those identified in candidate merging or pruning modules. Structural regions in the computation graph whose out-degree of nodes is greater than a set threshold and whose frequency of occurrence in recent rounds of model inference exceeds the average frequency of their respective layers are prioritized for use as high-density, high-utilization, and high-reach functional cores in the inference path, and are used to build the optimized module structure. In addition, to improve the operating efficiency of multiple modules on the accelerator hardware, a spatial remapping mechanism is introduced in the process of determining the operation order between modules. That is, based on the memory access mode, operator execution time, and resource type of the operation in each module, combined with the spatial topology of the multi-processing clusters (such as multi-core SM and multi-channel DSP arrays) in the accelerator, the operation path is dynamically mapped to different processing clusters. This achieves spatial load balancing of computing tasks among hardware clusters, avoids execution bottlenecks or resource conflicts caused by local computationally dense areas, and ensures that the entire inference process maintains high throughput and low resource congestion among parallel computing units.

[0036] To address resource bottlenecks during inference, a simulated execution graph builder is constructed to perform static and dynamic co-simulation analysis of multi-path scheduling behavior within the accelerator. Based on the modular task structure after model partitioning and the accelerator's execution architecture topology, this builder generates a scheduling graph including computational paths, data channel occupancy, operation granularity, and resource scheduling timing. This simulated graph is then used to predict and estimate bandwidth usage, peak memory consumption, and concurrency conflict regions for different paths during the runtime cycle, thereby identifying bottleneck paths with significant resource congestion. Furthermore, without disrupting the overall topology dependency logic, the execution order of these bottleneck paths is reorganized and optimized. This optimization replaces the original path with a rearranged path that is equivalent to the original but avoids bandwidth peak areas, improving the overall path bandwidth adaptability. Simultaneously, to effectively control the startup time and execution latency of bottleneck paths during scheduling... Furthermore, during the execution order adjustment phase, virtual dependency nodes are inserted before the critical nodes of the bottleneck path. These virtual nodes do not undertake actual computation tasks but are used to control the timing of the execution scheduler's computation start for that path, achieving delayed triggering. This allows resource-intensive tasks to be time-staggered with other lightly loaded paths, reducing the risk of execution congestion. In addition, to improve the robustness of the inference process under abnormal conditions, interruptible backup paths are introduced as a backoff mechanism in case of primary path scheduling failure or resource access denial. This mechanism reserves a set of backup execution paths with equivalent structures but different resource consumption characteristics during the model static graph generation phase. During runtime, the primary and backup paths are dynamically switched through execution status monitoring and path switching logic. When the primary path is interrupted due to sudden thread saturation, data transmission blockage, or increased cache error rate, the backup path is immediately activated to ensure the continuity of inference and the stability of results.

[0037] In the process of generating the adapted instruction sequence, a hybrid scheduling mode of static and dynamic scheduling is adopted to balance instruction execution efficiency and runtime adaptability. Specifically, the static part refers to the fixed execution instruction blocks generated during the model compilation stage based on general hardware configuration and conventional input distribution. This part is pre-compiled and embedded into the final deployed model instruction stream to cover the structurally stable and path-determined execution segments in model inference. The dynamic part, on the other hand, consists of variable instruction blocks generated and inserted in real time by the runtime scheduler during model execution based on the accelerator's current execution state information (including thread activity, cache utilization, execution unit saturation, etc.). The scheduler uses hardware performance counters or runtime state sampling modules to dynamically adjust instruction execution strategies, such as path selection, data loading order, or execution timing, thereby improving the inference process under resource fluctuation conditions. The robustness and throughput of the model are improved. In addition, the instruction generation logic also includes hardware primitive-level acceleration replacement of instruction blocks corresponding to critical paths in the model: when a path is identified to contain data operations that can be reconstructed in parallel or shared access, the original general memory access or serial instructions are replaced with low-latency shared memory operation instructions, thereby reducing read and write overhead and access conflict probability. If high-frequency accumulation, synchronous counting and other atomic operation logics are identified in the path, they are further replaced with atomic operation instructions (such as atomicAdd, atomicMax, etc.). This replacement process is based on structural pattern matching and runtime instruction block performance sampling results, and supports rollback judgment after replacement to ensure both functional correctness and performance improvement constraints, thereby realizing microstructure-aware instruction reconstruction for different inference paths.

[0038] Combined with appendix Figure 3To achieve optimal scheduling of instruction execution under different hardware states, a set of marker sequences with path semantics is embedded into the model instruction stream during the static pre-compilation phase. These marker sequences indicate the insertion position of subsequent dynamic instruction blocks at runtime, ensuring that the scheduler can accurately align the code injection operation during inference. Furthermore, a cascaded dynamic activation mechanism is introduced into this static structure: when an instruction block is successfully inserted and executed at runtime, the trigger embedded in that instruction block actively guides the scheduler to perform state matching checks on its neighboring path regions. If the conditions are met, the next round of insertion is triggered, making the path activation behavior chained, thereby achieving the layer-by-layer loading and activation of multiple dynamic path segments. To reduce the system overhead of branch judgment during scheduling, inference execution paths are constructed for the target model under various typical hardware configuration parameters during the pre-compilation phase. A conditional jump path tree is generated based on the principle of minimum state jumps. This path tree compresses the path depth through path merging and branch folding, reducing the number of branch judgments caused by state changes during the deployment phase. To achieve dynamic control of the activation and scheduling process described above, a path activation response function based on runtime feedback state is constructed to determine whether a preset path position should be dynamically activated in the current execution state. Its mathematical expression is as follows:

[0039]

[0040] in, Indicates path location Current hardware status parameters With status feedback The activation intention value is set below; the larger the value, the more the path should be activated. It is a path The expected latency prediction function under the current hardware state parameters is used to describe the execution cost of this path under the current resource configuration; This indicates the sensitivity of the prediction delay to changes in hardware state parameters, reflecting the adaptive pressure of the path to changes in system load or resources; It is a path For state feedback variables The matching score function, such as measuring whether the path is a hotspot or has a high hit rate, is used to determine the degree of matching between the path and the current execution context; while It is a path The jump cost penalty term, whose value is determined by a comprehensive evaluation of multiple factors such as the path's structural depth in the pre-built path tree, the number of failures in historical scheduling, and the number of backoffs, is used to suppress paths with high activation costs. During system runtime, the scheduler uses this function as the basis for activation decisions. At that time, the path is considered It should be activated, that is, trigger the insertion of a dynamic instruction block at that location and put its adjacent path into a check state; otherwise, the current static path remains unchanged.

[0041] To address the scheduling strategy for runtime dynamic instruction insertion and prevent excessive system execution latency from degrading inference performance, a latency residual evaluation mechanism is designed as the basis for pre-insertion judgment. Before the scheduler inserts a dynamic instruction block into a predefined marked position, a lightweight performance simulator or micro inference estimator is used to predict the execution latency required for the target region before and after the insertion, and to calculate the potential latency increment (i.e., the difference between the latency before and after insertion). If the latency residual exceeds a preset hardware tolerance threshold, the dynamic scheduler determines the insertion as non-optimal and instead selects an existing conservative execution path, i.e., bypassing the default path. The insertion position continues to use the static path flow for execution, thereby avoiding the problem of global inference task degradation due to local performance deterioration. In addition, in order to achieve feedback optimization of the long-term dynamic scheduling strategy, a set of shared hit counters is established between the static path and the dynamic path to record the actual hit frequency of each path during the inference process. The frequency is defined as the number of times the path is triggered and completes effective inference within a specified time window divided by the total number of inferences. After each round of inference, the scheduler updates the strategy weight of the corresponding path according to this ratio. If the hit rate of the dynamic path is lower than that of the static path for a long time, its insertion priority is reduced in subsequent inferences, and vice versa.

[0042] A hardware primitive-based acceleration replacement mechanism is introduced into the instruction generation logic to improve the execution efficiency of model path blocks on the accelerator architecture. The key implementation steps include three parts. After identifying the path instruction block, the system attempts to replace frequently accessed memory parts with shared memory operations to reduce memory access latency. Specifically, after instruction replacement, the system collects two core evaluation metrics for the path block in a shared memory usage scenario in real time: first, data residency time, i.e., the length of time a data block is retained in shared memory, to assess its effectiveness in multiple reuses; second, concurrent access conflict frequency, used to measure the degree of concurrent access conflict between different threads to the same shared memory unit during the execution cycle. If the above two metrics do not simultaneously reach the set thresholds (e.g., residency time below the lower limit or conflict frequency above the upper limit), it is considered that this replacement has no significant contribution to performance improvement, and the system will trigger a rollback mechanism to restore the original general-purpose memory instruction path, ensuring... The system prioritizes the stability and consistency of inference. Secondly, in the primitive acceleration replacement strategy, when traversing the path instruction graph, the system prioritizes identifying short cyclic dependency chains—that is, computational blocks with short data dependency paths and closed-loop structures—and attempts to reconstruct them into Single Instruction Loop structures. These are mapped to atomic operations or efficient aggregation instructions supported by the accelerator, thereby reducing scheduling overhead and branch jumps, and improving execution throughput. Finally, to enhance the adaptability of the replacement strategy, before each primitive replacement execution, the system constructs a set of candidate alternatives. This set is generated based on the structural differences of path blocks, operation density, and historical execution feedback. After each round of inference, the system collects the execution metrics (such as latency, bandwidth usage, and hit rate) of each alternative under the current input data distribution and accelerator state. Based on these metrics, the candidate set is then sorted and updated to ensure that subsequent replacement strategies evolve towards optimal performance.

[0043] To improve the stability and dynamic adaptability of the shared memory replacement strategy, after converting common memory operations to shared memory operations in the replacement path, the system constructs a joint performance evaluation mechanism to monitor and judge the replacement effect in real time. During model inference execution, the system collects the average waiting time metric within the current execution cycle. This metric measures the waiting time of threads accessing shared memory due to access conflicts or scheduling delays. Simultaneously, it combines this with the cache line hit rate metric, which reflects the utilization efficiency of shared memory resources by the replaced instruction blocks during execution. The system combines these two metrics to construct a replacement efficiency score function. For example, a unified score value is generated by normalization calculation followed by weighted summation. This score quantifies the overall execution effect of shared memory replacement in the current environment. Furthermore, to avoid misjudgment issues caused by static thresholds, this efficiency score is compared with a dynamically calculated reference threshold. The reference threshold is determined by the current batch input feature distribution and historical inference negative data. The system is jointly modeled based on load status and accelerator operating parameters (such as SM utilization and shared memory load), and has the ability to dynamically adjust over time and according to changes in task scenarios. If the current replacement efficiency score is lower than the reference threshold, the system determines that the current replacement has no significant acceleration benefit and causes resource waste or performance degradation. Therefore, it triggers a rollback operation, that is, cancels the current shared memory operation and restores the original general memory path to ensure system stability and consistency of inference results. To further improve the foresight and robustness of the replacement strategy, the system will also perform a micro-bypass simulation process on the replacement region after each rollback. This simulation simulates the behavior of the current instruction block under multiple batches of input data without affecting the main inference execution path, and records its execution latency, hit rate and resource contention under different data conditions. This is used to estimate the stability distribution characteristics of the replacement strategy in future execution cycles. Finally, the priority of the replacement strategy in the candidate set is adjusted based on the results.

[0044] To improve the overall efficiency and mapping effect when replacing path instruction blocks with hardware primitive operations, the system introduces a circular dependency chain evaluation mechanism based on weighted topology analysis for multiple computational path blocks in the model inference process. This mechanism combines operation type characteristics to determine the priority of primitive replacement: For the path region to be analyzed, the system performs topological sorting on its corresponding computation graph to determine the order of all dependent nodes. Based on this, it identifies circular dependency chains within the path block, i.e., subgraph structures with back edges. Subsequently, the system evaluates the execution time of each node in each circular dependency chain. This time value can be obtained through static analysis, micro-benchmarking, or dynamic sampling. The system uses the topological position of each node and its corresponding execution time as weighting factors to calculate the weighted length of the circular path, which measures its impact on the latency of the execution process. When multiple loop paths exist with the same or similar weighted lengths, the system needs to further select the optimal path for hardware primitive replacement. To this end, an operation type priority strategy is introduced, prioritizing paths containing atomic accumulation operations (such as atomicAdd, atomicMin, etc.) as replacement targets. This is because atomic operations often become performance bottlenecks in conventional execution paths due to their synchronization characteristics, while accelerators often have dedicated optimization channels or primitive support for such operations. Therefore, replacing them with corresponding primitive instructions can achieve more significant performance improvements. Finally, based on the above analysis results, the system selects the optimal loop dependency chain, maps it to a single-instruction loop structure, and completes hardware-level primitive acceleration replacement, thereby improving the execution efficiency and adaptability of computationally intensive paths without compromising the semantic correctness of the model.

[0045] Example 1:

[0046] Combined with appendix Figure 4 In this embodiment, the goal is to deploy a convolutional neural network model (such as a ResNet50 variant) for medical image diagnosis onto a heterogeneous computing platform (such as NVIDIA Jetson AGX Orin) to accelerate the intelligent analysis of lung CT images. In its original deployment state, the model contains multiple residual blocks, parallel path structures, and multi-branch feature fusion channels, with an overall computational load of up to 8.7 GFLOPs and an inference latency of 426ms before deployment. The system needs to perform pruning, modularization, and scheduling optimization on the model to adapt it to the accelerator architecture and improve its operating efficiency.

[0047] First, in the computational path identification stage, the system performs backpropagation on 50 representative lung CT samples to obtain the gradient contribution of each feature channel to the final classification output (such as lung nodule or normal). The third-to-last layer of the model contains 128 feature channels. The system records the normalized gradient mean and variance of each channel under different inputs. After sorting, it is found that the average gradient contribution of 32 channels is less than 60% of the average value of all channels, and the volatility (standard deviation) exceeds 80% of its own mean. Based on this, the system marks the computational path corresponding to these channels as a high volatility and low contribution path and uses it as a candidate region for pruning.

[0048] Then, the system enters the dynamic recoverable pruning stage. Instead of directly eliminating these paths, it uses a flag-preservation mechanism to encapsulate them into dynamically callable structures. When the complexity of the input image (such as density distribution and the number of texture boundaries) exceeds a certain threshold (e.g., the image complexity index Score calculated by the structure tensor index > 0.65), the reactivation of these paths will be triggered to improve the model's expressive power, thereby ensuring that key complex samples can still obtain complete computational representation.

[0049] During the subtask set construction phase, the original model has multiple parallel paths, especially in the residual structure of the fourth phase, which has a three-branch design. In order to adapt to the linear scheduling logic of the underlying accelerator, the system serializes the three-branch structure into three independent modules through equivalent transformation, and inserts data aggregation operations at the end of each module, thereby maintaining functional consistency while reducing thread splitting overhead.

[0050] Next, the system enters the modular partitioning and weight labeling stage. Based on the intermediate representation graph (IR) of the model, the system constructs a dependency graph for all operator modules, where nodes represent operators and edges represent data dependencies. The subgraph composed of nodes A, B, and C in a certain module is defined with the following characteristics: node A has an out-degree of 3, B has an out-degree of 2, and C has an out-degree of 1. In the most recent 200 model inference records, node A appears 96% of the time, which is higher than B's 83% and C's 41%. Therefore, the system determines that A is a substructure with a larger weight in this module and will prioritize retaining this structure in the subsequent module merging process to maintain the computational power of the main information path from being weakened.

[0051] Finally, during the operation space remapping stage, the system analyzed the SM cluster distribution structure of the accelerator and found that there were a total of 8 processing clusters. In the original allocation, the 3rd cluster accounted for 32% of the load, while the 6th cluster only accounted for 6%, indicating a serious load imbalance problem. The system rescheduled the operations between modules based on the module dependency graph, migrated several non-critical path modules to the low-load clusters, and introduced buffer nodes at the path boundaries to balance the execution rate. In the end, the measured task ratio difference between the highest and lowest load clusters decreased from the original 26% to 8%, the inference latency was reduced to 292ms, and the performance was improved by more than 31%.

[0052] The system enters the execution scheduling optimization phase, further improving runtime efficiency through resource bottleneck avoidance and a dynamic instruction hybrid scheduling mechanism. Based on the accelerator-side resource status monitoring unit, the system constructs a simulated execution graph builder. This builder takes a modular inference graph as input and combines attributes such as the average computation time, memory access bandwidth requirements, and thread concurrency level of each module to generate a scheduling estimation graph of multiple candidate execution paths under actual hardware resource constraints (such as the number of SMs, L2 cache bandwidth, and shared memory capacity). For example, in one estimation, it is found that simultaneous activation of paths P1 and P4 causes the peak L2 cache bandwidth utilization to exceed 92%, resulting in two rounds of instruction blocking for subsequent modules P7 and P9 in the scheduling window. The system then identifies P1-P4 as a conflict bottleneck area. To avoid this bottleneck, the system attempts to adjust the start time of P4 in the path graph, moving its order to be inserted after P1 has finished executing. However, this introduces additional computational waiting overhead. Therefore, the system introduces a virtual dependency node Vd_P4, forcing P4 to depend on the computation completion state of P1, thereby solidifying this delay structure at the compilation level and effectively avoiding conflicts and maintaining execution smoothness during model inference execution.

[0053] Furthermore, to prevent partial path scheduling prediction failures, the system introduces interruptible backup paths as a fault tolerance mechanism in all bottleneck paths. For example, a low-optimal strategy path, replaced by P1', is added to the master operator of path P1. If the runtime detects that the computational latency of P1 increases beyond a set threshold (e.g., 10ms) due to the current batch of data input, the runtime scheduler immediately switches to P1' and rolls back the corresponding state variables, ensuring uninterrupted execution and acceptable degradation. This mechanism improves system robustness in actual deployment, reducing the high-latency trigger rate from 8.7% to 1.2% in a large number of sample inferences.

[0054] Building upon this foundation, to further improve model instruction efficiency, the system implements a hybrid static and dynamic scheduling mechanism. For example, in the residual module of the second stage of the model, a large number of ReLU and BatchNorm operations have fixed structures, and the system pre-compiles them into the execution graph as static instructions. Meanwhile, certain path selection branches and input adaptive path blocks (such as dynamic convolution paths for high-texture samples) are defined as dynamic instruction blocks, whose triggering method is based on the runtime accelerator state. For instance, they are only inserted when the SM utilization is below 65% to improve efficiency per unit power consumption. The instruction sequence is generated and inserted in real time by the runtime scheduler after state monitoring, supporting personalized scheduling within each inference cycle.

[0055] Meanwhile, the instruction generation logic supports hardware primitive acceleration replacement for path instruction blocks. For example, the average pooling operation in the residual block has a high access latency problem. The system replaces it with a low-latency shared memory implementation. Actual measurements show that the execution time of this segment has decreased from 13.2ms to 7.8ms, a performance improvement of 41%. In another path, there is a multi-threaded accumulation operation. The original standard locking addition instruction was used. After the primitive replacement, an atomic accumulation instruction (such as atomicAdd()) is used, and the execution time is shortened from 4.6ms to 2.1ms, effectively reducing thread blocking.

[0056] Example 2:

[0057] Combined with appendix Figure 5 Based on Example 1, in the deployment of this medical image diagnostic model, to optimize the location and timing of inserting dynamic instruction blocks at runtime, the system embeds semantically tagged sequences into the model's instruction stream during the static pre-compilation stage. For example, tags are inserted between the input and output of the fourth residual module of the ResNet50 variant. , , These markers indicate the entry points that trigger dynamic paths. During static compilation, execution path models are also built for different hardware configurations (assuming the hardware has two states: state A and state B; state A: SM utilization 70%, shared memory usage 40%; state B: SM utilization 90%, shared memory usage 80%), and a minimum conditional jump path tree is generated. For example, dynamic paths triggered under state A and state B are merged, and only necessary branches are retained to reduce the number of state checks at runtime.

[0058] Then, in a practical reasoning exercise, the system collected runtime state parameters. Feedback Let a certain path location Hardware state parameter vector ,in =SM utilization rate =Shared memory utilization =Current thread activity level. Assuming it's measured during this inference. , , The delay prediction function obtained through reverse offline training. The form is:

[0059]

[0060] where constant The data was obtained by fitting from the training set, and the following settings were defined. , , , Substituting the above states:

[0061]

[0062] Then calculate the sensitivity (gradient) to the state parameters. Set the target here The sensitivity is greatest because changes in SM utilization have the greatest impact on latency, i.e.:

[0063]

[0064] The system normalizes these sensitivities, for example, by setting a weight vector. Weighted average sensitivity of each component , set to use ,but:

[0065]

[0066] Redefining the State Feedback Scoring Function Set the feedback vector It contains two items: cache hit rate and cache hit percentage. Has the path in the most recent batch been triggered a certain number of times? For example, measured (i.e., 80% hit rate) Number of triggers, let the scoring function be in the form of: set up Substitute:

[0067]

[0068] Redefine the jump cost weighting factor Set path The depth in the minimum jump path tree is (Root depth is 0), historical rollback count is Assigning a cost formula, for example in ,but:

[0069]

[0070] Finally, substitute into the path response function.

[0071]

[0072] because If the activation intention value is negative, the system determines that the user is not in the correct position. Insert dynamic instruction blocks while maintaining static path execution. Adjacent marker positions are as follows: It is then placed in an inspection state, where a similar evaluation is performed. If at another path location... Its state parameters Feedback Different, for example After recalculating the sensitivity, scoring function, and response function, a positive value is obtained (e.g., sensitivity 14.5ms, score 0.76, resulting in...). If the cost is high and activation is still not achieved, or if the cost is low and the path depth is 1 (positive value if the cost is low), then through these calculation mechanisms, the system avoids undesirable insertions in actual deployment. This reduces the latency penalty caused by invalid dynamic insertions by approximately 18% in the previous experiment. On top of the aforementioned reduction of 219ms in total inference latency, it saves an additional 30ms, resulting in a new inference latency of approximately 189ms, while maintaining the model accuracy (still 93.9%).

[0073] The model runtime environment incorporates dynamic instruction block insertion, path response scheduling, and primitive replacement optimization strategies. In a chest X-ray anomaly detection task, the inference model is based on a ResNet variant, nesting multiple residual modules, each containing replaceable convolutional-activation paths. Before determining whether to insert a dynamic path, the runtime scheduler first performs delayed residual evaluation and checks the marked positions. The insertion operation is compared with the predicted delay before and after. The average delay before insertion of this path is set to be... After inserting the candidate dynamic block, the simulation execution prediction is as follows: The residual is The system sets the latency increment tolerance threshold for this module to be [value missing]. ,because If the path is marked as inactive, it will fall back to the conservative static path.

[0074] To optimize path selection in the long term, the system establishes a shared hit counter between static and dynamic paths. and This records the number of hits in the past N rounds of reasoning. Assuming that in 100 rounds of data sample reasoning, the static path was successfully triggered 71 times and the dynamic path was successfully triggered 29 times, then the hit ratio is... The system is configured so that when the dynamic path hit rate falls below 0.35, the priority of the dynamic path will be reduced and its strategy weight adjusted in the next scheduling. The default value was reduced from 1.0 to 0.6, thereby raising the threshold for triggering dynamic paths during subsequent insertion checks.

[0075] Regarding instruction optimization, when a data path in a convolutional path is replaced with a shared memory operation, the system evaluates whether this replacement brings the expected performance gains. Evaluation metrics include data residency time and the frequency of concurrent access conflicts. The residency time of the currently replaced data block in shared memory is set to... The hardware platform requirements are less than Only then is the optimization considered successful, and the frequency of access conflicts within the same period is [missing information]. The threshold is 5%. Since neither metric meets the standard, the system immediately triggers a rollback mechanism to restore the original global memory access structure, thus avoiding the introduction of higher access overhead.

[0076] Primitive-accelerated replacement mechanisms prioritize short cyclic dependency chains when replacing path blocks, facilitating the construction of single-instruction loop structures. For example, in the convolution-activation-weighted summation path, three dependency chains are identified:

[0077] Path A: Input → Convolution 1 → Activation → Convolution 2 (dependency length = 3)

[0078] Path B: Input → Convolution 1 → Activation → Dropout → Convolution 2 (Length = 4)

[0079] Path C: Input → Convolution → Accumulation Atomic Operation (Length = 2)

[0080] Path C is not only short in its loop chain, but also contains atomic accumulation operations, making it ideal for hardware primitive instruction replacement, such as mapping to warp-level atomicAdd or memorycoalescing primitives, to improve efficiency and avoid register conflicts.

[0081] After the replacement is executed, the system records key metrics for each round of inference under different alternatives, such as latency ( ), power consumption ( ), number of memory accesses ( After setting path C for replacement, the following records are kept:

[0082] Round 1:

[0083] Round 2:

[0084] Round 3:

[0085] The system uses simple ranking metrics. ,set up The score for the second round is:

[0086]

[0087] Among all candidate replacements, the one with the lowest score represents the best performance, and the system will prioritize the candidate replacement path with the lowest score in the next round of inference.

[0088] The model has achieved initial acceleration through structural pruning, path rearrangement, and dynamic instruction scheduling. The current stage involves further optimization through shared memory replacement and weighted optimization of circular dependency paths. During inference, specific convolutional operation paths within the model are optimized. The system attempts to replace the default global memory access with a shared memory mode to reduce memory access latency. This replacement path involves three main nodes: input loading (0.8μs), local convolution kernel extraction (1.2μs), and activation function (0.4μs), forming a short cyclic dependency chain.

[0089] After the replacement was performed, the system collected the following data across 100 inference batches: the average waiting time was... Cache line hit rate The system design replacement efficiency score function is:

[0090]

[0091] in, For reference waiting time, Substituting the empirical weighting coefficients into the calculation, we get:

[0092]

[0093] The system sets the dynamic reference threshold for the current hardware load state to 0.55. This indicates that the shared memory replacement did not provide sufficient performance benefits, therefore the system reverted the replacement and restored the original path. Before the reversal, a mini-bypass simulation module was triggered to simulate the cache access and latency trends of this path under different input data distributions (lung shadow enhancement, edge sharpening, low contrast, etc.). Simulation results show that under high input resolution or local high-frequency region input conditions, the standard deviation of path latency fluctuation reaches 0.17 μs, significantly higher than the average of 0.09 μs for other stable paths, thus further supporting the decision that this path is unsuitable for replacement.

[0094] Subsequently, the system performs a circular dependency chain analysis operation on the three parallel candidate sub-paths in the current network structure. , , After establishing the topological graph structure and performing topological sorting, the following dependency structure is obtained:

[0095] Node A (0.6μs) → B (1.3μs) → C (0.9μs), length 3, total time weighted value =

[0096] Node D (1.0 μs) → E (1.2 μs) → F (0.8 μs), length 3, total time weighted value = 3.0

[0097] Node G (1.0 μs) → Atom accumulation H (1.1 μs) → I (0.7 μs), length 3, total time weighting = 2.8

[0098] Although the three paths are of the same length, the paths It contains atomic accumulation operations and has stronger primitive substitution mapping potential, such as being directly converted to GPU warp-scope operations like atomicAdd(). Therefore, the system prioritizes... Marked as a primitive replacement target, the scheduler maps it to a single-instruction loop structure for replacement testing in the next instruction generation phase.

[0099] Ultimately, the average inference latency of the alternative path on the test set decreased from the original 28.4ms to 26.7ms, and the computation per unit power increased from 6.2GFLOPS / W to 6.9GFLOPS / W.

[0100] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An accelerator adaptation method for AI algorithm inference, characterized in that, include: The AI ​​inference model to be deployed is structurally analyzed. Based on the target inference output, the computational paths that affect the inference results are identified, and the model structure is trimmed or reorganized to generate a set of subtasks to represent the main inference behavior. Based on the basic computing unit structure of the target accelerator, the set of subtasks is modularized so that the computing granularity of each module matches the execution unit supported by the accelerator, and the operation order between modules is adjusted. For the execution path of the modular task, the resource mapping relationship on the target accelerator is analyzed, and the execution order of the computation path located in the resource bottleneck region is reorganized. The resource bottleneck judgment is based on the multi-path scheduling estimation generated by the simulated execution graph builder, and the reordered path is selected to avoid the peak memory bandwidth conflict area. The path execution order reorganization includes inserting virtual dependency nodes to delay the computation start-up on the bottleneck path. The path adjustment is achieved by introducing an interruptible backup path as a backoff mechanism when the main path fails. Based on the current execution state of the accelerator, including thread activity, cache usage, and resource consumption, the system dynamically generates and optimizes the adaptation instruction sequence. The dynamically generated instruction sequence supports a hybrid static and dynamic scheduling mode, where the invariant parts are pre-compiled and the variable parts are inserted by the runtime scheduler according to the accelerator state. The instruction generation logic supports hardware primitive acceleration replacement of path instruction blocks in the model, including replacement with low-latency shared memory operations or atomic operation instructions.

2. The accelerator adaptation method for AI algorithm inference according to claim 1, characterized in that, The identification of the computation path includes calculating the inverse gradients of multiple input samples on different feature channels and establishing a channel importance ranking to filter out paths with fluctuating gradient contributions; the pruning operation includes dynamically constructing a recoverable structure, wherein the pruned path can be reactivated based on the input complexity at runtime to optimize the inference path of the input distribution. During the construction of the subtask set, parallel paths are converted into equivalent serial modules.

3. The accelerator adaptation method for AI algorithm inference according to claim 1, characterized in that, During the modular partitioning process, a dependency graph is established for each module and topological weights are labeled for the nodes, so that the substructures with larger weights are retained when the modules are merged in the future. The larger substructures are those substructures that are retained from the candidate substructures, where the out-degree of the nodes in the computation graph is greater than a preset threshold and the frequency of occurrence in the most recent model inferences is higher than the average. The operation sequence between modules includes spatial remapping of operations to balance the distributed load among the multi-processing clusters of the accelerator.

4. The accelerator adaptation method for AI algorithm inference according to claim 1, characterized in that, The pre-compiled static portion embeds a marker sequence, which serves as a trigger signal at runtime to guide the alignment of the insertion position of the dynamic instruction block. The dynamic scheduling insertion mechanism, in which dynamic instruction blocks are inserted by the runtime scheduler, supports the cascading activation of instruction blocks, where an inserted instruction block can trigger insertion condition checks and chain loading in adjacent regions. During the pre-compilation process, different dynamic execution scenarios are encoded as conditional jump paths. During the deployment phase, the minimum path tree is loaded according to the accelerator configuration to reduce instruction judgment overhead.

5. The accelerator adaptation method for AI algorithm inference according to claim 4, characterized in that, Before inserting a dynamic instruction, the runtime scheduler performs a latency residual assessment on the target region. If the potential latency increment after insertion exceeds a threshold, it switches to a conservative execution path. A shared counter is established between the static and dynamic paths to monitor the hit ratio of the two paths during the inference process and dynamically adjust the path strategy weights for the next iteration based on the ratio.

6. The accelerator adaptation method for AI algorithm inference according to claim 1, characterized in that, After the instruction block is replaced with a shared memory operation, the data residency time and the frequency of access conflicts in the same cycle are used as replacement evaluation indicators. If the indicators are not met, a rollback is triggered. The primitive acceleration replacement mechanism prioritizes matching short circular dependency chains in the path block so that the path can be mapped to a single instruction loop structure. When the primitive acceleration replacement is performed, a candidate replacement set is constructed. After the replacement, the execution indicators of different alternative schemes are recorded in each round of inference and used for the next round of candidate ranking.

7. The accelerator adaptation method for AI algorithm inference according to claim 6, characterized in that, After the shared memory replacement is performed, a replacement efficiency score is calculated based on the average waiting time and cache line hit rate during the actual execution cycle. If the score is lower than the dynamically calculated reference threshold, the replacement is revoked and the original instruction path is restored. The rollback is performed after the replacement to perform a micro bypass simulation to predict the stability distribution of the replacement region under different batches of input data.

8. The accelerator adaptation method for AI algorithm inference according to claim 6, characterized in that, The length of the circular dependency chain is calculated by weighting the results after topological sorting and the execution time of each dependent node; when there are multiple candidate circular paths with the same length, the path containing atomic accumulation operation is selected as the primitive replacement target.

Images