Large language model mixed precision quantization compression and execution graph compilation method and system for neural network processing unit (npu)

Abstract

The application relates to the field of artificial intelligence model compression and NPU compilation, and discloses a large language model mixed precision quantization compression and execution graph compilation method and system for a neural network processing unit (NPU). The method divides executable tiles according to the tile shape of the NPU matrix calculation unit, the DMA alignment granularity and the on-chip cache capacity, and determines precision tags, organizes adjacent tiles with similar dynamic ranges and error budgets into scale sharing domains, aggregates adjacent tiles with consistent execution attributes into precision islands, and determines the precision island boundaries according to an execution cost model containing scale reading, DMA alignment, SRAM occupation, kernel switching and execution graph fragmentation cost, and then compiles and generates an NPU execution graph, so as to reduce the additional overhead when a low-bit compression result is mapped to a continuous execution path of the NPU.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model compression and dedicated processor compilation technology, and in particular to a method, system, device and storage medium for large language model mixed precision quantization compression and execution graph compilation for neural network processing units (NPUs). Background Technology

[0002] As large language models are gradually deployed in scenarios such as smartphones, in-vehicle cockpits, edge inference servers, and localized private inference devices, neural network processing units (NPUs) need to perform a large number of matrix operations, such as Q / K / V / O projection, feedforward network projection, and vocabulary projection, within limited on-chip cache, fixed matrix computation units, and restricted DMA handling capabilities. For example, edge-side dialogue assistants need to achieve low-latency responses under limited storage bandwidth and power budgets, while data centers need to balance throughput and energy consumption per request when serving a large number of requests in batches. To reduce model storage, external storage access, and inference energy consumption, existing technologies typically employ low-bit quantization or mixed-precision quantization, allowing precision-sensitive parts to retain a higher bit width and insensitive parts to use a lower bit width.

[0003] However, in the actual execution environment of an NPU, mixed-precision configuration depends not only on model error but also on tile shape, DMA alignment, scale metadata reading, low-bit unpacking, on-chip SRAM usage, and kernel switching methods. Existing technologies suffer from the following problems: if bit width is still determined by layer, channel, or ordinary grouping, the bit width configuration may seem reasonable at first glance, but when converted to NPU execution, it can lead to frequent data rearrangement, scale lookup, and kernel switching; if the scale sharing range is excessively expanded, it may affect the quantization validity of local tiles; if too many minor precision differences are retained, the execution graph is easily fragmented by too many boundaries. In other words, existing methods lack a mechanism to directly organize mixed-precision quantization results into a continuously executable NPU layout and compile the execution graph accordingly, making it difficult to balance low-bit compression benefits with end-to-end execution efficiency. Therefore, a large language model mixed-precision quantization compression and execution graph compilation scheme for NPU hardware execution paths is urgently needed. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, device, and storage medium for mixed-precision quantization compression and execution graph compilation of large language models for neural network processing units (NPUs). This aims to alleviate the problems caused by the inconsistency between mixed-precision quantization results and the actual execution granularity of the NPU, fragmented scale metadata access, unaligned DMA access, and high kernel switching overhead, which result in the low-bit compression benefits being offset by additional execution overhead in scenarios where the NPU supports low-bit format and fixed-tile execution.

[0005] This application provides a method for mixed-precision quantization compression and execution graph compilation of large language models for neural network processing units (NPUs), including: The model matrix of the large language model is divided into multiple executable tiles according to the tile shape of the matrix calculation unit of the NPU, the granularity of direct memory access (DMA) alignment, and the on-chip cache capacity. Based on the statistical characteristics and error budget of each executable tile, a precision label is determined for each executable tile, the precision label indicating the bit width used for quantization of the corresponding executable tile; Adjacent executable tiles with similar dynamic ranges and error budgets are organized into a scale shared domain, and the scale metadata of each executable tile within the scale shared domain is described by a shared scaling factor or a shared scale page table. Adjacent executable tiles with the same bit width, accumulation type, and scale sharing method as indicated by the precision tag are aggregated into precision islands, so that the NPU can execute continuously within the precision islands; The boundary of the precision island is determined or adjusted according to the NPU execution cost model, which includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip static random access memory (SRAM) usage, kernel switching cost at the boundary of the precision island, and execution graph fragmentation cost. Based on the determined or adjusted precision islands, an NPU execution graph is generated for the large language model. The NPU execution graph enables the NPU to maintain continuous kernel execution within the precision islands and switch at the boundaries of the precision islands.

[0006] In a preferred embodiment, the determination of precision labels for each of the executable tiles includes: Based on the weight statistics, activation statistics, and output sensitivity of each executable tile, and in conjunction with the NPU execution cost model, the bit width that minimizes the score and satisfies the error budget is selected from the set of candidate bit widths supported by the NPU as the bit width indicated by the precision label of the executable tile; wherein, the first... The bit width used for the executable tile quantization. for: In the formula, The set of candidate bit widths supported by the NPU; For the first An executable tile has a bit width The hardware execution cost includes weight reads, scale reads, low-bit unpacking, DMA alignment, and on-chip cache usage; For the first An executable tile has a bit width Quantization error estimation under the following conditions; For the first Error budget for each executable tile; For quality weights.

[0007] In a preferred embodiment, organizing adjacent executable tiles with similar dynamic ranges and error budgets into a scale-shared domain includes: Adjacent executable tiles are merged into the scale shared domain, ensuring that the effective bit loss of each executable tile within the scale shared domain relative to the maximum dynamic range of the scale shared domain does not exceed a set threshold; where the scale shared domain... Inner Effective bit loss of each executable tile for: In the formula, For the scale shared domain The maximum dynamic range of each executable tile within the range; For the scale shared domain Inner The dynamic range of an executable tile; Represents the logarithm to the base 2; When the cumulative quantization error of the scale shared domain on the calibration set exceeds the error budget of the scale shared domain, the scale shared domain is split at the executable tile where the quantization error contributes the most. Furthermore, the method further includes organizing the scaling factors of the scale shared domain into a scale page table aligned to the DMA burst, wherein the scale page table records the scale shared domain number, starting tile number, number of tiles, scaling factor storage address, DMA alignment information, and checksum; wherein the scale shared domain The corresponding scale page table byte length for: In the formula, For the scale shared domain The number of executable tiles included; The number of bytes occupied by the metadata for each scaling factor; The number of bytes for the DMA burst alignment granularity; For floor operations, the NPU loads the corresponding scaling factor according to the scale page table before executing the corresponding precision island.

[0008] In a preferred embodiment, aggregating adjacent executable tiles with consistent bit width, accumulation type, and scale sharing mode indicated by the precision tags into precision islands includes: Adjacent executable tiles are aggregated into the same precision island only when their bit width, accumulation type, scale sharing method, dequantization position, and kernel template number are all the same. When there are inconsistencies in the above attributes of adjacent executable tiles, a precision island boundary is set between the adjacent executable tiles, and format conversion, accumulation conversion or requantization processing is inserted at the boundary of the precision island.

[0009] In a preferred embodiment, determining or adjusting the boundaries of the precision islands according to the NPU execution cost model includes cost arbitration of the boundaries of adjacent precision islands: For bit width First precision island The adjacent bit width is Second precision island The higher of the two is denoted as The island with the lower bit width among the two is denoted as Calculate the boundary arbitration value : In the formula, For the first precision island With the second precision island Kernel switching costs between them; This refers to the cumulative format conversion cost between the two. The cost of fragmenting the execution graph between the two; To make the precision island All executable tiles The bit width is changed from its original bit width Unified adjustment to the aforementioned higher bit width The additional execution costs incurred; For executable tile In-line width The execution cost below; For executable tile In its original position width The execution cost below; When the boundary arbitration value When the hysteresis threshold is greater than 1, the first precision island is... With the second precision island Merged into the higher bit width Execute the single precision island; otherwise, retain the first precision island. With the second precision island The boundary of the precision island between them.

[0010] In a preferred embodiment, the step of compiling and generating an NPU execution graph for the large language model includes: For different latency, power consumption, or accuracy budgets, multiple NPU execution graph versions are compiled and generated. These multiple NPU execution graph versions share the same external interface, key-value cache (KV Cache) format, and the boundaries of the accuracy islands; and... The tile block table, tile precision label table, scale shared domain page table, precision island boundary table, NPU kernel template number, DMA address and alignment information, accumulation type and output format description, execution graph version number, quality check summary and fallback version index are written into the mixed precision compressed package so that the NPU can directly perform scaling factor loading, weight unpacking, matrix calculation and output conversion based on the metadata in the mixed precision compressed package at runtime.

[0011] In a preferred embodiment, the method further includes an execution graph selection step performed during online model inference: Obtain at least one of the following technical status signals: current context length, batch size, NPU load, power consumption status, target latency, and output risk signal. The output risk signal includes at least one of the following: the logit margin of the top-1 and top-2 outputs, duplicate token risk, attention heat concentration, and recent window quality verification result. Based on the technical status signal, one NPU execution graph version is selected and loaded from the plurality of NPU execution graph versions. The selected NPU execution graph version... satisfy: In the formula, A set of candidate NPU execution graph versions; For NPU execution graph version The overall execution cost; For NPU execution graph version Task-level error estimation; For NPU execution graph version The delay index; For NPU execution graph version Required on-chip SRAM capacity; The upper limit of the allowable error; To delay the service target; The on-chip SRAM capacity available at runtime; When an NPU execution graph version switch is triggered, the switch is performed only at the boundary of the model layer, the boundary of the token window, or the boundary of the precision island; when the NPU execution graph version switch request falls inside the precision island, it is delayed until the boundary of the next precision island. When the verification digest in the mixed precision compressed package is inconsistent with the tile precision tag table, scale shared domain page table, precision island boundary table, or NPU kernel template number, or when the on-chip SRAM capacity available at runtime is less than the capacity required by the selected NPU execution graph version, the rollback NPU execution graph version is loaded according to the rollback version index.

[0012] This application provides a mixed-precision quantization compression and execution graph compilation system for large language models targeting neural network processing units (NPUs), including: The tile partitioning module is used to divide the model matrix of the large language model into multiple executable tiles according to the tile shape of the matrix calculation unit of the NPU, the direct memory access DMA alignment granularity, and the on-chip cache capacity. A precision label determination module is used to determine a precision label for each executable tile based on the statistical characteristics and error budget of each executable tile, wherein the precision label indicates the bit width used for quantization of the corresponding executable tile; The scale shared domain generation module is used to organize adjacent executable tiles with similar dynamic ranges and error budgets into scale shared domains, and to describe the scale metadata of each executable tile in the scale shared domain by a shared scaling factor or a shared scale page table. The precision island aggregation module is used to aggregate adjacent executable tiles that have the same bit width, accumulation type and scale sharing method indicated by the precision tag into precision islands, so that the NPU can execute continuously within the precision islands; A boundary determination module is used to determine or adjust the boundary of the precision island according to the NPU execution cost model. The NPU execution cost model includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip static random access memory (SRAM) usage, kernel switching cost at the boundary of the precision island, and execution graph fragmentation cost. An execution graph compilation module is used to compile and generate an NPU execution graph for the large language model based on the determined or adjusted precision islands. The NPU execution graph enables the NPU to maintain continuous kernel execution within the precision islands and switch at the boundaries of the precision islands.

[0013] This application provides an electronic device, including a neural network processing unit (NPU), a memory, and a processor. The memory stores program instructions, which, when executed by the processor, cause the electronic device to implement the above-described method.

[0014] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0015] This application divides the model matrix of a large language model into executable tiles that match the tile shape, DMA alignment granularity, and on-chip cache capacity of the NPU matrix computation unit. This ensures that the basic granularity of mixed-precision quantization corresponds to the actual execution granularity of the NPU, thereby helping to reduce the additional data rearrangement and execution misalignment caused when determining the bit width by layer, channel, or ordinary grouping and then mapping it to the NPU. Based on this, precision labels are determined according to the statistical characteristics, error budget, hardware execution cost, and quantization error estimation of each executable tile. This ensures that bit width selection is not only driven by reducing model error but also constrained by hardware execution factors such as weight reads, scale reads, low-bit unpacking, DMA alignment, and on-chip cache usage. This allows for a data layout more suitable for NPU execution within the error budget.

[0016] This application further organizes adjacent executable tiles with similar dynamic ranges and error budgets into a scale shared domain, and describes scale metadata through shared scaling factors or shared scale page tables. This allows scale access to multiple adjacent tiles to be organized in a more centralized manner, thereby helping to reduce the additional access overhead caused by scale metadata reading and table lookup during low-bit inference. By constraining the expansion of the scale shared domain through effective bit loss, and splitting the shared domain when the accumulated quantization error of the calibration set exceeds the domain-level error budget, an enforceable constraint relationship can be formed between reducing scale metadata access and limiting quantization errors caused by excessive sharing, preventing tiles with smaller dynamic ranges from losing too many effective quantization bits due to excessive sharing. Further organizing the scaling factors of the scale shared domain into scale page tables aligned to DMA bursts allows the NPU to load the corresponding scaling factors according to the page table before executing the corresponding precision island, thereby helping to convert fragmented scale access into page-based loading and reducing the resulting DMA start / stop and address jump overhead.

[0017] This application aggregates adjacent executable tiles with consistent bit width, accumulation type, scale sharing method, dequantization location, and kernel template number into precision islands. This allows the NPU to maintain continuous kernel execution within the precision islands and confines necessary format conversions, accumulation conversions, or requantization processes to the vicinity of the precision island boundaries. This helps reduce the execution overhead caused by frequent switching of kernel templates or accumulation paths between fragmented tiles. Furthermore, by using boundary arbitration values, the kernel switching cost, accumulation format conversion cost, and execution graph fragmentation cost are compared with the additional execution cost of moving low-bit-width precision islands to higher-bit-width execution. This is further processed in conjunction with a hysteresis threshold, ensuring that the retention or merging of precision island boundaries has a clear hardware cost basis and avoiding oscillations between repeated merging and splitting. This approach does not simply pursue finer precision differences but rather makes constrained compilation decisions between the NPU's continuous execution capability and the benefits of mixed precision compression.

[0018] This application compiles and generates an NPU execution graph based on the determined or adjusted precision islands, and writes the tile block table, tile precision label table, scale shared domain page table, precision island boundary table, NPU kernel template number, DMA address and alignment information, accumulation type and output format description, execution graph version number, quality check summary and fallback version index into a mixed-precision compressed package. This allows the NPU runtime to directly perform scaling factor loading, weight unpacking, matrix calculation and output conversion based on the metadata in the compressed package, without having to re-derive the mixed-precision layout during runtime. Multiple NPU execution graph versions with a shared external interface and key-value cache format are generated for different latency, power consumption or precision budgets. During inference, the execution graph version is selected based on context length, batch size, NPU load, power consumption status, target latency or output risk signals. This helps the same model use different compiled execution paths under different inference states such as short context, long context, small batch, high throughput and low power consumption. Further restricting execution graph switching to model layer boundaries, token window boundaries, or precision island boundaries, and delaying switching requests falling inside precision islands to the next precision island boundary, can reduce the impact of runtime switching on the consistency of accumulated state, scale page table, and kernel template. Loading a rollback execution graph version when the verification digest is inconsistent with the relevant metadata table or when on-chip SRAM is insufficient further improves runtime robustness, ensuring that the online use of multiple execution graph versions still conforms to the constraints of the NPU executable layout itself.

[0019] Overall, the technical effect of this application is not generated independently by a single bit width selection, a single scale sharing, or a single kernel compilation action, but is jointly formed by the hierarchical constraints between executable tiles, precision tags, scale sharing domains, precision islands, NPU execution cost model, mixed precision compressed packages, and NPU execution graph. In scenarios where the NPU supports the corresponding tile execution mechanism and low bit format, the above-mentioned synergistic relationship helps to make the benefits of low bit compression less likely to be offset by additional overhead such as scale reading, low bit unpacking, DMA unaligned access, kernel switching, and execution graph fragmentation, thereby achieving a better balance between bandwidth, latency, and energy consumption while ensuring that the generated quality meets the error budget.

[0020] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which are considered to have been described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a method for mixed-precision quantization compression and execution graph compilation of a large language model for neural network processing units according to an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a large language model mixed precision quantization compression and execution graph compilation system for neural network processing units according to an embodiment of this application. Detailed Implementation

[0022] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0023] Explanation of some concepts: A neural network processing unit (NPU) is a dedicated processor used to perform inference calculations for neural network models. It has a matrix calculation unit, on-chip cache, DMA transfer channel, and kernel execution path for executing different calculation templates. Its matrix calculation unit usually performs matrix operations according to a fixed tile size, a fixed data path, and a fixed DMA alignment method.

[0024] An executable tile refers to a matrix block obtained from the model matrix of a large language model according to the tile shape, DMA alignment granularity, and on-chip cache capacity of the NPU matrix calculation unit. This block can be loaded and calculated by the NPU as a basic execution granularity. In this application, it serves as the basic unit for hybrid precision configuration and precision labeling.

[0025] Mixed precision quantization refers to a quantization method that reserves a higher bit width for precision-sensitive layers, channels, or operators during the quantization process, while using a lower bit width for precision-insensitive regions.

[0026] A precision tag is an identifier used to indicate the bit width or quantization format used for the quantization of the corresponding executable tile. The bit width or format can be taken from the candidate set supported by the NPU, such as INT3, INT4, INT6, INT8, FP8, or FP16. The precision tag can be associated with the statistical characteristics, error budget, and NPU execution cost of the executable tile.

[0027] The scaling factor is a proportional factor used in the quantization process to map the quantized value back to the original numerical range. Together with the zero point, it determines the mapping relationship between quantization and dequantization.

[0028] A scale shared domain is a region consisting of adjacent executable tiles with similar dynamic ranges and error budgets. The scale metadata within this region is described by a shared scaling factor or a shared scale page table, which reduces the amount of scale metadata and the number of loads.

[0029] The scale page table is a table structure used to record metadata such as the scale shared domain number, starting tile number, number of tiles, scaling factor storage address, DMA alignment information, and checksum.

[0030] Precision islands refer to execution regions formed by the aggregation of adjacent executable tiles with the same bit width, accumulation type, scale sharing method, dequantization position, and kernel template number. Within this region, the NPU can maintain continuous kernel execution and switch only at the boundary of the region.

[0031] The NPU execution cost model is a model used to characterize the hardware-related costs generated when different mixed-precision layouts are executed on the NPU. The costs include at least quantization weight read volume, scale read volume, low bit unpacking cost, DMA alignment loss, on-chip SRAM usage, kernel switching cost, accumulation conversion cost, and execution graph fragmentation cost.

[0032] The NPU execution graph refers to the kernel sequence and its metadata dependencies generated by the compiler based on the precision islands, which can be loaded and executed by the NPU runtime. It enables the NPU to maintain continuous kernel execution within the precision islands and switch between them at the precision island boundaries.

[0033] A mixed-precision compressed packet refers to a data packet used to carry content such as tile block tables, tile precision tag tables, scale shared domain page tables, precision island boundary tables, NPU kernel template numbers, DMA addresses and alignment information, accumulation type and output format descriptions, execution graph version numbers, quality check summaries, and rollback version indexes.

[0034] Effective bit loss refers to the loss of effective quantization bits relative to the dynamic range of an executable tile when it uses the shared scaling factor of its scale shared domain. It can be expressed as the base-2 logarithm of the ratio of the maximum dynamic range of the scale shared domain to the dynamic range of the tile.

[0035] Dequantization refers to the process of mapping the quantized value back to an approximate range of the original value based on a scaling factor and zero point.

[0036] Accumulation type refers to the accumulation data format or accumulation method used by the NPU when performing matrix multiplication and accumulation operations.

[0037] Output risk signals refer to technical status signals used to reflect the current output state or inference quality state during the model inference process. They may include at least one of the following: the logit margin of the top-1 and top-2 outputs, duplicate token risk, attention heat concentration, and recent window quality verification results.

[0038] Key-value cache (KV Cache) refers to the storage structure used by large language models to cache the keys and values ​​of the attention mechanism during autoregressive inference.

[0039] The following is a brief summary of some of the innovative aspects of this application: In summary, the inventiveness of this application lies not in simply subdividing the mixed-precision quantization of a large language model into smaller particles, but in enabling the quantization particles to be pre-converged into executable tiles that can be continuously executed by the neural network processing unit (NPU). ( Figure 1 ), and make the tile sequence formed by the tile division module (10) and the precision label determined by the precision label determination module (20) The scale shared domains aggregated by the scale shared domain generation module (30) Precision islands formed by the precision island aggregation module (40) The NPU execution graph compiled by the boundary determination module (50) and the execution graph compilation module (60) constitutes a mutually constraining hardware compilation chain, rather than independent algorithm steps.

[0040] Specifically, precision labels It is not determined in isolation based on quantization error, but rather in the hardware execution cost. With quantization error estimation Under the common constraints, according to the criteria The selection ensures that bit width choice balances error and executability from the outset; scale shared domain It's not that the sharing scope can be expanded arbitrarily, but rather it's subject to dynamic range. Effective bit loss By limiting the error to no more than a threshold and by setting a domain-level error budget, and supplementing this with feedback that splits the shared domain at the point of maximum error contribution when the cumulative error exceeds the limit, a constrained balance is achieved between "reducing scale metadata readings" and "maintaining the effective bits of small dynamic range tiles"—this is precisely what distinguishes it from ordinary grouped quantization, which directly treats scale sharing as coarse grouping. Furthermore, precision islands... Furthermore, the consistency of bit width, accumulation type, scale sharing method, dequantization location, and kernel template number is used as aggregation conditions to ensure that the NPU maintains continuous kernel execution within the island and performs controlled switching only at the island boundary; boundary arbitration value The comparison of "the kernel switching, accumulation conversion, and execution graph fragmentation overhead saved by eliminating an island boundary" with "the additional execution cost increased after the low-bit-width tile is increased" is placed within the same cost framework, and overly fragmented precision islands are actively merged in reverse with the hysteresis threshold. Thus, "whether to retain finer precision differences" is no longer a simple model precision judgment, but is transformed into a compilation decision constrained by DMA alignment, on-chip SRAM usage, scale loading, and kernel continuity. This transformation is exactly the opposite of the existing technology's inertia of "the finer the granularity, the higher the precision," so it is difficult to say that it is obvious to those skilled in the art. Based on the mixed precision compressed package and NPU execution graph generated by this chain, low-bit compression, scale loading, weight unpacking, matrix calculation, and boundary switching can be directly consumed by the runtime according to the same metadata relationship; while the online execution graph selection module (70) selects the execution graph version At the same time, it is also based on a predetermined precision island structure, and performs controlled adaptation according to technical status signals such as context length, batch size, power consumption status, and target latency, as well as output risk signals including logit margin and attention heat concentration. The above features support and constrain each other, which together distinguishes this application from solutions that only search for bit width, only perform group quantization, or only rely on mixed precision hardware capabilities.

[0041] Furthermore, through long-term in-depth research, the inventors of this application discovered that when large language models are deployed on NPUs with low bit widths, the real limiting factor for end-to-end benefits is not whether the bit width of a single executable tile is low enough, but whether the quantized data layout can be consumed by the NPU with a stable tile sequence, aligned scale access, and continuous kernel paths. After a thorough analysis of the NPU's execution mechanism, the inventors realized that the traditional method of determining mixed precision by layer or channel separates the error sensitivity on the model side from the execution granularity on the NPU side. This results in a bit width selection that is acceptable at the algorithm level, but requires additional unpacking, data rearrangement, DMA fragmentation access, and kernel template switching at the hardware execution level. In low-bit scenarios, these overheads offset the benefits of weight compression, which is the root cause of the phenomenon of "reasonable bit width but inefficient hardware execution".

[0042] Through analysis of numerous real-world deployment cases, the inventors further discovered that scale sharing itself has a dual nature: if the sharing range is too small, the scale metadata itself becomes a new access burden; if the sharing range is too large, tiles with smaller dynamic ranges will lose effective quantization bits. Therefore, scale sharing cannot be simply equated with ordinary group quantization, but should be constrained by considering dynamic range, effective bit loss, and calibration error. More importantly, the inventors recognized that more mixed precision boundaries are not necessarily more advantageous, as each boundary may introduce kernel switching, accumulation format conversion, and execution graph fragmentation overhead. Therefore, actively merging adjacent low-bit-width regions into higher-bit-width precision islands at certain boundaries can actually help the NPU form continuous execution paths. Based on the above understanding, the inventors of this application propose to incorporate executable tiles, scale sharing domains, precision islands, and the NPU execution cost model into the same compilation chain, thereby generating a mixed precision compressed package and NPU execution graph that can be directly loaded by the runtime. The implementation process of this application is described in detail below through specific embodiments.

[0043] The technical solution of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0044] To facilitate understanding, several terms used in this document are explained below. The Neural Processing Unit (NPU) referred to in this document is a processing unit used to perform neural network operations. Its matrix computation unit typically performs matrix operations according to a fixed tile shape, a fixed data path, and a fixed Direct Memory Access (DMA) alignment. The tile (executable tile) referred to in this document is an executable block that matches the shape, DMA alignment granularity, and on-chip cache capacity of the matrix microinstruction block that the matrix computation unit can directly execute, rather than referring to arbitrary software tensor slices. The executable tile is the basic unit for hybrid precision configuration and precision labeling in this embodiment. The scaling factor (scale) referred to in this document is a scaling factor used during quantization and dequantization to convert between a low-ratio point representation and the original value. The scale shared domain referred to in this document is a group composed of several statistically similar adjacent executable tiles that share the same scaling factor or the same scale page table. Its purpose is to reduce the amount and number of loads of scale metadata. In this paper, "precision islands" refer to a group of adjacent executable tiles that share the same execution attributes, such as bit width, accumulation type, scale sharing method, dequantization location, and / or kernel template number, and can be executed by the NPU in a continuous kernel manner. Therefore, scale sharing domains are grouped according to statistical distribution to reduce metadata access, while precision islands are grouped according to execution attributes to ensure continuous kernel execution; their criteria for determination and their functions are different. The NPU execution graph, as referred to in this paper, refers to the sequence of kernels that the NPU runtime can load and execute, along with their metadata dependencies.

[0045] Example 1: A Mixed-Precision Quantization Compression and Execution Graph Compilation Method for Large Language Models Targeting NPU This embodiment provides a method for mixed-precision quantization compression and execution graph compilation of large language models for NPU. This method is preferably executed during the offline compilation stage, converting the model matrix in the large language model into a mixed-precision compressed package that can be directly loaded by the NPU, and further compiling it to generate an NPU execution graph. This allows the runtime to directly perform scaling factor loading, weight unpacking, matrix calculation, and output transformation based on the metadata within the mixed-precision compressed package after loading it, without needing to re-search or re-derive the mixed-precision layout at runtime. Figure 1 As shown, the method may include steps 100 to 800.

[0046] Step 100: Obtain the NPU hardware constraints and divide the model matrix of the large language model into multiple executable tiles according to the NPU hardware constraints.

[0047] Specifically, the NPU hardware constraints include the tile shape of the NPU matrix computation unit, the direct memory access (DMA) alignment granularity, and the on-chip cache capacity. The model matrix can include at least one of the following: Q-projection matrix, K-projection matrix, V-projection matrix, O-projection matrix from a large language model; gate projection matrix, up-projection matrix, down-projection matrix from a feedforward network (FFN); and vocabulary projection matrix. The compiler divides the model matrix into multiple executable tiles along the M×K or K×N direction according to the row-column partitioning method supported by the matrix computation unit, so that each executable tile corresponds to one or more matrix microinstruction execution units of the NPU. For regions at the matrix boundary that are less than a complete tile, zero padding, separate boundary tiles, or boundary masking methods supported by the NPU can be used, as long as the processing method remains consistent with the actual execution granularity of the NPU.

[0048] Furthermore, in step 100, the compiler not only records the position of each executable tile in the original model matrix, but also records the matrix index, layer index, tile row and column number, DMA start address, alignment status, and on-chip cache usage estimate corresponding to that executable tile. Through this processing, the executable tile is no longer just a grouping unit within the quantization algorithm, but a basic unit that simultaneously has quantization configuration significance and NPU hardware execution significance. It should be noted that the reason for using NPU executable tiles, rather than layers, channels, or operators, as the basic unit for subsequent mixed precision configuration is that the NPU's matrix calculation unit, data path, and DMA are usually executed using fixed tiles; if the granularity of the mixed precision is inconsistent with the tile, it is easy to encounter situations where the bit width selection itself is reasonable, but additional unpacking, data rearrangement, and kernel switching are still required in hardware, resulting in inefficient execution. Therefore, this step provides a unified index basis for subsequent precision label determination, scale shared domain generation, and precision island aggregation.

[0049] Step 200: Based on the statistical characteristics and error budget of each executable tile, determine the precision label for each executable tile. The precision label is used to indicate the bit width used for quantization of the corresponding executable tile, and can be further associated with the quantization format, inverse quantization position and accumulation type.

[0050] Specifically, this step may include the following sub-steps. Step 210: Collect weight statistics, activation statistics, and output sensitivity for each executable tile. The weight statistics may include at least one of the following: maximum weight, mean square weight, dynamic range, outlier proportion, and sign distribution; the activation statistics may include activation peak, mean square activation, activation dynamic range, or activation distribution in historical calibration samples; the output sensitivity may be determined based on the layer type of the executable tile, the attention head type, FFN channel contribution, vocabulary projection region, output logit margin, or error propagation in calibration samples.

[0051] Step 220: Under the constraint of error budget and considering the NPU execution cost, select the bit width with the smallest score that satisfies the error budget from the candidate bit width set supported by the NPU for each executable tile, and use it as the bit width indicated by the precision label of that executable tile. More specifically, for the first... The selection process for an executable tile can be represented as follows: In formula (Formula 1), Indicates the first The bit width of each executable tile is selected; This represents the set of candidate bit widths supported by the NPU; Indicates the first An executable tile has a bit width The hardware execution cost may include at least one of the following: weighted read cost, scale read cost, low bit unpacking cost, DMA alignment loss, and on-chip cache footprint. Indicates the first An executable tile has a bit width Quantization error estimation under the following conditions; Indicates the first Error budget for each executable tile; The quality weight is used to adjust the relative impact between hardware execution cost and quantization error estimation. Therefore, equation (Formula 1) does not simply aim for the minimum bit width or the minimum error, but rather incorporates both hardware execution cost and quantization error into the same objective function, thus ensuring that the selected bit width satisfies the error budget while also considering hardware execution efficiency. It should be noted that in equation (Formula 1)... It can be obtained through at least one of the following methods: analytical calculation, offline profile table, runtime counter, offline calibration table, or regression model; when using a regression model, its input features, output cost term, and calibration process should be agreed upon together.

[0052] Step 230, the candidate bit width set It can include at least one of integer low-bit format, floating-point format, block floating-point format or NPU custom low-bit format; optionally, for each candidate format, its weight packing method, scaling factor and zero point precision, symmetric or asymmetric quantization method, dequantization position and accumulation type can be pre-defined so that subsequent compilation and runtime can be processed according to a unified convention.

[0053] Step 300: Organize adjacent executable tiles with similar dynamic ranges and error budgets into a scale shared domain, and describe the scale metadata of each executable tile in the scale shared domain by a shared scaling factor or a shared scale page table.

[0054] Specifically, in low-bit quantization scenarios, if each executable tile stores its scaling factor separately, reading the scaling factor metadata may become a new bandwidth overhead. Furthermore, if the scale sharing range is excessively expanded, tiles with smaller dynamic ranges may lose effective quantization bits due to sharing an excessively large scaling factor. Therefore, this step can employ a two-stage approach to construct the scale sharing domain, limiting both scale metadata fragmentation and the compression of effective bits by the shared scaling factor.

[0055] Step 310 (Coarse Clustering Stage): The compiler attempts to merge adjacent executable tiles into the same scale shared domain one by one, and after merging, determines whether the effective bit loss of each executable tile relative to the maximum dynamic range of the shared domain does not exceed a set threshold. Here, let the scale shared domain be... The maximum dynamic range of each executable tile within is , No. The dynamic range of an executable tile is The effective bit loss of the executable tile under the shared scaling factor can be expressed as: In formula (Formula 2), Indicates the first Effective bit loss of an executable tile under a shared scaling factor; Indicates scale shared domain The maximum dynamic range of each executable tile within the range; This indicates the scale shared domain. Inner The dynamic range of an executable tile; This represents the logarithm to the base 2. It should be noted that in symmetric quantization scenarios, The maximum absolute value of the weight within the tile can be taken, and the shared scaling factor can be set accordingly. ( (This refers to the bit width used by the shared domain). In this case, the effective quantization bits of tiles with smaller dynamic ranges within the shared domain will be compressed; therefore, using... Using a set threshold as the merging condition limits the effective bit collapse of a single tile caused by sharing the scaling factor to a controllable range. When the effective bit loss of any executable tile exceeds the set threshold, the compiler closes the current scale shared domain and uses the current executable tile as the starting tile for the next scale shared domain. Optionally, the compiler can also limit the number of executable tiles contained in a single scale shared domain to no more than an upper limit.

[0056] Step 320 (Error Feedback Splitting Stage): Since generating the scale shared domain solely based on the dynamic range may not fully reflect the actual error on the calibration samples, the compiler also calculates or estimates the quantization error contribution of each executable tile on the calibration set and performs error feedback splitting on the scale shared domain. This constraint can be expressed as: In formula (3), Indicates scale shared domain The first One executable tile; The contribution of the executable tile to the quantization error on the calibration set can be represented by the norm of the output residual before and after quantization, the output residual of the local matrix, or the local contribution of the task-level verification index. This represents the error budget for the shared domain of that scale. When the cumulative quantization error of a shared domain on the calibration set exceeds the error budget of that shared domain (i.e., it does not satisfy equation (3)), the compiler splits the shared domain at the executable tile with the largest contribution to quantization error and recalculates the effective bit loss and cumulative quantization error for the split subdomains until each subdomain satisfies the error budget or is split into a single executable tile. It should be noted that since each split is performed at the point of maximum error contribution and the number of executable tiles is limited, the splitting process can terminate naturally within a finite number of steps. When a shared domain of a scale has been split into a single executable tile, the executable tile no longer bears the effective bit loss caused by the shared scaling factor. If its quantization error still exceeds the corresponding error budget, the compiler can increase the bit width of the executable tile, make it use the scaling factor independently, or hand it over to the rollback NPU execution graph version for processing, thereby entering an execution path that satisfies the error constraints. Therefore, step 310 first ensures that the shared scaling factor will not cause excessive loss of effective bits by using the dynamic range, and then step 320 corrects it by using the actual error on the calibration set. The two work together to ensure that while reducing the amount of metadata, the actual error of the grouping of the scale shared domain is constrained by the domain-level error budget.

[0057] Optionally, as another implementation of the scale shared domain grouping condition, in addition to limiting the effective bit loss, it is also possible to add a requirement that the difference in the dynamic range logarithms of any two executable tiles within the shared domain does not exceed a set threshold, i.e., satisfying: In formula (4), , Representing scale shared domains respectively Inner The, the The dynamic range of an executable tile; This represents the allowed dynamic range difference threshold. When this condition is not met, the scale shared domain is also split. Furthermore, the scale shared domain can also be generated by grouping consecutive tiles, the same attention head, or the same FFN, etc., but this embodiment does not limit this.

[0058] To facilitate understanding of the coarse clustering decision logic for the scale shared domain in step 300, a constructive numerical example is provided below. It should be noted that this example is only used to demonstrate the deterministic grouping results of the algorithm under given assumed inputs; the values ​​are assumed illustrative values ​​and not actual measured performance or error data. For example, suppose that there are 8 adjacent executable tiles in the up projection of a certain FFN, with dynamic ranges of 3.1, 3.3, 3.0, 3.2, 6.4, 6.6, 3.1, and 3.2 respectively, and the effective bit loss threshold is set to 0.5 bits. According to the decision in step 310: the maximum dynamic range of the first 4 tiles is 3.3, and its maximum effective bit loss within the domain is approximately... Since the effective bit loss is less than 0.5, it is assigned to the first shared domain. If the fifth tile is incorporated, the effective bit loss of the first tile will become approximately [missing value]. Since the dynamic range exceeds 0.5, the first shared domain is closed; the dynamic ranges of the 5th and 6th tiles are close, so they are assigned to the second shared domain; if the 7th tile is incorporated into the second shared domain, the effective bit loss of the 7th tile will become approximately Since the scaling factor exceeds 0.5, the 7th and 8th tiles are assigned to the third shared domain. As a result, the 8 originally independent scaling factors converge into 3 shared domains, and the 5th and 6th tiles, which have a significantly larger dynamic range, are isolated separately. This reduces the amount of scaling factor metadata and avoids effective bit collapse caused by large dynamic range tiles and small dynamic range tiles sharing the same scaling factor.

[0059] Step 400: Organize the scaling factors of the scale shared domain into scale page tables aligned to DMA bursts. Specifically, for each scale shared domain, the compiler generates a corresponding scale page table entry. This entry can record the scale shared domain number, starting tile number, number of tiles, scaling factor storage address, scale data length, quantization format descriptor, DMA alignment information, and checksum. An index relationship is established between the scale page table and the tile block table, tile precision tag table, and precision island boundary table, enabling the runtime to quickly locate and load the corresponding scale page based on the executable tiles involved in the precision island before executing it. In some implementations, the scale shared domain... The byte length of the corresponding scale page table can be expressed as: In formula (5), Indicates scale shared domain The aligned byte length of the corresponding scale page table; Indicates scale shared domain The number of executable tiles included; This indicates the number of bytes occupied by the metadata for each scaling factor; The number of bytes representing the DMA burst alignment granularity; This indicates a rounding up operation. For example, when each scaling factor occupies 2 bytes, a certain scale shared domain contains 12 executable tiles, and the DMA alignment granularity is 128 bytes, the page length of this scale can be obtained from equation (Formula 5) as follows: Each byte, representing the scaling factor of the entire shared domain, falls within a single aligned page. Therefore, by organizing the scaling factor into whole pages aligned to DMA bursts and loading them page by page according to the aforementioned indexing relationship, the NPU can transform potentially scattered and fragmented scaling factor metadata access into whole-block prefetching by aligned pages before executing the corresponding precision island. This reduces the overhead of DMA start / stop, address jumps, and scaling factor table lookups, making it less likely that the bandwidth gains from low-bit-weighted compression will be offset by scaling factor metadata access.

[0060] Step 500: Aggregate adjacent executable tiles with consistent bit width, accumulation type, and scale sharing mode as indicated by the precision label into precision islands, enabling the NPU to execute continuously within a precision island. More specifically, the compiler first generates initial precision islands according to the order of the executable tiles: adjacent executable tiles are only aggregated into the same precision island if their bit width, accumulation type, scale sharing mode, dequantization position, and kernel template number are all consistent; if at least one of the above attributes of adjacent executable tiles is inconsistent, a precision island boundary is set between them, and necessary format conversions, accumulation conversions, dequantization processing, or requantization processing are inserted at the precision island boundary to ensure the consistency of data format and accumulation state when executing across precision islands. Therefore, since the execution attributes of each tile within a precision island are completely consistent, the NPU can use the same kernel template to execute continuously within the precision island without repeatedly switching kernels or microinstruction templates between tiles. It should be noted that scale shared domains and precision islands belong to different levels of grouping: scale shared domains are mainly used to organize scaling factor metadata to reduce scale reads and DMA access fragmentation, while precision islands are mainly used to organize NPU kernel continuous execution regions to reduce kernel switching and execution graph fragmentation; a precision island can be associated with one or more scale shared domains, and a scale shared domain can also be referenced by one or more precision islands according to execution attributes. Both are uniformly scheduled through the index relationship in the mixed precision compressed package.

[0061] Step 600: Determine or adjust the boundaries of precision islands based on the NPU execution cost model. Specifically, the NPU execution cost model includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip static random access memory (SRAM) usage, kernel switching cost at the precision island boundaries, and execution graph fragmentation cost. Even after the initial precision islands are formed according to execution attribute consistency, there may still be too many boundaries. If each boundary leads to kernel template switching, accumulation format conversion, or an increase in execution graph scheduling nodes, the bandwidth gain from low-bit-width compression may be offset by boundary overhead. Therefore, this step further arbitrates the boundaries of adjacent precision islands based on cost. In some implementations, for bit-width... First precision island The adjacent bit width is Second precision island The higher of the two is denoted as The island with the lower bit width among the two is denoted as Calculate the boundary arbitration value: In formula (6), Indicates the first precision island With the second precision island The boundary arbitration value between them; Indicates the first precision island With the second precision island Kernel switching costs between them; This represents the cumulative format conversion cost between the two. This represents the cost of execution graph fragmentation between the two. Indicates the first precision island With the second precision island Precision islands with lower median width; Precision island The executable tile within; Indicates the higher bit width of the two; Indicates an executable tile In-situ width; and These represent executable tiles. In higher bit width Its original width The execution cost below; Indicates the precision island The bit width of all executable tiles within the same container is uniformly increased to a higher bit width. The additional execution costs incurred. When the boundary arbitration value When the value exceeds the hysteresis threshold, it indicates that the cost savings from kernel switching, accumulation conversion, and execution graph fragmentation by eliminating the precision island boundary outweigh the increased cost of upscaling the low-bit-width precision island to a higher-bit-width execution. The compiler will then remove the first precision island. With the second precision island Merged into a higher bit width The compiler executes a single precision island; otherwise, it preserves the precision island boundaries between them. It should be noted that the hysteresis threshold can be a value greater than or equal to zero and is determined by the offline profile table or compilation configuration; when it is greater than zero, it can prevent oscillations caused by repeated merging and splitting of boundaries under similar cost conditions. Furthermore, since cost arbitration only merges in the direction of higher bit width, merging will not negatively worsen the triggering parameters of the precision islands involved.

[0062] Step 700: Based on the determined or adjusted precision islands, compile and generate an NPU execution graph for the large language model. Specifically, the compiler generates a sequence of kernels that the NPU runtime can load and execute, along with corresponding metadata dependencies, based on the boundaries, bit width, accumulation type, scale sharing method, dequantization location, and kernel template number of each precision island. This NPU execution graph enables the NPU to maintain continuous kernel execution within the precision islands and perform necessary switching at the island boundaries. Compared to frequently switching bit widths within each tile or in overly fine-grained regions, this execution method helps reduce kernel start-up and shutdown overhead and execution graph fragmentation. Furthermore, in some implementations, the compiler can compile and generate multiple NPU execution graph versions for different latency, energy consumption, or precision budgets: for example, a high-throughput execution graph version uses more low-bit-width executable tiles and larger scale sharing domains within the error budget's allowable range; a balanced execution graph version retains higher precision for sensitive attention heads, high-contribution FFN channels, or vocabulary projection boundary regions; and a conservative execution graph version tends to use higher precision for output-sensitive layers, long contextual attention layers, or low logit margin states. The multiple NPU execution graph versions can share the same external interface, key-value cache (KV Cache) format, and precision island boundary table, thereby reducing the impact of runtime version switching on model state consistency.

[0063] Step 800: Write the compilation results into a mixed-precision compressed package. The mixed-precision compressed package may include a tile block table, a tile precision tag table, a scale shared domain page table, a precision island boundary table, an NPU kernel template number, DMA address and alignment information, accumulation type and output format description, execution graph version number, quality check summary, and fallback version index. After loading the mixed-precision compressed package, the runtime can determine the weights and activated block positions based on the tile block table, determine the unpacking format and calculation bit width based on the tile precision tag table, load the scaling factor based on the scale shared domain page table, determine the kernel continuous execution interval and switching point based on the precision island boundary table, and load the fallback NPU execution graph version based on the fallback version index when check failure or insufficient on-chip cache occurs. In this way, the runtime executes directly based on the metadata in the compressed package without needing to re-derive the mixed-precision layout at runtime.

[0064] Example 2: Online Execution Graph Selection and Boundary-Controlled Switching Based on Embodiment 1, this embodiment further provides an online execution graph selection method. It should be noted that the tile partitioning, precision label determination, scale shared domain generation, precision island aggregation, boundary cost arbitration, and execution graph compilation in Embodiment 1 can be implemented independently, and can independently produce the technical effects of reducing scale metadata reading, reducing DMA fragment access, and improving kernel continuous execution capabilities; the online execution graph selection in this embodiment is a further optional implementation method.

[0065] Specifically, during online model inference, the runtime can acquire at least one technical status signal from the following: current context length, batch size, NPU load, power consumption status, target latency, and output risk signal. It is important to emphasize that the output risk signal is not a business risk score for manual judgment, but rather a technical status signal used to drive the selection of the compiled NPU execution graph version (i.e., adjusting the allocation and configuration of hardware resources). The output risk signal can include at least one of the following: the logit margin of the top-1 and top-2 outputs, duplicate token risk, attention heat concentration, and recent window quality verification result. For example, a low logit margin may indicate high uncertainty in the current output distribution; high attention heat concentration may indicate that certain context regions have a strong influence on subsequent outputs; and the recent window quality verification result may indicate the quantization error status of the current execution graph version within the most recent few token windows.

[0066] In some implementations, the runtime selects and loads an NPU execution graph version that meets the constraints from a set of candidate NPU execution graph versions. The selected NPU execution graph version can be represented as: In formula (7), Indicates the selected NPU execution graph version; This represents the set of candidate NPU execution graph versions; Indicates the NPU execution graph version The overall execution cost; Indicates the NPU execution graph version Task-level error estimation; Indicates the NPU execution graph version The delay index; Indicates the NPU execution graph version Required on-chip SRAM capacity; Indicates the upper limit of the allowed error; Indicates the service delay target; This indicates the available on-chip SRAM capacity at runtime. Therefore, the runtime does not re-execute the quantization search online, but rather selects a version from multiple pre-compiled NPU execution graph versions that meets the constraints of error, latency, and on-chip cache. This allows the offline-built NPU execution cost model to be directly reused for online version selection.

[0067] Furthermore, to avoid frequent switching of execution graph versions near the threshold, the runtime can adopt a window-level hysteresis mechanism: It determines whether the current window is a high-risk window based on token windows. When several consecutive windows meet the high-risk condition and the current execution graph version is not conservative, it switches to a conservative version. When several consecutive windows do not meet the high-risk condition, and the downgraded execution graph version still satisfies the constraints of equation (7), it switches to a high-throughput version. Otherwise, it maintains the current version. When an NPU execution graph version switch is triggered, the runtime only performs the switch at the model layer boundary, token window boundary, or precision island boundary. If the switch request falls inside a precision island, the runtime delays the switch until the next precision island boundary, allowing the current precision island to continue executing according to the original kernel template, original scale page table, and original accumulation type. This avoids inconsistencies between the accumulation state, dequantization position, or kernel template and the current tile sequence caused by switching inside a precision island.

[0068] Furthermore, before loading the target NPU execution graph version, the runtime can perform consistency verification on the quality checksum and metadata checksum in the mixed-precision package: when the checksum is inconsistent with the tile precision tag table, scale shared domain page table, precision island boundary table, or NPU kernel template number, the runtime refuses to load the execution graph version and loads the fallback NPU execution graph version according to the fallback version index; when the available on-chip SRAM capacity at runtime is less than the capacity required by the selected NPU execution graph version, the runtime can also abandon the version and select the execution graph version with lower SRAM requirements or load the fallback version; when the NPU profile table is missing a bit width or a cost item for a kernel template, the compiler can retain the original precision island boundary without performing merge arbitration and mark the execution graph version as needing to be re-profiled.

[0069] Example 3: Mixed-precision quantization compression and execution graph compilation system for large language models targeting NPU This embodiment provides a mixed-precision quantization compression and execution graph compilation system for large language models targeting NPUs, corresponding to the methods described above. This system is used to execute the steps in the above method embodiments. Figure 2As shown, the system includes a tile partitioning module (10), a precision label determination module (20), a scale shared domain generation module (30), a precision island aggregation module (40), a boundary determination module (50), and an execution graph compilation module (60). Specifically, the tile partitioning module (10) is used to partition the model matrix of a large language model into multiple executable tiles based on the tile shape, DMA alignment granularity, and on-chip cache capacity of the NPU's matrix calculation unit; the precision label determination module (20) is used to determine a precision label for each executable tile based on its statistical characteristics and error budget, the precision label indicating the bit width used for quantization of the corresponding executable tile; the scale shared domain generation module (30) is used to organize adjacent executable tiles with similar dynamic ranges and error budgets into a scale shared domain, and to describe the scale metadata of each executable tile within the scale shared domain using a shared scaling factor or a shared scale page table; the precision island aggregation module (40) is used to... Adjacent executable tiles with consistent bit width, accumulation type, and scale sharing method indicated by the precision tag are aggregated into precision islands so that the NPU can execute continuously within the precision islands; the boundary determination module (50) is used to determine or adjust the boundary of the precision islands according to the NPU execution cost model, which includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip SRAM occupation, kernel switching cost at the precision island boundary, and execution graph fragmentation cost; the execution graph compilation module (60) is used to compile and generate an NPU execution graph for the large language model based on the determined or adjusted precision islands, which enables the NPU to maintain continuous kernel execution within the precision islands and switch at the precision island boundaries.

[0070] In some embodiments, the system may further include a mixed-precision compressed package generation module (80) and an online execution graph selection module (70). The mixed-precision compressed package generation module (80) is used to write the tile partitioning table, tile precision label table, scale shared domain page table, precision island boundary table, NPU kernel template number, DMA address and alignment information, accumulation type and output format description, execution graph version number, quality check summary and fallback version index into the mixed-precision compressed package; the online execution graph selection module (70) is used to select and load an NPU execution graph version from multiple NPU execution graph versions during online model inference based on at least one technical status signal among context length, batch size, NPU load, power consumption status, target latency and output risk signal. It should be noted that the above modules can be implemented by software programs or by a combination of software and hardware. For example, the tile partitioning module, precision label determination module, scale shared domain generation module, precision island aggregation module, boundary determination module and execution graph compilation module can be implemented as part of an offline compiler, and the online execution graph selection module can be implemented as part of the NPU runtime. The specific implementation methods, mutual cooperation methods, and resulting technical effects of the above modules are the same as or correspond to the above method embodiments. Moreover, the division of the above modules is only a logical functional division. In actual implementation, there may be other division methods, which will not be elaborated here.

[0071] Example 4: Electronic devices and computer-readable storage media This embodiment provides an electronic device, which includes a neural network processing unit (NPU), a memory, and a processor. The memory stores program instructions, which, when executed by the processor, cause the electronic device to implement the method described in Embodiment 1 or Embodiment 2. Optionally, the NPU can load scaling factors, unpack and quantize weights, call the corresponding NPU kernel template, and perform matrix calculations based on metadata in a mixed-precision compressed package; the processor can be used to perform at least some steps in offline compilation, runtime scheduling, metadata verification, or execution graph version selection; the memory can be used to store large language model parameters, mixed-precision compressed packages, NPU execution graph versions, profile tables, calibration data summaries, and rollback version indexes; the electronic device may also include a communication interface and bus connecting the processor, the memory, and the NPU.

[0072] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in Embodiment 1 or Embodiment 2 above. The computer-readable storage medium may be a non-transitory storage medium, such as a read-only memory, random access memory, flash memory, solid-state drive, magnetic disk, or other medium capable of storing program instructions.

[0073] Example 5: Parameter Determination, Boundary Case Handling, and Rollback Mechanism This embodiment further explains the definition of terms, parameter determination methods, boundary case handling, and rollback mechanism involved in the above embodiments. The following content is used to help understand this application and does not limit the scope of protection of this application.

[0074] In some implementations, the aforementioned "similar dynamic ranges and similar error budgets" can be defined by a preset threshold or a threshold determined by a calibration set. Similar dynamic ranges can include: after adjacent executable tiles are incorporated into the same scale shared domain, the effective bit loss of each executable tile relative to the maximum dynamic range of that scale shared domain does not exceed an effective bit loss threshold; or the logarithmic difference in the dynamic ranges of any two executable tiles within that scale shared domain does not exceed a dynamic range difference threshold. Similar error budgets can include: the difference in error budgets corresponding to adjacent executable tiles does not exceed a preset budget difference threshold, or the cumulative quantization error after being incorporated into the same scale shared domain does not exceed the error budget of that scale shared domain.

[0075] In some implementations, the tile shape supported by the NPU matrix calculation unit may include 16×64, 32×64, or other matrix block shapes supported by NPU microinstructions; the DMA burst alignment granularity may be 128 bytes or determined by the specific NPU platform configuration; the on-chip SRAM capacity is used to constrain the weights, scale pages, and intermediate accumulation results that need to be loaded before the execution of a single precision island.

[0076] In some implementations, the candidate bit-width set It can include several formats actually supported by the NPU, such as INT3, INT4, INT6, INT8, FP8, and FP16. For an NPU platform that does not support a certain candidate bit width, the compiler removes the bit width from the candidate bit width set and only performs the bit width selection described in formula (1) within the formats supported by the platform.

[0077] In some implementations, when there is no expression satisfying (Formula 1) When the candidate bit width is specified, the compiler can choose the highest precision format supported by the NPU platform, or mark the executable tile as a fallback tile and process it by the conservative execution graph version.

[0078] In some implementations, to avoid instability in the calculation of effective bits due to zero or near-zero dynamic range, the compiler can... The original dynamic range of an executable tile is denoted as . and in accordance with Determine the dynamic range used for calculation, where The lower limit is a preset positive limit. When an executable tile is an all-zero tile or a near-zero tile, the compiler can treat it as a separate scale shared domain, or make it use the scaling factor independently and the conservative scale page table referenced by the adjacent compatible precision island, thereby avoiding division by zero or numerical divergence in Equation (2).

[0079] In some implementations, the quantization error estimation and The residual L2 norm of the matrix output corresponding to the tile in the calibration sample before and after quantization can be used, or the propagation contribution of the residual to the final logit output can be used. When only offline fast estimation is required, an approximate error estimate can be obtained by looking up a table based on the weight dynamic range, activation peak, and candidate bit width. The output sensitivity can be synthesized using a weighted summation method, for example, by multiplying the normalized outlier ratio, activation peak, and the inverse of the output logit margin by their respective weights and then summing them. The weights and thresholds are determined by grid search on the calibration set.

[0080] In some implementations, the hardware execution cost It can be obtained by weighted summation of weighted read cost, scale read cost, low bit unpacking cost, DMA alignment loss, and on-chip SRAM usage cost, for example, expressed as: ,in , , , , They represent bit widths respectively. The costs associated with weighted reads, scale metadata reads, low-bit unpacking, DMA alignment loss, and on-chip SRAM usage are considered. to The weighting coefficients are determined by the NPU profile table, offline calibration, or compilation configuration; the weight read cost and scale read cost can be calculated from the number of aligned read bytes, the DMA alignment loss can be determined by the difference between the actual data length and the DMA burst aligned length, the low bit unpacking cost, kernel switching cost, and accumulation format conversion cost can be provided by the NPU profile table, and the execution graph fragmentation cost can be determined by looking up the table by the number of precision island boundaries or the number of newly added scheduling nodes.

[0081] In some implementations, the effective bit loss threshold, scale shared domain error budget, dynamic range difference threshold, and hysteresis threshold can be jointly determined by the calibration set and the NPU profile table. For example, the compiler iterates through several candidate threshold combinations on the calibration set, statistically analyzing scale metadata reads, DMA unaligned accesses, kernel switching counts, on-chip SRAM peak usage, average token latency, and task-level error metrics, and selects the threshold combination that satisfies the task-level error constraints and has a lower overall execution cost. For instance, the effective bit loss threshold can be approximately 0.5 bits, and the hysteresis threshold can be a value greater than or equal to zero, with the specific value determined by the calibration process. These thresholds can be written as compilation configurations into the metadata area of ​​the mixed-precision compressed package.

[0082] In some implementations, each tile number in the tile block table of the mixed precision compressed package can be indexed by the bit width and quantization format in the tile precision tag table, the scale page number in the scale shared domain page table, and the precision island number in the precision island boundary table; the precision island boundary table can be further indexed by the NPU kernel template number, accumulation type, and output format description; the runtime executes scale page loading, weight unpacking, kernel calls, and output conversion according to the above indexing relationship; when metadata verification fails, on-chip SRAM is insufficient, or candidate kernel templates are unavailable, the runtime loads the rollback NPU execution graph version according to the rollback version index.

[0083] In some implementations, during the online execution graph selection, the runtime calculates the logit margin, attention heat concentration, and recent window quality verification residual between the top-1 and top-2 nodes using a fixed token window unit. The number of consecutive windows can be 3 to 5. The high-throughput execution graph version increases the proportion of low-bit-width tiles by 10% to 20% and increases the average number of tiles in the scale shared domain compared to the balanced execution graph version. The conservative execution graph version increases the bit width by one level in the boundary region between the attention output projection and the vocabulary projection compared to the balanced execution graph version. The fallback NPU execution graph version can be a conservative execution graph version or an execution graph version pre-marked as having low SRAM requirements, fewer kernel templates, and stable verification summaries.

[0084] In some implementations, to verify the above technical effects, the layer-by-layer mixed precision scheme, the channel-level mixed precision scheme, and the tile-level precision island scheme of this application can be compared under the same NPU platform, the same large language model, and the same calibration / test sample conditions. The scale metadata reading volume, DMA unaligned access count, kernel template switching count, on-chip SRAM peak usage, first token latency, average token latency, and task-level error indicators are statistically analyzed, and the average values ​​of multiple runs are compared.

[0085] Technical effect The above embodiments have the following technical effects. By aligning the basic unit of mixed-precision quantization with the NPU executable tile, and using the executable tile as the basic unit of precision labeling, it helps to reduce hardware execution misalignment when configuring bit width by layer or by channel, and reduces the overhead of unpacking, data rearrangement, and kernel switching caused by inconsistency between precision granularity and hardware execution granularity. By constructing the scale shared domain in a two-stage manner, and using the effective bit loss not exceeding a threshold as a restricted sharing constraint, and splitting at the point of maximum error contribution when the cumulative error of the calibration set exceeds the budget as error feedback, it is possible to limit the effective bit compression of small dynamic range tiles by the shared scaling factor while reducing the amount of scaling factor metadata read and loading, and to subject the actual quantization error of the shared domain to domain-level budget constraints, so that the bandwidth gain of low bit weight compression is not easily offset by scaling factor metadata access. By organizing the scaling factor into a scale page table aligned by DMA burst and loading it page by page, it helps to transform the originally potentially fragmented scaling factor metadata access into aligned page loading that is more suitable for NPU runtime processing, thereby reducing the number of DMA start / stop times, address jump times, and scaling factor table lookup overhead.

[0086] Furthermore, by aggregating adjacent tiles with consistent execution attributes into precision islands and proactively merging excessively fine adjacent precision islands through a comparison of boundary arbitration values ​​and hysteresis thresholds, a more robust balance can be achieved between reducing kernel switching and execution graph fragmentation and avoiding unnecessary increases in bit width. This helps improve the NPU's continuous kernel execution capability within precision islands and reduces the degree of execution graph fragmentation. By compiling multiple NPU execution graph versions based on a shared interface and KV Cache format using precision islands, and selecting the compiled execution graph version during online runtime based on technical status signals such as context length, batch size, NPU load, power consumption status, target latency, and output risk signals, and by ensuring that forced switching only occurs at layer, token window, or precision island boundaries, it helps adapt to corresponding latency, power consumption, and quality constraints under different inference states, and reduces the risk of runtime state inconsistencies through hysteresis and rollback mechanisms. It should be noted that the above technical effects are all based on the collaborative processing between NPU tile, DMA, SRAM, scaling factor metadata, kernel template and execution graph, and do not depend on manual judgment or abstract rules themselves; and the above technical effects are obtained in scenarios where the NPU supports the corresponding tile shape and low bit format. This embodiment does not require the same degree of improvement to be produced in all context lengths, all batch sizes or all NPU platforms.

[0087] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not indicate or imply an actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In the application documents of this patent, if it refers to performing an action according to a certain element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0088] Furthermore, it should be understood that those skilled in the art can make appropriate modifications or substitutions to the above embodiments without departing from the technical concept of this application.

Claims

1. A method for mixed-precision quantization compression and execution graph compilation of large language models for neural network processing units (NPUs), characterized in that, include: The model matrix of the large language model is divided into multiple executable tiles according to the tile shape of the matrix calculation unit of the NPU, the direct memory access DMA alignment granularity, and the on-chip cache capacity. Based on the statistical characteristics and error budget of each executable tile, a precision label is determined for each executable tile, the precision label indicating the bit width used for quantization of the corresponding executable tile; Adjacent executable tiles with similar dynamic ranges and error budgets are organized into a scale shared domain, and the scale metadata of each executable tile within the scale shared domain is described by a shared scaling factor or a shared scale page table. Adjacent executable tiles with the same bit width, accumulation type, and scale sharing method as indicated by the precision tag are aggregated into precision islands, so that the NPU can execute continuously within the precision islands; The boundary of the precision island is determined or adjusted according to the NPU execution cost model, which includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip static random access memory (SRAM) usage, kernel switching cost at the boundary of the precision island, and execution graph fragmentation cost. Based on the determined or adjusted precision islands, an NPU execution graph is generated for the large language model. The NPU execution graph enables the NPU to maintain continuous kernel execution within the precision islands and switch at the boundaries of the precision islands.

2. The method according to claim 1, characterized in that, The aforementioned are precision labels for determining the executable tiles, including: Based on the weight statistics, activation statistics, and output sensitivity of each executable tile, and in conjunction with the NPU execution cost model, the bit width that minimizes the score and satisfies the error budget is selected from the set of candidate bit widths supported by the NPU as the bit width indicated by the precision label of the executable tile; wherein, the first... The bit width used for the executable tile quantization. for: In the formula, The set of candidate bit widths supported by the NPU; For the first An executable tile has a bit width The hardware execution cost includes weight reads, scale reads, low-bit unpacking, DMA alignment, and on-chip cache usage; For the first An executable tile has a bit width Quantization error estimation under the following conditions; For the first Error budget for each executable tile; For quality weights.

3. The method according to claim 1, characterized in that, Organizing adjacent executable tiles with similar dynamic ranges and error budgets into a scale-shared domain includes: Adjacent executable tiles are merged into the scale shared domain, ensuring that the effective bit loss of each executable tile within the scale shared domain relative to the maximum dynamic range of the scale shared domain does not exceed a set threshold; where the scale shared domain... Inner Effective bit loss of each executable tile for: In the formula, For the scale shared domain The maximum dynamic range of each executable tile within the range; For the scale shared domain Inner The dynamic range of an executable tile; Represents the logarithm to the base 2; When the cumulative quantization error of the scale shared domain on the calibration set exceeds the error budget of the scale shared domain, the scale shared domain is split at the executable tile where the quantization error contributes the most. Furthermore, the method further includes organizing the scaling factors of the scale shared domain into a scale page table aligned to the DMA burst, wherein the scale page table records the scale shared domain number, starting tile number, number of tiles, scaling factor storage address, DMA alignment information, and checksum; wherein the scale shared domain The corresponding scale page table byte length for: In the formula, For the scale shared domain The number of executable tiles included; The number of bytes occupied by the metadata for each scaling factor; The number of bytes for the DMA burst alignment granularity; For floor operations, the NPU loads the corresponding scaling factor according to the scale page table before executing the corresponding precision island.

4. The method according to claim 1, characterized in that, The step of aggregating adjacent executable tiles with the same bit width, accumulation type, and scale sharing method indicated by the precision tag into precision islands includes: Adjacent executable tiles are aggregated into the same precision island only when their bit width, accumulation type, scale sharing method, dequantization position, and kernel template number are all the same. When there are inconsistencies in the above attributes of adjacent executable tiles, a precision island boundary is set between the adjacent executable tiles, and format conversion, accumulation conversion or requantization processing is inserted at the boundary of the precision island.

5. The method according to claim 1, characterized in that, The process of determining or adjusting the boundaries of the precision islands based on the NPU execution cost model includes cost arbitration of the boundaries of adjacent precision islands: For bit width First precision island The adjacent bit width is Second precision island The higher of the two is denoted as The island with the lower bit width among the two is denoted as Calculate the boundary arbitration value : In the formula, For the first precision island With the second precision island Kernel switching costs between them; This refers to the cumulative format conversion cost between the two. For the cost of fragmenting the execution graph between the two; To make the precision island All executable tiles The bit width is changed from its original bit width Unified adjustment to the aforementioned higher bit width The additional execution costs incurred; For executable tile In-line width The execution cost below; For executable tile In its original position width The execution cost below; When the boundary arbitration value When the hysteresis threshold is greater than 1, the first precision island is... With the second precision island Merged into the higher bit width Execute the single precision island; otherwise, retain the first precision island. With the second precision island The boundary of the precision island between them.

6. The method according to claim 1, characterized in that, The process of compiling and generating an NPU execution graph for the large language model includes: For different latency, power consumption, or accuracy budgets, multiple NPU execution graph versions are compiled and generated. These multiple NPU execution graph versions share the same external interface, key-value cache (KV Cache) format, and the boundaries of the accuracy islands; and... The tile block table, tile precision label table, scale shared domain page table, precision island boundary table, NPU kernel template number, DMA address and alignment information, accumulation type and output format description, execution graph version number, quality check summary and fallback version index are written into the mixed precision compressed package so that the NPU can directly perform scaling factor loading, weight unpacking, matrix calculation and output conversion based on the metadata in the mixed precision compressed package at runtime.

7. The method according to claim 6, characterized in that, The method also includes an execution graph selection step performed during the online execution of model inference: Obtain at least one of the following technical status signals: current context length, batch size, NPU load, power consumption status, target latency, and output risk signal. The output risk signal includes at least one of the following: the logit margin of the top-1 and top-2 outputs, duplicate token risk, attention heat concentration, and recent window quality verification result. Based on the technical status signal, one NPU execution graph version is selected and loaded from the plurality of NPU execution graph versions. The selected NPU execution graph version... satisfy: In the formula, A set of candidate NPU execution graph versions; NPU execution graph version The overall execution cost; For NPU execution graph version Task-level error estimation; NPU execution graph version The delay index; For NPU execution graph version Required on-chip SRAM capacity; The upper limit of the allowable error; To delay the service target; The on-chip SRAM capacity available at runtime; When an NPU execution graph version switch is triggered, the switch is performed only at the boundary of the model layer, the boundary of the token window, or the boundary of the precision island; when the NPU execution graph version switch request falls inside the precision island, it is delayed until the boundary of the next precision island. When the verification digest in the mixed precision compressed package is inconsistent with the tile precision tag table, scale shared domain page table, precision island boundary table, or NPU kernel template number, or when the on-chip SRAM capacity available at runtime is less than the capacity required by the selected NPU execution graph version, the rollback NPU execution graph version is loaded according to the rollback version index.

8. A mixed-precision quantization compression and execution graph compilation system for large language models oriented towards neural network processing units (NPUs), characterized in that, include: The tile partitioning module is used to divide the model matrix of the large language model into multiple executable tiles according to the tile shape of the matrix calculation unit of the NPU, the direct memory access DMA alignment granularity, and the on-chip cache capacity. A precision label determination module is used to determine a precision label for each executable tile based on the statistical characteristics and error budget of each executable tile, wherein the precision label indicates the bit width used for quantization of the corresponding executable tile; The scale shared domain generation module is used to organize adjacent executable tiles with similar dynamic ranges and error budgets into scale shared domains, and to describe the scale metadata of each executable tile in the scale shared domain by a shared scaling factor or a shared scale page table. The precision island aggregation module is used to aggregate adjacent executable tiles that have the same bit width, accumulation type and scale sharing method indicated by the precision tag into precision islands, so that the NPU can execute continuously within the precision islands; A boundary determination module is used to determine or adjust the boundary of the precision island according to the NPU execution cost model. The NPU execution cost model includes at least the scale read volume of the scale shared domain, low bit unpacking cost, DMA alignment loss, on-chip static random access memory (SRAM) usage, kernel switching cost at the boundary of the precision island, and execution graph fragmentation cost. An execution graph compilation module is used to compile and generate an NPU execution graph for the large language model based on the determined or adjusted precision islands. The NPU execution graph enables the NPU to maintain continuous kernel execution within the precision islands and switch at the boundaries of the precision islands.

9. An electronic device, characterized in that, The device includes a neural network processing unit (NPU), a memory, and a processor. The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images