Neural network processor-oriented large language model residual flow equivalent replacement rotation quantization compression and online sorting arrangement method and system

Abstract

The application relates to the technical field of artificial intelligence model compression and neural network processor inference deployment, and discloses a residual stream equivalent replacement rotation quantization compression and online sorting arrangement method, system, equipment and storage medium for a neural network processor (NPU) large language model. The method models multiple residual positions as residual base state machines, sets a limited residual base composed of shared orthogonal transformation and position-level channel replacement for each residual position, degrades the conversion between adjacent residual positions to channel replacement without needing a dense rotation matrix multiplication; meanwhile, the residual base is absorbed and the space such as query, key and value projection output is kept from being rotated, online sorting and stability gate selection replacement are combined in layer-by-layer quantization, and a deployment package for NPU loading and verification is generated. Therefore, the grouping activation quantization error is reduced, the runtime conversion overhead is reduced, and the deployment consistency is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence model compression and neural network processor inference deployment technology, and in particular to a large language model residual stream equivalent permutation rotation quantization compression and online sorting and orchestration technology for neural network processors (NPUs). Background Technology

[0002] Large language models, in scenarios such as cloud-based inference services, edge-side intelligent terminals, enterprise-level private knowledge-based question answering, and robot or in-vehicle interaction, typically require low-latency, low-power inference deployment on neural network processors (NPUs) or dedicated AI accelerators. For example, in scenarios such as running local dialogue assistants on edge intelligent devices, providing low-latency text generation services on edge servers, and batch-running model inference on dedicated inference chips, models often need to continuously process long contexts, multi-turn dialogues, or batch requests, and are subject to strict constraints on storage capacity, memory access bandwidth, and power consumption. To reduce model storage requirements, external memory access pressure, and inference energy consumption, engineering practices typically employ low-bit quantization of weights and activations. For instance, linear projection, attention output projection, and multilayer perceptron projection are compressed using formats such as four-bit weights and eight-bit activations, four-bit weights and four-bit activations, eight-bit integers, or block floating-point numbers, and the weights and activations are read at runtime according to a grouping scale.

[0003] However, in Transformer-type large language models with residual streams, the residual activation distributions at different layers, different attention residual summing positions, and different multilayer perceptron residual summing positions are not consistent. Furthermore, hidden dimensions may contain channels with high amplitudes or uneven dynamic ranges. If activation quantization is performed in a fixed channel order, the scale distribution within the same quantization group can be affected by a few channels, amplifying quantization errors. While existing methods can use orthogonal transformations to disperse abnormal channels or perform channel rearrangement through offline statistics, several shortcomings remain: dense rotations at each residual position introduce additional matrix multiplication or dedicated kernel overhead; using only a single global rotation or static offline sorting is difficult to adapt to changes in residual activation distribution after layer-by-layer quantization; if channel rearrangement is not confined within safe residual boundaries, it may disrupt the semantic consistency of queries, key-value projection outputs, rotation position encoding, or key-value caching; and existing quantization strategies rarely explicitly consider the executability of neural network processors, making channel rearrangement prone to additional data transfer and kernel switching overhead during actual deployment.

[0004] Therefore, there is an urgent need for a technical solution that can coordinate low bit quantization error, runtime conversion overhead, and deployment consistency while maintaining the equivalent path of the residual stream and the executability of the neural network processor. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for equivalent permutation rotation quantization compression and online sorting of large language model residual streams for neural network processors, so as to coordinate low bit quantization error, runtime conversion overhead and deployment consistency while maintaining the equivalent path of the residual stream and the executability of the neural network processor, thereby solving the problems mentioned in the background art.

[0006] This application provides a method for equivalent permutation rotation quantization compression and online sorting of residual streams in large language models for neural network processors (NPUs), including: identifying multiple residual positions in the large language model, such as embedded outputs, attention residual sums, multilayer perceptron residual sums, and the final normalized or output layer inputs, and using these multiple residual positions as multiple states of a residual basis state machine; and setting residual bases for each of the multiple residual positions. ,in, For the first The residual basis at each residual location For all residual locations to be shared and satisfy The Hadamard transform, block Hadamard transform, or hardware-friendly orthogonal transform, For the first Channel permutation at each residual position It is the identity matrix. Number the residual positions; at least the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer are respectively absorbed by the residual basis of their corresponding residual positions, while keeping the output space, rotation position encoding space, and key-value cache space of query, key, and value projections unrotated; based on Generate permutation transformations between adjacent source residual locations and target residual locations. ,in, This is a permutation transformation from the source residual location to the adjacent target residual location. for transpose, For the channel permutation of the target residual position, the permutation transformation is configured to allow the runtime residual branch to perform channel rearrangement between adjacent residual positions using index permutation, column rearrangement, vector shuffling, direct memory access (DMA) aggregated reads, or tensor layout remapping, without performing dense rotation matrix multiplication; low-bit weight quantization is performed on the weights after absorbing the residual basis, and activation group quantization parameters are configured to obtain quantized weights, weight quantization scales, and activation quantization configurations, and a deployment package containing the channel permutation, the permutation transformation, the quantized weights, the weight quantization scales, the activation quantization configurations, and an NPU layout description is generated, so that the NPU can load the deployment package and perform low-bit weight-activation quantization inference.

[0007] In a preferred embodiment, the step of treating the plurality of residual positions as multiple states of the residual basis state machine includes: binding residual basis description information to each state, wherein the residual basis description information includes residual position number, layer number, position type, rotation type, index of the channel permutation, quantization group size, NPU layout identifier, source residual basis identifier, and target residual basis identifier; and binding transformation description information to the residual transition between adjacent states, wherein the transformation description information includes the permutation transition. The index, and a flag indicating whether the permutation transformation is to be fused to the next linear layer input read; the residual basis description information and the transformation description information are used for the generation of the deployment package.

[0008] In a preferred embodiment, the step of absorbing the residual basis at the corresponding residual positions includes: right-multiplying the embedding table by the residual basis at the initial residual position so that the embedding layer output is in the residual basis at the initial residual position; absorbing the residual basis at the current residual position by the input column of the attention projection and the multilayer perceptron projection; absorbing the residual basis at the target residual position by the output row of the attention output projection and the multilayer perceptron output projection; folding the final normalization into the output layer and absorbing the residual basis at the last residual position by the input of the output layer; and performing the permutation transformation on the main residual branch first when summing the residuals. Then add it to the output of the current sub-layer.

[0009] In a preferred embodiment, the channel permutation Generated by mean sorting, including: residual activation calculation at the residual position followed by the shared transformation. Activation after ,in, Let be the residual activation matrix at this residual location. For the shared transformation Activation after; Calculate the average amplitude for each channel in, For the number of sampled tokens, for The Middle The token, the Elements of each channel, For the first Average amplitude of each channel; based on the average amplitude of each channel The channels are sorted from largest to smallest to generate a channel permutation for the residual position, denoted as mean sort permutation. The mean sort permutation is used to improve the dynamic range grouping of grouped activation quantization.

[0010] In a preferred embodiment, the low-bit weight quantization is performed in a layer-by-layer quantization manner, and the channel permutation of at least some of the residual positions is determined online during the layer-by-layer quantization process, including: after quantization of each attention subunit or multilayer perceptron subunit is completed, the currently quantized prefix model is run on the scoring sample to collect the quantized residual activation of the next residual position. A candidate permutation set is generated for the next residual position. The candidate permutation set This includes identity permutation and activation of the quantized residuals according to the above-mentioned mean sorting method. The generated permutation includes at least two of the following: mean-sorted permutation, permutation generated by sorting by the 99th percentile of channel amplitude, permutation generated by sorting by the 50th percentile of channel amplitude, output-sensing permutation, and block-output-sensing permutation; for the candidate permutation set. Each candidate permutation in Calculate the overall score in, For candidate permutations, For candidate permutations The overall score, In candidate permutation Grouped quantization error with lower residual activation In candidate permutation The output error of subsequent linear layers, sub-units, or residual blocks. In candidate permutation The relative entropy of the small sample output logits. For candidate permutations The corresponding NPU rearrangement cost, , , , All are non-negative weight coefficients; From the candidate permutation set, according to the selection criteria... Select one candidate permutation as the channel permutation for the next residual position. This is then used to generate the permutation transformation between the residual position and its adjacent residual positions. .

[0011] In a preferred embodiment, the selection condition is a stability gating condition, and selecting a candidate permutation based on the selection condition includes: using the mean-sorted permutation as the default permutation. Calculate the comprehensive score for each candidate permutation on one or more scoring blocks, and take the average comprehensive score across the scoring blocks. And the candidate permutation with the smallest average comprehensive score is denoted as ; calculate Relative default substitution Benefits Return volatility across scoring blocks and adaptive switching threshold in, This is the default replacement. The candidate permutation with the lowest average overall score is selected. For candidate permutations The average composite score across the scoring blocks, for Compared to the benefits of the default replacement, For the fluctuation of returns across scoring blocks, For adaptive switching threshold, , All coefficients are non-negative; a corresponding candidate permutation is submitted as the channel permutation for the next residual position when any of the following conditions are met: the gain is greater than the adaptive switching threshold, i.e. Then submit ;or exist The winning percentage in each scoring block is no less than the voting threshold. Then submit ,in For the number of scoring blocks, This is the voting threshold; or the difference in relative entropy between the output logits does not exceed an approximately parallel threshold. From the approximate set of parallel groups, candidate permutations with low residual activation grouping quantization error and NPU rearrangement cost not exceeding the deployment cost constraint are selected. Approximate parallel threshold; otherwise, retain the default permutation. .

[0012] In a preferred embodiment, the deployment package further includes an equivalence certificate, which includes at least: a permutation checksum for each residual location, and each of the permutation transformations. The checksum, source residual basis identifiers of each weight tensor after absorbing the residual basis, target residual basis identifiers and weight checksums, non-rotation space whitelist, and floating-point equivalence check summary are provided. The non-rotation space whitelist covers the output space of query, key-value projection, rotation position encoding space, key-value cache space, and attention score space. The equivalence certificate is used by the NPU to check whether the number of residual positions matches the number of model layers, the checksums of each permutation, the source residual position identifiers and target residual position identifiers of each permutation transformation, and the source residual basis identifiers and target residual basis identifiers of each weight tensor are consistent when loading the deployment package and before performing inference. If any check fails, the NPU will refuse to load the deployment package, fall back to the deployment package using identity permutation or mean sort permutation, or trigger recompilation.

[0013] This application provides a system for equivalent permutation, rotation, quantization, compression, and online sorting of residual streams in large language models for neural network processors (NPUs), comprising: a residual position identification and modeling module, used to identify multiple residual positions in the large language model, including embedded outputs, attention residual sums, multilayer perceptron residual sums, and the final normalized or output layer inputs, and to treat the multiple residual positions as multiple states of a residual basis state machine; and a residual basis setting module, used to set residual bases for the multiple residual positions respectively. ,in, For the first The residual basis at each residual location For all residual locations to be shared and satisfy The Hadamard transform, block Hadamard transform, or hardware-friendly orthogonal transform, For the first Channel permutation at each residual position The identity matrix is ​​defined as follows: A weight absorption and whitelist protection module is used to absorb the residual basis of the corresponding residual positions of the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer, while maintaining the output space, rotation position encoding space, and key-value cache space of the query, key, and value projections unrotated; A permutation transformation generation module is used to... Generate permutation transformations between adjacent source residual locations and target residual locations. ,in, for transpose, For the channel permutation of the target residual position, the permutation transformation is configured so that the runtime residual branch can perform channel rearrangement between adjacent residual positions using index permutation, column rearrangement, vector shuffling, direct memory access (DMA) aggregated read, or tensor layout remapping, without performing dense rotation matrix multiplication; The quantization compilation and deployment package generation module is used to perform low-bit weight quantization on the weights after absorbing the residual basis, configure the activation group quantization parameters, obtain quantization weights, weight quantization scales, and activation quantization configurations, and generate a deployment package containing the channel permutation, the permutation transformation, the quantization weights, the weight quantization scales, the activation quantization configurations, and an NPU layout description, so that the NPU can load the deployment package and perform low-bit weight-activation quantization inference.

[0014] This application provides an electronic device, including a processor, a memory, and a neural network processor (NPU), or including a processor, a memory, and an interface communicating with the NPU, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the above-described method.

[0015] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0016] In this embodiment, by modeling the residual flow as a residual basis state machine and using a restricted residual basis composed of a combination of shared orthogonal transformation and position-level channel permutation, it helps to reduce group activation quantization error, reduce runtime conversion overhead, and improve deployment consistency, while helping to keep the model's floating-point output consistent within the numerical error range.

[0017] The residual location identification and modeling module identifies the embedded output, attention residual summation, multilayer perceptron residual summation, and final normalized or output layer input in the large language model as multiple residual locations. These residual locations are then organized into a residual basis state machine, transforming the residual stream from a mere intermediate tensor into a deployable object with states, descriptive information, and transformation edges. Each residual location can be bound to residual basis descriptive information such as residual location number, rotation type, channel permutation index, quantization group size, source residual basis identifier, and target residual basis identifier. Adjacent states can be bound to transformation descriptive information including permutation transformation index and fusion read flag. This provides a unified data foundation for subsequent weight absorption, permutation transformation, online sorting, and runtime loading verification, helping to reduce the risks of unclear state relationships, inconsistent layout relationships, or ambiguous transformation boundaries during the reparameterization of complex residual streams.

[0018] The residual basis setting module is used to set different residual positions. Constrained residual basis of form, and enabling shared transformation satisfy While retaining the ability of orthogonal transformation to disperse anomalous channels, it limits the differences between different residual positions to position-level channel permutation. Above. Therefore, the transformation between adjacent source residual positions and target residual positions can be achieved through permutation transformation. The representation is written into the permutation transformation table by the permutation transformation generation module, without introducing arbitrary dense rotation matrix multiplication at runtime; the transformation between adjacent states in the residual branch can be transformed into index permutation, column rearrangement, vector shuffling, direct memory access aggregation read, or tensor layout remapping, reducing the residual transformation overhead of a single token to approximately Multiplication and addition reduced to approximation Index access, when supported by hardware, can also integrate the transformation into the input reading of the next linear layer, which helps reduce the execution overhead caused by additional rotation calculations and explicit data handling during low-bit inference, and improves the feasibility of executing such quantized compilation results on neural network processors.

[0019] The weight absorption and whitelist protection module absorbs the residual basis at the corresponding residual positions of the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer, respectively, and performs the process on the main residual branch before residual summation. The formal permutation transformation enables the main residual branch and the current sub-layer output to be summed under the same target residual basis, thus helping to maintain the semantic consistency of the residual summation path. Simultaneously, the output spaces of query, key, and value projections, the rotation position encoding space, the key-value cache space, and the attention score space are whitelisted for non-rotation spaces. This allows the hidden dimensions carrying the residual flow to be permuted and rotated, while the semantics of the head dimension, position encoding relationships, and cache structure within the attention mechanism are not affected by this permutation and rotation propagation. This collaborative processing helps avoid compromising the consistency of the attention mechanism's internal space to reduce quantization errors, and reduces the risk of residual flow reparameterization disrupting equivalent paths in the model.

[0020] By activating the residual at the residual location Calculate the activation after the shared transformation and according to the average amplitude of each channel The mean-sorting permutation, generated from largest to smallest, reorganizes the channel order of grouped activated quantization based on the persistent channel amplitude characteristics at the residual positions. This mean-sorting permutation does not simply change the channel numbers, but rather integrates with the shared transformation. The combined effect of group activation quantization parameters and subsequent neural network processor layout makes the organization of high-amplitude channels in quantization groups more controllable, which helps to reduce scale waste caused by excessive differences in the dynamic range of channels within the same group. Compared with sorting methods that only focus on the tail fractions of a single extreme token, mean sorting permutation usually provides a more stable improvement to group activation quantization error, and no obvious side effects have been observed on weight quantization error.

[0021] The online sorting and orchestration module collects the quantized residual activation generated by the currently quantized prefix model during the layer-by-layer quantization process. Furthermore, a selection is made from candidates such as identity permutation, mean-sorted permutation, quantile-sorted permutation, output-sensing permutation, and block-output-sensing permutation. This ensures that the determination of channel permutation no longer relies solely on full-precision offline calibration activation, but rather reflects the impact of preceding quantization on the subsequent residual activation distribution, helping to reduce the mismatch between offline static statistics and the true distribution after quantization. Further, through comprehensive scoring... Simultaneously considering residual activation grouping quantization error, subsequent linear layer or residual block output error, small sample logits relative entropy, and neural network processor reordering cost. Placing accuracy-related metrics and deployment cost metrics in the same selection framework helps to make constrained choices between quantification error improvement and hardware execution costs.

[0022] Stability gating is used to select the mean-ranked permutation as the default permutation, and the results are based on the payoff of candidate permutations relative to the default permutation, payoff fluctuations across scoring blocks, and adaptive switching thresholds. The mechanism employs cross-score block voting and constraints on activation quantization error and rearrangement cost when logits relative entropy approximates parallel to determine whether to submit candidate permutations. This avoids frequent changes in residual position permutations due to small score fluctuations on a single score block. Instead of a greedy choice of submitting based solely on the minimum current score, this mechanism establishes a more robust submission condition that balances quantization gains, statistical stability, and neural network processor execution costs. This helps reduce layout switching without stable gains, mitigates the risk of duplicate deployment package generation or recompilation, and improves the reproducibility of permutation selection results under different score sample conditions.

[0023] The quantization compilation and deployment package generation module generates a deployment package containing descriptions of channel permutations, permutation transformations, quantization weights, weight quantization scales, activation quantization configurations, and neural network processor layouts. An equivalence certificate is further set within the deployment package to ensure that complex residual basis absorption, permutation transformations, and low-bit weight layouts are verified before runtime loading. The equivalence certificate includes permutation checksums, transformation checksums, source residual basis identifiers, target residual basis identifiers, weight checksums, a non-rotation space whitelist, and a floating-point equivalence checksum digest. This allows the neural network processor or related processors to detect inconsistencies in the number of residual positions, transformation source-target relationships, or weight basis identifiers before inference. If verification fails, the module refuses to load, rolls back to a deployment package using identity permutations or mean-sort permutations, or triggers recompilation. This helps reduce the risk of implicit inference errors caused by metadata inconsistencies during complex quantization compilation and deployment.

[0024] By implementing the above methods and steps into modules for residual location identification and modeling, residual basis setting, weight absorption and whitelist protection, permutation transformation generation, and quantization compilation and deployment package generation, and further including online sorting and orchestration and deployment verification modules, a complete processing chain can be formed from model parsing, residual basis setting, weight reparameterization, low-bit quantization compilation to neural network processor deployment and execution. Furthermore, when the electronic device includes a processor, memory, and a neural network processor or an interface communicating with the neural network processor, the processor can perform model parsing, permutation search, online sorting, and deployment package generation. The neural network processor can load the verified deployment package and execute low-bit weight-activation quantization inference, thereby establishing a clear correspondence between the software orchestration results and the hardware execution path. The aforementioned beneficial effects can also be achieved through a computer-readable storage medium storing the corresponding computer program, which is executed by the processor.

[0025] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a method for equivalent permutation, rotation, quantization, compression, and online sorting of large language models for neural network processors, according to an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a large language model residual flow equivalent permutation rotation quantization compression and online sorting and orchestration system for neural network processors according to an embodiment of this application. Detailed Implementation

[0027] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0028] Explanation of some concepts: Residual stream refers to the hidden state data stream that is continuously passed between the embedding output, attention sublayer, multilayer perceptron sublayer, and final output path in a large language model and participates in residual summation.

[0029] The residual location refers to the model location in the residual flow that is suitable for residual basis setting, permutation transformation, or residual summation. Examples include the location of the embedded output, the summation of attention residuals at each layer, the summation of residuals at each multilayer perceptron layer, and the location of the final normalized or output layer input.

[0030] A residual basis state machine is a data organization method or structure that treats multiple residual positions as states, binds residual basis description information to each state, and binds transition description information between adjacent states.

[0031] residual basis , refers to the The hidden dimension representation basis used for each residual location is of the form: ,in For an orthogonal transformation shared by all locations, Replace the channel at this location.

[0032] Shared transformation This refers to all residual locations sharing the same conditions and satisfying the following conditions: Hadamard transform, block Hadamard transform, or other hardware-friendly orthogonal transforms.

[0033] Channel replacement , refers to the The hidden dimension channel rearrangement relationship corresponding to each residual position can be represented as a permutation matrix, an integer index array, or a tensor layout descriptor.

[0034] Permutation and transformation This refers to the location of the source residual. To the adjacent target residual position The channel rearrangement relationship satisfies This is used to allow runtime residual branches to be transformed between adjacent residual positions in a channel rearrangement manner without performing dense rotation matrix multiplication.

[0035] Mean-order permutation refers to the activation of a residual at a specific residual position. First obtained through shared transformation Then calculate the average amplitude for each channel. and according to Channel permutations generated by sorting channels from largest to smallest.

[0036] Group quantization refers to a quantization method that divides multiple channels on a hidden dimension into several groups during quantization and allows channels within the same group to share the quantization scale or quantization parameters.

[0037] The non-rotation space whitelist refers to the set of spaces that remain unrotated during residual basis absorption and permutation arrangement, including the output space of query, key, and value projection, the rotation position encoding space, the key-value cache space, and the attention score space.

[0038] Online sorting and orchestration refers to the process of selecting channel permutations for the next residual position from the candidate permutation set during the layer-by-layer quantization process, based on the quantized residual activation, subsequent output error, logits relative entropy, or neural network processor rearrangement cost generated by the current quantized prefix model.

[0039] Stability gating refers to a mechanism that, during the candidate permutation selection process, decides whether to submit a candidate permutation or maintain the default permutation based on the payoff of the candidate permutation relative to the default permutation, the payoff fluctuation across scoring blocks, the adaptive switching threshold, the voting results, or the condition of near-parallelism.

[0040] Overall score It refers to the method used to evaluate candidate permutations. In the The score for the fit of a residual position can be obtained by weighting the residual activation group quantization error, subsequent output error, logits relative entropy, and neural network processor rearrangement cost.

[0041] Neural network processor reordering cost , refers to candidate permutation The deployment may incur execution costs such as index replacement, direct memory access aggregate reads, vector shuffling, tensor layout remapping, or core switching.

[0042] Quantized residual activation This refers to the process of layer-by-layer quantization, where the model generated by the pre-quantized model running on the scoring samples is located at the [number]th [level]. Residual activation at each residual location.

[0043] A deployment package refers to a data set that a neural network processor loads and executes low-bit weighted activation quantization inference. Examples of such packages include a channel permutation table, a permutation transformation table, quantization weights, weight quantization scales, activation quantization configurations, online location summaries, and a neural network processor layout description.

[0044] An equivalence certificate is a data structure in the deployment package used by the neural network processor to perform consistency checks during loading. Examples include permutation checksums for each residual location, checksums for each permutation transformation, source and target residual base identifiers for each weight tensor, a non-rotation space whitelist, and a floating-point equivalence check digest.

[0045] Low-bit weighted activation quantization inference refers to the large language model inference process executed on a neural network processor or a corresponding processor using low-bit weights, weight quantization scales, and activation group quantization configurations.

[0046] The following is a brief summary of some of the innovative aspects of this application: In summary, the inventiveness of this application does not lie in individually employing any one of the methods—such as Hadamard transform to disperse anomalous channels, statistical sorting by channel, or low-bit quantization of weights and activations—but rather in utilizing the overall technical concept of a residual base state machine to uniformly arrange the aforementioned, originally independent processing actions under the residual flow equivalence constraint for neural network processor inference. Specifically, the residual position recognition and modeling module first models the embedded output, attention residual summation, multilayer perceptron residual summation, and the final normalized or output layer input as multiple states in the residual base state machine. to The residual basis setting module then configures, with constraints, an orthogonal transformation shared by all locations for each state. Replacement with position-level channel Constrained residual basis formed by combination Shared transformation Used to disperse abnormal channels under orthogonal equivalence constraints, position-level channel permutation. The channel order for grouped quantization is reorganized according to the activation distribution at different residual positions, rather than introducing arbitrarily dense invertible matrices that are distinct at each position. This is because each residual position shares the same... of Adjacent source residual positions Residual position relative to target basis transformations between Only then can it mathematically degenerate into a permutation transformation that contains only permutations. The replacement transformation generation module writes the data into the replacement transformation table, so that when the runtime residual branch carrying the residual flow switches between adjacent positions, it is not necessary to perform an operation on each token with a scale of approximately [missing information]. The dense rotation matrix multiplication is replaced by a lightweight approach that approximates a single memory access address remapping, such as index permutation, column rearrangement, vector shuffling, or direct memory access aggregation reads. This coupling relationship, which trades shared orthogonality for runtime permutation executability, is difficult to foresee in existing technologies that only expose Hadamard rotations or only expose invertible matrix fusion weights. More importantly, the reason why the aforementioned restricted residual basis can hold without destroying the model semantics also depends on the fact that the weight absorption and whitelist protection module establishes source residual basis identifiers and target residual basis identifiers between the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer, and includes the output space of query, key, and value projection, rotation position encoding space, key-value cache space, and attention score space in the non-rotation space whitelist. Once this whitelist constraint is missing, position-level permutation rotations will propagate to attention scores, rotation position encoding, or key-value cache, making the equivalent path no longer valid. Therefore, there is a strong dependency between the whitelist and the restricted residual basis, where the absence of one leads to the failure of the other. Furthermore, during the layer-by-layer quantization process, the online sorting and orchestration module activates the quantized residuals generated by the quantized prefix model. Instead of relying on full-precision static activation, it combines a comprehensive score. Neural network processor reordering cost With stability gating threshold By selecting the best or back off from candidate permutations, the permutation selection is made to both fit the deployment state after the propagation of layer-by-layer quantization error and avoid frequent switching without stable returns due to small sample score fluctuations. Thus, shared orthogonality makes the transformation degenerate into a permutation, the permutation transformation table makes the scheme executable on the neural network processor, the non-rotation space whitelist makes the equivalent path valid, online sorting and stability gating make the permutation fit the real quantization state, and the verifiable deployment package makes the compilation result safe to be loaded by the neural network processor. The above features continuously transmit the same residual basis state in the three stages of quantization compilation, residual summation and runtime loading, and are mutually restrictive. If any stage is removed, it is difficult to simultaneously obtain the coordination effect between residual flow equivalence, group activation quantization error reduction and runtime executability. Therefore, the overall synergy is far beyond what can be achieved by simply superimposing the known methods.

[0047] Furthermore, the inventors of this application, through long-term in-depth research, have discovered that the main contradiction in the low-bit deployment of large language models is not simply the lack of a stronger rotation matrix or a more complex sorting index, but rather the lack of a common set of transferable and verifiable state constraints among residual flow equivalence, quantization group dynamic range, and the runtime executability of neural network processors.

[0048] After a thorough analysis of the principles of existing rotation quantization schemes, the inventors discovered that while arbitrarily dense transformations are applied to each residual position, theoretically increasing the degree of freedom of adaptation, adjacent residual branches require basis transformations before residual summation. This introduces additional matrix multiplications, data transport, or kernel switching during runtime. Conversely, if a fixed rotation is used for the entire model to reduce overhead, the activation differences at the residual summation points of different layers and attention levels compared to the multilayer perceptron cannot be fully utilized. Therefore, there exists a constraint relationship between the degree of freedom of rotation transformation and runtime overhead that needs to be coordinated. The root cause lies in the requirement that the rotation matrix be differentiated and transformed with low overhead between different residual positions, a constraint that existing schemes fail to address uniformly.

[0049] Furthermore, existing offline sorting typically determines the channel order based on full-precision calibrated activation. Through analysis of the layer-by-layer quantization process, the inventors realized that pre-quantization of weights has altered the true distribution of subsequent residual activations, leading to mismatches between static sorting and quantized activations. These mismatches manifest as scale organization instability and output error propagation in grouped activation quantization. More critically, through repeated experiments and theoretical derivations, the inventors recognized that residual flow permutations cannot arbitrarily traverse query, key-value projection output, rotation position encoding, and key-value cache space; otherwise, inconsistencies between the internal semantics of attention and cache layout can easily arise. Moreover, if channel rearrangement cannot be compressed into an index table, weight layout, and lightweight transformation, additional aggregation, rearrangement, and kernel switching overhead will still be introduced when deploying on a neural network processor.

[0050] Based on the above in-depth research, the inventors of this application propose a system with a residual-based state machine as the framework and a restricted residual basis. With as the core, For runtime transformation, and combining online sorting and orchestration, stability gating, and equivalence certificates in layer-by-layer quantization, this solution organizes rotation, permutation, quantization, and deployment verification into a single transitive and verifiable orchestration process. The implementation process of this application is described in detail below through specific embodiments.

[0051] To make the purpose, technical solution, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this application. Without departing from the technical concept of this application, those skilled in the art can make adaptive adjustments to the transformation type, quantization format, candidate permutation set, scoring objective, and runtime rearrangement method based on the actual large language model structure, the low-bit computing power of the neural network processor (NPU), and deployment constraints. For ease of description, the symbols used throughout the text are uniformly agreed upon at the beginning of the specific embodiments, and this agreement is followed throughout; unless otherwise stated, terms such as "residual position," "residual basis," "channel permutation," "permutation transformation," "shared transformation," "mean-sorted permutation," "default permutation," "stability gating," "deployment package," and "equivalence certificate" have the same meaning throughout the text, and the same technical feature uses the same technical name.

[0052] The core concept of this application is to model the residual flow of a large language model as a verifiable residual basis state machine. At each residual position, a restricted residual basis composed of a shared orthogonal transformation and position-level channel permutation is used. This causes the basis transformation between adjacent residual positions to degenerate into lightweight channel permutation at runtime, eliminating the need for dense rotation matrix multiplication. While maintaining a safe boundary that prevents rotation of query, key, and value projection output spaces, online sorting and stability gating during layer-by-layer quantization are combined to generate a deployment package that can be loaded, verified, and executed by the NPU. The large language model can be a Transformer-type decoder model with residual flow, including an embedding layer, multiple Transformer layers, attention projection, attention output projection, multilayer perceptron projection, residual summation, final normalization, and an output layer. The following describes the method implementation as the main thread, followed by corresponding system implementation, electronic device implementation, and storage medium implementation.

[0053] I. Method Implementation Examples like Figure 1 As shown, the method of this application may include steps 100 to 1000.

[0054] Step 100: Analyze the structure of the large language model, identify multiple residual positions, and model the multiple residual positions as multiple states of the residual basis state machine.

[0055] Specifically, the system first parses the network structure of the large language model, identifying the locations of the embedding layer, the summation of attention residuals at each layer, the summation of residuals in the multilayer perceptron, and the final normalization or output layer input. It should be noted that these locations are where the residual stream is consumed by subsequent sub-layers or updated by residual summation in the model, and therefore can serve as safe boundaries for residual basis switching and permutation orchestration. For example, for a large language model with… The model of the layer decoder can be denoted as the position of the residual after embedding the output as... , will the The position of the residual after summing the layer attention residuals is denoted as , will the The position of the residual after summing the residuals of the multilayer perceptron is denoted as . ,in The range of values ​​is to Therefore, the set of residual basic states can be expressed as: In Formula 1, Represents the set of residual basic states. This represents the initial residual state after embedding the output. Indicates the first The state after summing the attention residuals of each layer. Indicates the first The state after summing the residuals of a multilayer perceptron. Indicates the number of decoder layers.

[0056] More specifically, the system binds residual basis description information to each state. This residual basis description information may include residual location number, layer number, location type, rotation type, channel permutation index, quantization group size, NPU layout identifier, source residual basis identifier, and target residual basis identifier. The location type exemplifies embedding, attention residual summation, multilayer perceptron residual summation, or final location; the rotation type exemplifies Hadamard transform, block Hadamard transform, or permutation only. The system also binds transformation description information to residual transitions between adjacent states. This transformation description information may include source residual location number, target residual location number, the index of the permutation transition described below, and a flag indicating whether to fuse the permutation transition to the next linear layer input read. Through these states and description information, the residual stream is no longer treated as a regular intermediate tensor, but rather organized as a deployment object with verifiable base states and transformation edges. All of the above description information is stored as table entries for subsequent deployment package generation.

[0057] Step 200: Set residual bases for multiple residual locations and determine the permutation and transformation relationships between adjacent residual locations.

[0058] Specifically, for the first For each residual location, the system is configured with the following residual basis: In Formula 2, Indicates the first The residual basis at each residual location This represents an orthogonal transformation shared by all residual positions. Indicates the first Channel permutation at each residual position. It can be a Hadamard transform, a block Hadamard transform, or a hardware-friendly orthogonal transform, and must satisfy: In Formula 3, express transpose, This represents the identity matrix. For example, if a residual tensor is at the th... The original representation of each residual position is as follows: Then its representation under the residual basis is: Because each residual location shares the same The differences between different residual positions are mainly due to their respective channel permutations. This means that the ability to disperse abnormal channels through orthogonal transformation is retained, while avoiding the runtime matrix multiplication overhead caused by introducing different dense rotation matrices between adjacent residual positions.

[0059] More specifically, it is precisely because each residual position shares the same condition that satisfies Formula 3 When the residual branch needs to be generated by the first The residual position is transformed to the adjacent _th ... When there are several residual positions, the basis transformation can be derived as follows: In Formula 4, express inverse transform, The residual basis represents the location of the target residual. This represents the channel permutation at the target residual location. (See shared transformation) In the transformation of adjacent residual bases, they cancel each other out, leaving only combinations between channel permutations. Based on this, the system defines the permutation transformation from the source residual location to the target residual location: In Formula 5, Indicates from the first The source residual position is located at the adjacent first... The permutation transformation of the target residual positions. It should be noted that this constraint is one of the key differences between this application and the scheme using arbitrary dense invertible matrices: dense invertible matrices typically require an execution scale of approximately [missing information] for each token at runtime. ( The multiplication and addition operations (for the hidden dimension) are different from the permutation transformations in this application, which are essentially column rearrangements or index rearrangements of the hidden dimension. The runtime overhead is approximately the same as a single memory read / write address remapping, with a complexity of approximately [missing information]. Furthermore, the aforementioned It can be stored as a permutation matrix, or in the engineering implementation as an array of integer indices, a grouped index table, or a tensor layout descriptor that can be consumed by the NPU; for NPU deployment, it is preferable to store it as a permutation matrix, or as an array of integer indices, a grouped index table, or a tensor layout descriptor that can be consumed by the NPU. It is represented as an index table or layout descriptor so that the compiler or runtime can execute it through address remapping, vector shuffling, direct memory access (DMA) aggregated reads, etc., instead of implementing it as a general matrix multiplication.

[0060] Step 300: Determine the channel permutation at each residual position.

[0061] It should be noted that channel replacement It can be generated offline through statistical analysis, determined online during the layer-by-layer quantization process, or determined by a combination of both. The offline generation method is explained in two sub-steps below, while the online generation method is detailed in step 400.

[0062] Step 310: Generate channel permutations offline based on mean sorting. Specifically, for a given residual position, the system collects the residual activation matrix from the calibration phase. And calculate the shared transformation Activation afterward: In Formula 6, The residual activation matrix representing the location of the residual can typically be represented as a two-dimensional matrix consisting of the number of sampled tokens and the hidden dimension. This represents the residual activation after the shared transformation. Subsequently, the system... Calculate the average amplitude for each channel: In Formula 7, Indicates the number of sampled tokens. express The Middle The token, the Elements of each channel, Indicates the first The average amplitude of each channel This indicates taking the absolute value. The system calculates the average amplitude of each channel. The channels are sorted from largest to smallest to obtain the channel permutation at the residual position, which is denoted as mean-sort permutation in this paper. Optionally, when the average amplitudes of two channels are equal or the difference is less than the computational precision threshold, their relative order is determined by ascending order of the original channel numbers to ensure that the channel permutation can be reproduced during repeated compilation. More specifically, mean-sort permutation focuses on persistently high-amplitude channels. Its role is to ensure that persistently high-amplitude channels after the shared transformation enter the channel groups of grouped activation quantization in a defined order, thereby helping to control the dynamic range distribution within each quantization group and reducing the scale waste caused by excessive differences in channel amplitudes in grouped activation quantization. Compared to sorting methods that only focus on the tail digits of a single extreme token, mean-sort permutation generally provides more stable improvement to the error of grouped activation quantization, and no significant side effects were observed on the error of weighted quantization, indicating that it mainly improves the organization of activated groups.

[0063] Step 320: Generate a candidate permutation set. Besides mean-sorted permutations, the system can also provide the first... A set of candidate permutations is generated from each residual position: In Formula 8, Indicates the first A set of candidate permutations for each residual position. This indicates an identity permutation. This indicates a permutation based on the mean. This represents the permutation generated by sorting according to the 99th percentile of the channel amplitude. This represents the permutation generated by sorting according to the 50th percentile of the channel amplitude. This indicates the output perceptual permutation. This represents a block output-aware permutation. For example, the identity permutation preserves the channel order after the shared transformation and serves as a stable baseline; the output-aware permutation can be determined by combining the channel sensitivity of subsequent input projections, which can be determined by the weighted norm after folding normalization, the output error contribution, or the approximate diagonal term of the Hessian matrix; the block output-aware permutation can temporarily run the current residual block or the next sub-unit under candidate permutations, using the block output error as part of the selection criterion. It should be noted that in actual implementation, this set may only include at least two candidate permutations, and it is not required to include all candidates each time; by selecting on a small-scale candidate set, it is possible to avoid problems with a large-scale... (Right now factorial, representing A global search is performed on the permutation space of all permutations of the channels (if infeasible), while retaining the ability to adapt to different models, different layers and different quantization bit widths.

[0064] Step 400: Perform online sorting and orchestration during the layer-by-layer quantization process.

[0065] In a preferred embodiment, the low-bit weight quantization is performed using a layer-by-layer quantization method, such as GPTQ, RTN, or AWQ, and the channel permutation of at least some of the residual positions is determined online during this layer-by-layer quantization process. It should be noted that offline sorting, based on full-precision or static calibration activation, cannot fully reflect the changes in residual activation distribution after layer-by-layer quantization. To address this issue, the system determines the channel permutation online during the layer-by-layer quantization process. Specifically, the system processes the attention subunit and the multilayer perceptron subunit sequentially according to the model's forward order. After completing the weight quantization of each subunit, the system runs the currently quantized prefix model on the scoring samples to collect the quantized residual activation at the next residual position. In Formula 9, Indicates the first Quantized residual activation at each residual location This indicates the number of valid tokens collected at this location. This indicates the hidden dimension. More specifically, unlike offline sorting which is based on full-precision static activation, this method collects activations generated by the quantized prefix model. Therefore, it can reflect the impact of prior quantization errors on the subsequent residual activation distribution, making the selected permutation closer to the true activation distribution after layer-by-layer quantization error propagation, and reducing the mismatch between offline static statistics and the true distribution after quantization.

[0066] Furthermore, for the candidate permutation set Each candidate permutation in The system temporarily sets the current residual basis to A temporary rearrangement is performed on the current output projection row, the next input projection column, or the corresponding layout description, and then a comprehensive score is calculated on the scoring samples: In Formula 10, Indicates candidate permutation The overall score, Indicating in candidate permutation Grouped quantization error with lower residual activation Indicating in candidate permutation The output error of subsequent linear layers, sub-units, or residual blocks. Indicating in candidate permutation The relative entropy of the small sample output logits. Indicates candidate permutation The corresponding NPU rearrangement cost, , , , These are non-negative weighting coefficients, used to characterize the weights of different error terms and deployment cost terms in the current deployment objective. For example, when using block floating-point format (e.g., MXFP4, MXINT8), The cost can be calculated based on the hardware quantization group corresponding to the sharing index or sharing scale; if a candidate permutation prevents channels within the same quantization group from being accessed consecutively in the physical layout, thereby introducing additional DMA aggregate reads, explicit index rearrangements, or core layout switching, then this portion of the cost is included. The synergy among the various components of the overall score lies in: Directly target the activation of group quantization. and It reflects the impact of the permutation on subsequent outputs and even the overall output distribution. This incorporates deployment feasibility into the selection process, allowing for a balanced trade-off between accuracy gains and deployment costs. Furthermore, when the scoring sample is divided into multiple scoring blocks, the system can calculate the average comprehensive score across these blocks. In Formula 11, Indicates candidate permutation The average composite score across the scoring blocks, Indicates the number of scoring blocks. Indicates candidate permutation In the The system can record the candidate permutation with the smallest average comprehensive score across all scoring blocks as follows: ,Right now However, instead of submitting the candidate directly, it proceeds to the stability gating process in step 500.

[0067] Step 500: Submit or roll back the channel replacement selected online based on stability gating conditions.

[0068] It should be noted that relying solely on the minimum overall score on a single scoring block as the submission criterion is prone to frequent shifts in residual positions due to fluctuations in the relative entropy of small samples or local activation bias. Therefore, this step introduces stability gating. Specifically, the system uses mean-sorted permutations as the default permutation. Optionally, if the current residual position cannot generate a mean-sorted permutation, or if the mean-sorted permutation fails the equivalence check, the identity permutation can be used as the default permutation. System calculation Benefits relative to the default replacement: In Formula 12, The candidate with the smallest average composite score Relative default substitution The benefits, This represents the average overall score of the default permutation. Indicates candidate The system also calculates the average overall score of the return. In Formula 13, Indicates candidate The larger the value of the return fluctuation relative to the default permutation across multiple scoring blocks, the more unstable the return of the candidate. Furthermore, the system determines an adaptive switching threshold based on the return fluctuation and NPU rearrangement cost: In Formula 14, Indicates the adaptive switching threshold. This represents the coefficient indicating the impact of return volatility on the switching threshold. This represents the impact coefficient of NPU rearrangement cost on the switching threshold; both are non-negative coefficients. More specifically, this threshold is derived from the score fluctuation of the current scoring block and the candidate deployment cost, rather than being fixed to a certain empirical value, thereby reducing the unreproducible risk introduced by manually setting a threshold.

[0069] Based on this, the system will only submit the corresponding candidate permutation as the final channel permutation for the residual position if any of the following conditions are met. The first is the marginal benefit condition, i.e., when... When the score is determined to have exceeded the cost of score fluctuation and rearrangement, it is submitted. The second is the cross-scoring block voting condition, that is, when... exist The winning percentage in each scoring block is no less than the voting threshold. When it is determined that it has cross-sample consistency, it is submitted. ,in This represents the voting threshold. The third condition is the approximate parallel condition, meaning the difference between the relative entropies of the constructed output logits does not exceed the approximate parallel threshold. Approximate set of parallels: In Formula 15, This represents the candidate set whose relative entropy is approximately equal to that of logits. Indicates candidate permutation The relative entropy of logits, Indicates approximately parallel thresholds, Represents the candidate set The candidate permutations used for comparison. If... If a candidate permutation exists that results in the residual activation grouping quantization error being lower than the default permutation and the NPU rearrangement cost not exceeding the deployment cost constraint, then that candidate permutation is submitted. If none of the above conditions are met, the default permutation is retained. And record the stable backoff flag in the online location summary.

[0070] Furthermore, this step may include several deterministic exit processing steps to ensure deployable outputs under various boundary conditions: rollback to default substitution when the quantized residual activation results in a non-numerical value that cannot be corrected by discarding anomaly tokens; elimination of a candidate when its overall score cannot be calculated, and rollback to identity substitution when all scores cannot be calculated; no voting conditions when the number of valid score blocks is insufficient, only marginal benefit conditions and default rollback are used; priority is given to the candidate with the smaller average overall score when multiple candidates have the same winning percentage; elimination or improvement of the overall score by rearranging the cost term when the explicit rearrangement cost of a candidate exceeds the deployment constraint; no entry into the commit set when the source residual base identifier of a candidate after temporary rearrangement is inconsistent with the current residual state; and rollback when a candidate substitution is to be propagated to spaces such as query, key-value projection output, rotation position encoding, key-value cache, or attention score. It should be noted that the synergistic effect of the aforementioned stability gating and exit processing is to transform the seemingly conservative action of "reducing switching without stable benefits" into lower NPU layout recompilation costs and higher deployment consistency. This is the opposite of the greedy strategy of "submitting when the current score is the lowest" in conventional online optimization, and therefore cannot be directly predicted by conventional thinking in this field.

[0071] Step 600: Incorporate the residual basis into the model weights and maintain the whitelist of non-rotation space.

[0072] Specifically, to maintain the equivalence of model outputs, the system employs corresponding absorption methods for different structures: The embedding table is right-multiplied by the residual basis at the initial residual position, so that the embedding layer output is directly within the residual basis at the initial residual position; the final normalization scale is folded into the output layer preprocessing or output layer input layout to avoid additional inconsistencies between the residual basis and normalization; the input columns of the attention projection and the multilayer perceptron projection absorb the residual basis at the current residual position; the output rows of the attention output projection and the multilayer perceptron output projection absorb the residual basis at the target residual position; the final normalization is folded into the output layer, and the output layer input absorbs the residual basis at the last residual position; and during residual summation, the aforementioned permutation transformation is first performed on the main residual branch. Then, it is added to the output of the current sub-layer. For example, in a Transformer layer, the current residual state is... After the attention sublayer outputs, it enters the target residual state. Then the attention input projection is absorbed in the weight column layout. Attention output projection is absorbed in weighted row layout. The principal residual branch is first formed by through Switch to Then add it to the sub-layer output, so that the residual main branch and the sub-layer output are added under the same residual basis, maintaining the semantic consistency of residual summation.

[0073] More specifically, this step simultaneously maintains a whitelist of non-rotation spaces. That is, for query, key, and value projections, only the source residual basis is absorbed in the hidden dimensions of their inputs, while the output spaces, rotation position encoding spaces, and key-value cache spaces of the query, key, and value projections remain unrotated, as does the attention score space. Its purpose is to rotate only the hidden dimension input and output spaces carrying the residual flow, without affecting the internal structure of the attention head, the semantics of the head dimension encoded by the rotation position, or the structure of the key-value cache. This maintains the equivalent paths of the residual flow while avoiding disruption of the semantic consistency of the attention mechanism. It should be noted that for models containing grouped query attention, multi-query attention, linear attention, or special multilayer perceptron structures, only the input and output spaces of the hidden dimensions can be rotated according to their actual structure, while keeping the internal head, convolution, recursive state, or key-value state spaces unchanged. After the above processing, the floating-point outputs of the model before and after rotation can remain consistent within the numerical error range; for example, the cosine similarity is close to... The fact that the maximum absolute difference is on a relatively small order of magnitude demonstrates the feasibility of this equivalent path.

[0074] Step 700: Generate the substitution transformation and configure its runtime execution mode.

[0075] Specifically, after the channel permutation at each residual position is determined, the system generates permutation transformations for adjacent source residual positions and target residual positions according to Formula 5. This is stored as an array of integer indices or an equivalent layout descriptor. The permutation transformation is configured to allow runtime residual branches to perform channel rearrangement between adjacent residual positions using index permutations, column rearrangements, vector shuffling, DMA aggregate reads, or tensor layout remapping, without performing dense rotation matrix multiplication. More specifically, when the NPU supports vector shuffling, This can be accomplished using vector shuffling instructions; when the NPU supports DMA aggregated reads, This can be achieved by reading address order via DMA; when the compiler supports tensor layout remapping, The layout description that can be read as input to the next linear layer can be consumed without generating a separate rearrangement kernel; instead, it is directly consumed by the column layout description of the next matrix multiplication. When the above fusion conditions are not met, index permutation can also be explicitly performed. Accordingly, the system writes a fusion read flag or an explicit rearrangement flag to each permutation transformation and records the corresponding kernel identifier or execution method. Optionally, if the channel permutation at the next residual position has not yet been determined, the current permutation can be temporarily stored until it is determined before generating the corresponding permutation transformation.

[0076] Step 800: Perform low-bit quantization on the weights after absorbing the residual basis, configure the activation group quantization parameters, and generate the deployment package.

[0077] Specifically, the system performs low-bit weight quantization on the weights after absorbing the residual basis and configures the activation group quantization parameters to obtain the quantized weights, weight quantization scale, and activation quantization configuration. For example, this low-bit quantization can be applied to various bit widths and formats, such as four-bit weights and eight-bit activation, four-bit weights and four-bit activation, and can also be applied to block floating-point formats. Correspondingly, the shared transform can employ a global Hadamard transform or a block Hadamard transform. When using a block Hadamard transform, its transform granularity can be aligned with the quantization groups of the block floating-point format to reduce runtime and weight absorption costs. Optionally, the system can perform hardware-constrained grouping optimization after channel sorting. That is, it generates grouping allocation under the constraints of quantization group size, tile size and DMA alignment granularity supported by the NPU, so that channels within the same quantization group are physically continuous, and the rearrangement across quantization groups is described by permutation transformation. If a candidate group reduces the activation quantization error, but causes excessive explicit rearrangement cost, DMA aggregation read cost or core layout switching cost, it can be penalized by the NPU rearrangement cost term, or it can be rolled back to mean sort permutation or identity grouping.

[0078] Based on this, the system generates a deployment package, which is described in a field-based manner without relying on a structural diagram. Exemplarily, it includes at least: model identifier, number of layers, hidden dimension, rotation type, block size, number of residual positions, permutation table offset, permutation transformation table offset, quantization weight offset, scale table offset, online position summary offset, and overall checksum. Each residual position entry may include residual position number, layer number, position type, rotation type, permutation method, permutation index pointer, source residual base identifier, target residual base identifier, layout identifier, quantization group size, and position checksum. Each permutation transformation entry may include source residual position number, target residual position number, permutation transformation index pointer, transformation method, fusion read flag, explicit rearrangement flag, and transformation checksum. Furthermore, the deployment package may also include an online position summary, which records the selected permutation method, candidate score, number of scoring sample tokens, whether stable backoff is triggered, whether abnormal candidates exist, and NPU rearrangement cost summary for each residual position. The deployment package is used by the NPU to load and execute low-bit weighted activation quantization inference.

[0079] Step 900: Generate an equivalence certificate and configure NPU load verification and rollback.

[0080] Specifically, the system further generates an equivalence certificate within the deployment package to support consistency verification by the NPU when loading the deployment package. The equivalence certificate includes at least the permutation checksum at each residual location and the permutation transformation... The checksums, source residual base identifiers of each weight tensor after absorbing the residual base, target residual base identifiers and weight checksums, a non-rotation space whitelist, and floating-point equivalence checksums are all included. The non-rotation space whitelist covers the output space of queries, key-value projections, rotation position encoding space, key-value cache space, and attention score space. More specifically, when loading the deployment package and before executing inference, the NPU or a processor communicating with the NPU sequentially checks whether the number of residual positions matches the number of model layers, whether each permutation checksum, the source and target residual base identifiers of each permutation transformation, and the source and target residual base identifiers of each weight tensor are consistent. It also verifies whether the non-rotation space whitelist covers the aforementioned spaces and whether the transformation checksums match the target and target residual base identifiers. The consistency check verifies whether the quantized weights, scales, and layout identifiers match the current core configuration. It should be noted that if any check fails, the NPU can refuse to load the deployment package, fall back to a deployment package using identity permutation or mean-sort permutation, or trigger recompilation. This mechanism aims to detect inconsistencies between the weight layout and the residual state before loading in complex reparameterized deployments, thereby reducing the risk of implicit inference errors caused by inconsistencies between complex reparameterization, permutation transformations, and low-bit weight layouts.

[0081] Step 1000: The NPU loads the deployment package and performs low-bit weighted-activated quantization inference according to the residual basis state machine.

[0082] Specifically, after the NPU loads the quantized weights, scales, permutation tables, transformation tables, and equivalence certificates and passes the aforementioned loading verification, it performs inference according to the residual basis state machine: The embedding layer output enters the residual basis of the initial residual position; during the execution of each Transformer layer, the attention projection and multilayer perceptron projection read the reparameterized and quantized weights according to the current source residual basis identifier; the attention layer internally maintains the query, key, and value projection output space, rotation position encoding space, and key-value cache space without rotation; the output of the attention sublayer or multilayer perceptron sublayer enters the residual basis of the target residual position; before residual summation, the main residual branch performs index permutation, column rearrangement, vector shuffling, DMA aggregation read, or tensor layout remapping according to the corresponding permutation transformation. When the deployment package marks that the transformation can be fused to the input read of the next linear layer, the runtime may not generate an independent rearrangement kernel, but it can be directly consumed by the input read layout of the next linear layer; finally, the hidden state of the last residual position is finally normalized and combined with the output layer to obtain logits. Since the runtime residual transformation only involves channel index replacement or layout read order adjustment, the transformation complexity of a single token is approximately [missing information]. Compared to the approximately required dense rotation matrix multiplication for each residual position, By using multiplication and addition, this embodiment reduces runtime overhead while maintaining equivalent paths.

[0083] II. System Implementation Examples This embodiment also provides a large language model residual stream equivalent permutation rotation quantization compression and online sorting orchestration system for neural network processors, corresponding to the above method embodiments. Specifically, as Figure 2 As shown, the system includes a residual position identification and modeling module, a residual basis setting module, a weight absorption and whitelist protection module, a permutation transformation generation module, and a quantization compilation and deployment package generation module. The residual position identification and modeling module is used to identify multiple residual positions in a large language model, including the embedding output, attention residual summation, multilayer perceptron residual summation, and the final normalized or output layer input, and uses these multiple residual positions as multiple states of the residual basis state machine. The residual basis setting module is used to set residual bases for each of the multiple residual positions. The weight absorption and whitelist protection module is used to absorb the residual basis of the corresponding residual positions of the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer, respectively, and to keep the output space, rotation position encoding space, and key-value cache space of query, key, and value projections from being rotated; the permutation transformation generation module is used to... The system generates permutation transformations for adjacent source and target residual positions. The quantization compilation and deployment package generation module performs low-bit weight quantization on the weights after absorbing the residual basis, configures activation grouping quantization parameters, and generates a deployment package containing channel permutations, permutation transformations, quantization weights, weight quantization scales, activation quantization configurations, and NPU layout descriptions for the NPU to load and execute low-bit weight-activation quantization inference. In some embodiments, the system may further include an online sorting and orchestration module and a deployment verification module. The online sorting and orchestration module selects the channel permutation for the current residual position from the candidate permutation set during layer-by-layer quantization, based on the quantized residual activation, subsequent output error, logits relative entropy, or NPU rearrangement cost generated by the currently quantized prefix model. The deployment verification module performs pre-loading verification on the residual position, permutation transformation, source residual basis identifier, target residual basis identifier, weight checksum, and non-rotation space whitelist based on an equivalence certificate. The specific implementation methods, beneficial effects, and synergistic relationships between the above modules can be found in the corresponding descriptions in the method embodiments, and will not be repeated here. It should be noted that the above module division is only illustrative and can be merged or split as needed in actual implementation, as long as the corresponding functions can be achieved.

[0084] III. Examples of Electronic Devices and Storage Media This embodiment also provides an electronic device, which includes a processor, a memory, and a neural network processor, or includes a processor, a memory, and an interface communicatively connected to the neural network processor. The memory stores a computer program, which, when executed by the processor, implements the methods described in any of the foregoing method embodiments. The processor can be used to perform model parsing, residual basis setting, permutation search, online sorting and orchestration, quantization compilation, and deployment package generation. The neural network processor can be used to load the deployment package and perform low-bit weight-activation quantization inference. This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods described in any of the foregoing method embodiments. Those skilled in the art will understand that all or part of the processes of the above method embodiments can be implemented by computer program instructions and related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The memory and storage medium mentioned herein can include random access memory, read-only memory, flash memory, or other non-volatile storage media.

[0085] IV. Technical Effects of the Above Embodiments The above embodiments have the following technical effects: First, by modeling the residual flow of a large language model as a residual basis state machine, and setting different residual positions... The constrained residual basis allows each residual location to have position-level channel permutation degrees of freedom. While maintaining the model's floating-point output within the numerical error range, its adaptability to different layers and residual locations is superior to schemes using only a single fixed rotation for the entire model. Secondly, it achieves the desired results through shared orthogonal transformations. The properties of this property cause the transformation between adjacent residual positions to degenerate into For permutation transformations, the runtime residual branch does not require dense rotation matrix multiplication. Instead, channel rearrangement can be achieved through index permutation, column rearrangement, vector shuffling, DMA aggregate reads, or layout remapping. The overhead of a single token transformation is reduced to approximately [missing information]. Multiplication and addition reduced to approximation Index access, and the ability to fuse transformations to the next linear layer input reading when hardware supports it, helps to further reduce explicit data movement. Third, by absorbing the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer according to the source and target residual basis, and setting a whitelist of non-rotation spaces covering the query, key-value projection output space, rotation position encoding space, key-value cache space, and attention score space, it helps maintain consistency between the internal semantics of attention and the key-value cache structure, reducing the risk of residual flow reparameterization disrupting equivalent paths in the model. Fourth, through mean-sorting permutation, output-aware permutation, block-output-aware permutation, and online sorting and orchestration in layer-by-layer quantization, channel permutations can be selected based on residual activation magnitude, subsequent output error, logits relative entropy, and NPU rearrangement cost, helping to reduce errors caused by uneven dynamic range in grouped activation quantization; in particular, online sorting uses the quantized residual activation generated by the currently quantized prefix model, rather than just using full-precision offline statistics, which helps reduce the mismatch between static offline sorting and the true activation distribution after layer-by-layer quantization. Fifth, by employing an adaptive switching threshold derived from return volatility and rearrangement costs, cross-scoring block voting, optimization based on activation error and rearrangement costs when approximately tied, and a stability gating mechanism with default backoff for mean sorting, the system helps reduce unstable return permutation switching triggered by small sample score volatility, improving the consistency and reproducibility of the deployment package. Sixth, by utilizing a verifiable deployment package containing a permutation table, permutation transformation table, quantization weights, scaling table, NPU layout description, and equivalence certificate, along with a loading verification backoff mechanism, inconsistencies between the layout and residual states can be detected before loading and rejected, rolled back, or recompiled. This helps reduce the risk of implicit inference errors caused by layout inconsistencies during complex permutation rotation quantization compilation. Furthermore, the above scheme is compatible with various low bit widths and formats and is applicable to neural network processors and other processors that support low-bit matrix computation and channel rearrangement.

[0086] V. Supplementary Notes on Parameters, Boundary Conditions, and Deployment Verification To further explain the specific implementation methods of online sorting and orchestration, stability gating, weight absorption, and deployment verification, the following supplementary explanations are provided regarding the relevant parameters, calculation methods, and boundary cases.

[0087] In one implementation, the scoring samples consist of a portion of text from the calibration corpus that was not used for weight quantization fitting, and are divided into sequences, batches, or token blocks. Each rating block, among which It is a positive integer. When At that time, returns fluctuate Desirable And cross-scoring block voting conditions may not be enforced; when At that time, the comprehensive score of candidate replacements can be calculated separately for multiple scoring blocks, and whether to submit a candidate replacement is jointly determined by the average comprehensive score across scoring blocks, the profit fluctuation, and the voting results. (Minimum of sampled token count) The threshold can be determined based on the hidden dimension, quantization group size, and score sample size to ensure that each quantization group has a valid statistical sample. Approximate juxtaposition threshold. Voting threshold ,coefficient , and weighting coefficients , , , It can be pre-configured by the accuracy tolerance, stability requirements and NPU rearrangement cost constraints of the target deployment, and all of these can be written into the deployment configuration or online location summary.

[0088] In one implementation, candidate permutation Residual activation grouping quantization error under It can be calculated as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error. Quantized residual activation at each residual location By candidate permutation After rearrangement, we get The hidden dimension is divided into multiple quantization groups according to the activation quantization configuration. For each quantization group, pseudo-quantization is performed according to the corresponding integer format, block floating-point format, or shared scale format. The resulting pseudo-quantized dimension is then activated. and calculate in, Indicates the first The number of valid tokens collected at each residual location. Indicates the hidden dimension. This represents the Frobenius norm. For integer activation quantization using a shared scale, the scale of each quantization group can be determined by the maximum absolute value of activations within that group, the quantization quantization value, or a calibrated truncation threshold. For activation quantization using a block floating-point format, the shared exponent or shared scale of each quantization group can be determined according to the block floating-point rules supported by the target NPU.

[0089] In one implementation, candidate permutation Output error This can be obtained by temporarily applying candidate permutations and running subsequent linear layers, sub-cells, or residual blocks. Specifically, the reference output without applying candidate permutations can be denoted as... Let the output after applying candidate permutations and performing the corresponding temporary rearrangements be denoted as And using normalized mean square error, mean absolute error, or cosine distance as... Candidate permutations The small sample output logits relative entropy This can be obtained by comparing the reference logits distribution with the logits distribution under the candidate permutation, where the relative entropy can be calculated after softmax normalization of the logits distribution. The above... , and It can be computed on the same scoring sample or the same scoring block to ensure consistent comparison conditions between candidate permutations.

[0090] In one implementation, NPU rearrangement cost The cost can be determined from an offline cost table, compiler estimates, or runtime benchmark results. If the permutation transformation corresponding to a candidate permutation can be directly consumed by the input read layout of the next linear layer, then... Choose the lowest cost option; if explicit index replacement, DMA aggregate reads, vector shuffling, tensor layout remapping, or additional core switching is required, increase the cost based on the number of tokens being rearranged, hidden dimensions, memory access bandwidth, core startup overhead, whether tile alignment is broken, and whether it causes discontinuous access to channels within the quantization group. If a candidate replacement results in an invalid NPU layout, or its rearrangement cost exceeds deployment cost constraints, the candidate replacement can be discarded, or the cost can be increased. This will penalize it in the overall score.

[0091] In one implementation, when using row vector representation conventions, if the original residual hidden state is... In its first The representation of each residual basis is as follows: For the source residual location to the target residual position The linear mapping, the weights after absorbing the residual basis, can be expressed as: in, Indicates the weight before absorption. This represents the weights after absorbing the residual basis. For query, key-value projection, the input columns absorb the source residual basis, while the output space remains within the whitelist of the non-rotation space. The corresponding weights can be represented as... , , For attention output projection, its output rows can absorb the target residual basis; for multilayer perceptron gated projection and top projection, its input columns can absorb the source residual basis; for multilayer perceptron bottom projection, its output rows can absorb the target residual basis; for embedding tables, the embedding output can be right-multiplied by the residual basis at the initial residual position; for output layers, the output layer input can absorb the residual basis at the last residual position. If the actual model weights use a transposed layout in storage, the above relationships are adjusted according to the corresponding transpose method, but the correspondence between the source residual basis identifier, the target residual basis identifier, and the non-rotation space whitelist remains unchanged.

[0092] In one implementation, the non-rotation space whitelist includes at least the output space of query, key, and value projection, the rotation position encoding space, the key-value cache space, and the attention score space. Candidate permutations undergo validity checks before submission. If a candidate permutation would propagate to the aforementioned non-rotation space, or alter the internal dimensions of the attention head, the rotation position encoding dimension, the key-value cache layout, or the attention score calculation space, then the candidate permutation is deemed invalid and the system reverts to the default permutation or the identity permutation. For grouped query attention, multi-query attention, linear attention, hybrid expert structures, or special multilayer perceptron structures, residual basis absorption can be performed only on the hidden dimension input space or output space carrying the residual flow, while keeping the internal head dimension, key-value state, routing, or gating state unrotated.

[0093] In one implementation, deterministic rollback is performed when the overall score of a candidate permutation cannot be calculated, uncorrectable non-numerical values ​​appear in the scoring samples, the number of valid scoring blocks is insufficient, the hidden dimension cannot be divided by the quantized group size or NPU tile size, the candidate permutation violates NPU layout constraints, the source residual base identifier of the candidate permutation is inconsistent with the current residual state, or the candidate permutation touches the non-rotation space whitelist. The deterministic rollback includes at least one of the following: maintaining the default permutation, rolling back to the identity permutation, rolling back to the mean-sorted permutation, removing invalid candidates, padding or processing the tail blocks separately, regenerating the permutation transformation, or triggering recompilation. This allows online sorting and orchestration to still generate deployable channel permutations and deployment packages even in boundary cases.

[0094] In one implementation, the floating-point equivalence checksum in the equivalence certificate may include the maximum absolute difference, mean absolute difference, cosine similarity, or top-k consistency of small sample logits; the post-quantization checksum may include activation group quantization error, relative entropy of weight-only quantization, relative entropy of weight-activation pseudo-quantization, perplexity, or task log-likelihood difference. When loading the deployment package, the NPU or a processor communicating with the NPU sequentially checks the model layer number, number of residual locations, permutation table checksum, permutation transformation table checksum, source residual basis identifier, target residual basis identifier, weight checksum, quantization scale table, NPU layout identifier, and non-rotation space whitelist. If the checksum fails at the permutation table, transformation checksum, source residual basis identifier, or target residual basis identifier level, loading is rejected and an error code is output; if a pre-stored deployment package with identity permutation or mean-sorted permutation versions exists, it reverts to that version; if the deployment environment supports recompilation, the failure reason is recorded and a regeneration of the deployment package is triggered.

[0095] The specific implementation methods described above, without departing from the core concept of this application, can have various equivalent variations and substitutions. For example, shared transformations can employ global Hadamard, block Hadamard, Walsh-Hadamard, random orthogonal, approximately orthogonal, or hardware-defined reversible hybrid transformations; channel permutations can be generated according to residual position, layer, layer family, attention or multilayer perceptron type, head group, or NPU tile; online sorting can occur before layer-by-layer quantization, after each layer quantization, after each attention or multilayer perceptron subunit quantization, or after each residual block; permutation transformations can be explicitly performed using index permutations, or implicitly implemented by changing the next layer weight layout, input DMA address, register read order, or tensor layout descriptor. All of these should be covered within the scope of protection of this application.

[0096] It should be noted that in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if a reference is made to performing an action based on an element, it means performing the action at least based on that element, including two cases: performing the action only based on that element, and performing the action based on that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0097] All references to this application are considered to be incorporated integrally into the disclosure of this application so as to serve as a basis for modification if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application. It should be noted that in the application documents of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In the application documents of this patent, if it is mentioned that an action is performed based on a certain element, it means that the action is performed at least based on that element, including two cases: performing the action based solely on that element, and performing the action based on that element and other elements. Expressions such as multiple, repeated, and various include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0098] All documents mentioned in this application are considered to be incorporated in their entirety into the disclosure of this application so that they can serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A method for equivalent permutation, rotation, quantization, compression, and online sorting of residual streams in large language models for neural network processors (NPUs), characterized in that, include: Identify multiple residual positions in a large language model, including the embedding output, attention residual summation, multilayer perceptron residual summation, and the final normalized or output layer input, and use these multiple residual positions as multiple states of a residual basis state machine; set residual bases for each of these multiple residual positions. ,in, For the first The residual basis at each residual location For all residual locations to be shared and satisfy The Hadamard transform, block Hadamard transform, or hardware-friendly orthogonal transform, For the first Channel permutation at each residual position It is the identity matrix. Number the residual positions; at least the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer are respectively absorbed by the residual basis of their corresponding residual positions, while keeping the output space, rotation position encoding space, and key-value cache space of query, key, and value projections unrotated; based on Generate permutation transformations between adjacent source residual locations and target residual locations. ,in, This is a permutation transformation from the source residual location to the adjacent target residual location. for transpose, The channel permutation is performed at the target residual position; low-bit weight quantization is performed on the weights after absorbing the residual basis, and the activation group quantization parameters are configured to obtain the quantization weights, weight quantization scales and activation quantization configurations, and a deployment package containing the channel permutation, the permutation transformation, the quantization weights, the weight quantization scales, the activation quantization configurations and the NPU layout description is generated, so that the NPU can load the deployment package and perform low-bit weight-activation quantization inference.

2. The method according to claim 1, characterized in that, The step of using the multiple residual positions as multiple states of the residual base state machine includes: binding residual base description information to each state, wherein the residual base description information includes residual position number, layer number, position type, rotation type, index of the channel permutation, quantization group size, NPU layout identifier, source residual base identifier, and target residual base identifier; and binding transformation description information to the residual transformation between adjacent states, wherein the transformation description information includes the permutation transformation. The index, and a flag indicating whether the permutation transformation is to be fused to the next linear layer input read; the residual basis description information and the transformation description information are used for the generation of the deployment package.

3. The method according to claim 1, characterized in that, The step of absorbing the residual basis at the corresponding residual positions includes: multiplying the embedding table right-wise by the residual basis at the initial residual position, so that the embedding layer output is in the residual basis at the initial residual position; absorbing the residual basis at the current residual position by the input column of the attention projection and the multilayer perceptron projection; absorbing the residual basis at the target residual position by the output row of the attention output projection and the multilayer perceptron output projection; folding the final normalization into the output layer, and absorbing the residual basis at the last residual position by the input of the output layer; and performing the permutation transformation on the main residual branch first when summing the residuals. Then add it to the output of the current sub-layer.

4. The method according to claim 1, characterized in that, The channel replacement Generated by mean sorting, including: residual activation calculation at the residual position followed by the shared transformation. Activation after ,in, Let be the residual activation matrix at this residual location. For the shared transformation Activation after; Calculate the average amplitude for each channel in, For the number of sampled tokens, for The Middle The token, the Elements of each channel, For the first Average amplitude of each channel; based on the average amplitude of each channel The channels are sorted from largest to smallest to generate a channel permutation for the residual position, denoted as mean sort permutation. The mean sort permutation is used to improve the dynamic range grouping of grouped activation quantization.

5. The method according to claim 1, characterized in that, The low-bit weight quantization is performed in a layer-by-layer quantization manner, and the channel permutation of at least some of the residual positions is determined online during the layer-by-layer quantization process, including: after the quantization of each attention subunit or multilayer perceptron subunit is completed, the currently quantized prefix model is run on the scoring sample to collect the quantized residual activation of the next residual position. A candidate permutation set is generated for the next residual position. The candidate permutation set Including identity permutation and activation of the quantized residuals using the mean sorting method as described in claim 4. The generated permutation includes at least two of the following: mean-sorted permutation, permutation generated by sorting by the 99th percentile of channel amplitude, permutation generated by sorting by the 50th percentile of channel amplitude, output-sensing permutation, and block-output-sensing permutation; for the candidate permutation set. Each candidate permutation in Calculate the overall score in, For candidate permutations, For candidate permutations The overall score, In candidate permutation Grouped quantization error with lower residual activation In candidate permutation The output error of subsequent linear layers, sub-units, or residual blocks. In candidate permutation The relative entropy of the small sample output logits. For candidate permutations The corresponding NPU rearrangement cost, , , , All are non-negative weight coefficients; From the candidate permutation set, according to the selection criteria... Select one candidate permutation as the channel permutation for the next residual position. This is then used to generate the permutation transformation between the residual position and its adjacent residual positions. .

6. The method according to claim 5, characterized in that, The selection criteria are stability gating criteria, and the step of selecting a candidate permutation based on the selection criteria includes: using the mean-sorted permutation as the default permutation. Calculate the comprehensive score for each candidate permutation on one or more scoring blocks, and take the average comprehensive score across the scoring blocks. And the candidate permutation with the smallest average comprehensive score is denoted as ; calculate Relative default substitution Benefits Return volatility across scoring blocks and adaptive switching threshold in, This is the default replacement. The candidate permutation with the lowest average overall score is selected. For candidate permutations The average composite score across the scoring blocks, for Compared to the benefits of the default replacement, For the fluctuation of returns across scoring blocks, For adaptive switching threshold, , All coefficients are non-negative; a corresponding candidate permutation is submitted as the channel permutation for the next residual position when any of the following conditions are met: the gain is greater than the adaptive switching threshold, i.e. Then submit ;or exist The winning percentage in each scoring block is no less than the voting threshold. Then submit ,in For the number of scoring blocks, This is the voting threshold; or the difference in relative entropy between the output logits does not exceed an approximately parallel threshold. From the approximate set of parallel groups, candidate permutations with low residual activation grouping quantization error and NPU rearrangement cost not exceeding the deployment cost constraint are selected. Approximate parallel threshold; otherwise, retain the default permutation. .

7. The method according to claim 1, characterized in that, The deployment package also includes an equivalence certificate, which includes at least: a permutation checksum for each residual location, and each permutation transformation. The checksum, source residual basis identifiers of each weight tensor after absorbing the residual basis, target residual basis identifiers and weight checksums, non-rotation space whitelist, and floating-point equivalence check summary are provided. The non-rotation space whitelist covers the output space of query, key-value projection, rotation position encoding space, key-value cache space, and attention score space. The equivalence certificate is used by the NPU to check whether the number of residual positions matches the number of model layers, the checksums of each permutation, the source residual position identifiers and target residual position identifiers of each permutation transformation, and the source residual basis identifiers and target residual basis identifiers of each weight tensor are consistent when loading the deployment package and before performing inference. If any check fails, the NPU will refuse to load the deployment package, fall back to the deployment package using identity permutation or mean sort permutation, or trigger recompilation.

8. A system for equivalent permutation, rotation, quantization, compression, and online sorting of large language model residual streams for neural network processors (NPUs), characterized in that, include: The residual location identification and modeling module is used to identify multiple residual locations in the large language model, including the embedding output, attention residual summation, multilayer perceptron residual summation, and the final normalized or output layer input, and to treat these multiple residual locations as multiple states of the residual basis state machine; the residual basis setting module is used to set residual bases for each of the multiple residual locations. ,in, For the first The residual basis at each residual location For all residual locations to be shared and satisfy The Hadamard transform, block Hadamard transform, or hardware-friendly orthogonal transform, For the first Channel permutation at each residual position The identity matrix is ​​defined as follows: A weight absorption and whitelist protection module is used to absorb the residual basis of the corresponding residual positions of the weights of the embedding layer, attention projection, attention output projection, multilayer perceptron projection, and output layer, while maintaining the output space, rotation position encoding space, and key-value cache space of the query, key, and value projections unrotated; A permutation transformation generation module is used to... Generate permutation transformations between adjacent source residual locations and target residual locations. ,in, for transpose, The module performs channel permutation for the target residual position; a quantization compilation and deployment package generation module is used to perform low-bit weight quantization on the weights after absorbing the residual basis, configure the activation group quantization parameters, obtain the quantization weights, weight quantization scales and activation quantization configurations, and generate a deployment package containing the channel permutation, the permutation transformation, the quantization weights, the weight quantization scales, the activation quantization configurations and an NPU layout description, so that the NPU can load the deployment package and perform low-bit weight-activation quantization inference.

9. An electronic device comprising a processor, a memory, and a neural network processor (NPU), or comprising a processor, a memory, and an interface communicatively connected to the NPU, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images