System and method for trimming rotated and outlier-free large language model to achieve efficient weight-activated quantization

Abstract

A computing device includes at least one processor, one or more non-transitory computer-readable storage media, and a system for trimming a large language model at a low bit weight-activation quantization. The computing device also includes a graphics processing unit (GPU), a neural processing unit (NPU), or a tensor processing unit (TPU). And a hardware interface module of the system is used for loading the low-bit model representation form from the storage module and transmitting the low-bit model representation form to the GPU, the NPU or the TPU so as to perform inference execution operation.

Description

Technical Field

[0001] This invention relates to model compression and quantization techniques for large-scale neural networks; more specifically, it relates to a system and method for fine-tuning a rotated, outlier-free large language model (LLM) to achieve effective weight-activation quantization under low-bit conditions, thereby improving accuracy and reducing quantization error. Background Technology

[0002] Large language models (LLMs), such as GPT-4 and LLaMA, have achieved remarkable success across various tasks. However, the ever-increasing model size and training costs have spurred the development of methods for model compression and parameter-efficient fine-tuning (PEFT). Low-rank adaptation (LoRA) has become a widely adopted PEFT technique that improves fine-tuning efficiency by updating a finite set of parameters.

[0003] In recent years, quantization techniques that convert high-accuracy parameters to low-bit formats, such as INT4, have been combined with LoRA methods. Existing LoRA quantization schemes can save memory costs during fine-tuning, and some schemes can even reduce inference costs by directly generating quantized LLMs. However, these methods only perform weight quantization, and research on LoRA weight-activation quantization is currently insufficient. By quantizing weights and activations with low bits, runtime GPU memory can be further saved, and computationally intensive matrix multiplication operations can be accelerated. It has been found that in LLMs, using 4-bit or 6-bit weight-activation quantization with LoRA fine-tuning still leads to a significant drop in accuracy because outliers exist in the weight and activation distributions, which cause the quantization range to expand and increase quantization error.

[0004] Existing methods in post-training quantization research have attempted to address the outlier problem by mixing accuracy subgroups or transferring outliers from activations to weights. In recent years, it has been demonstrated that rotating the weight matrix of an LLM can effectively eliminate activation outliers while maintaining computational invariance. However, these methods all address the problem from a post-training perspective, neglecting the fact that outliers can arise and their distribution can change during pre-training and fine-tuning.

[0005] Therefore, a system and method are needed to achieve effective low-bit weighted activation quantization by eliminating outliers during the fine-tuning of large language models, while maintaining robustness throughout the fine-tuning process. Summary of the Invention

[0006] The purpose of this invention is to provide a system and method to address the aforementioned deficiencies and unmet needs in the prior art.

[0007] This invention proposes a method called Rotated Outlier-Free Low-Rank Adaptation (RoLoRA), a LoRA-based approach for achieving efficient weight-activation quantization. RoLoRA utilizes rotation to eliminate outliers and proposes a rotation-aware fine-tuning method to preserve the outlier-free characteristics of the rotated LLM. Experimental results show that RoLoRA can consistently improve the convergence of low-bit LoRA and the robustness of post-training quantization under weight-activation settings. RoLoRA has been evaluated on LLaMA2-7B / 13B and LLaMA3-8B models. In the commonsense reasoning task, RoLoRA achieves a maximum absolute accuracy improvement of 29.5% compared to the LoRA baseline on LLaMA2-13B with 4-bit weight-activation quantization. The effectiveness of RoLoRA has also been demonstrated on large multimodal models, including LLaVA-1.5-7B.

[0008] According to a first aspect of the invention, a system is provided for fine-tuning a large language model under low-bit weight-activation quantization. The system includes a storage module, a tensor structuring module, a model initialization module, a rotation configuration module, a rotation-aware fine-tuning module, a quantization module, and a hardware interface module. The storage module stores a structured representation associated with a model object, including a weight matrix, activation signals, rotation parameters, and intermediate or final representations. The tensor structuring module receives the structured representation of the model object from the storage module and processes it into a structured digital format suitable for computer processing, wherein the structured digital format is stored in the storage module. The model initialization module retrieves the structured digital format from the storage module and modifies the internal normalization components of the model object to maintain computational invariance during rotation. The rotation configuration module retrieves the modified model object from the storage module, applies an orthogonal rotation transformation to the weight matrix and activation signals of the model object, and stores the rotated model object in the storage module. The rotation-aware fine-tuning module retrieves the rotated model object and applies low-rank adaptation to it according to a selected fine-tuning strategy. This low-rank adaptation involves inserting and training a low-rank matrix while maintaining the basic weights unchanged. The quantization module retrieves the fine-tuned model object and quantizes the weights and activation signals using at least one quantization strategy to generate a low-bit model representation stored in the storage module. The hardware interface module exports the low-bit model representation stored in the storage module to the inference system for deployment.

[0009] According to a second aspect of the invention, a computing device is provided. The computing device includes at least one processor; one or more non-transitory computer-readable storage media; and further includes a system as described above for fine-tuning a large language model under low-bit weighted activation quantization. This includes a graphics processing unit (GPU), a neural processing unit (NPU), or a tensor processing unit (TPU). The system's hardware interface module is configured to load a low-bit model representation from the storage module and transfer this model representation to the GPU, NPU, or TPU for inference execution operations.

[0010] According to a third aspect of the invention, a method is provided for fine-tuning a large language model under low-bit weight-activation quantization. The method includes the following steps: receiving a structured representation associated with a model object, including a weight matrix, activation signals, and rotation parameters, via a tensor structuring module; processing the structured representation into a computer-processable structured digital format via the tensor structuring module and storing the structured digital format in a storage module; modifying the internal normalization components of the model object via a model initialization module to maintain computational invariance during rotation; applying an orthogonal rotation transformation to the weight matrix and activation signals of the model object via a rotation configuration module to generate a rotated model object; applying low-rank adaptation to the rotated model object according to a selected fine-tuning strategy via a rotation-aware fine-tuning module, wherein the low-rank adaptation includes inserting and training a low-rank matrix while maintaining the basic weights unchanged; quantizing the weights and activations using at least one quantization strategy via a quantization module to generate a low-bit model representation; and exporting the low-bit model representation to an inference system for deployment via a hardware interface module.

[0011] With the above configuration, the technical solution provided by this invention can achieve the following advantages:

[0012] RoLoRA was proposed, and the feasibility of combining rotation and quantization settings in LoRA was demonstrated.

[0013] • RoLoRA enables robust weight-activation quantization of finely tuned LLMs, especially in low-bit settings such as W4A4 and W6A6.

[0014] • The effectiveness of RoLoRA on the LLaMA series (2-7B, 2-13B, 3-8B) was verified, covering different quantizers (RTN / GPTQ), bit widths (W4A4 / W6A6), and benchmark tests (zero-shot commonsense reasoning, MMLU). Furthermore, the applicability of RoLoRA to large-scale multimodal models (LMMs) was also demonstrated. Attached Figure Description

[0015] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, wherein:

[0016] Figure 1 The activation distribution before and after rotation is shown;

[0017] Figure 2 Table 1 shows a comparison of the visual instruction outputs generated by different models;

[0018] Figure 3 An overall overview diagram of the RoLoRA method proposed according to an embodiment of the present invention is shown;

[0019] Figure 4 Two schemes for performing rotation-aware fine-tuning are demonstrated, including: (a) LAR; (b) LBR;

[0020] Figure 5 The approximate error plots of SVD under different LoRA-rotational integration schemes are shown.

[0021] Figure 6 A visualization of activation kurtosis and quantization error is shown, including the training dynamics of activation kurtosis and the fine-tuned quantization error, for comparison of LoRA and RoLoRA.

[0022] Figure 7 Table 2 shows a comparison of the average accuracy of the LLaMA series on seven ZCSR tasks and the MMLU benchmark.

[0023] Figure 8 Table 3 is presented, which compares the average accuracy of different low-bit LoRA methods on LLaMA2-7B on zero-shot commonsense reasoning tasks and MMLU benchmarks.

[0024] Figure 9 Table 4 shows a comparison of the W4A4 quantization performance of LLaVA-1.5-7B on LLaVA-Bench;

[0025] Figure 10 Table 5 shows the compatibility with DoRA on LLaMA2-7B;

[0026] Figure 11 Table 6 is shown, which presents ablation experiments regarding the timing of rotational application;

[0027] Figure 12 Table 7 shows the ablation experiments with rotational application locations;

[0028] Figure 13 Table 8 shows how to apply LoRA in ablation experiments;

[0029] Figure 14 Table 9 shows a comparison of the fine-tuning costs of LLaMA2-7B when the batch size is 8 on the GPU.

[0030] Figure 15 The average accuracy of W4A4 LLaMA2-7B at different ranks r is shown in the figure.

[0031] Figure 16 Table 10 shows a complete accuracy comparison of the LLaMA series on zero-shot commonsense reasoning tasks;

[0032] Figure 17 Table 11 shows the complete accuracy of the LLaMA series on the MMLU benchmark;

[0033] Figure 18 Table 12 shows the detailed hyperparameters for different LLM and LMM fine-tuning;

[0034] Figure 19A and Figure 19B The final activation distribution of the model after fine-tuning with RoLoRA and LoRA is shown; and

[0035] Figure 20 An architectural block diagram of a system according to an embodiment of the present invention is shown. Detailed Implementation

[0036] In the following description, a system and method, etc., will be illustrated by way of preferred examples for fine-tuning a large, outlier-free language model after rotation to achieve effective weight-activation quantization. Those skilled in the art will understand that various modifications, including additions and / or substitutions, can be made without departing from the scope and spirit of the invention. Specific details may be omitted to avoid obscuring the invention; however, this disclosure is intended to enable those skilled in the art to practice the teachings herein without extensive experimentation.

[0037] This disclosure proposes a method for outlier removal using rotation in a LoRA fine-tuning setup and investigates a method that dynamically integrates rotation with LoRA to maintain outlier-free properties and enhance the optimization of weight-activation quantization. Inspired by this goal, this application proposes a Rotated and Outlier-Free Low-Rank Adaptive (RoLoRA) framework, which first applies intra-block and inter-block rotations to a pre-trained LLM and then utilizes rotation-aware fine-tuning to generate an outlier-free fine-tuned LLM, such as... Figure 1 As shown, the activation distribution before and after rotation is illustrated. This allows us to explore the optimal rotation-aware fine-tuning scheme based on approximate error analysis.

[0038] Furthermore, extensive experimental results demonstrate the effectiveness of RoLoRA across various LLMs, tasks, and quantization settings. Compared to LoRA, RoLoRA achieves a performance improvement of up to 14.6 points in MMLU benchmarks with 4-bit quantization (W4A4) of weights and activations. Compared to existing low-bit LoRA methods, RoLoRA outperforms the previous state-of-the-art (SOTA) IR-QLoRA by up to 6.0 points in MMLU benchmarks. The proposed RoLoRA framework is more efficient than LoRA under the same settings, with negligible fine-tuning overhead. RoLoRA also improves the quantization robustness of large multimodal models (LMMs) such as LLaVA, and it has been observed that even after W4A4 quantization, the understanding of multimodalities is largely preserved. Figure 2 As shown in Table 1, where Figure 2 Table 1 presents the original language model operation flow, and therefore is expressed in the original text.

[0039] To make the solution provided by this invention easier to understand, relevant information will be provided below.

[0040] 1: Related work

[0041] Quantization. Quantization methods are powerful tools for improving training and inference efficiency. The core idea is to replace full-accuracy weights and activations with low-accuracy representations. Most existing LLM quantization techniques fall into the post-training quantization (PTQ) category, meaning the model can be quantized directly without extensive training. Among these LLM PTQ methods, most only employ weight quantization, while few explore weight-activation quantization. Compared to focusing solely on weight quantization, simultaneously quantizing weights and activations enables low-accuracy multiply-accumulation (MAC) units. The key challenge lies in the fact that outliers in activations can lead to high quantization errors. This paper focuses on weight-activation quantization placed within a LoRA pipeline.

[0042] LoRA. Considering the ever-growing scale of LLM development, the computationally impractical full-parameter fine-tuning framework has led to the introduction of Parameter-Efficient Fine-Tuning (PEFT) methods, which aim to reduce costs by training a relatively small subset of parameters. Low-Rank Adaptation (LoRA) is the most widely adopted PEFT method due to its flexibility and efficiency. In recent years, various LoRA variants have emerged to improve the effectiveness and efficiency of LoRA. Combining LoRA with quantization is also a promising direction, as quantization can further save GPU memory in LoRA fine-tuning. Furthermore, various improved QLoRA methods have been proposed to further reduce information distortion in low-bit fine-tuning. However, these methods only quantize the weights during fine-tuning to reduce memory consumption.

[0043] This invention proposes a quantized LoRA scheme that explicitly considers robustness under weighted activation quantization.

[0044] 2: Initial setup and motivation

[0045] 2.1. Low-Rank Adaptation (LoRA)

[0046] For the pre-trained weight matrix LoRA can use low-rank decomposition to update weights. The model is constructed and represented as AB, where and Let r represent two low-rank matrices, where r << min(d,k). Therefore, the fine-tuned weight W′ can be expressed as: W′=W0+ΔW=W0+AB, (1)

[0047] Among them, W0 remains unchanged during fine-tuning, while A and B are trainable parameters. Furthermore, based on formula (1), the learned ΔW can be merged with the pre-trained weights W0, and W′ can be obtained before deployment. Since both W′ and W0 are located at... Within the dimension of , LoRA and its related variants do not introduce any additional delay during inference compared to the original model.

[0048] 2.2. Outliers in Transformers

[0049] Starting with small-scale transformer (LLM) models such as BERT and ViT, researchers have discovered outliers in the weight and activation distributions. Outliers have also been observed in various studies. Figure 1As shown on the left, activation outliers can be distributed by channel. While these outliers enhance the transformer's representation capabilities, they also present challenges to quantization. Previous solutions to this outlier problem in quantization have mainly fallen into three categories: (1) isolating these outliers into subgroups with higher accuracy, such as LLM.int8, Atom, QuiK, and AdaDim. However, grouping and mixing accuracy incur additional overhead. (2) shifting the challenging task of quantization from activation to weights, such as SmoothQuant and Omni-Quant. However, these methods negatively impact the robustness of weights in quantization and fail in the W4A4 scenario. (3) rotating the activation or weight matrix to remove outliers, such as QuaRot and SpinQuant. Among these methods, recent rotation-based solutions have shown superior effectiveness. However, previous rotation-based methods primarily addressed the outlier problem from a post-training perspective and have not yet been developed or advanced in the PEFT setting.

[0050] This leads to a technical problem: can the outlier-free property of the rotated LLM be preserved and effectively utilized during PEFT? This disclosure addresses this problem and further investigates rotation-based fine-tuning strategies that preserve these rotated properties, thereby enhancing the robustness of weight activation quantization.

[0051] 2.3. Using rotation to eliminate outliers

[0052] Define the rotation matrix R as an orthogonal matrix, and |R| = 1, where R follows the property RR of orthogonal matrices. T =I. If RR T If the elements of I are +1 or -1, then it will become a Hadamard matrix H. By definition, a matrix H with 2k elements can be efficiently generated based on the Hadamard transform (also known as the Walsh-Hadamard transform, an example of a generalized Fourier transform):

[0053] in Represents the Kronecker product. Rotation transformations rely on efficient operation because of the d×d Hadamard matrix H. d Performing a matrix-vector product on X will require... This operation. Previous research has shown that rotating the weights of the prenorm transformer can maintain computational consistency and further reduce outliers in the weight and activation distributions. Specifically, the product of the weight matrix and the rotation matrix statistically mixes weights of different sizes together to form a distribution that is closer to a Gaussian distribution, resulting in activation jobs with fewer outliers and easier quantification.

[0054] 3: Method

[0055] Inspired by the activation outlier problem and rotation-based solutions, this paper proposes Rotated outlier-free Low-Rank Adaptation (RoLoRA). The RoLoRA framework initially performs intra-block and inter-block rotations on a pre-trained LLM, and then performs rotation-aware fine-tuning on the rotated LLM to preserve the outlier-free property, thereby producing a fine-tuned LLM that is highly robust to weight-activation quantization.

[0056] 3.1. Perform rotation

[0057] Before performing rotation fine-tuning, the model needs to be modified to maintain computational invariance before and after the rotation. First, any scaling operations in the regularization module need to be eliminated. For the LLaMA series, this can be achieved by absorbing the RMSNorm scaling parameter α into the weight matrix immediately after the RMSNorm layer.

[0058] Then, inter-block rotations are performed to eliminate outliers in inter-block activations. Figure 3 A general overview diagram of the RoLoRA method proposed according to an embodiment of the present invention is shown. Figure 3 As shown, the weight matrix in LLM is divided into two categories: left-hand weights, including W in the self-attention module. q W k W ( And W in the feedforward network module up and W +,-. (corresponding to) Figure 3 W in u and W + The right-hand weights, including W in the self-attention module; / W in the feedforward network module 0 / 12 Different rotation strategies can be used for these two sets of weights:

[0059] Where the rotation R is a randomly generated Hadamard matrix. Since the input X is also rotated before the embedding layer, i.e., X←XR. -1The output Y is also rotated after lm_head (Y←RY), so the final output of the model will be exactly the same as the original model. To avoid overflow during rotation, FP16 weights are converted to FP64 and then converted back after multiplication. The weight accuracy conversion only occurs once at the start of rotation merging, and the accuracy of the rotated weights will maintain FP16 accuracy during fine-tuning and inference. Since the accuracy during inference is always low bit (W4A4 / W6A6), no additional conversion overhead is incurred during actual inference. These rotation jobs can be performed before any training and inference, demonstrating that no additional overhead is incurred even after merging into the original weights.

[0060] Rotation applied directly to weights can effectively reduce outliers in inter-block activations; this can be called inter-block rotation (BBR). Figure 1 The effects of applying BBR are demonstrated, resulting in a smoother and more decentralized activation distribution. However, another challenge remains: activations in these modules are still affected by outliers, which is particularly prevalent in FFNs discussed in previous studies. Directly applying rotational operations similar to BBR is not feasible due to the nonlinear operations (e.g., Swiglu) in FFNs. To address this issue, the activation input is fed into W... 0 / 12 Previously, an online rotation node was employed. This online rotation was implemented after a fast Hadamard kernel, which can be viewed as a dynamically rotated activation layer. Due to the use of a fast Hadamard kernel on CUDA, this online rotation operation is highly efficient, and the additional overhead during training and inference is negligible. Such an operation is called In-Block Rotation (IBR). It is worth noting that IBR can also be applied to self-attention modules, but due to… Figure 12 The experiments shown in Table 7 indicate that this rotational operation did not result in any performance improvement.

[0061] 3.2. Rotation-sensing fine-tuning

[0062] After performing BBR and IBR, inter-block and intra-block activation outliers are eliminated. This property can be used to reduce quantization error during QLoRA training, resulting in more accurate gradient estimation and smoother fine-tuning. However, existing research shows that outliers may change their distribution or reappear during fine-tuning and pre-training. This presents a new challenge in dynamically incorporating rotation operations into LoRA to effectively maintain its outlier-free characteristic. To design an optimal rotation-aware fine-tuning scheme, the approximation difficulty during rotation is first analyzed. It is assumed that the optimal weight distribution for a specific downstream task is W. *It can be approximated by combining the LoRA weights and the pre-trained weights W0. The optimization of LoRA fine-tuning can be expressed as:

[0063] Among them ||·|| F The Frobenious norm is used to represent the LoRA module. To insert the LoRA module into the rotated model, two rotation-aware fine-tuning schemes are proposed: the LoRA After Rotation (LAR) framework and the LoRA Before Rotation (LBR) framework, as follows: Figure 4 As shown. Figure 4 Two schemes for performing rotation-aware fine-tuning are demonstrated, including: (a) LAR; (b) LBR.

[0064] In LAR, the rotation matrix is ​​first merged with the pre-trained weights, and W is approximated using R1W0+AB. * For LBR, first merge the LoRA weights, then rotate them to obtain R1(W0+AB). Assume the optimal weights are the complete fine-tuning result W. FT The optimization results of these two schemes can be expressed as follows:

[0065] Therefore, the final optimization results are completely different. For the approximate objective O... @<5 Perform singular value decomposition (SVD). Where O = USV T The LoRA weights of rank r can be initialized using the first r-dimensional principal singular values ​​and vectors, respectively. and We can obtain:

[0066] To simulate LoRA under two rotation schemes, the approximation error was evaluated under different rank choices r. In the experiments, a pre-trained LLaMA2-7B model was used as W0, and a fully fine-tuned model on the Alpaca dataset was used as W. FT ,like Figure 5 As shown, the results illustrate the SVD approximation errors under different LoRA-rotation ensemble schemes. The results show that LAR outperforms LBR in the low-rank setting with a lower approximation error, indicating that LAR is a better rotation-aware fine-tuning design. The superior approximation also demonstrates that after a two-stage merging of the rotation matrix and LoRA weights, the final weights still maintain the characteristic of being free of outliers, which is further verified in the ablation experiments presented in Section 4.5.

[0067] Under LAR settings, due to the adoption of the optimal rotation-aware fine-tuning scheme, the LLM fine-tuning process can effectively preserve the characteristic of no outliers, such as... Figure 6 As shown, the model is divided into three parts: left, middle, and right. The left part represents the training dynamics of the average kurtosis of activations; the middle part represents the distribution of activation kurtosis in each layer of the final model after LoRA and RoLoRA fine-tuning; and the right part represents the cumulative quantization error of the W4A4 GPTQ function across all layers in the final model after LoRA and RoLoRA fine-tuning.

[0068] 4: Experimental Section

[0069] 4.1. Settings

[0070] Model, LoRA, quantizer. Experimental models include LLaMA2-7B / 13B and LLaMA3-8B. The training process can be implemented based on the settings provided by LLaMA-Factory. The fine-tuning dataset is Alpaca, containing 52K samples. The weight PTQ method uses the benchmark Round-To-Nearest (RTN) and the widely used GPTQ, and the activation quantizer for all experiments is RTN. Channel-wise symmetric quantization is used for weights, and tensor-wise quantization is used for activations.

[0071] Tasks. This paper uses EleutherAI evaluation to validate the RoLoRA presented in this paper on seven zero-shot commonsense reasoning tasks. These tasks include BoolQ, PIQA, HellaSwag, WinoGrande, ARCeasy, ARC-challenge, and OBQA. Furthermore, this paper provides accuracy on large-scale multi-task language understanding (MMLU) benchmarks.

[0072] Baseline. This paper considers two experimental setups. The first is to use RoLoRA for FP16 fine-tuning, comparing the W4A4 and W6A6 quantization results with LoRA. The second is to use 4-bit weighted quantization for RoLoRA fine-tuning, called QRoLoRA, and compare the performance of W4A4 with other low-bit LoRA methods (including QLoRA, LoftQ, and IR-LoRA).

[0073] 4.2. Main Results

[0074] First, RoLoRA is compared with LoRA in FP16 fine-tuning, then weight-activation PTQ is applied to the fine-tuned LLM. To ensure a fair comparison, both RoLoRA and LoRA use the same settings (rank, period, learning rate, etc.). Figure 7As shown in Table 2, RoLoRA enhances the quantization robustness of the LLaMA series across various quantization settings in zero-shot commonsense reasoning and MMLU benchmark tests. Specifically, for the low-bit W4A4 setting, RoLoRA outperforms LoRA on ZCSR and MMLU, achieving absolute accuracies of 29.5% and 14.6%, respectively. Although MMLU consists of multiple-choice questions with four options, a relative accuracy below 25% is still significant because some low-bit quantized LLMs cannot even be instructed to choose from the four options. In this invention, the proposed method better preserves inference performance, ensuring that LLaMAs still answer questions as instructed in most cases, rather than generating meaningless symbols. Furthermore, RoLoRA makes near-lossless W6A6 quantization of the LLaMA series highly probable in multiple tasks.

[0075] RoLoRA was further compared with QLoRA and several baseline methods (including LoftQ and IR-QLoRA) on 4-bit fine-tuning, and then W4A4 PTQ was applied to LLaMA2-7B for low-bit fine-tuning. Figure 8 Table 3 details the performance of the seven commonsense reasoning tasks and four MMLU subtasks. It can be seen that RoLoRA consistently improves the performance of the quantized model while using the same quantizer. Specifically, for W4A4 GPTQ, RoLoRA achieves a 20.5% higher average accuracy on the commonsense reasoning task compared to QLoRA. In the FP16 and 4-bit fine-tuning experiments, RoLoRA demonstrates a superior overall performance improvement on GPTQ-quantized LLMs. Based on these observations, the following conclusion can be drawn: RoLoRA preserves outlier-free activations during fine-tuning, while GPTQ primarily reduces the quantization error of the weights, rather than the quantization error of the activations.

[0076] 4.3. Fine-tuning of visual instructions

[0077] The effectiveness of RoLoRA was validated on a visual instruction fine-tuning task using LLaVA-1.5-7B, which consists of a language model Vicuna-7B and a visual encoder CLIP ViT-L-336px. LLaVA-1.5-7B was fine-tuned on LLaVAInstruct-150K3. Quantization was applied only to the language model, and LLaVA was evaluated on the GPT-4 LLaVA-Bench (COCO) benchmark using the quantized Vicuna model and a full-precision visual encoder. Figure 9Table 4 shows the relative scores for dialogue, detailed description, and complex reasoning. The results demonstrate that RoLoRA helps improve quantification robustness and better preserves multimodal capabilities during the PTQ, resulting in an overall score improvement of up to 18.9 points. For example... Figure 2 Table 1 in the table also provides examples of detailed description tasks for a given image. The W4A4 LoRA model can only provide a coarse surface description of the image, while the proposed W4A4 RoLoRA model can fully describe the details, such as ingredients and containers.

[0078] 4.4. Compatibility with other LoRA variants

[0079] The proposed method can be further validated on the representative LoRA variant, DoRA. DoRA decomposes pre-trained weights into amplitude and orientation components and fine-tunes both. The rotation-aware fine-tuning stage described in this paper can also follow the same scheme, which is called RoDoRA. Figure 10 As shown in Table 5, RoDoRA, using RTN and GPTQ as quantizers, achieves accuracy improvements of 18.3% and 26.7% on W4A4 LLaMA2-7B, respectively. RoDoRA's results also outperform RoLoRA, demonstrating that the proposed method is compatible with state-of-the-art LoRA variants and has the potential to further improve weight-activation quantization performance.

[0080] 4.5. Ablation Research and Analysis

[0081] When to apply rotation? Unlike the rotation-aware fine-tuning (RAF) approach, which rotates the LLM before LoRA fine-tuning, rotation can be applied directly to the already fine-tuned LoRA model. This possible paradigm of LoRA → rotation → PTQ is called post-training rotation. In the evaluation of post-training rotation, the same training settings as RoLoRA in the LLaMA series are used. Figure 11 Table 6 lists the performance of W4A4 GPTQ on seven zero-shot commonsense reasoning tasks. The results show that applying rotation before LoRA can consistently enhance the quantization robustness of fine-tuned LLM.

[0082] Where should rotation be applied? Figure 3 The provided process introduces two types of rotations: inter-block rotations applied to all weight matrices and intra-block rotations applied to down_proj in FFN. As described in Section 3.1, a similar head-based IBR R3 self-attention mechanism can also be applied. Figure 3 In the middle, R3 will W v and W o Rotate as follows: The selection of these rotating targets has been validated on the LLaMA2-7B W4A4PTQ, and the results are as follows: Figure 12 As shown in Table 7, the results indicate that applying R1 and R2 simultaneously and only is the optimal choice for eliminating outliers.

[0083] How to apply LoRA? In Section 3.2, two rotation-aware fine-tuning schemes were proposed: the rotated LoRA (LAR) framework and the unrotated LoRA (LBR) framework, as follows: Figure 4 As shown. Based on approximation error analysis, LAR is a better paradigm compared to full fine-tuning. Figure 13 Table 8 quantitatively compares the W4A4 quantization performance and LLaMA2-7B fine-tuning effect of the two schemes. The LAR scheme shows better effectiveness, which is related to... Figure 5 The approximate analysis given is also consistent.

[0084] Outliers. The primary motivation for using RoLoRA during LLM fine-tuning is to maintain the outlier-free characteristic. To quantitatively verify the effectiveness of outlier elimination, activation kurtosis was used. The kurtosis is used to measure the presence or absence of outliers, where μ and σ are the empirical mean and standard deviation of the activation distribution, respectively. Generally, the larger the kurtosis value, the thicker the tail of the activation distribution, and the greater the likelihood of outliers. Figure 6 The dynamic changes in kurtosis are demonstrated when fine-tuning is performed using LoRA and RoLoRA. In the early stages of training, rotation operations effectively suppress outliers in activation. Rotation-aware fine-tuning preserves this optimal property. After fine-tuning using RoLoRA, the results are as follows: Figure 5 As shown, the kurtosis κ of all layers is significantly reduced, which further reduces the quantization error compared to the LoRA baseline. Figure 19A and Figure 19B It also shows a comparison of the activation distribution maps of each layer of RoLoRA and LoRA.

[0085] LoRA rank settings. During the fine-tuning of LLaMA2-7B, the robustness of LoRA and RoLoRA under different rank settings r∈{4,8,16,32,64} was also explored, and the models were evaluated on a zero-shot commonsense reasoning task. The optimal rank settings for RoLoRA and LoRA are 16 and 32, respectively. The lower optimal rank indicates that the proposed RoLoRA has the potential to save trainable parameters. Overall, regardless of the rank setting, RoLoRA consistently outperforms LoRA, demonstrating its robustness.

[0086] Efficiency. Regarding the fine-tuning efficiency of RoLoRA, the additional training time is solely due to the online rotation operation (i.e., Figure 3R2 in the equation is caused by another rotation operation (i.e., Figure 3 The R1 values ​​can be directly incorporated into the original weights. This requires only one additional matrix multiplication, and theoretically, the resulting increase in rotation parameters is negligible. Figure 14 Table 9 shows the fine-tuning cost of RoLoRA compared to LoRA under the same settings (rank r = 16, batch size 8, 3 training epochs). RoLoRA significantly improves the performance of W4A4 quantized LLaMA2-7B with extremely low overhead. Figure 15 As shown, it demonstrates the average accuracy of W4A4 LLaMA2-7B with different ranks r, fine-tuned using RoLoRA.

[0087] 5: Appendix Materials

[0088] This section provides supplementary support to the above discussion.

[0089] 5.1. Detailed Evaluation Results

[0090] Figure 16 Table 10 and Figure 17 Table 11 presents the complete evaluation results for the zero-shot commonsense reasoning task and the MMLU benchmark test. The evaluation was conducted using EleutherAI; in the given evaluation, "acc_norm" is used if an accuracy metric exists; otherwise, "acc" is used.

[0091] 5.2. Hyperparameters used for reproduction

[0092] exist Figure 18 Table 12 provides detailed hyperparameters for reproducing RoLoRA and LoRA results. No hyperparameter search was performed to improve accuracy.

[0093] 5.3. Activation Distribution Visualization

[0094] exist Figure 19A and Figure 19B The activation magnitudes of LLaMA2-7B after fine-tuning using LoRA and RoLoRA are visualized, and the final activation distributions of the fine-tuned models generated using RoLoRA and LoRA are shown. The output activation of q_proj is selected from layers with indices 0, 1, 6, 11, 16, 21, 26, and 31. The visualization results show that a large number of outliers exist in the LoRA-tuned model, but these outliers are significantly eliminated in the RoLoRA corresponding model.

[0095] Based on the aforementioned technical architecture and related fine-tuning and quantization workflows, this disclosure provides a specific computer implementation system for processing, rotating, fine-tuning, and quantizing large-scale neural network models. Specifically, a system is provided comprising multiple interconnected modules that can operate on the high-dimensional weights and activation tensors generated by LLM. The operations include matrix rotation using Hadamard transform, insertion of low-rank adaptive modules, rank-specific optimization based on SVD, and low-bit quantization of weights and activations. During these operations or processing, substantial computational resources are required, which cannot be accomplished through human thought processes or pen-and-paper calculations, especially the dynamic integration of rotation-aware fine-tuning and the memory persistence of multi-stage model objects. Therefore, the claimed system relies on computer technology and provides a practical framework designed to improve the deployment efficiency and robustness of quantized LLMs in practical applications, such as language inference and multimodal instruction tracing.

[0096] Figure 20 An architectural block diagram of a system 100 according to an embodiment of the present invention is shown. System 100 can be used to fine-tune a rotated, outlier-free LLM under low bit-weighted activation quantization, and includes a set of interconnect modules implemented on computer hardware. Specifically, system 100 includes a storage module 102, a tensor construction module 110, a model initialization module 120, a rotation configuration module 130, a rotation-aware fine-tuning module 140, a quantization module 150, and an evaluation and analysis module 160.

[0097] Storage module 102 may include one or more types of non-transitory computer-readable storage media, such as dynamic random access memory (DRAM), static random access memory (SRAM), and high-bandwidth memory (HBM). Storage module 102 is used to store intermediate and final model objects, including structured tensors, rotation transformation weights, fine-tuning parameters, and quantization representations, in a format accessible to other modules of system 100.

[0098] Tensor structure module 110 receives mathematical representations, such as weight matrices, activation signals, and rotation parameters, and processes them into a structured digital format suitable for computer processing. The structuring process includes encoding the high-dimensional tensor into a memory-aligned array, adjusting bit precision (e.g., FP16 to FP4), and preparing the data layout for efficient computation. The structured tensor data can be output to and stored in storage module 102, making it accessible to downstream modules.

[0099] The model initialization module 120 retrieves structured model data from the storage module 102 and modifies its internal architecture to maintain computational invariance during subsequent rotations. The initialization process includes removing scaling operations from the normalization layers and merging scaling parameters into adjacent weight matrices. The initialized model object is then stored back into the storage module 102 and used as input to the rotation configuration module 130.

[0100] The rotation configuration module 130 retrieves an initialized model object from the storage module 102 and applies an orthogonal rotation transformation to suppress outliers in the weights and activations. The initialized model object contains a set of structured weight matrices and activation flow definitions corresponding to different layers of a large language model, including attention projection weights, feedforward projection weights, and residual activation paths. After retrieval, the rotation configuration module 130 can parse the model object to identify specific sets of parameters that need to be transformed. Specifically, the rotation configuration module 130 can extract: (1) the attention projection weight matrix and the feedforward projection weight matrix (e.g., output projection, up projection, and down projection), and (2) the activation paths of the input nonlinear transformation (e.g., the SwiGLU activation function).

[0101] Then, rotation configuration module 130 generates orthogonal rotation matrices (e.g., Hadamard matrices or block diagonal variants) and further applies them: (i) to perform inter-block rotations on the identified projected weights by multiplying the original weight matrix by the rotation matrix and its transpose; and (ii) to perform intra-block rotations on the intermediate activation tensors by inserting a fast Hadamard transform after normalization. These application operations modify the internal tensor paths of the model object to suppress statistical outliers in the parameter and activation spaces while maintaining functional equivalence. Subsequently, the updated rotated model object is written back to storage module 102 for access by rotation-aware fine-tuning module 140.

[0102] The rotation-aware fine-tuning module 140 retrieves the rotated model object from the storage module 102 and performs parameter-efficient fine-tuning by inserting and updating low-rank adaptive components. The rotated model object, for example, contains rotation transformation weight matrices across attention and feedforward layers. After retrieval, the rotation-aware fine-tuning module 140 parses the model object to identify suitable insertion points for the low-rank modules and selects a fine-tuning strategy based on pre-configured settings, such as a rotated LoRA (LAR) framework or a pre-rotation LoRA (LBR) framework.

[0103] In one embodiment, the rotation-aware fine-tuning module 140 operates using a LAR strategy, where the low-rank module is inserted after the rotation is applied, such that the rotated weights are used as basis parameters. In this configuration, low-rank updates are computed and merged directly in the rotated weight space, thus preserving the outlier-free property introduced by the rotation configuration module 130. Therefore, the fundamental matrix for low-rank decomposition is the rotation alignment matrix.

[0104] In another embodiment, the rotation-aware fine-tuning module 140 operates using an LBR strategy, where the low-rank module is inserted before the rotation process. In this configuration, the fine-tuning operation is performed in the original, unrotated weight space, followed by transformation of the resulting adjusted weights through rotation. This configuration is compatible with existing LoRA implementations but does not retain the full statistical advantage of early outlier suppression during training.

[0105] In both strategies, the rotation-aware fine-tuning module 140 can apply singular value decomposition (SVD) to approximate the full fine-tuning update with low-rank components and initialize the trainable matrix accordingly. During the fine-tuning phase, only the low-rank matrix can be updated, while the underlying (rotated or unrotated) weights remain fixed / frozen.

[0106] The rotation-aware fine-tuning module 140 transforms the rotated model object into a task-adaptive model object. This model object initially consists of a weight matrix that suppresses outliers but is task-neutral, and it contains frozen base parameters and learned low-rank increments. Before fine-tuning, the model object only contains rotation transformation weights optimized for quantization robustness, but not task-specific adjustments. After inserting and training the low-rank components, the model object includes additional parameter matrices that encode task-adaptive knowledge while preserving the statistical advantages of rotation. Therefore, the fine-tuned model object exhibits higher inference accuracy under low-bit conditions and is stored as a deployable representation in storage module 102.

[0107] After training is complete, the rotation-aware fine-tuning module 140 can merge the base components and low-rank components to generate an updated model object, which is then stored in the storage module 102 for subsequent quantization by the quantization module 150.

[0108] Quantization module 150 retrieves the fine-tuned model object from storage module 102 and performs quantization on the weight parameters and activation tensors. The fine-tuned model object includes a combination of frozen base weights and learned low-rank update matrices, as well as defined activation paths corresponding to each model layer. After retrieval, quantization module 150 identifies and extracts relevant tensor components from the model object, including the weight matrix for each layer and representative activation ranges. Quantization module 150 applies channel-wise symmetric quantization to the weight matrix, aligning the quantization granularity with each output channel, and applies tensor-wise quantization to the activation tensors based on their statistical range characteristics. The quantization strategy can be selected from a configurable backend (e.g., Round-To-Nearest RTN or GPTQ), depending on deployment constraints. After quantization, the resulting model object contains low-bit encoded representations of the weights and activations, which can be stored in storage module 102 in a deployable format for inference or evaluation.

[0109] After quantization module 150 completes the quantization process, system 100 generates a fully structured model object residing in memory, which can be directly deployed in downstream inference systems. The model object contains low-bit encoded weights and activations, meets compatibility requirements with hardware accelerators and software inference frameworks, and represents the final output of the system's fine-tuning and quantization pipeline. Therefore, quantization module 150 is designed as the final stage of the model transformation process required to generate an executable, low-bit, large language model. In other words, the final output of system 100 is a quantized large language model stored in storage module 102, containing parameters for rotation, fine-tuning, and low-bit quantization, which are used for direct deployment in inference tasks.

[0110] In some embodiments, the evaluation analysis module 160 can be used to further validate and optimize the performance of the quantization model object before or during deployment. The evaluation analysis module 160 can serve as an auxiliary enhancement component, enabling quality assurance, performance benchmarking, and adaptive improvement of the quantization strategy without altering the functional readiness of the deployed model. The evaluation analysis module 160 retrieves the quantization model from the storage module 102 and evaluates its performance on benchmarking tasks, including but not limited to commonsense reasoning, multi-task learning, and visual instruction tracking. The evaluation analysis module 160 calculates metrics such as layer-by-layer activation kurtosis and quantization error to assess the robustness of outlier suppression. Evaluation results can be selectively written back to the storage module 102 for auditing, visualization, or further optimization.

[0111] In some embodiments, system 100 may further include a compatibility interface module 170 for adapting system 100 to LoRA variants, such as DoRA. The compatibility interface module 170 retrieves the decomposed weighted representation from storage module 102, performs variable-aware reintegration of the direction and amplitude components, and accordingly invokes rotation-aware fine-tuning module 140.

[0112] In some embodiments, system 100 further includes a digital conversion interface module 180 configured to convert symbolic or continuous mathematical structures (e.g., floating-point tensors, rotation matrices, and low-rank projected vectors) into a hardware-compatible, memory-aligned digital format. The digital conversion interface module 180 cooperates with the tensor structuring module 110 and includes accuracy conversion logic, alignment padding circuitry, and hardware buffer control components. Through the digital conversion interface module 180, the output can be a set of structured tensor blocks directly mapped to a non-transitory computer-readable storage area, allowing subsequent modules to operate on physical, byte-addressable objects, thus ensuring that the execution flow does not remain at the level of mathematical abstraction.

[0113] In some embodiments, system 100 further includes a hardware interface module 190 for interfacing memory-stored quantized model objects (e.g., model objects stored in storage module 102) with inference hardware (e.g., graphics processing unit (GPU), neural processing unit (NPU), or tensor processing unit (TPU)). Hardware interface module 190 includes driver-level logic components for model loading, tensor layout verification, and compatibility signaling for deployment to hardware accelerators.

[0114] System 100 can be implemented using computer equipment, including at least one processor and one or more non-transitory computer-readable storage media (e.g., DRAM, flash memory, or NVRAM). Each module of System 100 can be operated via executable logic stored in memory and executed by the processor. Intermediate and final model artifacts (including rotation weights, fine-tuning parameters, and quantization tensor objects) can be stored as structured digital artifacts in storage module 102. These generated artifacts are neither symbolic nor abstract; rather, they exist in an addressable binary format for downstream machine inference.

[0115] Through the coordinated interaction of the modules of System 100, symbolic or mathematical model descriptions are transformed into structured digital model artifacts, which can undergo physical tensor transformations, adaptive fine-tuning, and deployable low-bit quantization. These transformations include tensor rotation, LoRA merging, SVD initialization, and quantization, and are implemented as physical data transformations in memory, thereby generating machine-readable, memory-residing model objects containing low-bit encoded matrices and activation representations. Each module performs read and write operations through a storable module 102, ensuring that all intermediate and final states are concretely represented in memory. The generated model objects can be directly used in language model inference engines and hardware accelerators. Therefore, System 100 is not geared towards thought processes or mathematical abstractions, but rather provides a practical, computer-implemented solution for generating deployable LLM components and improving their efficiency and quantization robustness.

[0116] A feasible execution flow is proposed below. System 100 can be initialized first on a computing device containing at least one processor and memory, such as a server-class machine with GPU or NPU inference accelerator access capabilities. A pre-trained large language model (e.g., an LLaMA-based architecture) can be loaded into system 100 in its raw, uncompressed form. Tensor structuring module 110 receives the model's high-precision weight matrix, layer activation configuration, and optional rotation control parameters, and encodes these into a structured, memory-aligned digital format. These tensors are then stored in storage module 102 and made accessible to downstream modules.

[0117] The model initialization module 120 retrieves structured model tensors and performs internal architecture adjustments. For example, it examines and modifies normalization layers to remove floating-point scaling operations and absorbs scaling factors into adjacent weight matrices, thereby maintaining computational invariance under rotation conditions. The initialized model object, indicating it is ready for transformation, is then saved to the storage module 102.

[0118] Subsequently, the rotation configuration module 130 accesses the model object and applies an orthogonal rotation transformation using the generated Hadamard matrix or block diagonal matrix. Projection weights associated with the attention layer and feedforward layer are extracted and multiplied by the rotation matrix and its transpose to apply inter-block rotation; simultaneously, intra-block rotation is performed on specific points of the intermediate activation tensor after normalization and before nonlinearity. After this stage, the resulting object is a rotated model object with suppressed statistical outliers, which is then stored again in the storage module 102.

[0119] Next, the rotation-aware fine-tuning module 140 retrieves the rotated model and inserts a low-rank adaptive structure according to a selected fine-tuning strategy. For example, in LAR mode, a rank-constrained trainable matrix is ​​directly applied to the rotated weights. In LBR mode, fine-tuning is performed in the unrotated space, and then the resulting adaptive weights are rotated. In both cases, SVD is used to initialize the low-rank matrix. Fine-tuning is performed using a task-specific dataset (e.g., instruction tuning), and only the inserted low-rank components are updated during training. The resulting fine-tuned model objects are merged and also stably stored in the storage module 102.

[0120] After fine-tuning, the quantization module 150 retrieves the model object and performs channel-by-channel symmetric quantization on the weights and tensor-by-tensor quantization on the activations. The choice of quantization strategy (e.g., RTN or GPTQ) may depend on deployment constraints, such as inference latency or hardware targets. The output of this stage is a model object residing in memory, containing low-bit encoded weights and activation tensors suitable for inference.

[0121] Furthermore, at this stage, system 100 has generated a complete and executable LLM component. The model object used has been configured for deployment and can be exported or loaded into the inference runtime via hardware interface module 190. Evaluation and analysis module 160 can be selectively invoked to benchmark the performance of the quantized model on specific tasks, such as language inference or multimodal instruction tracing. Output metrics such as quantization error and activation distribution kurtosis can be recorded for future adjustments or auditing.

[0122] In some embodiments, system 100 may be integrated into a computing device for performing low-bit inference based on a fine-tuned large language model. The computing device may include at least one processor and one or more non-transitory computer-readable storage media for storing executable logic and model building blocks used by system 100. System 100 can run on the computing device to perform model building, rotation, fine-tuning, and quantization as described above. The computing device also includes at least one hardware accelerator selected from a GPU, NPU, or TPU. After quantization, the hardware interface module of system 100 can retrieve the low-bit model representation from the memory module and transfer the model representation to the designated hardware accelerator. This configuration enables the computing device to use the optimized and quantized large language model generated by system 100 to perform efficient inference tasks.

[0123] As described above, this invention provides the RoLoRA framework, designed to be combined with LoRA for weight-activation quantization. RoLoRA introduces rotation to eliminate outliers in the activation distribution and, combined with rotation-aware fine-tuning, maintains the outlier-free property throughout training. Theoretical analysis and empirical evaluation both support the integration of rotation into LoRA. RoLoRA improves the performance of large language models like W4A4 and W6A6 on various tasks without increasing training costs. Furthermore, RoLoRA demonstrates its applicability in visual instruction adjustment scenarios.

[0124] The functional units and modules of the apparatus and methods according to the embodiments disclosed herein can be implemented using computing devices, computer processors, or electronic circuits, including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of this disclosure. Those skilled in the art of software or electronics can readily write computer instructions or software code that execute in computing devices, computer processors, or programmable logic devices based on the teachings of this disclosure.

[0125] All or part of the methods according to the embodiments can be performed in one or more computing devices, including server computers, personal computers, laptops, and mobile computing devices (such as smartphones and tablets).

[0126] Embodiments may include computer storage media, transient and non-transient memory devices storing computer instructions or software code, which can be used to program or configure computing devices, computer processors, or electronic circuits to perform any of the processes of this invention. Storage media, transient and non-transient memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs, magneto-optical disks, ROMs, RAMs, flash memory devices, or any medium or device suitable for storing instructions, code, and / or data.

[0127] Each functional unit and module according to the various embodiments can also be implemented in a distributed computing environment and / or cloud computing environment, wherein all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0128] The above description of the present invention is provided for illustrative purposes. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0129] These embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling those skilled in the art to understand the various embodiments of the invention and the various modifications suitable for the intended particular use.

Claims

1. A system for fine-tuning a large language model under low bit weight-activation quantization, the system comprising: A storage module is used to store a structured representation associated with a model object, the structured representation including a weight matrix, activation signals, rotation parameters, and intermediate or final representations; The tensor structuring module is used to receive the structured representation of the model object from the storage module and process the structured representation into a structured digital format suitable for computer processing, wherein the structured digital format is stored in the storage module; A model initialization module is used to retrieve the structured digital format from the storage module and modify the internal normalization components of the model object to maintain computational invariance during rotation; A rotation configuration module is used to retrieve the modified model object from the storage module, apply an orthogonal rotation transformation to the weight matrix and activation signal of the model object, and store the rotated model object in the storage module; A rotation-aware fine-tuning module is used to retrieve the rotated model object and apply low-rank adaptation to the model object according to a selected fine-tuning strategy, wherein the low-rank adaptation includes inserting and training a low-rank matrix while keeping the basic weights unchanged. A quantization module is used to retrieve the fine-tuned model object and quantize the weights and activations using at least one quantization strategy to generate a low-bit model representation stored in the storage module. as well as A hardware interface module is used to export the low-bit model representation stored in the storage module to the inference system for deployment.

2. The system according to claim 1, wherein the orthogonal rotation transformation includes at least one of the Hadamard matrix or block diagonal matrix applied to the attention mechanism and the feedforward weight matrix.

3. The system according to claim 1, wherein the rotation configuration module is further configured to apply inter-block rotation to the projection weight matrix after the normalization layer and before the nonlinear function, and to apply intra-block rotation to the activation tensor.

4. The system according to claim 1, wherein the rotation-aware fine-tuning module is further configured to operate in LoRA (LoRA After Rotation) mode by inserting a low-rank matrix after rotation and training the low-rank matrix in the weight space after rotation.

5. The system according to claim 1, wherein the rotation-sensing fine-tuning module is further configured to operate in LoRA (LoRABefore Rotation; LBR) mode by applying the low-rank adaptation before rotation and then transforming the adapted weights by rotation.

6. The system according to claim 1, wherein the quantization module is further configured to perform channel-by-channel symmetric quantization on the weights and tensor-by-tensor quantization on the activations.

7. The system according to claim 1, further comprising: The evaluation and analysis module is used to retrieve the low-bit model representation and calculate evaluation metrics with activation kurtosis and quantization error.

8. The system according to claim 1, further comprising: The digital conversion interface module is used to convert floating-point tensors, rotation matrices, and low-rank vectors into memory-aligned digital formats compatible with hardware execution.

9. The system according to claim 1, further comprising: A compatibility interface module is provided to adapt the system to LoRA variants by reintegrating the decomposed direction and magnitude weight components into the fine-tuning process.

10. The system of claim 1, wherein the storage module comprises one or more non-transitory computer-readable storage media selected from at least the group consisting of: dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, and solid-state drive (SSD).

11. A computer device, comprising: At least one processor; One or more non-transitory computer-readable storage media; The system of claim 1 is used for fine-tuning large language models under low bit weight-activation quantization; and Graphics processing unit (GPU), neural processing unit (NPU), or tensor processing unit (TPU); The hardware interface module of the system is used to load the low-bit model representation from the storage module and transmit the low-bit model representation to the graphics processing unit, the neural processing unit, or the tensor processing unit for inference execution.

12. A method for fine-tuning a large language model under low bit weight-activation quantization, comprising: The tensor structuring module receives a structured representation associated with a model object, which includes a weight matrix, activation signals, and rotation parameters. The tensor structuring module processes the structured representation into a structured digital format suitable for computer processing, and stores the structured digital format in the storage module. The model initialization module modifies the internal normalization components of the model object to maintain computational invariance during rotation. The rotation configuration module applies an orthogonal rotation transformation to the weight matrix and activation signal of the model object to generate the rotated model object. The rotation-aware fine-tuning module applies low-rank adaptation to the rotated model object according to the selected fine-tuning strategy. The low-rank adaptation includes inserting and training a low-rank matrix while keeping the basic weights unchanged. The quantization module quantizes the weights and activations using at least one quantization strategy to generate a low-bit model representation. as well as The low-bit model representation is exported to the inference system for deployment via a hardware interface module.

13. The method of claim 12, wherein applying the orthogonal rotation transformation comprises applying at least one of the Hadamard matrix or block diagonal matrix to the attention mechanism and the feedforward weight matrix.

14. The method of claim 12, wherein applying the orthogonal rotation transformation comprises: Apply inter-block rotation to the projection weight matrix; as well as After the normalization layer and before the nonlinear function, apply an in-block rotation to the activation tensor.

15. The method of claim 12, wherein applying the low-rank adaptation includes operating in a LoRA (LoRAAfter Rotation; LAR) mode, which is achieved by inserting a low-rank matrix after rotation and training the low-rank matrix in the rotated weight space.

16. The method of claim 12, wherein applying the low-rank adaptation includes operating in LoRA (LoRABefore Rotation; LBR) mode, which involves applying low-rank adaptation before rotation and then adjusting the weights by rotation transformation.

17. The method of claim 12, wherein performing the quantization comprises: Perform channel-by-channel symmetric quantization on the weights; as well as Quantize each activation per tensor.

18. The method of claim 12, further comprising: The low-bit model representation is evaluated using a benchmarking task, and evaluation metrics including activation kurtosis and quantization error are calculated.

19. The method of claim 12, further comprising: Convert floating-point tensors, rotation matrices, and low-rank vectors into memory-aligned number formats compatible with hardware execution.

20. The method of claim 12, further comprising: The model object is adapted to the LoRA variant by reintegrating the decomposed direction and magnitude weight components into the fine-tuning process.

Images