Low-rank adaptive parameter selective updating method and system based on activation variance awareness

CN122655893APending Publication Date: 2026-08-28BEIJING LANGUAGE AND CULTURE UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611166362.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-03
Publication Date
2026-08-28

AI Technical Summary

Technical Problem

[0007]为了解决现有技术中基于统一参数更新策略的低秩适配微调方法在训练过程中未区分不同子模块对企业私有知识的激活响应差异、参数更新路径缺乏精细化控制、存在大量冗余梯度计算,导致计算资源浪费、训练效率低下的硬件执行效能的技术问题,本发明实施例提供了基于激活方差感知的低秩适配参数选择性更新方法及系统

Benefits of technology

[0017]本发明实施例提供的技术方案带来的有益效果至少包括:

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122655893A_ABST
    Figure CN122655893A_ABST
Patent Text Reader

Abstract

The application relates to the technical field of parameter efficient fine-tuning (PEFT) and large language model, and particularly relates to a low-rank adaptive parameter selective updating method and system based on activation variance perception. l The overall change range of the output activation of each sub-module of the layer is depicted under the distribution of enterprise private knowledge documents; the activation variance estimation values of each sub-module on the corresponding input sample are arranged in descending order, a threshold value is determined by calculating a quantile, and subset division is performed; the LoRA adapter is uniformly initialized for all sub-modules, and a selective back propagation constraint strategy is used to make the model concentrate the computing resources into the proportion parameter space which is most sensitive to the change of the enterprise private knowledge. The application can finely control the parameter updating process in a data-driven manner without changing the overall structure of the original pre-training model and introducing additional inference delay, and has good engineering realizability and universality across models and tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of parameter-efficient fine-tuning PEFT and large language model technology, and particularly to a method and system for selective updating of low-rank fitting parameters based on activation variance awareness. Background Technology

[0002] With the rise of the Transformer architecture, the parameter size of large-scale pre-trained language models has grown to hundreds of billions or even trillions. These models, pre-trained on massive amounts of data, exhibit powerful generalization capabilities and have become the core foundation of fields such as natural language processing. However, when adapting such large models to specific downstream tasks, traditional full-parameter fine-tuning faces problems such as high GPU memory consumption, high storage costs, and long training times. To address these issues, Parameter Efficient Fine-Tuning (PEFT) techniques have emerged. Among them, Low-Rank Adaptation (LoRA), due to its characteristic of freezing the original weights and training only a small number of low-rank matrices, has become one of the most widely used methods, significantly reducing the number of fine-tuned parameters and GPU memory usage.

[0003] Despite its great success, LoRA still has several limitations: First, in practical applications, high rank is often set to ensure performance, resulting in LoRA adapters containing a large number of redundant parameters and failing to fully exploit the potential of parameter efficiency; second, different layers or sub-modules (such as attention projection and feedforward network projection) have significant differences in sensitivity to specific tasks, but traditional LoRA uses a uniform training strategy for all target modules, ignoring this difference in importance; in addition, LoRA is prone to overfitting during training, leading to performance degradation.

[0004] In recent years, methods that utilize activation statistics to guide model pruning and quantization (such as SparseGPT, Wanda, and AWQ) have been proposed. These methods assess the importance of weights by analyzing the distribution of activation values ​​during forward propagation, offering greater accuracy compared to traditional weight-based methods. However, these methods primarily focus on the magnitude or norm information of activation values, evaluating the "static importance" of weights, which fails to reflect the response characteristics of submodules under different samples or input perturbations.

[0005] Improvements to LoRA, such as AdaLoRA, DoRA, and PrunedLoRA, have enhanced performance through adaptive parameter allocation, weight decomposition, or structured pruning. However, they all rely on gradients or second-order information for importance assessment, involving complex auxiliary computations (such as SVD decomposition and Hessian approximation). This leads to increased training time and higher GPU memory usage, and they cannot predict module importance at low cost before training. These drawbacks are particularly pronounced in resource-constrained scenarios, and they still cannot effectively reduce GPU memory usage and redundant gradient computations during training.

[0006] Therefore, existing technologies lack a method to assess the dynamic sensitivity of submodules to enterprise private knowledge with only a small amount of forward propagation data before fine-tuning at low cost. This makes it impossible to achieve fine-grained control over parameter update paths, resulting in a large amount of invalid gradient calculations and wasted resources. Summary of the Invention

[0007] To address the technical problems of existing low-rank adaptation fine-tuning methods based on a unified parameter update strategy, which fail to distinguish the differences in activation responses of different sub-modules to enterprise-specific knowledge during training, lack fine-grained control over parameter update paths, and involve a large amount of redundant gradient calculations, resulting in wasted computational resources and low training efficiency, this invention provides a low-rank adaptation parameter selective update method and system based on activation variance awareness. The technical solution is as follows: On the one hand, a method for selectively updating low-rank fitting parameters based on activation variance awareness is provided, characterized in that the method includes: S1. Obtain the model deployed in the enterprise's private knowledge documents and database; S2, Calculation Model l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; S3. Sort the activation variance estimates of each submodule in descending order on the corresponding input sample, and give a preset pruning ratio. Divide the entire set of submodules into two mutually exclusive subsets by calculating the threshold of quantiles. S4. Initialize the LoRA adapter for all sub-modules in a unified manner, and during the backpropagation of the model, use a selective backpropagation constraint strategy to concentrate the computational resources of the model on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

[0008] Optionally, in S2, the calculation model is... l The activation variance estimate for each submodule of the layer on the corresponding input sample includes: Obtain a subset of calibration data, and for each input sample in the calibration data subset, in the model's... l The hidden states corresponding to each module of the layer; among them, the calibration data subset comes from the enterprise's internal private knowledge documents to ensure that the subsequent fine-tuning process can closely meet the enterprise's specific domain needs. For each submodule, calculate its activation variance statistic on each input sample, and concatenate the token-level activations of all samples; Calculate the first l The mean vector of the submodule and the definition of the firstl Activation variance of layer submodules.

[0009] Optionally, in S2, the model is characterized by the activation variance estimate. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution, including: The larger the activation variance, the more significant the differences in the output of the submodule under the condition of processing diverse enterprise knowledge, and the more malleable its parameter update space is; Conversely, a smaller activation variance indicates that the submodule output is relatively stable and not sensitive to changes in internal inputs.

[0010] Optionally, in S3, the activation variance estimates of each submodule on the corresponding input samples are sorted in descending order, including: The set of activation variance scores for all sub-modules calculated based on enterprise proprietary knowledge documents and databases; The data in the activation variance score set of all sub-modules are sorted in descending order to obtain the difference in response sensitivity of each sub-module under the distribution of enterprise proprietary knowledge.

[0011] Optionally, in S3, a threshold is determined by calculating quantiles to divide the entire set of submodules into two mutually exclusive subsets, including: The activation variance score set is divided into two mutually exclusive subsets, high sensitivity and low sensitivity, by using a threshold, including the activation subset and the frozen subset; The activated subset contains sub-modules that are highly sensitive to changes in enterprise private knowledge; the frozen subset contains sub-modules whose outputs are relatively stable under the enterprise knowledge distribution.

[0012] Optionally, in S4, the LoRA adapter is initialized uniformly for all submodules. During the model's backpropagation, a selective backpropagation constraint strategy is used to concentrate computational resources on the proportional parameter space most sensitive to changes in enterprise proprietary knowledge. This completes parameter allocation and training resource optimization based on the LoRA adapter, including: During the fine-tuning phase targeting enterprise proprietary knowledge documents and databases, the LoRA adapter is initialized uniformly for all sub-modules, and the linear weight matrix of each target is reparameterized. For submodules belonging to the frozen subset, gradients will be strictly prohibited from flowing through these modules during backpropagation; and the forward propagation calculation formula for any module will be modified to establish gradient update rules in optimization iterations. By employing the aforementioned selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing parameter allocation and training resource optimization based on the LoRA adapter.

[0013] Optionally, the gradient update rule during optimization iterations includes: ; in, Represents the set of trainable parameters; Indicates the learning rate; This represents the loss function, which measures the difference between the model's predicted output and the true label on the current task; and This is the low-rank decomposition matrix of the LoRA adapter; Represents any module; This indicates the activation subset.

[0014] On the other hand, a selective update system for low-rank fitting parameters based on activation variance awareness is provided. This system is applied to the selective update method for low-rank fitting parameters based on activation variance awareness. The system includes: The initialization unit is used to obtain models deployed in the enterprise's private knowledge documents and databases; Activate the variance unit to calculate the model's first variance unit. l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; The module sorting and selection unit is used to sort the activation variance estimates of each sub-module in descending order on the corresponding input sample, and, given a preset pruning ratio, divide the entire set of sub-modules into two mutually exclusive subsets by calculating the threshold of quantiles. The backpropagation constraint unit is used to uniformly initialize the LoRA adapter for all sub-modules. During the backpropagation of the model, through a selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

[0015] On the other hand, a selective update device for low-rank adaptation parameters based on activation variance awareness is provided. The selective update device for low-rank adaptation parameters based on activation variance awareness includes: a processor; a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for selective update of low-rank adaptation parameters based on activation variance awareness.

[0016] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of selective update of low-rank adaptation parameters based on activation variance awareness.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention aims to provide a parameter update control method based on the activation statistics of intermediate layers in a model, significantly improving the targeting and efficiency of parameter fine-tuning in private knowledge document and database scenarios. By introducing an analysis mechanism for the activation statistics of intermediate layers in neural networks, it provides objective evidence for the training participation of different sub-modules, addressing the lack of quantitative evaluation of sub-module activation behavior in existing technologies. It also addresses the overly uniform parameter update strategies in existing low-rank adaptation fine-tuning methods, enabling differentiated control of sub-module-level parameter update paths and avoiding unnecessary gradient calculations and parameter updates. While ensuring model performance, it reduces computational overhead and training redundancy during fine-tuning, improving the adaptability and training stability of efficient parameter fine-tuning methods under different task and data distribution conditions. Through achieving these objectives, this invention can perform data-driven fine-tuning of the parameter update process without altering the overall structure of the original pre-trained model or introducing additional inference latency. This results in improved computer system performance, such as reduced redundant computation, lower memory usage, and accelerated convergence. It provides a novel and practical technical means for the efficient, secure, and sustainable customized fine-tuning of large-scale enterprise language models in private knowledge document and database scenarios. This method is particularly suitable for enterprise-level application scenarios that require data protection, control computational costs, and pursue rapid iteration. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the low-rank adaptation parameter selective update method based on activation variance awareness provided in the embodiments of the present invention; Figure 2 This is a block diagram of a low-rank adaptation parameter selective update system based on activation variance awareness provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0021] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0022] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] This invention provides a method for selectively updating low-rank adaptation parameters based on activation variance awareness. This method can be implemented by a device for selectively updating low-rank adaptation parameters based on activation variance awareness, which can be a terminal or a server. Figure 1 The flowchart shown is for a low-rank fitting parameter selective update method based on activation variance awareness. The processing flow of this method may include the following steps: S1. Obtain the model deployed in the enterprise's private knowledge documents and database; S2, Calculation Model l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; In one feasible implementation, in S2, the calculation model is... l The activation variance estimate for each submodule of the layer on the corresponding input sample includes: Obtain a subset of calibration data, and for each input sample in the calibration data subset, in the model's... l The hidden states corresponding to each module of the layer; among them, the calibration data subset comes from the enterprise's internal private knowledge documents to ensure that the subsequent fine-tuning process can closely meet the enterprise's specific domain needs. For each submodule, calculate its activation variance statistic on each input sample, and concatenate the token-level activations of all samples; Calculate the first l The mean vector of the submodule and the definition of the first l Activation variance of layer submodules.

[0025] Suppose the pre-trained language model contains Layer Transformer block, number The Multi-Attention Head module of the layer is denoted as:

[0026] in, This indicates single-head attention module computation; Represents the query matrix; Represents the key matrix; Represents a value matrix; Indicates the dimension of the key.

[0027]

[0028] in, Indicates the first Calculation of attention heads; Indicates the first The query projection matrix of the head; Indicates the first The key projection matrix of the head; Indicates the first The projection matrix of the values ​​of each head.

[0029]

[0030] in, This indicates calculations involving multiple attention points; This indicates a splicing operation along the feature dimension; Indicates the first One point of attention; This indicates the output projection matrix.

[0031] No. The FFN module of the layer is denoted as:

[0032] in, This represents the computation of the feedforward network module; Represents the input vector; Indicates the activation function; Represents the gated projection matrix; Represents the projection matrix; Let represent the downward projection matrix.

[0033] In enterprise application scenarios, for models deployed in enterprise private knowledge documents and databases, this invention provides a calibration data subset. This subset is derived from internal proprietary knowledge documents (such as company files, internal reports, and proprietary database records) to ensure that the fine-tuning process closely aligns with the company's specific domain needs. For each input sample... For each input In the model The hidden state corresponding to each module of the layer is denoted as , .

[0034] This invention first calculates the value of each submodule in... The activation variance statistic. Concatenate the token-level activations of all samples:

[0035] in, Indicates the first Layer The set of activation values ​​for each submodule; This represents the total number of samples in the calibration dataset; Indicates the first The first sample The token at the _ ... Layer The output hidden state (activation value) of each submodule; Indicates the first The length of the token sequence for each sample.

[0036] Calculate the mean vector:

[0037] in, Indicates the first The activation mean vector of a submodule; Indicates the first Layer The set of activation values ​​for each submodule; This represents the hidden state vector of a certain token output by this submodule.

[0038] This invention defines the first The activation variance of the submodule is:

[0039] in, Indicates the first Layer The activation variance of each submodule; Indicates the first Layer The set of activation values ​​for each submodule; This represents the hidden state vector of a certain token output by this submodule; Indicates the first The activation mean vector of the submodule.

[0040] In one feasible implementation, in S2, the model is characterized by activating the variance estimate. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution, including: The larger the activation variance, the more significant the differences in the output of the submodule under the condition of processing diverse enterprise knowledge, and the more malleable its parameter update space is; Conversely, a smaller activation variance indicates that the submodule output is relatively stable and not sensitive to changes in internal inputs.

[0041] In one feasible implementation, the activation variance described above is used to characterize the overall variation in the output activation of the submodule under the enterprise's proprietary knowledge document distribution. Its magnitude reflects the submodule's sensitivity to differences in specific input samples (such as internal document queries or database retrieval entries). In enterprise-specific model deployment, a larger activation variance indicates that the submodule's output varies significantly when handling diverse enterprise knowledge (such as reports from different departments or historical data changes), and its parameter update space is more flexible, helping to improve the model's adaptability to enterprise-specific tasks. Conversely, a smaller activation variance indicates that the submodule's output is relatively stable, insensitive to changes in internal enterprise input, and the benefits of parameter updates under the current enterprise knowledge distribution are limited, thus avoiding resource waste and optimizing deployment efficiency.

[0042] S3. Sort the activation variance estimates of each submodule in descending order on the corresponding input sample, and give a preset pruning ratio. Divide the entire set of submodules into two mutually exclusive subsets by calculating the threshold of quantiles. In one feasible implementation, in S3, the activation variance estimates of each sub-module on the corresponding input sample are sorted in descending order, including: The set of activation variance scores for all sub-modules calculated based on enterprise proprietary knowledge documents and databases; The data in the activation variance score set of all sub-modules are sorted in descending order to obtain the difference in response sensitivity of each sub-module under the distribution of enterprise proprietary knowledge.

[0043] In one feasible implementation, in S3, the entire set of submodules is divided into two mutually exclusive subsets by calculating quantiles to determine a threshold, including: The activation variance score set is divided into two mutually exclusive subsets, high sensitivity and low sensitivity, by using a threshold, including the activation subset and the frozen subset; The activated subset contains sub-modules that are highly sensitive to changes in enterprise private knowledge; the frozen subset contains sub-modules whose outputs are relatively stable under the enterprise knowledge distribution.

[0044] In one feasible implementation, the set of activation variance scores for all sub-modules calculated based on the enterprise's proprietary knowledge documents and database is obtained. Then (where K is the number of all target linear submodules in the model, such as attention projections and FFN projections at each layer), they are sorted in descending order to reflect the differences in response sensitivity of each submodule under the enterprise proprietary knowledge distribution. A preset pruning ratio is given. This invention determines the threshold by calculating quantiles. :

[0045] in, This represents the threshold for dividing the dataset into active and frozen subsets. Percentile function; This represents the set of activation variance scores for all submodules. This indicates the preset trimming ratio.

[0046] The threshold The activation variance score set is divided into two categories: high sensitivity and low sensitivity. Based on this, the present invention divides the entire submodule set into two mutually exclusive subsets: Activation subset This includes sub-modules that are highly sensitive to changes in enterprise proprietary knowledge, such as internal documents, proprietary database records, industry-specific terminology, and business processes. These modules exhibit significant output fluctuations when handling diverse enterprise queries, such as internal knowledge retrieval, compliance Q&A, and business decision support, and are better suited for handling task-related, adaptable updates.

[0047] Frozen subset These are sub-modules whose outputs are relatively stable under the enterprise knowledge distribution. These modules have limited response to input disturbances (such as changes in document styles across different departments or historical data changes), and their parameter update benefits are low. They are suitable for being selectively frozen during fine-tuning to save resources.

[0048] Through this dynamic sorting and thresholding mechanism based on activation variance, enterprises can objectively evaluate and prioritize the allocation of limited computing budgets to the sub-modules that contribute most to enterprise-specific tasks before fine-tuning, using only a small number of calibration samples (from real enterprise private knowledge bases). This significantly improves the targeting and resource utilization of parameter fine-tuning in private deployment scenarios.

[0049] S4. Initialize the LoRA adapter for all sub-modules in a unified manner, and during the backpropagation of the model, use a selective backpropagation constraint strategy to concentrate the computational resources of the model on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

[0050] In one feasible implementation, in S4, the LoRA adapter is uniformly initialized for all sub-modules. During the backpropagation of the model, a selective backpropagation constraint strategy is used to concentrate the model's computational resources on the proportional parameter space most sensitive to changes in enterprise proprietary knowledge. This completes the parameter allocation and training resource optimization based on the LoRA adapter, including: During the fine-tuning phase targeting enterprise proprietary knowledge documents and databases, the LoRA adapter is initialized uniformly for all sub-modules, and the linear weight matrix of each target is reparameterized. For submodules belonging to the frozen subset, gradients will be strictly prohibited from flowing through these modules during backpropagation; and the forward propagation calculation formula for any module will be modified to establish gradient update rules in optimization iterations. By employing the aforementioned selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing parameter allocation and training resource optimization based on the LoRA adapter.

[0051] In one feasible implementation, the gradient update rule during optimization iteration includes: ; in, Represents the set of trainable parameters; Indicates the learning rate; This represents the loss function, which measures the difference between the model's predicted output and the true label on the current task; and This is the low-rank decomposition matrix of the LoRA adapter; Represents any module; This indicates the activation subset.

[0052] During the fine-tuning phase targeting enterprise proprietary knowledge documents and databases, this invention uniformly initializes the LoRA adapter for all submodules, reparameterizing each target linear weight matrix as follows:

[0053] in, This represents the weight matrix actually used for forward propagation after fine-tuning; This represents the original weight matrix of the pre-trained model; This indicates the increment for weight updates; This represents the low-rank decomposition matrix of the LoRA adapter.

[0054] in, Keep the system completely frozen and train only on low-rank decomposition matrices A and B.

[0055] For those belonging to the frozen subset The present invention strictly prohibits gradient flow through these modules during the backpropagation process, thereby avoiding invalid or noisy updates to low-sensitivity areas under the enterprise knowledge distribution.

[0056] For any module Its forward propagation calculation formula is modified as follows:

[0057] in, Indicates the first The output of each submodule; Indicates the first The original pre-trained weight matrices of each submodule; Indicates the first Input tensors of each submodule; This represents the scaling hyperparameters of the LoRA adapter; Indicates the rank of the LoRA adapter; Indicates the first The low-rank decomposition matrix of the LoRA adapter for each submodule; Indicates the activation subset; This indicates that a subset is frozen.

[0058] in and Let be the low-rank decomposition matrix of LoRA. In the optimization iteration, the gradient update rule is:

[0059] in, Represents the set of trainable parameters; Indicates the learning rate; This represents the loss function, which measures the difference between the model's predicted output and the true label on the current task; and This is the low-rank decomposition matrix of the LoRA adapter; Represents any module; This indicates the activation subset.

[0060] Through the selective backpropagation constraint strategy described above, the model concentrates valuable computing resources (GPU memory, training time, and energy consumption) on the areas most sensitive to changes in enterprise proprietary knowledge. Within the proportional parameter space, a more efficient and stable fine-tuning process can be achieved.

[0061] In one feasible implementation, the method of the present invention is particularly suitable for resource-constrained enterprise private deployment environments. It can significantly reduce fine-tuning overhead and overfitting risk while ensuring the model's ability to understand company-specific documents, and improve generalization performance and deployment friendliness in real business scenarios (such as internal knowledge question answering systems and intelligent database retrieval).

[0062] Compared to existing efficient parameter fine-tuning methods based on empirical rules or manually specified fine-tuning positions, the LoRA backpropagation selective control method proposed in this invention, based on activation value variance, explicitly calculates the statistical variance of activation values ​​for each sub-module in each layer of the model by performing forward inference on a subset of calibration data before fine-tuning. This variance is then used as a quantitative indicator of module information utilization and dynamic sensitivity, thereby achieving adaptive adjustment of LoRA participation in the backpropagation process. This method can objectively assess the importance of different sub-modules based on the distribution of real task data, avoiding the limitations of relying on fixed layer numbers, fixed sub-module types, or manually setting fine-tuning positions, and effectively suppressing over-fine-tuning of sub-modules with small contributions to the task. By prohibiting sub-modules with activation value variance in the last p% from participating in backpropagation, it can reduce invalid or redundant gradient updates in low-information-density sub-modules and reduce the interference of gradient noise on high-information-density sub-modules, thereby improving the convergence stability and training efficiency of the LoRA fine-tuning process. Meanwhile, since the LoRA parameters corresponding to the low-variance submodules no longer participate in gradient calculation and intermediate activation storage, the computational overhead and memory usage during the fine-tuning stage are significantly reduced without changing the model's forward structure and inference behavior, making it more suitable for customized fine-tuning of large models in resource-constrained scenarios. Furthermore, by concentrating the limited parameter update budget on submodules that are more sensitive to input changes and have stronger task relevance, this invention helps improve the effectiveness of parameter updates, mitigates the risk of overfitting and performance degradation under small sample or distribution shift conditions, and achieves better or more stable downstream task performance with the same or fewer LoRA parameter scales. This method does not rely on special modifications to the model structure, nor does it require the introduction of additional supervision signals. It can be directly integrated into existing LoRA or parameter-efficient fine-tuning frameworks, and the statistical analysis of activation variance only requires one forward inference with a small number of calibration samples, demonstrating good engineering feasibility and cross-model and cross-task versatility.

[0063] In this embodiment of the invention, statistical analysis is performed on the activation values ​​of sub-modules based on calibration data, and the variance of activation values ​​is used as a quantitative indicator to measure the information utilization and task relevance of model sub-modules. Based on this, sub-modules are sorted and grouped, and their participation in backpropagation during LoRA fine-tuning is selectively controlled. Specifically, this invention obtains the activation value distribution of each layer and sub-module by performing at least one forward inference on the pre-trained model before fine-tuning, and calculates the corresponding statistical variance. Based on the magnitude of the activation value variance, low-variance sub-modules with low responsiveness to the current task are identified, and their corresponding LoRA parameters are prohibited or blocked from participating in gradient calculation or backpropagation. This achieves fine-tuning of the parameter update path without changing the model's forward structure and inference behavior. Through the synergistic effect of the above technical features, this invention can concentrate limited parameter update resources on high-variance sub-modules with stronger task relevance, suppressing invalid gradient updates in low-information-density sub-modules, thereby reducing training computation and memory overhead, and improving fine-tuning stability and generalization performance.

[0064] Figure 2 This is a block diagram of a selective update system 300 for low-rank adaptation parameters based on activation variance awareness, illustrated according to an exemplary embodiment. The system 300 is used in a selective update method for low-rank adaptation parameters based on activation variance awareness. (Refer to...) Figure 2 The system includes an initialization unit 310, an activation variance unit 320, a module sorting and selection unit 330, and a backpropagation constraint unit 340. Among them: Initialization unit 310 is used to obtain models deployed in enterprise private knowledge documents and databases; Activate variance unit 320 to calculate the model's variance unit 320. l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; The module sorting and selection unit 330 is used to sort the activation variance estimates of each sub-module in descending order on the corresponding input sample, and to divide the entire set of sub-modules into two mutually exclusive subsets by calculating the threshold through quantiles. The backpropagation constraint unit 340 is used to uniformly initialize the LoRA adapter for all sub-modules, and during the backpropagation of the model, through the selective backpropagation constraint strategy, the model concentrates the computing resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

[0065] Optionally, the variance unit 320 is activated to obtain a subset of calibration data, and the variance of each input sample in the calibration data subset in the model. l The hidden states corresponding to each module of the layer; among them, the calibration data subset comes from the enterprise's internal private knowledge documents to ensure that the subsequent fine-tuning process can closely meet the enterprise's specific domain needs. For each submodule, calculate its activation variance statistic on each input sample, and concatenate the token-level activations of all samples; Calculate the first l The mean vector of the submodule and the definition of the first l Activation variance of layer submodules.

[0066] Optionally, the larger the activation variance, the more significant the differences in the output of the submodule under the condition of processing diverse enterprise knowledge, and the more malleable its parameter update space is; Conversely, a smaller activation variance indicates that the submodule output is relatively stable and not sensitive to changes in internal inputs.

[0067] Optionally, the module sorting and selection unit 330 is used for

[0068] The set of activation variance scores for all sub-modules calculated based on enterprise proprietary knowledge documents and databases; The data in the activation variance score set of all sub-modules are sorted in descending order to obtain the difference in response sensitivity of each sub-module under the distribution of enterprise proprietary knowledge.

[0069] Optionally, the module sorting and selection unit 330 is used to divide the set of activation variance scores into two mutually exclusive subsets, high sensitivity and low sensitivity, by means of a threshold, including an activation subset and a frozen subset; The activated subset contains sub-modules that are highly sensitive to changes in enterprise private knowledge; the frozen subset contains sub-modules whose outputs are relatively stable under the enterprise knowledge distribution.

[0070] Optionally, the backpropagation constraint unit 340 is used to uniformly initialize the LoRA adapter for all submodules and reparameterize each target linear weight matrix during the fine-tuning phase for enterprise private knowledge documents and databases. For submodules belonging to the frozen subset, gradients will be strictly prohibited from flowing through these modules during backpropagation; and the forward propagation calculation formula for any module will be modified to establish gradient update rules in optimization iterations. By employing the aforementioned selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing parameter allocation and training resource optimization based on the LoRA adapter.

[0071] Optionally, the gradient update rule during optimization iterations includes: ; in, Represents the set of trainable parameters; Indicates the learning rate; This represents the loss function, which measures the difference between the model's predicted output and the true label on the current task; and This is the low-rank decomposition matrix of the LoRA adapter; Represents any module; This indicates the activation subset.

[0072] This invention aims to provide a parameter update control system based on the activation statistics of intermediate layers in a model, significantly improving the targeting and efficiency of parameter fine-tuning in scenarios involving private knowledge documents and databases. By introducing an analysis mechanism for the activation statistics of intermediate layers in neural networks, it provides objective evidence for the training participation of different sub-modules, addressing the lack of quantitative evaluation of sub-module activation behavior in existing technologies. It also addresses the problem of overly uniform parameter update strategies in existing low-rank adaptation fine-tuning methods, enabling differentiated control of sub-module-level parameter update paths and avoiding unnecessary gradient calculations and parameter updates. While ensuring model performance, it reduces computational overhead and training redundancy during fine-tuning, improving the adaptability and training stability of efficient parameter fine-tuning methods under different task and data distribution conditions. Through achieving these objectives, this invention enables data-driven fine-tuning of the parameter update process without altering the overall structure of the original pre-trained model or introducing additional inference latency. This provides a novel and practical technical means for efficient, secure, and sustainable customized fine-tuning of large-scale enterprise language models in scenarios involving private knowledge documents and databases. This method is particularly suitable for enterprise-level application scenarios that require data protection, control computational costs, and pursue rapid iteration.

[0073] Figure 3 This is a schematic diagram of a low-rank adaptation parameter selective update device based on activation variance awareness, provided in an embodiment of the present invention. Figure 3 As shown, a low-rank adaptation parameter selective update device based on activation variance awareness may include the above-mentioned... Figure 2 The diagram illustrates a selective update system for low-rank adaptation parameters based on activation variance awareness. Optionally, a selective update device 410 for low-rank adaptation parameters based on activation variance awareness may include a first processor 2001.

[0074] Optionally, a low-rank adaptation parameter selective update device 410 based on activation variance awareness may further include a memory 2002 and a transceiver 2003.

[0075] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0076] The following is combined Figure 3 A detailed description is given of each component of a low-rank adaptation parameter selective update device 410 based on activation variance awareness: The first processor 2001 is the control center of a low-rank adaptation parameter selective update device 410 based on activation variance awareness. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0077] Optionally, the first processor 2001 can perform various functions of a low-rank adaptation parameter selective update device 410 based on activation variance awareness by running or executing software programs stored in memory 2002 and calling data stored in memory 2002.

[0078] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0079] In a specific implementation, as one example, a low-rank adaptation parameter selective update device 410 based on activation variance awareness may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0080] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0081] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected to the interface circuit of the device 410 through a low-rank adaptation parameter selective update mechanism based on activation variance awareness. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0082] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0083] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0084] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and its interface circuitry can be selectively updated via a low-rank adaptation parameter based on activation variance awareness. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0085] It should be noted that, Figure 3 The structure of a low-rank adaptation parameter selective update device 410 based on activation variance awareness shown in the figure does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0086] Furthermore, the technical effect of a low-rank adaptation parameter selective update device 410 based on activation variance perception can be referred to the technical effect of the low-rank adaptation parameter selective update method based on activation variance perception described in the above method embodiments, and will not be repeated here.

[0087] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0088] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0089] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0090] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0091] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for selectively updating low-rank fitting parameters based on activation variance awareness, characterized in that, The method includes: S1. Obtain the model deployed in the enterprise's private knowledge documents and database; S2, Calculation Model l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; S3. Sort the activation variance estimates of each submodule in descending order on the corresponding input sample, and give a preset pruning ratio. Divide the entire set of submodules into two mutually exclusive subsets by calculating the threshold of quantiles. S4. Initialize the LoRA adapter for all sub-modules in a unified manner, and during the backpropagation of the model, use a selective backpropagation constraint strategy to concentrate the computational resources of the model on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

2. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 1, characterized in that, In S2, the calculation model number is... l The activation variance estimate for each submodule of the layer on the corresponding input sample includes: Obtain a subset of calibration data, and for each input sample in the calibration data subset, in the model's... l The hidden states corresponding to each module of the layer; among them, the calibration data subset comes from the enterprise's internal private knowledge documents to ensure that the subsequent fine-tuning process can closely meet the enterprise's specific domain needs. For each submodule, calculate its activation variance statistic on each input sample, and concatenate the token-level activations of all samples; Calculate the first l The mean vector of the submodule and the definition of the first l Activation variance of layer submodules.

3. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 2, characterized in that, In S2, the model is characterized by the activation variance estimate. l Each submodule of the layer outputs the overall change in activation under the enterprise proprietary knowledge document distribution, including: The larger the activation variance, the more significant the differences in the output of the submodule under the condition of processing diverse enterprise knowledge, and the more malleable its parameter update space is; Conversely, a smaller activation variance indicates that the submodule output is relatively stable and not sensitive to changes in internal inputs.

4. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 3, characterized in that, In S3, the activation variance estimates of each submodule on the corresponding input samples are sorted in descending order, including: The set of activation variance scores for all sub-modules calculated based on enterprise proprietary knowledge documents and databases; The data in the activation variance score set of all sub-modules are sorted in descending order to obtain the difference in response sensitivity of each sub-module under the distribution of enterprise proprietary knowledge.

5. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 4, characterized in that, In step S3, the entire set of submodules is divided into two mutually exclusive subsets by calculating quantiles to determine a threshold, including: The activation variance score set is divided into two mutually exclusive subsets, high sensitivity and low sensitivity, by using a threshold, including the activation subset and the frozen subset; The activated subset contains sub-modules that are highly sensitive to changes in enterprise private knowledge; the frozen subset contains sub-modules whose outputs are relatively stable under the enterprise knowledge distribution.

6. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 5, characterized in that, In step S4, the LoRA adapter is initialized uniformly for all sub-modules. During the model's backpropagation, a selective backpropagation constraint strategy is used to concentrate computational resources on the proportional parameter space most sensitive to changes in enterprise proprietary knowledge. This completes the parameter allocation and training resource optimization based on the LoRA adapter, including: During the fine-tuning phase targeting enterprise proprietary knowledge documents and databases, the LoRA adapter is initialized uniformly for all sub-modules, and the linear weight matrix of each target is reparameterized. For submodules belonging to the frozen subset, gradients will be strictly prohibited from flowing through these modules during backpropagation; and the forward propagation calculation formula for any module will be modified to establish gradient update rules in optimization iterations. By employing the aforementioned selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing parameter allocation and training resource optimization based on the LoRA adapter.

7. The method for selectively updating low-rank adaptation parameters based on activation variance awareness according to claim 6, characterized in that, The gradient update rules during optimization iterations include: ; in, Represents the set of trainable parameters; Indicates the learning rate; This represents the loss function, which measures the difference between the model's predicted output and the true label on the current task; and This is the low-rank decomposition matrix of the LoRA adapter; Represents any module; This indicates the activation subset.

8. A selective update system for low-rank adaptation parameters based on activation variance awareness, wherein the selective update system for low-rank adaptation parameters based on activation variance awareness is used to implement the selective update method for low-rank adaptation parameters based on activation variance awareness as described in any one of claims 1-7, characterized in that, The system includes: The initialization unit is used to obtain models deployed in the enterprise's private knowledge documents and databases; Activate the variance unit to calculate the model's first variance unit. l The activation variance estimate of each submodule in the layer on the corresponding input sample is used to characterize the model's first-order model. l Each submodule of the layer outputs the overall change in activation under the enterprise private knowledge document distribution; The module sorting and selection unit is used to sort the activation variance estimates of each sub-module in descending order on the corresponding input sample, and, given a preset pruning ratio, divide the entire set of sub-modules into two mutually exclusive subsets by calculating the threshold of quantiles. The backpropagation constraint unit is used to uniformly initialize the LoRA adapter for all sub-modules. During the backpropagation of the model, through a selective backpropagation constraint strategy, the model concentrates its computational resources on the proportional parameter space that is most sensitive to changes in enterprise private knowledge, thereby completing the parameter allocation and training resource optimization based on the LoRA adapter.

9. A selective update device for low-rank adaptation parameters based on activation variance awareness, wherein the selective update device for low-rank adaptation parameters based on activation variance awareness comprises: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement any one of the methods in the low-rank adaptation parameter selective update method based on activation variance awareness as described in any one of claims 1-7.

10. A computer-readable storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the methods in the low-rank adaptation parameter selective update method based on activation variance awareness as described in any one of claims 1-7.