Ultralow parameter efficient fine tuning method based on sparse frequency domain projection

Abstract

The invention relates to the field of artificial intelligence deep learning model optimization, and discloses an ultra-low parameter efficient fine tuning method based on sparse frequency domain projection. Comprising the steps of setting an upper projection matrix which is frozen after random initialization in an attention mechanism layer or a feedforward mapping layer of a pre-training model for mapping a low-dimensional reconstruction result to an original weight dimension; constructing a sparse spectrum matrix in a frequency domain, generating a non-zero index set according to a frequency priority and a random sparse mixed strategy, and keeping the whole training process fixed; performing inverse transformation on the sparse spectrum matrix to obtain a time domain reconstruction matrix, and generating a weight increment through linear mapping of an upper projection matrix; only updating the non-zero spectrum coefficient under the limited parameter budget, and keeping the pre-training weight and the upper projection matrix frozen; optionally, spectrum regularization, energy constraint and gradient clipping are applied in the training stage, and quantification and distillation joint optimization is performed in the reasoning stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence deep learning model optimization, in particular to a super-low parameter efficient fine-tuning method based on sparse frequency domain projection. BACKGROUND

[0002] With the wide application of deep neural networks (DNN) in natural language processing and computer vision, the model size grows exponentially, and the number of parameters reaches billions. To achieve transfer between different tasks, traditional fine-tuning methods usually require retraining and storing all model parameters, which not only leads to huge training overhead, but also occupies a large amount of storage and computing resources in model distribution and online deployment stages. For example, in the Transformer series model, full-parameter fine-tuning needs to load complete attention layer and feedforward layer weights, which puts significant pressure on memory and bandwidth, making it difficult to meet the needs of end-side devices or multi-task rapid adaptation.

[0003] To reduce the fine-tuning cost, existing research has proposed various parameter-efficient fine-tuning methods, such as adapter insertion, low-rank decomposition (LoRA), prefix tuning, etc. Although these methods can reduce some trainable parameters, they still have the following shortcomings: first, low-rank decomposition and insertion-type structure rely on fixed rank constraints, making it difficult to further compress the parameter space while ensuring accuracy; second, most methods only perform linear mapping in the time domain or feature dimension, failing to fully utilize frequency domain characteristics, resulting in limited expression ability of model update signals; third, existing techniques still need to load multiple independent adaptation modules in parameter sharing and multi-task switching, lacking a unified management mechanism for cross-task parameters, thus limiting the model's lightweight deployment and real-time migration capabilities.

[0004] In summary, existing fine-tuning methods still have limitations in parameter compression rate, frequency domain expression ability, and cross-task generalization. Therefore, a super-low parameter efficient fine-tuning method based on sparse frequency domain projection is proposed. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a super-low parameter efficient fine-tuning method based on sparse frequency domain projection to solve the technical problems of parameter update redundancy, insufficient utilization of frequency domain information, large computation in model fine-tuning process, poor training stability, and low deployment efficiency in the prior art.

[0006] The above technical purpose of the present application is achieved by the following technical scheme:

[0007] The super-low parameter efficient fine-tuning method based on sparse frequency domain projection comprises the following steps:

[0008] S1: In the attention mechanism projection layer or the feedforward mapping layer of the pre-trained model, set the upper projection matrix that is randomly initialized in a Gaussian distribution or column orthogonal manner and then frozen, for mapping the low-dimensional reconstruction result to the original weight dimension, and the upper projection matrix can be shared among multiple layers or multiple tasks;

[0009] S2: Construct a sparse spectrum matrix in the frequency domain, generate a non-zero spectrum index set according to a frequency distribution priority and a random sparse mixed strategy, and the non-zero spectrum index set is determined in the training initialization stage and remains fixed in the whole training process;

[0010] S3: Divide the sparse spectrum matrix into multiple frequency domain sub-blocks along the row direction, and perform two-dimensional inverse Fourier transform or its equivalent linear reversible transform on each sub-block to obtain a corresponding time domain reconstruction matrix;

[0011] S4: Concatenate the time domain reconstruction matrix by row, and perform linear mapping through the upper projection matrix to generate a weight increment;

[0012] S5: Under the condition of a limited trainable parameter budget, only update the non-zero spectrum coefficients in the sparse spectrum matrix through back propagation based on the task loss, keep the pre-trained weights and the upper projection matrix in a frozen state, and apply spectrum regularization, energy constraint, learning rate grouping and gradient clipping in the training stage to form a parameter update closed loop and realize model fine-tuning.

[0013] Preferably, the upper projection matrix is shared among multiple linear layers and multiple tasks to reduce the total amount of adapters and the distribution volume. Different tasks share the upper projection matrix, and only the non-zero spectrum index set and the corresponding non-zero spectrum coefficients are switched to realize the parameter second-level switching and gray rollback update among tasks.

[0014] Preferably, the sparse frequency domain structure is inserted into the query layer and the value layer of the attention mechanism, and can be extended to the key layer and the feedforward network layer. The number of sub-blocks is set to 2 to 4, the dimension of the upper projection matrix is 64 or 128, and the spectrum sparsity is less than 10% to ensure the spectrum sparsity and reconstruction stability.

[0015] Preferably, the determination method of the non-zero spectrum index set includes a low-frequency priority and random sparse mixed strategy, a random sampling strategy with a fixed random seed, a cross-layer shared index strategy, or a learnable mask strategy based on gradient feedback.

[0016] Preferably, the inverse transform can be replaced by discrete cosine transform, discrete wavelet transform or Hadamard transform, and the above equivalent transforms all maintain the spectral domain fidelity characteristic.

[0017] ​Preferably, L1 or L2 regularization, energy constraint, learning rate grouping and gradient clipping are applied to the spectrum matrix during the training phase, and quantization and distillation joint optimization are performed on the weight increment or non-zero spectrum coefficient, wherein the quantization bit width is INT8 or INT4, and the distillation joint optimization is used to constrain the difference between the quantized student model output and the pre-trained model output.

[0018] Preferably, the weight increment is pre-calculated and cached after the training phase enters the inference phase, and only one addition operation is performed to combine to the original weight during model inference, thereby improving the inference efficiency.

[0019] Preferably, the trainable parameters account for 0.01% to 0.05% of the total amount of model parameters.

[0020] In summary, the present application mainly has the following beneficial effects:

[0021] By setting the upper projection matrix randomly initialized in the attention mechanism layer and the feedforward mapping layer of the pre-trained model in a Gaussian distribution or column orthogonal manner and then frozen, and linearly mapping the low-dimensional sparse frequency domain reconstruction result back to the original weight space through the matrix, a parameter decoupling structure of low-dimensional training and high-dimensional mapping is realized. In the training phase, only the non-zero coefficients in the frequency domain are updated, while the pre-trained weights and the projection matrix remain in a frozen state, thereby effectively reducing parameter redundancy and memory occupation, and significantly reducing training complexity and resource consumption. Through the above setting, the present application realizes high-fidelity weight reconstruction and stable model fine-tuning under the condition of ultra-low parameter budget.

[0022] By constructing a sparse spectrum matrix in the frequency domain and introducing a low-frequency priority and random sparse hybrid strategy, and combining an energy constraint term and a gradient mask update mechanism, the spectrum energy is dynamically and evenly distributed among the subblocks. This design effectively prevents numerical oscillation caused by gradient concentration in local frequency bands, and realizes adaptive expansion of high-contribution frequency domains through a learnable mask. Through the above setting, the model can realize gradient smoothing update at the frequency domain level during backpropagation, thereby achieving the technical effects of stable training convergence, balanced energy distribution and efficient optimization process.

[0023] By introducing a quantization and distillation joint optimization mechanism after training is completed, a joint objective function including task loss, quantization error and distillation loss is constructed, and the performance of the compressed model under low-bit precision is maintained. This mechanism enables the quantized model to learn the teacher model output distribution through distillation constraint, thereby compensating for the numerical error introduced by quantization. Through the above setting, only a weight fusion operation needs to be performed during the inference phase to realize efficient inference, and different tasks can be quickly switched, thereby achieving the comprehensive effects of low-bit deployment, high-speed inference and flexibility improvement of multi-task migration. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1is a flow chart of the method of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] Embodiment one

[0027] Reference Figure 1 The super-low parameter efficient fine-tuning method based on sparse frequency domain projection includes the following steps:

[0028] S1: In the attention mechanism projection layer or the feedforward mapping layer of the pre-trained model, an upper projection matrix randomly initialized by a Gaussian distribution or column orthogonal method is set to be frozen, which is used to map the low-dimensional reconstruction result to the original weight dimension and can be shared among multiple layers or multiple tasks;

[0029] S2: A sparse spectrum matrix is constructed in the frequency domain, and a non-zero spectrum index set is generated according to a frequency distribution priority and a random sparse hybrid strategy. The non-zero spectrum index set is determined in the training initialization stage and remains fixed in the whole training process.

[0030] S3: The sparse spectrum matrix is divided into multiple frequency domain sub-blocks along the row direction, and a two-dimensional inverse Fourier transform or its equivalent linear reversible transform is performed on each sub-block to obtain a corresponding time domain reconstruction matrix;

[0031] S4: The time domain reconstruction matrix is spliced by row, and linear mapping is performed through the upper projection matrix to generate a weight increment;

[0032] S5: Under the condition of limited trainable parameter budget, only the non-zero spectrum coefficients in the sparse spectrum matrix are updated through the task loss-based back propagation, the pre-trained weight and the upper projection matrix are kept in the frozen state, and the spectrum regularization, energy constraint, learning rate grouping and gradient clipping are applied in the training stage to form a parameter update closed loop and realize model fine-tuning.

[0033] Establishment and freezing of the upper projection matrix: in the attention mechanism projection layer or the feedforward network layer of the pre-trained model, an upper projection matrix is set, which is used to map the low-dimensional reconstruction result back to the original weight space;

[0034] The initialization of the matrix is generated by a Gaussian distribution or a column orthogonal method:

[0035]

[0036] where, is the standard deviation, is is the identity matrix of order Keep frozen during the whole training process, share among multiple layers and tasks;

[0037] Fixing the upper projection matrix is equivalent to defining a stable linear mapping subspace, which can reversibly map low-dimensional features to high-dimensional weights. By freezing the parameters, the distribution consistency between the model backbone weights and the adaptive layers is maintained.

[0038] This achieves a multi-task shared mapping structure, significantly reducing parameter storage volume and task switching time.

[0039] Construction and index generation of sparse spectral matrix Perform two-dimensional Fourier transform to get the frequency domain weight matrix:

[0040]

[0041] where, is the two-dimensional fast Fourier transform operator;

[0042] To realize the frequency domain sparsity, only the non-zero coefficients corresponding to the index set are retained:

[0043]

[0044] where, is the original frequency domain matrix; is the non-zero spectral index set; is the sparse spectral matrix after mask screening;

[0045] Index set is determined by the low-frequency priority and random sparse hybrid strategy:

[0046]

[0047] where, represents the energy concentration area, represents the high-frequency area sampled with a probability The index set is fixed at initialization, and can also be dynamically adjusted during training;

[0048] At the beginning of training (the first 30% of the iteration stage), the mask set is fixed to ensure stable spectral mapping. When the loss decrease rate is lower than the threshold , the gradient-aware module is activated to update the mask:

[0049]

[0050] where, is a Sigmoid function for smoothing gradient response, denotes the frequency component in the th row and th column of the frequency domain matrix, is the gradient of the loss function with respect to the frequency coefficient, is the gradient-aware score, is a threshold parameter. This mechanism enables the spectral mask to adaptively expand to the high-contribution frequency region;

[0051] The spectral sparse representation maps high-dimensional weights to the energy-concentrated frequency domain, preserving dominant components, and the learnable mask further optimizes the non-zero positions dynamically with training.

[0052] The block-wise inverse transform and time-domain reconstruction divide the sparse spectral matrix into sub-blocks where, perform a two-dimensional inverse Fourier transform or an equivalent linear invertible transform for each sub-block to obtain a time-domain reconstruction matrix:

[0053]

[0054] where, is an inverse Fourier transform, a discrete cosine (DCT) transform, a discrete wavelet (DWT) transform, or a Hadamard transform;

[0055] The reconstruction matrix is concatenated by rows as:

[0056]

[0057] where, denotes the complete time-domain reconstruction matrix composed of all sub-block matrices;

[0058] This step restores the sparse spectrum to a continuous time-domain structure, establishes a mapping relationship between the spectral sparsity and the time-domain weight distribution, ensures the continuity and differentiability of the sparse weight mapping, and avoids the fault phenomenon.

[0059] Mapping generation and weight fusion, after linear mapping by the upper projection matrix, the weight increment is obtained:

[0060]

[0061] and add it to the frozen pre-trained weights:

[0062]

[0063] where, The weight increment generated after projection mapping This is the updated weight matrix after weight fusion. This is the frozen weight matrix of the original pre-trained model;

[0064] Linear mapping enables reversible reconstruction from low-dimensional sparse representation to high-dimensional parameter space. By fixing the projection matrix, a unified mapping basis can be shared among different tasks.

[0065] Closed-loop optimization and parameter update: During the training phase, only the non-zero spectral coefficients in the sparse spectral matrix are updated. The loss function is defined as:

[0066]

[0067] in, For mission losses; This is an energy constraint term; This is the total loss function; It is a sparse spectral matrix; , These are the regularization weight coefficients;

[0068] The energy constraint term is defined as follows:

[0069]

[0070] in, For the first One spectrum sub-block;

[0071] The gradient update rule is as follows:

[0072]

[0073] in, For the mask matrix, The learning rate;

[0074] To ensure that the frequency domain gradient can flow back, the gradient calculation path is defined as follows:

[0075]

[0076] in, Corresponding inverse transform The Jacobian matrix is ​​obtained by inverse FFT; this chain gradient backflow achieves closed-loop optimization of frequency domain, time domain, and task loss.

[0077] By restricting gradient propagation paths with sparse masks and applying energy regularization and L1 sparsity constraints, efficient differentiable optimization is achieved, which significantly reduces computational and communication costs while maintaining model accuracy.

[0078] Quantization and distillation joint optimization, after training, the spectral parameters are quantized by INT8 or INT4, and the distillation constraint is introduced, and the joint loss function is:

[0079]

[0080] Wherein, is the task loss; is the quantization error loss; is the distillation loss; is the quantization item weight coefficient; distillation item weight coefficient; , used to keep the student model and the teacher model output consistent, wherein, is the output result of the student model (the model after quantization and compression), is the output result of the teacher model (unquantized, full-precision model);

[0081] After training, the sparse spectral parameters are quantized by INT8 or INT4, and the model storage volume is further compressed, in order to verify the structural compression effect of the method under the condition of "ultra-low parameter", the parameter size and the trainable proportion under different fine-tuning methods are compared, and the results are shown in Table 1:

[0082] Table 1: Comparison table of parameter size and trainable proportion under different fine-tuning methods

[0083]

[0084] As shown in Table 1, the present application reduces the trainable parameter proportion to 0.03% under the premise of keeping the model structure complete, which is significantly better than the existing LoRA and Adapter methods, verifying the "ultra-low parameter budget" feature;

[0085] In order to verify the model performance retention ability of the sparse frequency domain projection under the condition of low parameters, typical natural language understanding and image classification tasks are selected for comparison test, and the results are shown in Table 2:

[0086] Table 2: Comparison test table of typical natural language understanding and image classification tasks

[0087]

[0088] As shown in Table 2, the performance retention rate of each task is more than 99% when the parameter amount of the present application is only 0.03% of the full amount fine-tuning, which fully shows that the frequency domain sparse projection method can keep the performance stable while compressing the parameters;

[0089] In order to further verify the efficient characteristics of the present application in the model deployment stage, the training efficiency, inference delay and task switching time of each fine-tuning method are counted, and the results are shown in Table 3:

[0090] Table 3: Training efficiency, inference delay and task switching time data table of each fine-tuning method

[0091]

[0092] As can be seen from Table 3, the present application only needs one addition operation to complete weight fusion, realizes about 40% delay reduction compared with the traditional method in the inference stage, and controls the task switching time within 1.2 seconds, which significantly improves the online deployment performance of the model;

[0093] The quantization term constraint parameter value is discretized, and the distillation term ensures that the output of the quantized model remains semantically consistent.

[0094] The fast fusion mechanism in the inference stage pre-caches the weight increment , and performs single weight fusion:

[0095]

[0096] After the model is loaded, no inverse transformation or spectrum calculation is performed; by static weight merging, calculation redundancy during inference is avoided, the inference delay is reduced by more than 30%, and task-level switching is supported.

[0097] Architecture adaptation and hybrid deployment, the method can be deployed in a model containing an encoder, a decoder or a visual Transformer structure to perform the frequency domain fine-tuning operation; sparse frequency domain mapping is used in key layers to compress parameters, the remaining layers maintain the original adaptation mechanism, high layers use sparse frequency domain mapping to realize parameter compression, shallow layers retain low-rank adaptation to keep features stable, and multi-layer collaborative optimization and cross-task compatibility are realized.

[0098] The present scheme initializes and freezes the upper projection matrix;

[0099] A frequency domain sparse matrix is constructed and an index set is generated;

[0100] Inverse transform to reconstruct the time domain weight;

[0101] Linear mapping generates weight increment;

[0102] Sparse closed-loop optimization and gradient update are performed;

[0103] Quantization and distillation joint optimization are performed;

[0104] The weight is cached and quickly loaded in the inference stage.

[0105] Each stage is connected through a chain gradient backflow, forming a differentiable closed-loop structure of frequency domain, time domain and weight;

[0106] The test results on GLUE, COCO, ImageNet and other public benchmarks show that:

[0107] In the case of trainable parameters accounting for 0.01% to 0.05% of the whole model, the method achieves an accuracy comparable to that of the low-rank adaptation method, and reduces inference delay by 40% and energy consumption by about 35% under multi-task conditions.

[0108] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A super low parameter efficient fine-tuning method based on sparse frequency domain projection, characterized in that, The method comprises the following steps: S1: setting an upper projection matrix in the attention mechanism projection layer or the feedforward mapping layer of the pre-trained model, which is randomly initialized by a Gaussian distribution or column orthogonality and then frozen, for mapping a low-dimensional reconstruction result to an original weight dimension and being shared among multiple layers or multiple tasks; S2: constructing a sparse spectrum matrix in a frequency domain, generating a non-zero spectrum index set according to a frequency distribution priority and a random sparsity mixed strategy, and determining the non-zero spectrum index set in a training initialization stage and keeping the non-zero spectrum index set fixed in a whole training process; S3: dividing the sparse spectrum matrix into multiple frequency domain sub-blocks along a row direction, performing two-dimensional inverse Fourier transform or an equivalent linear reversible transform on each sub-block to obtain a corresponding time domain reconstruction matrix; S4: splicing the time domain reconstruction matrix by rows and performing linear mapping on the time domain reconstruction matrix through the upper projection matrix to generate a weight increment; S5: under the condition of a limited trainable parameter budget, only updating non-zero spectrum coefficients in the sparse spectrum matrix through back propagation based on a task loss, keeping the pre-trained weight and the upper projection matrix in a frozen state, and applying spectrum regularization, energy constraint, learning rate grouping and gradient clipping in the training stage to form a parameter update closed loop and realize model fine-tuning.

2. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 1, characterized in that, The upper projection matrix is shared among multiple linear layers and multiple tasks to reduce the total amount of adapters and the distribution volume, and different tasks share the upper projection matrix, only switching the non-zero spectrum index set and the corresponding non-zero spectrum coefficients to realize parameter second-level switching and gray rollback updating among tasks.

3. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 2, characterized in that, The sparse frequency domain structure preferentially inserts an attention mechanism query layer and a numerical layer, which can be extended to a key layer and a feedforward network layer, and the number of blocks The upper projection matrix dimension is 64 or 128, and the spectral sparsity is less than 10%, to ensure spectral sparsity and reconstruction stability.

4. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 3, characterized in that, The determination mode of the non-zero spectrum index set includes a low-frequency priority and a random sparsity mixed strategy, a random sampling strategy with a fixed random seed, a cross-layer shared index strategy or a learnable mask strategy based on gradient feedback.

5. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 4, characterized in that, The linear reversible transform can be replaced by a discrete cosine transform, a discrete wavelet transform or a Hadamard transform, and the above equivalent transforms all maintain the spectral domain fidelity characteristic.

6. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 5, characterized in that, In the training stage, L1 or L2 regularization, energy constraint, learning rate grouping and gradient clipping are applied to the spectrum matrix, and quantization and distillation joint optimization are performed on the weight increment or the non-zero spectrum coefficient, wherein the quantization bit width is INT8 or INT4, and the distillation joint optimization is used to constrain the difference between the output of the quantized student model and the output of the pre-trained model.

7. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 6, characterized in that, In the inference stage after the training stage, the weight increment is pre-calculated and cached, and only one addition operation is performed to combine the original weight during model inference, thereby improving the inference efficiency.

8. The sparse frequency domain projection based ultra-low parameter high efficiency fine-tuning method according to claim 7, characterized in that, The trainable parameters account for 0.01% to 0.05% of the total amount of model parameters.

Images