Model optimization method and device for low-resource scene, electronic equipment and medium

Abstract

The invention relates to a model optimization method and device for a low-resource scene, electronic equipment and a medium, and the method comprises the steps: obtaining a pre-training model, the pre-training model being a Transform model, for any linear layer in the Transform model, connecting a pair of low-rank matrixes in parallel in the linear layer to form a LoRA increment path, and freezing a backbone network in the pre-training model; for any linear layer in the Transform model, a learnable gate is added on a LoRA increment path, and an improved LoRA module is formed; and obtaining training data, training the pre-training model according to the training data to obtain a target model, and performing semantic analysis on the to-be-processed text according to the target model. According to the method provided by the invention, through collaborative design of structural innovation and an optimization mechanism, the performance and stability of the efficient parameter fine tuning method in a low-resource scene are remarkably improved, and the method has a good industrial application prospect.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and more specifically, to a model optimization method, apparatus, electronic device, and medium for low-resource scenarios. Background Technology

[0002] With the widespread application of large-scale pre-trained language models in natural language processing, computer vision, and multimodal tasks, the number of model parameters continues to grow, ranging from hundreds of millions to hundreds of billions. Although large models exhibit strong generalization capabilities, full fine-tuning for specific downstream tasks requires updating all model parameters, resulting in huge computational resource consumption and high storage costs, making them difficult to deploy, especially on edge devices or in low-resource computing environments.

[0003] In summary, existing LoRA methods still have the following technical shortcomings: First, the gradient propagation path is fixed across different tasks, layers, or input samples, lacking dynamic adjustment capabilities, which makes it sensitive to noisy data or distribution shifts in low-resource scenarios.

[0004] Secondly, the standard LoRA module does not take into account the differences in gradient magnitude between different adapted modules, which may lead to gradient conflicts or unstable optimization.

[0005] Finally, under conditions of extremely low data volume, LoRA may still overfit to a limited number of samples, thus limiting its generalization ability.

[0006] Therefore, there is an urgent need in the existing technology for a new fine-tuning framework that can further improve parameter efficiency, optimize gradient propagation behavior, and enhance model robustness in low-resource environments. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a model optimization method, apparatus, electronic device and medium for low resource scenarios, aiming to solve at least one of the above-mentioned technical problems.

[0008] Firstly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a model optimization method for low-resource scenarios, the method comprising: Obtain the pre-trained model, which is a Transformer model. For any linear layer in the Transformer model, connect a pair of low-rank matrices in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. For any linear layer in the Transformer model, a learnable gate is added to the LoRA incremental path to form an improved LoRA module. Acquire training data. The number of samples in the training data is less than a set value. The training data includes multiple sample texts and the annotation results corresponding to each sample text. The pre-trained model is trained based on the training data to obtain the target model, which is then used to perform semantic analysis on the text to be processed. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

[0009] The beneficial effects of this invention are: in scenarios with a low sample size less than a set value, by freezing the backbone network and introducing LoRA incremental paths with learnable gating only next to each linear layer, extremely lightweight fine-tuning of the number of trainable parameters can be achieved, significantly reducing the demand for GPU memory and computing power, and improving the fine-tuning efficiency and model performance under conditions of scarce data and limited computing power.

[0010] Based on the above technical solution, the present invention can be further improved as follows.

[0011] Furthermore, the above-mentioned training of the pre-trained model based on the training data to obtain the target model includes: By extracting features from the training data through the frozen backbone network in the pre-trained model, the initial features corresponding to each sample text are obtained. The text features corresponding to each sample text are input into a linear layer of the pre-trained model, and the LoRA increment corresponding to each sample text is determined through the LoRA increment path in the linear layer. For each sample text, the increment is scaled element-wise according to the LoRA increment corresponding to the sample text and the learnable gate in the linear layer to obtain the scaled increment. The scaled increment is then added to the initial feature corresponding to the sample text to obtain the intermediate feature. The pre-trained model is trained based on the intermediate features corresponding to all sample texts to obtain the target model.

[0012] The beneficial effect of adopting the above-mentioned further scheme is that, during the training process, the initial features are first extracted by freezing the backbone network, and then the improved LoRA module is used to perform gated scaling on the increments before superposition. This makes the early stage completely dependent on the backbone expression with strong generalization ability, and then gradually injects low-rank adaptive information in the later stage, realizing a smooth transition from "backbone-dominated to LoRA-assisted". This further reduces the risk of early overfitting under the limitation of less training data, speeds up the convergence speed, and maintains stable performance gains under extremely low sample conditions.

[0013] Furthermore, the total loss value corresponding to the aforementioned pre-trained model includes the main task loss value, the gradient magnitude equalization loss value, and the L2 regularization of the gating parameters. The main task loss value represents the difference between the prediction result and the labeled result output by the pre-trained model. The gradient magnitude equalization loss value represents the difference between the gradient magnitudes of two low-rank matrices within the same improved LoRA module. The L2 regularization of the gating parameters represents the distance between the gating parameters and the zero vector. The model parameters of the pre-trained model include the gating parameters.

[0014] The beneficial effects of adopting the above-mentioned further scheme are that by coupling the main task loss, gradient magnitude equalization loss and gated L2 regularization in the training objective, on the one hand, the gradient magnitudes of the two low-rank matrices within the same improved LoRA module are automatically aligned, significantly reducing gradient conflicts and optimization oscillations, and improving training stability; on the other hand, L2 constraints are applied to the gating parameters to prevent the gate value from deviating too far from zero and causing overfitting, thereby maintaining performance improvement in scenarios with lower training data and maintaining a controllable convergence trajectory of continuously decreasing gradient variance.

[0015] Furthermore, the gradient magnitude equalization loss value corresponds to the first weight, and the L2 regularization of the gating parameter corresponds to the second weight. During the training process of the pre-trained model, the above method also includes: In the initial stage, the first and second weights are increased so that the features of the training data include those extracted by the backbone network but not those extracted by the improved LoRA module. During the transition phase, the first and second weights are reduced so that the features of the training data also include those extracted by the improved LoRA module.

[0016] The beneficial effect of adopting the above-mentioned further scheme is that by dynamically adjusting the first and second weights in a course-based manner, the strength of the two gating regularizations is simultaneously amplified in the initial stage of training, so that the improved LoRA module is almost turned off and the backbone network independently undertakes feature expression, which significantly reduces the risk of early overfitting. Subsequently, the two weights are simultaneously decayed in the transition stage, gradually releasing the LoRA increment and gating ability, realizing a smooth transition from "freeze-regulation" to "free fine-tuning". In scenarios with less training data, it maintains the generalization advantage of the backbone while making full use of the flexibility of low-rank adaptation, thereby improving the convergence speed of the model under extremely low sample size conditions.

[0017] Furthermore, the pre-trained model also includes a hybrid adapter module, and the method further includes: The hybrid adapter module freezes the original parameters corresponding to each linear layer in the pre-trained model.

[0018] The beneficial effect of adopting the above-mentioned further scheme is that by inserting a "hybrid adapter module" next to each linear layer, the incremental parameters and gating parameters can be trained independently without changing the number of parameters in the backbone network. This module provides dynamic scaling and gradient equalization functions during the training phase, and can be merged or turned off as a whole during the inference phase to keep the model volume zero expansion or compress it in real time, which significantly reduces the memory usage and computational overhead, and improves the convergence speed in low-resource scenarios.

[0019] Secondly, in order to solve the above-mentioned technical problems, the present invention also provides a model optimization device for low-resource scenarios, the device comprising: The first acquisition module is used to acquire the pre-trained model, which is a Transformer model. For any linear layer in the Transformer model, a pair of low-rank matrices are connected in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. An additional module is added to add a learnable gate to the LoRA incremental path for any linear layer in the Transformer model, forming an improved LoRA module. The second acquisition module is used to acquire training data. The number of samples in the training data is less than a set value. The training data includes multiple sample texts and the annotation results corresponding to each sample text. The training module is used to train the pre-trained model based on the training data to obtain the target model, which is then used to perform semantic analysis on the text to be processed. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

[0020] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the model optimization method for low-resource scenarios of the present application.

[0021] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the model optimization method for low-resource scenarios of the present application.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0024] Figure 1 This is a flowchart illustrating a model optimization method for low-resource scenarios according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a model optimization device for low-resource scenarios provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0025] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0026] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0027] The solution provided in this invention can be applied to any application scenario that requires optimized training of the Transformer model. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.

[0028] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of a model optimization method for low-resource scenarios. This method can be executed by any electronic device, such as a terminal device, or jointly by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject. Figure 1 The flowchart shown indicates that the method may include the following steps: S10, Obtain the pre-trained model. The pre-trained model is a Transformer model. For any linear layer in the Transformer model, connect a pair of low-rank matrices in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. S20: For any linear layer in the Transformer model, add a learnable gate to the LoRA incremental path to form an improved LoRA module. S30, Obtain training data. The number of samples in the training data is less than a set value. The training data includes multiple sample texts and the annotation results corresponding to each sample text. S40, the pre-trained model is trained based on the training data to obtain the target model, and semantic analysis of the text to be processed is performed based on the target model. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

[0029] The proposed solution achieves extremely lightweight fine-tuning of trainable parameters in scenarios with low sample sizes (less than a set value) by freezing the backbone network and introducing LoRA incremental paths with learnable gating only next to each linear layer. This significantly reduces the demand for GPU memory and computing power, and improves fine-tuning efficiency and model performance under conditions of scarce data and limited computing power.

[0030] The following specific embodiments further illustrate the solution of the present invention. In this embodiment, to address the problems existing in the prior art, the present invention proposes a parameter-efficient fine-tuning framework for low-resource scenarios. By combining a low-rank adaptive (LoRA) structure with a gradient path modulation mechanism, a hybrid adapter is constructed to achieve fine-grained control over the model fine-tuning process. This framework is designed collaboratively from two dimensions: model structure innovation and optimization mechanism innovation, significantly improving fine-tuning efficiency and model performance under conditions of scarce data and limited computing power. The core idea of ​​parameter-efficient fine-tuning (PEFT) is to train only a small number of newly added or adjustable parameters while freezing the model parameters of the backbone network, thereby achieving performance comparable to full fine-tuning. An improved LoRA module is also introduced. Low-rank adaptive (LoRA) has a simple structure and superior performance. LoRA introduces incremental matrices (A and B) of low-rank decomposition next to the original weight matrix to achieve an approximate expression of weight updates, significantly reducing the number of trainable parameters while maintaining the model's expressive power.

[0031] Based on this, the model optimization method for low-resource scenarios provided in this embodiment may include: S10, Obtain the pre-trained model. The pre-trained model is a Transformer model. For any linear layer in the Transformer model, connect a pair of low-rank matrices in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. The Transformer architecture consists of several overlapping layers with different functions. If we break these layers down logically, they can be categorized as follows (from input to output): 1. Input Embedding layer is used to map discrete token IDs into continuous dense vectors; it is usually added to positional encoding to provide order information to the model.

[0032] 2. Multi-Head Self-Attention layer is used to calculate the attention weights between all pairs of positions within a sequence, and learns different types of correlations in parallel through multiple "heads".

[0033] 3. Add & Normalize layers, where residual connections are used to directly add the sub-layer input to the sub-layer output to alleviate gradient vanishing.

[0034] Layer Normalization: Used to normalize across the feature dimension for stable training.

[0035] 4. Position-wise Feed-Forward Network (FFN) layer: This layer is used to apply two fully connected layers independently to each location (typically with the middle dimension magnified by 4 times).

[0036] Nonlinearity (ReLU, GELU, SwiGLU, etc.) can also be introduced to enhance expressive power.

[0037] 5. Encoder-Decoder Attention Layer (only needed in the decoder part): Used to query the current position from the decoder, with the key / value coming from the encoder output, achieving "source-target" alignment. It also consists of multi-head attention and Add & Norm.

[0038] 6. Output Projection (Softmax) layer: This layer linearly maps the hidden states of the decoder's top layer to the vocabulary dimension. A softmax layer is then applied to obtain the probability distribution of the next token.

[0039] The core layer of Transformer is "Embedding → Multi-head Self-attention → Add & Norm → Feedforward → Add & Norm", with the encoder and decoder stacked N times each; the decoder has an additional "encoder-decoder attention" layer; and finally, an output projection layer is added.

[0040] In Transformer, a "linear layer" refers to a fully-connected (dense) layer that performs only a linear transformation y = xW + b. It contains no non-linear activations and no complex operators such as LayerNorm, Dropout, or Attention. Therefore, linear layers in Transformer include: input / output embedding layers, the three projections in attention: Q, K, and V linear mappings, and the two linear mappings in the feedforward network (FFN). Apart from these, LayerNorm, residual connections, Softmax, and Attention weight calculations are not considered linear layers.

[0041] The freezing of the backbone network in the pre-trained model means that during the entire training process, except for a small number of newly added or specified trainable parameters (LoRA low-rank matrices A and B and gating parameters U, V, b_g, etc.), all other original weights (including embedding layers, all self-attention layers, feedforward networks, and layer normalization, etc.) of the Transformer model are set to non-trainable (requires_grad=False). Their parameter values ​​remain unchanged in the pre-trained state at the time of loading, neither receiving backpropagation gradients nor being updated by the optimizer, and are only used as fixed feature extractors.

[0042] S20: For any linear layer in the Transformer model, add a learnable gate to the LoRA incremental path to form an improved LoRA module. Optionally, the original weight matrix of the pre-trained model is W0. A pair of low-rank matrices is appended to the linear layer to form the LoRA incremental path. That is, a pair of low-rank matrices is added to the standard LoRA module for updating, which can be expressed as: in, , All are low-rank matrices. It is the rank dimension, usually set to 4~64.

[0043] Building upon this, the gradient path modulation unit introduced in this invention, based on the standard LoRA module, is a learnable soft-gated structure directly embedded within the LoRA incremental path, and its structure is as follows: 1. Input: The original input hidden vector of this linear layer. .

[0044] 2. Low-rank increment: Δ = α·BA h, where α is the scaling factor.

[0045] 3. Gating signal: g =σ( LayerNorm( h U^TV^T ) + b_g ), in, , and bias All parameters are learnable, σ is the sigmoid function, and the output is... .

[0046] 4. Output: That is, first scale Δ element by element using g, and then add it to the main output.

[0047] During forward propagation, g controls the contribution of the LoRA increment to the final output; during backward propagation, the gradient of g also adjusts the magnitude of the gradient flows of A and B, achieving "gradient path modulation".

[0048] As an example, for the input vector The output of the improved LoRA module is: in: Output after adaptation; The LoRA scaling factor is usually set as a function of rank r (e.g., =r); This represents the Hadamard product (element-by-element multiplication). A learnable gating function with parameters. , used to modulate the activation intensity of the LoRA branch.

[0049] Among them, the gate function The specific implementation is as follows: in: To create a learnable weight matrix, a low-rank decomposition can be used in practice. To control the amount of additional parameters; For bias terms; Use the Sigmoid activation function to ensure the output is in This interval enables soft gating of LoRA paths. This is a layer normalization operation to stabilize the input distribution.

[0050] This gating mechanism controls the contribution intensity of the LoRA branch during forward propagation and adjusts the gradient flows of A and B accordingly during backward propagation, thus achieving "gradient path modulation".

[0051] S30, Obtain training data. The number of samples in the training data is less than a set value. The training data includes multiple sample texts and the annotation results corresponding to each sample text. In this context, a smaller number of training data samples than a set value indicates low-quality training data. In the realm of large models, the annotation results can represent different meanings for different application scenarios. As an example, the specific form of the annotation results directly corresponds to the downstream task. Examples include: 1. Text classification (single label): Sample text: "This movie has a tight plot and stunning special effects." Annotation result: positive (positive sentiment category label); 2. Text multi-label classification: Sample text: "The patient reported headache, fever, and dry cough." Annotation result: [headache, fever, dry cough] (multiple medical symptom labels are simultaneously valid); 3. Named entity recognition (sequence labeling): Sample text: "A company released a new mobile phone in City A." Annotation result: [B-LOC, I-LOC, B-ORG, I-ORG, I-ORG, O, B-LOC, O, O, O, O, O, O, O, O] (B-start, I-middle, O-non-entity BIO label sequence); 4. Question answering (extractive): Sample text: "On September 2, 2025, Kimi released the latest large model." Question: Release date of the large model? Annotation result: September 2, 2025 (correct answer text fragment). 5. Regression / scoring task, sample text: full text of an English essay, annotation result: 7.5 (essay score, a real number between 0 and 10); 6. Generative summarization, sample text: a long news article, annotation result: a manually written summary text: Successful launch of a new generation of meteorological satellites will improve typhoon monitoring accuracy.

[0052] As long as the number of samples is less than the set value (e.g., 100 samples per class), any of the above-mentioned "text + annotation" formats can be used as training data for this scheme.

[0053] S40, the pre-trained model is trained based on the training data to obtain the target model, and semantic analysis of the text to be processed is performed based on the target model. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

[0054] Adjusting a few model parameters during training can improve model training efficiency.

[0055] Optionally, in step S40 above, training the pre-trained model based on the training data to obtain the target model includes: S401, the initial features corresponding to each sample text are obtained by extracting features from the training data through the frozen backbone network in the pre-trained model. S402, input the text features corresponding to each sample text into a linear layer of the pre-trained model, and determine the LoRA increment corresponding to each sample text through the LoRA increment path in the linear layer; S403, for each sample text, the increment is scaled element-wise according to the LoRA increment corresponding to the sample text and the learnable gate in the linear layer to obtain the scaled increment. The scaled increment is added to the initial feature corresponding to the sample text to obtain the intermediate feature. S404: Train the pre-trained model based on the intermediate features corresponding to all sample texts to obtain the target model.

[0056] This process can be broken down into five consecutive steps: "freezing the backbone → low-rank increment → gating scaling → residual stacking → global training". Its core idea is to fine-tune the output of each linear layer using a very small number of trainable parameters without affecting the backbone parameters (the model parameters corresponding to the backbone network). A point-by-point analysis follows: 1. Frozen Backbone Feature Extraction: The backbone network is completely frozen and only serves as a fixed general feature extractor; the training data can obtain stable initial features after passing through it once, ensuring that the prior knowledge of the large model is not destroyed and minimizing memory / computation overhead.

[0057] 2. LoRA Incremental Path: Next to each target linear layer, Δh = α·B·A·h is calculated using low-rank matrices A and B, compressing the number of parameters to <1% of the original weights. This step reduces the problem of "how to fine-tune" to "learning a low-rank perturbation", avoiding full updates.

[0058] 3. Learnable gated scaling: Introduced Then, Δh is multiplied element by element by g∈(0,1). The gating value changes dynamically with the input, which can suppress overfitting by squeezing g close to 0 on noisy samples, and amplify g on high-confidence samples to make full use of LoRA's expressive power, thus achieving "sample-level / time-level" adaptation.

[0059] 4. Residual stacking: The scaled increment is added to the initial feature in the form of residuals, following the Transformer residual philosophy of "preserving the original expression + injecting controllable corrections". The gradient can be directly backpropagated to at least LoRA and gating parameters, while the backbone still has no gradient.

[0060] Thus, the proposed solution achieves a unified approach to "parameter minimization, adaptive adjustment, and training stability" under extremely low resource conditions. It inherits the general representation of pre-trained large models and injects task-specific information in a refined manner through gated LoRA.

[0061] Furthermore, the total loss value corresponding to the pre-trained model includes the main task loss value, the gradient magnitude equalization loss value, and the L2 regularization of the gating parameters. The main task loss value represents the difference between the prediction result and the labeled result output by the pre-trained model. The gradient magnitude equalization loss value represents the difference between the gradient magnitudes of the two low-rank matrices within the same improved LoRA module. The L2 regularization of the gating parameters represents the distance between the gating parameters and the zero vector. The model parameters of the pre-trained model include the gating parameters.

[0062] The total loss value mentioned above can be expressed as: in, The main task loss (such as cross-entropy); These are hyperparameters that control the strength of the regularization term, which are also the first and second weights in the scheme of this application. L2 regularization of the gate parameters is used to prevent overfitting.

[0063] The gradient magnitude equalization loss described above can be expressed as: in, This represents the total number of improved LoRA modules in the pre-trained model; and The first In an improved LoRA module, the matrix and Gradient in the current batch; This represents the L2 norm (i.e., the gradient magnitude). For smoothing terms (such as...) This prevents division by zero; the loss term encourages internal division within each LoRA module. and The gradient magnitudes tend to be consistent, which alleviates optimization oscillations caused by differences in gradient scale.

[0064] Furthermore, the gradient magnitude equalization loss value corresponds to the first weight, and the L2 regularization of the gating parameter corresponds to the second weight. During the training process of the pre-trained model, the above method also includes: In the initial stage, a first weight and a second weight are added so that the features of the training data include those extracted by the backbone network but exclude those extracted by the improved LoRA module; wherein, the above initial stage may include a forward propagation stage and a backpropagation stage; During the transition phase, the first and second weights are reduced so that the features of the training data also include those extracted by the improved LoRA module.

[0065] In this application, the processing procedure is essentially a course-based regularization scheduling strategy, which dynamically changes the first weight. (Gradient magnitude balancing) and the second weight The value of (gated L2) is used to control the actual participation of the "improved LoRA module" in each training step.

[0066] In the initial stage, increasing the first and second weights results in a strong regularization that forces the L2 term of g to approximately zero. The gradient balancing term further reduces the gradients of A and B, thus the LoRA increment Δ is approximately zero. The improved LoRA module contributes almost no features in the forward pass and receives almost no gradient updates in the backward pass. The training signal is entirely handled by the frozen backbone network, allowing the model to learn to utilize pre-trained general representations first, avoiding early overfitting.

[0067] During the transition phase, the first and second weights are gradually reduced (the first and second weights can be reduced gradually according to a set step size). The regularization intensity decays synchronously, causing the gating g to gradually loosen, and the gradient begins to flow heavily to A and B. As a result, the weight of Δ increases with the increase of g, and LoRA low-rank adaptation begins to explicitly participate in the forward output. At the same time, with the reduction of gradient balancing loss, the update magnitudes of A and B tend to be consistent, reducing conflicts. The backbone features and LoRA corrections transition from "complete isolation" to "cooperative expression," achieving smooth knowledge injection.

[0068] The training process also includes: In the final stage, the first and second weights are adjusted to gradually approach 0, the two gating regularizations almost disappear, the gating g≈1, and LoRA gains full freedom. The model enters a "free fine-tuning" state, which can make full use of the backbone generalization and accurately fit the downstream task with a very small number of parameters.

[0069] In this application, all linear layers share a single loss mechanism (main task + gradient equalization + gated L2), which is optimized end-to-end within a relatively small trainable parameter space; curriculum-based decay is employed. , This allows the model to smoothly transition from "almost frozen" to "fully fine-tuned," balancing early generalization with later fitting.

[0070] Optionally, the pre-trained model also includes a hybrid adapter module, and the method further includes: The hybrid adapter module freezes the original parameters corresponding to each linear layer in the pre-trained model, allowing training only on the pre-trained model. Model parameters, etc.

[0071] Here, the original parameters refer to the model parameters corresponding to each linear layer, which can be the original weight matrix W0 mentioned earlier. The hybrid adapter module is a miniature plugin that simultaneously contains LoRA low-rank increments and gradient path modulation gating. It is inserted into the bypass of a specified linear layer in the Transformer, and its structure is as follows: Δ = α·B·A (LoRA low-rank increment) (Gated, 0~1) Output = h + g⊙Δ Among them, A, B, U, V, and b_g are the only trainable parameters, while the remaining weights remain frozen. This module performs both "low-rank adaptation" and "gradient flow adjustment", hence it is called a "hybrid adapter".

[0072] Experiments show that, under low-resource settings (100 samples per class) on the GLUE benchmark, this invention outperforms the standard LoRA by an average of 3.8 percentage points while keeping the total fine-tuning parameters below 0.5%, and improves the training convergence speed by approximately 22%.

[0073] The core difference between this invention and the prior art lies in the following: 1. At the model structure level, a learnable gradient path modulation gating mechanism is proposed and embedded inside the LoRA module to dynamically adjust the gradient flow intensity of the adapter parameters during backpropagation, thereby achieving adaptive response to different input samples or task stages.

[0074] 2. At the optimization mechanism level, a loss regularization term based on gradient magnitude normalization is designed to coordinate the gradient scale among multiple LoRA modules, alleviate gradient conflicts, and improve the stability of joint optimization.

[0075] 3. At the training strategy level, a phased course fine-tuning mechanism is introduced, combined with the state evolution of the gradient modulation gate, to achieve a smooth transition from "strong regularization" to "free fine-tuning", thereby enhancing the model's generalization ability under low resource conditions.

[0076] Through the above improvements, the present invention achieves the following objective improvement effects: 1. Under the same parameter budget, compared with the standard LoRA method, it improves the performance by an average of 2.3–5.7 percentage points on multiple low-resource text classification and question answering tasks; 2. Significantly reduces gradient variance during training (average decrease of 41%), improving optimization stability; 3. It can still maintain good convergence and generalization ability even with extremely low resource settings using only 100 training samples; 4. Supports flexible configuration of modulation intensity, and the modulation module can be turned off during the inference stage to further compress the model size.

[0077] Therefore, this invention significantly improves the performance and stability of the efficient parameter fine-tuning method in low-resource scenarios through the synergistic design of structural innovation and optimization mechanism, and has good prospects for industrial application.

[0078] Based on and Figure 1 Using the same principle as the method shown, this embodiment of the invention also provides a model optimization device 20 for low-resource scenarios, such as... Figure 2 As shown, the model optimization device 20 for low-resource scenarios may include a first acquisition module 210, an addition module 220, a second acquisition module 230, and a training module 240, wherein: The first acquisition module 210 is used to acquire the pre-trained model, which is a Transformer model. For any linear layer in the Transformer model, a pair of low-rank matrices are connected in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. Add module 220 to add a learnable gate to the LoRA incremental path for any linear layer in the Transformer model, forming an improved LoRA module; The second acquisition module 230 is used to acquire training data. The number of samples in the training data is less than a set value. The training data includes multiple sample texts and the annotation results corresponding to each sample text. The training module 240 is used to train the pre-trained model based on the training data to obtain the target model, and to perform semantic analysis on the text to be processed based on the target model. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

[0079] Optionally, when the training module 240 trains the pre-trained model based on the training data to obtain the target model, it is specifically used for: By extracting features from the training data through the frozen backbone network in the pre-trained model, the initial features corresponding to each sample text are obtained. The text features corresponding to each sample text are input into a linear layer of the pre-trained model, and the LoRA increment corresponding to each sample text is determined through the LoRA increment path in the linear layer. For each sample text, the increment is scaled element-wise according to the LoRA increment corresponding to the sample text and the learnable gate in the linear layer to obtain the scaled increment. The scaled increment is then added to the initial feature corresponding to the sample text to obtain the intermediate feature. The pre-trained model is trained based on the intermediate features corresponding to all sample texts to obtain the target model.

[0080] Optionally, the total loss value corresponding to the pre-trained model includes the main task loss value, the gradient magnitude equalization loss value, and the L2 regularization of the gating parameters. The main task loss value represents the difference between the prediction result and the labeled result output by the pre-trained model. The gradient magnitude equalization loss value represents the difference between the gradient magnitudes of two low-rank matrices within the same improved LoRA module. The L2 regularization of the gating parameters represents the distance between the gating parameters and the zero vector. The model parameters of the pre-trained model include the gating parameters.

[0081] Optionally, the gradient magnitude equalization loss value corresponds to the first weight, and the L2 regularization of the gating parameter corresponds to the second weight. During the training process of the pre-trained model, the device further includes: The phase training module is used to increase the first weight and the second weight in the initial phase so that the features of the training data include the features extracted by the backbone network but not the features extracted by the improved LoRA module; in the transition phase, the first weight and the second weight are decreased so that the features of the training data also include the features extracted by the improved LoRA module.

[0082] Optionally, the pre-trained model also includes a hybrid adapter module, and the above-mentioned device further includes: The freeze module is used to freeze the original parameters corresponding to each linear layer in the pre-trained model via the hybrid adapter module.

[0083] The low-resource scenario model optimization device of this invention can execute the low-resource scenario model optimization method provided in this invention. The implementation principle is similar. The actions performed by each module and unit in the low-resource scenario model optimization device in each embodiment of this invention correspond to the steps in the low-resource scenario model optimization method in each embodiment of this invention. For detailed functional descriptions of each module of the low-resource scenario model optimization device, please refer to the descriptions in the corresponding low-resource scenario model optimization methods shown above, which will not be repeated here.

[0084] The aforementioned low-resource scenario model optimization device can be a computer program (including program code) running on a computer device, for example, the low-resource scenario model optimization device is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0085] In some embodiments, the model optimization device for low-resource scenarios provided in this invention can be implemented using a combination of hardware and software. As an example, the model optimization device for low-resource scenarios provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the model optimization method for low-resource scenarios provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0086] In other embodiments, the model optimization device for low-resource scenarios provided in this invention can be implemented in software. Figure 2 A model optimization apparatus for low-resource scenarios stored in memory is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including a first acquisition module 210, an addition module 220, a second acquisition module 230, and a training module 240, for implementing the model optimization method for low-resource scenarios provided in the embodiments of the present invention.

[0087] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0088] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0089] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0090] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0091] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0092] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0093] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0094] Among these, electronic devices can also be terminal devices. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0095] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0096] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0097] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0101] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A model optimization method for low-resource scenarios, characterized in that, include: Obtain a pre-trained model, which is a Transformer model. For any linear layer in the Transformer model, connect a pair of low-rank matrices in parallel with the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. For any linear layer in the Transformer model, a learnable gate is added to the LoRA incremental path to form an improved LoRA module; Acquire training data, wherein the number of samples in the training data is less than a set value, and the training data includes multiple sample texts and the annotation results corresponding to each sample text; The pre-trained model is trained based on the training data to obtain a target model, which is then used to perform semantic analysis on the text to be processed. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

2. The method according to claim 1, characterized in that, The step of training the pre-trained model based on the training data to obtain the target model includes: The training data is used to extract features from the frozen backbone network in the pre-trained model to obtain the initial features corresponding to each sample text. The text features corresponding to each sample text are input into a linear layer of the pre-trained model, and the LoRA increment corresponding to each sample text is determined through the LoRA increment path in the linear layer. For each sample text, the increment is scaled element-wise according to the LoRA increment corresponding to the sample text and the learnable gate in the linear layer to obtain the scaled increment. The scaled increment is then added to the initial feature corresponding to the sample text to obtain the intermediate feature. The pre-trained model is trained based on the intermediate features corresponding to all sample texts to obtain the target model.

3. The method according to claim 1, characterized in that, The total loss value corresponding to the pre-trained model includes the main task loss value, the gradient magnitude equalization loss value, and the L2 regularization of the gating parameter. The main task loss value represents the difference between the prediction result and the labeled result output by the pre-trained model. The gradient magnitude equalization loss value represents the difference between the gradient magnitudes of two low-rank matrices within the same improved LoRA module. The L2 regularization of the gating parameter represents the distance between the gating parameter and the zero vector. The model parameters of the pre-trained model include the gating parameter.

4. The method according to claim 3, characterized in that, The gradient magnitude equalization loss value corresponds to the first weight, and the L2 regularization of the gating parameter corresponds to the second weight. During the training process of the pre-trained model, the method further includes: In the initial stage, the first weight and the second weight are increased so that the features of the training data include features extracted by the backbone network but exclude features extracted by the improved LoRA module. During the transition phase, the first and second weights are reduced so that the features of the training data also include features extracted by the improved LoRA module.

5. The method according to any one of claims 1 to 3, characterized in that, The pre-trained model also includes a hybrid adapter module, and the method further includes: The hybrid adapter module freezes the original parameters corresponding to each linear layer in the pre-trained model.

6. A model optimization device for low-resource scenarios, characterized in that, include: The first acquisition module is used to acquire a pre-trained model, which is a Transformer model. For any linear layer in the Transformer model, a pair of low-rank matrices are connected in parallel to the linear layer to form a LoRA incremental path. The backbone network in the pre-trained model is frozen. An additional module is added to the LoRA incremental path for any linear layer in the Transformer model, forming an improved LoRA module. The second acquisition module is used to acquire training data, wherein the number of samples in the training data is less than a set value, and the training data includes multiple sample texts and the annotation results corresponding to each sample text. The training module is used to train the pre-trained model based on the training data to obtain a target model, and to perform semantic analysis on the text to be processed based on the target model. The model parameters adjusted during the training process of the pre-trained model are the model parameters corresponding to the network layers in the pre-trained model, excluding the frozen backbone network.

7. The apparatus according to claim 6, characterized in that, When the training module trains the pre-trained model based on the training data to obtain the target model, it is specifically used for: The training data is used to extract features from the frozen backbone network in the pre-trained model to obtain the initial features corresponding to each sample text. The text features corresponding to each sample text are input into a linear layer of the pre-trained model, and the LoRA increment corresponding to each sample text is determined through the LoRA increment path in the linear layer. For each sample text, the increment is scaled element-wise according to the LoRA increment corresponding to the sample text and the learnable gate in the linear layer to obtain the scaled increment. The scaled increment is then added to the initial feature corresponding to the sample text to obtain the intermediate feature. The pre-trained model is trained based on the intermediate features corresponding to all sample texts to obtain the target model.

8. The apparatus according to claim 6, characterized in that, The total loss value corresponding to the pre-trained model includes the main task loss value, the gradient magnitude equalization loss value, and the L2 regularization of the gating parameter. The main task loss value represents the difference between the prediction result and the labeled result output by the pre-trained model. The gradient magnitude equalization loss value represents the difference between the gradient magnitudes of two low-rank matrices within the same improved LoRA module. The L2 regularization of the gating parameter represents the distance between the gating parameter and the zero vector. The model parameters of the pre-trained model include the gating parameter.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images