Efficient parameter fine tuning method and system based on interleaving memory of twin large language model and application

Abstract

The invention discloses an efficient parameter fine tuning method based on interleaving memory of twin large language models. The method comprises the following steps: step 1, pre-constructing two large language models sharing the same structure and pre-training parameters; wherein one large language model keeps original parameters frozen, and the other large language model is finely adjusted by using an efficient parameter fine adjustment method; 2, respectively processing given input by using the two large language models constructed in the step 1, and respectively generating different memories; the memory refers to the last layer of hidden state generated by the large language model; and step 3, introducing a twinning large language model interleaving memory mechanism, and adjusting the contribution of the memory of the large language model with frozen original parameters and the large language model finely adjusted by using the efficient parameter fine adjustment method to the generation of the next lexical element. According to the method, excellent performance is achieved on downstream tasks, the problem of disastrous forgetting is effectively solved, and extra fine adjustment time does not need to be increased.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and technology, and relates to a method, system and application for efficient parameter fine-tuning based on interleaved memory of a twin large language model. Background Art

[0002] Parameter-Efficient Fine-Tuning (PEFT) refers to a technique that reduces the amount of parameter updates when fine-tuning a large language model (LLM) to reduce computational costs and resource consumption. This method can be divided into four categories: [1] : Additive, selective, reparameterized and hybrid methods. Additive PEFT, such as prefix-tuning [2] and(IA)3 [3] , by introducing additional modules into the large language model to adapt to specific downstream tasks. Selective PEFT, such as BitFit [4] , focusing on selecting a small set of parameters from the base model for fine-tuning. Reparameterizing PEFT [5] The method aims to solve the problem that additive methods increase the depth of the model, which leads to reasoning delays. [6] Aims to combine different PEFT methods to provide a comprehensive solution. PEFT provides a promising alternative to Full Fine-Tuning (FFT), which can achieve comparable or even better performance while requiring less time and resources. Current PEFT methods allow aligning LLM to new domains at the parameter level. However, modifying or adding parameters may destroy LLM's existing world knowledge and reasoning ability. This will not only affect LLM's performance on other tasks, but also lead to a performance crisis when processing inputs from new domains.

[0003] LLM faces the challenge of catastrophic forgetting during both pre-training and fine-tuning, i.e., the model tends to forget previously acquired knowledge after fine-tuning. Empirical studies have shown that even efficient fine-tuning techniques such as LoRA are affected by catastrophic forgetting. One of the scenarios explored to address the problem of catastrophic forgetting is the continuous learning scenario. Several effective strategies have been proposed to address this challenge, including constrained parameter updates, data replay, and parameter isolation. However, for LLM, obtaining extensive pre-training data is an expensive task. In addition, it is also challenging to ensure that the existing knowledge learned by LLM is not overwritten at the parameter level. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a method, system and application for efficient parameter fine-tuning based on interleaved memory of a twin large language model, which can improve the effect of efficient parameter fine-tuning and alleviate catastrophic forgetting caused by fine-tuning.

[0005] The present invention discloses a simple and effective parameter-efficient fine-tuning (PEFT) method, namely, interweaving memories of Siamese large language model (IMSM). This is a model-independent framework that can theoretically be applied to all open source large language models (LLM) and existing PEFT methods. Specifically, the twin LLM can be regarded as two LLMs that share the same structure and pre-trained parameters, one of which remains frozen and the other performs PEFT. Given an input, the twin large language model generates different memories, namely the last layer of hidden states, and then IMSM introduces an interleaving mechanism to regulate the contribution of the memories of the frozen LLM and the fine-tuned LLM to the generation of the next word. This paper proposes two memory interleaving mechanisms: a simple interleaving mechanism and a gate-based interleaving mechanism. Extensive experiments were conducted to evaluate IMSM compared with state-of-the-art PEFT methods such as LoRA, (IA) 3 , and AdaLoRA. Three popular LLMs were used as the backbone large models, including ChatGLM3, Qwen1.5, and Llama3, and fine-tuned on three datasets. The results show that IMSM not only achieves excellent performance on downstream tasks, but also effectively alleviates the catastrophic forgetting problem without adding additional fine-tuning time.

[0006] The present invention provides a method for efficient parameter fine-tuning based on interleaved memory of a twin large language model, the method comprising the following steps:

[0007] Step 1: Pre-build two large language models that share the same structure and pre-trained parameters; one of the large language models keeps the original parameters frozen, and the other large language model is fine-tuned using the PEFT method;

[0008] Step 2: using the two large language models constructed in step 1 to process the given inputs respectively, and generate different memories respectively; the memory refers to the last hidden state generated by the large language model;

[0009] Step 3: Introduce the twin large language model interleaved memory mechanism to adjust the contribution of the memory of the large language model with frozen original parameters and the large language model fine-tuned using the PEFT method to the generation of the next word.

[0010] In step 1, the parameters are updated by adding an adapter to the original parameters to obtain a fine-tuned large language model;

[0011] The update of the parameters is expressed by the following formula:

[0012] W=W o +△W=W o +BA,

[0013] Among them, W o are the original parameters, kept frozen, and and represents a trainable low-rank matrix, and r' is much smaller than d and k; △W represents the adapter, and W represents the parameters of the large language model after fine-tuning.

[0014] In step 3, the twin large language model interleaving memory mechanism includes a simple memory interleaving mechanism and a gate-based memory interleaving mechanism;

[0015] The simple memory interleaving mechanism directly adds the logical values ​​of the frozen large language model and the fine-tuned large language model to calculate and generate the probability distribution of the next word unit;

[0016] For each time step t, access the logical values ​​output by the frozen large language model and the fine-tuned large language model respectively. The logical value refers to the non-normalized score of each word in the vocabulary set V as the next word, which is used to reflect the model's confidence level in predicting the next word.

[0017] Based on input query x and response y <t , calculate the final logical value:

[0018]

[0019] Among them, x and y belong to the vocabulary set V, and Respectively and The resulting logical value is Represents the original parameter W o The frozen large language model, while Represents the fine-tuned large language model after efficient fine-tuning of the parameters of the adapter △W.

[0020] The gate-based memory interleaving mechanism splices the last hidden states of the frozen large language model and the fine-tuned large language model, and uses a low-rank linear layer to generate a gate value to dynamically balance the contribution of the two models to the next generation, thereby optimizing the performance of the model;

[0021] At each time step, access and The last hidden state of the layer is connected along the feature dimension. A low-rank linear layer is constructed on the top of the model to generate a gate, which determines the contribution of each large language model LLM in the next step of the logical value generation process, which is expressed as follows:

[0022]

[0023] Among them, ⊕ represents vector concatenation, f represents a low-rank linear layer, and Represent the last hidden state of freezing the large language model and fine-tuning the large language model, respectively. Where d represents the dimension of the hidden layer state. In addition, W A and W B are the weight matrices of the linear layer, respectively; where r is a hyperparameter. It is worth noting that r is much smaller than d to ensure the number of parameters is limited. In addition, the gate is obtained through sigmoid activation.

[0024] Last hidden state after update And the logits can be calculated as follows:

[0025]

[0026]

[0027] Among them, ° represents element-by-element multiplication, W out represents the parameters of the linear layer of the twin large language model used to predict the next word. In step 3, the probability distribution of the next word is expressed as follows:

[0028] p(y t |x,y <t )=softmax(logits).

[0029] In the present invention, the parameter adapter △W in the fine-tuning large language model and the weight matrix W of the linear layer in the memory interleaving mechanism of the twin large language model based on gating are A and W B Optimization is performed using the cross entropy loss function:

[0030]

[0031] Where T is the length of the output sequence.

[0032] The present invention also provides an adjustment system for implementing the above-mentioned parameter efficient fine-tuning method, the adjustment system comprising: a twin large language model generation module, a memory interleaving module, and a fine-tuning module;

[0033] The twin large language model generation module is used to generate two twin large language models that share the same structure and pre-trained parameters;

[0034] The memory interleaving module is used to adjust the contribution of the memory of the frozen model and the fine-tuned model to the generation of the next word;

[0035] The fine-tuning module is used to efficiently fine-tune the parameters of one of the twin large language models.

[0036] The present invention also provides applications of the above-mentioned efficient parameter fine-tuning method or adjustment system in domain specialization, machine translation, text summarization, etc.

[0037] Time and space complexity of the present invention: Compared with using only ordinary PEFT methods, IMSM performs comparable in space and time complexity. In theory, despite the need to output two different memories, the twin LLM does not require double the storage resources. Although the gate-based memory interleaving mechanism introduces some additional trainable parameters, these effects are minimal compared to the original PEFT parameters. For example, in ChatGLM with a hidden layer dimension of 4096, when the ranks of LoRA and the gate are both 16, the trainable parameters of the gate account for only 5.04% of LoRA. In addition, the two memories can be generated in parallel. During fine-tuning and inference, the time required to interleave the two memories at each step remains constant. This ensures that IMSM is comparable to existing PEFT methods in time complexity.

[0038] The beneficial effects of the present invention include: the method of the present invention generates different memories (i.e., the last hidden state) by keeping one in a frozen state and fine-tuning the other in two twin large language models that share the same structure and pre-trained parameters. Then, through a simple memory interleaving mechanism or a gating-based memory interleaving mechanism, the memories of the two models are combined to dynamically balance their contributions to the generation of the next word unit, thereby improving performance in downstream tasks while maintaining the original knowledge of the model. In the present invention, the IMSM with two interleaving mechanisms outperforms the ordinary fine-tuning method to varying degrees, with average improvements of 1.53% and 2.21%, respectively. In a specific embodiment, the two memory interleaving mechanisms of the present invention both show better performance for catastrophic forgetting on non-target datasets. Among them, the simple IMSM reduces LoRA forgetting by 1.90, while the gate-based IMSM reduces it by 3.70. For AdaLoRA, both mechanisms reduce it by 1.86. For (IA) 3, simple IMSM reduces forgetting by 1.11, and gated IMSM reduces forgetting by 1.29. In different tasks, the IMSM method achieves a good balance between stability and adaptability by adjusting the rank of the gate, ensuring optimal performance and minimal knowledge forgetting. At the same time, the method of the present invention shows high adaptability in various tasks and PEFT frameworks, and can flexibly adjust parameters to achieve the best effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is a schematic diagram comparing the efficient fine-tuning of common parameters of the present invention and the interleaved memory of a simple twin large language model.

[0041] Figure 2 It is a schematic diagram of the memory interleaving mechanism of the twin large language model based on gating of the present invention.

[0042] Figure 3 Schematic diagram of the effect of the rank of the gate of the present invention on three data sets.

[0043] Figure 4 It is a schematic diagram of the result of summing the logical values ​​of three data sets at different ratios according to the present invention.

[0044] Figure 5 It is a schematic diagram of knowledge forgetting of different ranks in the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0046] The present invention proposes a model-independent framework called Interweaving Memories of Siamese Large Language Model (IMSM), and develops a parameter-efficient fine-tuning method based on the framework. Twin LLM can be regarded as two LLMs with the same size, structure and pre-trained parameters, such as Llama3. One of them keeps the original parameters frozen, while the other will be fine-tuned using the PEFT method. The frozen LLM and the fine-tuned LLM process the same input word at the same time, resulting in different memories, namely the last layer of hidden states. Finally, two memory interleaving mechanisms are proposed: a simple interleaving mechanism or a gate-based interleaving mechanism. The next word of the response output comes from the interleaved memory. In this framework, the parameters related to PEFT and the parameters related to the gate mechanism will be fine-tuned using downstream task data, while the remaining parameters will be frozen. Specifically, the parameters related to PEFT include a small part of the parameters adjusted during the fine-tuning process, such as the low-rank matrix introduced in a specific layer; the parameters related to the gate mechanism include the linear layer parameters used in the gating mechanism. The goal of this invention is to maintain the world knowledge and general reasoning ability inherent in the model while improving its performance in downstream tasks. The proposed method of using interweaving memories of Siamese Large Language Model (IMSM) can be applied to various open source LLM and PEFT methods.

[0047] 1. Task definition:

[0048] The main goal of the IMSM proposed in this invention is to control the output generated by the LLM. From a mathematical point of view, the language model aims to model the probability distribution of natural language:

[0049]

[0050] Among them, P(x) represents the joint probability of generating sequence x, x i represents the i-th token in the sequence, and W represents the parameters of the language model;

[0051] In order to determine the next token in the generation process, multiple strategies can be adopted. A popular approach is greedy decoding, that is, the model selects the token with the highest probability:

[0052]

[0053] Currently, most LLMs are autoregressive models that rely mainly on a decoder framework. During the generation process, LLMs take into account the input and previously generated tokens. The proposed IMSM is able to interleave two different memories before each generation step. Then, the probability of the next token is predicted based on this updated memory.

[0054] 2. Twin Large Language Model

[0055] The twin LLM can be viewed as two LLMs that share the same Transformer structure and pre-trained parameters. When using the PEFT method, one of them remains frozen to preserve the original knowledge, while the other is fine-tuned with downstream data to align with the new target. Taking LoRA as an example, by adding the original parameters Add an adapter △W to update the parameter W:

[0056] W=W o +△W=W o +BA

[0057] Among them, W o Remain frozen, while and represents a trainable low-rank matrix. And r' is much smaller than d and k. This configuration greatly reduces the number of trainable parameters. o and W denote the parameters of the frozen LLM and the fine-tuned LLM, respectively.

[0058] 3. Memory interleaving mechanism

[0059] During fine-tuning and reasoning, the twin LLM proposed in the present invention generates different hidden states and corresponding logical values ​​according to the input and its generated word-units. The last hidden layer state can be used as a feature representation to encapsulate the LLM's memory of previous inputs. In addition, the logical value represents the model's confidence level in predicting the next word-unit. The present invention uses two methods, a simple memory interleaving mechanism and a gate-based memory interleaving mechanism, to interleave different memories together.

[0060] make Indicates that the parameter W o The frozen LLM Denotes a fine-tuned LLM with a partial fine-tuning parameter △W. Two memory interleaving mechanisms are proposed: a simple memory interleaving mechanism and a gate-based interleaving mechanism.

[0061] Mode 1: Simple memory interleaving mechanism

[0062] At each time step t, we can access and The logical value of the output of is the non-normalized score of each word in the vocabulary set V as the next word. Then, based on the input query x and the response y <t , the final logical value can be calculated:

[0063]

[0064] Among them, x and y belong to the vocabulary set V, and Respectively and The resulting logical value.

[0065] The vocabulary V is the set of all possible tokens in the language model. In the simple memory interleaving mechanism, the frozen large language model and the fine-tuned large language model calculate the logical value of each token in the vocabulary based on the input and previously generated tokens. By adding these logical values, the model can predict and generate the next token.

[0066] Note that for the prediction of the next token, directly adding two hidden states has the same effect as adding two logical values. Therefore, for the sake of clarity and intuition, this strategy is explicitly described as the addition of two logical values.

[0067] Mode 2: Gating-based memory interleaving mechanism

[0068] Considering that the logic value is calculated from the last hidden state, the last hidden state can capture more potential information. The present invention also provides an alternative memory interleaving mechanism for the twin LLM, namely, a gate-based memory interleaving mechanism. At each time step, you can access and The last layer of hidden states of , these hidden states are connected along the feature dimension. A low-rank linear layer is constructed on top of the model to generate gates that determine the contribution of each LLM in the next step of the logical value generation process:

[0069]

[0070] Among them, ⊕ represents vector concatenation, f represents a low-rank linear layer, and They represent the last hidden state of the frozen LLM and the fine-tuned LLM, respectively. Where d represents the dimension of the hidden layer state. In addition, Among them, r is a hyperparameter. It is worth noting that r is much smaller than d to ensure the number of parameters is limited. In addition, the gate is obtained through sigmoid activation.

[0071] Last hidden state after update And the logits can be calculated as follows:

[0072]

[0073]

[0074] Among them, ° represents element-by-element multiplication, W out represents the parameters of the twin LLM linear layer used to predict the next token.

[0075] 4. Model training and inference

[0076] Finally, the probability distribution of the next token can be calculated as follows:

[0077] p(y t |x,y <t )=softmax(logits)

[0078] In the fine-tuning stage, the cross entropy loss function is used to optimize the parameters △W,W A and W B :

[0079]

[0080] Where T is the length of the output sequence.

[0081] Figure 2 Schematic diagram of the memory interleaving mechanism of the twin large language model based on gate control in the present invention. In the figure,

[0082] 1. Input:

[0083] The input text sequence is fed into both the frozen large model and the fine-tuned large model.

[0084] 2. Twin Large Language Model:

[0085] The figure contains two models: the frozen large model and the fine-tuned large model. Both have the same structure and pre-trained parameters, but the fine-tuned large model has additional parameters that are efficiently fine-tuned.

[0086] 3. The last hidden state:

[0087] Freezing the large model and fine-tuning the large model generate their last hidden states respectively. These hidden states contain the model's understanding and memory of the input sequence.

[0088] 4. Hidden state after splicing:

[0089] The last hidden state of the frozen large model and the fine-tuned large model is concatenated along the feature dimension to form a concatenated hidden state.

[0090] 5. Linear Layer:

[0091] The concatenated hidden state passes through two linear layers A and B to generate the input of the gating mechanism. Both linear layers A and B are fine-tunable parameters.

[0092] 6. Gating mechanism:

[0093] The gating mechanism generates a gate value through the sigmoid function to determine the contribution ratio of the frozen model and the fine-tuned model to the final output. The specific formula is as follows:

[0094]

[0095] Among them, σ represents the sigmoid function, ⊕ represents the concatenation operation, and W A and W B are the weight matrices of the linear layer respectively;

[0096] 7. Updated hidden status:

[0097] Based on the generated gate value, the updated hidden state is calculated, which combines the memories of the two together:

[0098]

[0099] Among them, ° represents element-by-element multiplication, W out Represents the parameters of the linear layer of the Twin Large Language Model used to predict the next token.

[0100] 8. Output Generation:

[0101] Finally, the updated hidden state is used to generate the probability distribution of the next token, and the final output is generated based on the probability distribution.

[0102] In the inference phase, as shown in Algorithm 1, the present invention uses a greedy search strategy to generate the output sequence. The token with the highest probability is selected as the next generated token, added to the input sequence, and then the process is repeated until the complete sequence is generated.

[0103]

[0104]

[0105] Effective results compared with existing technologies

[0106] 1. Dataset

[0107] Experiments are conducted on three benchmark datasets to evaluate the alignment capability of the proposed IMSM in downstream tasks, as well as its general reasoning ability in other tasks.

[0108] The MRPC dataset aims to determine whether two given sentences have the same meaning.

[0109] CoLA is a dataset on language annotations designed to evaluate the ability of models to identify grammatical acceptability in sentences.

[0110] ROPES is a reading comprehension dataset designed to evaluate the model’s ability to make multi-step inferences about new situations.

[0111] 2. Evaluation indicators

[0112] Different metrics are used for different downstream tasks. For MRPC, Accuracy (Acc.) and F1 are usually used as indicators. For CoLA, Matthews Correlation Coefficient (MCC) is used as the main indicator. For ROPES, Exact Match (EM) and F1 token overlap between answer text and gold are used as indicators.

[0113] 3. Benchmark PEFT Method

[0114] The following PEFT method was utilized as a baseline and the proposed IMSM method was applied.

[0115] LoRA: It freezes the pre-trained model weights and incorporates removable plugins (trainable rank factorization matrices) into the layers of the Transformer architecture.

[0116] (IA) 3 : This method uses the learned vectors to rescale the internal activations and has fewer trainable parameters than LoRA.

[0117] AdaLoRA: It adaptively adjusts the ranks of different modules according to the importance of the weight matrix.

[0118] 4. Performance comparison

[0119]

[0120]

[0121] The table above shows the results of IMSM and baseline models on three datasets. Overall, the fine-tuned LLM performs better than the LLM enhanced only by prompts. This highlights the importance of fine-tuning in aligning LLM to unfamiliar tasks compared to contextual learning. IMSM with two interleaving mechanisms outperforms the ordinary fine-tuning method to varying degrees, with average improvements of 1.53% and 2.21% respectively. In most cases, the interleaving method based on the gating mechanism performs better than the simple interleaving method.

[0122] The effectiveness of fine-tuning is affected by multiple factors, including the complexity of the natural language task, the reasoning ability of the original LLM, and the adaptability of the adopted fine-tuning method. Specifically, the improvements observed on the MRPC dataset are relatively small. MRPC focuses on determining the semantic similarity between two sentences and relies more on general language capabilities rather than highly specialized downstream models. The effectiveness of fine-tuning is closely related to the quality and capabilities of the base LLM. As one of the most advanced open source LLMs, the LLama3-based model almost achieves the best performance. In addition, LoRA includes low-rank decomposition, highlighting its effectiveness in aligning with downstream tasks.

[0123] However, IMSM, as a simple and powerful model-agnostic framework, excels in combining the stability of the original LLM with the adaptability of the fine-tuned LLM. In addition, IMSM introduces a gating mechanism that allows selective control of whether the understanding comes from the original knowledge or the adjusted knowledge. As a result, it achieves consistent improvements in various downstream tasks.

[0124] 5. Catastrophic Forgetting Verification

[0125]

[0126] Fine-tuning inevitably introduces a certain degree of forgetting. The present invention fine-tunes LLMs on one dataset and then evaluates their performance on other datasets. In particular, ChatGLM3-6B is used as the backbone model in a specific embodiment of the present invention, and then fine-tuned on three datasets using the existing PEFT method and IMSM. For each target task, two datasets for evaluating forgetting are introduced. The first dataset shared by all tasks is a context-free question-answering dataset WebQ, which is used to evaluate the degree of forgetting of world knowledge. The present invention evaluates performance on WebQ by checking whether the generated response contains the golden answer, rather than strictly requiring an exact match. The second dataset varies depending on the target task. For simplicity, CoLA, MRPC, and MultiRC are used respectively.

[0127] From the above table, we can see that the three common PEFT methods all prioritize adaptability and ignore stability. 3 After fine-tuning MRPC on CoLA, the performance of CoLA drops from 43.99 to 29.54, 28.46, and 35.86, respectively. Similarly, the performance on WebQ drops from 25.49 to 24.02, 22.19, and 23.52. Overall, LoRA exhibits the highest degree of catastrophic forgetting, followed by AdaLoRA and (IA) 3This can be attributed to the number of parameters of the fine-tuning process. The fewer parameters involved, the less impact it has on the general capabilities of the model.

[0128] The IMSM method in the present invention can effectively mitigate the performance degradation of other datasets caused by fine-tuning. On average, compared with ordinary PEFT, the two memory interleaving mechanisms of the present invention show better performance on non-target datasets. Among them, the simple IMSM reduces LoRA forgetting by 1.90, while the gate-based IMSM reduces it by 3.70. For AdaLoRA, both mechanisms reduce it by 1.86. For (IA) 3 , simple IMSM reduces forgetting by 1.11, and gate-based IMSM reduces forgetting by 1.29.

[0129] By summing the logical values ​​from the frozen and fine-tuned LLMs, the extent to which the probability of the initially correct token on the non-target dataset decreases due to fine-tuning can be mitigated. The last layer of hidden states serves as a memory of the LLM’s understanding of the input sequence. By considering the memory from the original and fine-tuned LLMs, the twin LLM is ensured to maintain a balance between its prior knowledge and adaptation to new data. In addition, it shows better performance in mitigating catastrophic forgetting compared to the gate-based interleaving mechanism. By dynamically balancing the knowledge of the frozen model and the fine-tuned model, the gate mechanism can effectively recall previous memories, helping the twin LLM prevent overfitting to new data while retaining the model’s original knowledge.

[0130] 6. Hyperparameter Experiments

[0131] Figure 3 The effect of the gate rank on three datasets is shown. The present invention evaluates the performance of MRPC and CoLA using gate ranks of 2, 4, 8, and 16, and evaluates the performance of ROPES using gate ranks of 4, 8, 16, and 32. First, all selected ranks exceed or match the corresponding ordinary PEFT performance. The best rank varies for different PEFT methods and tasks. This emphasizes the effectiveness and adaptability of IMSM in various PEFT frameworks and tasks. For MRPC, the overall performance improves with the increase of rank. Specifically, the peak performance of LoRA, (IA)3, and AdaLoRA is achieved at ranks 8, 16, and 16, respectively. For CoLA, the performance reaches its peak when the rank is 8. For ROPES, it is obvious that higher ranks actually tend to reduce performance. This suggests that lower ranks may be more conducive to improving the performance of these tasks.

[0132] Figure 4Results of summing logical values ​​in different ratios for the three datasets are shown. The results show that satisfactory performance is achieved when the logical values ​​of frozen knowledge and fine-tuned knowledge in the twin LLM are added in equal proportions. For example, when the LLM is fine-tuned on the CoLA dataset using the LoRA-based IMSM method, the performance reaches 65.53 when the logical values ​​of the frozen model and the fine-tuned model are combined in a 1:1 ratio. However, when the ratio is changed to 2:3 or 3:2, the performance drops significantly, to 63.81 and 63.70, respectively. This result highlights the potential benefits of leveraging the pre-trained knowledge stored in the frozen LLM and the new knowledge obtained from the fine-tuned LLM. By combining these memories in a balanced manner, the IMSM shows enhanced performance.

[0133] In addition, the present invention also tests the knowledge forgetting of different ranks, such as Figure 5 As shown, from left to right, they are based on LoRA, (IA) 3 and IMSM of AdaLoRA. We use MRPC as the target dataset and report the average performance degradation of the other two datasets. When the gate rank is 8, the performance of the IMSM based on LoRA and (IA) 3 The forgetfulness of IMSM based on AdaLoRA is the lowest, while when the rank is 16, the forgetfulness of IMSM based on AdaLoRA is the lowest. In addition, considering the hyperparameter experimental results of different ranks on the performance of downstream tasks, the method of the present invention achieves a trade-off in stability and adaptability, and even the best performance. In addition, when the rank of the gate is relatively low, such as 2, the degree of forgetfulness will be aggravated. Gates with lower ranks cannot capture the contribution of original knowledge and new knowledge.

[0134] Example 1

[0135] like Figure 1As shown, the user's input and the word unit generated by the model are "Please answer the following question. Question: What family of animals does sheep belong to? Answer: Sheep belongs to". For the model with efficient fine-tuning of ordinary parameters, after receiving the input, the model generates corresponding logical values ​​for the entire vocabulary word unit (including "sheep", "dog", "cow", etc.), and these logical values ​​represent the non-normalized score of each word unit as the next word unit. It is converted into a probability distribution through the softmax function, and the model makes a prediction based on the word unit with the highest probability. The model may obtain an incorrect predicted word unit, such as "sheep". For the twin large language model, the input word unit is simultaneously passed to a fine-tuned large model and a frozen large model, and they each generate a set of logical values. Then, the logical values ​​of the fine-tuned large model and the frozen large model are interleaved, as shown in the figure, and simply added to form a new set of logical values. The combined logical value is converted into a probability distribution through the softmax function, and the model makes a prediction based on the new probability distribution, thereby generating the correct predicted word unit "cow". The above steps are iterated repeatedly until the final end word unit is generated. Through the above method steps, the present invention can effectively improve the prediction accuracy of the model and achieve more accurate word unit prediction.

[0136] The present invention includes a twin large language model, which can be viewed as two LLMs that share the same structure and pre-trained parameters. The parameters of one of the models remain frozen, while the other model uses existing efficient parameter fine-tuning techniques to fine-tune a small number of parameters. The same output passes through the twin LLM to obtain two different logical values. The simplest way to interleave is to fuse the logical values ​​of the two models in a 1:1 ratio, and then use the softmax function to predict the next word. The model output and the original input are spliced ​​together, and the above process is iterated repeatedly until the entire sequence is generated. This method ensures the stability and original performance of the model by keeping the parameters of one model frozen, and provides flexibility and adaptability by fine-tuning a small number of parameters of the other model. The design of this twin large model helps to improve its performance on specific tasks while maintaining the original capabilities of the model.

[0137] References:

[0138] [1]Han, Z., Gao, C., Liu, J., Zhang, SQ, et al. (2024). Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608.

[0139] [2]Li,X.L.,&Liang,P.(2021).Prefix-tuning:Optimizing continuousprompts for generation.arXiv preprint arXiv:2101.00190.

[0140] [3]Bansal,M.,&Raffel,C.A.(2022).Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in NeuralInformation Processing Systems,35,1950-1965.

[0141] [4]Ben Zaken,E.,Goldberg,Y.,&Ravfogel,S.(2022).BitFit:Simpleparameter-efficient fine-tuning for transformer-based masked languagemodels.In Proceedings of the 60th Annual Meeting of the Association forComputational Linguistics(Volume 2:Short Papers)(pp.1–9).

[0142] [5]Hu,E.J.,Shen,Y.,Wallis,P.,Allen-Zhu,Z.,Li,Y.,Wang,S.,Wang,L.,&Chen,W.(2021).LoRA:Low-rank adaptation of large language models.arXivpreprint arXiv:2106.09685.

[0143] [6]Mao,Y.,Mathias,L.,Hou,R.,Almahairi,A.,Ma,H.,Han,J.,Yih,S.,&Khabsa,M.(2022).UniPELT: A unified framework for parameter-efficient language modeltuning.In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers)(pp.6253-6264).

[0144] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. An efficient parameter fine-tuning method based on interleaved memory of twin large language models, characterized in that: The steps include: Step 1: Pre-build two large language models that share the same structure and pre-trained parameters; one of the large language models keeps the original parameters frozen, and the other large language model is fine-tuned using a parameter-efficient fine-tuning method; Step 2: using the two large language models constructed in step 1 to process the given inputs respectively, and generate different memories respectively; the memory refers to the last hidden state generated by the large language model; Step 3: Introduce the twin large language model interleaved memory mechanism to adjust the contribution of the memory of the large language model with frozen original parameters and the large language model fine-tuned using the efficient parameter fine-tuning method to the generation of the next word.

2. The efficient parameter fine-tuning method according to claim 1, characterized in that: In step 1, the parameters are updated by adding an adapter to the original parameters to obtain a fine-tuned large language model; The update of the parameters is expressed by the following formula: W=W o +△W=W o +BA, Among them, W o are the original parameters, kept frozen, and and represents a trainable low-rank matrix, and r' is less than d and k; △W represents the adapter, and W represents the parameters of the large language model after fine-tuning.

3. The efficient parameter fine-tuning method according to claim 1, characterized in that: In step 3, the twin large language model interleaving memory mechanism includes a simple memory interleaving mechanism and a gate-based memory interleaving mechanism; The simple memory interleaving mechanism directly adds the logical values ​​of the frozen large language model and the fine-tuned large language model to calculate and generate the probability distribution of the next word unit; The gate-based memory interleaving mechanism optimizes the performance of the model by splicing the last hidden states of the frozen large language model and the fine-tuned large language model, and then using a low-rank linear layer to generate a gate value to dynamically balance the contribution of the two models to the next step of generation.

4. The efficient parameter fine-tuning method according to claim 3, characterized in that: In the simple memory interleaving mechanism, for each time step t, the logical values ​​output by the frozen large language model and the fine-tuned large language model are accessed respectively, and the logical value refers to the non-normalized score of each word in the vocabulary set V as the next word, which is used to reflect the confidence level of the model in predicting the next word; Based on input query x and response y <t , calculate the final logical value: Among them, x and y belong to the vocabulary set V, and Respectively and The resulting logical value is Represents the original parameter W o The frozen large language model, while Represents the large language model after efficient fine-tuning of the parameters of the adapter △W.

5. The efficient parameter fine-tuning method according to claim 3, characterized in that: In the gated memory interleaving mechanism, at each time step, access and The last hidden state of the model is connected along the feature dimension; a low-rank linear layer generation gate is constructed at the top of the model, which determines the contribution of each large language model in the next step of the logical value generation process, which is expressed as follows: in, represents vector concatenation, f represents a low-rank linear layer, and Represent the last hidden state of the frozen large language model and the fine-tuned large language model respectively; Where d represents the dimension of the hidden layer state; W A and W B are the weight matrices of the linear layer respectively; r is a hyperparameter; Last hidden state after update And the logits can be calculated as follows: in, represents element-by-element multiplication, W out Represents the parameters of the linear layer of the Twin Large Language Model used to predict the next token.

6. The efficient parameter fine-tuning method according to claim 1, characterized in that: In step 3, the probability distribution of the next word is expressed as follows: p(and t |x,y <t)=softmax(logits)。 7. The efficient parameter fine-tuning method according to claim 1, characterized in that: Fine-tuning the parameter adapter △W in the large language model, the weight matrix W of the linear layer in the memory interleaving mechanism of the gating-based twin large language model A and W B Optimization is performed using the cross entropy loss function: Where T is the length of the output sequence.

8. Application of the efficient parameter fine-tuning method as described in any one of claims 1 to 7 in domain specialization, machine translation, and text summarization.

9. An adjustment system for implementing the parameter efficient fine-tuning method as described in any one of claims 1 to 7, characterized in that: The adjustment system includes: a twin large language model generation module, a memory interleaving module, and a fine-tuning module; The twin large language model generation module is used to generate two twin large language models that share the same structure and pre-trained parameters; The memory interleaving module is used to adjust the contribution of the memory of the frozen model and the fine-tuned model to the generation of the next word; The fine-tuning module is used to efficiently fine-tune the parameters of one of the twin large language models.

10. Or the application of the parameter efficient fine-tuning system as claimed in claim 9 in domain specialization, machine translation, and text summarization.

Images