An adapter-based language model knowledge injection method and system

Abstract

The application discloses an adapter-based language model knowledge injection method and system, the language model is a language model based on a Transformer architecture, and belongs to the fields of efficient optimization and knowledge injection of language model parameters. The method comprises the following steps: obtaining text data to be injected with knowledge, performing word segmentation on the text data and mapping the text data into corresponding identifiers in a word table, and cutting the text data into subsequences; inserting an adapter module with a gating mechanism before at least one Transformer layer of a target language model; inputting the subsequences into the target language model containing the adapter module as training samples, predicting the subsequence identifiers according to the final hidden layer representation of the output, and updating the parameters of all the adapter modules; and taking the target language model containing the adapter module after training as the result after knowledge injection. The application can inject knowledge in a specific field into language models of various parameter scales, and improve the performance of the language model in downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of efficient optimization of language model parameters and knowledge injection, and in particular to an adapter-based method and system for knowledge injection into language models. Background Technology

[0002] With the upgrade of computing power and the accumulation of massive amounts of data, large-scale language models have achieved great success. Although large language models can achieve good results in some general domains without fine-tuning, they are difficult to achieve ideal results in more specialized fields, such as power and medicine, without fine-tuning. Research also shows that large language models generally have poor memory for factual knowledge.

[0003] If full parameter fine-tuning of a large model is required, the time cost and computing resources required are enormous, which is very detrimental to research in this field and its application in industry. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the lack of domain knowledge and difficulty in full parameter fine-tuning of the above-mentioned large language models. The present invention provides a method for knowledge injection language models based on adapters to break through the limitations of the prior art, so that large language models can memorize domain knowledge and achieve efficient parameter optimization of downstream tasks.

[0005] Therefore, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides an adapter-based language model knowledge injection method, wherein the language model is a language model based on the Transformer architecture, comprising:

[0007] Obtain the text data of the knowledge to be injected, segment the text data into words and map them to the corresponding identifiers in the vocabulary, and divide the mapped identifier sequence into several sub-sequences according to the maximum input sequence length of the target language model;

[0008] Insert an adapter module with gating mechanism before at least one Transformer layer of the target language model;

[0009] The subsequences are used as training samples and input into a target language model containing an adapter module. Based on the final hidden layer representation output by the target language model, the subsequence identifiers are predicted.

[0010] The loss is calculated based on the input subsequence identifiers and the predicted subsequence identifiers. The initial target language model parameters are frozen, and the parameters of all adapter modules are updated. The trained target language model containing adapter modules is used as the result after knowledge injection.

[0011] Furthermore, the adapter module with gated mechanism consists of an adapter part and a gated part. The input of the adapter module with gated mechanism is calculated by both the adapter part and the gated part. Then, the output results of the two parts are multiplied together and added to the input of the adapter module with gated mechanism as a residual link to obtain the output of the adapter module with gated mechanism.

[0012] Furthermore, the adapter portion consists of two linear layers and an activation function, calculated as follows:

[0013] AP l (x)=W2σ(W1x+b1)+b2

[0014] Among them, AP l (.) represents the adapter part of the adapter module inserted before the l-th Transformer layer of the target language model, x represents the input of the adapter module, W1 and W2 represent the parameter matrices of the two linear layers of the adapter part, b1 and b2 represent the offsets of the two linear layers of the adapter part, and σ represents the activation function of the adapter part.

[0015] Furthermore, the gating part consists of a linear layer and an activation function, calculated as follows:

[0016] G l (x)=ξ(W3x+b3)

[0017] Among them, G l (.) represents the gating part of the adapter module inserted before the l-th Transformer layer of the target language model, W3 represents the parameter matrix of the linear layer of the gating part, b3 represents the offset of the linear layer of the gating part, and ξ represents the activation function of the gating part.

[0018] Furthermore, the output of the adapter module with gating mechanism is represented as follows:

[0019]

[0020] Among them, GA l (.) indicates an adapter module inserted before the l-th Transformer layer of the target language model. l (.) indicates the adapter part of the adapter module inserted before the l-th Transformer layer of the target language model, G l (.) indicates the gating part of the adapter module inserted before the l-th Transformer layer of the target language model, and x represents the input of the adapter module. This indicates the output of the adapter module.

[0021] Furthermore, an adapter module with a gating mechanism is inserted before all Transformer layers of the target language model.

[0022] Furthermore, in a target language model containing adapter modules, the parameters of the inserted adapter modules are not shared.

[0023] Furthermore, when calculating the loss based on the input subsequence identifier and the predicted subsequence identifier, the cross-entropy loss function is used.

[0024] Secondly, the present invention provides an adapter-based language model knowledge injection system for implementing the above method.

[0025] The beneficial effects of this invention are as follows: The knowledge injection method proposed in this invention is based on an adapter module containing an adapter part and a gating part, which breaks through the limitations of the prior art. It does not require full parameter fine-tuning of the large language model, and has low time overhead and low computing power requirements. This enables the large language model to memorize knowledge in professional fields and achieves efficient parameter optimization of downstream tasks. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the architecture of a target language model containing an adapter module, as shown in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the adapter module structure with gating mechanism shown in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the gating section in the adapter module shown in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0030] The adapter-based language model knowledge injection method proposed in this invention includes the following steps:

[0031] Step 1: Prepare text data in a specific domain as training data for knowledge injection.

[0032] In this embodiment, we chose to inject professional knowledge in the field of power into the language model LLaMA-13B. Therefore, the prepared text data includes 100,000 articles, magazines and papers related to the field of power.

[0033] Step two: Segment the text data obtained in step one. Use the word segmenter built into the language model to segment the text data and map it to the corresponding labels in the vocabulary.

[0034] The word segmentation result is denoted as T = Tokenizer(C) = {t1, t2, ... t}. i …,t M}, where Tokenizer(·) represents the built-in tokenizer of the language model, T represents the token sequence after tokenization, i.e., the tokenization result; t i This represents the token in the vocabulary corresponding to the i-th word, and M represents the number of words in the text data, corresponding to the total number of tokens after word segmentation. In this embodiment, the final total number of tokens is approximately 270M.

[0035] Step 3: Based on the word segmentation results obtained in Step 2, truncate the segments according to the maximum sequence length (max_length) of the language model. Subsequences This indicates the rounding up sign. In this embodiment, max_length is set to 2048.

[0036] Step four: Insert an adapter module with a gating mechanism before each Transformer layer in the target language model, such as... Figure 1 As shown, in this embodiment, the target language model LLaMA-13B adopts the Transformer decoder architecture;

[0037] Each subsequence obtained in step three is input as a sample S into the embedding layer of the language model to obtain an embedding vector, i.e. Where V represents the total number of words in the vocabulary, and E represents the dimension of the embedding layer. In this embodiment, V = 49953, and E = 4096.

[0038] The embedding vector e is input into the first adapter module with a gating mechanism, and the corresponding hidden representation is obtained through the calculation of the adapter module. Here, GA1(.) represents the first adapter module with a gating mechanism.

[0039] The aforementioned gating adapter module consists of two parts: an adapter section and a gating section. The embedding vector e input to the first gating adapter module is processed by both the adapter and gating sections. The outputs of the two sections are then multiplied and the embedding vector e itself is added as a residual link to obtain the hidden representation.

[0040] In an alternative implementation, the adapter section consists of two linear layers sandwiching an activation function ReLU, with the specific structure as follows: Figure 2 As shown, the calculation process of the embedding vector e through the adapter part can be expressed as:

[0041] AP(e)=W2σ(W1e+b1)+b2

[0042] Where AP(.) represents the adapter part, and W1e+b1 represents the computation of the embedding vector after the first linear layer. b1 and b2 represent the parameter matrix and offset of the first linear layer, respectively; d represents the intermediate dimension of the adapter part; σ represents the activation function, and the activation function used in this embodiment is ReLU; W2(·)+b2 represents the calculation of the second linear layer. b2 and b3 are the parameter matrix and offset of the second linear layer, respectively.

[0043] In one alternative implementation, the gating part consists of a linear layer and an activation function, with the specific structure as follows: Figure 3 As shown, the calculation process of the embedding vector e through the gating part can be expressed as:

[0044] G(e)=ξ(W3e+b3)

[0045] Where G(.) represents the gating part, and W3e+b3 represents the calculation of the embedding vector after the linear layer. b1 and b2 are the parameter matrix and offset of the linear layer, respectively. This linear layer does not change the dimension of the vector; ξ represents the activation function, and the activation function used in this embodiment is Sigmoid.

[0046] Multiply the adapter part's calculation result AP(e) and the gating part's calculation result G(e), and add the embedding vector itself to obtain the output result of the gated adapter module, expressed as:

[0047]

[0048] Then hide the representation The input is fed into the first Transformer layer of the language model, and the corresponding hidden representation is obtained through the computation of the Transformer layer. Where l represents the number of layers, which is 1 in this step.

[0049] Step 5: Input the hidden representation h1 output by the first Transformer layer obtained in Step 4 into the subsequent N-1 gating adapter modules and Transformer layers, where N represents the number of Transformer layers in the target language model. In this embodiment, the selected language model has a total of 32 layers. The final hidden representation h1 is calculated sequentially according to the following formula. 32 The calculation formula is as follows:

[0050] h l =Transformer l (GAl (h l-1 ))

[0051] GA l (h l-1 )=G(h l-1 )·AP(h l-1 )+h l-1

[0052] AP(h l-1 )=W2σ(W1h l-1 +b1)+b2

[0053] G(h l-1 )=ξ(W3h l-1 +b3)

[0054] Step six: Based on the final hidden representation obtained in step five, predict the identifier of the input sequence. In this embodiment, through... Figure 1 The lm_head layer in the middle measures the identifier of the input sequence;

[0055] Step 7: Based on the predicted labels obtained in Step 6, the true labels of the input sequence are used as labels, the loss is calculated, and the parameters of all adapter modules with gating mechanisms are optimized through gradient descent backpropagation. The parameters of the target language model are then frozen.

[0056] The training process described above involves predicting the nth identifier based on the first n-1 identifiers of the input sequence, and calculating the loss based on the prediction result logits for each identifier and the input identifier sequence labels.

[0057] loss=CrossEntropyLoss(logits,labels)

[0058] CrossEntropyLoss(·) indicates that the loss is calculated using the cross-entropy loss function.

[0059] Training the model for three cycles reduces the loss to a suitable range. The trained target language model of the insertion adapter is then used as the knowledge-injected language model to perform downstream tasks. This invention eliminates the need for full parameter fine-tuning of large language models, minimizing time and computational resource requirements. It solves the problems of large language models lacking domain-specific knowledge and being difficult to fine-tune with all parameters.

[0060] In this embodiment, 500 question-and-answer pairs related to the power industry were manually constructed to evaluate the effectiveness of the present invention. The present invention was compared with the original LLaMA-13B model and the LLaMA-13B model pre-trained on the dataset from step one using the LoRA method. Two evaluation methods were used: manual evaluation and ChatGPT evaluation. Manual evaluation involved power industry professionals scoring the model's generated answers, with a score range of [0, 10]. ChatGPT evaluation involved calling the ChatGPT service interface and setting instructions for ChatGPT to score the model's answers, with a score range of [0, 10]. The evaluation results of the above three models without fine-tuning are shown in the table below (all scores are displayed as a percentage):

[0061] LLaMA-13B 22.3 20.6 LLaMA-13B+LoRA 38.2 36.2 LLaMA-13B+GA (This invention) 40.5 41.2

[0062] The results show that the present invention achieves better results than the currently popular LoRA method for training models.

[0063] This embodiment also provides an adapter-based language model knowledge injection system for implementing the above embodiments; details already described will not be repeated. The terms "module," "unit," etc., used below refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.

[0064] The system includes:

[0065] The knowledge acquisition module is used to acquire the text data of the knowledge to be injected.

[0066] The preprocessing module is used to segment text data into words and map them to corresponding identifiers in the vocabulary. Based on the maximum input sequence length of the target language model, the mapped identifier sequence is divided into several sub-sequences.

[0067] An adapter insertion module is used to insert a gated adapter module before at least one Transformer layer of the target language model.

[0068] The training module is used to take the subsequence as training samples, input it into the target language model containing the adapter module, predict the subsequence identifier based on the final hidden layer representation output by the target language model, calculate the loss based on the input subsequence identifier and the predicted subsequence identifier, freeze the initial target language model parameters, and update the parameters of all adapter modules.

[0069] The knowledge injection result output module is used to take the trained target language model containing the adapter module as the result after knowledge injection.

[0070] In an optional implementation, the adapter module with gated mechanism consists of an adapter part and a gated part. The input of the adapter module with gated mechanism is calculated by both the adapter part and the gated part. Then, the output results of the two parts are multiplied and added to the input of the adapter module with gated mechanism as a residual link to obtain the output of the adapter module with gated mechanism. The parameters of the inserted adapter modules are not shared.

[0071] The specific implementation process of the functions and roles of each module in the above system is detailed in the corresponding steps of the above method, and will not be repeated here. For the system embodiment, since it basically corresponds to the method embodiment, relevant parts can be referred to in the description of the method embodiment. The system embodiment described above is merely illustrative; the modules described as separate components may or may not be physically separated, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0072] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0073] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A language model knowledge injection method based on an adapter, wherein the language model is a language model based on the Transformer architecture, characterized in that, include: Obtain the text data of the knowledge to be injected, segment the text data into words and map them to the corresponding identifiers in the vocabulary, and divide the mapped identifier sequence into several sub-sequences according to the maximum input sequence length of the target language model; An adapter module with a gated mechanism is inserted before at least one Transformer layer of the target language model; the adapter module with a gated mechanism consists of an adapter part and a gated part. The input of the adapter module with a gated mechanism is processed by both the adapter part and the gated part. Then, the outputs of the two parts are multiplied together and the input of the adapter module with a gated mechanism is added as a residual link to obtain the output of the adapter module with a gated mechanism. The adapter part consists of two linear layers and an activation function, calculated as follows: ； in, Indicates the insertion of the target language model. The adapter section of the adapter module preceding the Transformer layer. This indicates the input of the adapter module. These represent the parameter matrices of the two linear layers in the adapter section. These represent the offsets of the two linear layers in the adapter section. This represents the activation function of the adapter part; The gating part consists of a linear layer and an activation function, calculated as follows: ； in, Indicates the insertion of the target language model. The gating section of the adapter module preceding each Transformer layer. This represents the parameter matrix of the linear layer in the gating part. This indicates the offset of the linear layer in the gating section. This represents the activation function of the gating part; The output of the adapter module with gating mechanism is expressed as follows: ； in, Indicates the insertion of the target language model. The adapter module preceding the Transformer layer Indicates the insertion of the target language model. The adapter section of the adapter module preceding the Transformer layer. Indicates the insertion of the target language model. The gating section of the adapter module preceding each Transformer layer. This indicates the input of the adapter module. Indicates the output of the adapter module; The subsequences are used as training samples and input into a target language model containing an adapter module. Based on the final hidden layer representation output by the target language model, the subsequence identifiers are predicted. The loss is calculated based on the input subsequence identifiers and the predicted subsequence identifiers. The initial target language model parameters are frozen, and the parameters of all adapter modules are updated. The trained target language model containing adapter modules is used as the result after knowledge injection.

2. The language model knowledge injection method based on an adapter according to claim 1, characterized in that, An adapter module with gating mechanism is inserted before all Transformer layers of the target language model.

3. The language model knowledge injection method based on an adapter according to claim 1, characterized in that, In a target language model containing adapter modules, the parameters of the inserted adapter modules are not shared.

4. The language model knowledge injection method based on an adapter according to claim 1, characterized in that, When calculating the loss based on the input subsequence identifier and the predicted subsequence identifier, the cross-entropy loss function is used.

5. A language model knowledge injection system based on an adapter, wherein the language model is a language model based on the Transformer architecture, characterized in that, include: The knowledge acquisition module is used to acquire the text data of the knowledge to be injected. The preprocessing module is used to segment text data into words and map them to corresponding identifiers in the vocabulary. Based on the maximum input sequence length of the target language model, the mapped identifier sequence is divided into several sub-sequences. An adapter insertion module is used to insert a gated adapter module before at least one Transformer layer of the target language model. The gated adapter module consists of an adapter part and a gate part. The input of the gated adapter module is processed by both the adapter part and the gate part. The outputs of the two parts are then multiplied and added to the input of the gated adapter module as a residual link to obtain the output of the gated adapter module. The adapter part consists of two linear layers and an activation function, calculated as follows: ； in, Indicates the insertion of the target language model. The adapter section of the adapter module preceding the Transformer layer. This indicates the input of the adapter module. These represent the parameter matrices of the two linear layers in the adapter section. These represent the offsets of the two linear layers in the adapter section. This represents the activation function of the adapter part; The gating part consists of a linear layer and an activation function, calculated as follows: ； in, Indicates the insertion of the target language model. The gating section of the adapter module preceding each Transformer layer. This represents the parameter matrix of the linear layer in the gating part. This indicates the offset of the linear layer in the gating section. This represents the activation function of the gating part; The output of the adapter module with gating mechanism is expressed as follows: ； in, Indicates the insertion of the target language model. The adapter module preceding the Transformer layer Indicates the insertion of the target language model. The adapter section of the adapter module preceding the Transformer layer. Indicates the insertion of the target language model. The gating section of the adapter module preceding each Transformer layer. This indicates the input of the adapter module. Indicates the output of the adapter module; The training module is used to take the subsequence as training samples, input it into the target language model containing the adapter module, predict the subsequence identifier based on the final hidden layer representation output by the target language model, calculate the loss based on the input subsequence identifier and the predicted subsequence identifier, freeze the initial target language model parameters, and update the parameters of all adapter modules. The knowledge injection result output module is used to take the trained target language model containing the adapter module as the result after knowledge injection.

6. The language model knowledge injection system based on an adapter according to claim 5, characterized in that, The parameters of the inserted ligand modules are not shared.

Images