Optimization method for large language models based on layer-by-layer parameter editing

Abstract

The present invention discloses an optimization method for a large language model based on layer-by-layer parameter editing, wherein the large language model is composed of a multi-layer Transformer decoder, wherein the Transformer decoder includes a feedforward network module, and the optimization method includes: in the Transformer decoder, from the first key layer to the last key layer, performing parameter editing on each key layer; the parameter editing method for each key layer includes the following steps: 1) estimating the key C of the knowledge encoded by the large language model; 0 , 2) The batch target knowledge representation on the feedforward network module is used as the value V 1 ; 3) The hidden state of the batch target knowledge before entering the feedforward network module is used as the key K 1 ; 4) Update the output weight matrix of the feedforward network module. The present invention completes parameter update layer by layer, realizes accurate parameter editing of large language models, and eliminates the information loss problem caused by knowledge dissemination.

Description

Technical Field

[0001] The present invention relates to the technical field of text knowledge processing based on a large language model, and in particular to an optimization method for a large language model based on layer-by-layer parameter editing. Background Art

[0002] The large language model first pre-trains on a massive corpus to learn the knowledge contained in the corpus, and then further fine-tunes to align with human preferences, such as GPT-3, ChatGPT, GPT4, Gemini, Llama, ChatGLM and other large language models. The pre-training part is the core link for the large language model to learn massive knowledge, which often requires a lot of computing resources. For example, OpenAI used more than 30,000 A100 GPUs to pre-train ChatGPT. However, some harmful, personal privacy and biased content will inevitably appear in the pre-training corpus. To prevent the large language model from generating these contents, you can clean the corpus and re-pre-train the large model, but this will consume a lot of computing power again. ① How to prevent it from generating harmful content at a low cost has become a problem. There is a delay between the corpus and the real world, that is, the knowledge possessed by the large language model cannot be synchronized with the world in real time. ② How to keep the large language model synchronized with the world is also a challenge. In addition, the large language model will also give a probabilistically correct answer to questions that cannot be answered, which leads to the "hallucination" phenomenon of the large language model. How to improve or even solve the hallucination of the large language model has become a difficult problem.

[0003] For problem ①, the current mainstream method is to conduct human feedback reinforcement learning or add security protection to the large language model to prevent it from generating harmful content. For problems ② and ③, the current solution is to use retrieval enhancement technology to retrieve relevant knowledge and then use the knowledge as input to allow the large language model to temporarily learn the knowledge. However, this method increases the length of the input text, increases the reasoning burden of the GPU, and may cause conflicts between new knowledge and knowledge within the large model. None of the current methods fundamentally solve the above problems. The parameters of the large language model are the essence of knowledge storage. To fundamentally solve the above problems, it is necessary to edit the parameters of the large language model.

[0004] Among the current parameter editing methods for large language models, ROME and MEMIT regard the feedforward network module (FFN) in the Transformer module of the large language model as the main component for storing knowledge, and achieve the purpose of changing the knowledge of the large language model by editing the parameters of the feedforward network module. Specifically, they assume that the weight W of the feedforward network module is a key-value memory, and the hidden state of the text representation before entering the feedforward network module is used as the key k to query the knowledge in the parameters of the feedforward network module, and the output hidden state of the feedforward network module is returned as the query value v. The above process can be formally expressed as kW=v. In order to update the weight W of the feedforward network module, they first optimize the objective function to obtain the layer hidden state of the Transformer module as the value, and then extract the hidden state of the text representation before entering the feedforward network module as the key k to complete the update of the weight of the feedforward network module. However, the layer hidden state of the Transformer module is the sum of the residual, AM (Attention Module) and the output hidden state of the feedforward network module in the parallel Transformer module, and the sum of the residual and the output hidden state of the feedforward network module in the serial Transformer module. If the layer hidden state of the Transformer module is used as a value to update the weight of the feedforward network module, it is inaccurate, which affects the accuracy of parameter editing. In addition, since MEMIT needs to write new knowledge in batches through parameter editing, it obtains the layer hidden state of the last key layer and then propagates the layer hidden state to other key layers in an average propagation manner. Although this method reduces the optimization cost, this average propagation method further blurs the update information, which ultimately affects the accuracy of parameter editing, resulting in editing failure and even affecting other irrelevant edited knowledge in the model. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an optimization method for a large language model based on layer-by-layer parameter editing.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an optimization method of a large language model based on layer-by-layer parameter editing, wherein the large language model converts an input word into a probability distribution on a vocabulary, and the large language model is composed of a multi-layer Transformer decoder, and the Transformer decoder includes a feedforward network module. The optimization method includes: in the Transformer decoder, from the first key layer to the last key layer, parameter editing is performed on each key layer; the parameter editing method of each key layer includes the following steps:

[0007] 1) Sampling multiple sample articles on the online encyclopedia dataset to form sample vectors as input words of the large language model, and estimating the key C of the knowledge encoded by the large language model based on the input words 0 ,

[0008] 2) Batch target knowledge representation on the feedforward network module as value V 1 ;

[0009] 3) The hidden state of the batch target knowledge before entering the feedforward network module is used as the key K 1 ;

[0010] 4) Use W 1 =W 0 +Δ Update the output weight matrix of the feedforward network module, where W 1 represents the expected target weight, W 0 is the original weight; Δ is the weight increment;

[0011] in

[0012] Preferably, in step 1), a sample vector is formed by selecting a text of the maximum input length of the model of the sample article as an input word of the large language model. in, is the second-order moment, and λ is a hyperparameter.

[0013] Preferably, the step 2) further includes optimizing the target knowledge representation on the feedforward network module, and the optimization method of the target knowledge representation on the feedforward network module is:

[0014] The Transformer decoder also includes a self-attention module, which adds a learnable vector to the output hidden state of the feedforward network module and the self-attention module respectively;

[0015] Let the target knowledge of editing be (x e ,y e ), and optimize the learnable vector using the following loss function:

[0016]

[0017] Among them, μ represents the hyperparameter that controls the degree of parameter deviation, The KL divergence is used to measure the difference between the probability distribution of the edited model and the original model. represents the hyperparameter that controls the intensity of learning new knowledge, It means taking the average value of the input containing the prefixes generated by P models, Represents the output hidden state of the self-attention module Added learnable vectors And the output hidden state of the feedforward network module Added learnable vectors The output of the large language model after that; Indicates that when input x e The prefix pref is concatenated before j ;

[0018] After completing the optimization of the learnable vectors, only save As the target knowledge representation, after optimizing the batch target knowledge representation, Push value V onto the stack 1 .

[0019] Preferably, in step 3), the key K 1 Depend on Push the stack to form, let the target knowledge to be edited be (x e ,y e ),

[0020]

[0021] in Represents input The text representation before entering the feedforward network module, Indicates that when input x e The prefix pref is concatenated before j .

[0022] Preferably, the large language model θ takes as input the word x 1 ,...,x z The process of converting to probability distribution on the vocabulary is expressed as:

[0023]

[0024] Among them, W p and γ are the projection weight matrix and layer normalization respectively; and are the output hidden states of the self-attention module and the feedforward network module of the L-th layer Transformer module on the z-th input, is the layer output hidden state on the z-th input of the L-1-th layer Transformer module.

[0025] Preferably, the output hidden state of the feedforward network module on the i-th word unit of the l-th layer of the Transformer decoder is

[0026] in, is the output weight matrix of the feedforward network module of the lth layer of the Transformer decoder; is the input weight matrix of the feedforward network module of the lth layer of the Transformer decoder; σ is the nonlinear activation function; γ represents layer normalization; represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

[0027] Preferably, the output hidden state of the self-attention module on the i-th word unit of the l-th layer of the Transformer decoder is for:

[0028] in, is the output weight matrix of the self-attention module of the lth layer of the Transformer decoder; SA l represents the self-attention module of layer l, γ represents layer normalization, represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

[0029] The beneficial effects of the present invention are as follows: the present invention calculates the target knowledge representation for each key layer separately, completes the parameter update layer by layer, and finally realizes the accurate parameter editing of the large language model, eliminating the information loss problem caused by knowledge propagation. The target knowledge representation is calculated for each key layer and the parameter is updated so that the output weight matrix of the feedforward network module of each layer of the Transformer module is more accurate, which effectively improves the batch editing stability of the parameter editing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of a parameter editing method for each key layer in the optimization method of the present invention;

[0031] Figure 2 It is a principle block diagram of a parallel Transformer module using the method of the present invention;

[0032] Figure 3 It is a principle block diagram of a serial Transformer module using the method of the present invention. DETAILED DESCRIPTION

[0033] The present invention will now be described in further detail in conjunction with the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0034] A method for optimizing a large language model based on layer-by-layer parameter editing, wherein the large language model converts input word units into probability distributions on a vocabulary, and the large language model is composed of a multi-layer Transformer decoder, wherein the Transformer decoder includes a feed forward network module (Feed Forward Network, FFN) and a self-attention module (Self-Attention, SA). The optimization method includes: in the Transformer decoder, from the first key layer to the last key layer, parameter editing is performed on each key layer. The key layer mentioned in this article is a key layer that stores factual knowledge, and the key layer of the large language model is obtained by an existing causal intervention method, such as "Locating and editing factual associations in GPT" published by Meng Kevin et al., recorded in "Advances in Neural Information Processing Systems 35(2022):17359-17372".

[0035] Specifically, in an optional implementation, the large language model θ inputs the word x 1 ,...,x z The process of converting to probability distribution on the vocabulary is expressed as:

[0036]

[0037] Among them, W p and γ are the projection weight matrix and layer normalization respectively; and are the output hidden states of the self-attention module and the feedforward network module of the L-th layer Transformer module on the z-th input, is the layer output hidden state on the z-th input of the L-1-th layer Transformer module.

[0038] Specifically, in an optional implementation, the output hidden state of the self-attention module on the i-th word unit of the l-th layer of the Transformer decoder is for: in, is the output weight matrix of the self-attention module of the lth layer of the Transformer decoder; SA l represents the self-attention module of layer l, γ represents layer normalization, represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

[0039] Specifically, in an optional implementation, the output hidden state of the feedforward network module on the i-th word unit of the l-th layer of the Transformer decoder is

[0040] in, is the output weight matrix of the feedforward network module of the lth layer of the Transformer decoder; is the input weight matrix of the feedforward network module of the lth layer of the Transformer decoder; σ is the nonlinear activation function; γ represents layer normalization; represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

[0041] In an optimization method for a large language model based on layer-by-layer parameter editing, Figure 1 As shown, the parameter editing method of each key layer includes the following steps:

[0042] 1) Sampling multiple sample articles on the online encyclopedia dataset to form sample vectors as input words of the large language model, and estimating the key C of the knowledge encoded by the large language model based on the input words 0 ,

[0043] 2) Batch target knowledge representation on the feedforward network module as value V 1 ;

[0044] 3) The hidden state of the batch target knowledge before entering the feedforward network module is used as the key K 1 ;

[0045] 4) Use W 1 =W 0 +Δ Update the output weight matrix of the feedforward network module, where W 1 represents the expected target weight, W 0 is the original weight; Δ is the weight increment;

[0046] in

[0047] In the feedforward network module, the output weight matrix of the feedforward network module is updated by calculating the closed-form solution of the least squares problem. The least squares problem is:

[0048]

[0049] Where W 1 represents the expected target weight; k i and v i It is a key-value pair that encodes knowledge; Indicates that n pieces of knowledge need to be retained. It means that u>>1 piece of knowledge needs to be edited. Let W 0 is the original weight and Δ=W 1 -W 0 , then the closed-form solution to the above least squares problem is: and Key k i Sum value v i The stacked matrix.

[0050] Specifically, in an optional implementation, in step 1), a sample vector is formed by selecting a text of the maximum input length of the model of the sample article as an input word unit of the large language model. in, is the second-order moment, and λ is a hyperparameter. Specifically, in an optional implementation, 100,000 sample articles are sampled from the Wikipedia dataset, and the texts of the maximum input length of the model of these articles are selected as inputs, and the hyperparameter λ is usually set to 6,000.

[0051] Specifically, in an optional implementation, the step 2) further includes optimizing the target knowledge representation on the feedforward network module, and the optimization method of the target knowledge representation on the feedforward network module is:

[0052] The Transformer decoder also includes a self-attention module, which adds a learnable vector to the output hidden state of the feedforward network module and the self-attention module respectively;

[0053] Let the target knowledge of editing be (x e ,y e ), and optimize the learnable vector using the following loss function:

[0054]

[0055] Among them, μ represents the hyperparameter that controls the degree of parameter deviation, The KL divergence is used to measure the difference between the probability distribution of the edited model and the original model. represents the hyperparameter that controls the intensity of learning new knowledge, It means taking the average value of the input containing the prefixes generated by P models, Represents the output hidden state of the self-attention module Added learnable vectors And the output hidden state of the feedforward network module Added learnable vectors The output of the large language model after that; Indicates that when input x e The prefix pref is concatenated before j ;

[0056] After completing the optimization of the learnable vectors, only save As the target knowledge representation, after optimizing the batch target knowledge representation, Push value V onto the stack 1 .

[0057] Specifically, in an optional implementation, in step 3), key K 1 Depend on Push the stack to form, let the target knowledge to be edited be (x e ,y e ),

[0058]

[0059] in Represents input The text representation before entering the feedforward network module, Indicates that when input x e The prefix pref is concatenated before j .

[0060] Figure 2 It is a principle block diagram of a parallel Transformer module using the method of the present invention; Figure 3 The schematic diagram is a principle block diagram of a serial Transformer module using the method of the present invention. Both parallel Transformer modules and serial Transformer modules can use the optimization method of the present invention.

[0061] The optimization method of the large language model based on layer-by-layer parameter editing of the present invention is adopted for targeted and efficient editing. During the parameter editing process of the large language model, it is found that the different propagation methods of the target knowledge representation will greatly affect the effect of parameter editing. The possible reason is that the propagation of the target knowledge representation may lose some update information. Therefore, the present invention abandons the previous method of propagating the target knowledge representation. The present invention calculates the target knowledge representation for each key layer separately, completes the parameter update layer by layer, and finally realizes the precise parameter editing of the large language model, eliminating the information loss problem caused by knowledge propagation. The present invention calculates the target knowledge representation for each key layer separately and updates the parameters, so that the output weight matrix of the feedforward network module of each layer of the Transformer module is more accurate, which effectively improves the batch editing stability of the parameter editing method. Compared with the retrieval enhancement method in the prior art, the parameter editing method directly updates the model parameters, does not increase the reasoning burden caused by the increase in input content, and does not have problems such as conflicts between the retrieved knowledge and the internal knowledge of the model.

[0062] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation modes described above without departing from the essence and scope of the invention.

Claims

1. An optimization method for a large language model based on layer-by-layer parameter editing, wherein the large language model converts input word units into probability distributions on a vocabulary, wherein the large language model is composed of a multi-layer Transformer decoder, wherein the Transformer decoder includes a feedforward network module, and wherein: The optimization method comprises: in a Transformer decoder, from the first key layer to the last key layer, performing parameter editing on each key layer; the key layer is a key layer storing factual knowledge, and the key layer of the large language model is obtained by a causal intervention method, and the parameter editing method of each key layer comprises the following steps: 1) Sampling multiple sample articles on the online encyclopedia dataset to form sample vectors as input words of the large language model, and estimating the key C0 of the knowledge encoded by the large language model based on the input words, In the step 1), the text of the maximum input length of the model of the sample article is selected to form a sample vector as the input word of the large language model. in, is the second-order moment, λ is a hyperparameter; 2) The batch target knowledge representation on the feedforward network module is taken as the value V1; at the output hidden state of the feedforward network module Add learnable vectors As the target knowledge representation, Push the value V1 onto the stack; 3) The hidden state of the batch target knowledge before entering the feedforward network module is used as the key K1; Key K1 is composed of Push the stack to form, let the target knowledge to be edited be (x e ,y e ), in Represents input The text representation before entering the feedforward network module, Indicates that when input x e The prefix pref is concatenated before j ; 4) Use W1=W0+Δ to update the output weight matrix of the feedforward network module, where W1 represents the expected target weight, W0 is the original weight, and Δ is the weight increment; in 2. The optimization method of a large language model based on layer-by-layer parameter editing according to claim 1, characterized in that: The step 2) further includes optimizing the target knowledge representation on the feedforward network module. The optimization method of the target knowledge representation on the feedforward network module is: The Transformer decoder also includes a self-attention module, which adds a learnable vector to the output hidden state of the feedforward network module and the self-attention module respectively; The learnable vector is optimized using the following loss function: Among them, μ represents the hyperparameter that controls the degree of parameter deviation, The KL divergence is used to measure the difference between the probability distribution of the edited model and the original model. represents the hyperparameter that controls the intensity of learning new knowledge, It means taking the average value of the input containing the prefixes generated by P models, Represents the output hidden state of the self-attention module Added learnable vectors And the output hidden state of the feedforward network module Added learnable vectors The output of the large language model after that; Indicates that when input x e The prefix pref is concatenated before j ; After completing the optimization of the learnable vectors, only save As the target knowledge representation, after optimizing the batch target knowledge representation, Push the value V1 onto the stack.

3. The optimization method of a large language model based on layer-by-layer parameter editing according to claim 1, characterized in that: The large language model θ takes as input the word x1,...,x z The process of converting to probability distribution on the vocabulary is expressed as: Among them, W p and γ are the projection weight matrix and layer normalization respectively; and are the output hidden states of the self-attention module and the feedforward network module of the L-th layer Transformer module on the z-th input, is the layer output hidden state on the z-th input of the L-1-th layer Transformer module.

4. The optimization method of a large language model based on layer-by-layer parameter editing according to claim 1 or 2, characterized in that: The output hidden state of the feedforward network module on the i-th word unit of the l-th layer of the Transformer decoder is in, is the output weight matrix of the feedforward network module of the lth layer of the Transformer decoder; is the input weight matrix of the feedforward network module of the lth layer of the Transformer decoder; σ is the nonlinear activation function; γ represents layer normalization; represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

5. The optimization method of a large language model based on layer-by-layer parameter editing according to claim 2, characterized in that: The output hidden state of the self-attention module on the i-th token of the l-th layer of the Transformer decoder for: in, is the output weight matrix of the self-attention module of the lth layer of the Transformer decoder; SA l represents the self-attention module of layer l, γ represents layer normalization, represents the layer output hidden state on the i-th input of the l-1-th layer Transformer module.

Images