A language model knowledge distillation method and system based on selective intervention

Abstract

The present invention discloses a language model knowledge distillation method and system based on selective intervention. The method adopted by the present invention includes: initializing a teacher model and a student model; inputting a text prompt and calculating a corresponding exponential decay threshold; obtaining the output distribution of the student model and the teacher model; calculating the divergence between the output distribution of the student model and the output distribution of the teacher model; determining the label sequence according to the divergence and the exponential decay threshold; and updating the student model parameters according to the label sequence. By introducing an exponential decay threshold, the present invention dynamically adjusts the degree of participation of the teacher model in the student model sequence generation process as the generated sequence advances, so as to reduce the error accumulation caused by the noise generated by the student. Compared with the traditional student-generated output method, the present invention can significantly improve the performance of the student model in generating long sequences, and effectively reduce the prediction deviation between the teacher model and the student model, thereby improving the effect of the language model in practical applications.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation technology, and in particular to a language model knowledge distillation method and system based on selective intervention. Background Art

[0002] In recent years, large language models have achieved significant performance improvements in natural language processing. However, their large size imposes high resource requirements, limiting their widespread adoption in practical applications. Consequently, model compression techniques based on knowledge distillation have emerged to effectively reduce model complexity. In current language model training, the generated output of the student model is often affected by noise, leading to the accumulation of errors and severely impacting the quality of text generation. Therefore, effectively performing knowledge distillation, particularly in long sequence generation tasks, has become a pressing technical challenge.

[0003] Mainstream knowledge distillation methods typically utilize the output of a student model as training sequences to improve text generation performance. However, these methods often focus too much on efficient student model training and overlook the importance of precise guidance from the teacher model. The inherent performance gap between the teacher and student models can be misleading during training, becoming increasingly pronounced as training progresses. Furthermore, the student model's biases can accumulate during long sequence generation, and its autoregressive nature can further exacerbate these issues. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the problems existing in the above-mentioned prior art and provide a language model knowledge distillation method and system based on selective intervention, which aims to strategically integrate the teacher model into the generation process of the student model output sequence to ensure that the teacher model provides accurate guidance, thereby effectively avoiding the misleading phenomenon of the student model in the generation of long sequences.

[0005] To this end, the present invention adopts the following technical solution: a language model knowledge distillation method based on selective intervention, which includes:

[0006] Step 1), initialize the teacher model and student model;

[0007] Step 2), enter the text prompt and calculate the corresponding exponential decay threshold;

[0008] Step 3), obtain the output distribution of the student model and the teacher model;

[0009] Step 4), calculate the divergence between the output distribution of the student model and the output distribution of the teacher model;

[0010] Step 5), determine the label sequence based on the divergence and exponential decay threshold;

[0011] Step 6) Update the student model parameters based on the label sequence.

[0012] This method dynamically controls the involvement of the teacher model. By introducing an exponential decay threshold, it allows the student model to learn autonomously in the early stages of sequence generation, while gradually increasing the teacher model's involvement in the later stages of the sequence, thereby enhancing guidance accuracy. Across different student model sizes, the proposed method effectively reduces misleading information caused by over-reliance on student model output, significantly outperforming existing baseline methods.

[0013] Furthermore, the step 1) is specifically as follows: loading the teacher model and its pre-trained parameters into the memory, initializing the student model, the optimizer and the learning rate scheduler, and loading the training data.

[0014] Furthermore, the calculation formula of the exponential decay threshold in step 2) is: ,in, express The initial threshold value when is the decay rate that controls how fast the threshold decreases over time.

[0015] Furthermore, the specific process of step 3) is as follows:

[0016] 3.1) Loading input text prompt And the corresponding true label ;

[0017] 3.2) Obtaining the output distribution of the teacher model ;

[0018] 3.3) Obtaining the output distribution of the student model And the output sequence ;

[0019] in, Represents the time step The token sequence generated when Represents the updated label sequence.

[0020] Furthermore, in step 4), the divergence calculation content is: calculating the JS divergence between the output distributions of the teacher model and the student model in each time step, the formula is as follows:

[0021]

[0022] in, is the average distribution of the teacher model output distribution and the student model output distribution, represents the KL divergence, Represents JS divergence.

[0023] Furthermore, in step 5), the specific process of determining the tag sequence according to the divergence and the exponential decay threshold is as follows:

[0024] At the current time step If the JS divergence between the output distribution of the teacher model and the student model is calculated Exceeds the corresponding exponential decay threshold , will switch from the student model output token to the teacher model output token when generating the next token, that is:

[0025] .

[0026] The present invention selectively intervenes in the output of the student model by detecting the difference in next token probability distribution between the teacher model and the student model.

[0027] Furthermore, in step 6), the specific process of updating the student model parameters is as follows:

[0028] The student model is trained by gradient descent using the following training objective:

[0029]

[0030] in, represents the KL divergence between the output distribution of the student model and the teacher model, Represents a vocabulary.

[0031] The present invention also provides a language model knowledge distillation system based on selective intervention, which is used to implement the above-mentioned language model knowledge distillation method, and includes:

[0032] Model initialization unit: initialize the teacher model and student model;

[0033] Exponential decay threshold calculation unit: input text prompt and calculate the corresponding exponential decay threshold;

[0034] Output distribution acquisition unit: obtains the output distribution of the student model and the teacher model;

[0035] Divergence calculation unit: calculates the divergence between the output distribution of the student model and the output distribution of the teacher model;

[0036] Tag sequence determination unit: determines the tag sequence according to the divergence and exponential decay threshold;

[0037] Student model parameter updating unit: updates the student model parameters according to the label sequence.

[0038] The present invention has the following beneficial effects: it strategically integrates the teacher model into the student model's sequence generation process, effectively reducing the student model's accumulated errors during long sequence generation and ensuring the accuracy of the generated output. This can effectively alleviate the performance degradation caused by deviations in the student model's long sequence output. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a language model knowledge distillation method based on selective intervention according to the present invention;

[0040] Figure 2 This is a composition diagram of a language model knowledge distillation system based on selective intervention in the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific implementation methods.

[0042] Example 1

[0043] like Figure 1 As shown, the present invention is a language model knowledge distillation method based on selective intervention, comprising the following steps:

[0044] Step 1: Initialize the teacher model and student model.

[0045] Load the teacher model and its pre-trained parameters into memory, initialize the student model, optimizer, and learning rate scheduler, and load the training data.

[0046] Step 2: Enter the text prompt and calculate the corresponding exponential decay threshold.

[0047] The present invention introduces an exponential decay threshold ,in, express The default initial threshold is 1. is the decay rate that controls how fast the threshold decreases over time.

[0048] Step 3: Get the output distribution of the student model and the teacher model.

[0049] First, load the input text prompt And the corresponding true label ;

[0050] Then, the text prompt Input into the teacher model to obtain the output distribution of the teacher model ;

[0051] Next, the text prompt Input into the student model to obtain the output distribution of the student model and the output sequence of the student model ,

[0052] in, Represents the time step The token sequence generated when Represents the updated label sequence.

[0053] Step 4: Calculate the divergence between the output distribution of the student model and the output distribution of the teacher model.

[0054] The Jensen-Shannon (JS) divergence between the output distributions of the teacher model and the student model is calculated at each time step, as follows:

[0055]

[0056] in, is the average distribution of the teacher model output distribution and the student model output distribution, represents the Kullback-Leibler (KL) divergence, Represents JS divergence.

[0057] Step 5: Determine the label sequence based on the divergence and exponential decay threshold.

[0058] At the current time step If the JS divergence between the output distribution of the teacher model and the student model is calculated Exceeds the corresponding exponential decay threshold , when generating the next token, the output token of the student model is switched to the output token of the teacher model, that is:

[0059] .

[0060] Step 6: Update the student model parameters according to the label sequence.

[0061] The student model is trained by gradient descent using the following training objective:

[0062]

[0063] in, represents the KL divergence between the output distribution of the student model and the teacher model, Represents a vocabulary.

[0064] This embodiment adopts a language model knowledge distillation method based on selective intervention, which aims to improve the performance of long sequence generation and reduce the prediction deviation between the teacher and student models. By introducing an exponential decay threshold, the method proposed in this embodiment dynamically adjusts the degree of intervention of the teacher model in the student model sequence generation process to reduce the error accumulation caused by the noise generated by the student model. Compared with the existing knowledge distillation method based on student-generated output, the present invention effectively balances the needs of the student model learning its own output and preventing error accumulation, and significantly improves the accuracy of the generated results through timely intervention of the teacher model.

[0065] Application Examples

[0066] The language model knowledge distillation method described in Example 1 is applied as follows.

[0067] The training process of this paper uses the Databricks-dolly-15k dataset, which is divided into 14K training samples and 500 validation samples. In addition, the paper introduces a language modeling loss using the OpenWebText corpus, mixes generated sequences with real sequences (with a ratio of 0.5), and sets the decay factor parameter during training to 0.1.

[0068] The present invention evaluates the performance of the language model on five datasets, including Dolly, Selfnst, Vicuna, S-NI, and UnNI, to verify the wide applicability and significant effectiveness of the present invention. The Dolly dataset contains 500 samples extracted from Databricks-dolly-15k; the Selfnst dataset contains 252 user instruction following set samples; the Vicuna dataset contains 80 questions in nine categories: writing, role-playing, mathematics, programming, and knowledge; the S-NI dataset is a 9K test subset of the SuperNatural Instructions dataset; and the UnNI dataset includes a 10K test subset randomly selected from the Unnatural Instructions dataset.

[0069] This paper uses objective evaluation methods to evaluate the performance of models trained using this method. Specifically, the ROUGE-L score and the GPT-4 feedback score are used as evaluation metrics. GPT-2 XL (1.5 billion parameters) is used as the teacher model, and the student models include GPT-2 Base (120M parameters), GPT-2 Medium (340M parameters), and GPT-2 Large (760M parameters).

[0070] Table 1 shows the GPT4 and ROUGE-L scores of the present invention on the Dolly dataset; Table 2 shows the GPT4 and ROUGE-L scores of the present invention on the SelfInst dataset; Table 3 shows the GPT4 and ROUGE-L scores of the present invention on the VicunaEval dataset; and Table 4 shows the ROUGE-L scores of the present invention on the UnNI and Unnatural datasets. In these tables, GPT4 represents the GPT4 score, and RL represents the ROUGE-L score.

[0071] Table 1: GPT4 score and RL score results on the Dolly dataset

[0072]

[0073] Table 2: GPT4 score and RL score results on the SelfInst dataset

[0074]

[0075] Table 3: GPT4 score and RL score results on the Vicuna dataset

[0076]

[0077] Table 4: RL score results on S-NI and Unnatural datasets

[0078]

[0079] Results show that our method consistently outperforms state-of-the-art methods across a variety of teacher-student model configurations and evaluation metrics. In particular, the performance improvement is even more pronounced on smaller student models, demonstrating that our method can effectively bridge the gap between student and teacher models.

[0080] Example 2

[0081] This embodiment provides a language model knowledge distillation system based on selective intervention, which is used to implement the language model knowledge distillation method described in Example 1. Figure 2 As shown in the figure, it consists of a model initialization unit, an exponential decay threshold calculation unit, an output distribution acquisition unit, a divergence calculation unit, a label sequence determination unit and a student model parameter update unit.

[0082] Model initialization unit: initializes the teacher model and the student model; used to implement the function of step 1 in Example 1, which will not be repeated here.

[0083] Exponential decay threshold calculation unit: input text prompt, calculate the corresponding exponential decay threshold; used to implement the function of step 2 in embodiment 1, no further details will be given here.

[0084] Output distribution acquisition unit: obtains the output distribution of the student model and the teacher model; used to implement the function of step three in Example 1, which will not be repeated here.

[0085] Divergence calculation unit: calculates the divergence between the output distribution of the student model and the output distribution of the teacher model; used to implement the function of step 4 in Example 1, which will not be repeated here.

[0086] Tag sequence determination unit: determines the tag sequence according to the divergence and the exponential decay threshold; is used to implement the function of step 5 in embodiment 1, and will not be repeated here.

[0087] Student model parameter updating unit: updates the student model parameters according to the label sequence; used to implement the function of step six in Example 1, which will not be repeated here.

[0088] This paper introduces an exponential decay threshold to dynamically adjust the teacher model's involvement in the student model's sequence generation process as the generated sequence progresses, thereby reducing the accumulation of errors caused by student-generated noise. This effectively balances the need for the student model to learn its own output while preventing error accumulation. Through timely intervention by the teacher model, the accuracy of generated results is significantly improved.

[0089] It should be noted that each unit in the above-mentioned language model knowledge distillation system can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units. For the specific definition of a language model knowledge distillation system, please refer to the definition of a language model knowledge distillation method (i.e., Example 1) above. The two have the same functions and effects and will not be repeated here.

[0090] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A language model knowledge distillation method based on selective intervention, characterized in that: include: Step 1), initialize the teacher model and student model; Step 2), enter the text prompt and calculate the corresponding exponential decay threshold; Step 3), obtain the output distribution of the student model and the teacher model; Step 4), calculate the divergence between the output distribution of the student model and the output distribution of the teacher model; Step 5), determine the label sequence based on the divergence and exponential decay threshold; Step 6) Update the student model parameters based on the label sequence.

2. The language model knowledge distillation method based on selective intervention according to claim 1, characterized in that: The step 1) is specifically as follows: loading the teacher model and its pre-trained parameters into memory, initializing the student model, optimizer and learning rate scheduler, and loading the training data.

3. The language model knowledge distillation method based on selective intervention according to claim 1, characterized in that: The calculation formula of the exponential decay threshold in step 2) is: ,in, express The initial threshold value when is the decay rate that controls how fast the threshold decreases over time.

4. The language model knowledge distillation method based on selective intervention according to claim 1, characterized in that: The specific process of step 3) is as follows: 3.1) Loading input text prompt And the corresponding true label ; 3.2) Obtaining the output distribution of the teacher model ; 3.3) Obtaining the output distribution of the student model And the output sequence ; in, Represents the time step The token sequence generated when Represents the updated label sequence.

5. The language model knowledge distillation method based on selective intervention according to claim 4, characterized in that: In step 4), the divergence calculation content is: calculate the JS divergence between the output distribution of the teacher model and the student model in each time step, and the formula is as follows: ， in, is the average distribution of the teacher model output distribution and the student model output distribution, represents the KL divergence, Represents JS divergence.

6. The language model knowledge distillation method based on selective intervention according to claim 5, characterized in that: In step 5), the specific process of determining the tag sequence according to the divergence and exponential decay threshold is as follows: At the current time step If the JS divergence between the output distribution of the teacher model and the student model is calculated Exceeds the corresponding exponential decay threshold , will switch from the student model output token to the teacher model output token when generating the next token, that is: 。 7. The language model knowledge distillation method based on selective intervention according to claim 5, characterized in that: In step 6), the specific process of updating the student model parameters is as follows: The student model is trained by gradient descent using the following training objective: in, represents the KL divergence between the output distribution of the student model and the teacher model, Represents a vocabulary.

8. A language model knowledge distillation system based on selective intervention, which is used to implement the language model knowledge distillation method according to any one of claims 1 to 7, characterized in that: include: Model initialization unit: initialize the teacher model and student model; Exponential decay threshold calculation unit: input text prompt and calculate the corresponding exponential decay threshold; Output distribution acquisition unit: obtains the output distribution of the student model and the teacher model; Divergence calculation unit: calculates the divergence between the output distribution of the student model and the output distribution of the teacher model; Tag sequence determination unit: determines the tag sequence according to the divergence and exponential decay threshold; Student model parameter updating unit: updates the student model parameters according to the label sequence.

Images