A language large model full fine-tuning method and device based on a knowledge distillation effect

Abstract

The application discloses a language large model full fine-tuning method and device based on a knowledge distillation effect. Training data sets required for automatic generation of instructions are constructed; the training data sets are combined with preset prompt word templates, and corresponding reply data is generated by using a preset teacher model; the reply data is constructed into instruction fine-tuning data sets according to a preset data construction format; and the preset student model is fully fine-tuned through the instruction fine-tuning data sets, which can help the student model learn more general language expression modes, thereby better adapting to various application scenarios, and enabling the student model to have higher performance and understanding ability while maintaining light weight.

Description

Technical Field

[0001] This application relates to the field of fine-tuning technology for large language models, specifically to a method and apparatus for full-scale fine-tuning of a large language model based on the knowledge distillation effect. Background Technology

[0002] With the rapid development of large-scale pre-training and high-performance computing technologies, numerous large language models pre-trained on massive amounts of unsupervised text have emerged, demonstrating outstanding performance in multiple domains and even surpassing human capabilities in some tasks. However, these models typically have billions or even tens of billions of parameters, requiring significant computational resources and memory during deployment, and are relatively slow. In contrast, smaller-scale language models offer a more energy-efficient solution. Through careful design and optimization, these smaller-scale models can significantly reduce computational resource and memory requirements while maintaining relatively reasonable performance levels. However, due to the reduced number of parameters, smaller-scale language models may not perform as well as larger-scale models when handling complex reasoning tasks, and may lag behind in understanding deep language structures and subtle contextual changes, leading to decreased accuracy in understanding complex texts. Therefore, maintaining deep understanding capabilities while reducing the number of parameters in large-scale language models has become a pressing need.

[0003] With the growing attention given to the ability of large language models to handle complex reasoning tasks, the challenges and demands for these tasks are also increasing. To avoid retraining these massive models from scratch, related technologies have proposed instruction fine-tuning techniques to improve their performance and applicability in complex reasoning. Traditional machine learning models require task-specific fine-tuning, which limits their generalization ability. Compared to traditional fine-tuning, instruction fine-tuning aims to improve the model's adaptability to different tasks by using a broad set of instructions, while reducing reliance on large amounts of task-specific training data, making the model more adaptable to specific tasks. However, fine-tuning large language models requires significant computational resources and time, posing a challenge in resource-constrained environments. Therefore, diverse and high-quality instruction fine-tuning datasets are crucial, as they significantly impact the model's generalization and understanding capabilities. Thus, effectively addressing these issues and improving the generalization and understanding capabilities of large language models is a pressing problem in the field of complex reasoning using large language models. Summary of the Invention

[0004] This application provides a method and apparatus for full fine-tuning of a large language model based on the knowledge distillation effect, addressing the technical problem in related technologies that fine-tuning a large language model requires a large amount of computational resources and time.

[0005] In a first aspect, embodiments of this application provide a method for full fine-tuning of a large language model based on the knowledge distillation effect, the method comprising:

[0006] The build command automatically generates the required training dataset;

[0007] The training dataset is combined with a preset prompt word template, and a preset teacher model is used to generate corresponding response data;

[0008] The response data is constructed into an instruction fine-tuning dataset according to a preset data construction format;

[0009] The preset student model is fine-tuned by fully adjusting the dataset using the instructions.

[0010] In conjunction with the first aspect, in one implementation, the construction instructions automatically generate the required training dataset, including:

[0011] Data sets for text summarization, text paraphrasing, and reading comprehension tasks of a large language model are collected as the training dataset.

[0012] The questions and answers in the training dataset are separated to obtain question data and answer data. In one implementation, the training dataset is combined with a preset prompt word template, and a preset teacher model is used to generate corresponding response data, including:

[0013] The prompt word template is appended to the end of the question data as input data;

[0014] The input data is input into the teacher model to obtain the response data;

[0015] The prompt word template is a statement used to guide the teacher model to output comprehensive, clear, and easy-to-understand response data.

[0016] In one implementation, the response data is constructed into an instruction fine-tuning dataset according to a preset data construction format, including:

[0017] Using the prompt word template as the instruction, the question data in the training dataset as the input, and the response data from the teacher model as the output, instruction fine-tuning data is constructed.

[0018] The instruction fine-tuning data is stored in JSON file format to generate the instruction fine-tuning dataset.

[0019] In one implementation, fine-tuning the preset student model across the entire dataset via the instruction includes:

[0020] The instruction fine-tuning data in the instruction fine-tuning dataset is segmented using the word segmenter corresponding to the student model to obtain the corresponding tensor.

[0021] The instruction fine-tuning dataset after word segmentation is used as training data and input into the student model. The gradient is calculated using the Adam optimizer and cross-entropy loss function, and the parameters of the student model are updated by full fine-tuning.

[0022] Secondly, embodiments of this application provide a full-scale fine-tuning device for a large language model based on the knowledge distillation effect, the device comprising:

[0023] The first module is used to automatically generate the required training dataset using the construction instructions;

[0024] The second module is used to combine the training dataset with the preset prompt word template and use the preset teacher model to generate corresponding response data;

[0025] The third module is used to construct the response data into an instruction fine-tuning dataset according to a preset data construction format;

[0026] The fourth module is used to fine-tune the preset student model by fully adjusting the dataset through the instructions.

[0027] In conjunction with the second aspect, in one implementation, the first module is further configured to:

[0028] Data sets for text summarization, text paraphrasing, and reading comprehension tasks of a large language model are collected as the training dataset.

[0029] The questions and answers in the training dataset are separated to obtain question data and answer data.

[0030] In one implementation, the second module is further configured to:

[0031] The prompt word template is appended to the end of the question data as input data;

[0032] The input data is input into the teacher model to obtain the response data;

[0033] The prompt word template is a statement used to guide the teacher model to output comprehensive, clear, and easy-to-understand response data.

[0034] In one implementation, the third module is further configured to:

[0035] Using the prompt word template as the instruction, the question data in the training dataset as the input, and the response data from the teacher model as the output, instruction fine-tuning data is constructed.

[0036] The instruction fine-tuning data is stored in JSON file format to generate the instruction fine-tuning dataset.

[0037] In one implementation, the fourth module is further configured to:

[0038] The instruction fine-tuning data in the instruction fine-tuning dataset is segmented using the word segmenter corresponding to the student model to obtain the corresponding tensor.

[0039] The instruction fine-tuning dataset after word segmentation is used as training data and input into the student model. The gradient is calculated using the Adam optimizer and cross-entropy loss function, and the parameters of the student model are updated by full fine-tuning.

[0040] The method and apparatus for full-scale fine-tuning of a large language model based on the knowledge distillation effect provided in this application embodiment automatically generates the required training dataset by constructing instructions; combines the training dataset with preset prompt word templates, and uses a preset teacher model to generate corresponding response data; constructs the response data into an instruction fine-tuning dataset according to a preset data construction format; and fully fine-tunes the preset student model through the instruction fine-tuning dataset, which can help the student model learn more general language expressions, thereby better adapting to various application scenarios, and enabling the student model to have higher performance and comprehension ability while maintaining lightweight design. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the full-scale fine-tuning method for the large language model based on the knowledge distillation effect in this application.

[0042] Figure 2 Examples of training datasets collected for this application;

[0043] Figure 3 This is a schematic diagram illustrating the construction of the instruction fine-tuning dataset of this application;

[0044] Figure 4 This is a schematic diagram of the technical framework for fully fine-tuning the student model in this application;

[0045] Figure 5 This is a schematic diagram of the functional modules of the language large model full-scale fine-tuning device based on the knowledge distillation effect in this application. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] In a first aspect, embodiments of this application provide a method for full fine-tuning of a large language model based on the knowledge distillation effect.

[0049] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the full-scale fine-tuning method for the large language model based on the knowledge distillation effect in this application. Figure 1 As shown, the full-scale fine-tuning method for a large language model based on the knowledge distillation effect includes:

[0050] Step S101: The construction command automatically generates the required training dataset.

[0051] Step S102: Combine the training dataset with the preset prompt word template, and use the preset teacher model to generate the corresponding response data.

[0052] Step S103: Construct the response data into an instruction fine-tuning dataset according to the preset data construction format.

[0053] Step S104: Fine-tune the preset student model by fully adjusting the dataset using the instructions.

[0054] It is worth noting that the instruction fine-tuning dataset constructed in this embodiment is named the Paraphrasing Easy-to-Understand dataset. The teacher model in this embodiment refers to a language model with large-scale parameters, which can be GPT-3.5 or GPT-4. The student model refers to a language model with small-scale parameters.

[0055] Specifically, knowledge distillation is a method for training smaller models by extracting knowledge from larger models. It generates high-quality outputs using large-parameter language models (such as GPT-3.5 or GPT-4) that reflect the deep understanding and knowledge of these large-parameter models on a specific task. We then leverage these responses generated by the large-parameter language models to create a paraphrasable dataset, aiming to capture and convey the high-level knowledge and understanding of the large model. Full-scale fine-tuning then improves the limitations of smaller-parameter models in fully understanding the problem.

[0056] It is important to understand that a deep understanding of the problem is a crucial part of the reasoning process for solving it. Strengthening the model's understanding of the entire problem can significantly improve its performance on reasoning tasks. Therefore, this application, based on the knowledge distillation effect, uses instruction fine-tuning techniques to allow a language model with fewer parameters (the student model) to learn the thinking methods of a language model with more parameters (the teacher model). This allows the student model to benefit from the teacher model's deeper understanding, further enhancing its comprehension ability. This application can effectively improve the performance of the student model in multi-step logical reasoning, deep causal relationship analysis, or abstract understanding tasks, while maintaining efficient reasoning.

[0057] In one embodiment, the automatic generation of the required training dataset by the construction instructions includes: collecting datasets from text summarization, text paraphrasing, and reading comprehension tasks of a large language model as the training dataset; then separating the questions and answers in the training dataset to obtain question data and answer data, such as... Figure 2 As shown.

[0058] Furthermore, in one embodiment, the training dataset is combined with a preset prompt word template, and a preset teacher model is used to generate corresponding response data, including: appending the prompt word template to the end of the question data as input data; inputting the input data into the teacher model to obtain the response data; wherein, the prompt word template is a statement used to guide the teacher model to output comprehensive, clear and easy-to-understand response data.

[0059] Exemplary, such as Figure 3As shown, the prompt template in this embodiment is "Please teach the above information comprehensively, clearly, and easily to someone who has never been exposed to this field before, and provide your guidance details!" Input data is formed by appending this prompt template to the end of the question data. The input data is then fed into the teacher model, where the guiding prompt template prompts the model to generate response data. The response data contains a simplified and easily understandable paraphrase of the complex information in the original question data, making it easily comprehensible even to those unfamiliar with the field. This information should contain the key information from the original text while being expressed in a clear and simple manner.

[0060] Furthermore, the response data is constructed into an instruction fine-tuning dataset according to a preset data construction format, including: using the prompt word template as an instruction, using the question data in the training dataset as input, and using the response data from the teacher model as output to construct instruction fine-tuning data; storing the instruction fine-tuning data in Jsonline file format to generate the instruction fine-tuning dataset.

[0061] As an example, the template for constructing quality fine-tuning data in this embodiment is: {“Instruction”: “Please teach the above information comprehensively, clearly, and easily to someone who has never been exposed to this field before, and provide your guidance details!”, “Input”: [Original Question], “Output”: [Response generated by the teacher model]}. The content of the input slot [Original Question] is the original question data from the collected training dataset, and the content of the input slot [Response from the Teacher Model] is the response data output by the teacher model.

[0062] Furthermore, a fine-tuning dataset (i.e., a paraphrased dataset) was constructed based on the data template. Each line in the JSON file format represents one fine-tuning instruction, and the content of each data entry is constructed in the same manner as the fine-tuning dataset described above. This dataset will be used for full-scale fine-tuning of the student model in subsequent steps.

[0063] Furthermore, the preset student model is fully fine-tuned using the instruction fine-tuning dataset, including: segmenting the instruction fine-tuning data in the instruction fine-tuning dataset into words using the word segmenter corresponding to the student model to obtain the corresponding tensors; inputting the segmented instruction fine-tuning dataset into the student model as training data, and calculating the gradient using the Adam optimizer and cross-entropy loss function to fully fine-tune and update the parameters of the student model.

[0064] As an example, after obtaining the instruction fine-tuning dataset (the easily understandable paraphrase dataset), the data is preprocessed. The student model's corresponding tokenizer is used to segment the data, resulting in tensors. Tokenization is the process of dividing the original text into smaller units (such as words, phrases, or symbols), these smaller units are called "tags." When processing the segmented text, each tag is typically further converted into a number (e.g., through a vocabulary index), and then arranged into arrays of one or more dimensions, i.e., tensors. The segmented instruction fine-tuning dataset (the easily understandable paraphrase dataset) is then used as the training dataset and input into the student model. Simultaneously, the Adam optimizer and cross-entropy loss function are used to calculate gradients and fully fine-tune and update the model parameters.

[0065] It is worth noting that the language model with a relatively small number of parameters fine-tuned in this application primarily uses the cross-entropy loss function, which characterizes the distance between the actual output (probability) and the expected output (probability). In other words, the smaller the cross-entropy value, the closer the two probability distributions are. In the feature space, the formula for the cross-entropy loss function is as follows:

[0066]

[0067] Where p(x) i Let q(x) be the true probability distribution. i Let be the probability distribution predicted by the model. The formula above shows that the smaller the cross-entropy value, the more accurate the model's prediction.

[0068] Furthermore, the gradient descent algorithm used in this invention employs the Adam optimizer. It combines the advantages of AdaGrad and RMSProp, comprehensively considering the mean and variance of the gradient to calculate the update step size. The former references the concept of momentum in physics, ensuring that past gradient descent results influence the current gradient, while the latter, to some extent, avoids divergence in parameter updates. The Adam update rule is as follows:

[0069] Algorithm: Adam

[0070] Begin:

[0071] m0←0,v0←0,t←0

[0072] While θ t Do not converage:

[0073] t←t+1

[0074]

[0075] m t ←β1·mt-1 +(1-β1)·g t

[0076]

[0077]

[0078]

[0079]

[0080] End;

[0081] Where m0←0, v0←0, t←0 is the initialization of the Adam algorithm, and m0←0, v0←0 are the initializations of the first moment estimate (momentum) and the second moment estimate (scaled gradient squared). t←0 initializes the iteration count to 0. Subsequent iterations increase the count, calculating at θ... t-1 The gradient at point A is given by the following formula:

[0082]

[0083] Next, update the momentum and scaled gradient squared, which are respectively the previous momentum decay plus the current gradient scaling and the previous value decay plus the current gradient squared scaling, as shown in the following formulas:

[0084] m t =β1·m t-1 +(1-β1)·g t

[0085]

[0086] Then, bias correction is applied to the scaling of momentum and squared gradient to reduce the impact of bias on the early stages of training, as shown in the following formula:

[0087]

[0088] Finally, the corrected momentum and scaled gradient parameters are used to update the parameters, as shown in the following formula:

[0089]

[0090] The Adam optimizer adjusts its parameters at each step, based on a weighted average of past gradients (momentum) and a weighted average of past gradient magnitudes (scaled gradient squared). Through bias correction and adaptive learning rate features, it can efficiently train deep learning models.

[0091] As an example, the implementation framework of the method based on the knowledge distillation effect for full fine-tuning of a large language model in this embodiment is as follows: Figure 4 As shown.

[0092] This application provides a method for full fine-tuning of a large language model based on the knowledge distillation effect.

[0093] Fine-tuning a small model by constructing a new dataset from the responses generated by a large model is a knowledge distillation method. In this process, the deep knowledge of the large model is transferred to the small model. By learning from the responses generated by the large model, the small model not only improves its ability to handle specific tasks but also inherits the deep understanding of the large model, enabling it to improve its performance and understanding capabilities while maintaining its lightweight nature.

[0094] Secondly, embodiments of this application also provide a device for full fine-tuning of a large language model based on the knowledge distillation effect.

[0095] In one embodiment, reference is made to Figure 5 , Figure 5 This is a schematic diagram of the functional modules of the full-scale fine-tuning device for the language large model based on the knowledge distillation effect in this application. Figure 5 As shown, the full-scale fine-tuning device for a large language model based on the knowledge distillation effect includes:

[0096] The first module is used to automatically generate the required training dataset using the construction instructions;

[0097] The second module is used to combine the training dataset with the preset prompt word template and use the preset teacher model to generate corresponding response data;

[0098] The third module is used to construct the response data into an instruction fine-tuning dataset according to a preset data construction format;

[0099] The fourth module is used to fine-tune the preset student model by fully adjusting the dataset through the instructions.

[0100] Furthermore, in one embodiment, the first module is further configured to:

[0101] Data sets for text summarization, text paraphrasing, and reading comprehension tasks of a large language model are collected as the training dataset.

[0102] The questions and answers in the training dataset are separated to obtain question data and answer data.

[0103] Furthermore, in one embodiment, the second module is also used for:

[0104] The prompt word template is appended to the end of the question data as input data;

[0105] The input data is input into the teacher model to obtain the response data;

[0106] The prompt word template is a statement used to guide the teacher model to output comprehensive, clear, and easy-to-understand response data.

[0107] Furthermore, in one embodiment, the third module is also used for:

[0108] Using the prompt word template as the instruction, the question data in the training dataset as the input, and the response data from the teacher model as the output, instruction fine-tuning data is constructed.

[0109] The instruction fine-tuning data is stored in JSON file format to generate the instruction fine-tuning dataset.

[0110] Furthermore, in one embodiment, the fourth module is also used for:

[0111] The instruction fine-tuning data in the instruction fine-tuning dataset is segmented using the word segmenter corresponding to the student model to obtain the corresponding tensor.

[0112] The instruction fine-tuning dataset after word segmentation is used as training data and input into the student model. The gradient is calculated using the Adam optimizer and cross-entropy loss function, and the parameters of the student model are updated by full fine-tuning.

[0113] The functions of each module in the above-mentioned language large model full fine-tuning device based on the knowledge distillation effect correspond to the steps in the above-mentioned language large model full fine-tuning method embodiment based on the knowledge distillation effect. Their functions and implementation processes will not be described in detail here.

[0114] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0115] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0116] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0117] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0118] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0120] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for full-scale fine-tuning of a large language model based on the knowledge distillation effect, characterized in that, The method for full fine-tuning of a large language model based on the knowledge distillation effect includes: The build command automatically generates the required training dataset; The training dataset is combined with a preset prompt word template, and a preset teacher model is used to generate corresponding response data; The response data is constructed into an instruction fine-tuning dataset according to a preset data construction format; The preset student model is fine-tuned by fully adjusting the dataset using the aforementioned instructions. The training dataset automatically generated by the construction instructions includes: Data sets for text summarization, text paraphrasing, and reading comprehension tasks of a large language model are collected as the training dataset. The questions and answers in the training dataset are separated to obtain question data and answer data; Specifically, the training dataset is combined with a preset prompt word template, and a preset teacher model is used to generate corresponding response data, including: The prompt word template is appended to the end of the question data as input data; The input data is input into the teacher model to obtain the response data; The prompt word template is a statement used to guide the teacher model to output comprehensive, clear and easy-to-understand response data; The process of constructing a fine-tuning instruction dataset from the response data according to a preset data construction format includes: Using the prompt word template as the instruction, the question data in the training dataset as the input, and the response data from the teacher model as the output, instruction fine-tuning data is constructed. The instruction fine-tuning data is stored in JSON file format to generate the instruction fine-tuning dataset.

2. The method for full fine-tuning of a large language model based on the knowledge distillation effect as described in claim 1, characterized in that, Fine-tuning the preset student model by fully adjusting the dataset using the instructions includes: The instruction fine-tuning data in the instruction fine-tuning dataset is segmented using the word segmenter corresponding to the student model to obtain the corresponding tensor. The instruction fine-tuning dataset after word segmentation is used as training data and input into the student model. The gradient is calculated using the Adam optimizer and cross-entropy loss function, and the parameters of the student model are updated by full fine-tuning.

3. A device for full-scale fine-tuning of a large language model based on the knowledge distillation effect, characterized in that, The language large model full fine-tuning device based on the knowledge distillation effect includes: The first module is used to automatically generate the required training dataset using the construction instructions; The second module is used to combine the training dataset with the preset prompt word template and use the preset teacher model to generate corresponding response data; The third module is used to construct the response data into an instruction fine-tuning dataset according to a preset data construction format; The fourth module is used to fine-tune the preset student model by fully adjusting the dataset through the instructions; The first module is further configured to: Data sets for text summarization, text paraphrasing, and reading comprehension tasks of a large language model are collected as the training dataset. The questions and answers in the training dataset are separated to obtain question data and answer data; The second module is also used for: The prompt word template is appended to the end of the question data as input data; The input data is input into the teacher model to obtain the response data; The prompt word template is a statement used to guide the teacher model to output comprehensive, clear and easy-to-understand response data; The third module is further used for: Using the prompt word template as the instruction, the question data in the training dataset as the input, and the response data from the teacher model as the output, instruction fine-tuning data is constructed. The instruction fine-tuning data is stored in JSON file format to generate the instruction fine-tuning dataset.

4. The language large model full-scale fine-tuning device based on the knowledge distillation effect as described in claim 3, characterized in that, The fourth module is also used for: The instruction fine-tuning data in the instruction fine-tuning dataset is segmented using the word segmenter corresponding to the student model to obtain the corresponding tensor. The instruction fine-tuning dataset after word segmentation is used as training data and input into the student model. The gradient is calculated using the Adam optimizer and cross-entropy loss function, and the parameters of the student model are updated by full fine-tuning.

Images