A distillation method and device of a large language model
By introducing knowledge distillation technology into a large language model and training student models using loss functions for both domain-specific and general-specific data, the problem of language style convergence during domain-specific data training is solved, achieving a balance between professionalism and generality, as well as natural output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
- Filing Date
- 2026-01-29
- Publication Date
- 2026-06-05
AI Technical Summary
When large language models are trained with supervised fine-tuning using domain data, the language style may converge to the style of the training data, resulting in written and templated outputs that affect user experience and weaken the model's generality.
We employ knowledge distillation techniques, combining domain data, general data, and student models, and train the model using style loss, KL divergence loss, and supervised fine-tuning loss to optimize the model's style output and general capabilities.
The trained model is able to maintain domain-specific knowledge while producing natural (non-written) outputs, thus enhancing its versatility.
Smart Images
Figure CN122154775A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model training technology, and in particular to a method and apparatus for distilling large language models. Background Technology
[0002] With the widespread application of large language models, the usage requirements of large language models in different business domains are becoming increasingly personalized. Different business domains usually need to use domain data specific to their business to train the model in a targeted manner in order to obtain a large language model that is more suitable for the current business domain.
[0003] However, simply using domain data for traditional supervised fine-tuning (SFT) training can cause the language style of large language models to converge to the style of the training data, resulting in written and templated outputs that negatively impact user experience. Furthermore, extensive use of domain data for training weakens the general capabilities of the original large language model, making it impossible to balance the model's specialization and versatility. Therefore, there is an urgent need for a training method that can balance the model's domain knowledge specialization and versatility while also providing natural (non-written) output. Summary of the Invention
[0004] This invention provides a distillation method and apparatus for a large language model. By introducing knowledge distillation technology into the large language model, it is possible to obtain style loss corresponding to the domain data, KL divergence loss corresponding to the general data, and supervised fine-tuning loss of the student model when both domain data and general data are introduced simultaneously. The student model is trained simultaneously using the three losses, so that the trained model can balance the domain knowledge specialization and general ability of the model, while also having natural (non-written) output.
[0005] This invention provides a distillation method for a large language model, comprising the following steps: acquiring domain data, general data, a pre-trained teacher model, and a student model to be trained; training the student model based on a style loss corresponding to the domain data, a KL divergence loss corresponding to the general data, and a supervised fine-tuning loss of the student model; wherein the style loss is calculated by inputting the domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively; and the supervised fine-tuning loss is calculated by inputting the general data and the domain data into the student model.
[0006] Optionally, the style loss is calculated as follows: the domain data is input into the teacher model and the student model respectively to obtain a first feature matrix output by the intermediate layer of the teacher model and a second feature matrix output by the intermediate layer of the student model; the style loss is calculated based on the distance between the first feature matrix and the second feature matrix.
[0007] Optionally, the intermediate layer of the teacher model is a neural network layer that accounts for 60% to 80% of the total number of layers in the teacher model; and / or, the intermediate layer of the student model is a neural network layer that accounts for 60% to 80% of the total number of layers in the student model.
[0008] Optionally, calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: determining the cosine similarity loss and the variance similarity loss between the first feature matrix and the second feature matrix, respectively; and generating the style loss based on the cosine similarity loss and the variance similarity loss.
[0009] Optionally, when there are multiple intermediate layers, the step of calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: for each intermediate layer, performing the step of calculating the sub-distance corresponding to each intermediate layer based on the distance between the first feature matrix and the second feature matrix; determining the weight corresponding to each intermediate layer respectively; and performing a weighted summation of each sub-distance based on the weight to obtain the style loss.
[0010] Optionally, the KL divergence loss is calculated through the following process: inputting the general data into the teacher model and the student model respectively to obtain the first confidence level output by the teacher model and the second confidence level output by the student model; and calculating the KL divergence loss based on the first confidence level and the second confidence level.
[0011] Optionally, the amount of general data is 10% to 20% of the amount of domain data.
[0012] The present invention also provides a distillation apparatus for a large language model, comprising the following modules: The acquisition module is used to acquire domain data, general data, pre-trained teacher models, and student models to be trained. The training module is used to train the student model based on the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model. The style loss is calculated by inputting the domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively.
[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distillation method for a large language model as described above.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distillation method for large language models as described above.
[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distillation method for large language models as described above.
[0016] This invention provides a distillation method and apparatus for a large language model. By introducing knowledge distillation technology into the large language model, it is possible to obtain style loss corresponding to the domain data, KL divergence loss corresponding to the general data, and supervised fine-tuning loss of the student model when both domain data and general data are introduced simultaneously. The student model is trained using the three losses simultaneously, so that the trained model can balance the domain knowledge specialization and general ability of the model, while also having natural (non-written) output. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is one of the flowcharts illustrating the distillation method for a large language model provided by this invention.
[0019] Figure 2 This is a schematic diagram of the style loss calculation process in the distillation method for large language models provided by this invention.
[0020] Figure 3 A schematic diagram illustrating the specific process of calculating style loss in step 202 provided by this invention.
[0021] Figure 4This is a schematic diagram of the calculation process of KL divergence loss in the distillation method of the large language model provided by the present invention.
[0022] Figure 5 This is a schematic diagram of the distillation apparatus for the large language model provided by the present invention.
[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0026] Figure 1 This is one of the flowcharts illustrating the distillation method for large language models provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain domain data, general data, pre-trained teacher models, and student models to be trained; Domain data refers to data specific to a particular business domain, while general data is data applicable across multiple business domains. The pre-defined teacher model is a large language model that provides natural and non-written responses; specifically, it is an open-source foundational model that can be obtained based on existing technologies.
[0027] Specifically, the core of large-model knowledge distillation is to transfer knowledge from the teacher model (large model) to the student model (small model). Based on access permissions to the teacher model, it can be divided into white-box distillation, black-box distillation, and a hybrid strategy combining both. The difference between white-box and black-box distillation lies in whether or not the internal information of the teacher model can be accessed. In this embodiment of the invention, since the teacher model is subsequently used to calculate the KL divergence loss, white-box distillation is preferred.
[0028] In an optional embodiment, in order to avoid excessive general data affecting the domain specialization of the model, the amount of general data in this embodiment of the invention is 10% to 20% of the amount of domain data, such as 10%, 12%, 15%, 18%, 20%, etc.
[0029] Step 102: Train the student model based on the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model. The style loss is calculated by inputting domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting general data into the teacher model and the student model respectively; and the supervised fine-tuning loss is calculated by inputting general data and domain data into the student model.
[0030] During the training of a neural network model, it is desirable for the network's output to be as close as possible to the desired predicted value. Therefore, it is necessary to compare the current predicted value with the actual target value, and then update the weight vector of each layer of the neural network based on this difference. For example, if the network's predicted value is too high, the weight vector is adjusted to predict a lower value. Through multiple rounds of adjustments, the neural network is trained until it can predict the actual target value or is very close to it. The difference between the predicted value and the actual target value is the loss function, and the value calculated by the loss function is the loss.
[0031] In this invention, the selection of loss is the core inventive point. The loss in this embodiment considers both domain-specific expertise and general applicability, while also taking into account the expressive style of the model's output, enabling the trained student model to possess multi-dimensional advantages. Specifically, the style loss in this embodiment optimizes the model's style output capability. By simultaneously calculating based on both the teacher and student models, the student model's style becomes closer to the teacher model, maintaining the naturalness and non-formalism of the output responses. Secondly, a KL divergence loss is generated for general data, allowing the trained model to be supervised with more knowledge information in a general domain, rather than being limited to a single business domain, greatly improving the model's versatility. Finally, the supervised fine-tuning loss based on general and domain data ensures that the trained student model possesses a certain level of understanding of both general and domain knowledge, avoiding situations where there is a lack of understanding of either.
[0032] The specific calculation process for the three types of losses involved in the embodiments of the present invention will be explained in detail below: In one alternative embodiment, such as Figure 2As shown, the style loss can be calculated in the following way: Step 201: Input the domain data into the teacher model and the student model respectively to obtain the first feature matrix output by the intermediate layer of the teacher model and the second feature matrix output by the intermediate layer of the student model. In this invention, the intermediate layer refers to the neural network layer located in the middle of the multi-layered structure of the neural network model. In one optional embodiment, the intermediate layer of the teacher model is the neural network layer located in the 60%~80% range of the total number of layers in the teacher model; the intermediate layer of the student model is the neural network layer located in the 60%~80% range of the total number of layers in the student model. The reason why the intermediate layer is chosen to calculate the style loss in this embodiment is that in the multiple neural network layers of the neural network model, neural network layers located at different positions usually have different capabilities. For example, the lower-level neural network layers are mainly responsible for basic grammar, the intermediate-level neural network layers are mainly responsible for output style, and the higher-level neural network layers are mainly responsible for knowledge and content. Therefore, this invention selects the features output by the intermediate layer to calculate the style loss, which can effectively improve the output style of the model by utilizing the specific functions of the intermediate layer.
[0033] Specifically, the domain data is in the format of standard question-answer pairs, and preprocessing is required before it is input into the teacher and student models. In one optional embodiment, the data preprocessing includes: concatenating the questions and answers in the question-answer pairs to form input data that conforms to the format specification of Transformer-type models; and then using a word segmenter consistent with the teacher and student models to segment the concatenated text, generating a word sequence and corresponding attention masks.
[0034] After generating the lexical sequence and the corresponding attention mask, the lexical sequence and the corresponding attention mask can be input into the teacher model and the student model respectively. The intermediate layer of the teacher model and the intermediate layer of the student model output the first intermediate layer hidden state of the teacher model and the second intermediate layer hidden state of the student model respectively. Then, by extracting features from the first intermediate layer hidden state and the second intermediate layer hidden state, the first feature matrix and the second feature matrix representing the style features can be obtained.
[0035] Step 202: Calculate the style loss based on the distance between the first feature matrix and the second feature matrix.
[0036] Specifically, in one optional embodiment, step 202 may include: determining the cosine similarity loss and variance similarity loss between the first feature matrix and the second feature matrix, respectively; and generating a style loss based on the cosine similarity loss and variance similarity loss. Here, the cosine similarity loss represents the similarity distance between the first feature matrix and the second feature matrix, and the variance similarity loss represents the distance between the variances of the first feature matrix and the second feature matrix.
[0037] In an optional embodiment, when there are multiple intermediate layers, step 202 can proceed as follows: Figure 3 As shown, it includes: Step 301: For each intermediate layer, perform the step of determining the sub-distance corresponding to each intermediate layer based on the distance between the first feature matrix and the second feature matrix; Step 302: Determine the weights corresponding to each intermediate layer; Step 303: Sum the sub-distances according to their weights to obtain the style loss.
[0038] The following sections provide a detailed explanation of the calculation process for cosine similarity loss and variance similarity loss, specifically for cases with multiple intermediate layers: (a) Cosine similarity loss For example, cosine similarity can be calculated based on the lexical-level features of the intermediate hidden states (first and second intermediate hidden states) of the teacher and student models to ensure style details for each lexical and achieve fine-grained style alignment. Specifically, the hidden states of the k-th intermediate layer of the teacher and student models are first normalized according to the lexical dimension to eliminate scale differences between the teacher and student models. Then, the cosine similarity of each lexical is calculated, and a scalar similarity value is output. Further, an attention mask is introduced, and the average similarity of effective lexical units within a batch is calculated to obtain the cosine similarity loss of the k-th intermediate layer.
[0039] (ii) Variance Similarity Loss For example, the variance similarity loss is first calculated for a single intermediate layer. Specifically, the number of effective words and the mean of effective words are calculated for the k-th intermediate layer of the teacher model and the k-th intermediate layer of the student model. Then, based on the calculated mean of effective words and the number of effective words, the first feature matrix (first variance matrix) corresponding to the teacher model and the second feature matrix (second variance matrix) corresponding to the student model are further calculated. After obtaining the first variance matrix and the second variance matrix, the mean squared error (MSE) algorithm is used to calculate the variance similarity loss between the student model and the teacher model.
[0040] After obtaining the cosine similarity loss and variance similarity loss for each intermediate layer, the arithmetic mean of all intermediate layers is taken to obtain the final style loss. The style constraint weights for each intermediate layer are equal, ensuring that the style loss comprehensively covers the style features of each intermediate layer.
[0041] In one alternative embodiment, such as Figure 4 As shown, the KL divergence loss can be calculated as follows: Step 401: Input the general data into the teacher model and the student model respectively to obtain the first confidence score output by the teacher model and the second confidence score output by the student model; Step 402: Calculate the KL divergence loss based on the first confidence level and the second confidence level.
[0042] For example, before inputting general data into the teacher model and student model, the general data also needs to be preprocessed. In one optional embodiment, the preprocessing process of general data is the same as that of domain data. That is, the data preprocessing process includes: concatenating the questions and answers in the question-answer pair to form input data that conforms to the Transformer class model format specification, and then using a word segmenter consistent with the teacher model and student model to segment the concatenated text to generate word sequence and corresponding attention mask.
[0043] After generating the token sequence and its corresponding attention mask, these sequences and masks can be input into the teacher model and student model, respectively. The outputs are the first confidence score for the teacher model and the second confidence score for the student model. The calculation of the confidence scores can be implemented using conventional methods in this field, and will not be elaborated upon here. After obtaining the first and second confidence scores, a token-wise KL divergence calculation method is used to calculate the KL divergence for each token and each sample, and finally, the KL divergence loss is obtained by averaging.
[0044] In one alternative embodiment, the supervised fine-tuning loss of the student model can be achieved using cross-entropy loss. This is done by inputting the word sequence of the domain data into the student model, using the word sequence of the answer as the label, and calculating the difference between the student model's predicted value and the true value to obtain the supervised fine-tuning loss of the student model.
[0045] In summary, this embodiment of the invention, by inputting domain data and general data into the teacher model and student model respectively, obtains style loss, KL divergence loss, and supervised fine-tuning loss with different characteristics. The total training loss of the student model is obtained by weighted summing of these three losses. It is evident that this embodiment of the invention simultaneously utilizes three losses to train the student model, enabling the trained model to balance domain knowledge specialization and general capabilities while also possessing natural (non-written) output.
[0046] The distillation apparatus for a large language model provided by the present invention will be described below. The distillation apparatus for a large language model described below can be referred to in correspondence with the distillation method for a large language model described above.
[0047] like Figure 5 As shown, the distillation apparatus 500 for a large language model provided in this embodiment of the invention includes: The acquisition module 501 is used to acquire domain data, general data, pre-trained teacher models, and student models to be trained; Training module 502 is used to train the student model based on the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model. The style loss is calculated by inputting the domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively; and the supervised fine-tuning loss is calculated by inputting the general data and the domain data into the student model.
[0048] In an optional embodiment, the style loss is calculated as follows: the domain data is input into the teacher model and the student model respectively to obtain a first feature matrix output by the intermediate layer of the teacher model and a second feature matrix output by the intermediate layer of the student model; the style loss is calculated based on the distance between the first feature matrix and the second feature matrix.
[0049] In one optional embodiment, the intermediate layer of the teacher model is a neural network layer that accounts for 60% to 80% of the total number of layers in the teacher model; and / or, the intermediate layer of the student model is a neural network layer that accounts for 60% to 80% of the total number of layers in the student model.
[0050] In an optional embodiment, calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: determining the cosine similarity loss and the variance similarity loss between the first feature matrix and the second feature matrix, respectively; and generating the style loss based on the cosine similarity loss and the variance similarity loss.
[0051] In an optional embodiment, when there are multiple intermediate layers, the step of calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: for each intermediate layer, performing the step of obtaining a sub-distance corresponding to each intermediate layer based on the distance between the first feature matrix and the second feature matrix; determining the weight corresponding to each intermediate layer respectively; and performing a weighted summation of each sub-distance based on the weight to obtain the style loss.
[0052] In an optional embodiment, the KL divergence loss is calculated by the following process: inputting the general data into the teacher model and the student model respectively to obtain a first confidence level output by the teacher model and a second confidence level output by the student model; and calculating the KL divergence loss based on the first confidence level and the second confidence level.
[0053] In one optional embodiment, the amount of general data is 10% to 20% of the amount of domain data.
[0054] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a large language model distillation method, which includes: acquiring domain data, general data, a pre-trained teacher model, and a student model to be trained; training the student model based on a style loss corresponding to the domain data, a KL divergence loss corresponding to the general data, and a supervised fine-tuning loss of the student model; wherein the style loss is calculated by inputting the domain data into the teacher model and the student model respectively; and the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively.
[0055] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0056] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the distillation method for a large language model provided by the above methods. The method includes: acquiring domain data, general data, a pre-trained teacher model, and a student model to be trained; training the student model according to the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model; wherein the style loss is calculated by inputting the domain data into the teacher model and the student model respectively; and the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively.
[0057] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a distillation method for a large language model provided by the methods described above. This method includes: acquiring domain data, general data, a pre-trained teacher model, and a student model to be trained; training the student model based on a style loss corresponding to the domain data, a KL divergence loss corresponding to the general data, and a supervised fine-tuning loss of the student model; wherein the style loss is calculated by inputting the domain data into the teacher model and the student model respectively; and the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively.
[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A distillation method for a large language model, characterized in that, include: Acquire domain data, general data, pre-trained teacher models, and student models to be trained; The student model is trained based on the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model. The style loss is calculated by inputting the domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively; and the supervised fine-tuning loss is calculated by inputting the general data and the domain data into the student model.
2. The distillation method for a large language model according to claim 1, characterized in that, The style loss is calculated in the following manner: The domain data is input into the teacher model and the student model respectively to obtain the first feature matrix output by the intermediate layer of the teacher model and the second feature matrix output by the intermediate layer of the student model. The style loss is calculated based on the distance between the first feature matrix and the second feature matrix.
3. The distillation method for a large language model according to claim 2, characterized in that, The intermediate layer of the teacher model is the neural network layer that accounts for 60% to 80% of the total number of layers in the teacher model; And / or, The intermediate layer of the student model is the neural network layer that accounts for 60% to 80% of the total number of layers in the student model.
4. The distillation method for a large language model according to claim 2, characterized in that, The step of calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: The cosine similarity loss and variance similarity loss between the first feature matrix and the second feature matrix are determined respectively. The style loss is generated based on the cosine similarity loss and the variance similarity loss.
5. The distillation method for a large language model according to claim 3, characterized in that, When the intermediate layers are multiple, calculating the style loss based on the distance between the first feature matrix and the second feature matrix includes: For each of the intermediate layers, the step of obtaining the sub-distance corresponding to each intermediate layer based on the distance between the first feature matrix and the second feature matrix is performed; Determine the weights for each intermediate layer; The style loss is obtained by weighting and summing the sub-distances according to the weights.
6. The distillation method for a large language model according to claim 1, characterized in that, The KL divergence loss is calculated through the following process: The general data is input into the teacher model and the student model respectively to obtain the first confidence level output by the teacher model and the second confidence level output by the student model; The KL divergence loss is calculated based on the first confidence level and the second confidence level.
7. The distillation method for a large language model according to claim 1, characterized in that, The amount of general data is 10% to 20% of the amount of domain data.
8. A distillation apparatus for a large language model, characterized in that, include: The acquisition module is used to acquire domain data, general data, pre-trained teacher models, and student models to be trained. The training module is used to train the student model based on the style loss corresponding to the domain data, the KL divergence loss corresponding to the general data, and the supervised fine-tuning loss of the student model. The style loss is calculated by inputting the domain data into the teacher model and the student model respectively; the KL divergence loss is calculated by inputting the general data into the teacher model and the student model respectively; and the supervised fine-tuning loss is calculated by inputting the general data and the domain data into the student model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the distillation method for a large language model as described in any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the distillation method for a large language model as described in any one of claims 1 to 7.