A knowledge distillation method, apparatus, device, medium and product

Abstract

The application discloses a knowledge distillation method, device, equipment, medium and product, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting a text under a current iteration number corresponding to the tth cycle number into a teacher model and a student model respectively to obtain a probability distribution set; calculating the values of forward KL divergence and reverse KL divergence according to the probability distribution set to obtain a loss function value; if a mark word is a non-end mark, the mark word is spliced to the back of the text, and then the next iteration corresponding to the tth cycle number is entered; otherwise, the parameters of the student model are updated according to the loss function value, then the cycle number t is increased by 1, the iteration number corresponding to the next cycle number is initialized, the next cycle is entered, and the parameters of the student model and the architecture of the student model corresponding to the last cycle number are saved until the texts in a training data set are extracted, thereby improving the precision and generalization ability of the student model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a knowledge distillation method, apparatus, equipment, medium, and product. Background Technology

[0002] The emergence of large language models has driven social change and brought profound impacts to human society. However, the massive number of parameters in large language models places higher demands on the computing resources of the deployed devices. Currently, for the general public, the mainstream solution is to utilize the computing resources provided by cloud service providers or build a high-performance computing cluster to perform deep learning tasks, and finally transmit the results to customer or enterprise terminal devices via the network. However, in certain specific fields, relevant personnel have a need to use large language models in offline scenarios, such as in the field where there is no network coverage. Therefore, when targeting specific fields, it is inevitable to deploy large language models directly on embedded devices. However, due to their large model size and high demand for computing resources, large language models have low usability on resource-constrained embedded devices.

[0003] The primary reason why large language models are difficult to deploy and run on embedded devices is the lack of computational and storage resources. This is mainly due to the sheer size of the models, which typically contain billions to hundreds of billions of parameters. A typical Llama-7B model, loaded with data type float16, requires 14GB of memory or GPU memory, which is unacceptable. To address this issue, the most common approach is knowledge distillation. The basic idea is to construct a smaller model (student model) with a much smaller number of parameters than the original model, and then align the outputs of the original model (teacher model) and the smaller model using machine learning methods. Specifically, the teacher model generates richer predictive information when processing input data, while the student model mimics the teacher model's output, especially its class distribution or predicted probability distribution, thereby reducing computational complexity and storage requirements while preserving model accuracy. However, traditional knowledge distillation methods have shortcomings, resulting in low accuracy and poor generalization ability in the final student model. Summary of the Invention

[0004] The purpose of this application is to provide a knowledge distillation method, apparatus, equipment, medium, and product that can improve the accuracy and generalization ability of student models.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a knowledge distillation method, comprising:

[0007] In the t-th iteration, a text is extracted without replacement from the training dataset as the text for the t-th iteration; the training dataset includes multiple texts.

[0008] At the current iteration number corresponding to the t-th iteration number, the text at the current iteration number corresponding to the t-th iteration number is input into the teacher model and the student model corresponding to the (t-1)-th iteration number, respectively, to obtain the probability distribution set at the current iteration number corresponding to the t-th iteration number; the probability distribution set includes the probability distribution of the output logit of the teacher model, the vector representation of the hidden features of each intermediate layer of the teacher model, the probability distribution of the output logit of the student model, and the vector representation of the hidden features of each intermediate layer of the student model; the text at the initial iteration number corresponding to the t-th iteration number is the text at the t-th iteration number; the teacher model is a pre-trained large language model;

[0009] The loss function value for the current iteration corresponding to the t-th iteration is obtained by summing the first loss function value and the loss function value for the previous iteration corresponding to the t-th iteration. The first loss function value is the value obtained by inputting the probability distribution set for the current iteration corresponding to the t-th iteration into the loss function. The loss function includes forward kL divergence and backward kL divergence.

[0010] Determine whether the marker is a non-terminal marker. If so, append the marker to the text of the current iteration corresponding to the t-th loop number to obtain the text of the next iteration corresponding to the t-th loop number. Then, increment the iteration number corresponding to the t-th loop number by 1 and enter the next iteration corresponding to the t-th loop number. The marker is obtained based on the probability distribution of the teacher model's output logit in the probability distribution set corresponding to the current iteration number.

[0011] If not, the parameters of the student model corresponding to the (t-1)th iteration number are updated according to the loss function value of the current iteration number corresponding to the tth iteration number to obtain the student model corresponding to the tth iteration number. Then, the iteration number t is incremented by 1 and the iteration number corresponding to the next iteration number is initialized. The next iteration begins until all the text in the training dataset has been extracted. The parameters of the student model corresponding to the last iteration number and the architecture of the student model are saved.

[0012] Secondly, this application provides a knowledge distillation apparatus, comprising:

[0013] An extraction module is used to extract a text without replacement from the training dataset in the t-th iteration as the text for the t-th iteration; the training dataset includes multiple texts.

[0014] The probability distribution determination module is used to input the text at the current iteration number corresponding to the t-th iteration number into the teacher model and the student model corresponding to the (t-1)-th iteration number, respectively, to obtain the probability distribution set at the current iteration number corresponding to the t-th iteration number. The probability distribution set includes the probability distribution of the output logit of the teacher model, the vector representation of the hidden features of each intermediate layer of the teacher model, the probability distribution of the output logit of the student model, and the vector representation of the hidden features of each intermediate layer of the student model. The text at the initial iteration number corresponding to the t-th iteration number is the text at the t-th iteration number. The teacher model is a pre-trained large language model.

[0015] The loss function value calculation module is used to calculate the first loss function value and the sum of the loss function value under the previous iteration number corresponding to the t-th iteration number to obtain the loss function value under the current iteration number corresponding to the t-th iteration number; the first loss function value is the value obtained by inputting the probability distribution set under the current iteration number corresponding to the t-th iteration number into the loss function; the loss function includes forward kL divergence and backward kL divergence.

[0016] The judgment module is used to determine whether the marker word is a non-terminal marker. If so, the marker word is appended to the text of the current iteration number corresponding to the t-th loop number to obtain the text of the next iteration number corresponding to the t-th loop number. Then, the iteration number corresponding to the t-th loop number is incremented by 1, and the next iteration corresponding to the t-th loop number is started. The marker word is obtained based on the probability distribution of the teacher model's output logit in the probability distribution set corresponding to the current iteration number.

[0017] The parameter update module is used to update the parameters of the student model corresponding to the (t-1)th iteration number based on the loss function value corresponding to the current iteration number corresponding to the tth iteration number if no, then increment the iteration number t by 1 and initialize the iteration number corresponding to the next iteration number, and enter the next iteration until the text in the training dataset has been completely extracted, and save the parameters and architecture of the student model corresponding to the last iteration number.

[0018] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge distillation method described above.

[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge distillation method described above.

[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the knowledge distillation method described above.

[0021] According to the specific embodiments provided in this application, this application has the following technical effects:

[0022] This application provides a knowledge distillation method, apparatus, device, medium, and product. Traditional knowledge distillation schemes using forward KL divergence as the loss function often result in poor fitting of the student model to the teacher's output distribution in certain regions. Furthermore, the logit-based distillation mode used in traditional knowledge distillation is relatively simple to implement but lacks a fine-grained understanding of knowledge. These shortcomings affect the overall fitting effect of the student model to the teacher model, thus impacting the prediction accuracy of the distilled student model. This application improves the performance of the student model after distillation by combining forward and backward KL divergence as the loss function and combining logit-based and feature-based knowledge distillation modes, thereby enhancing the accuracy and generalization ability of the student model. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A comparison of the effects of fitting a mixture Gaussian distribution to FKL divergence and RKL divergence;

[0025] Figure 2 This is a schematic flowchart of a knowledge distillation method according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] To better understand the physical meaning of forward Kullback-Leibler Divergence (FKL) and reverse Kullback-Leibler Divergence (RKL), this application uses a Python program to visually illustrate the difference between FKL and RKL in their distribution fitting performance, such as... Figure 1 As shown, when using FKL and RKL to try to fit a mixture Gaussian distribution, we can see that RKL tends to fit the highest peak of the overall distribution, that is, the fitting mode, while FKL tends to fit the overall average value of the distribution.

[0030] Traditional knowledge distillation schemes using the FKL formula as the loss function can easily lead to a large deviation in the fitting effect of the student model on the output distribution of the teacher model. Specifically, after knowledge distillation, the student model may show a higher probability in regions that should be low.

[0031] The explanation from the perspective of formula analysis is as follows:

[0032] KL divergence is a commonly used metric for calculating the difference between two distributions. Using the Forward KL formula as the loss function essentially relies on the output probability distribution of the teacher model to guide the student model's knowledge learning. By studying the forward KL divergence formula, it's easy to see that in the low to medium probability region, the value of p(x) is very small. The contribution to the overall divergence is also small. This means that KL divergence pays less attention to the low-to-medium probability regions during the optimization process. This directly leads to insufficient fitting in the low-to-medium probability regions, resulting in situations where the fitting values ​​are too high in regions that should be low-to-medium probability, and too low in regions that should be high probability. Therefore, the student model distilled based on this loss function tends to cover all probability regions of the teacher distribution on an average basis, affecting the student model's fit to the overall distribution and thus leading to poor inference performance. Specifically, this manifests as sampling low-to-medium probability words, leading to answer bias.

[0033] Using the inverse KL divergence formula as the loss function focuses on the relative relationships of the learning distributions. During training, the model no longer relies on an exact match to the teacher's output distribution, but instead emphasizes the relative relationships within the entire prediction distribution. Since q(x) is initially random, the RKL-based distillation process pays attention to all underexplored regions in the student distribution. It encourages the student model to try fitting all regions in the teacher model's distribution, lacking focus. This results in a slow convergence speed of the model's loss function. Therefore, when the training dataset is insufficient, the student model often produces undervalued fits to low-probability regions and overvalued fits to high-probability regions.

[0034] Furthermore, traditional knowledge distillation schemes mostly use a logit-based approach, where the student model's output logit and the teacher model's output logit are used as the loss function to optimize the student model. While knowledge distillation allows the student model to inherit knowledge from the teacher model, lightweight models often have fewer parameters than the teacher model. Therefore, when transferring knowledge for complex tasks, it may struggle to fully capture the deep features of the teacher model. This distillation approach only focuses on the difference between the teacher and student models' final outputs, lacking an understanding of the deep relationships between input and output, resulting in weak generalization ability. In contrast, feature-based knowledge distillation helps improve the student model's expressive power at intermediate feature layers. By combining these two methods, the student model can not only obtain useful information from the teacher model's output but also learn more low-level knowledge through feature transfer from intermediate layers, resulting in a model with higher accuracy and generalization ability, thereby improving the overall performance of the model.

[0035] Based on this, embodiments of this application provide a knowledge distillation method, such as... Figure 2 As shown, it includes the following steps, wherein:

[0036] Step 201: In the t-th iteration, extract one text without replacement from the training dataset as the text for the t-th iteration; the training dataset includes multiple texts.

[0037] Step 202: Under the current iteration number corresponding to the t-th loop number, input the text under the current iteration number corresponding to the t-th loop number into the teacher model and the student model corresponding to the (t-1)-th loop number respectively, to obtain the probability distribution set under the current iteration number corresponding to the t-th loop number; the probability distribution set includes the probability distribution of the output logit of the teacher model, the vector representation of the hidden features of each intermediate layer of the teacher model, the probability distribution of the output logit of the student model, and the vector representation of the hidden features of each intermediate layer of the student model; the text under the initial iteration number corresponding to the t-th loop number is the text under the t-th loop number. The teacher model is a pre-trained large language model. The probability distribution of the teacher model's output logit, obtained by inputting text into the teacher model, includes the probability that each word in the pre-defined vocabulary is the next word in the text. For example, if the pre-defined vocabulary includes three words (ABCDEF), and the text MJH is input into the teacher model, the probability distribution of the teacher model's output logit is p(A) = 0.1, p(B) = 0.1, p(C) = 0.2, p(D) = 0.1, p(E) = 0.1, and p(F) = 0.4. Here, p(A) represents the probability that word A in the pre-defined vocabulary is the next word in the text MJH. The next word in the text means the word to be added at the end of the text. The hidden feature vectors of each intermediate layer of the teacher model are represented as the feature vectors of each word in the current text.

[0038] Step 203: Calculate the first loss function value and the sum of the loss function values ​​under the previous iteration number corresponding to the t-th iteration number to obtain the loss function value under the current iteration number corresponding to the t-th iteration number; the first loss function value is the value obtained by inputting the probability distribution set under the current iteration number corresponding to the t-th iteration number into the loss function; the loss function includes forward kl divergence and backward kl divergence.

[0039] Step 204: Determine whether the marker word is a non-terminal marker. If so, append the marker word to the text of the current iteration number corresponding to the t-th loop number to obtain the text of the next iteration number corresponding to the t-th loop number. Then increment the iteration number corresponding to the t-th loop number by 1 and enter the next iteration corresponding to the t-th loop number. The marker word is obtained based on the probability distribution of the teacher model's output logit in the probability distribution set corresponding to the current iteration number.

[0040] Step 205: If not, then assume that the knowledge transfer work of the text has been completed. Update the parameters of the student model corresponding to the (t-1)th iteration number according to the loss function value of the current iteration number corresponding to the tth iteration number to obtain the student model corresponding to the tth iteration number. Then increment the iteration number t by 1 and initialize the iteration number corresponding to the next iteration number. Enter the next iteration until the text in the training dataset has been completely extracted. Save the parameters of the student model corresponding to the last iteration number and the architecture of the student model.

[0041] By implementing steps 201 to 205 above, and using the unlabeled training dataset to complete the knowledge transfer from the teacher model (large model) to the student model (small and medium-sized model), the accuracy and generalization ability of the student model can be improved.

[0042] In another exemplary embodiment of this application, step 201 further includes:

[0043] Step 1: Select the teacher model and the student model. The teacher model is a fully pre-trained and fine-tuned open-source large language model. The student model can be a manually constructed model or an open-source model with fewer parameters. It is necessary to ensure that each basic layer of the student model and the teacher model is isomorphic and that the number of layers in the teacher model is greater than or equal to the number of layers in the student model.

[0044] Step 2: Prepare text data in the relevant field to form a training dataset. An unlabeled dataset is sufficient. This dataset will be used to transfer knowledge from the teacher model to the student model during the knowledge distillation process.

[0045] Step 3: Load the teacher model and student model into the computer's video memory or RAM, freeze all parameters of the teacher model, and unlock all parameters of the student model.

[0046] In another exemplary embodiment of this application, the calculation process of the first loss function value is as follows:

[0047] Calculate the forward kl divergence and backward kl divergence values ​​based on the probability distributions of the teacher model's output logit and the student model's output logit in the probability distribution set corresponding to the current iteration number at the t-th iteration number.

[0048] The feature loss function is calculated based on the vector representations of the hidden features of each intermediate layer of the teacher model and the vector representations of the hidden features of each intermediate layer of the student model in the probability distribution set corresponding to the current iteration number at the t-th iteration number.

[0049] The first loss function value is obtained based on the values ​​of the forward KL divergence, the backward KL divergence, and the feature loss function.

[0050] In another exemplary embodiment of this application, the loss function formula is specifically as follows:

[0051] Loss kd =λ[μD RKKL (p||q)+(1-μ)D FKL [(p||q)]+(1-λ)Loss feature Among them, Loss kd Let λ represent the loss function, and let λ represent the first hyperparameter, the purpose of which is to minimize the loss. logit and Loss feature Transformed to the same order of magnitude, Loss logit The value represents the harmonic Kullback-Leibler (HKL) divergence, and the loss is... logit =μD RKL (p||q)+(1-μ)D FKL (p||q), where μ represents the second hyperparameter, and μ = 0.05 is recommended, λ = n / (1+n). Optimal hyperparameter settings can also be obtained through grid search. D RKL (p||q) represents the reverse Kl divergence, D FKL (p||q) represents the forward KL divergence, and Loss feature This represents the feature loss function.

[0052] In another exemplary embodiment of this application, the forward KL divergence and backward KL divergence are calculated based on the probability distribution of the teacher model's output logit and the probability distribution of the student model's output logit in the probability distribution set corresponding to the current iteration number at the t-th iteration number. Specifically, this includes:

[0053] The forward KL divergence and backward KL divergence values ​​are calculated based on the probability distribution set of each word in the preset vocabulary at the current iteration number corresponding to the t-th iteration, which corresponds to the probability distribution of the teacher model's output logit, and the probability distribution set of each word in the preset vocabulary at the current iteration number corresponding to the t-th iteration, which corresponds to the probability distribution of the student model's output logit.

[0054] The formula for forward KL divergence is:

[0055] The formula for the inverse KL divergence is:

[0056] Where p(x) and q(x) represent the probability that a word x in the pre-defined vocabulary is the next word in the text, i.e., the probability of a word x in the pre-defined vocabulary in the probability distribution of the output logit of the teacher model, and the probability that a word x in the pre-defined vocabulary is the next word in the text, i.e., the probability of a word x in the pre-defined vocabulary in the probability distribution of the output logit of the student model, respectively. Here, X represents the pre-defined vocabulary.

[0057] In another exemplary embodiment of this application, the feature loss function formula is:

[0058] Among them, Q l P represents the vector representation of the hidden features in the intermediate layer l of the student model. l′ The vector representation of the hidden feature of the intermediate layer l′ of the teacher model corresponds to the intermediate layer l of the student model. L represents the set of intermediate layers of the student model, n represents the total number of intermediate layers of the student model, and MSE() is the mean squared error loss function.

[0059] In another exemplary embodiment of this application, after determining whether the marker is a non-terminal marker, if so, the marker is appended to the text of the current iteration corresponding to the t-th loop number to obtain the text of the next iteration corresponding to the t-th loop number. Then, the iteration number corresponding to the t-th loop number is incremented by 1, and the next iteration corresponding to the t-th loop number is entered. Before this, the process also includes:

[0060] The probability sequence is obtained by sorting the probabilities of the teacher model's output logit probability distribution in the probability distribution set corresponding to the current iteration number at the t-th iteration number from largest to smallest. For example, sorting p(A) = 0.1, p(B) = 0.1, p(C) = 0.2, p(D) = 0.1, p(E) = 0.1, and p(F) = 0.4 from largest to smallest.

[0061] Select any one word from all words corresponding to the first M probabilities in the probability sequence as the marker word. M can be 3. Select the word with the highest probability, F, as the marker word and append it to the current text, so the current text becomes MJHF.

[0062] In another exemplary embodiment of this application, the parameters of the student model corresponding to the (t-1)th iteration number are updated according to the loss function value corresponding to the current iteration number for the tth iteration number to obtain the student model corresponding to the tth iteration number. Then, the iteration number t is incremented by 1 and the iteration number corresponding to the next iteration number is initialized, and the next iteration begins. Specifically:

[0063] The accumulated gradient is backpropagated using the loss function value corresponding to the current iteration number at the t-th iteration number, and the parameters of the student model corresponding to the (t-1)-th iteration number are updated. After clearing the accumulated gradient of the parameters, the iteration number t is incremented by 1, and the iteration number corresponding to the next iteration number is initialized, and the next iteration begins. This is a well-known process.

[0064] In another exemplary embodiment of this application, the parameters and architecture of the student model corresponding to the last loop count are specifically saved as follows:

[0065] Save the parameters and architecture of the student model corresponding to the last loop count to the hard disk, thus completing the transfer process of domain-specific knowledge from the teacher model to the student model.

[0066] The knowledge distillation method provided in this application can improve the fitting ability of the student model to the teacher model, thereby achieving performance closer to the teacher model (large language model) on the student model with a much smaller number of parameters than the teacher model.

[0067] This application also provides an application scenario where the knowledge distillation method described above is applied. Specifically, the knowledge distillation method provided in this embodiment can be applied to the lightweighting of models in a specific domain. The scenario includes three steps: model selection, model pruning, and model domain knowledge transfer. The process begins with model selection, followed by model pruning to obtain a lightweight model, and then proceeds to model domain knowledge transfer. The knowledge distillation method provided in this embodiment belongs to the knowledge transfer step. When the knowledge distillation method is applied to the question-answering domain, the text in the above knowledge distillation method specifically refers to a question-answering text dataset.

[0068] Based on the same inventive concept, this application also provides a knowledge distillation apparatus for implementing the knowledge distillation method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations in one or more knowledge distillation apparatus embodiments provided below can be found in the limitations of the knowledge distillation method described above, and will not be repeated here.

[0069] In one exemplary embodiment, a knowledge distillation apparatus is provided, comprising:

[0070] An extraction module is used to extract a text without replacement from the training dataset in the t-th iteration as the text for the t-th iteration; the training dataset includes multiple texts.

[0071] The probability distribution determination module is used to input the text at the current iteration number corresponding to the t-th iteration number into the teacher model and the student model corresponding to the (t-1)-th iteration number, respectively, to obtain the probability distribution set at the current iteration number corresponding to the t-th iteration number. The probability distribution set includes the probability distribution of the output logit of the teacher model, the vector representation of the hidden features of each intermediate layer of the teacher model, the probability distribution of the output logit of the student model, and the vector representation of the hidden features of each intermediate layer of the student model. The text at the initial iteration number corresponding to the t-th iteration number is the text at the t-th iteration number. The teacher model is a pre-trained large language model.

[0072] The loss function value calculation module is used to calculate the first loss function value and the sum of the loss function value under the previous iteration number corresponding to the t-th iteration number to obtain the loss function value under the current iteration number corresponding to the t-th iteration number; the first loss function value is the value obtained by inputting the probability distribution set under the current iteration number corresponding to the t-th iteration number into the loss function; the loss function includes forward kl divergence and backward kl divergence.

[0073] The judgment module is used to determine whether the marker word is a non-terminal marker. If so, the marker word is appended to the text of the current iteration number corresponding to the t-th loop number to obtain the text of the next iteration number corresponding to the t-th loop number. Then, the iteration number corresponding to the t-th loop number is incremented by 1, and the next iteration corresponding to the t-th loop number is entered. The marker word is obtained based on the probability distribution of the output logit of the teacher model in the probability distribution set of the current iteration number corresponding to the t-th loop number.

[0074] The parameter update module is used to update the parameters of the student model corresponding to the (t-1)th iteration number based on the loss function value corresponding to the current iteration number corresponding to the tth iteration number if no, then increment the iteration number t by 1 and initialize the iteration number corresponding to the next iteration number, and enter the next iteration until the text in the training dataset has been completely extracted, and save the parameters and architecture of the student model corresponding to the last iteration number.

[0075] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores knowledge distillation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a knowledge distillation method.

[0076] Those skilled in the art will understand that Figure 3 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0077] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0078] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0080] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0081] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of knowledge distillation, the method comprising: The knowledge distillation method comprises the following steps: In the tth cycle, a text in the training data set is extracted as a text in the tth cycle without replacement; In the current iteration corresponding to the tth cycle, the text in the current iteration corresponding to the tth cycle is input into the teacher model and the student model corresponding to the (t-1)th cycle respectively to obtain a probability distribution set in the current iteration corresponding to the tth cycle; the probability distribution set comprises a probability distribution of the output logit of the teacher model, a vector representation of the hidden feature of each intermediate layer of the teacher model, a probability distribution of the output logit of the student model, and a vector representation of the hidden feature of each intermediate layer of the student model; the text in the initial iteration corresponding to the tth cycle is the text in the tth cycle; the teacher model is a pre-trained large language model; The sum of the first loss function value and the loss function value in the previous iteration corresponding to the tth cycle is calculated to obtain the loss function value in the current iteration corresponding to the tth cycle; the first loss function value is a value obtained by inputting the probability distribution set in the current iteration corresponding to the tth cycle into a loss function; the loss function comprises a forward kl divergence and a reverse kl divergence; If the mark word is a non-end marker, the mark word is spliced to the back of the text in the current iteration corresponding to the tth cycle to obtain a text in the next iteration corresponding to the tth cycle, and then the iteration number corresponding to the tth cycle is increased by 1 to enter the next iteration corresponding to the tth cycle; the mark word is a word obtained according to the probability distribution of the output logit of the teacher model in the probability distribution set in the current iteration corresponding to the tth cycle; If not, the parameters of the student model corresponding to the (t-1)th cycle are updated according to the loss function value in the current iteration corresponding to the tth cycle to obtain the student model corresponding to the tth cycle, then the cycle number t is increased by 1 and the iteration number corresponding to the next cycle is initialized to enter the next cycle, until the texts in the training data set are extracted, the parameters and the architecture of the student model corresponding to the last cycle are saved.

2. The knowledge distillation method of claim 1, wherein, The first loss function value calculation process is as follows: The value of the forward kl divergence and the value of the reverse kl divergence are calculated according to the probability distribution of the output logit of the teacher model and the probability distribution of the output logit of the student model in the probability distribution set in the current iteration corresponding to the tth cycle; The value of the feature loss function is calculated according to the vector representation of the hidden feature of each intermediate layer of the teacher model and the vector representation of the hidden feature of each intermediate layer of the student model in the probability distribution set in the current iteration corresponding to the tth cycle; The first loss function value is obtained according to the value of the forward kl divergence, the value of the reverse kl divergence, and the value of the feature loss function.

3. The knowledge distillation method of claim 2, wherein, The loss function formula is as follows: Loss kd = λ [μD RKL (p||q) + (1 - μ)D FKL (p||q)] + (1 - λ) Loss feature where Loss kd denotes a loss function, λ denotes a first hyperparameter, μ denotes a second hyperparameter, D RKL (p||q) denotes a reverse kl divergence, D FKL (p||q) denotes a forward kl divergence, and Loss feature denotes a feature loss function.

4. The knowledge distillation method of claim 3, wherein, The feature loss function formula is: where Q l denotes the vector representation of the hidden features of the student model intermediate layer l, P l' denotes the vector representation of the hidden features of the teacher model intermediate layer l', the teacher model intermediate layer l' corresponding to the student model intermediate layer l, L denotes the set of student model intermediate layers, n denotes the total number of student model intermediate layers, and MSE() is a mean squared error loss function.

5. The knowledge distillation method of claim 1, wherein, If the mark word is a non-end marker, the mark word is spliced to the text under the current iteration number corresponding to the tth loop number, to obtain the text under the next iteration number corresponding to the tth loop number, and then the iteration number corresponding to the tth loop number is incremented by 1 to enter the next iteration corresponding to the tth loop number, and the previous iteration number corresponding to the tth loop number is also included: The probabilities in the probability distribution of the output logit of the teacher model in the probability distribution set under the current iteration number corresponding to the tth loop number are sorted from large to small to obtain a probability sequence; Any one of all the words corresponding to the first M probabilities in the probability sequence is selected as the mark word.

6. The knowledge distillation method of claim 2, wherein, The values of the forward kl divergence and the reverse kl divergence are calculated according to the probability distribution of the output logit of the teacher model and the probability distribution of the output logit of the student model in the probability distribution set under the current iteration number corresponding to the tth loop number, and specifically include: The values of the forward kl divergence and the reverse kl divergence are calculated according to the probabilities corresponding to the teacher model in the probability distribution set under the current iteration number corresponding to the tth loop number and the probabilities corresponding to the student model in the probability distribution set under the current iteration number corresponding to the tth loop number.

7. A knowledge distillation apparatus, characterized by, The knowledge distillation device includes: The extraction module is configured to extract a text from the training data set without replacement as the text under the tth loop number at the tth loop number; the training data set includes a plurality of texts; The probability distribution determination module is configured to input the text under the current iteration number corresponding to the tth loop number into the teacher model and the student model corresponding to the t-1th loop number to obtain a probability distribution set under the current iteration number corresponding to the tth loop number; the probability distribution set includes the probability distribution of the output logit of the teacher model, the vector representation of the hidden feature of each intermediate layer of the teacher model, the probability distribution of the output logit of the student model, and the vector representation of the hidden feature of each intermediate layer of the student model; the text under the initial iteration number corresponding to the tth loop number is the text under the tth loop number; the teacher model is a pre-trained large language model; The loss function value calculation module is configured to calculate the sum of the first loss function value and the loss function value under the previous iteration number corresponding to the tth loop number to obtain the loss function value under the current iteration number corresponding to the tth loop number; the first loss function value is the value obtained by inputting the probability distribution set under the current iteration number corresponding to the tth loop number into the loss function; the loss function includes the forward kl divergence and the reverse kl divergence; A judgment module is configured to judge whether the marked word is a non-end marker. If yes, the marked word is concatenated to the text under the current iteration number corresponding to the tth loop number to obtain the text under the next iteration number corresponding to the tth loop number, and then the iteration number corresponding to the tth loop number is increased by 1 to enter the next iteration corresponding to the tth loop number. The marked word is obtained according to the probability distribution of the output logit of the teacher model in the probability distribution set under the current iteration number corresponding to the tth loop number. A parameter updating module is configured to update the parameters of the student model corresponding to the (t-1)th loop number according to the loss function value under the current iteration number corresponding to the tth loop number to obtain the student model corresponding to the tth loop number, then increase the loop number t by 1 and initialize the iteration number corresponding to the next loop number, and enter the next loop until the text in the training data set is extracted, and save the parameters and the architecture of the student model corresponding to the last loop number.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge distillation method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the knowledge distillation method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the knowledge distillation method of any one of claims 1-6. The computer program is executed by the processor to implement the knowledge distillation method of any one of claims 1-6.

Images