Knowledge distillation method and device based on AI large model and electronic equipment

Abstract

The invention provides a distillation method and device based on AI large model knowledge and electronic equipment, and belongs to the technical field of artificial intelligence, in the method, a dynamic adjustment mechanism is adopted for the temperature, and along with the increase of a training period, the preset initial temperature corresponding to the training period is gradually reduced; besides, in each training period, the preset initial temperature is finely adjusted based on the distillation loss, so that the target temperature corresponding to each training sample is obtained, namely the temperature most suitable for the current distillation effect. The target distillation loss and the student loss obtained by recalculating based on the target temperature are more scientific, so that the total loss for updating the weight parameters of the student model is more accurate and robust, the student model is prompted to learn the knowledge of the teacher model more effectively, and a scientific and efficient learning process is learned. The finally obtained student model has good generalization ability, and the robustness of the model is significantly enhanced, so that the overall effect of knowledge distillation is significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular to a method, device, and electronic equipment based on AI large model knowledge distillation. Background Art

[0002] In today's large-scale model landscape, outstanding models undoubtedly shine. Leveraging their powerful capabilities, large models excel in handling complex tasks, producing high-quality results and delivering stunning visuals. However, this brilliance also carries significant risks. The sheer volume of parameters in large models consumes significant computational and storage resources, resulting in high costs and slow inference speeds. These issues, like heavy shackles, significantly limit the widespread application of large models and hinder their full potential in a wider range of fields.

[0003] Furthermore, achieving high accuracy and performance with large models often requires extensive training with massive amounts of data. While this process can empower the models, it also comes with extremely high computing costs and lengthy training times. For many companies, such a massive investment presents an insurmountable obstacle. Even if they see the potential of large models, practical cost and time constraints prevent them from venturing into the market, forcing them to resort to cloud-based large models.

[0004] In contrast, small models exhibit distinct advantages. They offer significant advantages in computational efficiency, responsiveness, and cost control. They can respond quickly to tasks while effectively controlling costs, making them highly desirable in scenarios requiring high resource and speed. However, due to their limited parameters, small models often fall short in terms of accuracy, comprehensiveness, semantic capture, contextual logic, and knowledge capacity when faced with complex tasks, making them less capable than large models.

[0005] Against this backdrop, the emergence of knowledge distillation technology undoubtedly provides an effective solution to this contradiction. The core concept of knowledge distillation lies in cleverly and efficiently transferring the vast and rich knowledge contained in a large model, the teacher model, to a smaller model, the student model. Through this transfer, the distilled smaller model can achieve excellent performance similar to that of the larger model while significantly reducing computing resource requirements and deployment costs. This feature enables the smaller model to be easily adapted to mobile embedded devices and various resource-constrained application scenarios, breaking away from the traditional limitation of pre-training models from scratch and greatly expanding the application space and practicality of the smaller model.

[0006] To better understand large-scale model distillation technology, let’s first introduce what distillation is and how the student model learns the teacher model.

[0007] Knowledge distillation is a technique for transferring knowledge learned by a large model to a smaller model. The large model with many parameters and superior performance serves as the teacher model, while the smaller model with fewer parameters serves as the student model. During this distillation process, the teacher model outputs a series of word probability distributions that align with human preferences based on the input data. The student model then learns from these output probability distributions, identifying differences and simultaneously updating its own neural network weight parameters. Through repeated training, learning, and updating, the student model gradually learns to mimic the teacher model's behavior, generating high-quality, coherent text that better conforms to linguistic patterns and semantic logic, ultimately successfully absorbing the teacher model's knowledge.

[0008] Here are some examples: For example, if you input the sentence "What do you think is the core technological driving force behind the current development of the artificial intelligence industry?", the probability of the words (temperature is 5) output by the Softmax function of the teacher model neural network is as follows: Table 1

[0009] According to the above table data, it can be clearly seen that after the Transformer neural network inference, the optional words returned show a specific probability distribution (such as Figure 2 ). This probability distribution is calculated by applying the Softmax function to the output layer.

[0010] Among these possible words, "deep learning" has the highest probability, reaching 0.525. Based on the principle of probability priority, we selected "deep learning" as the next word in the sentence. In addition, some other words also have relatively high probabilities. Specifically, the probabilities of "machine learning" and "reinforcement learning" are significantly higher than those of "natural language processing" and "big data." This phenomenon clearly demonstrates that "machine learning" and "reinforcement learning" have a closer semantic correlation with the input sentence. These two words and the internal neural network parameters they refer to are actually the "hidden knowledge" contained in the teacher model. We hope that the student model can effectively absorb this implicit knowledge from the teacher model, rather than having the student model start from scratch during training. This is the core idea of ​​large-scale model distillation.

[0011] Knowledge Distillation (KD) is a technique that transfers knowledge learned by a large teacher model to a smaller student model. The core idea is to have the student model mimic the output of the teacher model. The student model learns not only the true labels used for its own training but also the soft labels output by the teacher model. This reduces the model's computational and storage requirements, improves its accuracy and generalization, and makes the student model more suitable for deployment and operation in resource-constrained environments.

[0012] Traditional distillation methods use a fixed temperature to distill knowledge from the student model. In the early stages of distillation, the student model differs significantly from the teacher model, and a fixed temperature makes it difficult for the student model to effectively learn the teacher model's complex knowledge. In the later stages of distillation, the student model gradually approaches the teacher model, and a fixed temperature fails to fully tap the student model's potential. This ultimately results in inefficient knowledge transfer and fails to fully leverage the advantages of knowledge distillation, limiting the student model's performance output and generalization capabilities.

[0013] In summary, in the traditional large-model knowledge distillation method, the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and needs to be improved in robustness, that is, the effect of knowledge distillation is poor. Summary of the Invention

[0014] In view of this, the purpose of the present invention is to provide a method, device and electronic device based on AI large model knowledge distillation to alleviate the technical problems in traditional large model knowledge distillation methods, in which the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and robustness that needs to be improved.

[0015] In a first aspect, an embodiment of the present invention provides a method for knowledge distillation based on an AI large model, the method comprising: Input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, and obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; Determine a preset initial temperature corresponding to the current training cycle, and scale the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature to respectively calculate soft labels of the teacher model and soft predictions of the student model, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft labels and soft predictions are probability distributions of each predicted word; Calculating the distillation loss based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; Fine-tuning the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; After scaling the original output logits of the teacher model and the original output logits of the student model by the target temperature, the target soft labels of the teacher model and the target soft predictions of the student model are recalculated respectively; Recalculate the target distillation loss based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; Obtaining a hard prediction of the current training sample output by the student model, and calculating a student loss based on the hard prediction and a hard label corresponding to the current training sample; The target distillation loss and the student loss are weightedly calculated to obtain the total loss of the current training sample, and the weight parameters of the student model are updated based on the total loss of all training samples in the current batch of training samples.

[0016] Furthermore, before inputting the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, the method further includes: Obtaining a current batch of training samples from the training sample set; Each word in each training sample in the current batch of training samples is converted into an embedding vector, and position information of each word in the corresponding training sample is determined.

[0017] Furthermore, the original output logits of the teacher model and the original output logits of the student model are scaled by the preset initial temperature, and the soft labels of the teacher model and the soft predictions of the student model are calculated respectively, including: Calculate the formula based on the soft label of the teacher model Compute the soft labels of the teacher model, where represents the probability of the i-th word output by the teacher model, Represents the original output logits of the i-th word output by the teacher model, represents the preset initial temperature, Represents the original output logits of the jth word output by the teacher model, Represents the total number of words output by the teacher model for the training samples; Soft prediction calculation formula based on the student model Compute the soft predictions of the student model, where represents the probability of the i-th word output by the student model, Represents the original output logits of the i-th word output by the student model, represents the preset initial temperature, Represents the original output logits of the jth word output by the student model, Represents the total number of words output by the student model for the training samples.

[0018] Furthermore, the distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model, including: Calculation formula based on distillation loss The distillation loss is calculated, wherein, represents the distillation loss, represents the preset initial temperature, represents the soft label of the i-th word output by the teacher model, represents the soft prediction of the i-th word output by the student model, and N represents the total number of output words.

[0019] Further, fine-tuning the preset initial temperature based on the distillation loss to obtain the target temperature of the current training sample includes: If the distillation loss is not within the preset distillation loss range and the number of updates of the temperature corresponding to the current training sample has not reached the preset maximum number of updates, the preset initial temperature is updated based on the preset temperature adjustment factor to obtain the updated temperature, and the updated temperature is used as the preset initial temperature. Return to the step of scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, until the distillation loss is within the preset distillation loss range, or the number of updates of the temperature corresponding to the current training sample reaches the preset maximum number of updates, and the finally obtained temperature is used as the target temperature of the current training sample.

[0020] Furthermore, updating the preset initial temperature based on a preset temperature adjustment factor includes: If the distillation loss is less than the lower limit of the preset distillation loss range, the calculation formula is updated according to the first temperature The preset initial temperature is updated to obtain the updated temperature, wherein: represents the updated temperature, represents the preset temperature adjustment factor, , represents the preset initial temperature, Indicates the lower limit of the preset temperature range; If the distillation loss is greater than the upper limit of the preset distillation loss range, the second temperature update formula is used. The preset initial temperature is updated to obtain the updated temperature, wherein: represents the updated temperature, represents the preset temperature adjustment factor, , represents the preset initial temperature, Indicates the upper limit of the preset temperature range.

[0021] Furthermore, calculating the student loss according to the hard prediction and the hard label corresponding to the current training sample includes: Calculation formula based on student losses Calculate the student loss, where represents the student loss, represents the i-th hard tag, represents the probability of the i-th word in the hard prediction, and N represents the total number of output words.

[0022] In a second aspect, an embodiment of the present invention further provides a device based on AI large model knowledge distillation, the device comprising: An input unit is used to input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, to obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; a determination and calculation unit, configured to determine a preset initial temperature corresponding to a current training cycle, and scale the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature to calculate soft labels of the teacher model and soft predictions of the student model, respectively, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft labels and soft predictions are probability distributions of the predicted words; a computing unit for computing a distillation loss based on a KL divergence between the soft labels of the teacher model and the soft predictions of the student model; a fine-tuning unit, configured to fine-tune the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; A first recalculation unit is configured to recalculate the original output logits of the teacher model and the original output logits of the student model after scaling them by the target temperature to obtain a target soft label of the teacher model and a target soft prediction of the student model; a second recalculation unit, for recalculating the target distillation loss based on the KL divergence between the target soft label of the teacher model and the target soft prediction of the student model; an acquisition and calculation unit, configured to acquire a hard prediction of the current training sample output by the student model, and calculate a student loss based on the hard prediction and a hard label corresponding to the current training sample; A calculation and update unit is used to perform weighted calculation on the target distillation loss and the student loss to obtain the total loss of the current training sample, and update the weight parameters of the student model based on the total loss of all training samples in the current batch of training samples.

[0023] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0024] In an embodiment of the present invention, a method for knowledge distillation based on an AI large model is provided, the method comprising: inputting the embedding vector of each word of a current training sample and the position information of each word in the current training sample into a teacher model and a student model, respectively, to obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in a current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; determining a preset initial temperature corresponding to the current training cycle, and scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft label and the soft prediction are the probability distribution of each word obtained by prediction; The distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; the preset initial temperature is fine-tuned based on the distillation loss to obtain the target temperature of the current training sample, where the target temperature is the ideal temperature of the current training sample; the original output logits of the teacher model and the original output logits of the student model are scaled by the target temperature, and then the target soft labels of the teacher model and the target soft predictions of the student model are recalculated respectively; the target distillation loss is recalculated based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; the hard prediction of the current training sample output by the student model is obtained, and the student loss is calculated based on the hard prediction and the hard label corresponding to the current training sample; the target distillation loss and the student loss are weighted to obtain the total loss of the current training sample, and the weight parameters of the student model are updated based on the total loss of all training samples in the current batch of training samples. From the above description, it can be seen that in the method based on AI large model knowledge distillation of the present invention, the temperature adopts a dynamic adjustment mechanism. As the training cycle increases, the preset initial temperature corresponding to the training cycle gradually decreases; in addition, in each training cycle, the preset initial temperature is finely adjusted based on the distillation loss to obtain the target temperature corresponding to each training sample, that is, the temperature that is most suitable for the current distillation effect. The target distillation loss and student loss recalculated based on the target temperature are more scientific, and the total loss used to update the student model weight parameters is also more accurate and robust, which enables the student model to effectively learn the knowledge of the teacher model and learn a scientific and efficient learning process. The final student model has good generalization ability and significantly enhances the robustness of the model, thereby significantly improving the overall effect of knowledge distillation, alleviating the technical problems in the traditional large model knowledge distillation method that the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and the robustness needs to be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A flowchart of a method for knowledge distillation based on an AI large model provided by an embodiment of the present invention; Figure 2 A schematic diagram of probability distribution at different temperature values ​​provided by an embodiment of the present invention; Figure 3 A schematic diagram of the process of large-scale model knowledge distillation provided by an embodiment of the present invention; Figure 4 Another schematic diagram of the process of large model knowledge distillation provided by an embodiment of the present invention; Figure 5 Another schematic diagram of the process of large model knowledge distillation provided by an embodiment of the present invention; Figure 6 A schematic diagram of temperature fine-tuning provided by an embodiment of the present invention; Figure 7 An overall schematic diagram of knowledge distillation provided by an embodiment of the present invention; Figure 8 A schematic diagram of a device based on AI large model knowledge distillation provided by an embodiment of the present invention; Figure 9 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions of the present invention are described clearly and completely below with reference to the embodiments. It is obvious that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0028] In the traditional large-model knowledge distillation method, the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and needs to be improved in robustness.

[0029] Based on this, in the method of knowledge distillation based on AI large models of the present invention, the temperature adopts a dynamic adjustment mechanism. As the training cycle increases, the preset initial temperature corresponding to the training cycle gradually decreases; in addition, within each training cycle, the preset initial temperature is fine-tuned based on the distillation loss to obtain the target temperature corresponding to each training sample, that is, the temperature that is most suitable for the current distillation effect. The target distillation loss and student loss recalculated based on the target temperature are more scientific, and the total loss used to update the student model weight parameters is also more accurate and robust, which enables the student model to effectively learn the knowledge of the teacher model and learn a scientific and efficient learning process. The final student model has good generalization ability and significantly enhances the robustness of the model, thereby significantly improving the overall effect of knowledge distillation.

[0030] To facilitate understanding of this embodiment, a method based on AI large model knowledge distillation disclosed in an embodiment of the present invention is first introduced in detail.

[0031] Example 1: According to an embodiment of the present invention, an embodiment of a method for knowledge distillation based on an AI large model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Figure 1 is a flow chart of a method based on AI large model knowledge distillation according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S102: Input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, and obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; In the embodiment of the present invention, the description is made by taking one training sample in a batch of training samples (a batch of training samples in a training sample set) in one epoch training cycle as an example.

[0033] The original output logits of the teacher model are actually the original output logits of the teacher model for each word, and the original output logits of the student model are actually the original output logits of the student model for each word. The original output logits are the output scores of the prediction data of the last fully connected layer of the neural network, which are the original prediction values ​​without the softmax function (not normalized). The calculation of logits is to convert the hidden layer of the last layer of the Transformer of the model into Input to the fully connected layer (the upper layer calculation process is similar, After Q / K / V self-attention, multi-head attention, activation function, residual connection and layer normalization, the linear transformation is mapped back to the original dimension to obtain hidden layer state), and finally get logits.

[0034]

[0035] in, is the weight matrix, is the bias vector, are logits, is the hidden layer state obtained from the previous layer.

[0036] Assume that the last hidden layer state has 10 neurons, the batch size is 3, and the output has 2 neurons. That is: The last hidden state has a shape of (3×10); is the weight matrix, shape is (10×2); is the bias vector, with shape (1×2); is the output, with shape (3×2).

[0037] The calculation formula and process are as follows: Hidden state layer Neurons:

[0038] Hidden layer transposed matrix :

[0039] Bias vector :

[0040] Matrix multiplication :

[0041] Calculate logits:

[0042] in, , substituting it into, we get 6 logits, Represents the i-th training sample in the batch (not the feature dimension).

[0043]

[0044] Here, there are three batches, each with two logits. For example, based on the example in the background technology section, "The core technological driving force behind the development of the artificial intelligence industry is," there are five logits (in practice, there may be more). These logits correspond to each word in the vocabulary. The larger the logits value, the higher the probability that the word will become the next. These raw prediction values ​​are not probabilities themselves.

[0045] The original output logits of the teacher model are scores (logits) that have not been normalized by softmax. These scores represent the predicted value of each word in the text generation task.

[0046] The original output logits of the student model are the same as the teacher's logits. The goal of the student model is to imitate the teacher model by learning values ​​similar to the teacher model's output (logits).

[0047] Step S104: Determine a preset initial temperature corresponding to the current training cycle, and scale the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature to obtain the soft labels of the teacher model and the soft predictions of the student model, respectively. As the training cycle increases, the preset initial temperature corresponding to the training cycle decreases. The soft labels and soft predictions are the probability distributions of the predicted words. Specifically, the temperature parameter plays a role in modulating the softness of the probability distribution of words in the model's output. Specifically, higher temperature values ​​result in a smoother and more dispersed probability distribution of generated words. This means the model's predictions for each word are more uniform, with a higher probability of selection, exhibiting a softer, less uncertain tendency. This allows for greater creativity and diversity in the output. Conversely, lower temperature values ​​result in a sharper and more concentrated probability distribution of words, and the model tends to provide more specific and unambiguous predictions.

[0048] Regarding the learning process, specifically, the teacher model will select a word with a probability close to 1, such as 0.981, and the probabilities of other words are close to 0. If the student model needs to learn hidden knowledge with a probability close to 0, it is necessary to add a temperature coefficient T to Softmax to smooth the probability value. When the temperature T value is small, the probability distribution will be sharper, and the probability value of high-probability words will be much higher than other words. On the contrary, when the T value is large, the probability distribution will be smoother, so that the probability value of the original high-probability words will be lowered, and the probability value of the low-probability words will increase, so that the probability difference between each word is reduced, so that the final output probability distribution will no longer be concentrated on one or several words, but will be more evenly distributed on each word. With such temperature adjustment, the student model will be able to tap into more hidden knowledge during the learning and training process, rather than just being limited to a few words with extremely high probability values. If Figure 2 As shown in the figure, the probability distribution under different temperature values ​​is the most uniform when T=10, with no obvious preference; when T=5, the probability distribution gradually differentiates; and when T=1, the probability distribution is sharply concentrated.

[0049] In summary, temperature regulation is crucial for student model learning. The temperature regulation strategy we designed below is to balance the model's overly "creative and diverse" or overly "accurate and stable" prediction outputs, making the prediction results automatically smoother or sharper based on the actual situation. This avoids the model's excessive bias towards certain words, thereby absorbing more hidden knowledge from the teacher model and capturing more subtle semantic information and features, ultimately improving the stability, logical coherence, and generalization ability of the student model.

[0050] Temperature adjustment strategy: Use different preset initial temperatures for different training cycles. Initially (i.e., the first few training cycles) use a higher preset initial temperature, and then gradually reduce it later. (That is, as the training cycle increases, the corresponding preset initial temperature decreases. The preset initial temperature for each training cycle can be determined by setting a linear function, specifically a linear function between the training cycle and the preset initial temperature.) If the total number of epochs is n, such as 100 (an epoch is the number of times the model trains the entire training set. In each training cycle, the model iterates over all batches in the data loader), set a higher preset initial temperature at the beginning of training. As the training cycle increases, gradually reduce the preset initial temperature until it reaches 1. For example, the preset initial temperature could be 10 for the first training cycle, 9.9 for the second training cycle, 9.8 for the third training cycle, and so on. During each epoch of training, the preset initial temperature will be dynamically adjusted according to the progress of the current training cycle, so that the student model can repeatedly train and learn, find the differences with the teacher model, and correct the loss in time.

[0051] At the beginning of training, a higher preset initial temperature is set (making the soft labels of the teacher model smoother and the student model can learn from a wider distribution of words). As training progresses, the preset initial temperature is gradually reduced (making the student model begin to imitate the behavior of the teacher model more accurately). At the end of training, the preset initial temperature will gradually decrease to 1 (the soft labels of the teacher model become sharper and the student model begins to focus on learning hard labels (true labels)).

[0052] In addition to periodic adjustments to the preset initial temperature during epochs, a more precise secondary temperature adjustment (i.e., fine-tuning the preset initial temperature based on the distillation loss) is also performed during this process. Batching involves dividing the training set into multiple batches, each containing a certain number of training examples.

[0053] The process of calculating the soft labels of the teacher model and the soft predictions of the student model is described in detail below and will not be repeated here.

[0054] Step S106, calculating the distillation loss based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; Specifically, distillation loss is calculated by comparing the soft labels generated by the teacher model with the soft predictions generated by the student model. It measures the difference between the probability distributions of the student model's output and the teacher model's output, using the KL (Kullback-Leibler Divergence) algorithm. The goal is to make the student model mimic the teacher model's behavior by minimizing this loss function. A smaller loss function indicates more accurate predictions and a smaller difference in the probability distributions of the teacher and student models' outputs.

[0055] This process will be described in detail below and will not be repeated here.

[0056] Step S108, fine-tuning the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; Specifically, this process is to fine-tune the preset initial temperature according to the KL divergence value (i.e., distillation loss) to obtain the target temperature of the current training sample. The fine-tuning process is described in detail below and will not be repeated here.

[0057] Step S110, after scaling the original output logits of the teacher model and the original output logits of the student model by the target temperature, recalculate the target soft label of the teacher model and the target soft prediction of the student model respectively; Specifically, the process of recalculating the target soft labels of the teacher model and the target soft predictions of the student model is the same as the process of calculating the soft labels of the teacher model and the soft predictions of the student model. For details, please refer to the relevant description below and will not be repeated here.

[0058] Step S112, recalculating the target distillation loss based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; Specifically, the process of recalculating the target distillation loss is the same as the process of calculating the distillation loss. For details, please refer to the relevant description below and will not be repeated here.

[0059] Step S114: Obtain the hard prediction of the current training sample output by the student model, and calculate the student loss based on the hard prediction and the hard label corresponding to the current training sample; Specifically, the hard prediction is the predicted word obtained by the softmax function of the student model, which is used to calculate the cross entropy loss with the hard label (the true result). By converting the probability distribution output by the student model into a word label, the word with the highest probability is selected as the prediction result (that is, the hard prediction of the student model), and the temperature T is 1.

[0060] The student's hard labels (i.e., hard labels) are the ground truth labels of the training set. They are used to calculate the cross-entropy loss between the student model's predictions (i.e., the student model's hard predictions) and the hard labels (the ground truth). Hard labels are typically represented in one-hot encoding. For an input label example, the hard label represents the label of the correct word, with a label value of 1, and the label value of all other words is 0.

[0061] Step S116, perform weighted calculation on the target distillation loss and the student loss to obtain the total loss of the current training sample, and update the weight parameters of the student model based on the total loss of all training samples in the current batch of training samples.

[0062] Specifically, the total loss is the weighted sum of the target distillation loss and the student loss. The backpropagation algorithm is then used to update the weight parameters of the student model. The goal is to minimize the target distillation loss and the student loss so that the student model is close to the output of the teacher model and the output of the student model is close to the true label (i.e., hard label).

[0063] Formula: The total loss is the weighted sum of the target distillation loss and the student loss.

[0064]

[0065] in, represents the total loss, represents the target distillation loss, Indicates student loss, is the weight coefficient ( =0.5).

[0066] The alpha parameter dynamically adjusts the weighting strategy. By dynamically adjusting the weighting between the KL divergence loss (i.e., the target distillation loss) and the CE cross entropy loss (i.e., the student loss) based on the performance and progress of the student model during training, the flexibility and effectiveness of the distillation process can be significantly enhanced.

[0067] Specifically, the alpha parameter balances the impact of the two losses. When calculating the total loss, alpha determines the proportion of the KL divergence loss (target distillation loss) and the CE cross entropy loss (i.e., the student loss) to the total loss. When the alpha value is close to 0, the cross entropy loss is weighted more heavily, meaning that the student model places greater emphasis on accurate matching with hard labels. When alpha is close to 1, the KL divergence loss is weighted more heavily, and the student model tends to learn more extensive knowledge and smoother predictions from the teacher model. If alpha is set to 0.5, the impact of these two losses on the total loss is balanced. In summary, the application of alpha in the actual training and learning process can flexibly adjust the learning focus according to specific needs, such as different stages or different model performance. This ensures the accuracy of hard predictions during training while capturing the hidden knowledge of the teacher model. Finding the optimal balance between accuracy and knowledge transfer effectively improves the quality and efficiency of the distillation process.

[0068] In addition to the total loss calculation described above, weighted loss can also be combined with intermediate layer losses such as attention loss and hidden layer loss, and with or without CoT (Chain-of-Thought) loss to integrate into the comprehensive calculation (such as Figure 3 ). Or adopt multiple teacher models at the same time, so that the student model can learn from multiple angles, directions, and fields, so that the student model can achieve a more comprehensive understanding.

[0069] When updating the student model's weight parameters based on the total loss of all training samples, backpropagation can be used to update the student model's weight parameters. Specifically, after calculating the total loss, backpropagation is used to calculate the gradient of the loss function with respect to the student model parameters. These gradients are then used to update the student model's weights and biases, essentially updating the model parameters in the opposite direction of the student model's gradient to minimize the loss function. This accelerates the distillation training process and reduces the total loss.

[0070] Note: This process changes the internal parameters of the student model itself, and is completely different from the traditional RAG retrieval enhancement generation method. RAG is essentially an external knowledge base of the model, which has no direct relationship with the large model itself and does not change the model.

[0071] During the back propagation process, the loss It is used to calculate the gradient and update the weight of the student model. Specifically, the weight of the student model is updated by the following formula:

[0072] Right now

[0073] in: are the parameters in the student model, namely the weight matrix W and the bias b vector from the i-th layer to the i+1 layer in the neural network. The weight matrix determines the strength of the connection between different neurons in the neural network. are the parameters of the student model.

[0074] Is the learning rate, a hyperparameter used to control the step size of each weight update. A suitable learning rate will enable the model loss function to converge quickly to fit the data, but too large or too small a learning rate will have an adverse effect on model training. When the learning rate is too large, the step size of each update will be too large, causing the parameter update to directly skip the lowest point of the loss function, making it impossible for the loss function to converge and making it difficult for the model to learn effective features. If the step size is too small, the step size of each parameter update is very small. Although this can ensure the stability of the parameter update, it also means that the model needs to go through more iterations and the longest training time to converge to the lowest point of the loss function, resulting in a waste of computing resources and low training efficiency.

[0075] It is the total loss function, which includes the distillation loss (the difference between the teacher model and the student model) and the student loss (the difference between the hard prediction of the student model and the true label), and is used to measure the gap between the model prediction and the true label.

[0076] W is the neuron connection weight, b is the bias, is the input feature of the i-th training sample, is the target value (label) of the i-th training sample, and N is the number of samples. Represents the prediction error of the model for the i-th training sample.

[0077] is the loss function For neural network parameters The gradient of w (the gradient represents the rate of change of the loss function with respect to the current weight value), or the partial derivative with respect to weight w, represents the direction and magnitude of the update, with the direction being the fastest increasing direction of the loss function. In the backpropagation algorithm, weights must be updated in the opposite direction of the gradient to reduce the loss function. This gradient is calculated by backpropagating from the output layer using the chain rule. is the loss function Gradient of the student model parameters. By repeatedly iterating forward to calculate the loss, and backpropagating to calculate the gradient of the loss relative to the neural network parameters, and finally updating the weights and biases in the opposite direction of the gradient, the student model loss value can be continuously reduced and converged, and the performance will gradually improve.

[0078] In addition to the above, during backpropagation, you can try using various optimization methods such as Dropout to randomly discard neurons, adjust batch size, adjust learning rate, early interruption, RL reinforcement learning, random / small batch / adaptive Adam / AdamW gradient descent, different activation functions, and weight regularization to achieve better distillation results. It is important to note that the teacher model needs to be set to no-grad (frozen teacher model). This is to avoid consuming computing resources by calculating the teacher model's gradients and to prevent the teacher model's weight parameters from being updated during backpropagation, so that the update process only updates the student model.

[0079] After completing the above process, the training of the student model for the current batch of training samples is completed. The training of the student model for all batches of training samples is completed, and the training of the current training cycle of the student model for the training sample set is completed. The training of all training cycles is completed, or the total loss converges to the preset value.

[0080] In an embodiment of the present invention, a method for knowledge distillation based on an AI large model is provided, the method comprising: inputting the embedding vector of each word of a current training sample and the position information of each word in the current training sample into a teacher model and a student model, respectively, to obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in a current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; determining a preset initial temperature corresponding to the current training cycle, and scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft label and the soft prediction are the probability distribution of each word obtained by prediction; The distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; the preset initial temperature is fine-tuned based on the distillation loss to obtain the target temperature of the current training sample, where the target temperature is the ideal temperature of the current training sample; the original output logits of the teacher model and the original output logits of the student model are scaled by the target temperature, and then the target soft labels of the teacher model and the target soft predictions of the student model are recalculated respectively; the target distillation loss is recalculated based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; the hard prediction of the current training sample output by the student model is obtained, and the student loss is calculated based on the hard prediction and the hard label corresponding to the current training sample; the target distillation loss and the student loss are weighted to obtain the total loss of the current training sample, and the weight parameters of the student model are updated based on the total loss of all training samples in the current batch of training samples. From the above description, it can be seen that in the method based on AI large model knowledge distillation of the present invention, the temperature adopts a dynamic adjustment mechanism. As the training cycle increases, the preset initial temperature corresponding to the training cycle gradually decreases; in addition, in each training cycle, the preset initial temperature is finely adjusted based on the distillation loss to obtain the target temperature corresponding to each training sample, that is, the temperature that is most suitable for the current distillation effect. The target distillation loss and student loss recalculated based on the target temperature are more scientific, and the total loss used to update the student model weight parameters is also more accurate and robust, which enables the student model to effectively learn the knowledge of the teacher model and learn a scientific and efficient learning process. The final student model has good generalization ability and significantly enhances the robustness of the model, thereby significantly improving the overall effect of knowledge distillation, alleviating the technical problems in the traditional large model knowledge distillation method that the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and the robustness needs to be improved.

[0081] The above content briefly introduces the method of knowledge distillation based on AI large model of the present invention. The specific contents involved are described in detail below.

[0082] In an optional embodiment of the present invention, before inputting the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, the method further includes the following steps: (1) Obtain the current batch of training samples in the training sample set; (2) Convert each word in each training sample in the current batch of training samples into an embedding vector and determine the position information of each word in the corresponding training sample.

[0083] Specifically, such as Figure 3 、 Figure 4 and Figure 5 As shown in the figure, tokenized text data (words, i.e., each word in each training example) is input. These input tokens are then passed through an embedding layer, which maps each token into a vector (embedding vector) in a high-dimensional space. Words with similar semantics are placed close together. The corresponding sine and cosine positional information is then added to represent the position of each word in the corresponding training example. This allows the Transformer to better capture both local and global information in the input training example sequence. This data is then passed to the model.

[0084] like Figure 4 As shown in the figure, in the model's structure, the input layer is on the far left, and the output layer is on the far right, with multiple hidden layers located between them. The value of each neuron in the hidden and output layers is calculated by multiplying the value of each node in the previous layer by the corresponding weight, then accumulating the result and adding the bias. The algorithm's calculation process is as follows: starting from the input layer, calculations are performed from left to right, layer by layer. During this process, if a neuron is "activated," it is sent to the next layer and affects the result. Otherwise, the neuron will not affect the result (the weight and bias contribute to the result to zero), until the final calculation result is obtained at the output layer.

[0085] Teacher Model: A teacher model is a pre-trained, high-performance model. It's typically large and has numerous parameters, enabling more accurate predictions. The teacher model's neural network consists of multiple layers (Layer 1 through Layer X). These layers perform tensor processing on the input data through self-attention, multi-head attention mechanisms, feedforward networks, and normalization layers, ultimately generating soft labels, which are the model's output probability distributions.

[0086] Student Model: The student model is a smaller version of the teacher model, with fewer layers (Layer 1 to Layer Y) and fewer parameters. Through knowledge distillation, the student model learns the output probability distribution of the teacher model to improve its generalization ability and performance.

[0087] In an optional embodiment of the present invention, the original output logits of the teacher model and the original output logits of the student model are scaled by a preset initial temperature, and then the soft labels of the teacher model and the soft predictions of the student model are calculated respectively, which specifically includes the following steps: (1) Calculate the formula based on the soft label of the teacher model Compute the soft labels of the teacher model, where represents the probability of the i-th word output by the teacher model, Represents the original output logits of the i-th word output by the teacher model, Indicates the preset initial temperature. Represents the original output logits of the jth word output by the teacher model, Represents the total number of words output by the teacher model for the training samples; (2) Soft prediction calculation formula based on the student model Compute the soft predictions of the student model, where represents the probability of the i-th word output by the student model, Represents the original output logits of the i-th word output by the student model, Indicates the preset initial temperature. Represents the original output logits of the jth word output by the student model, Represents the total number of words output by the student model for the training samples.

[0088] Specifically, the teacher model converts logits (the raw output logits) into a softmax function, where the sum of all values ​​is 1. During this process, the temperature parameter (the preset initial temperature) is applied to obtain the teacher model's soft labels, which are the probability distribution of the words predicted by the teacher model. The temperature parameter acts on the softmax function. If the temperature parameter is greater than 1, the output probability distribution will be smoother and more uniform. The specific formula is:

[0089] in, Represents the raw output logits of the i-th word output by the teacher model, which represents the unnormalized score of the model for each word category. is the exponential sum of all word logits, C is the size of the vocabulary, that is, the total number of words output by the teacher model for the training sample, e is the base of the natural logarithm of exp, Indicates the preset initial temperature. The higher the temperature, The lower the value, the more uniform the distribution will be. Conversely, the greater the value, the more uniform the distribution will be. This formula ensures that the result is a reasonable probability distribution.

[0090] Soft predictions are the probability distributions obtained by the student model using the softmax function, which is the logarithm of the logits. The formula is similar to that used by the teacher. They represent the model's predicted probability for each word. They are also calculated based on the scores of the model's logits normalized by the softmax function.

[0091] In an optional embodiment of the present invention, the distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model, specifically comprising the following steps: Calculation formula based on distillation loss Calculate the distillation loss, where represents the distillation loss, Indicates the preset initial temperature. represents the soft label of the i-th word output by the teacher model, represents the soft prediction of the i-th word output by the student model, and N represents the total number of output words.

[0092] Specifically, the distillation loss is expressed by calculating the KL divergence between the output of the student model and the output of the teacher model.

[0093]

[0094] and They are the Log_Softmax of the student model and the Softmax probability output of the teacher model. That is:

[0095] in, It is used to balance the effect of temperature on the gradient. When the temperature T is large, the output probability distribution will become very smooth, and the KL divergence loss will be very small, which may cause the student model to not be able to effectively learn the knowledge of the teacher model. The loss can be appropriately amplified so that the student model pays more attention to the relative probability information output by the teacher model, thereby better balancing the relationship between different words during the learning process. Denotes the distillation loss, which is used to encourage the student model to imitate the output distribution of the teacher model. P(i) is the soft label of the i-th word output by the teacher model, that is, the teacher model's predicted probability of the i-th word output. Q(i) is the soft prediction of the i-th word output by the student model, that is, the student model's predicted probability of the i-th word output. is the KL divergence, which measures the difference between two distributions P and Q. It is a logarithmic term that measures the degree of distribution difference between the teacher model and the student model at each value of i. The smaller the KL divergence, the closer the student model's prediction is to the teacher model. If the KL divergence is zero, it means that the student model and the teacher model predict the same probability, that is, .

[0096] The ultimate goal of calculating the distillation loss is to minimize the KL divergence between the student model and the teacher model, so that the output distribution of the teacher model and the student model is neither too sharp nor too flat, thereby better transferring knowledge.

[0097] In an optional embodiment of the present invention, the preset initial temperature is fine-tuned based on the distillation loss to obtain the target temperature of the current training sample, which specifically includes the following steps: If the distillation loss is not within the preset distillation loss range and the number of updates of the temperature corresponding to the current training sample has not reached the preset maximum number of updates, the preset initial temperature is updated based on the preset temperature adjustment factor to obtain the updated temperature, and the updated temperature is used as the preset initial temperature. Return to execute the steps of scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, until the distillation loss is within the preset distillation loss range, or the number of updates of the temperature corresponding to the current training sample reaches the preset maximum number of updates, and the final temperature is used as the target temperature of the current training sample.

[0098] Specifically, such as Figure 6 As shown, the configuration parameters and process are as follows: Assume that the preset initial temperature T = 5.0 corresponding to the current training cycle, and use the temperature value to adjust the logits of the teacher model and the student model.

[0099] Preset temperature adjustment factor Control the variation of T, constrain the stability of temperature, ensure that the temperature strategy does not change significantly and maintains relatively high accuracy.

[0100] Temperature range, , avoid T being too large or too small, in the process of dynamic temperature adjustment, the final target temperature will not be lower than , at the same time, the target temperature will not be higher than .

[0101] Adjust the preset maximum number of updates (The maximum number of temperature updates corresponding to a training sample in a training cycle) to avoid infinite adjustments.

[0102] KL divergence loss calculated for the current batch .

[0103] The lower limit of the KL range (i.e. the preset distillation loss range) Configurable, default setting is 0.5. When the calculated KL divergence is less than this lower limit, it means that the output distributions of the teacher model and the student model are too similar. In this case, it is necessary to reduce the temperature to increase the sharpness of the distribution.

[0104] The upper limit of the KL range (i.e. the preset distillation loss range) Configurable, default setting is 1.5. When the calculated KL divergence is greater than this upper limit, it means that the output distribution of the teacher model and the student model is too different. In this case, the temperature needs to be increased to smooth the distribution.

[0105] When implemented, the preset initial temperature is updated based on the preset temperature adjustment factor, specifically including: if the distillation loss is less than the lower limit of the preset distillation loss range, then according to the first temperature update formula The preset initial temperature is updated to obtain an updated temperature, where: Indicates the updated temperature, Indicates the preset temperature adjustment factor, , Indicates the preset initial temperature. Indicates the lower limit of the preset temperature range; if the distillation loss is greater than the upper limit of the preset distillation loss range, the formula is updated according to the second temperature The preset initial temperature is updated to obtain an updated temperature, where: Indicates the updated temperature, Indicates the preset temperature adjustment factor, , Indicates the preset initial temperature. Indicates the upper limit of the preset temperature range.

[0106] That is , indicating that the distribution is too smooth and the temperature needs to be lowered. , reduce the temperature according to the preset temperature adjustment factor, but not lower than .

[0107] when , indicating that the distribution difference is too large and the temperature needs to be increased. , increase the temperature according to the preset temperature adjustment factor, but not higher than .

[0108] when Not less than , no greater than , indicating that the current temperature is appropriate, that is, the target temperature.

[0109] Through the above fine-tuning process, Converge to a suitable value, and then determine The target temperature corresponding to convergence.

[0110] Determine whether the number of updates of the temperature corresponding to the current training sample has reached the preset maximum number of updates If it is not reached, the distillation loss (KL divergence) is recalculated based on the updated temperature.

[0111] Effect example: Assuming the preset initial temperature T of the current training cycle is 5 and the KL divergence is greater than 1.5, then fine-tune to increase the temperature; if the KL divergence is less than 0.5, then fine-tune to decrease the temperature. The preset temperature adjustment factor is 1.01, the minimum adjustment is 1, the maximum adjustment is 10, and the adjustment range is as shown in Table 2: Table 2

[0112] By adaptively and dynamically adjusting the temperature parameter T, the student model can more easily learn the knowledge of the teacher model in the early stages of training and gradually focus on the true labels in the later stages of training. This significantly improves the effect of knowledge distillation and generalization capabilities, especially in complex tasks and when data distribution is uneven.

[0113] In an optional embodiment of the present invention, the student loss is calculated based on the hard prediction and the hard label corresponding to the current training sample, specifically including the following steps: Calculation formula based on student losses Calculate student losses, where Indicates student loss, represents the i-th hard label, represents the probability of the i-th word in the hard prediction, and N represents the total number of output words.

[0114] Specifically, the difference between the hard prediction of the student model and the hard label, that is, the true label, is calculated using cross-entropy loss (Cross-Entropy Loss is used to measure the difference between the probability distribution of the model prediction and the probability distribution of the true label). Figure 7 shown.

[0115] Formula: Student loss is expressed as:

[0116] in, is the cross entropy loss of the student model. It measures the difference between the probability distribution of the output words predicted by the model and the true hard-labeled target words. The loss function calculates the difference between the hard predicted probability distribution Q and the one-hot distribution y of the true output words. is the hard prediction probability of the student model for the i-th word, obtained by softmax, is the value of the ith bit in the one-hot encoding of the true output word. If the true word is the ith word in the vocabulary, then is equal to 1, and the rest are 0. is the i-th probability in the hard prediction. is the raw (unnormalized) score of the i-th output word in the logits of the hard prediction output of the student model, given by composition. Convert logits into probabilities, representing the predicted probability of the i-th output word, represents the probability of the i-th word in the hard prediction, and N represents the total number of output words, that is, the total number of words in the output vocabulary.

[0117] Since the natural logarithm function log(x) (logarithm with base e) has a value range of (−∞,0] when x∈(0,1]), in order to obtain a non-negative loss value, it is necessary to add a minus sign before the formula to convert the negative value to a positive value.

[0118] The student model generated by the present invention's method for AI large-model knowledge distillation can be applied to mobile embedded devices. Large-model knowledge distillation is a technique for compressing a large pre-trained language model (often called a "teacher model") into a smaller, more efficient model (called a "student model"). The core of this process is that the student transfers the rich knowledge learned by the large model to a smaller model by mimicking the teacher model's behavior, predictions, or intermediate outputs, while preserving the performance of the original model as much as possible.

[0119] The method of AI large model knowledge distillation of the present invention includes a temperature adjustment strategy. The temperature adjustment strategy dynamically adjusts the temperature of the distillation process in real time based on multiple indicators such as the training progress of the student model (i.e., training cycle) and the distillation loss with the teacher model. In the early stage of training, the temperature is appropriately increased to make the soft label distribution output by the teacher model smoother and more uniform, so as to help the student model learn the probability distribution of the teacher model knowledge more easily; as the training progresses, the temperature is gradually lowered. In this process, the temperature is dynamically fine-tuned according to the KL divergence (distillation loss between the teacher model and the student model), so that the student model can focus more on learning the details of the teacher model knowledge and accurately imitate the output of the teacher model. Practice has shown that this method can significantly improve the flexibility and accuracy of the distillation process, and improve the learning process and generalization performance of the student model.

[0120] The purpose of this invention is to dynamically adjust the temperature according to the performance of the model during the training process, so as to better transfer the knowledge of the teacher model to the student model and improve the stability, logical coherence and generalization ability of the student model.

[0121] 1. Improving the student model's learning ability: By dynamically adjusting the temperature T, the student model gradually learns from the teacher model. In the early stages of training, the teacher model outputs smoother soft labels, allowing the student model to learn more relative semantic relationships between words. As training progresses, the temperature is gradually lowered, allowing the student model to focus more on the teacher model's precise behavior.

[0122] 2. Dynamically balance learning of soft and hard labels: Dynamic temperature adjustment automatically balances the learning ratio of soft labels (teacher model output) and hard labels (true labels) based on the training progress. Initially, the teacher model's soft labels provide rich knowledge, while later, the true hard labels improve accuracy, allowing the model to focus on learning true labels. This stabilizes the training process and improves the model's generalization ability.

[0123] 3. Effectively control the precision of training: By dynamically fine-tuning the temperature change amplitude based on the KL divergence using a preset temperature adjustment factor, the learning accuracy of the student model is effectively controlled, allowing the student model to more accurately imitate the behavior of the teacher model and avoid overly smooth or overly sharp soft labels, thereby improving stability and training effects.

[0124] 4. Improved robustness to noisy data: A higher temperature parameter can smooth the effects of noisy data, making the student model more robust on noisy data. As the temperature gradually decreases, the student model can gradually learn a more accurate distribution, thereby improving the overall robustness of the student model.

[0125] 5. Adaptability to Different Tasks and Data: Different tasks and data distributions are complex and diverse, and their sensitivity to temperature parameters varies, which can also affect distillation results. Dynamic temperature adjustment allows for flexible configuration of different temperature parameters based on task characteristics to adjust the smoothness of soft labels, thereby better adapting to different tasks and data requirements.

[0126] Compared to fixed temperature, a dynamic temperature adjustment strategy can significantly improve knowledge distillation. By adjusting the temperature in stages during the training cycle and dynamically fine-tuning it based on KL divergence, it stabilizes the training process, accelerates convergence, and thus improves model generalization.

[0127] The flexible temperature adjustment mechanism enables the student model to better inherit the "wisdom" of the teacher model, significantly reducing the computing power resources required for the student model to run or deploy, and providing strong support for the widespread application of AI technology in resource-constrained environments.

[0128] Key points of the present invention: A dynamic temperature regulator was constructed to continuously optimize the student model's learning process by dynamically adjusting the temperature parameter during the large-scale model knowledge distillation process. Specifically, the temperature parameter T is set to different initial values ​​at different training cycles, starting at a higher value at the beginning of training and gradually decreasing to a value close to 1 at the end. This phased temperature adjustment strategy helps the student model obtain different learning signals at different training stages, thereby improving the model's generalization and accuracy.

[0129] In the early stages of training, using a higher temperature parameter smoothes the teacher model's soft labels, allowing the student model to learn from a wider distribution of word categories. This smoothed label distribution helps the student model avoid prematurely falling into local optima, allowing it to better explore the parameter space. As training progresses, the temperature parameter is gradually lowered, and the student model begins to more accurately mimic the teacher model's behavior.

[0130] At the end of training, the temperature parameter approaches 1, the soft labels become sharper, and the student model begins to focus on learning the hard labels (the true labels). The goal of this stage is to make the student model's output closer to the true labels, thereby improving model accuracy. This phased temperature adjustment strategy allows the student model to obtain different learning signals at different training stages, thereby better balancing generalization and accuracy.

[0131] In addition to the phased temperature adjustment, the temperature regulator also performs secondary dynamic fine-tuning based on the value of the KL divergence loss of the batch. KL divergence is used to measure the difference between the output distribution of the student model and the output distribution of the teacher model. By dynamically adjusting the temperature parameter, the output distribution of the student model can be ensured to be consistent with the output distribution of the teacher model. Specifically, if the KL divergence is greater than the preset maximum value, it means that the output of the student model is significantly different from the output of the teacher model. In this case, the temperature parameter needs to be increased to smooth the distribution. Conversely, if the KL divergence is less than the preset minimum value, it means that the output of the student model is close to the output of the teacher model. In this case, the temperature parameter needs to be decreased to make the distribution sharper.

[0132] Through this dynamic adjustment mechanism, the temperature regulator can effectively control the learning process of the student model, ensuring that it always maintains consistency with the output distribution of the teacher model during training. In addition, to prevent the temperature parameter from being too large or too small, the temperature regulator also sets a temperature range (e.g., T∈[0.1, 10.0]) to ensure that the temperature parameter varies within a reasonable range.

[0133] In summary, the temperature regulator plays a crucial role in the large-scale model knowledge distillation process. Through phased temperature adjustment and dynamic temperature fine-tuning based on KL divergence, the temperature regulator can effectively optimize the student model's learning process, improving the model's generalization and accuracy. Ultimately, through the knowledge distillation process, not only does the lightweight student model inherit the "intelligence" of the teacher model and maintain its performance, but it also significantly reduces the computing power and storage resources required for model operation and deployment. This allows the student model to be independent of the large cloud model and run efficiently in resource-limited environments, while also avoiding security and legal issues such as copyright, privacy, and confidential information leakage. In short, this technology opens up new possibilities for the popularization and application of AI technology in areas such as autonomous driving, smart homes, and mobile embedded devices, providing important support for the popularization and promotion of AI technology in practical applications.

[0134] Example 2: An embodiment of the present invention also provides a device based on AI large model knowledge distillation. The device based on AI large model knowledge distillation is mainly used to execute the method based on AI large model knowledge distillation provided in Example 1 of the present invention. The following is a detailed introduction to the device based on AI large model knowledge distillation provided in an embodiment of the present invention.

[0135] Figure 8 is a schematic diagram of a device based on AI large model knowledge distillation according to an embodiment of the present invention, such as Figure 8As shown, the device mainly includes: an input unit 10, a determination and calculation unit 20, a calculation unit 30, a fine-tuning unit 40, a first recalculation unit 50, a second recalculation unit 60, an acquisition and calculation unit 70, and a calculation and update unit 80, wherein: An input unit is used to input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, to obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; a determination and calculation unit, configured to determine a preset initial temperature corresponding to the current training cycle, and to calculate the soft labels of the teacher model and the soft predictions of the student model after scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft labels and soft predictions are the probability distributions of the predicted words; A computation unit for computing the distillation loss based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; A fine-tuning unit, configured to fine-tune the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; The first recalculation unit is used to recalculate the target soft label of the teacher model and the target soft prediction of the student model after scaling the original output logits of the teacher model and the original output logits of the student model with the target temperature; The second recalculation unit is used to recalculate the target distillation loss based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; An acquisition and calculation unit, which is used to obtain the hard prediction of the current training sample output by the student model and calculate the student loss based on the hard prediction and the hard label corresponding to the current training sample; The calculation and update unit is used to perform weighted calculation on the target distillation loss and the student loss to obtain the total loss of the current training sample, and to update the weight parameters of the student model based on the total loss of all training samples in the current batch of training samples.

[0136] In an embodiment of the present invention, a device based on AI large model knowledge distillation is provided, the device comprising: inputting the embedding vector of each word of the current training sample and the position information of each word in the current training sample into a teacher model and a student model respectively, obtaining the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in a current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; determining a preset initial temperature corresponding to the current training cycle, and scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft label and the soft prediction are the probability distribution of each word obtained by prediction; The distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; the preset initial temperature is fine-tuned based on the distillation loss to obtain the target temperature of the current training sample, where the target temperature is the ideal temperature of the current training sample; the original output logits of the teacher model and the original output logits of the student model are scaled by the target temperature, and then the target soft labels of the teacher model and the target soft predictions of the student model are recalculated respectively; the target distillation loss is recalculated based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; the hard prediction of the current training sample output by the student model is obtained, and the student loss is calculated based on the hard prediction and the hard label corresponding to the current training sample; the target distillation loss and the student loss are weighted to obtain the total loss of the current training sample, and the weight parameters of the student model are updated based on the total loss of all training samples in the current batch of training samples. From the above description, it can be seen that in the device based on AI large model knowledge distillation of the present invention, the temperature adopts a dynamic adjustment mechanism. As the training cycle increases, the preset initial temperature corresponding to the training cycle gradually decreases; in addition, within each training cycle, the preset initial temperature is finely adjusted based on the distillation loss to obtain the target temperature corresponding to each training sample, that is, the temperature that is most suitable for the current distillation effect. The target distillation loss and student loss recalculated based on the target temperature are more scientific, and the total loss used to update the student model weight parameters is also more accurate and robust, which enables the student model to effectively learn the knowledge of the teacher model and learn a scientific and efficient learning process. The final student model has good generalization ability and significantly enhances the robustness of the model, thereby significantly improving the overall effect of knowledge distillation, alleviating the technical problems in the traditional large model knowledge distillation method that the student model cannot learn the knowledge of the teacher model well, the learning process is poor, and the final student model has poor generalization ability and the robustness needs to be improved.

[0137] Optionally, the device is further used to: obtain a current batch of training samples in a training sample set; convert each word in each training sample in the current batch of training samples into an embedding vector, and determine the position information of each word in the corresponding training sample.

[0138] Optionally, the determining and calculating unit is further configured to: calculate the formula according to the soft label of the teacher model Compute the soft labels of the teacher model, where represents the probability of the i-th word output by the teacher model, Represents the original output logits of the i-th word output by the teacher model, Indicates the preset initial temperature. Represents the original output logits of the jth word output by the teacher model, Represents the total number of words output by the teacher model for the training sample; calculated based on the soft prediction formula of the student model Compute the soft predictions of the student model, where represents the probability of the i-th word output by the student model, Represents the original output logits of the i-th word output by the student model, Indicates the preset initial temperature. Represents the original output logits of the jth word output by the student model, Represents the total number of words output by the student model for the training samples.

[0139] Optionally, the calculation unit is further configured to: calculate the distillation loss according to the formula Calculate the distillation loss, where represents the distillation loss, Indicates the preset initial temperature. represents the soft label of the i-th word output by the teacher model, represents the soft prediction of the i-th word output by the student model, and N represents the total number of output words.

[0140] Optionally, the fine-tuning unit is also used to: if the distillation loss is not within the preset distillation loss range and the number of updates of the temperature corresponding to the current training sample has not reached the preset maximum number of updates, then the preset initial temperature is updated based on the preset temperature adjustment factor to obtain the updated temperature, and the updated temperature is used as the preset initial temperature, and the step of scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature is returned to execute, and the soft label of the teacher model and the soft prediction of the student model are calculated respectively, until the distillation loss is within the preset distillation loss range, or the number of updates of the temperature corresponding to the current training sample reaches the preset maximum number of updates, and the final temperature is used as the target temperature of the current training sample.

[0141] Optionally, the fine-tuning unit is further configured to: if the distillation loss is less than the lower limit of the preset distillation loss range, update the calculation formula according to the first temperature The preset initial temperature is updated to obtain an updated temperature, where: Indicates the updated temperature, Indicates the preset temperature adjustment factor, , Indicates the preset initial temperature. Indicates the lower limit of the preset temperature range; if the distillation loss is greater than the upper limit of the preset distillation loss range, the formula is updated according to the second temperature The preset initial temperature is updated to obtain an updated temperature, where: Indicates the updated temperature, Indicates the preset temperature adjustment factor, , Indicates the preset initial temperature. Indicates the upper limit of the preset temperature range.

[0142] Optionally, the acquisition and calculation unit is further configured to: calculate the student loss formula Calculate student losses, where Indicates student loss, represents the i-th hard label, represents the probability of the i-th word in the hard prediction, and N represents the total number of output words.

[0143] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0144] like Figure 9As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the above-mentioned method based on AI large model knowledge distillation.

[0145] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memory and processor, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned method based on AI large model knowledge distillation.

[0146] The processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 601 or by instructions in the form of software. The above-mentioned processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 602, and processor 601 reads the information in memory 602 and performs the steps of the above method in conjunction with its hardware.

[0147] The device based on AI large model knowledge distillation provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present application are the same as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0148] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] For another example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0151] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method based on AI large model knowledge distillation described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0153] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0154] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method based on AI large model knowledge distillation, characterized in that: The method comprises: Input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, and obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; Determine a preset initial temperature corresponding to the current training cycle, and scale the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature to respectively calculate soft labels of the teacher model and soft predictions of the student model, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft labels and soft predictions are probability distributions of each predicted word; Calculating the distillation loss based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model; Fine-tuning the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; After scaling the original output logits of the teacher model and the original output logits of the student model by the target temperature, the target soft labels of the teacher model and the target soft predictions of the student model are recalculated respectively; Recalculate the target distillation loss based on the KL divergence between the target soft labels of the teacher model and the target soft predictions of the student model; Obtaining a hard prediction of the current training sample output by the student model, and calculating a student loss based on the hard prediction and a hard label corresponding to the current training sample; The target distillation loss and the student loss are weightedly calculated to obtain the total loss of the current training sample, and the weight parameters of the student model are updated based on the total loss of all training samples in the current batch of training samples.

2. The method according to claim 1, characterized in that Before inputting the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, the method further includes: Obtaining a current batch of training samples from the training sample set; Each word in each training sample in the current batch of training samples is converted into an embedding vector, and position information of each word in the corresponding training sample is determined.

3. The method according to claim 1, characterized in that After scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, the soft labels of the teacher model and the soft predictions of the student model are calculated respectively, including: Calculate the formula based on the soft label of the teacher model Compute the soft labels of the teacher model, where represents the probability of the i-th word output by the teacher model, Represents the original output logits of the i-th word output by the teacher model, represents the preset initial temperature, Represents the original output logits of the jth word output by the teacher model, Represents the total number of words output by the teacher model for the training samples; Soft prediction calculation formula based on the student model Compute the soft predictions of the student model, where represents the probability of the i-th word output by the student model, Represents the original output logits of the i-th word output by the student model, represents the preset initial temperature, Represents the original output logits of the jth word output by the student model, Represents the total number of words output by the student model for the training samples.

4. The method according to claim 1, wherein The distillation loss is calculated based on the KL divergence between the soft labels of the teacher model and the soft predictions of the student model, including: Calculation formula based on distillation loss The distillation loss is calculated, wherein, represents the distillation loss, represents the preset initial temperature, represents the soft label of the i-th word output by the teacher model, represents the soft prediction of the i-th word output by the student model, and N represents the total number of output words.

5. The method according to claim 1, characterized in that Fine-tuning the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample includes: If the distillation loss is not within the preset distillation loss range and the number of updates of the temperature corresponding to the current training sample has not reached the preset maximum number of updates, the preset initial temperature is updated based on the preset temperature adjustment factor to obtain the updated temperature, and the updated temperature is used as the preset initial temperature. Return to the step of scaling the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature, and respectively calculating the soft label of the teacher model and the soft prediction of the student model, until the distillation loss is within the preset distillation loss range, or the number of updates of the temperature corresponding to the current training sample reaches the preset maximum number of updates, and the finally obtained temperature is used as the target temperature of the current training sample.

6. The method according to claim 5, characterized in that Updating the preset initial temperature based on a preset temperature adjustment factor includes: If the distillation loss is less than the lower limit of the preset distillation loss range, the calculation formula is updated according to the first temperature The preset initial temperature is updated to obtain the updated temperature, wherein: represents the updated temperature, represents the preset temperature adjustment factor, , represents the preset initial temperature, Indicates the lower limit of the preset temperature range; If the distillation loss is greater than the upper limit of the preset distillation loss range, the second temperature update formula is used. The preset initial temperature is updated to obtain the updated temperature, wherein: represents the updated temperature, represents the preset temperature adjustment factor, , represents the preset initial temperature, Indicates the upper limit of the preset temperature range.

7. The method according to claim 1, characterized in that Calculating a student loss based on the hard prediction and the hard label corresponding to the current training sample, including: Calculation formula based on student losses Calculate the student loss, where represents the student loss, represents the i-th hard tag, represents the probability of the i-th word in the hard prediction, and N represents the total number of output words.

8. A device based on AI large model knowledge distillation, characterized in that: The device comprises: An input unit is used to input the embedding vector of each word in the current training sample and the position information of each word in the current training sample into the teacher model and the student model respectively, to obtain the original output logits of the teacher model and the original output logits of the student model, wherein the current training sample is a training sample in the current batch of training samples, and the current batch of training samples is obtained by dividing the training samples in the training sample set into batches; a determination and calculation unit, configured to determine a preset initial temperature corresponding to a current training cycle, and scale the original output logits of the teacher model and the original output logits of the student model by the preset initial temperature to calculate soft labels of the teacher model and soft predictions of the student model, respectively, wherein as the training cycle increases, the preset initial temperature corresponding to the training cycle decreases, and the soft labels and soft predictions are probability distributions of the predicted words; a computing unit for computing a distillation loss based on a KL divergence between the soft labels of the teacher model and the soft predictions of the student model; a fine-tuning unit, configured to fine-tune the preset initial temperature based on the distillation loss to obtain a target temperature of the current training sample, wherein the target temperature is an ideal temperature of the current training sample; A first recalculation unit is configured to recalculate the original output logits of the teacher model and the original output logits of the student model after scaling them by the target temperature to obtain a target soft label of the teacher model and a target soft prediction of the student model; a second recalculation unit, for recalculating the target distillation loss based on the KL divergence between the target soft label of the teacher model and the target soft prediction of the student model; an acquisition and calculation unit, configured to acquire a hard prediction of the current training sample output by the student model, and calculate a student loss based on the hard prediction and a hard label corresponding to the current training sample; A calculation and update unit is used to perform weighted calculation on the target distillation loss and the student loss to obtain the total loss of the current training sample, and update the weight parameters of the student model based on the total loss of all training samples in the current batch of training samples.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images