Large language model distillation method, device and equipment and storage medium

Abstract

The invention discloses a large language model distillation method, device and equipment and a storage medium, and relates to the technical field of computers, and the large language model distillation method comprises the steps: determining the teacher occupancy measure of a teacher model and the student occupancy measure of a student model based on a Markov decision process and a teacher model generation process; performing loss function definition through the teacher occupation measure, the student occupation measure and the JS divergence, and determining a training loss function of the student model; and training the student model according to an expert data set, a target discriminator and the training loss function to obtain a target distillation model, and performing a language processing task through the target distillation model. Through the mode, the distillation process combines the advantages of black box knowledge distillation and white box knowledge distillation, the time cost and hardware cost of model distillation are reduced, and the model performance and universality of the small distillation model are ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a large language model distillation method, device, equipment and storage medium. Background Art

[0002] Large Language Models (LLMs) is a technology that has made significant breakthroughs in the field of artificial intelligence in recent years. Since the launch of OpenAI's ChatGPT, a landmark achievement, the research on large language models has been attracting much attention. Among the many methods for miniaturizing large language models, knowledge distillation refers to transferring the knowledge in a large teacher model to a relatively small student model and meeting the constraints of the real world. Recently, knowledge distillation has been divided into two types: black-box knowledge distillation and white-box knowledge distillation. However, black-box knowledge distillation can only be a distillation of a specific task, and when faced with different tasks, a new model must be distilled again, and a general small model cannot be obtained; white-box knowledge distillation requires a large number of pre-training steps to achieve the distillation of a general small model, and the hardware cost and time cost are very high. Summary of the invention

[0003] The main purpose of this application is to provide a large language model distillation method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that when a large language model is miniaturized, it is impossible to take into account the distillation cost while ensuring the universality of the model.

[0004] To achieve the above objectives, the present application proposes a large language model distillation method, which includes: Determining a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; Defining a loss function by using the teacher occupancy measure, the student occupancy measure and the JS divergence, and determining a training loss function of the student model; The student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform a language processing task through the target distillation model.

[0005] In one embodiment, the determining of the teacher occupancy measure of the teacher model and the student occupancy measure of the student model based on the Markov decision process and the generation process of the teacher model includes: The generation process of the teacher model is formalized based on the Markov decision process to obtain the process mapping result of the teacher model; A measurement calculation is performed based on the process mapping result of the teacher model to determine the teacher occupancy measurement of the teacher model and the student occupancy measurement of the student model.

[0006] In one embodiment, the loss function is defined by the teacher occupancy measure, the student occupancy measure and the JS divergence to determine the training loss function of the student model, including: Calculate the gap according to the teacher occupation measure, the student occupation measure and the JS divergence to obtain a JS divergence measurement indicator; The loss function is defined according to the JS divergence metric, weight parameters and initial definition parameters of the target discriminator to determine the training loss function of the student model.

[0007] In one embodiment, the student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, and before the target distillation model is used to perform a language processing task, the method further includes: Inputting the sample training set into the teacher model to obtain sample output data corresponding to each sample training data; The sample output data corresponding to each sample training data are aggregated to obtain the expert data set.

[0008] In one embodiment, the student model is trained according to the expert data set, the target discriminator and the training loss function to obtain the target distillation model, including: Inputting the sample training set into the student model to obtain training output data corresponding to each sample training data; Determine the sample output data corresponding to each sample training data according to the expert data set; The student model is trained according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain a target distillation model.

[0009] In one embodiment, the student model is trained according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain the target distillation model, including: Input the sample output data corresponding to each sample training data and the training output data corresponding to each sample training data into the target discriminator to obtain a decision reward signal; Update the parameters of the student model according to the decision reward signal and the training loss function to obtain an initial updated model; The initial updated model is iteratively optimized based on the target discriminator and the expert dataset to obtain a target distillation model.

[0010] In one embodiment, the student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, and after the target distillation model is used to perform a language processing task, the method further includes: When receiving a language processing task, extracting text from the language processing task using the target distillation model to determine a text to be processed; The text to be processed is subjected to content analysis according to the target distillation model, and a task response text is generated based on the content analysis result to respond to the language processing task.

[0011] In addition, to achieve the above purpose, the present application also proposes a large language model distillation device, the large language model distillation device comprising: A processing module, configured to determine a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; A definition module, used to define a loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determine the training loss function of the student model; A training module is used to train the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform a language processing task through the target distillation model.

[0012] In addition, to achieve the above-mentioned objectives, the present application also proposes a large language model distillation device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the large language model distillation method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the large language model distillation method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the large language model distillation method described above.

[0015] This application determines the teacher occupancy measure of the teacher model and the student occupancy measure of the student model based on the Markov decision process and the generation process of the teacher model; defines the loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determines the training loss function of the student model; trains the student model according to the expert data set, the target discriminator and the training loss function to obtain the target distillation model, so as to perform language processing tasks through the target distillation model. Through the above method, the occupancy measure of the model is determined based on the Markov decision process and the generation process of the teacher model, and the training loss function of the student model is determined based on the occupancy measure of the model and the JS divergence. The student model is trained in combination with the expert data set and the target discriminator to obtain the target distillation model. The distillation process combines the advantages of black box knowledge distillation and white box knowledge distillation, reduces the time cost and hardware cost of model distillation, and ensures the model performance and versatility of the small distillation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a process flow provided for Example 1 of the large language model distillation method of the present application; Figure 2 A schematic diagram of a flow chart provided for Example 2 of the large language model distillation method of the present application; Figure 3 A schematic diagram of a simplified process of a large language model distillation method provided in Example 2 of the present application; Figure 4 This is a schematic diagram of the module structure of a large language model distillation device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the large language model distillation method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: based on the Markov decision process and the generation process of the teacher model, determine the teacher occupancy measure of the teacher model and the student occupancy measure of the student model; define the loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determine the training loss function of the student model; train the student model according to the expert data set, the target discriminator and the training loss function to obtain the target distillation model, so as to perform language processing tasks through the target distillation model.

[0023] Among the many methods for miniaturizing large language models, knowledge distillation refers to transferring the knowledge in a large teacher model to a relatively small student model and meeting the constraints of the real world. Recently, knowledge distillation has been divided into two types: black-box knowledge distillation and white-box knowledge distillation. However, black-box knowledge distillation can only be a distillation of a specific task, and when faced with different tasks, a new model must be distilled again, and a general small model cannot be obtained; white-box knowledge distillation requires a large number of pre-training steps to achieve distillation of a general small model, and the hardware cost and time cost are very high.

[0024] The present application provides a solution, which determines the occupancy measure of the model based on the Markov decision process and the generation process of the teacher model, and determines the training loss function of the student model based on the occupancy measure of the model and the JS divergence, and trains the student model in combination with the expert data set and the target discriminator to obtain the target distillation model. The distillation process combines the advantages of black-box knowledge distillation and white-box knowledge distillation, reduces the time cost and hardware cost of model distillation, and ensures the model performance and versatility of the distilled small model.

[0025] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a large language model distillation device that can realize the above functions. The following takes the large language model distillation device as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0026] Based on this, the present application embodiment provides a large language model distillation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the large language model distillation method of the present application.

[0027] In this embodiment, the large language model distillation method includes steps S10 to S30: Step S10, determining a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; It should be noted that the currently popular large language models include OpenAI's GPT series, Gemini, Claude, LLaMA3, etc. The parameters of these large models are very large. For example, LLaMA 3 has 70B parameters, and others such as PaLM have up to 500B parameters. Such a large number of parameters makes the training of large models consume huge computing resources and storage resources; even the fine-tuning, deployment and reasoning process of large models require sufficient resource support, which obviously severely limits the practical application of large models. Because in many scenarios, especially in applications with private data or security requirements, it is necessary to fine-tune and deploy large models locally, rather than simply using the API provided by commercial large models. Therefore, the miniaturization of large models is imperative.

[0028] It is understandable that among the many methods of miniaturizing large models, knowledge distillation refers to transferring the knowledge in a large teacher model to a relatively small student model and meeting the constraints in the real world. Recently, knowledge distillation has been divided into two types: black-box knowledge distillation and white-box knowledge distillation. Among them, white-box knowledge distillation is carried out in accordance with the traditional distillation method, and the student model can access the parameters of the teacher model; while for black-box knowledge distillation, the parameters of the teacher model are not open to the students, and the students cannot access the inside of the teacher model, but can use the data output by the teacher model. It can be seen that black-box and white-box knowledge distillation each have their own advantages and disadvantages. Therefore, this embodiment provides a solution that combines the advantages of black-box and white-box distillation to specifically perform knowledge distillation on large models, that is, only use the data generated by the large model for distillation training without other pre-training, and finally obtain a general distillation small model. In this embodiment, a large model knowledge distillation framework based on generative adversarial imitation learning is proposed, which can distill the large model using only the output data of the large model.

[0029] In the specific implementation, the Markov decision process (MDP) is a four-tuple ,in is the set of state spaces; A is the set of action spaces; is the action in state s at time t Leading to the state at time t+1 The probability of is the state s after the action Transition to state In addition, the strategy From the state space Mapping to the action space A.

[0030] It should be noted that the classic knowledge distillation architecture consists of a teacher model T and a student model S, whose output is converted into a probability distribution ,in is a temperature value that is 1 after regularization. The distillation loss is a measure of the KL distance between two probability distributions, as shown below: ,in, The real distribution provided to teachers, is the distribution of the student model.

[0031] It is understandable that the goal of imitation learning is to learn a strategy by imitating the expert behaviors from the expert data. In the setting of imitation learning, is a strategy that needs to be learned, yes The expert strategies that need to be imitated, is the dynamic transfer function of the environment, is the cost function for the expert example. As a discount factor, the expectation of the cost function can be defined as . There are two ways to implement imitation learning: one is behavior cloning and the other is inverse reinforcement learning. Behavior cloning is a supervised learning method that directly maps the state in the expert data to the action through the model. Inverse reinforcement learning is different from this method of directly obtaining the expert strategy. Its goal is to first restore the expert's loss function and then use reinforcement learning to extract a strategy from this loss function. This embodiment focuses on the imitation learning method based on inverse reinforcement learning. Among them, the Maximum Causal Entropy-IRL (MCE-IRL) inverse reinforcement learning method provides an infrastructure that stitches a loss function into an optimization problem, namely: ,in, It’s a strategy of - then deduct causal entropy. This method can set a low loss for the expert strategy and a higher loss for other strategies. Then, running a special reinforcement learning algorithm can extract the expert strategy. The process is as follows: Compared with the behavior cloning method, the advantage of the above method is that less expert data can be used and there is no problem of covariate shift. In this embodiment, this framework still requires an inner loop process to run the reinforcement learning algorithm.

[0032] In the specific implementation, the knowledge distillation framework is mapped into the imitation learning framework, that is, a student model S imitates the expert behavior from the teacher model T. In this embodiment, the teacher model is used as the large language model that needs to be distilled, and the student model is used to imitate the expert behavior from the teacher model, and finally a small model is obtained after distillation. The text generation process of the teacher model is formalized as a T-step Markov process. After completing the above formalization, the occupancy measure of the model can be defined.

[0033] It should be noted that the occupancy measure can be understood as the distribution of state-action pairs, which is the distribution of a sentence in the current context. At this time, the occupancy measure of the student model is ,in, It can be seen as the probability distribution of a word output by the student model, playing the role of the student strategy; here is the parameter of the student model. Similarly, the occupancy measure of the expert data can be defined as In this embodiment, That is, the student occupancy measure of the student model and the occupancy measure of the expert data That is the teacher occupancy measure of the teacher model.

[0034] Step S20, defining a loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determining a training loss function of the student model; It should be noted that the knowledge distillation loss is a measure of the difference between two distributions: in this embodiment, it is hoped that the distance between the output distribution from the teacher model and the output distribution from the student is as small as possible, so that the teacher model can guide the student model training. JS divergence refers to Jensen-Shannon divergence. Since KL divergence is not symmetrical and is not a distance in a mathematical sense, in order to achieve the purpose, a new probability distribution measurement method is used as distillation loss in this embodiment. In this embodiment, Jensen-Shannon divergence can be used as a measure of distillation loss. Jensen-Shannon divergence is a symmetric distance metric.

[0035] It can be understood that the Jensen-Shannon divergence can be expressed as: , and the relationship between JS divergence and Generative Adversarial Networks (GAN) is known. The optimal discriminator calculates JS divergence, and they can be converted to each other. Therefore, based on the teacher occupancy measure, the student occupancy measure and JS divergence, the loss function of the student model training can be obtained.

[0036] In a feasible implementation, step S20 may include steps A21-A22: Step A21, performing gap calculation according to the teacher occupancy measure, the student occupancy measure and the JS divergence to obtain a JS divergence measurement index; It should be noted that the difference calculation is based on the teacher occupancy measure, the student occupancy measure and the JS divergence, and the JS divergence metric is obtained as a measure between the two distributions. .

[0037] Step A22, defining a loss function according to the JS divergence metric, weight parameters, and initial definition parameters of the target discriminator, and determining a training loss function of the student model.

[0038] It should be noted that the weight parameter Used to balance the impact of the entropy term on the total loss, target discriminator The initial definition parameters are , the task of the target discriminator is to distinguish the data generated by the student model from the expert data. Based on the JS divergence metric, weight parameters and the initial definition parameters of the target discriminator, the loss function is defined to obtain the training loss function of the student model .

[0039] Step S30, training the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform a language processing task through the target distillation model.

[0040] It should be noted that the expert data set refers to the data set composed of the data output by the model after the sample training data is input into the teacher model. In this embodiment, the PaLM model can be used as the teacher model and the OPT-125m model can be used as the student model. , and the target discriminator OPT-125m is also used. In this embodiment, the Adam algorithm is used to identify the target The parameters of the student model are updated. If the PPO algorithm is used to update the training data, the student model finally trained is the target distillation model for subsequent language processing tasks. In this embodiment, the student model is regarded as a generator.

[0041] In a feasible implementation manner, step S30 may include steps A31 to A32: Step A31, inputting the sample training set into the teacher model to obtain sample output data corresponding to each sample training data; It should be noted that the sample training set contains multiple sample training data, which contain questions and true labels. Examples of questions are as follows: Input: '"There are 10 apples on an apple tree. Three fall off. Now there are X apples." What is this an example of?\nAnswerChoices:\n(a) park\n(b) coloring book\n(c) garden center\n(d) math problem\n(e) gravity'. Answer: math problem.

[0042] It can be understood that the questions in each sample training data are spliced ​​into the prepared CoT (Chain of Thought) template and input into the teacher model to obtain the corresponding CoT and answers. The CoT and answer of each sample training data are the sample output data corresponding to each sample training data. The specific example is: CoT+answer: The answer must be something that has to do with the above problem. Of the above choices, only math problem has to do with the above problem. So the answer is (d) math problem.

[0043] Step A32, summarizing the sample output data corresponding to each sample training data to obtain an expert data set.

[0044] It should be noted that the sample output data corresponding to each sample training data set are aggregated to obtain an expert data set.

[0045] In a feasible implementation, step S30 may include steps B31 to B33: Step B31, inputting the sample training set into the student model to obtain training output data corresponding to each sample training data; It should be noted that each sample training data in the sample training data set is input into the student model so that the student model outputs the CoT and answer corresponding to each sample training data. In this embodiment, the CoT and answer corresponding to each sample training data output by the student model are the training output data corresponding to each sample training data.

[0046] Step B32, determining the sample output data corresponding to each sample training data according to the expert data set; Step B33, training the student model according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain a target distillation model.

[0047] It should be noted that the sample output data corresponding to each sample training data is extracted from the expert data set. The sample output data corresponding to each sample training data is matched with the training output data corresponding to each sample training data, and the matching result is input into the target discriminator for discrimination. The target discriminator is trained by the Adam algorithm. The parameters of the student model are updated. To update the target distillation model, the PPO algorithm is used to update the student model through the reward signal fed back by the target discriminator and the training loss function, so as to obtain the target distillation model.

[0048] In a feasible implementation, step B33 may include steps C31 to C33: Step C31, inputting the sample output data corresponding to each sample training data and the training output data corresponding to each sample training data into the target discriminator to obtain a decision reward signal; It should be noted that the sample output data of each sample training data and the training output data corresponding to each sample training data are input into the target discriminator. In the generative adversarial network, the task of the target discriminator is to distinguish between the expert data and the data generated by the student model. It completes this task by evaluating the authenticity of the input data and outputting a probability value, which reflects the possibility that the input data is considered to be expert data. When the student model generates a new sample, the target discriminator will judge its authenticity and give a corresponding score.

[0049] It can be understood that, under the above framework, the score of the target discriminator can be regarded as a kind of "reward" signal to the student model. If the probability that the sample generated by the student model is judged to be true is high, then the "reward" received by the student model is large, and the corresponding loss function value is small; conversely, if the probability that the sample generated by the student model is judged to be false is high, then the "penalty" received by the student model is large, and the loss function value increases accordingly. The "reward" or "penalty" here is actually reflected by the output of the target discriminator converted into the form of a loss function. In this embodiment, the score of the target discriminator is the decision reward signal.

[0050] Step C32, updating the parameters of the student model according to the decision reward signal and the training loss function to obtain an initial updated model; It should be noted that the parameter update of the student model depends on the decision reward signal fed back by the target discriminator. During the training process, the goal of the student model is to minimize its training loss function, that is, to increase the probability that the generated data is judged to be true. The student model calculates the gradient of the training loss function relative to its parameters through the back-propagation algorithm, and adjusts the parameters according to this gradient, in the hope of generating more realistic data in the future, thereby reducing losses. In this embodiment, the student model after the parameter update is the initial update model.

[0051] Step C33, iteratively optimizing the initial update model based on the target discriminator and the expert dataset to obtain a target distillation model.

[0052] It should be noted that as training progresses, both the generator and the target discriminator continue to improve. The generator tries to generate samples that are more difficult to distinguish, while the target discriminator strives to improve its ability to distinguish. This competitive dynamic makes the generator gradually learn to generate samples that are highly similar to the expert data until it reaches an equilibrium point, at which point the quality of the generated samples is so high that the discriminator cannot easily distinguish them from real or fake.

[0053] It can be understood that the initial update model is iteratively optimized based on the target discriminator and the expert dataset, and the target discriminator is optimized by the Adam algorithm. The parameters of the student model are updated. If the update is completed, the PPO algorithm is used to finally complete the training of the student model and obtain the target distillation model.

[0054] In a feasible implementation, step S30 may include steps A301-A302: Step A301, when receiving a language processing task, extracting text from the language processing task through the target distillation model to determine a text to be processed; Step A302, performing content analysis on the text to be processed according to the target distillation model, and generating a task response text based on the content analysis result to respond to the language processing task.

[0055] It should be noted that when the language processing task is a question-answering task, the target distillation model extracts the language processing task to obtain the text to be processed, performs content analysis on the text to be processed, and generates a task response text based on the content analysis results, thereby responding to the language processing task.

[0056] It is understandable that the target distillation model can also be applied to other language processing tasks, including but not limited to text creation and generation, speech recognition and synthesis, etc., which is not limited in this embodiment.

[0057] This embodiment determines the teacher occupancy measure of the teacher model and the student occupancy measure of the student model based on the Markov decision process and the generation process of the teacher model; defines the loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determines the training loss function of the student model; trains the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform language processing tasks through the target distillation model. In the above manner, the occupancy measure of the model is determined based on the Markov decision process and the generation process of the teacher model, and the training loss function of the student model is determined based on the occupancy measure of the model and the JS divergence. The student model is trained in combination with the expert data set and the target discriminator to obtain the target distillation model. The distillation process combines the advantages of black box knowledge distillation and white box knowledge distillation, reduces the time cost and hardware cost of model distillation, and ensures the model performance and versatility of the distillation small model.

[0058] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 In the large language model distillation method, step S10 further includes steps S11-S12: Step S11, formalizing the generation process of the teacher model based on the Markov decision process to obtain a process mapping result of the teacher model; Step S12, performing measurement calculation based on the process mapping result of the teacher model to determine the teacher occupancy measurement of the teacher model and the student occupancy measurement of the student model.

[0059] It should be noted that the generation process of the teacher model is formalized based on the Markov decision process, as follows: 1. The teacher model can be regarded as a policy that maps the state to the probability distribution in the action space; a state is a partial sequence ,in ; Initial state is the input sequence of the teacher model ,in is the initial state distribution; an action is defined as the next character generated by the teacher model ; The action space is the entire dictionary; given a state (i.e., a partial sequence) and a corresponding action selected from the policy (next character), then the state transition equation is determined: the next state is the concatenation of this state and the action, that is, In this embodiment, after the formalization of the text generation process of the teacher model is completed, the process mapping result of the teacher model can be obtained.

[0060] It can be understood that the occupancy measure of the model can be calculated based on the process mapping result of the teacher model. The occupancy measure can be understood as the distribution of state-action pairs, which is the distribution of a sentence in the current context. At this time, the occupancy measure of the student model is ,in, It can be seen as the probability distribution of a word output by the student model, playing the role of the student strategy; here is the parameter of the student model. Similarly, the occupancy measure of the expert data can be defined as In this embodiment, That is, the student occupancy measure of the student model and the occupancy measure of the expert data That is the teacher occupancy measure of the teacher model.

[0061] This embodiment formalizes the generation process of the teacher model based on the Markov decision process to obtain the process mapping result of the teacher model; based on the process mapping result of the teacher model, the measurement calculation is performed to determine the teacher occupancy measure of the teacher model and the student occupancy measure of the student model. In this way, the occupancy measure of the model can be accurately defined, laying the foundation for subsequent model distillation.

[0062] For example, to help understand the implementation process of the large language model distillation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flow chart of a large language model distillation method is provided, specifically: The large language model distillation method of this embodiment specifically includes the following steps: 1) Formalizing the Markov decision process of the problem to be solved: Formalizing the text generation process of the large model into a T-step Markov process (Markov Decision Process, MDP), as shown below: The large model can be regarded as a policy that maps the state to the probability distribution in the action space; a state is a partial sequence ,in ; Initial state The input sequence of the large model ,in is the initial state distribution; an action is defined as the next character generated by the large model ; The action space is the entire dictionary; given a state (i.e., a partial sequence) and a corresponding action selected from the policy (next character), then the state transition equation is determined: the next state is the concatenation of this state and the action, that is, .

[0063] 2) Knowledge distillation based on generative adversarial imitation learning: a) Occupancy measure of large models: Occupancy measure can be understood as the distribution of state-action pairs, which is the distribution of a sentence in the current context. ,in, It can be seen as the probability distribution of a word output by the student model, playing the role of the student strategy; here is the parameter of the student model. Similarly, the occupancy measure of the expert data can be defined as b) Loss function: From the above, we can see that the knowledge distillation loss is a measure between two distributions: In this embodiment, we hope that the distance between the output distribution from the teacher model and the output distribution from the student is as small as possible, so that the teacher model can guide the student model training. In this embodiment, Jensen-Shannon divergence is used as a measure between two distributions. Jensen-Shannon divergence can be expressed as: , since the relationship between JS divergence and Generative Adversarial Networks (GAN) is known, the optimal discriminator calculates JS divergence, and they can be converted to each other, then the loss function of the student model can be expressed as: ,in For is the discriminator of parameters.

[0064] 3) Training method: In this embodiment, the PaLM model is used as the teacher, and its data output is used as the expert data; the OPT-125m model is used as the student model , and the discriminator model OPT-125m is also used. The Adam algorithm is used for the discriminator The parameters of the generator are updated, and The update uses the PPO algorithm.

[0065] This embodiment proposes a GAIL-KD-based generative adversarial imitation learning framework to solve the distillation problem of large models; the framework of this embodiment combines the advantages of black-box knowledge distillation and white-box knowledge distillation, that is, only the data generated by the large model is used for distillation training, and finally a general distillation small model effect is obtained.

[0066] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the large language model distillation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0067] This application also provides a large language model distillation device, please refer to Figure 4 , the large language model distillation device comprises: A processing module 10, for determining a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; A definition module 20, used to define a loss function by using the teacher occupancy measure, the student occupancy measure and the JS divergence, and determine a training loss function of the student model; The training module 30 is used to train the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model so as to perform a language processing task through the target distillation model.

[0068] Optionally, the processing module 10 is further used for: The generation process of the teacher model is formalized based on the Markov decision process to obtain the process mapping result of the teacher model; A measurement calculation is performed based on the process mapping result of the teacher model to determine the teacher occupancy measurement of the teacher model and the student occupancy measurement of the student model.

[0069] Optionally, the definition module 20 is further used to: Calculate the gap according to the teacher occupation measure, the student occupation measure and the JS divergence to obtain a JS divergence measurement indicator; The loss function is defined according to the JS divergence metric, weight parameters and initial definition parameters of the target discriminator to determine the training loss function of the student model.

[0070] Optionally, the training module 30 is further used to: Inputting the sample training set into the teacher model to obtain sample output data corresponding to each sample training data; The sample output data corresponding to each sample training data are aggregated to obtain the expert data set.

[0071] Optionally, the training module 30 is further used to: Inputting the sample training set into the student model to obtain training output data corresponding to each sample training data; Determine the sample output data corresponding to each sample training data according to the expert data set; The student model is trained according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain a target distillation model.

[0072] Optionally, the training module 30 is further used to: Input the sample output data corresponding to each sample training data and the training output data corresponding to each sample training data into the target discriminator to obtain a decision reward signal; Update the parameters of the student model according to the decision reward signal and the training loss function to obtain an initial updated model; The initial updated model is iteratively optimized based on the target discriminator and the expert dataset to obtain a target distillation model.

[0073] Optionally, the training module 30 is further used to: When receiving a language processing task, extracting text from the language processing task using the target distillation model to determine a text to be processed; The text to be processed is subjected to content analysis according to the target distillation model, and a task response text is generated based on the content analysis result to respond to the language processing task.

[0074] The large language model distillation device provided by the present application adopts the large language model distillation method in the above embodiment, which can solve the technical problem in the prior art that when the large language model is miniaturized, it is impossible to take into account the distillation cost while ensuring the versatility of the model. Compared with the prior art, the beneficial effects of the large language model distillation device provided by the present application are the same as the beneficial effects of the large language model distillation method provided by the above embodiment, and the other technical features of the large language model distillation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0075] The present application provides a large language model distillation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model distillation method in the above-mentioned embodiment 1.

[0076] Reference below Figure 5 , which shows a schematic diagram of the structure of a large language model distillation device suitable for implementing the embodiment of the present application. The large language model distillation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The large language model distillation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0077] like Figure 5As shown, the large language model distillation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the large language model distillation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the large language model distillation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a large language model distillation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0078] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0079] The large language model distillation device provided by the present application adopts the large language model distillation method in the above embodiment, which can solve the technical problem in the prior art that when the large language model is miniaturized, it is impossible to take into account the distillation cost while ensuring the versatility of the model. Compared with the prior art, the beneficial effects of the large language model distillation device provided by the present application are the same as the beneficial effects of the large language model distillation method provided by the above embodiment, and the other technical features of the large language model distillation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0080] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0081] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0082] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the large language model distillation method in the above-mentioned embodiment.

[0083] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0084] The computer-readable storage medium may be included in the large language model distillation device; or may exist independently without being assembled into the large language model distillation device.

[0085] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the large language model distillation device, the large language model distillation device: determines the teacher occupancy measure of the teacher model and the student occupancy measure of the student model based on the Markov decision process and the generation process of the teacher model; defines the loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determines the training loss function of the student model; trains the student model according to the expert data set, the target discriminator and the training loss function to obtain the target distillation model, so as to perform language processing tasks through the target distillation model.

[0086] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0089] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large language model distillation method, and can solve the technical problem in the prior art that when the large language model is miniaturized, it is impossible to take into account the distillation cost while ensuring the universality of the model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the large language model distillation method provided in the above-mentioned embodiment, and will not be repeated here.

[0090] The present application also provides a computer program product, including a computer program, which implements the steps of the large language model distillation method as described above when executed by a processor.

[0091] The computer program product provided in this application can solve the technical problem in the prior art that when a large language model is miniaturized, it is impossible to take into account the distillation cost while ensuring the universality of the model. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the large language model distillation method provided in the above embodiment, and will not be repeated here.

[0092] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A large language model distillation method, characterized in that: The large language model distillation method comprises: Determining a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; Defining a loss function by using the teacher occupancy measure, the student occupancy measure and the JS divergence, and determining a training loss function of the student model; The student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform a language processing task through the target distillation model.

2. The method according to claim 1, characterized in that The generation process based on the Markov decision process and the teacher model determines the teacher occupancy measure of the teacher model and the student occupancy measure of the student model, including: The generation process of the teacher model is formalized based on the Markov decision process to obtain the process mapping result of the teacher model; A measurement calculation is performed based on the process mapping result of the teacher model to determine the teacher occupancy measurement of the teacher model and the student occupancy measurement of the student model.

3. The method according to claim 1, characterized in that The loss function is defined by the teacher occupancy measure, the student occupancy measure and the JS divergence to determine the training loss function of the student model, including: Calculate the gap according to the teacher occupation measure, the student occupation measure and the JS divergence to obtain a JS divergence measurement indicator; The loss function is defined according to the JS divergence metric, weight parameters and initial definition parameters of the target discriminator to determine the training loss function of the student model.

4. The method according to claim 1, characterized in that The method further includes: training the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, and performing a language processing task through the target distillation model; Inputting the sample training set into the teacher model to obtain sample output data corresponding to each sample training data; The sample output data corresponding to each sample training data are aggregated to obtain the expert data set.

5. The method according to claim 1, characterized in that The student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, including: Inputting the sample training set into the student model to obtain training output data corresponding to each sample training data; Determine the sample output data corresponding to each sample training data according to the expert data set; The student model is trained according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain a target distillation model.

6. The method according to claim 5, characterized in that The student model is trained according to the sample output data corresponding to each sample training data, the training output data corresponding to each sample training data, the target discriminator and the training loss function to obtain the target distillation model, including: Input the sample output data corresponding to each sample training data and the training output data corresponding to each sample training data into the target discriminator to obtain a decision reward signal; Update the parameters of the student model according to the decision reward signal and the training loss function to obtain an initial updated model; The initial updated model is iteratively optimized based on the target discriminator and the expert dataset to obtain a target distillation model.

7. The method according to any one of claims 1 to 6, characterized in that After the student model is trained according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, and the language processing task is performed by the target distillation model, the method further includes: When receiving a language processing task, extracting text from the language processing task using the target distillation model to determine a text to be processed; The text to be processed is subjected to content analysis according to the target distillation model, and a task response text is generated based on the content analysis result to respond to the language processing task.

8. A large language model distillation device, characterized in that: The large language model distillation device comprises: A processing module, configured to determine a teacher occupancy measure of the teacher model and a student occupancy measure of the student model based on a Markov decision process and a generation process of the teacher model; A definition module, used to define a loss function through the teacher occupancy measure, the student occupancy measure and the JS divergence, and determine the training loss function of the student model; A training module is used to train the student model according to the expert data set, the target discriminator and the training loss function to obtain a target distillation model, so as to perform a language processing task through the target distillation model.

9. A large language model distillation device, characterized in that: The large language model distillation device includes: a memory, a processor, and a large language model distillation program stored in the memory and executable on the processor, wherein the large language model distillation program is configured to implement the large language model distillation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a large language model distillation program, which, when executed by the processor, implements the large language model distillation method according to any one of claims 1 to 7.

Images