Method, related apparatus and computer program product for distillation model knowledge

Abstract

The present disclosure provides a method for distilling model knowledge, a related device and a computer program product, relating to the technical field of artificial intelligence such as deep learning, distillation training, training sample optimization, etc. A specific embodiment of the method includes: generating a quality evaluation parameter for a student model output based on a current state parameter of the student model and a student model output generated by the student model processing a sample input; in response to the quality evaluation parameter being greater than a first parameter threshold, generating an output score of the student model output by a teacher model; training the student model based on the output score; and in response to a loss function for supervising the training process of the student model reaching a training target, ending the training to obtain a target student model trained by the student model. Thus, not only can the requirements for teacher model resource usage and model capability be reduced, enabling more models to be compressed and lightened through distillation, but also the efficiency and quality of the model knowledge distillation process can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to artificial intelligence technologies such as deep learning, distillation training, and training sample optimization, and particularly to methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for distillation model knowledge. Background Technology

[0002] As the scale of large models continues to grow, the number of model parameters, computational complexity, and inference resource consumption are constantly increasing, leading to higher requirements for computing power, storage, and latency in the actual deployment and application of models.

[0003] Against this backdrop, to reduce the difficulty of model deployment and make models applicable to more application scenarios, model compression techniques such as model knowledge distillation and model distillation have gradually been widely used. Model knowledge distillation, as a typical model compression technique, can transfer the model knowledge possessed by a large model (the teacher model) to a lightweight student model. This allows the student model to maintain the capabilities of the large model as much as possible while reducing the resource requirements and implementation costs for model deployment.

[0004] Therefore, how to complete the distillation process with lower resource consumption while ensuring the effectiveness of knowledge distillation and knowledge transfer, and how to reduce the performance requirements of the teacher model during the distillation process, are issues that deserve attention and are urgently needed. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for distillation model knowledge.

[0006] In a first aspect, embodiments of this disclosure propose a method for distilling model knowledge, comprising: generating quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated by the student model processing sample input; generating an output score for the student model output through a teacher model in response to the quality evaluation parameters being greater than a first parameter threshold; training the student model based on the output score; and ending the training in response to the loss function used to supervise the training process of the student model reaching the training objective, thereby obtaining a target student model trained by the student model.

[0007] Secondly, embodiments of this disclosure propose an apparatus for distilling model knowledge, comprising: an evaluation parameter generation unit configured to generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated from the sample input processed by the student model; an output score output unit configured to generate an output score for the student model output through a teacher model in response to the quality evaluation parameters being greater than a first parameter threshold; a first student training unit configured to train the student model based on the output score; and a training termination unit configured to terminate training in response to the loss function used to supervise the training process of the student model reaching the training objective, thereby obtaining a target student model trained by the student model.

[0008] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement a method for distillation model knowledge as described in any implementation of the first aspect.

[0009] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for enabling a computer to implement a method for distillation model knowledge as described in any implementation of the first aspect.

[0010] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, enables a method for implementing distillation model knowledge as described in any implementation of the first aspect.

[0011] The distillation model knowledge method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure firstly generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated from the processed sample input; then, in response to the quality evaluation parameters being greater than a first parameter threshold, an output score for the student model output is generated through a teacher model; next, the student model is trained based on the output score; finally, in response to the loss function used to supervise the training process of the student model reaching the training objective, the training ends, and a target student model trained by the student model is obtained.

[0012] This disclosure not only reduces the requirements for teachers' use of model resources and model capabilities, enabling more models to be compressed and lightweighted through distillation, but also improves the efficiency and quality of the model knowledge distillation process.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart illustrating a process for obtaining distillation model knowledge as provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating a process for generating quality evaluation parameters is provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating the process of implementing distillation model knowledge in a specific application scenario as provided in this embodiment of the disclosure; Figure 5 A structural block diagram of a distillation model knowledge apparatus provided in this disclosure embodiment; Figure 6 This is a schematic diagram of an electronic device suitable for performing a method of distillation model knowledge, provided as an embodiment of the present disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] Furthermore, if the technical solutions disclosed herein involve information that is of the type of user personal information, such as student models and teacher models in some embodiments of this disclosure that can be processed using user personal information as input and material, then in such cases, the acquisition, storage, use, processing, transportation, provision and disclosure of such information comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0017] Figure 1 An exemplary system architecture 100 is shown, illustrating embodiments of methods, apparatuses, electronic devices, and computer-readable storage media to which the distillation model knowledge of this disclosure can be applied.

[0018] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0019] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include model distillation applications, model deployment applications, and instant messaging applications.

[0020] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0021] Server 105 can provide various services through its built-in applications. For example, it can provide a model distillation application that distills the model knowledge from the teacher model into the student model to form a target student model. When running this model distillation application, server 105 can achieve the following effects: First, it obtains sample input from terminal devices 101, 102, and 103 via network 104. Then, server 105 calls the student model to process the sample input and generate the corresponding student model output. Next, based on the current state parameters of the student model and the student model output generated by processing the sample input, server 105 generates quality evaluation parameters for the student model output. Then, in response to the quality evaluation parameters exceeding a first parameter threshold, server 105 generates an output score for the student model output through the teacher model. Then, server 105 trains the student model based on the output score. Finally, in response to the loss function used to supervise the training process of the student model reaching the training objective, server 105 ends the training, obtaining the target student model trained by the student model.

[0022] It should be noted that, in addition to being obtained from terminal devices 101, 102, and 103 via network 104, sample inputs can also be pre-stored locally on server 105 through various means. Therefore, when server 105 detects that this data is already stored locally (e.g., when starting to process previously retained model distillation tasks or student model training tasks), it can choose to directly obtain this data from locally. In this case, the exemplary system architecture 100 may also exclude terminal devices 101, 102, and 103 and network 104.

[0023] Since training and distilling model knowledge often require significant computational resources and power, the methods for distilling model knowledge provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computational power and abundant resources. Correspondingly, the device for distilling model knowledge is also generally located in the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computational power and resources, they can also perform the aforementioned calculations performed by the server 105 through their installed model distillation applications, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computational capabilities exist simultaneously, but the model distillation application determines that its terminal device has strong computational power and abundant remaining resources, the terminal device can perform the aforementioned calculations, thereby appropriately reducing the computational burden on the server 105. Accordingly, the device for distilling model knowledge can also be located in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0024] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0025] Please refer to Figure 2 , Figure 2 A flowchart of a process for providing distillation model knowledge according to an embodiment of this disclosure, including process 200.

[0026] Process 200 specifically includes the following steps: Step 201: Based on the current state parameters of the student model and the student model output generated by processing sample inputs, generate quality evaluation parameters for the student model output; In embodiments of this disclosure, this step aims to have the student model process the sample input, generate the corresponding student model output, and then have the method for distilling model knowledge executed by the subject (e.g., Figure 1 The server 105 shown generates quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated from the sample input processed by the student model.

[0027] As discussed above, the student model can be a smaller model than the teacher model (e.g., a large model). This allows for the distillation and training of the teacher model's knowledge into the student model to obtain a target student model that is smaller in size but has processing logic and capabilities close to the teacher model. Subsequently, this target student model can be deployed, providing capabilities close to the teacher model while reducing deployment resource consumption and requirements. In practice, depending on different scenarios and needs, this teacher model can be used to process text, images, audio, and video.

[0028] For example, the teacher model could be a Large Language Model (LLM). An LLM is an artificial intelligence model designed to understand and generate human language, and it can perform processing operations based on its understanding to obtain corresponding results. LLMs can be trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. A key characteristic of LLMs is their large scale; they typically include a large number of parameters to help them learn complex patterns in language data. These models are often based on deep learning architectures, such as transformers, which helps them deliver better performance on various natural language processing tasks.

[0029] For example, a teacher model could be an LLM that executes corresponding generative actions based on input textual instructions in natural language. For instance, a teacher model can generate text, images, audio, and video content that meets user requirements based on its held and maintained model knowledge, provided by the user (text, speech, etc.). Correspondingly, after a distillation process, the executing entity can distill the teacher model's generative capabilities and model knowledge into a student model to obtain a target student model with capabilities approaching those of the teacher model. As another example, a teacher model could be an image recognition model used to process images and identify specific objects within them.

[0030] In embodiments of this disclosure, the sample input may correspond to the model knowledge to be distilled. For example, in the example above, the sample input may be a "text instruction in natural language form," and the corresponding student model output may be text, image, audio, or video content actually output by the student model based on the sample input. As another example, the sample input may be a sample image that includes positive sample properties of the aforementioned specific object, and a sample image that does not include negative sample properties of the aforementioned specific object.

[0031] It should be understood that the process of the student model processing sample inputs and generating student model outputs can be completed by the executing entity actively calling the student model, or by the student model based on the calling action of other entities. This disclosure is not intended to limit this.

[0032] Furthermore, in different implementations and scenarios, for a single sample input, the student model can choose to process it into a unique output for one student model, achieving one-to-one training. Alternatively, the student model can process it into at least two or more student model outputs, enabling the student model to fully learn the teacher model's model knowledge from that sample input. For example, for a single sample input... x Student Model s This sample input can be processed. x ,get K The student model outputs, for example, to obtain the sample model output sequence. ,in, K It is a positive integer.

[0033] In this step, after obtaining the student model output, the executing entity can generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated by processing sample inputs.

[0034] The current state parameter can be a feature vector of the student model's current training state. This feature vector can be determined based on metrics such as the moving average of the loss function used for supervised learning models (e.g., the loss function used to supervise the training process of the student model, which will be discussed below), the current training epoch, and the parameter values ​​of specific parameters in the student model.

[0035] The quality evaluation parameters for student model output can be used to assess the quality of student model output in terms of usage and generation. This allows the implementing entity to determine the difficulty of teacher model correction and guidance based on the "quality" of the student model output in subsequent steps, and whether it is appropriate to use the teacher model to guide the student model and distill model knowledge by correcting the student model output. This avoids the teacher model correcting all student model outputs in one go, thus avoiding the waste of resources caused by such a method of knowledge distillation (e.g., avoiding the teacher model correcting student model outputs of very poor quality that lack correction value).

[0036] In practice, the above-mentioned "correction" method of training and distillation can correspond to the guidance method and distillation method discussed in this paper, which uses the output score output by the teacher model to train the student model.

[0037] For ease of understanding, the following discussion can be conducted using a student model. s Processing sample input x The sample model output sequence is obtained. The scenario will be discussed as an example.

[0038] Accordingly, in such a case, the sample model output The corresponding quality evaluation parameters can then be characterized as ,in, i ∈[1, K ].

[0039] And that : (1) in, For example, the definition of an activation function can be: ; For example, it can be a routing network, which can be a multilayer perceptron (MLP) with a preset number of layers. Input for sample The feature representation vector; This is a feature vector derived from the current state parameters of the student model. For example, it can be represented by the moving average of the loss function mentioned above, the current training epoch, or the parameter values ​​of specific parameters in the student model. The feature vectors obtained after vectorization.

[0040] Next, after obtaining the quality evaluation parameters The implementing entity can then compare it with a first parameter threshold. This first parameter threshold can typically be pre-set based on a standard that allows the teacher model to "guide" through correction, or in other words, it can be set based on a standard that allows the teacher model to consider it valuable to distill model knowledge through correction and guidance.

[0041] Accordingly, if the quality evaluation parameter of the specific student model output is greater than the first parameter threshold, the student model output can be considered to be worthy of being used in the "guidance and correction" manner (for example, although there are errors and flaws in the student model output, its overall quality is better, and these errors and flaws can be overcome through the student model's self-iteration and learning). The executing entity can respond to this and continue to execute step 202.

[0042] Step 202: Generate the output score from the student model using the teacher model; In the embodiments of this disclosure, if the quality evaluation parameter output by the student model is greater than the first parameter threshold based on step 201, then as discussed above, it indicates that it is "better" in quality and is worth using the teacher model to "guide" and "distill" the model knowledge lacking in the middle school student model into the student model through correction and guidance.

[0043] Accordingly, for student model outputs that are "worthy of correction and guidance," the executing entity can, for example, use a teacher model to generate an output score for the student model output. For instance, the executing entity can provide the sample input along with the student model output to the teacher model, which can then use the teacher model to generate an output score characterizing the student model output based on the degree of matching between the training samples and the student model output, the correctness, completeness, logical consistency, and / or satisfaction of task constraints.

[0044] Step 203: Train the student model based on the output score; In the embodiments of this disclosure, after obtaining the output score based on step 202 above, the executing entity can train the student model based on the output score. For example, the executing entity can use an increase in the output score as a reward signal to guide the student model to train and iterate in the direction of increasing or improving the matching degree, the correctness and completeness of the student model's output, or in other words, increasing or improving the output score.

[0045] For example, the executing entity can continuously train the student models using sample inputs and the corresponding output scores of each student model, so that the student models can be adjusted and trained in the direction of improving the output scores of the student models for a given sample input. For example, the executing entity can perform the distillation training process iteratively according to multiple training rounds and steps. For example, the executing entity can use an output score as a round or step.

[0046] Accordingly, in each training epoch, the executing agent generates a new student model output using the student model. If the quality evaluation parameter of the student model output is greater than the first parameter threshold, the executing agent continues to score the student model output using the teacher model. Then, based on the student model output and corresponding score obtained in the current training epoch, the executing agent trains the student model to guide the generation of student model output and the updating of model parameters in subsequent training epochs.

[0047] During the training and iteration of the student model, a loss function can be used to supervise the training and iteration process. For example, a loss function can be constructed using KL (Kullback-Leibler divergence) loss. L Then, it is used to supervise the training and iteration process; for example, the loss function. L It can be expressed as follows: (2) (2) in, For the teacher model in sample input Prefix generation for student models Under the condition of position The probability distribution of the next token. For student models, position under the same conditions The probability distribution of the next word term. Measurement using distribution Approximate distribution Information loss at that time , These are the conditional probability distribution functions for the teacher model and the student model, respectively.

[0048] Accordingly, if after a round of training corresponding to an output score, the loss function reaches the training objective, for example, the loss function (value) has been minimized, or converges to the target value, converges to a predetermined numerical range, etc., then the execution entity can respond to this and execute step 204.

[0049] Step 204: End training and obtain the target student model trained from the student model.

[0050] In the embodiments of this disclosure, if the loss function used to monitor the training process of the student model reaches the training objective in this step, the executing entity can respond by choosing to end or exit the training of the student model and fix the model parameters of the current student model to obtain the target student model trained by the student model.

[0051] The method for distilling model knowledge provided in this disclosure first generates quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated from the processed sample input. Then, in response to the quality evaluation parameters exceeding a first parameter threshold, an output score for the student model output is generated using a teacher model. Next, the student model is trained based on the output score. Finally, in response to the loss function used to supervise the training process of the student model reaching the training objective, training ends, resulting in the target student model trained by the student model. This not only reduces the requirements for teacher model resources and model capabilities, enabling more models to be compressed and lightweighted through distillation, but also improves the efficiency and quality of the model knowledge distillation process.

[0052] In some embodiments, if the quality evaluation parameter of the student model output obtained after step 201 is less than or equal to the first parameter threshold, it can be considered that the student model output lacks adjustment value from the teacher model (for example, the teacher model needs to use more than the average resource consumption to correct and score, or even if the teacher model scores and outputs a score, it will be difficult to effectively guide the training process of the student model due to the low output score). In such cases, in order for the student model to learn the model knowledge of the teacher model in this part and in the model's logical direction (i.e., according to the logical direction used by the student model when generating the student model output based on the sample input), the executing entity can respond by obtaining the teacher model output generated by the teacher model processing the sample input. In practice, the teacher model can pre-process the sample input in a similar manner to the student model to generate one or more corresponding teacher model outputs.

[0053] In some embodiments, if multiple "(student) model outputs" are used for the same sample input, the executing entity can also select the teacher model output with the highest similarity based on the similarity between the student model output and the teacher model output, so as to obtain the teacher model output with the same and similar logical direction to distill the student model's ability and model knowledge in the model logic and output it.

[0054] Then, the implementing entity can use the teacher model's output to train the student model. For example, using the processed sample input as input and the teacher model's output as the corresponding output, the student model can be trained. Thus, by reasonably allocating teacher model correction resources and reducing the requirements for the teacher model's correction capabilities, the teacher model's output can ensure that the student model's currently deficient capabilities and corresponding model knowledge are fully distilled into the student model.

[0055] Correspondingly, this approach also enables more effective and higher-quality distillation of model knowledge into student models in scenarios such as cold starts (e.g., when the student model has limited training and lacks basic processing capabilities for sample inputs).

[0056] Accordingly, in this situation, since the teacher model's knowledge can be distilled through output scores (i.e., corrections) and also through the teacher model's output, the aforementioned loss function can be constructed and composed of a first loss sub-function and a second loss sub-function. This allows the loss function to simultaneously supervise both directions and strategies, coordinating the training effects under both training methods and ensuring distillation quality.

[0057] The first loss function is used to supervise the first training process of training the student model using the output scores, and the second loss function is used to supervise the second training process of training the student model based on the teacher model output.

[0058] It should be understood that, in some embodiments, based on differences in training objectives, such as supervising only a first training process that performs model distillation in a "corrective" manner, or supervising only a second training process that performs model distillation based on the teacher model output, the executing agent may also construct the loss function separately using the first loss subfunction and the second loss subfunction.

[0059] For example, in an embodiment of process 200 above, the loss function shown in formula (2) is used. L It can then be understood that it is constructed solely based on the first loss function.

[0060] As for the loss function constructed solely using the second loss function... L It can be expressed by the following formula (3): (3) The same parameters can be referred to in the parameter description of formula (2) and will not be repeated here. Parameters not appearing in formula (2) This indicates the position in the output sequence pre-generated by the teacher model. All previous morphemes.

[0061] Accordingly, if it is desired to supervise two training and distillation methods simultaneously, the loss function can be constructed from the first loss sub-function exemplified by formula (2) and the second loss sub-function exemplified by formula (3). For example, in such a case, the executing entity can combine formula (2) and formula (3) by direct addition, proportional addition, weighted addition, etc., to obtain the loss function. Accordingly, for ease of distinction, such a loss function including formula (2) and formula (3) is called " L " can be described as .Right now This can be understood as being derived from formulas (2) and (3) by " L The result of the combination of "".

[0062] In some embodiments, in order to enable student models to better form their own model knowledge rather than simply imitating teacher models, the participation of teacher models can be controlled and managed based on an annealing mechanism. This allows teacher models to gradually withdraw as the number of rounds increases and training deepens during the distillation of model knowledge, thereby reducing the dependence of student models on teacher models.

[0063] In this case, the loss function described above can be based on the aforementioned... The engagement coefficients of the teacher model are constructed. For ease of understanding, in this case, the loss function used to supervise the training process of the student model can be generally characterized as follows: And that It can be determined by the following formula (4): (4) in, For the first The teacher participation weights in the step-by-step model have a range of values. It gradually decreases with training.

[0064] In some embodiments, if during the training of the student model, in addition to supervising the distillation process of its model knowledge of the teacher model, it is also desired to supervise the native capabilities of the student model, the supervision weight of the native capabilities can be increased accordingly as the reference weight of the teacher model decreases, thereby ensuring that the native capabilities are effectively trained while enabling them to be better distilled in terms of model knowledge.

[0065] Accordingly, the It can be determined by the following formula (5): (5) in, This is the loss function for the original task.

[0066] In some embodiments, to allow the teacher model to exit more smoothly, a cosine annealing mechanism can be used to perform the annealing process described above. Accordingly, in such cases, the participation weights... It can be determined by the following formula (6): (6) in, This represents the upper bound of teacher model engagement, corresponding to the state of full-fledged guidance by the teacher model in the early stages of training. This represents the lower bound of the teacher model's involvement, corresponding to the minimum level of teacher model intervention in the later stages of training. This represents the current number of training steps and rounds. The total number of training steps and rounds (for example, if the distillation process is trained using a defined number of rounds, then that defined number of rounds can be used as the training step number). T Thus, in this way, The decay curve is smooth, starting slowly, then accelerating, and then slowing down again, so that the teacher model can smoothly and quickly reduce the intensity of guidance.

[0067] In some embodiments, in addition to the cosine annealing mechanism described above, the executing agent may also employ staged annealing (e.g., dividing the training rounds into multiple groups sequentially, and then each group has a corresponding...). The value of the annealing intensity can be used to control the guidance intensity of the teacher model.

[0068] In some embodiments, a specific timing for initiating the annealing action can be configured to allow for more effective use of the teacher model for guidance early in the training process. For example, the implementing agent can estimate the student model's capabilities based on the dispersion between the student model's output and the teacher model's data, and determine the annealing round to begin based on the specific value of that dispersion.

[0069] In some embodiments, if the training process is supervised using at least a second loss function (e.g., the loss function is constructed by the second loss function alone or in conjunction with the first loss function), the executing agent can continuously obtain the magnitude of the decrease in the teacher model output of the second loss function.

[0070] If the decrease in the second loss function over a consecutive number of teacher model outputs is less than or equal to a threshold (which is usually preset based on the standard that it has little effect on the model's ability), that is, after training with a consecutive number of teacher model outputs, the decrease in each round is less than or equal to this threshold, then the executing agent can respond by choosing to lower the first parameter threshold (for example, by adjusting its specific value to a predetermined value lower than the current value, or by removing a value based on a predetermined adjustment strategy to lower the first parameter threshold). This approach allows the teacher model output to quickly and significantly exit the training process using the teacher model output when it can no longer provide effective reference or guidance, allowing the student model to be trained more in a corrective manner.

[0071] This allows student models to be trained more efficiently and with higher quality, while also enabling them to better develop their own, localized model knowledge.

[0072] In some embodiments, as discussed above, since using output scoring often enables student models to better form "self-localized model knowledge," the implementing entity may prefer to use "correction" methods for model knowledge distillation rather than teacher model output when the student model's ability is sufficient to generate high-quality student model output. This is to help student models better form self-localized model knowledge and achieve higher-quality model knowledge distillation.

[0073] Therefore, in this case, the first parameter threshold can be determined based on the total number of output scores and teacher model outputs already used, for example, before the start of each round or at regular intervals. Correspondingly, this first parameter threshold exhibits a negative numerical correlation with the total number of scores. That is, as the total number of scores increases, the first parameter threshold adaptively decreases. Typically, a correspondence function between the two can be constructed, or a pre-determined table of their relationship can be used, so that after determining the total number of scores, the executing entity can determine the corresponding first parameter threshold based on this function and the table of relationships.

[0074] Therefore, as the number of training steps and rounds increases, it is more inclined to use "correction" to distill the model, so as to better form the student model's own and localized model knowledge, enabling it to better have the ability to "independently" process tasks and inputs, rather than simply "imitating" the existing content of the teacher model.

[0075] Similarly, in some embodiments, in order to improve the formation of student model self-localized model knowledge and enable it to better have the ability to "independently" process tasks and inputs, the executing entity can also generate adjustment parameters based on the total number of output scores already used and teacher model outputs, so as to update and adjust the quality evaluation parameters based on the adjustment parameters (for example, using adjustment parameters that can be greater than or less than 1 to amplify or reduce the original quality evaluation coefficient).

[0076] Correspondingly, the adjustment parameter can be positively correlated with the total quantity; that is, as the total quantity increases, the adjustment parameter can be larger so that it can amplify the quality evaluation parameters more.

[0077] As a result, with the increase in the number of samples used and training rounds, even for the same quality evaluation parameters, the implementing entity is more inclined to use the "correction" method to train the student model, so as to better form the student model's own, localized model knowledge and reduce its dependence on the teacher model.

[0078] In some embodiments, as discussed above, the implementing entity may also “evaluate” the current capabilities of the student model by the student model output to determine whether it is more suitable to support more “corrective” training and distillation.

[0079] In this case, a second parameter threshold, numerically greater than the first parameter threshold, can also be set. Then, if a student model outputs a quality evaluation parameter greater than the second parameter threshold, the implementing entity can respond by lowering the (current) first parameter threshold (e.g., by lowering it to a specific value as discussed above, or by lowering it incrementally) to allow the student model to be trained more in a corrective manner.

[0080] It should be understood that if the "first parameter threshold" can be dynamically determined using the "total quantity" as described in some of the embodiments above, then in such cases, the range that the first parameter threshold can be used can be determined first, and based on that range, a second parameter threshold greater than the upper limit of that range can be selected to avoid the two overlapping.

[0081] Therefore, in this way, the second parameter threshold can be used to indicate and calibrate "high-quality student model output". When the student model has the ability to output such high-quality student model output, the distillation and training strategies can be adjusted so that after the student model has a certain processing ability, it can be distilled and trained in the direction of better forming its local model knowledge and capabilities.

[0082] In some embodiments, similarly, the number of already generated output scores can be managed to control and adjust the ratio of distillation and training rounds between the correction method and the teacher model output method, so as to avoid the distillation and training process from completely tilting towards the correction process, causing the teacher model output method to become ineffective and affecting the distillation effect.

[0083] Specifically, in such a case, after the executing entity generates the output score of the student model through the teacher model in response to the quality evaluation parameter being greater than the first parameter threshold, it can simultaneously determine whether the number of output scores already generated is less than the quantity threshold (which can be determined in advance based on the expected usage ratio of the two methods), that is, whether the number of output scores used in the previous round is less than the quantity threshold.

[0084] If the quality evaluation parameter is greater than the first parameter threshold, and the number of output scores already generated is less than the quantity threshold, then the executing entity can generate the output scores of the student model through the teacher model, as discussed above. This allows for more effective training using correction methods within the desired range, thereby forming the model's own model knowledge.

[0085] If the quality evaluation parameter is greater than the first parameter threshold, and the number of output scores already generated is greater than or equal to the quantity threshold, the executing entity can choose to obtain the teacher model output generated by the teacher model processing the sample input. Then, based on the teacher model output, the student model is trained. This avoids the distillation and training processes completely reverting to the correction process, which would render the teacher model output method ineffective and affect the distillation effect.

[0086] In some embodiments, to avoid resource loss and impact on training results caused by using model knowledge that the student model already fully possesses, or knowledge that the student model cannot fully reach, to train the student model, the executing entity can also manage and filter the student model output, so as to exclude student model outputs that can reflect that the student model already fully possesses the capabilities, and student model outputs that are completely unreachable from flowing into the training process.

[0087] For ease of understanding, please refer to the following: Figure 3 . Figure 3A flowchart of a process for generating quality evaluation parameters provided in an embodiment of this disclosure includes process 300. For example, process 300 can be an alternative or alternative implementation of step 201 described above.

[0088] Process 300 specifically includes the following steps: Step 301: Based on the initial student model output generated from the student model processing sample input and the uncertainty index of the student model relative to the initial student model, the initial student model whose uncertainty index is within the preset numerical range is determined as the student model output; Specifically, the executing entity can first determine the results generated and output by the student model from the sample input as the initial student model output. Then, by adding a classification head or similar method to the student model, the student model can provide an uncertainty indicator for that initial student model output along with the initial student model output. For example, regarding the initial student model output... The student model can output its uncertainty index. .

[0089] In some embodiments, uncertainty indicators It can be determined by the following formula (7): (7) in, It is a scalar value, obtained by averaging the entropy over all positions in the sequence. The first generation generated for the student model The output of the initial student model ; The information entropy function is defined as follows: ,in For vocabulary list; Let be the conditional probability distribution function of the student model; Input for the sample; Indicates the first Position in candidate sequences All previously generated lexical units, i.e. ; For the student model, input a given sample and the generated prefix Under the condition of position The probability distribution of the next word, with a dimension equal to the size of the vocabulary. .

[0090] Then, the executing entity can Located at the upper and lower limits and The initial student model within the preset numerical range determines the student model output that can be used subsequently. This can be understood as a lower bound threshold for uncertainty; below this value, it indicates that the student model has fully grasped the content, and the teacher model's guidance is redundant. This can be understood as an upper bound threshold for uncertainty. A value higher than this indicates that the sample exceeds the current learnable range of the student model, and the teacher's guidance is of little value.

[0091] Accordingly, after the “student model output” is determined in this step, the executing entity can continue to generate quality evaluation parameters for the student model output by executing step 302 as in step 201 above. The same parts can be referred to the situation discussed in step 201 above, and will not be repeated here.

[0092] Step 302: Based on the current state parameters of the student model and the student model output generated from the processed sample input, generate quality evaluation parameters for the student model output.

[0093] To enhance understanding, this disclosure also provides an implementation scheme for a specific application scenario. Please refer to this for further clarification. Figure 4 . Figure 4 A flowchart illustrating the process of implementing distillation model knowledge in a specific application scenario according to embodiments of this disclosure includes process 400. For example, process 400 may be... Figure 1 In the example architecture 100, server 105 is implemented as the "executor".

[0094] Specifically, in process 400, server 105 can first execute S401 by example to call student model 410 to process sample input 421 in order to generate corresponding student model outputs 431, 432 and 433.

[0095] It should be understood that the number of student model outputs mentioned above (e.g., the "3" student model outputs 431, 432, and 433) is merely an arbitrary choice made for the convenience of the example and is not intended to limit the number of student model outputs generated.

[0096] Then, server 105 can generate quality evaluation parameters 441 for model output 431, quality evaluation parameters 442 for student model output 432, and quality evaluation parameters 443 for student model output 433 by executing S402 based on the current state parameters 422 of student model 410 and student model outputs 431, 432, and 433.

[0097] To facilitate a clearer illustration and discussion of the technical solutions disclosed herein, in process 400, the quality evaluation parameters 441 and 442 may have specific values ​​greater than the first parameter threshold, while the quality evaluation parameter 443 may be less than the first parameter threshold.

[0098] Accordingly, in such a case, server 105 can generate output scores 451 and 452 of student model outputs 431 and 432 through teacher model 415 by executing S403 and S404 respectively (the two can be executed in any order or simultaneously), and obtain teacher model output 460 generated by teacher model 415 (so as to use the teacher output model 460 to replace or substitute student model 433).

[0099] Next, server 105 can train student model 410 based on output scores 451 and 452 and teacher model output 460 by executing S405. In practice, as discussed above, sample input 421, etc., can also be used in this training process, which will not be repeated here.

[0100] Accordingly, during this training process, supervision can be provided using a loss function 470. For example, the loss function 470 can consist of three parts: a first loss subfunction 471 for supervising the training process based on output scores 451 and 452, a second loss subfunction 472 for supervising the training process based on the teacher model output 460, and an engagement coefficient 473 for incorporating an "annealing mechanism" for the teacher model 460.

[0101] Subsequently, if the loss function 470 reaches the training objective, the server 105 can execute S406 to end the training and obtain the target student model 410' trained by the student model 410.

[0102] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a distillation model knowledge apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0103] like Figure 5As shown, the distillation model knowledge apparatus 500 of this embodiment may include: an evaluation parameter generation unit 501, an output score output unit 502, a first student training unit 503, and a training termination unit 504. The evaluation parameter generation unit 501 is configured to generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated from the processed sample inputs. The output score output unit 502 is configured to generate an output score for the student model output through a teacher model in response to the quality evaluation parameters exceeding a first parameter threshold. The first student training unit 503 is configured to train the student model based on the output score. The training termination unit 504 is configured to terminate training in response to the loss function used to supervise the training process of the student model reaching the training objective, thereby obtaining the target student model trained by the student model.

[0104] In this embodiment, the specific processing and technical effects of the evaluation parameter generation unit 501, the output score output unit 502, the first student training unit 503, and the training end unit 504 in the distillation model knowledge apparatus 500 can be referred to respectively. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiments will not be repeated here.

[0105] In some optional implementations of this embodiment, the apparatus 500 further includes: a first teacher model output acquisition unit, configured to acquire the teacher model output generated by the teacher model processing the sample input in response to the quality evaluation parameter being less than or equal to a first parameter threshold; and a second student training unit, configured to train the student model based on the teacher model output.

[0106] In some optional implementations of this embodiment, the loss function is constructed from a first loss subfunction and a second loss subfunction. The first loss subfunction is used to supervise the first training process of training the student model using the output score, and the second loss subfunction is used to supervise the second training process of training the student model based on the teacher model output.

[0107] In some optional implementations of this embodiment, the loss function is constructed from a first loss subfunction, a second loss subfunction, and the participation coefficient of the teacher model. The participation coefficient is determined based on the total number of output ratings that have been used and the total number of outputs from the teacher model. The participation coefficient is negatively correlated with the total number of outputs.

[0108] In some optional implementations of this embodiment, the apparatus 500 further includes: a parameter threshold first adjustment unit, configured to reduce the first parameter threshold in response to the fact that the magnitude of the decrease of the second loss function on the teacher model output of a continuous number of targets is less than or equal to the magnitude threshold.

[0109] In some optional implementations of this embodiment, the apparatus 500 further includes: a parameter threshold determination unit, configured to determine a first parameter threshold based on the total number of output scores already used and the total number of teacher model outputs, wherein the first parameter threshold has a negative numerical correlation with the total number.

[0110] In some optional implementations of this embodiment, the apparatus 500 further includes: an adjustment parameter generation unit configured to generate adjustment parameters based on the total number of output scores already used and the total number of teacher model outputs, wherein the adjustment parameters are positively correlated with the total number; and an evaluation parameter update unit configured to update the quality evaluation parameters based on the adjustment parameters.

[0111] In some optional implementations of this embodiment, the apparatus 500 further includes: a second parameter threshold adjustment unit, configured to reduce the first parameter threshold in response to a quality evaluation parameter being greater than a second parameter threshold, wherein the second parameter threshold is greater than the first parameter threshold.

[0112] In some optional implementations of this embodiment, the output score output unit 502 is further configured to generate the output score of the student model through the teacher model in response to the quality evaluation parameter being greater than the first parameter threshold and the number of output scores already generated being less than the number threshold.

[0113] In some optional implementations of this embodiment, the apparatus 500 further includes: a second teacher model output acquisition unit, configured to acquire the teacher model output generated by the teacher model processing the sample input in response to the quality evaluation parameter being greater than the first parameter threshold and the number of output scores currently generated being greater than or equal to the number threshold; and a third student training unit, configured to train the student model based on the teacher model output.

[0114] In some optional implementations of this embodiment, the evaluation parameter generation unit 501 includes: a student model output determination subunit, configured to determine the initial student model output as the student model output based on the initial student model output generated by the student model processing sample input and the uncertainty index of the student model relative to the initial student model; and an evaluation parameter generation subunit, configured to generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated by the student model processing sample input.

[0115] This embodiment exists as a device embodiment corresponding to the above method embodiment. The device for distilling model knowledge provided in this embodiment can not only reduce the requirements for teachers' model resource usage and model capabilities, enabling more models to be compressed and lightweighted through distillation, but also improve the efficiency and quality of the model knowledge distillation process.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0117] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0119] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method of distilling model knowledge. For example, in some embodiments, the method of distilling model knowledge may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method of distilling model knowledge described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method of distilling model knowledge by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service ecosystem to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. Servers can also be categorized as distributed system servers or servers incorporating blockchain technology.

[0127] According to the technical solutions of the embodiments of this disclosure, not only can the requirements for the use of teacher model resources and model capabilities be reduced, enabling more models to be compressed and lightweighted through distillation, but also the efficiency and quality of the model knowledge distillation process can be improved.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for knowledge of distillation models, comprising: Based on the current state parameters of the student model and the student model output generated by the student model through the sample input, quality evaluation parameters for the student model output are generated. In response to the quality evaluation parameter being greater than the first parameter threshold, the output score of the student model is generated through the teacher model. The student model is trained based on the output score. In response to the loss function used to supervise the training process of the student model reaching the training objective, the training ends, and the target student model trained by the student model is obtained.

2. The method according to claim 1, further comprising: In response to the quality evaluation parameter being less than or equal to the first parameter threshold, the teacher model output generated by the teacher model processing the sample input is obtained; The student model is trained based on the output of the teacher model.

3. The method according to claim 2, wherein, The loss function is constructed from a first loss sub-function and a second loss sub-function. The first loss sub-function is used to supervise a first training process in which the student model is trained using the output score, and the second loss sub-function is used to supervise a second training process in which the student model is trained based on the output of the teacher model.

4. The method according to claim 3, wherein, The loss function is constructed from the first loss sub-function, the second loss sub-function, and the participation coefficient of the teacher model. The participation coefficient is determined based on the total number of output ratings that have been used and the total number of outputs of the teacher model. The participation coefficient is negatively correlated with the total number of outputs.

5. The method according to claim 4, further comprising: In response to the fact that the decrease in the second loss function on the teacher model output for a continuous number of targets is less than or equal to the magnitude threshold, the first parameter threshold is reduced.

6. The method according to claim 2, further comprising: Based on the total number of output scores already used and the total number of outputs from the teacher model, a first parameter threshold is determined, wherein the first parameter threshold is negatively correlated with the total number.

7. The method according to claim 2, further comprising: Based on the output scores that have already been used and the total number of outputs from the teacher model, adjustment parameters are generated, wherein the adjustment parameters are positively correlated with the total number. The quality evaluation parameters are updated based on the adjustment parameters.

8. The method according to claim 1, further comprising: In response to the quality evaluation parameter being greater than the second parameter threshold, the first parameter threshold is reduced, wherein the second parameter threshold is greater than the first parameter threshold.

9. The method according to claim 1, wherein, The step of generating the output score of the student model through the teacher model in response to the quality evaluation parameter being greater than the first parameter threshold includes: In response to a quality evaluation parameter being greater than a first parameter threshold and the number of output scores already generated being less than a quantity threshold, the output score output by the student model is generated through the teacher model.

10. The method according to claim 9, further comprising: In response to a quality evaluation parameter being greater than a first parameter threshold, and the number of output scores already generated being greater than or equal to the number threshold, the teacher model output generated by the teacher model processing the sample input is obtained; The student model is trained based on the output of the teacher model.

11. The method according to any one of claims 1-10, wherein, The quality evaluation parameters for the student model output are generated based on the current state parameters of the student model and the student model output generated by processing sample inputs, including: Based on the initial student model output generated from the sample input processed by the student model and the uncertainty index of the student model relative to the initial student model, the initial student model whose uncertainty index is within a preset numerical range is determined as the student model output; Based on the current state parameters of the student model and the student model output generated by the student model in processing the sample input, quality evaluation parameters for the student model output are generated.

12. An apparatus for generating distillation model knowledge, comprising: The evaluation parameter generation unit is configured to generate quality evaluation parameters for the student model output based on the current state parameters of the student model and the student model output generated by the student model processing sample inputs. The output score output unit is configured to generate the output score of the student model through the teacher model in response to the quality evaluation parameter being greater than the first parameter threshold. The first student training unit is configured to train the student model based on the output score; The training termination unit is configured to terminate the training in response to the loss function used to supervise the training process of the student model reaching the training objective, thereby obtaining the target student model trained by the student model.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the method of distillation model knowledge as described in any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a method of distillation model knowledge according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method of distillation model knowledge according to any one of claims 1-11.

Images