Model training methods, devices, equipment and media based on knowledge distillation

Abstract

The present application discloses a model training method, apparatus, equipment and medium based on knowledge distillation, which belongs to the field of artificial intelligence technology. The method includes: obtaining a first training data set and a second training data set; training a first deep learning model based on the first training data set to obtain a first teacher model; in the training process of the student model, obtaining the knowledge distillation intensity according to the first type of indicators; wherein the knowledge distillation intensity is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicators include the training status and model performance of the student model, the temperature parameters during the knowledge distillation process, and at least one of the model training parameters; under the constraint of the knowledge distillation intensity, based on the second training data set and the output probability distribution of the first teacher model, training a second deep learning model for performing the target deep learning task to obtain a student model. The present application can improve the training efficiency and performance of the student model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, equipment and medium based on knowledge distillation. Background Art

[0002] Knowledge distillation (KD), as a model compression technique, is based on the idea of transferring knowledge learned from a large neural network (also known as a general model or teacher model) to a small neural network (also known as a vertical model or student model). In other words, the goal of knowledge distillation is to transfer knowledge learned from the general model to the vertical model, so that the vertical model has lower computational complexity while maintaining high performance.

[0003] Knowledge distillation technology has been applied in numerous fields, such as natural language processing. Regardless of the application, achieving more efficient knowledge transfer and better vertical model performance has always been a research hotspot. Specifically, improving the training efficiency and performance of vertical models through novel model training methods is a key focus in this field. Summary of the Invention

[0004] The present application provides a model training method, apparatus, device, and medium based on knowledge distillation, which improves the training efficiency and performance of the student model. The technical solution is as follows.

[0005] On the one hand, a model training method based on knowledge distillation is provided, and the method comprises the following steps.

[0006] Obtain a first training data set and a second training data set;

[0007] Training a first deep learning model based on the first training data set to obtain a first teacher model;

[0008] During the training of the student model, obtaining the knowledge distillation strength according to a first type of indicator; wherein the knowledge distillation strength is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicator includes at least one of the training status and model performance of the student model, a temperature parameter during the knowledge distillation process, and a model training parameter;

[0009] Under the constraint of the knowledge distillation strength, based on the second training data set and the output probability distribution of the first teacher model, a second deep learning model for performing the target deep learning task is trained to obtain the student model.

[0010] In one possible implementation, the first teacher model is used to perform multiple deep learning tasks; the training data in the second training data set is derived from the first training data set; the training data in the first training data set is unlabeled; the training data in the second training data set is labeled; or,

[0011] The first training data set and the second training data set are the same data set that matches the target deep learning task.

[0012] In a possible implementation, the method further includes:

[0013] During the training of the student model, a knowledge distillation rate is obtained according to a second type of indicator; wherein the knowledge distillation rate is used to control the speed of knowledge distillation; the second type of indicator includes at least one of the training progress, model performance, and model training parameters of the student model;

[0014] The step of training a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model includes:

[0015] Under the constraints of the knowledge distillation strength and the knowledge distillation rate, the second deep learning model is trained based on the second training data set and the output probability distribution of the first teacher model to obtain the student model.

[0016] In one possible implementation, training a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model to obtain the student model includes:

[0017] Constructing a first loss function based on the output probability distribution of the first teacher model and the output probability predicted by the second deep learning model;

[0018] Constructing a second loss function based on the second training data set and the output probability predicted by the second deep learning model;

[0019] Obtaining a first weight of the first loss function and a second weight of the second loss function;

[0020] Weighting the first loss function and the second loss function based on the first weight and the second weight to obtain a target loss function;

[0021] The loss value is obtained by iteratively obtaining the target loss function until the training stop condition is met, thereby obtaining the student model.

[0022] In a possible implementation, obtaining a first weight of the first loss function and a second weight of the second loss function includes:

[0023] Periodically obtaining performance changes of the student model on a specified data set; determining the first weight and the second weight based on the performance changes; wherein the specified data set is a validation data set or a test data set; or,

[0024] Determining the first weight and the second weight based on the model training parameters; or,

[0025] Configure weight parameters during the model training process; and determine the first weight and the second weight during the model training process based on the weight parameters.

[0026] In a possible implementation, the training status includes the training progress and loss change of the student model; and obtaining the knowledge distillation intensity in the knowledge distillation process according to the first type of indicator includes:

[0027] Obtaining a knowledge distillation strength according to each item in the first category of indicators, and obtaining a plurality of knowledge distillation strengths that match the number of indicator items included in the first category of indicators;

[0028] Obtain the influence weight of each of the first category of indicators on the strength of knowledge distillation;

[0029] Determine the current knowledge distillation intensity based on the knowledge distillation intensity and influence weight corresponding to each item in the first category of indicators.

[0030] In one possible implementation, the step of training a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model includes:

[0031] Obtaining an intermediate layer feature representation of the first teacher model and an intermediate layer feature representation of the second deep learning model;

[0032] Under the constraint of the knowledge distillation strength, the second deep learning model is trained based on the second training data set, the output probability distribution and intermediate layer feature representation of the first teacher model, and the intermediate layer feature representation of the second deep learning model to obtain the student model.

[0033] In a possible implementation, the method further includes:

[0034] Constructing a meta-learning task, the meta-learning task comprising a support set and a query set; wherein the support set comprises a plurality of subtasks, each subtask being configured with a different first-category indicator; and the query set comprises task samples for training the subtasks;

[0035] Training the meta-learning model based on the support set and performing model performance evaluation based on the query set to obtain a trained meta-learning model;

[0036] During the training process of the student model, the trained meta-learning model is called to obtain the knowledge distillation intensity in the knowledge distillation process based on the task characteristics of the target deep learning task or the data distribution of the second training data set.

[0037] In a possible implementation, the method further includes:

[0038] Obtain multiple training data sets;

[0039] Training multiple deep learning models based on the multiple training data sets to obtain multiple second teacher models;

[0040] The step of training a second deep learning model for performing a target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain the student model includes:

[0041] Obtaining a weight for each of the first teacher model and the plurality of second teacher models according to the training progress and model performance of the student model;

[0042] Based on the obtained weight of each model, weight the output probabilities of the first teacher model and the multiple second teacher models to obtain a fused output probability distribution;

[0043] Based on the second training data set and the fused output probability distribution, the second deep learning model is trained to obtain the student model.

[0044] In a possible implementation, the method further includes:

[0045] In the case of newly added training data, fine-tuning the first teacher model based on the newly added training data to obtain an updated first teacher model;

[0046] The step of training a second deep learning model for performing a target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain the student model includes:

[0047] Based on the second training data set and the output probability distribution of the updated first teacher model, the second deep learning model is trained to obtain the student model.

[0048] In a possible implementation, the method further includes:

[0049] Based on the newly added training data, the trained student model is fine-tuned to obtain an updated student model.

[0050] In a possible implementation, the method further includes:

[0051] Receive input data matching the target deep learning task; wherein the input data is at least one of text, image, audio, and video;

[0052] The student model is called to perform the target deep learning task according to the input data.

[0053] On the other hand, a model training device based on knowledge distillation is provided, which includes the following modules.

[0054] A first acquisition module is configured to acquire a first training data set and a second training data set;

[0055] A first training module is configured to train a first deep learning model based on the first training data set to obtain a first teacher model;

[0056] a second acquisition module configured to acquire, during the training process of the student model, knowledge distillation strength based on a first type of indicator; wherein the knowledge distillation strength is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicator includes at least one of the training state and model performance of the student model, a temperature parameter during the knowledge distillation process, and a model training parameter;

[0057] The second training module is configured to train a second deep learning model for performing the target deep learning task based on the second training data set and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model.

[0058] In one possible implementation, the first teacher model is used to perform multiple deep learning tasks; the training data in the second training dataset is derived from the first training dataset; the training data in the first training dataset is unlabeled; the training data in the second training dataset is labeled; or, the first training dataset and the second training dataset are the same dataset that matches the target deep learning task.

[0059] In one possible implementation, the first acquisition module is further configured to acquire a knowledge distillation rate according to a second type of indicator during the training of the student model; wherein the knowledge distillation rate is used to control the speed of knowledge distillation; and the second type of indicator includes at least one of the training progress, model performance, and model training parameters of the student model;

[0060] The second training module is configured to train the second deep learning model based on the second training data set and the output probability distribution of the first teacher model under the constraints of the knowledge distillation intensity and the knowledge distillation rate to obtain the student model.

[0061] In a possible implementation, the second training module is configured to:

[0062] Constructing a first loss function based on the output probability distribution of the first teacher model and the output probability predicted by the second deep learning model;

[0063] Constructing a second loss function based on the second training data set and the output probability predicted by the second deep learning model;

[0064] Obtaining a first weight of the first loss function and a second weight of the second loss function;

[0065] Weighting the first loss function and the second loss function based on the first weight and the second weight to obtain a target loss function;

[0066] The loss value is obtained by iteratively obtaining the target loss function until the training stop condition is met, thereby obtaining the student model.

[0067] In a possible implementation, the second training module is configured to:

[0068] Periodically obtaining performance changes of the student model on a specified data set; determining the first weight and the second weight based on the performance changes; wherein the specified data set is a validation data set or a test data set; or,

[0069] Determining the first weight and the second weight based on the model training parameters; or,

[0070] Configure weight parameters during the model training process; and determine the first weight and the second weight during the model training process based on the weight parameters.

[0071] In a possible implementation, the training status includes the training progress and loss change of the student model; and the second acquisition module is configured to:

[0072] Obtaining a knowledge distillation strength according to each item in the first category of indicators, and obtaining a plurality of knowledge distillation strengths that match the number of indicator items included in the first category of indicators;

[0073] Obtain the influence weight of each of the first category of indicators on the strength of knowledge distillation;

[0074] Determine the current knowledge distillation intensity based on the knowledge distillation intensity and influence weight corresponding to each item in the first category of indicators.

[0075] In a possible implementation, the second training module is configured to:

[0076] Obtaining an intermediate layer feature representation of the first teacher model and an intermediate layer feature representation of the second deep learning model;

[0077] Under the constraint of the knowledge distillation strength, the second deep learning model is trained based on the second training data set, the output probability distribution and intermediate layer feature representation of the first teacher model, and the intermediate layer feature representation of the second deep learning model to obtain the student model.

[0078] In a possible implementation, the apparatus further includes: a third acquisition module; the third acquisition module is configured to:

[0079] Constructing a meta-learning task, the meta-learning task comprising a support set and a query set; wherein the support set comprises a plurality of subtasks, each subtask being configured with a different first-category indicator; and the query set comprises task samples for training the subtasks;

[0080] Training the meta-learning model based on the support set and performing model performance evaluation based on the query set to obtain a trained meta-learning model;

[0081] During the training process of the student model, the trained meta-learning model is called to obtain the knowledge distillation intensity in the knowledge distillation process based on the task characteristics of the target deep learning task or the data distribution of the second training data set.

[0082] In a possible implementation, the first acquisition module is further configured to acquire multiple training data sets;

[0083] The first training module is further configured to train multiple deep learning models based on the multiple training data sets to obtain multiple second teacher models;

[0084] The second training module is configured to:

[0085] Obtaining a weight for each of the first teacher model and the plurality of second teacher models according to the training progress and model performance of the student model;

[0086] Based on the obtained weight of each model, weight the output probabilities of the first teacher model and the multiple second teacher models to obtain a fused output probability distribution;

[0087] Based on the second training data set and the fused output probability distribution, the second deep learning model is trained to obtain the student model.

[0088] In a possible implementation, the first training module is further configured to, when new training data is added, fine-tune the first teacher model based on the new training data to obtain an updated first teacher model;

[0089] The second training module is further configured to train the second deep learning model based on the second training data set and the output probability distribution of the updated first teacher model to obtain the student model.

[0090] In a possible implementation, the second training module is further configured to fine-tune the student model based on the newly added training data to obtain an updated student model.

[0091] In a possible implementation, the apparatus further includes: a processing module;

[0092] The processing module is configured to receive input data matching the target deep learning task; wherein the input data is at least one of text, image, audio and video; and call the student model to perform the target deep learning task according to the input data.

[0093] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the above-mentioned model training method based on knowledge distillation.

[0094] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the above-mentioned model training method based on knowledge distillation.

[0095] On the other hand, a computer program product or computer program is provided, which includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the above-mentioned knowledge distillation-based model training method.

[0096] The model training scheme based on knowledge distillation provided in the embodiment of the present application can automatically adjust the knowledge distillation intensity during the training process of the student model, which enables the student model to obtain appropriate supervision signals at different stages of the knowledge distillation process, thereby improving the training efficiency and model performance of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0098] Figure 1 Schematic diagram of an implementation environment involved in a model training method based on knowledge distillation provided in an embodiment of the present application;

[0099] Figure 2 This is a flow chart of a model training method based on knowledge distillation provided in an embodiment of the present application;

[0100] Figure 3 This is a flowchart of another model training method based on knowledge distillation provided in an embodiment of the present application;

[0101] Figure 4 Schematic diagram of a model training device based on knowledge distillation provided in an embodiment of the present application;

[0102] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0103] Figure 6 It is a structural diagram of another computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0105] In this application, the terms "first," "second," and the like are used to distinguish identical or similar items having substantially the same role and function. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms.

[0106] These terms are simply used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of various examples. Both the first element and the second element can be elements, and in some cases, can be separate and different elements.

[0107] Here, at least one refers to one or more than one. For example, at least one element can be one element, two elements, three elements, or any other integer greater than or equal to one. And multiple refers to two or more than two. For example, multiple elements can be two elements, three elements, or any other integer greater than or equal to two.

[0108] The term "and / or" used in this document indicates that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0109] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0110] Before explaining in detail the model training scheme based on knowledge distillation provided in the embodiment of the present application, knowledge distillation is first introduced.

[0111] As mentioned earlier, knowledge distillation is a technique for transferring knowledge learned by a teacher model to a student model. This technique reduces computational complexity while maintaining model performance, enabling the model to run efficiently in resource-constrained environments or devices. In other words, the goal of knowledge distillation is to transfer knowledge learned by the teacher model to the student model, enabling the student model to maintain high performance while having lower computational complexity.

[0112] Knowledge transfer in knowledge distillation involves both soft and hard targets. The soft target generally refers to the output probability distribution of the teacher model. The hard target generally refers to the true value corresponding to the output probability predicted by the student model, i.e., the labeled data. Soft and hard targets can be combined when training the student model to improve its generalization ability. Specifically, the optimization objective in knowledge distillation typically consists of two parts: the loss of the student model on the hard target and the loss of the student model on the soft target. By balancing these two losses, a balance can be achieved to maintain the performance of the student model while reducing computational complexity.

[0113] Figure 1 This is a schematic diagram of the implementation environment involved in a model training method based on knowledge distillation provided in an embodiment of the present application.

[0114] See also Figure 1 In the training phase, the first computer device 110 is used to perform model training based on knowledge distillation to obtain a teacher model and a student model. In the inference phase, the second computer device 120 performs deep learning tasks using the trained student model.

[0115] The first computer device 110 and the second computer device 120 are computer devices with machine learning capabilities. For example, the computer devices may be personal computers or servers, or they may be resource-constrained edge computing devices, such as smartphones or embedded devices.

[0116] It should be noted that the first computer device 110 and the second computer device 120 may be the same device, or the first computer device 110 and the second computer device 120 may be different devices, which is not limited in this application.

[0117] Based on the above implementation environment, the model training method based on knowledge distillation provided in the embodiment of the present application realizes adaptive knowledge distillation, which can more efficiently transfer knowledge and train a student model with better performance, that is, obtain a high-precision, low-computational complexity student model after training and optimization.

[0118] For example, during the training process, the embodiments of the present application include but are not limited to applying the following technologies:

[0119] Adaptive knowledge distillation strength adjustment, dynamic loss weight adjustment, hierarchical knowledge distillation, and meta-learning-based knowledge distillation strength adjustment, etc.; in addition, during the training process, the following extended optimization techniques can be selectively applied: efficient training technology, online knowledge distillation, multi-model knowledge distillation, incremental knowledge distillation, combination of model compression and distillation, etc., as described below.

[0120] The following implementation method is used to introduce in detail the model training scheme based on knowledge distillation provided in the embodiment of the present application.

[0121] Figure 2 This is a flow chart of a model training method based on knowledge distillation provided in an embodiment of the present application. The execution subject of this method is a computer device. Figure 2 The method flow provided in the embodiment of the present application includes the following steps.

[0122] 201. A computer device obtains a first training data set and a second training data set.

[0123] In the embodiment of the present application, the first training data set is used to train the teacher model of knowledge distillation, and the second training data set is used to train the student model of knowledge distillation. In order to distinguish it from the teacher model that appears later, the teacher model here is also referred to as the first teacher model.

[0124] In one possible implementation, the first teacher model is a global, general-purpose model that can be used to perform a variety of deep learning tasks. In other words, the first teacher model is a global, general-purpose model, typically trained on a large dataset (i.e., the first training dataset is a large dataset) to capture a wide range of knowledge and features. While a global, general-purpose model performs well across the entire task domain, it may underperform on specific subtasks or localized data distributions.

[0125] The first point to be explained is that, when the first teacher model is a global general model, the second training data set can be a subset of the first training data set, that is, the training data in the second training data set is derived from the first training data set, and this application does not limit this. In addition, the training data in the first training data set is unlabeled, that is, the first teacher model is usually trained using a self-supervised learning method. The training data in the second training data set is labeled, that is, the student model is usually trained using a supervised learning method.

[0126] In another possible implementation, the first teacher model and the student model are related to the target deep learning task. In this case, the first training data set and the second training data set are the same data set that matches the target deep learning task, which is not limited in this application.

[0127] In another possible implementation, to improve the generalization ability of the student model, the present embodiment also introduces a privately deployed universal model strategy. In this strategy, the student model not only needs to learn the knowledge of the global universal model, but also needs to learn the knowledge of a privately deployed universal model for a specific sample subset. In this way, the student model can adapt to the local data distribution while maintaining global performance.

[0128] Second, the specific sample subset refers to the local data that the student model needs to adapt to. This subset is typically selected from the training dataset used to train the global general model. By introducing a privately deployed general model, the student model can better adapt to the characteristics of the specific sample subset, thereby improving the model's performance on the local data distribution.

[0129] 202. The computer device trains a first deep learning model based on the first training data set to obtain a first teacher model for knowledge distillation.

[0130] In an embodiment of the present application, the first deep learning model can be any type of deep learning model, such as a convolutional neural network, a recurrent neural network, or a Transformer structure, etc., and this application does not limit this.

[0131] Taking the supervised learning method to train the first teacher model as an example, the model parameters are learned by minimizing the loss value of the loss function based on the training data and the corresponding annotation labels in the first training data set to obtain the trained first teacher model.

[0132] Taking the self-supervised learning method to train the first teacher model as an example, the self-supervised learning method is to train the first teacher model in an unsupervised or semi-supervised environment, rather than using explicitly labeled data. Among them, semi-supervised learning often uses a portion of real labeled data, in addition to which, it also includes a large amount of unlabeled data. For this part of unlabeled data, pseudo labels can be generated, and then these pseudo labels are used to train the model, which is not limited in this application.

[0133] 203. During the training process of the student model of knowledge distillation, the computer device obtains the knowledge distillation intensity based on the first category of indicators; wherein the knowledge distillation intensity is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first category of indicators includes the training status and model performance of the student model, the temperature parameters in the knowledge distillation process, and at least one of the model training parameters.

[0134] This step involves adaptively adjusting the strength of knowledge distillation. During the knowledge distillation process, the present embodiment dynamically adjusts the strength of knowledge distillation using an adaptive strategy. In this embodiment, the strategy automatically adjusts the strength of knowledge distillation based on the first category of indicators, enabling the student model to obtain appropriate supervisory signals at different stages, thereby improving training efficiency and model performance.

[0135] The knowledge distillation intensity is used to indicate the degree to which the student model has learned the knowledge of the teacher model. It can be used to adjust the strength of knowledge distillation. Generally speaking, the higher the knowledge distillation intensity, the more the student model strives to imitate the behavior of the teacher model, and the more accurate the knowledge transfer.

[0136] Exemplarily, the above-mentioned training status includes the training progress of the student model, loss changes, etc.; the above-mentioned model training parameter can be a learning rate, which is not limited in this application.

[0137] In response to the change in loss, the embodiment of the present application will monitor the first loss (also known as knowledge distillation loss or soft target loss). If the first loss is large, it indicates that the student model has not fully captured the knowledge of the teacher model, and the knowledge distillation intensity can be increased at this time. If the first loss is small, the knowledge distillation intensity can be reduced. In addition, the embodiment of the present application will regularly evaluate the performance of the student model on the validation data set or the test data set. If the performance is improved, the model training can continue with the current knowledge distillation intensity. If the performance stagnates or decreases, the knowledge distillation intensity can be increased to help the model learn better. In addition, the embodiment of the present application will also monitor the progress of model training. Generally, the student model requires a greater knowledge distillation intensity in the early stages of training in order to learn faster. As the training progresses, the knowledge distillation intensity can be gradually reduced to avoid overfitting. In addition, the knowledge distillation intensity can also be controlled by adaptively adjusting the learning rate or temperature parameters.

[0138] In the embodiments of this application, the strength of knowledge distillation affects the knowledge distillation process, primarily by balancing model performance and training efficiency. A stronger knowledge distillation strength typically results in faster learning for the student model, but may also lead to overfitting. An appropriate knowledge distillation strength can balance model performance and generalization capabilities.

[0139] Regarding training speed, a higher knowledge distillation strength results in faster training because the student model learns useful knowledge more quickly. Regarding model size, a higher knowledge distillation strength typically results in a larger student model because it needs to store more knowledge. This can increase memory and computing resource requirements. Regarding generalization, excessive knowledge distillation strength can cause the model to perform poorly on test data because it relies too heavily on the specific training data of the teacher model.

[0140] 204. Under the constraint of knowledge distillation strength, the computer device trains a second deep learning model for performing the target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain a trained student model.

[0141] During the training process of the student model, the output of the teacher model is used as an additional supervisory signal. In terms of model scale, the teacher model is larger than the student model. Therefore, compared with the first deep learning model, the structure of the second deep learning model is relatively simple. In addition, the embodiment of the present application trains the student model based on a supervised learning method. That is, the training data in the second training data set is labeled training data. In addition, the initial parameters of the second deep learning model can be randomly initialized, and this application does not limit this.

[0142] In summary, the model training scheme based on knowledge distillation provided in the embodiment of the present application can automatically adjust the knowledge distillation intensity during the training process of the student model, which enables the student model to obtain appropriate supervision signals at different stages of the knowledge distillation process, thereby improving the training efficiency and model performance of the student model.

[0143] The above briefly introduces the model training scheme provided by the embodiment of the present application. Figure 3 The task processing solution based on the pre-trained language model provided in the embodiment of the present application is introduced in detail.

[0144] Figure 3 This is a flowchart of another model training method based on knowledge distillation provided in an embodiment of the present application. The execution subject of this method is a computer device. Figure 3 The method flow provided in the embodiment of the present application includes the following steps.

[0145] 301. A computer device obtains a first training data set and a second training data set.

[0146] This step can refer to the aforementioned step 201 and will not be repeated here.

[0147] 302. The computer device trains a first deep learning model based on the first training data set to obtain a first teacher model for knowledge distillation.

[0148] This step can refer to the aforementioned step 202 and will not be repeated here.

[0149] 303. During the training process of the student model of knowledge distillation, the computer device obtains the knowledge distillation intensity based on the first category of indicators; wherein the knowledge distillation intensity is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first category of indicators includes the training status and model performance of the student model, the temperature parameters in the knowledge distillation process, and at least one of the model training parameters.

[0150] In a possible implementation, the knowledge distillation intensity in the knowledge distillation process is obtained according to the first type of indicators, including but not limited to the following methods.

[0151] The knowledge distillation strength is obtained according to each item in the first category of indicators, and multiple knowledge distillation strengths that match the number of indicator items included in the first category of indicators are obtained; the influence weight of each item in the first category of indicators on the knowledge distillation strength is obtained; and the current knowledge distillation strength is determined according to the knowledge distillation strength and influence weight corresponding to each item in the first category of indicators.

[0152] For example, the weight of each indicator in the first category on the strength of knowledge distillation can be set as needed. One of the setting principles is that the greater the impact of the indicator on the training process, the greater its influence weight. This application does not limit this.

[0153] 304. Under the constraint of knowledge distillation strength, the computer device trains a second deep learning model for performing the target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain a trained student model.

[0154] In this embodiment, to improve the training efficiency of the student model, efficient training techniques such as gradient accumulation, mixed-precision training, and knowledge distillation rate adjustment are employed. Gradient accumulation reduces the number of updates and computational overhead during training; mixed-precision training balances computational accuracy and resources, increasing training speed; and knowledge distillation rate adjustment dynamically adjusts the speed of knowledge distillation based on the student model's training progress and performance, accelerating the training process.

[0155] The knowledge distillation rate is used to control the speed of knowledge distillation. It determines how closely the student model should imitate the teacher model in each training step. For example, the knowledge distillation rate can be a fixed hyperparameter or an adjustable hyperparameter, which is not limited in this application. Taking the dynamic adjustment of the knowledge distillation rate as an example, the following methods can be used to adjust the knowledge distillation rate:

[0156] During the training process of the student model, the knowledge distillation rate is obtained according to the second type of indicators; wherein the second type of indicators include at least one of the training progress, model performance and model training parameters of the student model.

[0157] It should be noted that, with respect to the training progress, the performance of the student model on the training data, such as changes in the loss function, will be monitored during the knowledge distillation process. If the student model performs poorly in the initial stage of training, the knowledge distillation rate can be increased to learn the knowledge of the teacher model faster. As the training progresses, the knowledge distillation rate can be gradually reduced to avoid overfitting. In addition, the embodiment of the present application will also regularly evaluate the performance of the student model on a validation dataset or a test dataset. If the performance improves, it means that the knowledge distillation rate is suitable for the current training progress. If the performance stagnates or decreases, the knowledge distillation rate can be increased to promote faster learning. In addition, the speed of knowledge distillation can be affected by adjusting the learning rate. Among them, a larger learning rate will result in faster knowledge transfer, and a smaller learning rate will result in slower knowledge transfer.

[0158] Accordingly, the above step 304 can be replaced by "under the constraints of knowledge distillation intensity and knowledge distillation rate, based on the second training data set and the output probability distribution of the first teacher model, training the second deep learning model for performing the target deep learning task to obtain a trained student model."

[0159] In one possible implementation, based on the second training data set and the output probability distribution of the first teacher model, a second deep learning model for performing the target deep learning task is trained to obtain a trained student model, including but not limited to the following methods.

[0160] A first loss function is constructed based on the output probability distribution (soft target) of the first teacher model and the output probability predicted by the second deep learning model. Furthermore, a second loss function (also known as hard target loss) is constructed based on the second training dataset (labeled data, i.e., hard targets) and the output probability predicted by the second deep learning model. The output probability distribution and output probability are obtained under the same temperature parameters. The second loss function is used to measure the difference between the predicted value and the true value of the student model. Subsequently, a first weight for the first loss function and a second weight for the second loss function are obtained. Based on the first and second weights, the first and second loss functions are weighted to obtain a target loss function. Furthermore, the target loss function is iteratively used to obtain a loss value until the training stop condition is met, resulting in a trained student model.

[0161] In another possible implementation, the present embodiment also includes dynamic loss weight adjustment. That is, during the training process of the student model, the weight between the hard target loss and the soft target loss is dynamically adjusted. This can reduce the computational complexity of the student model while maintaining performance.

[0162] Exemplarily, obtaining the first weight of the first loss function and the second weight of the second loss function includes, but is not limited to, adopting the following method.

[0163] Method 1: Periodically obtain the performance changes of the student model on a specified data set; determine the first weight and the second weight based on the obtained performance changes.

[0164] For example, the designated dataset is a validation dataset or a test dataset. In this approach, the student model's performance on the validation dataset or test dataset can be regularly evaluated, for example, by monitoring performance metrics such as accuracy, F1 score, or loss. If model performance remains high, the current weight settings can be maintained. If model performance degrades or stagnates, the weights need to be adjusted, i.e., the loss weights need to be increased.

[0165] Method 2: Determine the first weight and the second weight based on the model training parameters.

[0166] For example, the model training parameter may be a learning rate, which is not limited in this application. In this approach, the loss weight is controlled by adjusting the learning rate. If model performance degrades, the learning rate can be reduced, i.e., the loss weight is increased to slow down the model's parameter updates and adapt it more cautiously to the data. This can prevent the model from falling into a local minimum prematurely.

[0167] Method 3: configuring weight parameters during the model training process; based on the weight parameters, determining the first weight and the second weight during the model training process.

[0168] In this approach, this parameter dynamically adjusts the loss weight between the hard target loss and the soft target loss during training. For example, this parameter is proportional to the degree of performance degradation of the model. Additionally, during training, this parameter can be updated using gradient descent or other optimization algorithms.

[0169] 305. The computer device receives input data that matches the target deep learning task; wherein the input data is at least one of text, image, audio and video; and calls the trained student model to perform the target deep learning task according to the input data.

[0170] It should be noted that the target deep learning tasks include, but are not limited to, audio processing tasks (such as speech recognition), image processing tasks (such as medical image analysis), text processing tasks, or video processing tasks, and this application does not limit this. In the case of different tasks, the type of input data input into the trained student model is also different.

[0171] In another possible implementation, during the student model training process, the intermediate-layer feature representations of the teacher and student models can be used as additional supervisory signals to guide the student model to learn more effective feature representations. By combining hierarchical knowledge distillation with knowledge distillation intensity adjustment, the performance and generalization ability of the student model can be further improved.

[0172] Accordingly, the above step 304 can also be replaced by "obtaining the intermediate-layer feature representation of the first teacher model and the intermediate-layer feature representation of the second deep learning model; based on the second training data set, the output probability distribution and intermediate-layer feature representation of the first teacher model, and the intermediate-layer feature representation of the second deep learning model, training the second deep learning model to obtain a student model."

[0173] In another possible implementation, embodiments of the present application can also adjust the knowledge distillation intensity based on meta-learning. For example, a meta-learning model can be trained to predict the optimal knowledge distillation intensity, thereby achieving more efficient knowledge transfer across different tasks and datasets. The following describes how to adjust the knowledge distillation intensity based on meta-learning.

[0174] First, it is necessary to construct a meta-learning model, which can be a deep neural network, and this application does not limit this. Among them, the input of the trained meta-learning model is the task characteristics of the current task or the data distribution of the data set, and the output is the predicted knowledge distillation intensity. In addition, in order to train the meta-learning model, it is also necessary to construct a meta-learning task. Among them, the meta-learning task includes a support set and a query set. The support set includes multiple small tasks, each of which is set with different knowledge distillation hyperparameters (corresponding to the first type of indicators mentioned above). The query set is used to evaluate model performance, including task samples, and does not involve the setting of knowledge distillation hyperparameters. Next, the support set is used to train the meta-learning model. During the training process, the meta-learning model learns how to predict the optimal knowledge distillation intensity by observing the knowledge distillation hyperparameter settings of multiple small tasks. After the meta-learning model is trained, the trained meta-learning model can be used to predict the optimal knowledge distillation intensity.

[0175] Based on the above description, the method provided in the embodiment of the present application also includes: first, constructing a meta-learning task, which includes a support set and a query set; wherein the support set includes multiple subtasks, each subtask is configured with a different first-category indicator; the query set includes task samples for training subtasks; then, training the meta-learning model based on the support set and performing model performance evaluation based on the query set to obtain a trained meta-learning model; in the training process of the aforementioned student model, calling the trained meta-learning model, and obtaining the knowledge distillation intensity in the knowledge distillation process based on the task characteristics of the target deep learning task or the data distribution of the second training data set. For how task characteristics and data distribution affect the knowledge distillation intensity, please refer to the subsequent task-adapted knowledge distillation section.

[0176] In another possible implementation, in addition to introducing a single teacher model for knowledge distillation, multiple teacher models can be introduced, distilling the knowledge of multiple teacher models into the student model. This approach can fully leverage the advantages of multiple teacher models and improve the generalization ability of the student model.

[0177] In addition, the embodiment of the present application dynamically adjusts the weights of each teacher model in the knowledge distillation process. For example, if the performance of the student model decreases during training, the weight of the high-performance teacher model can be increased to help the student model restore performance. Among them, the weights of the teacher models in the knowledge distillation process are used to calculate the loss function. In detail, each teacher model provides a soft target for the student model, and these soft targets are compared with the predictions of the student model. The above-mentioned weights are used to weight the soft targets of different teacher models to generate the final loss function. For each teacher model, the higher the weight of the teacher model, the more influence its soft target has on the training of the student model.

[0178] Based on the above description, the method provided in the embodiment of the present application also includes: obtaining multiple training data sets; training multiple deep learning models based on the multiple training data sets to obtain multiple second teacher models.

[0179] Accordingly, the above step 304 can also be replaced by "according to the training progress and model performance of the student model, obtain the weight of each model in the first teacher model and multiple second teacher models; based on the weight of each model, weight the output probability of the first teacher model and multiple second teacher models to obtain the fused output probability distribution; based on the second training data set and the fused output probability distribution, train the second deep learning model to obtain a trained student model."

[0180] For example, the first training dataset and each of the multiple training datasets correspond to different deep learning tasks. That is, the outputs of the teacher models for multiple deep learning tasks can be used as supervisory signals to simultaneously train the student model. This can improve the generalization ability of the student model in multi-task scenarios.

[0181] In another possible implementation, in the context of incremental learning, incremental knowledge distillation can be used to enable the student model to gradually learn the knowledge of new tasks while retaining the knowledge of previously learned tasks. For example, implementing incremental knowledge distillation includes but is not limited to the following steps.

[0182] 1. Prepare new data or tasks. These data may belong to new fields, new tasks, or new samples, which are not limited in this application.

[0183] 2. Update the teacher model. If a teacher model has been built, you can consider using it to initialize new data or tasks. For example, this can be achieved by fine-tuning the teacher model to adapt to the new data or task.

[0184] 3. Knowledge distillation: Using knowledge distillation technology, the knowledge of the updated teacher model is transferred to the student model. For example, this step usually involves the following steps.

[0185] 3-1. Define a loss function to minimize the difference between the output of the student model and the output of the teacher model. Exemplarily, the defined loss function includes soft target loss and hard target loss.

[0186] 3-2. Perform knowledge distillation on new data or tasks so that the student model can learn the knowledge of the teacher model.

[0187] 3-3. After knowledge distillation is completed, the student model can be further incrementally trained to adapt it to new data or tasks. This can be fine-tuning of the student model to ensure its performance.

[0188] 3-4. Evaluate the updated student model and deploy it in real-world applications to cope with new data or tasks.

[0189] Based on the above description, the method provided in the embodiment of the present application further includes: in the case of newly added training data, fine-tuning the first teacher model based on the newly added training data to obtain an updated first teacher model; responsively, the embodiment of the present application further trains a second deep learning model based on the second training data set and the output probability distribution of the updated first teacher model to obtain a trained student model. In addition, the embodiment of the present application further fine-tunes the trained student model based on the newly added training data to obtain an updated student model.

[0190] In another possible implementation, embodiments of the present application allow the student model to obtain the supervisory signal of the teacher model in real time during training, thereby avoiding the overhead of pre-calculating and storing the teacher model output. This approach can achieve more efficient knowledge transfer and model training.

[0191] In another possible implementation, the embodiment of the present application may also utilize model compression technology to further reduce the computational complexity and storage overhead of the model while maintaining high performance.

[0192] In another possible implementation, to further improve training efficiency, embodiments of the present application can also parallelize the training process of the teacher model and the student model. By using distributed computing and communication technologies, large-scale parallel training and knowledge distillation can be achieved, thereby significantly shortening the model training time.

[0193] In another possible implementation, when labeled data is scarce, knowledge distillation can be performed by combining semi-supervised and unsupervised learning methods. By leveraging unlabeled or partially labeled data and introducing additional self-supervisory signals, the generalization ability and robustness of the student model can be enhanced.

[0194] In another possible implementation, in a multimodal learning scenario, embodiments of the present application support obtaining knowledge from teacher models of different modalities and integrating it into a student model. This approach can fully utilize multimodal information, thereby improving the performance and generalization ability of the student model. Exemplarily, integrating knowledge obtained from teacher models of different modalities into the student model includes, but is not limited to, the following steps.

[0195] a. Feature fusion: Teacher models of different modalities may have learned feature representations of different modalities. Therefore, before fusing knowledge, it is possible to consider fusing these feature representations. Exemplarily, feature fusion methods include but are not limited to concatenation, weighted summation, or concatenation, which are not limited in this application. In addition, after generating the fused multimodal feature representation, it can be used as input to the student model.

[0196] b. Knowledge distillation: This step uses knowledge distillation technology to transfer the knowledge of teacher models of different modalities to the student model.

[0197] c. Weight adjustment: This embodiment of the application supports adjusting the weights of teacher models of different modalities based on task and performance requirements. Among them, better modalities can be assigned higher weights to ensure that more knowledge of their corresponding teacher models is transferred to the student model.

[0198] d. Fusion strategy: This step is used to integrate knowledge from different modalities. For example, the fusion strategy can be hard fusion (using only the output of one teacher model) or soft fusion (weighted fusion of the outputs of multiple teacher models), etc., which is not limited in this application.

[0199] e. Training and tuning: This step is used to train and tune the student model. Exemplarily, this step includes end-to-end training of the entire system to maximize the use of knowledge from different modalities.

[0200] Another possible implementation involves performing knowledge distillation between domains. This involves transferring knowledge from a teacher model in one domain to a student model in another. Domain adversarial training (DAT) allows for effective cross-domain knowledge transfer, improving the generalization and robustness of the student model. DAT is a technique used for domain adaptation and transfer learning, designed to help models achieve better generalization across domains or distributions. This approach is often used to address the following problems:

[0201] Domain Adaptation: If a model is trained on one domain and needs to be tested or applied on another domain, it may face the problem of domain shift. DAT aims to mitigate domain shift so that the model can perform well on different domains.

[0202] Domain transfer: If a model is trained on one domain and its knowledge needs to be transferred to a model in another domain, DAT can help the model better adapt to the target domain.

[0203] In summary, the core idea of DAT is to introduce a domain classifier or domain discriminator, whose task is to identify whether the features output by the model belong to the source domain or the target domain. At the same time, the model should also deceive the domain classifier as much as possible so that it cannot accurately distinguish features from different domains.

[0204] In another possible implementation, task-adaptive knowledge distillation strategies can be designed for different tasks and datasets. For example, by analyzing task characteristics and data distribution, factors such as the knowledge distillation intensity, loss function, or optimization algorithm can be adjusted according to actual needs to achieve more efficient knowledge transfer and model training. In other words, a task-adaptive knowledge distillation strategy refers to adjusting the knowledge distillation strategy according to different tasks and datasets to achieve more efficient knowledge transfer and model training. This means that the knowledge distillation intensity, loss function, or optimization algorithm in the knowledge distillation process can be customized.

[0205] Taking the strength of knowledge distillation as an example, for task characteristics, if the task is complex, a stronger knowledge distillation strength may be required to ensure that the student model can fully capture the knowledge of the teacher model. Regarding data distribution, if the data distribution is uneven in certain fields, the knowledge distillation strength can be adjusted according to the data density of different fields to meet the needs of different fields. For practical needs, considering the actual application needs of the task, the knowledge distillation strength may need to be adjusted according to performance requirements and resource constraints. In some cases, training speed may be more important than model performance, so the knowledge distillation strength can be reduced.

[0206] Taking loss functions as an example, different types of loss functions can be selected based on the nature of the task. For example, the cross-entropy loss function can be used for classification tasks, while the mean squared error loss function can be used for regression tasks. Regarding data distribution, variations of loss functions, such as weighted loss functions, can be used based on the data distribution. For practical needs, considering the actual application requirements of the task, it may be necessary to weigh the weights of different loss terms to achieve a balance between model performance and training speed.

[0207] Taking optimization algorithms as an example, different optimization algorithms can be selected based on task complexity and optimization difficulty, such as stochastic gradient descent, Adam, or L-BFGS. Regarding data distribution, which can affect the convergence speed of gradient descent, appropriate optimization algorithms and hyperparameters can be selected based on the characteristics of the data distribution. For practical needs, the selection of an optimization algorithm can also take into account the availability of computing resources and training time constraints.

[0208] In summary, the knowledge distillation-based model training solution provided in the embodiments of this application can effectively improve the training efficiency and performance of the student model. In addition, the embodiments of this application also introduce a variety of extended optimization technologies, such as efficient training technology, online knowledge distillation, multi-model knowledge distillation, incremental knowledge distillation, and the combination of model compression and distillation, which further improve the training efficiency and performance of the student model.

[0209] In addition, the embodiments of the present application have broad application prospects, including but not limited to the following aspects.

[0210] Edge computing devices: This solution can reduce the computational complexity and storage requirements of the model, enabling student models to run efficiently on resource-constrained edge computing devices (such as smartphones, embedded devices, etc.), thereby expanding the scope of deep learning applications.

[0211] Real-time analysis and efficient reasoning: By improving the training efficiency and performance of the student model, the student model has higher performance in real-time analysis and efficient reasoning tasks, meeting the real-time application requirements of low latency and high throughput.

[0212] Personalized and customized applications: By introducing technologies such as privately deployed teacher models, incremental knowledge distillation, and task-adaptive knowledge distillation, this solution can achieve personalized and customized training of student models to meet the specific needs of different users, scenarios, and applications.

[0213] General and domain-specific applications: This solution is applicable to a variety of general and domain-specific deep learning tasks and can significantly improve the performance and generalization ability of the model.

[0214] Multimodal and cross-domain learning: Through multimodal knowledge distillation and cross-domain knowledge distillation, this solution can achieve more efficient multimodal information fusion and cross-domain knowledge transfer, thereby expanding the potential of deep learning in complex scenarios and cross-domain applications.

[0215] Semi-supervised and unsupervised learning: Combining semi-supervised and unsupervised knowledge distillation techniques, this solution can use unlabeled data or partially labeled data for model training when labeled data is scarce, improving the model's generalization ability and robustness.

[0216] It should be noted that the embodiments of the present application can be flexibly applied in scenarios and are also applicable to various deep learning models, and can be appropriately adjusted and improved according to actual needs.

[0217] Figure 4 This is a schematic diagram of the structure of a model training device based on knowledge distillation provided in an embodiment of the present application. Figure 4 , the device includes the following modules.

[0218] A first acquisition module 401 is configured to acquire a first training data set and a second training data set;

[0219] A first training module 402 is configured to train a first deep learning model based on the first training data set to obtain a first teacher model;

[0220] The second acquisition module 403 is configured to acquire the knowledge distillation strength according to the first type of indicators during the training process of the student model; wherein the knowledge distillation strength is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicators includes at least one of the training status and model performance of the student model, the temperature parameter during the knowledge distillation process, and the model training parameters;

[0221] The second training module 404 is configured to train a second deep learning model for performing the target deep learning task based on the second training data set and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model.

[0222] The model training scheme based on knowledge distillation provided in the embodiment of the present application can automatically adjust the knowledge distillation intensity during the training process of the student model, which enables the student model to obtain appropriate supervision signals at different stages of the knowledge distillation process, thereby improving the training efficiency and model performance of the student model.

[0223] In one possible implementation, the first teacher model is used to perform multiple deep learning tasks; the training data in the second training dataset is derived from the first training dataset; the training data in the first training dataset is unlabeled; the training data in the second training dataset is labeled; or, the first training dataset and the second training dataset are the same dataset that matches the target deep learning task.

[0224] In one possible implementation, the first acquisition module is further configured to acquire a knowledge distillation rate according to a second type of indicator during the training of the student model; wherein the knowledge distillation rate is used to control the speed of knowledge distillation; and the second type of indicator includes at least one of the training progress, model performance, and model training parameters of the student model;

[0225] The second training module is configured to train the second deep learning model based on the second training data set and the output probability distribution of the first teacher model under the constraints of the knowledge distillation intensity and the knowledge distillation rate to obtain the student model.

[0226] In a possible implementation, the second training module is configured to:

[0227] Under the constraint of the knowledge distillation strength, constructing a first loss function based on the output probability distribution of the first teacher model and the output probability predicted by the second deep learning model;

[0228] Constructing a second loss function based on the second training data set and the output probability predicted by the second deep learning model;

[0229] Obtaining a first weight of the first loss function and a second weight of the second loss function;

[0230] Weighting the first loss function and the second loss function based on the first weight and the second weight to obtain a target loss function;

[0231] The loss value is obtained by iteratively obtaining the target loss function until the training stop condition is met, thereby obtaining the student model.

[0232] In a possible implementation, the second training module is configured to:

[0233] Periodically obtaining performance changes of the student model on a specified data set; determining the first weight and the second weight based on the performance changes; wherein the specified data set is a validation data set or a test data set; or,

[0234] Determining the first weight and the second weight based on the model training parameters; or,

[0235] Configure weight parameters during the model training process; and determine the first weight and the second weight during the model training process based on the weight parameters.

[0236] In a possible implementation, the training status includes the training progress and loss change of the student model; and the second acquisition module is configured to:

[0237] Obtaining a knowledge distillation strength according to each item in the first category of indicators, and obtaining a plurality of knowledge distillation strengths that match the number of indicator items included in the first category of indicators;

[0238] Obtain the influence weight of each of the first category of indicators on the strength of knowledge distillation;

[0239] Determine the current knowledge distillation intensity based on the knowledge distillation intensity and influence weight corresponding to each item in the first category of indicators.

[0240] In a possible implementation, the second training module is configured to:

[0241] Obtaining an intermediate layer feature representation of the first teacher model and an intermediate layer feature representation of the second deep learning model;

[0242] Under the constraint of the knowledge distillation strength, the second deep learning model is trained based on the second training data set, the output probability distribution and intermediate layer feature representation of the first teacher model, and the intermediate layer feature representation of the second deep learning model to obtain the student model.

[0243] In a possible implementation, the apparatus further includes: a third acquisition module; the third acquisition module is configured to:

[0244] Constructing a meta-learning task, the meta-learning task comprising a support set and a query set; wherein the support set comprises a plurality of subtasks, each subtask being configured with a different first-category indicator; and the query set comprises task samples for training the subtasks;

[0245] Training the meta-learning model based on the support set and performing model performance evaluation based on the query set to obtain a trained meta-learning model;

[0246] During the training process of the student model, the trained meta-learning model is called to obtain the knowledge distillation intensity in the knowledge distillation process based on the task characteristics of the target deep learning task or the data distribution of the second training data set.

[0247] In a possible implementation, the first acquisition module is further configured to acquire multiple training data sets;

[0248] The first training module is further configured to train multiple deep learning models based on the multiple training data sets to obtain multiple second teacher models;

[0249] The second training module is configured to:

[0250] Obtaining a weight for each of the first teacher model and the plurality of second teacher models according to the training progress and model performance of the student model;

[0251] Based on the obtained weight of each model, weight the output probabilities of the first teacher model and the multiple second teacher models to obtain a fused output probability distribution;

[0252] Based on the second training data set and the fused output probability distribution, the second deep learning model is trained to obtain the student model.

[0253] In a possible implementation, the first training module is further configured to, when new training data is added, fine-tune the first teacher model based on the new training data to obtain an updated first teacher model;

[0254] The second training module is further configured to train the second deep learning model based on the second training data set and the output probability distribution of the updated first teacher model to obtain the student model.

[0255] In a possible implementation, the second training module is further configured to fine-tune the student model based on the newly added training data to obtain an updated student model.

[0256] In a possible implementation, the apparatus further includes: a processing module;

[0257] The processing module is configured to receive input data matching the target deep learning task; wherein the input data is at least one of text, image, audio and video; and call the student model to perform the target deep learning task according to the input data.

[0258] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0259] It should be noted that the model training device based on knowledge distillation provided in the above embodiment only uses the division of the above functional modules as an example when performing model training based on knowledge distillation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device based on knowledge distillation provided in the above embodiment and the model training method embodiment based on knowledge distillation belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0260] Figure 5 It is a structural diagram of a computer device 500 provided in an embodiment of the present application.

[0261] Typically, the computer device 500 includes a processor 501 and a memory 502 .

[0262] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In one possible implementation, the processor 501 may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In one possible implementation, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0263] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In one possible implementation, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one program code, which is used to be executed by the processor 501 to implement the model training method based on knowledge distillation provided in the method embodiment of the present application.

[0264] In one possible implementation, computer device 500 further includes a peripheral device interface 503 and at least one peripheral device. Processor 501, memory 502, and peripheral device interface 503 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 503 via a bus, signal lines, or circuit boards. The peripheral device may include at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, a positioning assembly 508, and a power supply 509.

[0265] The peripheral device interface 503 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 501 and the memory 502. In one possible implementation, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in another possible implementation, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 can be implemented on separate chips or circuit boards, which is not limited in this application.

[0266] The RF circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In one possible implementation, the RF circuit 504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 504 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In one possible implementation, the RF circuit 504 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0267] The display screen 505 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 505 is a touch screen, it is also capable of collecting touch signals on or above the surface of the display screen 505. These touch signals can be input as control signals to the processor 501 for processing. In this case, the display screen 505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In one possible implementation, there can be one display screen 505, located on the front panel of the computer device 500. In another possible implementation, there can be at least two display screens 505, located on different surfaces of the computer device 500 or in a foldable design. In another possible implementation, the display screen 505 can be a flexible display, located on a curved or foldable surface of the computer device 500. Furthermore, the display screen 505 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0268] The camera assembly 506 is used to capture images or videos. In one possible implementation, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In one possible implementation, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In another possible implementation, the camera assembly 506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0269] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 501 for processing, or input into the radio frequency circuit 504 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the computer device 500. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as distance measurement. In one possible implementation, the audio circuit 507 may also include a headphone jack.

[0270] The positioning component 508 is used to locate the current geographic location of the computer device 500 to implement navigation or LBS (Location Based Service). The positioning component 508 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0271] Power supply 509 is used to power various components in computer device 500. Power supply 509 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0272] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0273] Figure 6: It is a structural diagram of another computer device 600 provided in an embodiment of the present application. The computer 600 can be a server. The computer device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 601 and one or more memories 602, wherein the memory 602 stores at least one program code, and the at least one program code is loaded and executed by the processor 601 to implement the model training method based on knowledge distillation provided by the above-mentioned various method embodiments. Of course, the computer device 600 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device 600 may also include other components for realizing the functions of the device, which will not be described here.

[0274] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code. The program code can be executed by a processor in a computer device to perform the model training method based on knowledge distillation in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0275] In an exemplary embodiment, a computer program product or computer program is also provided, which includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the above-mentioned model training method based on knowledge distillation.

[0276] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0277] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model training method based on knowledge distillation, characterized in that: The method comprises: Obtain a first training data set and a second training data set; Training a first deep learning model based on the first training data set to obtain a first teacher model; During the training of the student model, obtaining the knowledge distillation strength according to a first type of indicator; wherein the knowledge distillation strength is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicator includes at least one of the training status and model performance of the student model, a temperature parameter during the knowledge distillation process, and a model training parameter; Under the constraint of the knowledge distillation strength, based on the second training dataset and the output probability distribution of the first teacher model, training a second deep learning model for performing the target deep learning task to obtain the student model; Receive input data matching the target deep learning task; wherein the input data is at least one of text, image, audio, and video; The student model is called to perform the target deep learning task according to the input data.

2. The method according to claim 1, characterized in that The first teacher model is used to perform multiple deep learning tasks; the training data in the second training data set is derived from the first training data set; The training data in the first training data set are not labeled; the training data in the second training data set are labeled; or, The first training data set and the second training data set are the same data set that matches the target deep learning task.

3. The method according to claim 1, characterized in that The method further comprises: During the training of the student model, a knowledge distillation rate is obtained according to a second type of indicator; wherein the knowledge distillation rate is used to control the speed of knowledge distillation; the second type of indicator includes at least one of the training progress, model performance, and model training parameters of the student model; The step of training a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model includes: Under the constraints of the knowledge distillation strength and the knowledge distillation rate, the second deep learning model is trained based on the second training data set and the output probability distribution of the first teacher model to obtain the student model.

4. The method according to claim 1, wherein The step of training a second deep learning model for performing a target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain the student model includes: Constructing a first loss function based on the output probability distribution of the first teacher model and the output probability predicted by the second deep learning model; Constructing a second loss function based on the second training data set and the output probability predicted by the second deep learning model; Obtaining a first weight of the first loss function and a second weight of the second loss function; Weighting the first loss function and the second loss function based on the first weight and the second weight to obtain a target loss function; The loss value is obtained by iteratively obtaining the target loss function until the training stop condition is met, thereby obtaining the student model.

5. The method according to claim 4, characterized in that The obtaining of a first weight of the first loss function and a second weight of the second loss function includes: Periodically obtaining performance changes of the student model on a specified data set; determining the first weight and the second weight based on the performance changes; wherein the specified data set is a validation data set or a test data set; or, Determining the first weight and the second weight based on the model training parameters; or, Configure weight parameters during the model training process; and determine the first weight and the second weight during the model training process based on the weight parameters.

6. The method according to claim 1, characterized in that The training status includes the training progress and loss change of the student model; the knowledge distillation intensity obtained in the knowledge distillation process according to the first type of indicators includes: Obtaining a knowledge distillation strength according to each item in the first category of indicators, and obtaining a plurality of knowledge distillation strengths that match the number of indicator items included in the first category of indicators; Obtain the influence weight of each of the first category of indicators on the strength of knowledge distillation; Determine the current knowledge distillation intensity based on the knowledge distillation intensity and influence weight corresponding to each item in the first category of indicators.

7. The method according to claim 1, characterized in that The step of training a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model includes: Obtaining an intermediate layer feature representation of the first teacher model and an intermediate layer feature representation of the second deep learning model; Under the constraint of the knowledge distillation strength, the second deep learning model is trained based on the second training data set, the output probability distribution and intermediate layer feature representation of the first teacher model, and the intermediate layer feature representation of the second deep learning model to obtain the student model.

8. The method according to claim 1, characterized in that The method further comprises: Constructing a meta-learning task, the meta-learning task comprising a support set and a query set; wherein the support set comprises a plurality of subtasks, each subtask being configured with a different first-category indicator; and the query set comprises task samples for training the subtasks; Training a meta-learning model based on the support set and evaluating model performance based on the query set to obtain a trained meta-learning model; During the training process of the student model, the trained meta-learning model is called to obtain the knowledge distillation intensity in the knowledge distillation process based on the task characteristics of the target deep learning task or the data distribution of the second training data set.

9. The method according to claim 1, characterized in that The method further comprises: Obtain multiple training data sets; Training multiple deep learning models based on the multiple training data sets to obtain multiple second teacher models; The step of training a second deep learning model for performing a target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain the student model includes: Obtaining a weight for each of the first teacher model and the plurality of second teacher models according to the training progress and model performance of the student model; Based on the obtained weight of each model, weight the output probabilities of the first teacher model and the multiple second teacher models to obtain a fused output probability distribution; Based on the second training data set and the fused output probability distribution, the second deep learning model is trained to obtain the student model.

10. The method according to claim 1, characterized in that The method further comprises: In the case of newly added training data, fine-tuning the first teacher model based on the newly added training data to obtain an updated first teacher model; The step of training a second deep learning model for performing a target deep learning task based on the second training data set and the output probability distribution of the first teacher model to obtain the student model includes: Based on the second training data set and the output probability distribution of the updated first teacher model, the second deep learning model is trained to obtain the student model.

11. The method according to claim 10, characterized in that The method further comprises: Based on the newly added training data, the trained student model is fine-tuned to obtain an updated student model.

12. A model training device based on knowledge distillation, characterized in that: The device comprises: A first acquisition module is configured to acquire a first training data set and a second training data set; A first training module is configured to train a first deep learning model based on the first training data set to obtain a first teacher model; a second acquisition module configured to acquire, during the training process of the student model, knowledge distillation strength based on a first type of indicator; wherein the knowledge distillation strength is used to reflect the degree of knowledge transfer during the knowledge distillation process; the first type of indicator includes at least one of the training state and model performance of the student model, a temperature parameter during the knowledge distillation process, and a model training parameter; A second training module is configured to train a second deep learning model for performing a target deep learning task based on the second training dataset and the output probability distribution of the first teacher model under the constraint of the knowledge distillation strength to obtain the student model; A processing module is configured to receive input data matching the target deep learning task; wherein the input data is at least one of text, image, audio and video; and call the student model to perform the target deep learning task according to the input data.

13. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the model training method based on knowledge distillation as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the model training method based on knowledge distillation as described in any one of claims 1 to 11.

Images