Model training method, device, electronic device and storage medium

Abstract

The present specification provides a model training method, device, electronic device and storage medium, the method comprising: inputting training data into a teacher model, an auxiliary model and a student model respectively, wherein the teacher model is trained. The teacher model extracts features from the training data and transfers the extracted teacher intermediate features to the auxiliary model. The auxiliary model extracts features from the training data and transfers the extracted auxiliary intermediate features to the student model; and, the auxiliary model is trained based on a first loss function, wherein the first loss function includes a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features. The student model extracts features from the training data to obtain student intermediate features; and, the student model is trained based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to model training methods, devices, electronic devices and storage media. Background Art

[0002] With its huge model capacity and large-scale high-quality training data, large visual language models have demonstrated strong performance in multimodal understanding and generation tasks, promoting the deep integration of computer vision and natural language processing. However, the high computing requirements of these models make them face huge challenges in practical applications. Due to limited computing resources, they are usually difficult to deploy directly, especially in scenarios with high requirements for real-time and efficiency. Therefore, how to reduce resource overhead while maintaining model performance has become a key issue in the implementation of large visual language models.

[0003] Knowledge Distillation (KD) has become an effective solution for deploying large models in resource-constrained environments. By transferring the knowledge of a large-scale parameter model (teacher model) to a smaller and more efficient model (student model), the student model can imitate the performance of the teacher model with significantly reduced resource usage, making it deployable in resource-constrained environments.

[0004] Traditional knowledge distillation technology usually directly transfers the knowledge of the teacher model to the student model. However, due to the small parameters and limited learning ability of the student model, the traditional teacher-student model may not be able to fully learn the rich multi-level features of the teacher model. As a result, the student model trained based on traditional knowledge distillation technology can reduce resource overhead, but it is difficult to reproduce the model performance of the teacher model. Summary of the invention

[0005] To overcome the problems existing in the related art, this specification provides a model training method, device, electronic device and storage medium.

[0006] According to a first aspect of an embodiment of this specification, a model training method is provided, the method comprising:

[0007] Inputting training data into a teacher model, an auxiliary model and a student model respectively; the teacher model is trained;

[0008] The teacher model extracts features from the training data and transmits the extracted teacher intermediate features to the auxiliary model;

[0009] The auxiliary model extracts features from the training data and transfers the extracted auxiliary intermediate features to the student model; and, training the auxiliary model based on a first loss function, the first loss function comprising a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features;

[0010] The student model extracts features from the training data to obtain student intermediate features; and the student model is trained based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

[0011] According to a second aspect of an embodiment of this specification, a model training device is provided, the device comprising:

[0012] A training data input module, used to input training data into the teacher model, the auxiliary model and the student model respectively; the teacher model is trained;

[0013] A teacher intermediate feature extraction module, used for the teacher model to extract features from training data and transfer the extracted teacher intermediate features to the auxiliary model;

[0014] an auxiliary model training module, configured for the auxiliary model to extract features from the training data and transfer the extracted auxiliary intermediate features to the student model; and, training the auxiliary model based on a first loss function, wherein the first loss function includes a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features;

[0015] A student model training module is used for the student model to extract features from the training data to obtain student intermediate features; and to train the student model based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

[0016] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect are implemented.

[0017] According to a fourth aspect of the embodiments of this specification, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0018] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0019] In the embodiments of this specification, when using the teacher model to perform knowledge distillation on the student model, an auxiliary model is introduced as an intermediary, and the first loss function is used to train the auxiliary model. The first loss function includes the first feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model from the training data and the teacher intermediate features extracted by the teacher model from the training data, so that the knowledge extracted by the teacher model from the training data can be distilled into the auxiliary model. The second loss function is used to train the student model, and the second loss function includes the second feature distillation loss between the student intermediate features extracted by the student model from the training data and the auxiliary intermediate features, so that the knowledge extracted by the auxiliary model from the teacher model can be further distilled into the student model. It can be seen that this scheme distills the knowledge of the teacher model to the student model step by step through the auxiliary model, reduces the learning pressure of the student model, and helps the student model to get closer to the model performance of the teacher model under conditions of limited resources.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0022] Figure 1 It is a flowchart of a model training method shown in this specification according to an exemplary embodiment.

[0023] Figure 2 It is a schematic diagram of a model training process shown in this specification according to an exemplary embodiment.

[0024] Figure 3 is a schematic diagram of a different distillation framework shown in this specification according to an exemplary embodiment.

[0025] Figure 4 It is a schematic diagram of a teacher-assistant-student framework A shown in this specification according to an exemplary embodiment.

[0026] Figure 5 It is a schematic diagram of a teacher-assistant-student framework B shown in this specification according to an exemplary embodiment.

[0027] Figure 6 This specification is based on a first loss function according to an exemplary embodiment. Schematic diagram of the training auxiliary model Ai.

[0028] Figure 7 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification.

[0029] Figure 8 It is a block diagram of a model training device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] With its huge model capacity and large-scale high-quality training data, large visual language models have demonstrated strong performance in multimodal understanding and generation tasks, promoting the deep integration of computer vision and natural language processing. However, the high computing requirements of these models make them face huge challenges in practical applications. Due to limited computing resources, they are usually difficult to deploy directly, especially in scenarios with high requirements for real-time and efficiency. Therefore, how to reduce resource overhead while maintaining model performance has become a key issue in the implementation of large visual language models.

[0031] Knowledge Distillation (KD) has become an effective solution for deploying large models in resource-constrained environments. By transferring the knowledge of a large-scale parameter model (teacher model) to a smaller and more efficient model (student model), the student model can imitate the performance of the teacher model with significantly reduced resource usage, making it deployable in resource-constrained environments.

[0032] Traditional knowledge distillation technology usually directly transfers the knowledge of the teacher model to the student model. However, due to the small parameters and limited learning ability of the student model, the traditional teacher-student model may not be able to fully learn the rich multi-level features of the teacher model. As a result, the student model trained based on traditional knowledge distillation technology can reduce resource overhead, but it is difficult to reproduce the model performance of the teacher model.

[0033] In response to the above technical problems, this specification provides a model training method to achieve a model performance of a trained student model that is closer to that of a teacher model under limited computing resources.

[0034] Next, the embodiments of this specification are described in detail.

[0035] like Figure 1 As shown, Figure 1 1 is a flowchart of a model training method according to an exemplary embodiment of the present specification, including steps 101-104:

[0036] Step 101: input the training data into the teacher model, auxiliary model and student model respectively; the teacher model is trained.

[0037] Step 102: The teacher model extracts features from the training data and transfers the extracted teacher intermediate features to the auxiliary model.

[0038] Step 103: the auxiliary model performs feature extraction on the training data and transfers the extracted auxiliary intermediate features to the student model; and, trains the auxiliary model based on a first loss function, wherein the first loss function includes a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features.

[0039] Step 104: the student model performs feature extraction on the training data to obtain student intermediate features; and, trains the student model based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

[0040] like Figure 2 As shown, Figure 2 2 is a schematic diagram of a model training process according to an exemplary embodiment of the present specification. The teacher model 21 may be a trained model, the auxiliary model 22 and the student model 23 may be untrained models, and the teacher model 21 is used to train the auxiliary model 22 and the student model 23 respectively.

[0041] The training data 20 are input to the teacher model 21, the auxiliary model 22 and the student model 23 respectively. The teacher model 21 extracts features from the training data 20 and transmits the extracted teacher intermediate features to the auxiliary model 22.

[0042] The auxiliary model 22 performs feature extraction on the training data 20 and transfers the extracted auxiliary intermediate features to the student model 23. The first loss function for training the auxiliary model 22 may include a first feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model and the corresponding teacher intermediate features extracted by the teacher model. For example, the first feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model 22 and the teacher intermediate features extracted by the teacher model 21 may be measured by distance loss. The distance loss may be calculated using distance loss functions such as mean square error and mean absolute error.

[0043] The student model 23 performs feature extraction on the training data 20 to obtain the student intermediate features. The second loss function used to train the student model 23 may include a second feature distillation loss between the student intermediate features extracted by the student model 23 and the auxiliary intermediate features. For example, the distance loss may be used to measure the second feature distillation loss between the student intermediate features extracted by the student model 23 and the auxiliary intermediate features extracted by the auxiliary model 22. The distance loss may also be calculated using distance loss functions such as mean square error and mean absolute error.

[0044] Based on this, when using the teacher model to perform knowledge distillation on the student model, an auxiliary model is introduced as an intermediary, and the first loss function is used to train the auxiliary model. The first loss function includes the first feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model from the training data and the teacher intermediate features extracted by the teacher model from the training data, so that the knowledge extracted by the teacher model from the training data can be distilled into the auxiliary model. The second loss function is used to train the student model. The second loss function includes the second feature distillation loss between the student intermediate features extracted by the student model from the training data and the auxiliary intermediate features, so that the knowledge extracted by the auxiliary model from the teacher model can be further distilled into the student model. It can be seen that this scheme distills the knowledge of the teacher model to the student model step by step through the auxiliary model, reduces the learning pressure of the student model, and helps the student model to get closer to the model performance of the teacher model under limited resource conditions.

[0045] This specification does not limit the specific content of the training data, which depends on the training task. In addition, the knowledge distillation technology can be applied to the training of deep learning models in various application fields. This specification does not limit the selection of teacher models, auxiliary models and student models, which can be determined according to the specific application field. For example, in a large visual language model, the training data can be multimodal data (including images, videos and text labels), or pure visual data or pure text data. The teacher model, auxiliary model and student model can be different types of models used to process similar tasks but with different parameter amounts. For example, in computer vision tasks, the teacher model can be a deep convolutional neural network (such as ResNet (Residual Network)-152, with a parameter amount of about 60M); the auxiliary model can be a more efficient visual transformer model (such as ViT (Vision Transformer)-Small, with a parameter amount of about 22M); and the student model can be a small CNN (Convolutional Neural Network) model (such as MobileNet (Mobile Neural Network), with a parameter amount of about 4M). Of course, the teacher model, auxiliary model, and student model can also be different variants of the same model, and the number of parameters of different variants decreases in turn. For example, taking the CLIP (Contrastive Language-Image Pretraining) model as an example, the teacher model can be a CLIP model with a ViT-L14 structure for the visual encoder, with a parameter number of about 305M; the auxiliary model can be a CLIP model with a ViT-B32 structure for the visual encoder, with a parameter number of about 151M; the student model can be a CLIP model with a ViT-S16 structure for the visual encoder, with a parameter number of about 50M.

[0046] like Figure 3 As shown, Figure 3 2 is a schematic diagram of a different distillation framework according to an exemplary embodiment of the present specification. In the existing distillation technology, a teacher-student framework is proposed, that is, the knowledge of the teacher model 21 is directly distilled to the student model 23. Because the parameters of the student model are small and the learning ability is limited, the traditional one-to-one teacher-student model may not be able to fully utilize the rich multi-level features in the teacher model 21.

[0047] In order to solve the limitations of traditional knowledge distillation methods, this solution is improved on the basis of the teacher-student framework, using auxiliary model 22 to bridge the gap between teacher model 21 and student model 23. Auxiliary model 22 acts as an intermediary to extract specific knowledge of teacher model 21 and pass it to student model 23, so that student model 23 can learn more detailed and diverse features, thereby understanding the data more comprehensively. The exemplary teacher-assistant-student framework provided by this solution can be two types: teacher-assistant-student framework A and teacher-assistant-student framework B.

[0048] For the teacher-assistant-student framework A, the auxiliary model 22 may be a module of a neural network, which is divided into n auxiliary modules. , the teacher model 21 is divided into n teacher modules And the student module is divided into n student modules , Teacher Module , Student Module and auxiliary modules One-to-one correspondence, n is not less than 2, i=1, 2, ..., n. The arrows indicate the direction of knowledge distillation, and the various teacher modules of the teacher model 21 Knowledge can be distilled into various auxiliary modules , auxiliary modules The student module that can distill knowledge into the student model 23 Of course, the teacher model 21 can also perform knowledge distillation directly to the student model 23 while performing knowledge distillation to the auxiliary model 22.

[0049] The difference between the teacher-assistant-student framework B and the teacher-assistant-student framework A is that the auxiliary model 22 can be an independent neural network, that is, the n auxiliary models can be n mutually independent neural networks, and each auxiliary model can output probability distribution and prediction results independently. The model structures of the n auxiliary models can be at least partially the same, or they can be completely different, and this specification does not impose any restrictions on this. The teacher model 21 is divided into n teacher modules And the student model is divided into n student modules , Teacher Module , Student Module and auxiliary models One-to-one correspondence, n is not less than 2, i=1, 2, ..., n. The arrows indicate the direction of knowledge distillation, and the various teacher modules of the teacher model 21 Knowledge can be distilled into each auxiliary model separately , each auxiliary model The knowledge can be distilled into the student modules corresponding to the student model 23 respectively Of course, the teacher model 21 can also be used to provide While performing knowledge distillation, knowledge distillation is also performed directly on the student model 23.

[0050] Next, we introduce the teacher-assistant-student framework A and the teacher-assistant-student framework B respectively:

[0051] like Figure 4 As shown, Figure 4 2 is a schematic diagram of a teacher-assistant-student framework A according to an exemplary embodiment of the present specification. The auxiliary model 22 may be an auxiliary module of the neural network 41, and the neural network 41 may be divided into n auxiliary modules. , the teacher model 21 can be divided into n teacher modules And the student model 23 is divided into n student modules , Teacher Module , Student Module and auxiliary modules One-to-one correspondence, n is not less than 2, i=1, 2, ..., n.

[0052] When dividing the teacher model 21 into modules, it can be divided according to the functions of different network layers in the teacher model 21. For example, the teacher model 21 can be divided into 3 teacher modules, and the first teacher module is used to extract low-level intermediate features of the training data. For example, extract features such as pixel color, edge and texture of the input image. The second teacher module is used to extract mid-level intermediate features of the training data. For example, extract features of the local area of ​​the input image. The third teacher module is used to extract high-level intermediate features of the training data. For example, extract more abstract global features of the input image. At this time, the number of network layers contained in different teacher modules may be different, and different teacher modules are used to extract teacher intermediate features of different levels of the training data. Of course, you can also choose to divide each module into the same number of network layers. For example, assuming that the teacher model 21 contains an n-layer network and needs to be divided into 4 teacher modules, the network from the 1st to the n / 4th layers can be used as the first teacher module. ; Use the network from layer n / 4 to layer n / 2 as the second teacher module ; Use the n / 2 to 3n / 4 layer network as the third teacher module ; Finally, the network from 3n / 4 layers to the last layer is used as the fourth teacher module Similarly, the auxiliary model is an auxiliary module in the neural network 41 In the case of , the neural network 41 and the student model 23 are divided into modules in a similar manner, such as Figure 4 As shown, the neural network 41 is divided into n auxiliary modules And the student model 23 is divided into n student modules , this manual will not go into details here.

[0053] In one embodiment, the first loss function may include various auxiliary modules The extracted auxiliary intermediate features are respectively compared with the corresponding teacher modules The average of the distillation losses between the extracted teacher intermediate features, the first loss function See formula 1:

[0054] Formula 1

[0055] in, It indicates that the teacher model distills knowledge to the auxiliary model. Indicates the number of modules divided. Indicates that the teacher module of the teacher model The extracted intermediate features are distilled into the auxiliary module of the auxiliary model . Represents the average of the distillation losses between the auxiliary intermediate features extracted by the auxiliary model and the teacher intermediate features extracted by the teacher model. Indicates auxiliary module Extracted auxiliary intermediate features and teacher module Distillation loss between the extracted teacher intermediate features. By aligning the intermediate features of different levels extracted by the auxiliary model and the teacher model, the auxiliary model can extract the complete knowledge of the teacher model from shallow to deep layers.

[0056] It can be calculated using Formula 2:

[0057] Formula 2

[0058] in, Indicates auxiliary module Extracted auxiliary intermediate features Modules with teachers Extracted teacher intermediate features The feature distillation loss between and Of course, other distance functions can also be used for calculation, such as cosine distance loss function, which is not limited in this specification. Can be an auxiliary module The intermediate features output by the last feature extraction layer of Can be a teacher module The intermediate features output by the last feature extraction layer.

[0059] In another embodiment, the first loss function may also include a distribution distillation loss between the auxiliary output distribution output by the auxiliary model and the teacher output distribution output by the teacher model; and / or a label loss between the true label of the training data and the predicted result predicted by the auxiliary model. Specifically, the first loss function The deformation of can be seen in formula 3:

[0060] Formula 3

[0061] in, Represents each auxiliary module The extracted auxiliary intermediate features are respectively compared with the corresponding teacher modules The average of the distillation losses among the extracted teacher intermediate features. Represents the distribution distillation loss between the auxiliary output distribution output by the auxiliary model and the teacher output distribution output by the teacher model. Represents the label loss between the true labels of the training data and the predicted results predicted by the auxiliary model. , and is a hyperparameter to measure these three losses. It can be seen that Formula 3 also aligns the auxiliary output distribution of the auxiliary model with the teacher output distribution of the teacher model, as well as the prediction results of the auxiliary model with the true labels. By aligning with the output distribution representation of the teacher model, it can be ensured that the auxiliary model can learn the predicted distribution of the teacher model, and by aligning with the true label, the prediction ability of the auxiliary model can be improved. Through the multi-target alignment of Formula 3, the deviation problem guided by a single loss can be avoided, thereby enhancing the stability of training.

[0062] It can be calculated using Formula 4:

[0063] Formula 4

[0064] in, The teacher output distribution representing the output of the teacher model can be the probability distribution of the output of the last distribution output layer of the teacher model. The auxiliary output distribution representing the output of the auxiliary model may be the probability distribution of the output of the last distribution output layer of the auxiliary model. represents the distillation loss between the auxiliary output distribution output by the auxiliary model and the teacher output distribution output by the teacher model, which can be expressed as and The distance loss can also be calculated using a cosine distance function, and this specification does not limit the calculation method of the distance loss.

[0065] It can be expressed by formula 5:

[0066] Formula 5

[0067] Where C is the number of categories, is the true label of category c, is the prediction result predicted by the auxiliary model. The above formula calculates the loss between the true label and the prediction result predicted by the auxiliary model, which can be measured by the cross entropy loss function.

[0068] In one embodiment, when training the student model based on the second loss function, the second loss function may include each student module The extracted student intermediate features are respectively combined with the corresponding auxiliary modules The average value of the distillation loss between the extracted auxiliary intermediate features. The second loss function can be seen in Formula 6:

[0069] = Formula 6

[0070] in, represents the second loss function. Indicates auxiliary module Extracted auxiliary intermediate features and student modules Feature distillation loss between the extracted student intermediate features. n represents the number of modules. Represents each auxiliary module Extracted auxiliary intermediate features and student modules The average value of the total distillation loss between the extracted student intermediate features. By aligning the intermediate features of different levels extracted by the auxiliary model and the student model, the knowledge from shallow to deep layers extracted by the auxiliary model from different teacher modules of the teacher model can be further transferred to the corresponding student modules of the student model.

[0071] It can be calculated using Formula 7:

[0072] Formula 7

[0073] in, Represents the student module Extracted student median features With auxiliary modules Extracted auxiliary intermediate features The feature distillation loss between , which can be used and The student intermediate feature can be the student module The intermediate features output by the last feature extraction layer.

[0074] In another embodiment, the second loss function may also include a distribution distillation loss between a student output distribution output by the student model and a teacher output distribution output by the teacher model; and / or a label loss between the true label of the training data and the predicted result predicted by the student model. Specifically, The deformation of can be seen in formula 8:

[0075] Formula 8

[0076] in, Represents each student module The extracted student intermediate features are respectively combined with the corresponding auxiliary modules The average of the distillation losses among the extracted auxiliary intermediate features. Represents the distribution distillation loss between the student output distribution output by the student model and the teacher output distribution output by the teacher model. Represents the loss between the true label and the prediction result predicted by the student model. , and is a hyperparameter to measure these three losses. Formula 8 also aligns the student output distribution output by the student model with the teacher output distribution output by the teacher model, as well as the alignment of the prediction results predicted by the student model with the true label. The alignment with the teacher output distribution output by the teacher model can encourage the student model to closely approximate the teacher output distribution of the teacher model. By aligning with the true label, the prediction accuracy of the student model can be improved. Through the multi-objective alignment of Formula 8, the deviation guided by a single loss can be avoided, thereby enhancing the stability of training.

[0077] It can be calculated using formula 9:

[0078] Formula 9

[0079] It can be measured by the KL divergence of formula 9. Where J is the number of categories, and denote the predicted probabilities of the teacher model and the student model for category j, respectively. This KL divergence loss encourages the student model to closely approximate the final output distribution of the teacher model.

[0080] It can be calculated using formula 10:

[0081] Formula 10

[0082] in, is the number of categories, is the true label of category j, is the prediction result of the student model for category j, which is measured by the prediction probability. This loss strengthens the alignment of the student model with the true label and improves its prediction accuracy.

[0083] In the teacher-assistant-student framework A, this scheme divides the teacher model, auxiliary model and student model into multiple modules, so that the auxiliary model and student model can learn feature representations of different levels of the teacher model.

[0084] In one embodiment, the auxiliary model may have m layers, and the number of parameters of the auxiliary model of each layer gradually decreases as the number of layers increases. m is greater than or equal to 2. The n auxiliary models of the jth layer correspond to the n auxiliary modules into which the neural network of the layer is divided. , j = 1, 2, ..., m. The teacher model is divided into n teacher modules And the student model is divided into n student modules , Teacher Module , Student Module and auxiliary modules One-to-one correspondence, n is not less than 2, i=1, 2, ..., n.

[0085] The training data is input into the trained teacher model, the auxiliary model of the m layers, and the student model respectively. For the auxiliary model of the first layer, the auxiliary model of the layer is trained based on the third loss function, and the third loss function includes the feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model of the layer and the teacher intermediate features extracted by the teacher model. For each auxiliary model other than the first layer, the auxiliary model of the layer is trained based on the fourth loss function, and the fourth loss function includes the feature distillation loss between the auxiliary intermediate features extracted by the auxiliary model of the layer and the auxiliary intermediate features extracted by the auxiliary model of the previous layer. The student model is trained based on the fifth loss function, and the fifth loss function includes the feature distillation loss between the student intermediate features extracted by the student model and the auxiliary intermediate features extracted by the auxiliary model of the last layer. In this embodiment, compared with the single-layer auxiliary model, the multi-layer auxiliary model can learn the knowledge representation of the model of the previous layer, so that the knowledge can be refined layer by layer and the complex knowledge transfer process can be decomposed.

[0086] Next, we will introduce the teacher-assistant-student framework B:

[0087] In the teacher-assistant-student framework B, the number of auxiliary models is n, and the auxiliary models are independent neural networks. The teacher model is divided into n teacher modules And the student model is divided into n student modules , Teacher Module , Student Module and auxiliary models One-to-one correspondence, n is not less than 2, i=1, 2, ..., n. Among them, the teacher module and student modules The division method is the same as that of framework A, which will not be repeated here.

[0088] like Figure 5 As shown, Figure 5 2 is a schematic diagram of a teacher-assistant-student framework B according to an exemplary embodiment of the present specification. The training data 20 is input into the teacher model 21, the student model 23 and each auxiliary model Different auxiliary models The training processes between them are independent of each other, and the training methods between different auxiliary models are the same. Next, based on the first loss function Training auxiliary models For example, the training of other auxiliary models can refer to the auxiliary model , I will not elaborate on them one by one.

[0089] like Figure 6 As shown, Figure 6 This specification is based on a first loss function according to an exemplary embodiment. Training auxiliary models Schematic diagram of the first loss function. Training auxiliary models , the first loss function Can include: Auxiliary models The feature distillation loss between the extracted auxiliary intermediate features and the teacher intermediate features extracted by the teacher module Ti. The first loss function See formula 11:

[0090] Formula 11

[0091] in, Representation auxiliary model The first loss function is Representation auxiliary model Extracted auxiliary intermediate features and teacher module The feature distillation loss between the extracted teacher intermediate features. It should be noted that in framework A, each auxiliary module of the same neural network Align Teacher Module , while in framework B each independent auxiliary model Align Teacher Module Since each auxiliary model in framework B Independent Alignment Teacher Module , each auxiliary model can learn the features of the corresponding level of the teacher model independently, reduce interference with each other, and effectively capture the different complexities of the features. Each auxiliary model exists independently, and the structure can be designed separately according to the characteristics of the corresponding teacher module. For example, for the distillation of low-level features of the teacher model, a neural network with strong ability to extract low-level features can be selected, while for the distillation of high-level features of the teacher model, a neural network with strong ability to extract high-level features can be selected.

[0092] It can be calculated using formula 12:

[0093] Formula 12

[0094] in, Representation auxiliary model Extracted auxiliary intermediate features Modules with teachers Extracted teacher intermediate features The feature distillation loss between , which can be used and Of course, in addition to formula 12, the cosine distance can also be used for measurement, and this specification does not limit the measurement method. The auxiliary intermediate feature in framework B can be an auxiliary model The feature representation output by the final feature extraction layer, the teacher intermediate feature can be the teacher module The feature representation output by the last feature extraction layer.

[0095] In another embodiment, the first loss function Also includes: Auxiliary Models Distribution distillation loss between the auxiliary output distribution output by the output and the teacher output distribution output by the teacher model; and / or, the true labels of the training data and the auxiliary model The loss between the predicted results of the prediction. Specifically, The deformation of can be seen in formula 13:

[0096] Formula 13

[0097] in, Representation auxiliary model The extracted auxiliary intermediate features and the corresponding teacher module Feature distillation loss between the extracted teacher intermediate features. Representation auxiliary model The distribution distillation loss between the auxiliary output distribution output by the output and the teacher output distribution output by the teacher model. Represents the true label and auxiliary model The loss between the predicted results. , and is the hyperparameter to measure these three losses. It can be seen that the auxiliary model is also aligned in formula 13 The auxiliary output distribution of the output and the teacher output distribution of the teacher model output and the auxiliary model The predicted result is aligned with the true label. Through the multi-target alignment of formula 13, the deviation problem of single loss guidance can be avoided, thereby enhancing the stability of training. And each auxiliary model in formula 13 Focus only on the teacher module they are studying , which can reduce the learning difficulty and improve the alignment accuracy.

[0098] It can be calculated using formula 14:

[0099] = Formula 14

[0100] in, Representation auxiliary model The distribution distillation loss between the auxiliary output distribution output by the output and the teacher output distribution output by the teacher model can be expressed as and Measured by distance. The teacher output distribution representing the output of the teacher model can be the probability distribution of the output of the last feature extraction layer of the teacher model. Representation auxiliary model The auxiliary output distribution of the output can be an auxiliary model Probability distribution of the output of the last feature extraction layer.

[0101] It can be calculated using formula 15:

[0102] Formula 15

[0103] Among them, C is the number of categories, is the true label of category c, For auxiliary model The above formula calculates the actual label and the auxiliary model. The loss between the predicted and predicted results can be measured by the cross entropy loss function of formula 15.

[0104] When training the student model based on the second loss function, the second loss function specifically includes each student module The extracted student intermediate features are respectively combined with the auxiliary model The average value of the distillation loss between the extracted auxiliary intermediate features. The second loss function can be seen in Formula 16:

[0105] Formula 16

[0106] in, represents the second loss function. Representation auxiliary model The extracted auxiliary intermediate features and the student module of the student model Feature distillation loss between the extracted student intermediate features. Represents each auxiliary model The average of the distillation losses between the extracted auxiliary intermediate features and the student intermediate features extracted by the student model. n is the number of modules. It can be calculated using formula 17:

[0107] Formula 17

[0108] in, Represents the student module Extracted student median features With auxiliary model Extracted auxiliary intermediate features The feature distillation loss between , which can be used and Measured by the distance between.

[0109] In another embodiment, the second loss function further includes a distribution distillation loss between a student output distribution output by the student model and a teacher output distribution output by the teacher model; and / or a loss between the true label of the training data and the predicted result output by the student model. Specifically, The deformation of can be seen in formula 18:

[0110] Formula 18

[0111] in, Represents each student module The extracted student intermediate features are respectively compared with the corresponding auxiliary models The average of the distillation losses among the extracted auxiliary intermediate features. Represents the distribution distillation loss between the student output distribution output by the student model and the teacher output distribution output by the teacher model. Represents the loss between the true label and the prediction result predicted by the student model. , and is a hyperparameter to measure these three losses. Formula 18 also aligns the student output distribution of the student model output with the teacher output distribution of the teacher model output, as well as the alignment of the prediction results predicted by the student model with the true label. See formula 9, Please refer to formula 10.

[0112] In the teacher-assistant-student framework B, different auxiliary models can learn different feature segments of the teacher model respectively. The alignment between the auxiliary models and the blocks of the student model allows the student model to learn the specific level features learned by the auxiliary models. The student learning part focuses on transferring the knowledge of the teacher model and multiple auxiliary models to the student model. This hierarchical approach ensures that the student model can benefit from the comprehensive learning process, and can both efficiently acquire low-level features and obtain high-level feature representations.

[0113] In one embodiment, the auxiliary model may have m layers, each layer may have n auxiliary models, and the number of parameters of the auxiliary model of each layer gradually decreases as the number of layers increases. m is greater than or equal to 2, and the jth layer has n auxiliary models. , j = 1, 2, ..., m. The teacher model is divided into n teacher modules And the student model is divided into n student modules , Teacher Module , Student Module and auxiliary models One-to-one correspondence, n is not less than 2, i=1, 2, ..., n.

[0114] The training data is input into the trained teacher model, each auxiliary model of the m layer and the student model respectively. , based on the sixth loss function to train the auxiliary model of this layer , the sixth loss function includes the auxiliary model The feature distillation loss between the extracted auxiliary intermediate features and the teacher intermediate features extracted by the teacher model. , based on the seventh loss function to train the auxiliary model , the seventh loss function includes the auxiliary model The extracted auxiliary intermediate features and the corresponding positions of the previous layer The feature distillation loss between the extracted auxiliary intermediate features. The student model is trained based on the eighth loss function, which includes the student intermediate features extracted by the student model and the auxiliary models of the last layer. The average value of the distillation loss between the learned auxiliary intermediate features. In this embodiment, compared with the single-layer auxiliary model, the multi-layer auxiliary model can learn the knowledge representation of the previous layer model, so that the knowledge can be refined layer by layer and the complex knowledge transfer process can be decomposed.

[0115] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0116] like Figure 7 As shown, Figure 7 It is a structural diagram of an electronic device shown in this specification according to an exemplary embodiment. At the hardware level, the electronic device 700 includes a processor 702, an internal bus 704, a network interface 706, a memory 708 and a non-volatile memory 710, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0117] like Figure 8 As shown, Figure 8 is a block diagram of a model training device according to an exemplary embodiment of the present specification. The device can be applied to Figure 7 In the electronic device 700 shown in the figure, the technical solution of this specification is implemented. The device includes:

[0118] A training data input module 802, used to input training data into the teacher model, the auxiliary model and the student model respectively; the teacher model is trained;

[0119] The teacher intermediate feature extraction module 804 is used for the teacher model to extract features from the training data and transfer the extracted teacher intermediate features to the auxiliary model;

[0120] an auxiliary model training module 806, for extracting features from the training data using the auxiliary model and transferring the extracted auxiliary intermediate features to the student model; and, training the auxiliary model based on a first loss function, wherein the first loss function includes a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features;

[0121] The student model training module 808 is used for the student model to extract features from the training data to obtain student intermediate features; and to train the student model based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

[0122] Optionally, the teacher intermediate feature extraction module 804 is specifically used for the teacher model to extract features of multiple levels of the training data, and to transfer the features of the multiple levels to different auxiliary models respectively, wherein the auxiliary model is an independent neural network or a module in a neural network.

[0123] Optionally, the auxiliary model training module 806 is specifically used for each auxiliary model to transfer the extracted auxiliary intermediate features to the corresponding student module, and the student module is a module in the student model.

[0124] Optionally, the second feature distillation loss is the average value of the distillation losses between the student intermediate features output by each student module and the auxiliary intermediate features output by each auxiliary model; when the auxiliary model is a module in a neural network, the first feature distillation loss is the average value of the distillation losses between the teacher intermediate features of each level and the auxiliary intermediate features output by the corresponding auxiliary model.

[0125] Optionally, the first loss function also includes: a distribution distillation loss between the auxiliary output distribution output by the auxiliary model and the teacher output distribution output by the teacher model, and / or a loss between the true label of the training data and the prediction result predicted by the auxiliary model.

[0126] Optionally, the second loss function also includes: a distribution distillation loss between a student output distribution output by the student model and a teacher output distribution output by the teacher model, and / or a loss between a true label of the training data and a prediction result predicted by the student model.

[0127] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0128] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0129] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned model training methods provided in this application.

[0130] Specifically, computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.

Claims

1. A model training method, characterized in that: The method comprises: Inputting training data into a teacher model, an auxiliary model and a student model respectively; the teacher model is trained; wherein the training data is multimodal data, pure visual data or pure text data; The teacher model extracts features from the training data and transmits the extracted teacher intermediate features to the auxiliary model, including: the teacher model extracts features of multiple levels of the training data and transmits the features of the multiple levels to different auxiliary models respectively, wherein the auxiliary models are independent neural networks, and the training processes of different auxiliary models are independent of each other; The auxiliary model extracts features from the training data and transmits the extracted auxiliary intermediate features to the student model; and, training the auxiliary model based on a first loss function, the first loss function including a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features, including: for the first loss function of any auxiliary model, the first loss function includes a first feature distillation loss between the auxiliary intermediate features extracted by any auxiliary model and the corresponding teacher intermediate features; The student model extracts features from the training data to obtain student intermediate features; and the student model is trained based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

2. The method according to claim 1, characterized in that The second feature distillation loss is the average of the distillation losses between the student intermediate features output by each student module and the auxiliary intermediate features output by each auxiliary model.

3. The method according to claim 1, characterized in that The first loss function also includes: a distribution distillation loss between an auxiliary output distribution output by the auxiliary model and a teacher output distribution output by the teacher model, and / or a loss between a true label of the training data and a prediction result predicted by the auxiliary model.

4. The method according to claim 1, characterized in that: The second loss function also includes: a distribution distillation loss between a student output distribution output by the student model and a teacher output distribution output by the teacher model, and / or a loss between a true label of the training data and a prediction result predicted by the student model.

5. A model training device, characterized in that: The device comprises: A training data input module, used to input training data into the teacher model, the auxiliary model and the student model respectively; the teacher model is trained; wherein the training data is multimodal data, pure visual data or pure text data; The teacher intermediate feature extraction module is used for the teacher model to extract features from the training data and transfer the extracted teacher intermediate features to the auxiliary model, including: the teacher model extracts features of multiple levels of the training data and transfers the features of the multiple levels to different auxiliary models respectively, wherein the auxiliary models are independent neural networks, and the training processes of different auxiliary models are independent of each other; an auxiliary model training module, for the auxiliary model to extract features from the training data and transfer the extracted auxiliary intermediate features to the student model; and, training the auxiliary model based on a first loss function, the first loss function including a first feature distillation loss between the auxiliary intermediate features and the teacher intermediate features, including: for the first loss function of any auxiliary model, the first loss function including a first feature distillation loss between the auxiliary intermediate features extracted by any auxiliary model and the corresponding teacher intermediate features; A student model training module is used for the student model to extract features from the training data to obtain student intermediate features; and to train the student model based on a second loss function, wherein the second loss function includes a second feature distillation loss between the student intermediate features and the auxiliary intermediate features.

6. The device according to claim 5, characterized in that The second feature distillation loss is the average of the distillation losses between the student intermediate features output by each student module and the auxiliary intermediate features output by each auxiliary model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images