A model training method, device and electronic equipment

Abstract

Embodiments of the present application provide a model training method and device and electronic equipment. The method comprises: obtaining at least two teacher models and constructing a student model; the student model is used to process all target tasks of the two teacher models; obtaining current input data, including a training data corresponding to each target task and a label value corresponding to each training data; inputting the teacher model corresponding to the target task and the student model; calculating a first loss value based on the output result of the student model and the soft label, calculating a second loss value based on the output result of the student model and the hard label, and calculating a target loss value based on the first loss value and the second loss value; updating the parameters of the student model based on the target loss value, and obtaining a trained student model when the iteration stop condition is met. The present application can improve the model training efficiency, reduce resource consumption, reduce the mutual interference between multiple tasks, and improve the performance of knowledge distillation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a model training method, apparatus, and electronic device. Background Technology

[0002] Knowledge distillation is the process of transferring the knowledge learned by a complex, high-performance large model (usually called the teacher model) to a relatively simple small model (usually called the student model). This allows the small model to learn as much effective information as possible from the large model, thereby reducing computational complexity, the number of parameters, and improving inference speed while maintaining good performance.

[0003] A multi-task model is a machine learning model designed to handle multiple different but related tasks simultaneously. A multi-task model can accept various types of input data and output corresponding results for different task objectives. For example, in an image-related multi-task model, multiple tasks can be performed simultaneously, such as image classification (determining the category of objects in an image), object detection (finding objects in an image and determining their location and category), and image semantic segmentation (labeling the category of each pixel in the image).

[0004] In the knowledge distillation scenario of multi-task models, the first step is to train a multi-task teacher model. This teacher model has been fully trained on various tasks and can handle different tasks well, outputting relatively accurate results. For example, by training a multi-task teacher model that can simultaneously perform image classification, object detection, and semantic segmentation, and then using this multi-task teacher model for knowledge distillation, a multi-task student model that can simultaneously perform image classification, object detection, and semantic segmentation can be obtained.

[0005] However, training a multi-task teacher model is inefficient and resource-intensive. Furthermore, the various tasks within a multi-task teacher model may interfere with each other, thus affecting the performance of knowledge distillation. Summary of the Invention

[0006] This invention provides a model training method, apparatus, and electronic device that can improve model training efficiency, reduce resource consumption, reduce mutual interference between multiple tasks, and improve the performance of knowledge distillation.

[0007] In a first aspect, embodiments of the present invention disclose a model training method, the method comprising:

[0008] Obtain at least two teacher models and construct a student model; wherein each teacher model is used to handle at least one target task; the student model is used to handle all target tasks of the at least two teacher models;

[0009] Obtain the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data;

[0010] The training data from the input data is input into the teacher model corresponding to the target task, and all the training data from the input data is input into the student model; a target loss value is calculated based on a first loss value and a second loss value; the first loss value is calculated based on the difference between the output of the student model and the output of the teacher model; the second loss value is calculated based on the difference between the output of the student model and the label value corresponding to each training data point;

[0011] The parameters of the student model are updated based on the target loss value, and the next round of training begins. Training stops when the iteration stopping condition is met, and the trained student model is obtained.

[0012] Secondly, embodiments of the present invention disclose a model training apparatus, the apparatus comprising:

[0013] A model building module is used to acquire at least two teacher models and build student models; wherein each teacher model is used to handle at least one target task; and the student model is used to handle all target tasks of the at least two teacher models.

[0014] The data acquisition module is used to acquire the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data.

[0015] The loss calculation module is used to input the training data from the input data into the teacher model corresponding to the target task, and to input all the training data from the input data into the student model; calculate the target loss value based on a first loss value and a second loss value; the first loss value is calculated based on the difference between the output of the student model and the output of the teacher model; the second loss value is calculated based on the difference between the output of the student model and the label value corresponding to each training data.

[0016] The iterative training module is used to update the parameters of the student model based on the target loss value, enter the next round of training, and stop training when the iteration stopping condition is met, thus obtaining the trained student model.

[0017] Thirdly, embodiments of the present invention disclose an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the steps of the model training method as described above.

[0018] The embodiments of the present invention have the following advantages:

[0019] The model training method provided in this invention can jointly perform knowledge distillation on multiple single-task teacher models and / or small-scale multi-task teacher models to train a single multi-task model (student model), without pre-training a large-scale multi-task teacher model. This improves training efficiency, reduces resource consumption, and minimizes learning conflicts between tasks caused by training a large-scale multi-task teacher model, thus enhancing the performance of knowledge distillation. Furthermore, this invention calculates a first loss value based on the student model's output and soft labels, a second loss value based on the student model's output and hard labels, and a target loss value based on the first and second loss values. Combining soft and hard labels during the learning process, and utilizing the guidance of the soft labels output by the teacher model, reduces learning conflicts caused by hard labels and minimizes interference between multiple tasks. Moreover, the combination of soft and hard labels allows the student model to learn both the "experienced knowledge" of the teacher model and accurately classify based on real label information, enabling the student model to learn richer feature representations and improving its accuracy and generalization ability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of an embodiment of the model training method of the present invention;

[0022] Figure 2 This is a schematic diagram illustrating how the present invention automatically classifies and processes input data through a data filtering module;

[0023] Figure 3 This is a schematic diagram of the model architecture of an independent Head in one example of the present invention;

[0024] Figure 4 yes Figure 3A schematic diagram illustrating the calculation of the first loss value in the case of an independent head;

[0025] Figure 5 This is a schematic diagram illustrating the calculation of the first loss value in a shared Head case, as shown in one example of the present invention.

[0026] Figure 6 This is a structural block diagram of an embodiment of the model training device of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0030] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a model training method according to the present invention, the method comprising the following steps:

[0031] Step 101: Obtain at least two teacher models and construct a student model; wherein each teacher model is used to handle at least one target task; the student model is used to handle all target tasks of the at least two teacher models;

[0032] Step 102: Obtain the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data.

[0033] Step 103: Input the training data from the input data into the teacher model corresponding to the target task, and input all the training data from the input data into the student model; calculate the target loss value based on the first loss value and the second loss value; the first loss value is calculated based on the difference between the output of the student model and the output of the teacher model; the second loss value is calculated based on the difference between the output of the student model and the label value corresponding to each training data.

[0034] Step 104: Update the parameters of the student model based on the target loss value, enter the next round of training, and stop training when the iteration stopping condition is met to obtain the trained student model.

[0035] In one example, suppose we need to obtain a small model that can simultaneously handle tasks A, B, and C. Related technologies first require obtaining a pre-trained large model capable of handling tasks A, B, and C as a teacher model. Then, through knowledge distillation, the knowledge from this large model (teacher model) is transferred to a relatively simpler, smaller model (student model). This allows the student model to handle tasks A, B, and C simultaneously while maintaining a relatively small number of parameters and computational cost, and to achieve performance as close as possible to that of the teacher model, thus enabling it to run on resource-constrained hardware.

[0036] Pre-training a large model (teacher model) capable of handling tasks A, B, and C simultaneously is inefficient and resource-intensive. Furthermore, pre-training a large multi-task model may introduce learning conflicts between tasks. For example, an imbalance in the number of training samples (i.e., training data) across tasks can lead to unbalanced gradient updates, with tasks having more samples being trained more thoroughly and those having fewer samples being undertrained. Additionally, the high feature similarity between related tasks can cause interference between them.

[0037] The model training method provided in this embodiment of the invention can perform knowledge distillation on multiple single-task teacher models to train a single multi-task model (student model) without pre-training a large multi-task teacher model. This can improve training efficiency, reduce resource consumption, and reduce problems such as learning conflicts between multiple tasks caused by training a multi-task teacher model.

[0038] For example, embodiments of the present invention can obtain single-task models for processing task A, single-task models for processing task B, and single-task models for processing task C, which are designated as teacher models, denoted as Teacher Model A, Teacher Model B, and Teacher Model C, respectively. Knowledge distillation is then performed using these three teacher models to obtain a small model (student model) capable of simultaneously processing tasks A, B, and C. Since Teacher Model A, Teacher Model B, and Teacher Model C are single-task models, they are much smaller in scale compared to the large model capable of simultaneously processing tasks A, B, and C. Therefore, pre-training the small single-task models has higher training efficiency and lower resource consumption. Furthermore, since multiple single-task teacher models are trained separately, the problem of mutual interference between tasks, which affects knowledge distillation performance, caused by training a large multi-task teacher model, can be avoided.

[0039] It should be noted that the embodiments of the present invention do not limit the number of tasks that the student model can handle; the three tasks A, B, and C mentioned above are merely illustrative examples. Furthermore, in these embodiments, each teacher model used for knowledge distillation can be a single-task model, such as Teacher Model A, Teacher Model B, and Teacher Model C mentioned above, or it can be a smaller-scale multi-task model (such as two tasks). For example, in the above example, a multi-task model for simultaneously processing Task A and Task B, and a single-task model for processing Task C can also be obtained as teacher models, denoted as Teacher Model AB and Teacher Model C, respectively. These two teacher models are then used together for knowledge distillation to ultimately obtain a small model (student model) that can simultaneously process Task A, Task B, and Task C.

[0040] Specifically, in this embodiment of the invention, at least two teacher models are first obtained, and a student model is constructed; wherein each teacher model is used to process at least one target task; and the student model is used to process all target tasks of the at least two teacher models.

[0041] For example, to obtain a small model that can simultaneously handle tasks A, B, and C, we first need to acquire at least two teacher models, each handling at least one target task. For instance, we could acquire teacher model A, teacher model B, and teacher model C; or, teacher model AB (which can handle both tasks A and B) and teacher model C; or, teacher model A and teacher model BC (which can handle both tasks B and C); or, teacher model AC (which can handle both tasks A and C) and teacher model B. Tasks A, B, and C are the target tasks. The ultimate goal is to train a student model that can simultaneously handle all target tasks from all teacher models; in this example, this student model is denoted as student model ABC.

[0042] This invention does not limit the type of the target task. The target task includes, but is not limited to, face recognition, image classification, object detection, semantic segmentation, speech recognition, machine translation, natural language processing, and recommendation systems. In one example, task A and task B are both object detection tasks; task A is used to detect the target object "person," and task B is used to detect the target object "vehicle." Task C is an image classification task. In another example, task A is an object detection task, task B is a semantic segmentation task, and task C is an image classification task.

[0043] After obtaining the teacher model, a student model can be constructed. The student model and the teacher model can be deep neural network models with the same structure, but the student model has a smaller size and fewer parameters compared to the teacher model. The deep neural network model can fuse multiple neural networks. These neural networks include, but are not limited to, at least one or more combinations, superpositions, or nestings of the following: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory) network, RNN (Simple Recurrent Neural Network), attention neural network, etc.

[0044] For example, both the teacher model and the student model include a backbone network, a neck network, and a head network.

[0045] The backbone network is the foundational architecture of the model, primarily responsible for initial, general feature extraction from the input data. In computer vision, for image data, the backbone uses a series of convolutional layers, pooling layers (common in convolutional neural networks), or multi-head attention mechanisms in Transformers to progressively transform pixel-level information into semantically meaningful feature representations. For example, in image classification tasks, the backbone can extract features such as the shape and texture of objects in an image; in object detection tasks, the backbone provides basic features for subsequent detection of the target's location and category.

[0046] The neck network, located between the backbone and the head network, plays a crucial bridging role. Its main function is to further fuse, adjust, or optimize the features extracted by the backbone network, making them more suitable for the subsequent head network to perform specific tasks. For example, in object detection tasks, targets of different scales require features of different scales for effective detection. The neck network can fuse features from different levels in the backbone network to construct a structure with multi-scale features, allowing the model to better detect targets of different sizes.

[0047] The head network is the rearmost part of the model architecture and is directly related to the specific task. The head network makes the final prediction output based on features optimized by the neck network. For example, in image classification tasks, the head network typically consists of fully connected layers that map the processed feature vectors to the dimension corresponding to the number of classes, and then output the probability of each class through activation functions such as Softmax, thus determining which class the input image is most likely to belong to. In object detection tasks, the head network can output the location information (bounding box coordinates, etc.) and class information of the target object.

[0048] In practice, the backbone and neck networks of the teacher and student models are the same. The difference is that the output of the head network of the student model is the set of the outputs of the head networks of all teacher models obtained in step 101.

[0049] Next, a training set for knowledge distillation is constructed, which includes training data for each target task and a label value for each training data, referred to as a hard label.

[0050] In practical implementation, for different task scenarios, training data required for the corresponding target task can be collected to obtain a training set. For example, for the target task of image classification, a large number of images can be collected and labeled to obtain training data for the image classification task. Similarly, for the target task of speech recognition, a large number of recordings can be collected and labeled to obtain training data for the speech recognition task. Furthermore, the training set of the teacher model can be used as the training set for knowledge distillation. For example, obtaining the training set for training teacher model AB and the training set for training teacher model C, the dataset consisting of these two training sets is the training set used for knowledge distillation to train student model ABC.

[0051] Next, the current input data is obtained from the constructed training set for knowledge distillation, which includes a training data for each target task and the label value corresponding to the training data.

[0052] For example, in the example above, we need to train student model ABC, and the obtained teacher models include teacher model AB and teacher model C. Assume that the current input data includes: data A, data B, and data C, and the corresponding label values ​​for data A, data B, and data C are label A, label B, and label C, respectively.

[0053] The training data from the input data is input into the teacher model corresponding to the target task, and all the training data from the input data is input into the student model. For example, data A and label A, and data B and label B are input into teacher model AB; data C and label C are input into teacher model C; and data A and label A, data B and label B, and data C and label C are input into student model ABC.

[0054] In an optional embodiment of the present invention, the method may further include:

[0055] Step S11: Input the input data into the data filtering module, and use the data filtering module to identify the task category of each training data in the input data;

[0056] Step S12: Input the training data corresponding to the task category and the target task into the corresponding teacher model through the data filtering module, and input all the training data in the input data into the student model.

[0057] Before knowledge distillation, the current input data can be fed into a data filtering module for automatic classification processing. The data filtering module identifies the task category of each training data in the input data.

[0058] Reference Figure 2 This diagram illustrates an embodiment of the present invention where an input data is automatically categorized and processed by a data filtering module. For example... Figure 2 As shown, after the current input data (datasets) is fed into the data filtering module (filter), data A and data B are input into teacher model AB, data C is input into teacher model C, and data A, data B, and data C are input into student model ABC. It is understandable that the input data also includes hard labels. Figure 2 Not shown in the image.

[0059] The current input data (datasets) is fed into a data filtering module, which identifies the task category of each training data point in the input data. For example, if the data filtering module identifies that the task category of data A and data B corresponds to teacher model AB, and the task category of data C corresponds to teacher model C, then data A and label A, and data B and label B are input into teacher model AB, and data C and label C are input into teacher model C. Furthermore, the data filtering module inputs all training data (data A and label A, data B and label B, and data C and label C) contained in the current input data into student model ABC.

[0060] Furthermore, the method may further include: during the construction of the training set, labeling the training data corresponding to different target tasks with corresponding task categories. For example, for data A, which is used to train task A, the task category of data A can be labeled as task A, and task A corresponds to teacher model AB. Similarly, the task category of data B is labeled as task B, and task B corresponds to teacher model AB. And so on. Thus, the data filtering module can identify the task category of the received training data based on the labeling, thereby determining which teacher model's target task the training data corresponds to, and then inputting the training data into the teacher model corresponding to the corresponding target task.

[0061] After inputting the data into the corresponding teacher and student models, the model loss (target loss value) is calculated based on the output results of the teacher and student models. Furthermore, in this embodiment of the invention, hard labels and soft labels are combined to calculate the model loss during the knowledge distillation process.

[0062] Hard labels are a form of label used in supervised learning, and can be either one-hot encoding or a specific category index. For example, in a multi-class classification task with N categories, the hard label for each sample is a vector of length N, where only the position corresponding to the correct category is 1, and the rest are 0. For example, in a multi-class classification task involving cats, dogs, and pigs, if one-hot encoding is used, the hard label for "cat" is [1,0,0], the hard label for "dog" is [0,1,0], and the hard label for "pig" is [0,0,1].

[0063] Soft labels are labels output by the teacher model that include a category probability distribution. Instead of simply indicating that a sample belongs to a specific category, a soft label provides the probability value of the sample belonging to each category. These probability values ​​are usually processed, such as using the Softmax function, to ensure that the sum of all category probabilities is 1. For example, in the animal classification task mentioned above, after Softmax processing, the teacher model's output for a certain image yields a probability of 0.7 for "cat," 0.2 for "dog," and 0.1 for "pig." Therefore, the soft label for this image would be [0.7, 0.2, 0.1].

[0064] In this embodiment of the invention, a first loss value is calculated based on the difference between the output of the student model and the output of the teacher model (soft label); and a second loss value is calculated based on the difference between the output of the student model and the label value (hard label) corresponding to each training data.

[0065] Taking the student model ABC as an example, data A and data B are input into teacher model AB, data C is input into teacher model C, and data A, B, and C are input into the student model. Assume the teacher model outputs results A1 and B1, teacher model C outputs result C1, and the student model outputs results A2, B2, and C2. Based on the differences between results A1 and A2, B1 and B2, and C1 and C2, the first loss value is calculated. Based on the differences between result A2 and label A, B2 and label B, and C2 and label C, the second loss value is calculated.

[0066] In machine learning models, the difference between the predicted value and the true value for each sample is called the loss. The smaller the difference, the smaller the loss, and the better the learning model. The loss function is a non-negative real-valued function used to calculate the loss value. This invention does not restrict the loss function used to calculate the loss value. For example, any loss function such as cross-entropy, KL divergence, L2 loss, MGD loss, or FGD loss can be used.

[0067] The model loss (target loss value) of the student model can be calculated based on the first loss value and the second loss value. For example, the sum or weighted sum of the first loss value and the second loss value can be used as the target loss value. The parameters of the student model are updated based on the target loss value, so that the student model continuously moves closer to the knowledge of the teacher model.

[0068] Since the target loss value is calculated based on both soft and hard labels, the soft labels contain more information. Besides indicating the most likely category of the sample, they also reflect the teacher model's "confidence" in each category. In knowledge distillation, the student model can learn this probability distribution information from the soft labels, thereby leveraging the teacher model's generalization ability and knowledge to help the student model better learn feature representations. The hard labels explicitly indicate the true category of the sample. The combination of soft and hard labels allows the student model to learn both the teacher model's "experienced knowledge" and accurately classify based on the true label information, enabling the student model to learn richer feature representations and improving its accuracy and generalization ability.

[0069] In this embodiment of the invention, the student model and the teacher model adopt the same network structure, which, exemplarily, includes a backbone network, a neck network, and a head network. In specific implementations, the head network can include two forms: an independent head network and a shared head network.

[0070] Independent head networks refer to a multi-task model where each task is equipped with a dedicated, independent head network. That is, the input data passes through the backbone network to extract features, then through the neck network to process the features, and finally, different heads are constructed for different tasks to complete the final prediction output.

[0071] A shared head network refers to a network where multiple tasks, after feature extraction via the backbone and feature processing via the neck, share a common head for the final prediction output. This shared head receives common features from different tasks and uses these common features to perform predictions for multiple different tasks.

[0072] By sharing a head, the overall number of parameters in the model can be reduced. Multiple tasks share the same set of parameters for final prediction, saving computational resources and improving computational efficiency during the training and inference phases. It also makes it easier to deploy the model to resource-constrained hardware environments such as mobile devices and embedded devices. With independent heads, gradient updates for different tasks only occur in their respective head parts and do not affect each other, which helps to train each task more stably.

[0073] In practice, the type of Head can be determined by comprehensively considering various factors such as the specific task combination, hardware resources, and the performance requirements of each task. This embodiment of the invention does not impose any restrictions on this.

[0074] Reference Figure 3 This illustrates a schematic diagram of the model architecture for an independent Head in an example. Figure 3As shown, after the current input data (datasets) is fed into the data filtering module (filter), data A and data B are input into teacher model AB, data C is input into teacher model C, and data A, data B, and data C are input into student model ABC. It is understandable that the input data also includes hard labels. Figure 3 The diagram is not shown. Taking teacher model AB as an example, teacher model AB includes a backbone network, a neck network, and a head network. Teacher model AB uses independent heads, that is, teacher model AB includes Head A and Head B. Head A is used to process task A, and Head B is used to process task B.

[0075] Assume task A is an object detection task, specifically designed to detect whether an input image contains the object "cat". Furthermore, since object detection aims to identify target objects in an image and determine their location and category, HeadA can output the following detection information: obj (probability of object presence), cls (object category), and loc (object location). obj represents the probability or confidence level of the object's presence, used to determine if an object exists at a specific location in the image. cls represents the object's category information, used to determine which category the detected object belongs to, such as car, pedestrian, or animal. loc represents the object's location information in the image, typically described in the form of a bounding box, including the object's coordinates, width, and height.

[0076] For example, the prediction result output by HeadA is the product of obj and cls, which represents the confidence level that a target object exists in the image.

[0077] The structures of teacher model C and student model ABC are similar to those of teacher model AB. Figure 3 Not shown in the image.

[0078] Reference Figure 4 , showed Figure 3 This example illustrates the calculation of the first loss value in the case of an independent head. Figure 4 As shown, teacher models AB, C, and ABC all use independent heads. Teacher model AB includes HeadA and HeadB, which are used to handle tasks A and B, respectively. Teacher model C includes HeadC, which is used to handle task C. Student models ABC include HeadA, HeadB, and HeadC, which are used to handle tasks A, B, and C, respectively.

[0079] The output of the teacher model is also called the soft label. When calculating the first loss value, based on the difference between the output of HeadA of teacher model AB and HeadA of student model ABC, and the difference between the output of HeadB of teacher model AB and HeadB of student model ABC, the loss value between the output of student model ABC and the soft label output by teacher model AB can be calculated (as denoted as lossAB); based on the difference between the output of HeadC of teacher model C and HeadC of student model ABC, the loss value between the output of student model ABC and the soft label output by teacher model C can be calculated (as denoted as lossC).

[0080] When the student model uses independent heads, a first loss value can be calculated based on the loss value between the output of each student model head and the soft label output of each teacher model. For example, the first loss value can be calculated based on lossAB and lossC. Exemplarily, the sum or weighted sum of lossAB and lossC can be used as the first loss value.

[0081] In an optional embodiment of the present invention, the method may further include:

[0082] Step S21: If the student model uses a shared head network, then the output of the shared head network is sliced ​​to obtain the output of the corresponding target task of each output layer.

[0083] Step S22: Calculate the first loss value based on the difference between the output of each output layer of each teacher model and the output of each output layer of the student model.

[0084] If the student model uses a shared head, a dedicated output layer can be added for each target task, such as a multilayer perceptron (MLP) or a convolutional layer. This output layer can perform more complex feature transformations and combinations on the shared features to adapt to the needs of different tasks.

[0085] For example, the student model handles multiple tasks including semantic segmentation and object detection. For semantic segmentation, a series of convolutional and upsampling layers can be added as output layers after the shared head to map shared features to pixel-level class probability distributions, enabling class prediction for each pixel in the image. For object detection, specific convolutional and regression layers can be added after the shared head to predict the class and location of the target. The output of the shared head includes the output of the corresponding output layer for each target task. For example, in this example, the output of the shared head includes the output of the semantic segmentation task and the output of the object detection task. This embodiment of the invention does not limit the specific form of the output of the shared head; for example, it can be an array.

[0086] Taking the student model ABC as an example, the output of the shared head can be an array containing the outputs of task A, task B, and task C. The outputs of task A and task B need to be compared with the soft labels output by the teacher model AB, and the output of task C needs to be compared with the soft labels output by the teacher model C. Therefore, when calculating the first loss value, the output of the shared head of the student model needs to be sliced ​​to obtain the output of the corresponding target task for each output layer.

[0087] Reference Figure 5 This illustrates a diagram of calculating the first loss value in an example with a shared head. Figure 5 As shown, the student model ABC uses a shared head, meaning that tasks A, B, and C are processed through a single head. When calculating the first loss value, the output of the shared head needs to be segmented into the outputs of tasks A, B, and C. The outputs of tasks A and B are compared with the soft labels output by the teacher model AB to calculate loss AB; the output of task C is compared with the soft labels output by the teacher model C to calculate loss C; and then the first loss value is calculated based on loss AB and loss C.

[0088] In an optional embodiment of the present invention, the method may further include:

[0089] Step S31: Calculate the output layer loss value based on the difference between the output results of the student model and the output results of the teacher model;

[0090] Step S32: Calculate the intermediate layer loss value based on the intermediate layer features of each teacher model and the intermediate layer features of the student model;

[0091] Step S33: Calculate the first loss value based on the output layer loss value and the intermediate layer loss value.

[0092] In this embodiment of the invention, the loss value calculated based on the model's output is called the output layer loss value, and the first loss value can be directly calculated based on the output layer loss value. Taking the student model ABC as an example, the output layer loss values ​​include lossAB and lossC, so the first loss value can be calculated based on lossAB and lossC, such as by adding lossAB and lossC or by weighted addition. Here, lossAB and lossC are calculated based on the difference between the output results of the student model's output layer and the teacher model's output layer, and are therefore called output layer losses.

[0093] Furthermore, embodiments of the present invention can also calculate intermediate layer loss values, and combine intermediate layer loss values ​​and output layer loss values ​​to jointly calculate a first loss value.

[0094] In this embodiment of the invention, the student model and the teacher model have the same network structure, and can be divided into N stages, where N is a positive integer. Each stage can include at least one network layer. For the same training data, features from the last layer output of each of the N stages in both the teacher and student models are extracted; these are called intermediate layer features. For the teacher model, intermediate layer features can be extracted from N stages; similarly, for the student model, intermediate layer features can be extracted from N stages.

[0095] Based on the intermediate layer features of the i-th segment of the teacher model and the intermediate layer features of the i-th segment of the student model, the intermediate layer loss value of the i-th segment can be calculated, where i takes the value from 1 to N, thus obtaining N intermediate layer loss values. Based on these N intermediate layer loss values ​​and the output layer loss value, the first loss value can be calculated. This can be achieved by adding the N intermediate layer loss values ​​to the output layer loss value or by weighted summation.

[0096] The purpose of the intermediate layer loss value is to make the features output by each stage of the student model as close as possible to the features output by the corresponding stage in the teacher model. Thus, in addition to knowledge distillation of the output layer, this embodiment of the invention also realizes hierarchical knowledge distillation of the teacher model, so that the intermediate layer output distribution of the student model is constrained by the high-precision teacher model, thereby further improving the accuracy of the student model.

[0097] In an optional embodiment of the present invention, the method may further include:

[0098] Step S41: When constructing the student model, assign corresponding task weights to each target task; the task weights are used to calculate the first loss value and the second loss value.

[0099] Step S42: During the training of the student model, the task weight of the target task is dynamically adjusted according to the scale of the target task and / or the loss value generated by the target task.

[0100] In this embodiment of the invention, the student model is a multi-task model capable of processing multiple target tasks simultaneously. In specific implementations, different tasks may have varying degrees of difficulty, data volume, or importance. Therefore, corresponding task weights can be assigned to the multiple target tasks processed by the student model, allowing for the reasonable allocation of resources among different target tasks during student model training.

[0101] Specifically, when building a student model, initial task weights can be set, and a dynamic adjustment mechanism can be introduced. During training, the task weights can be dynamically adjusted according to the student model's learning progress and difficulty on each target task, thereby dynamically adjusting the allocation of training resources to ensure balanced development of multi-task learning.

[0102] This invention does not limit the method for initially setting task weights. For example, all target tasks can be set to equal weights (uniform weights), such as all being set to 1. Alternatively, task weights can be set according to the difficulty level of the target tasks; for example, higher difficulty tasks can be given higher task weights. The difficulty level of the target tasks can be determined based on the complexity and volume of the data processed by the target tasks.

[0103] Furthermore, the training process can be divided into multiple stages, with dynamic adjustments made to the task weights and loss calculation methods at different stages. For example, in the early training stage, uniform weights are used to enable the student model to learn each target task initially; in the later training stage, the task weights corresponding to the target tasks are dynamically adjusted according to the scale of the target tasks and performance indicators, and a weighted summation loss is calculated.

[0104] Assume L i Let w be the loss function for the i-th objective task. i Let L be the task weight corresponding to the i-th objective task. There are a total of n objective tasks, then the model loss L is:

[0105]

[0106] For example, the loss function of each target task can be weighted and summed based on the task weight of each target task to calculate the first loss value and the second loss value. Taking the example of training the student model ABC using teacher model AB and teacher model C, when calculating the first loss value, lossAB and lossC are calculated first. Assume L A Let w be the loss function for task A.A L represents the task weight corresponding to task A. B Let w be the loss function for task B. B L represents the task weight corresponding to task B. C Let w be the loss function for task C. C Let w be the task weight corresponding to task C; then calculate lossAB = w A L A +w B L B ; and calculate loss C = w C L C For example, the first loss value is calculated as lossAB + lossC.

[0107] The scope of a target task can include its importance level and difficulty level. For example, a higher importance level and / or a higher difficulty level can increase the task weight, while a lower importance level and / or a lower difficulty level can decrease the task weight.

[0108] Furthermore, tasks with higher difficulty levels converge more slowly during training, and their loss decreases more difficult. Therefore, task weights can be dynamically adjusted based on the changes in the loss value generated by each target task. For example, assigning greater task weights to target tasks with slow loss decreases (usually higher difficulty levels) allows the model training to focus more on higher difficulty tasks, accelerating the training process for those tasks.

[0109] Furthermore, in addition to adjusting the target task's size and the changes in the loss value, task weights can also be dynamically adjusted based on metrics that reflect the actual performance of the target task, such as accuracy, recall (in tasks like object detection), and F1 score. If the performance metrics of a certain target task improve slowly or remain at a low level, the corresponding task weight can be appropriately increased to guide the model to optimize the performance of that target task more.

[0110] In practice, training a large multi-task teacher model may lead to learning conflicts due to the correlation between multiple tasks, affecting model performance. This invention utilizes multiple single-task teacher models, or small-scale multi-task teacher models (e.g., one teacher model with no more than two tasks), to jointly perform knowledge distillation and train a small multi-task student model. This reduces mutual interference between multiple tasks, and the guidance provided by the soft labels output by the teacher model can mitigate learning conflicts caused by hard labels.

[0111] After updating the parameters of the student model based on the target loss value, new input data is obtained from the training set, and the next round of training is entered. The above steps are repeated, and training is stopped when the iteration stopping condition is met, resulting in a trained student model.

[0112] This invention does not impose limitations on the iteration stopping condition. The iteration stopping condition may include any one of the following:

[0113] The target loss value is less than a preset threshold; or...

[0114] The number of iterations for training reaches the preset maximum number of iterations, or...

[0115] The performance metrics of key tasks in each target task must meet the required standards. In this embodiment of the invention, the student model is a multi-task model, and the target loss value is the comprehensive loss value calculated based on multiple target tasks from multiple teacher models. Training can be stopped when the comprehensive loss value drops to a preset lower threshold. Alternatively, a maximum number of iterations can be set, and training can be stopped after completing a specified number of iterations. Alternatively, an early stopping method can be used, recording the performance metrics of each target task on the validation set. If the performance metrics of each target task or its key tasks meet the required standards, training can be stopped early. Specifically, key tasks can be identified among the target tasks, and thresholds for performance metrics such as accuracy and recall can be set for the key tasks. When the performance metrics of the key tasks on the validation or test set reach or exceed the preset performance metric thresholds, the performance metrics are considered to be met, and training can be stopped early.

[0116] Furthermore, other optimization methods can be employed during the training of the student model to improve its overall performance. For example, periodic performance evaluations can be performed on the student model, and strategies such as cross-validation can be used to monitor its performance, adjust training parameters in a timely manner, and avoid overfitting.

[0117] Furthermore, after the student model has been trained, it can be deployed on the target device to perform the target task. For example, after the student model ABC has been trained, it can be deployed on the target device. The image to be processed is input into the student model ABC, and the student model ABC can output the results of task A, task B, and task C.

[0118] The trained student model can be deployed on resource-constrained hardware devices and can quickly infer correct prediction results with low power consumption.

[0119] The target devices include, but are not limited to: smart home terminals (including air conditioners, refrigerators, rice cookers, water heaters, etc.), smart business terminals (including video phones, smart conference desktop terminals, etc.), wearable devices (including smartwatches, smart glasses, etc.), smart financial terminals, as well as smartphones, tablets, personal digital assistants (PDAs), in-vehicle devices, computers, etc.

[0120] In summary, the model training method provided in this invention can jointly perform knowledge distillation on multiple single-task teachers and / or small-scale multi-task teacher models to train a single multi-task model (student model), without pre-training a large-scale multi-task teacher model. This improves training efficiency, reduces resource consumption, and minimizes learning conflicts between tasks caused by training a large-scale multi-task teacher model, thus enhancing the performance of knowledge distillation. Furthermore, this invention calculates a first loss value based on the student model's output and soft labels, a second loss value based on the student model's output and hard labels, and a target loss value based on the first and second loss values. Combining soft and hard labels during the learning process, and utilizing the guidance of the soft labels output by the teacher model, reduces learning conflicts caused by hard labels and minimizes interference between multiple tasks. Moreover, the combination of soft and hard labels allows the student model to learn both the "experienced knowledge" of the teacher model and accurately classify based on real label information, enabling the student model to learn richer feature representations and improving its accuracy and generalization ability.

[0121] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0122] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of a model training device according to the present invention. The device may include:

[0123] The model building module 601 is used to acquire at least two teacher models and build student models; wherein each teacher model is used to handle at least one target task; and the student model is used to handle all target tasks of the at least two teacher models.

[0124] The data acquisition module 602 is used to acquire the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data.

[0125] The loss calculation module 603 is used to input the training data from the input data into the teacher model corresponding to the target task, and to input all the training data from the input data into the student model; calculate the target loss value based on the first loss value and the second loss value; the first loss value is calculated based on the difference between the output result of the student model and the output result of the teacher model; the second loss value is calculated based on the difference between the output result of the student model and the label value corresponding to each training data.

[0126] The iterative training module 604 is used to update the parameters of the student model based on the target loss value, enter the next round of training, and stop training when the iteration stopping condition is met, thus obtaining the trained student model.

[0127] Optionally, the device further includes:

[0128] The slicing calculation module is used to slice the output of the shared head network if the student model uses a shared head network, to obtain the output of the corresponding target task of each output layer; and to calculate the first loss value based on the difference between the output of each output layer of each teacher model and the output of each output layer of the student model.

[0129] Optionally, the device further includes:

[0130] The output layer loss calculation module is used to calculate the output layer loss value based on the difference between the output results of the student model and the output results of the teacher model.

[0131] The intermediate layer loss calculation module is used to calculate the intermediate layer loss value based on the intermediate layer features of each teacher model and the intermediate layer features of the student model.

[0132] The first loss calculation module is used to calculate a first loss value based on the output layer loss value and the intermediate layer loss value.

[0133] Optionally, the device further includes:

[0134] The weight setting module is used to assign corresponding task weights to each target task when constructing the student model; the task weights are used to calculate the first loss value and the second loss value.

[0135] The weight adjustment module is used to dynamically adjust the task weights of the target task based on the size of the target task and / or the loss value generated by the target task during the training of the student model.

[0136] Optionally, the student model and the teacher model have the same network structure, and the output of the head network of the student model is the set of the output of the head network of the teacher model.

[0137] Optionally, the iteration stopping condition includes:

[0138] The target loss value is less than a preset threshold; or...

[0139] The number of iterations for training reaches the preset maximum number of iterations, or...

[0140] The performance indicators of the key tasks in each target task have met the standards.

[0141] Optionally, the device further includes:

[0142] The data filtering module is used to receive the input data and identify the task category of each training data in the input data; input the training data corresponding to the task category and the target task into the corresponding teacher model, and input all the training data in the input data into the student model.

[0143] The model training apparatus provided in this invention can jointly perform knowledge distillation on multiple single-task teacher models and / or small-scale multi-task teacher models to train a single multi-task model (student model), without pre-training a large-scale multi-task teacher model. This improves training efficiency, reduces resource consumption, and minimizes learning conflicts between tasks caused by training a large-scale multi-task teacher model, thus enhancing the performance of knowledge distillation. Furthermore, this invention calculates a first loss value based on the student model's output and soft labels, a second loss value based on the student model's output and hard labels, and a target loss value based on the first and second loss values. Combining soft and hard labels during the learning process, and utilizing the guidance of the soft labels output by the teacher model, reduces learning conflicts caused by hard labels and minimizes interference between multiple tasks. Moreover, the combination of soft and hard labels allows the student model to learn both the "experienced knowledge" of the teacher model and accurately classify based on real label information, enabling the student model to learn richer feature representations and improving its accuracy and generalization ability.

[0144] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0147] Reference Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 7 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to execute the steps of the model training method of the aforementioned embodiment.

[0148] This invention provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a terminal's program or processor, enables the terminal to perform the steps of the model training method described in the foregoing embodiments.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0150] Furthermore, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program products or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0151] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0152] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0154] The above provides a detailed description of the model training method, apparatus, and electronic device provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A model training method, characterized in that, The method includes: Obtain at least two teacher models and construct a student model; wherein each teacher model is used to handle at least one target task; the student model is used to handle all target tasks of the at least two teacher models; Obtain the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data; The training data from the input data is input into the teacher model corresponding to the target task, and all the training data from the input data is input into the student model; a target loss value is calculated based on a first loss value and a second loss value; the first loss value is calculated based on the difference between the output of the student model and the output of the teacher model; the second loss value is calculated based on the difference between the output of the student model and the label value corresponding to each training data point; The parameters of the student model are updated based on the target loss value, and the next round of training begins. Training stops when the iteration stopping condition is met, and the trained student model is obtained.

2. The method according to claim 1, characterized in that, The method further includes: If the student model uses a shared head network, the output of the shared head network is sliced ​​to obtain the output of the corresponding target task of each output layer. The first loss value is calculated based on the difference between the output of each output layer of each teacher model and the output of each output layer of the student model.

3. The method according to claim 1, characterized in that, The method further includes: The output layer loss value is calculated based on the difference between the output results of the student model and the output results of the teacher model. The intermediate layer loss value is calculated based on the intermediate layer features of each teacher model and the intermediate layer features of the student model. The first loss value is calculated based on the output layer loss value and the intermediate layer loss value.

4. The method according to claim 1, characterized in that, The method further includes: When constructing the student model, a corresponding task weight is assigned to each target task; the task weight is used to calculate the first loss value and the second loss value. During the training of the student model, the task weights of the target task are dynamically adjusted based on the size of the target task and / or the loss value generated by the target task.

5. The method according to claim 1, characterized in that, The student model and the teacher model have the same network structure, and the output of the head network of the student model is the set of the output of the head network of the teacher model.

6. The method according to claim 1, characterized in that, The iteration stopping conditions include: The target loss value is less than a preset threshold; or... The number of iterations for training reaches the preset maximum number of iterations, or... The performance indicators of the key tasks in each target task have met the standards.

7. The method according to claim 1, characterized in that, The method further includes: The input data is fed into the data filtering module, which identifies the task category of each training data in the input data. The data filtering module inputs the training data corresponding to the task category and the target task into the corresponding teacher model, and inputs all the training data in the input data into the student model.

8. A model training device, characterized in that, The device includes: A model building module is used to acquire at least two teacher models and build student models; wherein each teacher model is used to handle at least one target task; and the student model is used to handle all target tasks of the at least two teacher models. The data acquisition module is used to acquire the current input data, which includes a training data corresponding to each target task and a label value corresponding to each training data. The loss calculation module is used to input the training data from the input data into the teacher model corresponding to the target task, and to input all the training data from the input data into the student model; calculate the target loss value based on a first loss value and a second loss value; the first loss value is calculated based on the difference between the output of the student model and the output of the teacher model; the second loss value is calculated based on the difference between the output of the student model and the label value corresponding to each training data. The iterative training module is used to update the parameters of the student model based on the target loss value, enter the next round of training, and stop training when the iteration stopping condition is met, thus obtaining the trained student model.

9. The apparatus according to claim 8, characterized in that, The device further includes: The slicing calculation module is used to slice the output of the shared head network if the student model uses a shared head network, to obtain the output of the corresponding target task of each output layer; and to calculate the first loss value based on the difference between the output of each output layer of each teacher model and the output of each output layer of the student model.

10. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the model training method as described in any one of claims 1 to 7.

Images