Model training method and device, and readable storage medium

Abstract

The present disclosure relates to a model training method and device and a readable storage medium, wherein the method performs knowledge distillation on each first intermediate subnetwork of a teacher model, thereby avoiding the selection and setting problems of the distillation position and the number of distillation layers, and achieving better distillation effect in the intermediate subnetwork distillation task. In addition, the knowledge distillation is performed by using the boundary mean square error loss, which not only enables the student model to learn the output of the teacher model, but also enables the student model to learn whether the teacher model activates the output, so that the student model learns more knowledge and is beneficial to improve the performance of the target student model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, apparatus, and readable storage medium. Background Technology

[0002] Deep neural networks are being used in a variety of tasks. The more complex the task, the larger the scale of the deep neural network, and the greater the computational resource consumption. Therefore, model compression technology is receiving more and more attention due to practical needs.

[0003] Knowledge distillation is a key method for compressing deep neural network models. Specifically, a large-scale model is pre-trained as the teacher model, and a small-scale model is selected as the student model. The student model learns the output of the teacher model, resulting in a well-trained student model. This student model has performance close to that of the teacher model but is smaller in scale. Currently, for tasks such as regression and generation, knowledge distillation of intermediate subnetworks in deep neural network models is often required to achieve model compression or performance improvement.

[0004] However, determining the distillation location is a pressing issue when performing knowledge distillation on intermediate subnetworks of deep neural network models. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a model training method, apparatus, and readable storage medium.

[0006] In a first aspect, embodiments of this disclosure provide a model training method, including:

[0007] Obtain the sample dataset, teacher model, and student model corresponding to the target task; wherein the teacher model is a pre-trained model;

[0008] The sample dataset is input into the teacher model, and the student model is trained by knowledge distillation using the output of each first intermediate subnetwork of the teacher model to obtain the target student model; wherein, the teacher model includes a first input subnetwork, at least one first intermediate subnetwork, and a first output subnetwork.

[0009] In some possible designs, the sample dataset is input into the teacher model, and the student model is trained using knowledge distillation based on the output of each first intermediate subnetwork of the teacher model to obtain the target student model, including:

[0010] (a) Input the sample data included in the sample dataset into the teacher model and the student model respectively, and obtain the first result output by each first intermediate subnetwork of the teacher model and the second result output by each second intermediate subnetwork of the student model respectively;

[0011] The student model includes a second input subnetwork, at least one second intermediate subnetwork, and a second output subnetwork, wherein the first intermediate subnetwork corresponds one-to-one with the second intermediate subnetwork.

[0012] (b) Update the student model based on the truth labels of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks;

[0013] Repeat steps (a) to (b) until the preset convergence condition is met to obtain the target student model.

[0014] In some possible designs, if the last layer of the intermediate subnetwork is an activation function layer, then the output of the intermediate subnetwork is the output of the penultimate layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

[0015] In some possible designs, if the last layer of the intermediate subnetwork is not an activation function layer, then the output of the intermediate subnetwork is the output of the last layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

[0016] In some possible designs, updating the student model based on the ground truth labeling of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks includes:

[0017] Based on the ground truth labeling of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks, the first loss information is obtained.

[0018] The student model is updated based on the first loss information.

[0019] In some possible designs, obtaining the first loss information based on the ground truth labeling of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks includes:

[0020] The second loss information is obtained based on the boundary mean square error between the first result output by each of the first intermediate sub-networks and the second result output by the corresponding second intermediate sub-network.

[0021] Based on the ground truth labeling of the sample data and the second results output by each of the second intermediate sub-networks, the third loss information is obtained;

[0022] The first loss information is obtained based on the second loss information and the third loss information.

[0023] In some possible designs, obtaining the second loss information based on the boundary mean square error between the first result output by each of the first intermediate sub-networks and the second result output by the corresponding second sub-network includes:

[0024] For each of the first intermediate sub-networks, channel mapping is performed on the second result output by the corresponding second intermediate sub-network according to the number of channels of the first intermediate sub-network to obtain the mapped second result;

[0025] The first result output by the first intermediate sub-network is corrected according to a preset boundary threshold to obtain the corrected first result corresponding to the first intermediate sub-network.

[0026] Based on the mean square error between the corrected first result and the mapped second result corresponding to the first intermediate sub-network, the fourth loss information corresponding to the first intermediate sub-network is obtained.

[0027] The second loss information is obtained by summing or averaging the fourth loss information corresponding to each of the first intermediate sub-networks.

[0028] In some possible designs, the step of performing boundary correction on the first result output by the first intermediate sub-network according to a preset boundary threshold, and obtaining the corrected first result corresponding to the first intermediate sub-network, includes:

[0029] If the feature value included in the first result is greater than or equal to the preset boundary threshold, then the feature value remains unchanged;

[0030] If the feature value included in the first result is less than the preset boundary threshold, then the feature value is modified to the preset boundary threshold.

[0031] Secondly, embodiments of this disclosure provide a model training apparatus, comprising:

[0032] The acquisition module is used to acquire the sample dataset, teacher model, and student model corresponding to the target task; wherein the teacher model is a model obtained through pre-training.

[0033] The model training module is used to input the sample dataset into the teacher model and use the output of each first intermediate subnetwork of the teacher model to perform knowledge distillation training on the student model to obtain the target student model.

[0034] Thirdly, another embodiment of this disclosure also provides an electronic device, including: a memory, a processor, and computer program instructions;

[0035] The memory is configured to store the computer program instructions;

[0036] The processor is configured to execute the computer program instructions to implement the method as described in any of the first aspects.

[0037] Fourthly, another embodiment of this disclosure also provides a readable storage medium, comprising: a computer program; wherein, when executed by at least one processor of an electronic device, the computer program implements the method as described in any of the first aspects.

[0038] Fifthly, another embodiment of this disclosure also provides a program product comprising a computer program stored in a readable storage medium, wherein at least one processor of the model training apparatus can read the computer program from the readable storage medium, and the at least one processor executes the computer program to implement the method as described in any of the first aspects.

[0039] This disclosure provides a model training method, apparatus, and readable storage medium. The method avoids the problems of selecting and setting the distillation position and the number of distillation layers by selecting each first intermediate subnetwork of the teacher model for knowledge distillation, and can achieve better distillation results in the intermediate subnetwork distillation task.

[0040] In addition, this scheme uses boundary mean squared error loss for knowledge distillation, which not only enables the student model to learn the output of the teacher model, but also enables the student model to learn whether the teacher model activates that output, thus enabling the student model to learn more knowledge and improving the performance of the target student model. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0042] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of a model training method provided in an embodiment of this disclosure;

[0044] Figure 2 This is a schematic diagram of the model training framework when using the model training method provided in this disclosure;

[0045] Figure 3 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure;

[0046] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0047] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0048] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0049] Currently, knowledge distillation mostly involves distilling the output subnetworks of deep neural network models, i.e., the student model learning the output of the teacher model. Output subnetwork distillation performs well in classification tasks, but it is not suitable for tasks such as regression and generation. These tasks typically require distilling the intermediate subnetworks of the deep neural network model. However, intermediate subnetwork distillation is relatively complex, requiring careful consideration of the distillation location and the number of distillation layers.

[0050] Based on this, this disclosure provides a model training method, apparatus, and readable storage medium. By selecting each intermediate subnetwork of the teacher model for knowledge distillation, the problem of selecting and setting the distillation location and the number of distillation layers is avoided, and a better distillation effect can be achieved in the intermediate subnetwork distillation task.

[0051] The model training method disclosed herein is executed by a model training device. The model training device may include tablet computers, mobile phones (such as foldable phones, large-screen phones, etc.), wearable devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart TVs, smart screens, high-definition TVs, 4K TVs, smart speakers, smart projectors, and other Internet of Things (IoT) devices, independent servers, server clusters, etc. This disclosure does not impose any restrictions on the specific type of model training device.

[0052] Based on the foregoing description, this disclosure will use an embodiment as an example, combined with the accompanying drawings and application scenarios, to elaborate in detail on the model training method provided by this disclosure.

[0053] Figure 1 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 1 As shown, the method provided in this embodiment includes:

[0054] S101. Obtain the sample dataset, teacher model, and student model corresponding to the target task.

[0055] The model training device can acquire a sample dataset corresponding to the target task. The sample dataset includes at least one sample data, and each sample data contains a corresponding ground truth label. This disclosure does not limit the target task; for example, the target task can be an image recognition task.

[0056] The model training device can also acquire a teacher model. The teacher model is a pre-trained model. It can be pre-trained and stored in the model training device, or it can be obtained by training an initial teacher model using the sample dataset after receiving the knowledge distillation task and acquiring the sample dataset for the target task.

[0057] The model training device can also acquire student models. In this scheme, the student model can be a model obtained through pre-training or an untrained student model. This disclosure does not impose any restrictions on this.

[0058] Specifically, when the student model is a pre-trained model, it can be obtained by the model training device receiving the knowledge distillation task and acquiring the sample dataset of the target task, and then training the initial student model using the sample dataset. When the student model is not pre-trained, the weight coefficients of each parameter in the student model can be determined through random initialization or preset.

[0059] In this scheme, the teacher model includes a first input subnetwork, at least one first intermediate subnetwork, and a first output subnetwork. The student model includes a second input subnetwork, at least one second intermediate subnetwork, and a second output subnetwork. The number of first intermediate subnetworks is the same as the number of second intermediate subnetworks, and there is a one-to-one correspondence between the first and second intermediate subnetworks.

[0060] S102. Input the sample dataset into the teacher model, and use the output of each first intermediate subnetwork of the teacher model to perform knowledge distillation training on the student model to obtain the target student model.

[0061] For example, the target student model can be obtained through the following steps:

[0062] Step (a): The model training device inputs the sample data included in the sample dataset into the teacher model and the student model respectively, and obtains the first result output by each first intermediate subnetwork of the teacher model and the second result output by each second intermediate subnetwork of the student model.

[0063] Step (b) involves obtaining first loss information based on the ground truth labels carried by the sample data, the first results output by each first intermediate sub-network, and the second results output by each second intermediate sub-network, using a pre-constructed loss function; and updating the student model based on the first loss information.

[0064] Repeat steps (a) to (b) until the preset convergence condition is met to obtain the target student model. The target student model is the model that meets the preset convergence condition.

[0065] The preset convergence conditions may include, but are not limited to, the number of training iterations, model evaluation metrics, etc.

[0066] In this scheme, the distillation location can include the following two cases:

[0067] (1) The last layer of the first intermediate subnetwork is the activation function layer.

[0068] Specifically, when the last layer of a certain first intermediate sub-network is an activation function layer, the distillation position for the first intermediate sub-network is located in the layer before the activation function layer. In other words, the first output of the first intermediate sub-network is the output of the layer before the activation function layer, that is, the output of the penultimate layer of the first intermediate sub-network.

[0069] When the last layer of a first intermediate subnetwork is an activation function layer, the last layer of the second intermediate subnetwork corresponding to that first intermediate subnetwork is also an activation function layer.

[0070] Similarly, for the corresponding second intermediate subnetwork, the distillation position is located in the layer before the activation function. That is, the second result output by the second intermediate subnetwork is the output of the layer before the activation function layer, i.e., the output of the penultimate layer of the second intermediate subnetwork.

[0071] (2) The last layer of the first intermediate subnetwork is not an activation function layer.

[0072] Specifically, when the last layer of a certain first intermediate sub-network is not an activation function layer, the distillation position is located at the last layer of the first intermediate sub-network. In other words, the first output of the first intermediate sub-network is the output of the last layer of the first intermediate sub-network.

[0073] When the last layer of a first intermediate subnetwork is not an activation function layer, the last layer of the second intermediate subnetwork corresponding to the first intermediate subnetwork is also not an activation function layer. In other words, the second result output by the second intermediate subnetwork is the output result of the last layer of the second intermediate subnetwork.

[0074] Similarly, for the corresponding second intermediate subnetwork, the distillation site is located in the last layer of the second intermediate subnetwork.

[0075] Since the feature values ​​contained in the first result can be either negative or positive, they contain knowledge regardless of whether they are positive or negative. In order to ensure that the student model can learn the knowledge mapped by the part of the first result with negative feature values, the distillation position is set before the activation function layer of the intermediate sub-network when the last layer of the intermediate sub-network is an activation function layer.

[0076] In this scheme, the pre-built loss function for the student model (referring to the overall loss function of the student model) is as follows:

[0077] The overall loss function of the student model consists of two parts: the knowledge distillation loss function and the original loss function of the student model. The overall loss function of the student model is denoted as Loss. total The original loss function of the student model is denoted as Loss. gt The knowledge distillation loss function is denoted as Loss. distill The above three satisfy formula (1).

[0078] Loss total =λ1*Loss distill +λ2*Loss gt Formula (1)

[0079] In formula (1), λ1 represents Lossdistill The weighting coefficient; λ2 represents the loss. gt The weighting coefficients.

[0080] Optionally, λ2 can be a constant, for example, a constant of 1. Furthermore, during knowledge distillation, the ratio between the knowledge distillation loss function and the original loss function of the student model can be adjusted by changing the value of λ1.

[0081] Knowledge distillation loss function Loss distill :

[0082] Among them, the knowledge distillation loss function Loss distill It is obtained by summing or averaging the boundary mean square errors between the first and second intermediate subnetworks that have a corresponding relationship in each group.

[0083] When using a summation method, the knowledge distillation loss function can be expressed by the formula:

[0084]

[0085] In formula (2), i represents the index of the intermediate subnetwork, i = 1, 2…n; n is the total number of intermediate subnetworks, n is an integer greater than or equal to 1; MarginMSE() represents the boundary mean square loss function; T i S represents the first result output by the i-th first intermediate sub-network; i This represents the second result output by the i-th second intermediate sub-network.

[0086] For each pair of corresponding first and second intermediate subnetworks, the first result T is calculated using the boundary mean square loss function MarginMSE(). i Second result S i Before determining the boundary mean square error, the following two steps need to be performed:

[0087] Step 1: Adjust the second result S according to the number of channels in the first intermediate sub-network. i Perform channel mapping so that the mapped second result S i ′ and the first result T i The number of channels is the same.

[0088] Since the student model can be a compressed model, the first result T output by the first intermediate subnetwork of the teacher model is... i The second result S output by the second intermediate subnetwork of the student model i The number of channels may differ for each case, making it impossible to calculate the first result T. i With the second result S iThe mean square error of the boundary between them. Therefore, the second result S needs to be adjusted based on the number of channels in the first intermediate sub-network. i Perform channel mapping so that the mapped second result S i ′ and the first result T i The number of channels is the same.

[0089] Step 2: Apply the first result T output by the first intermediate sub-network to the preset boundary threshold. i Perform boundary correction to obtain the corrected first result T corresponding to the first intermediate sub-network. i ′.

[0090] Specifically, if the first result T i If the included feature value is greater than or equal to the preset boundary threshold, then the feature value remains unchanged; if the first result T i If the included feature value is less than the preset boundary threshold, then the feature value is modified to the preset boundary threshold.

[0091] This can be expressed by the formula:

[0092]

[0093] In formula (3), M represents the preset boundary threshold; Y represents the first result T. i Included feature values. Optionally, the preset boundary threshold is negative and greater than the smallest feature value in the first result. Optionally, M = -1.

[0094] By setting a boundary threshold, the negative value boundary in the first result is narrowed, changing the learning constraints. This allows the student model to learn the knowledge mapped by the negative feature values ​​in the first result, reducing the learning burden on the student model and helping to prevent overfitting. In other words, this scheme uses boundary mean square loss not only to enable the student model to learn the knowledge mapped by the negative values ​​of the teacher model's output, but also to learn whether the teacher model activates this output.

[0095] After performing the channel mapping in step one and the boundary correction in step two, the boundary mean square loss function MarginMSE() is used to calculate and obtain the first result T. i Second result S i The boundary mean square error between them can be denoted as the fourth loss information. Specifically, it can be calculated using formula (4):

[0096] MarginMSE(T i ,S i ) = MSE(T i ′,S i ′) Formula (4)

[0097] In formula (4), MSE() represents the mean square error function.

[0098] By calculating the first result output by the first intermediate subnetwork and the second result output by the second intermediate subnetwork for each corresponding group, n fourth loss information is obtained. Then, the n fourth loss information is substituted into the above formula (2) to obtain the second loss information.

[0099] The original loss function of the student model. gt :

[0100] Among them, the original loss function of the student model is Loss gt It is calculated based on the truth labels carried by the sample data and the second result output by the second intermediate subnetwork.

[0101] The original loss function of the student model. gt For example, it could be the KL divergence function, the mean square loss function, etc.

[0102] The model training method provided in this disclosure avoids the problems of selecting and setting the distillation position and the number of distillation layers by selecting each first intermediate subnetwork of the teacher model for knowledge distillation, and can achieve better distillation results in the intermediate subnetwork distillation task.

[0103] In addition, this embodiment uses boundary mean square error loss for knowledge distillation, which not only enables the student model to learn the output of the teacher model, but also enables the student model to learn whether the teacher model activates the output, thus enabling the student model to learn more knowledge and improving the performance of the target student model.

[0104] exist Figure 1 Based on the illustrated embodiment, after S102, the target student model can also be deployed online to perform the corresponding target task.

[0105] To more clearly illustrate the model training method provided in this disclosure, the following example uses a convolutional neural network, where both teacher model 10 and student model 20 are used to perform image recognition tasks, with teacher network 10 including 8 first intermediate subnetworks and student model 20 including 8 second intermediate subnetworks, to provide a detailed introduction to the model training method provided in this disclosure.

[0106] Specifically, Figure 2 This is a schematic diagram of the model training framework corresponding to the model training method provided in this disclosure. For example... Figure 2As shown, the teacher model 10 includes a first input subnetwork 11, eight first intermediate subnetworks 12, and a first output subnetwork 13, and the above subnetworks are connected sequentially from the first input subnetwork 11 to the first output subnetwork 13; the student model 20 includes a second input subnetwork 21, eight second intermediate subnetworks 22, and a second output subnetwork 23, and the above subnetworks are connected sequentially from the second input subnetwork 21 to the second output subnetwork 23.

[0107] It should be noted that the first intermediate subnetwork 12 included in the teacher network 10 corresponds one-to-one with the second intermediate subnetwork 22 included in the student model 20, and for each pair of first intermediate subnetwork 12 and second intermediate subnetwork 22 with corresponding relationship, the complexity of the first intermediate subnetwork 12 is higher than that of the second intermediate subnetwork 22.

[0108] It should be noted that the network structures of the first input subnetwork 11 included in the teacher model 10 and the second input subnetwork 21 included in the student model 20 may be the same or different. Similarly, the first output subnetwork 13 included in the teacher model 10 and the second output subnetwork 23 included in the student model 20 may be the same or different. This disclosure does not impose any restrictions in this regard.

[0109] The teacher model 10 includes eight second intermediate sub-networks 22, each of which is connected to the mapping module 30. The mapping module 30 is used to perform channel mapping on the second result output by the second intermediate sub-network 22 according to the number of channels of each first intermediate sub-network 12.

[0110] In practical applications, the mapping module 30 may include multiple mapping sub-modules 31. In this embodiment, the mapping module 30 includes eight mapping sub-modules 31. Each mapping sub-module 31 is used to process the channel mapping between the first result output by the first intermediate sub-network 12 and the second result output by the second intermediate sub-network 22, which have a corresponding relationship.

[0111] A loss function calculation module 40 is also included between the teacher model 10 and the student model 20. The loss function calculation module 40 is connected to the mapping module 30 and is used to obtain the second result after mapping. The loss function calculation module 40 is also connected to the eight first intermediate sub-networks 12 included in the teacher model 10 and is used to obtain the first result output by each first intermediate sub-network 12. The loss function calculation module 40 is used to calculate and obtain the first loss information based on the second result after mapping, the first results output by each first intermediate sub-network 12, and the ground truth label of the sample data.

[0112] Both the mapping module 30 and the loss function calculation module 40 can be deployed in the model training device.

[0113] In this embodiment, it is assumed that among the eight first intermediate sub-networks 12 included in the teacher model 10, the first to fourth first intermediate sub-networks 12 are all sub-networks that perform downsampling, and the fifth to eighth first intermediate sub-networks 12 are sub-networks that perform upsampling.

[0114] In this case, the last layer of the first intermediate sub-network 12 is not an activation function layer, the last layer of the second and third intermediate sub-networks 12 is an activation function layer, and the last layer of the fourth to eighth intermediate sub-networks 12 is not an activation function layer.

[0115] Similarly, among the eight second intermediate subnetworks 22 included in the student model 20, the first to fourth second intermediate subnetworks 22 are all subnetworks that perform downsampling, and the fifth to eighth second intermediate subnetworks 22 are subnetworks that perform upsampling.

[0116] In this case, the last layer of the first second intermediate sub-network 22 is not an activation function layer, the last layer of the second and third second intermediate sub-networks 22 is an activation function layer, and the last layer of the fourth to eighth second intermediate sub-networks 22 is not an activation function layer.

[0117] It should be understood that in practical applications, whether the first intermediate sub-network 12 and the second intermediate sub-network 22 are equipped with activation function layers, and the location of the activation function layers, can be set according to actual needs; this embodiment is only for illustrating the solution and is not a specific limitation on setting activation function layers in the first intermediate sub-network 12 and the second intermediate sub-network 22.

[0118] Based on the relationship between the aforementioned teacher model 10 and student model 20, it can be seen that for an intermediate subnetwork whose last layer is an activation function layer, the distillation position is the output position of the layer preceding the activation function layer; for an intermediate subnetwork whose last layer is not an activation function layer, the distillation position is the output position of the last layer of that intermediate subnetwork.

[0119] Specifically, the distillation positions of each first intermediate sub-network 12 and each second intermediate sub-network 22 are as follows: Figure 2 Marked in. In Figure 2 In the diagram, the dashed boxes represent activation function layers, the black solid circles mark the output positions of the layers preceding the activation function layers, and the black solid triangles mark the output positions of the last layers of the intermediate subnetworks.

[0120] When training the model, the sample data corresponding to the target task, i.e., image sample A1, is input into the first input subnetwork 11 of the teacher model 10 and the second input subnetwork 21 of the student model 20, respectively.

[0121] Specifically, the first input sub-network 11 of the teacher model 10 is used to perform feature extraction and other processing on the image sample A1 to obtain a feature map, and input the feature map to the first first intermediate sub-network 12 so that the first first intermediate sub-network 12 performs downsampling convolution and other processing on the input feature map. The first first intermediate sub-network 12 inputs the processed feature map to the second first intermediate sub-network 12, which performs downsampling convolution and other processing on the input feature map. And so on. Each first intermediate sub-network 12 obtains the input feature map from the connected previous sub-network, performs relevant processing, and outputs it to the connected next sub-network. Finally, the first output sub-network 13 outputs the image recognition result.

[0122] In the above process, the first result is output at the distillation position corresponding to each of the first intermediate sub-networks 12, and is denoted as T1, T2, T3, T4, T5, T6, T7, and T8 in sequence.

[0123] Specifically, the second input sub-network 21 of the student model 20 is used to perform feature extraction and other processing on the image sample A1 to obtain a feature map, and input the feature map to the first second intermediate sub-network 22 so that the first second intermediate sub-network 22 performs downsampling convolution and other processing on the input feature map. The first second intermediate sub-network 22 inputs the processed feature map to the second second intermediate sub-network 22, which performs downsampling convolution and other processing on the input feature map. And so on. Each first intermediate sub-network 12 obtains the input feature map from the connected previous sub-network, performs relevant processing, and outputs it to the connected next sub-network. Finally, the second output sub-network 23 outputs the image recognition result.

[0124] In the above process, the distillation position corresponding to each of the second intermediate sub-networks 22 outputs the second result, which is denoted as S1, S2, S3, S4, S5, S6, S7, and S8 in sequence.

[0125] Each mapping submodule 31 in the mapping module 30 performs channel mapping on the above-mentioned second results S1, S2, S3, S4, S5, S6, S7, and S8 according to the number of channels in the corresponding first intermediate sub-network, and obtains the mapped second results, denoted as S1', S2', S3', S4', S5', S6', S7', and S8'.

[0126] The loss function calculation module 40 obtains the first results T1, T2, T3, T4, T5, T6, T7, and T8 from each of the first intermediate sub-networks 12; and obtains the mapped second results S1', S2', S3', S4', S5', S6', S7', and S8' from the mapping module 30. Then, according to the method shown in the above formula (3), each of the first results T1, T2, T3, T4, T5, T6, T7, and T8 is corrected according to the preset boundary threshold to obtain the corrected first results T1', T2', T3', T4', T5', T6', T7', and T8'; based on the corrected first results and the mapped second results, the knowledge distillation loss value, i.e., the second loss information, is calculated using the above formulas (4) and (2). The loss function calculation module 40 is also used to calculate the original loss value of the student model, i.e. the third loss information, based on the truth label carried by the image sample A1 and each second result; and to calculate the first loss information based on the second loss information and the third loss information according to the above formula (1).

[0127] Afterwards, the loss function calculation module 40 can pass the first loss information to the student model 20, driving the student model 20 to update according to the first loss information. The updated student model 20 then serves as the student model for the next knowledge distillation.

[0128] Next, image samples are input into the teacher model and the updated student model 20, and the above knowledge distillation training process is executed again. By inputting a large number of image samples, knowledge distillation of the intermediate subnetwork is continuously performed until the preset convergence condition is met, and the student model obtained after the last knowledge distillation is used as the target student model.

[0129] After obtaining the target student model, it can be deployed online to perform target tasks such as image recognition.

[0130] Figure 3 This is a schematic diagram of the structure of a model training apparatus provided in an embodiment of this disclosure. (Refer to...) Figure 3 As shown, the model training device 300 provided in this embodiment includes:

[0131] The acquisition module 301 is used to acquire the sample dataset, teacher model, and student model corresponding to the target task; wherein the teacher model is a model obtained through pre-training.

[0132] The model training module 302 is used to input the sample dataset into the teacher model and use the output of each first intermediate subnetwork of the teacher model to perform knowledge distillation training on the student model to obtain the target student model; wherein, the teacher model includes a first input subnetwork, at least one first intermediate subnetwork and a first output subnetwork.

[0133] In some possible designs, the model training module 302 is specifically used to obtain the target student model through the following steps:

[0134] (a) Input the sample data included in the sample dataset into the teacher model and the student model respectively, and obtain the first result output by each first intermediate subnetwork of the teacher model and the second result output by each second intermediate subnetwork of the student model respectively; wherein, the student model includes a second input subnetwork, at least one second intermediate subnetwork and a second output subnetwork, and the first intermediate subnetwork corresponds one-to-one with the second intermediate subnetwork;

[0135] (b) Update the student model based on the truth labels of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks;

[0136] Repeat steps (a) to (b) until the preset convergence condition is met to obtain the target student model.

[0137] In some possible designs, if the last layer of the intermediate subnetwork is an activation function layer, then the output of the intermediate subnetwork is the output of the penultimate layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

[0138] In some possible designs, if the last layer of the intermediate subnetwork is not an activation function layer, then the output of the intermediate subnetwork is the output of the last layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

[0139] In some possible designs, the model training module 302 is specifically used to obtain first loss information based on the ground truth labeling of the sample data, the first result output by each of the first intermediate sub-networks, and the second result output by each of the second intermediate sub-networks; and to update the student model based on the first loss information.

[0140] In some possible designs, the model training module 302 is specifically used to obtain second loss information based on the boundary mean square error between the first result output by each of the first intermediate sub-networks and the second result output by the corresponding second intermediate sub-network; to obtain third loss information based on the ground truth label of the sample data and the second result output by each of the second intermediate sub-networks; and to obtain the first loss information based on the second loss information and the third loss information.

[0141] In some possible designs, the model training module 302 is specifically used to, for each of the first intermediate sub-networks, perform channel mapping on the second result output by the corresponding second intermediate sub-network according to the number of channels of the first intermediate sub-network, to obtain the mapped second result; perform boundary correction on the first result output by the first intermediate sub-network according to a preset boundary threshold, to obtain the corrected first result corresponding to the first intermediate sub-network; obtain the fourth loss information corresponding to the first intermediate sub-network according to the mean square error between the corrected first result corresponding to the first intermediate sub-network and the mapped second result; and sum or average the fourth loss information corresponding to each of the first intermediate sub-networks to obtain the second loss information.

[0142] In some possible designs, if the feature value included in the first result is greater than or equal to the preset boundary threshold, the feature value is kept unchanged; if the feature value included in the first result is less than the preset boundary threshold, the feature value is modified to the preset boundary threshold.

[0143] The model training device provided in this embodiment can be used to execute any of the above method embodiments. Its implementation principle and technical effect are similar, and can be referred to the description of the foregoing embodiments, which will not be repeated here.

[0144] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. (Refer to...) Figure 4 As shown, the electronic device 400 provided in this embodiment includes a memory 401 and a processor 402.

[0145] The memory 401 can be a separate physical unit, connected to the processor 402 via a bus 403. Alternatively, the memory 401 and processor 402 can be integrated together, implemented in hardware, etc.

[0146] The memory 401 is used to store program instructions, which the processor 402 calls to execute the operations according to any of the above method embodiments.

[0147] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 400 may also include only the processor 402. The memory 401 for storing programs is located outside the electronic device 400, and the processor 402 is connected to the memory via circuits / wires to read and execute the programs stored in the memory.

[0148] Processor 402 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0149] The processor 402 may further include a hardware chip. This hardware chip may be an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), or any combination thereof.

[0150] Memory 401 may include volatile memory, such as random-access memory (RAM); memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0151] This disclosure also provides a readable storage medium including a computer program that, when executed by at least one processor of an electronic device, performs the method of any of the above embodiments.

[0152] This disclosure also provides a program product comprising a computer program stored in a readable storage medium, wherein at least one processor of the model training apparatus can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the model training apparatus to perform any of the above method embodiments.

[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that, include: Obtain the sample dataset, teacher model, and student model corresponding to the target task; wherein, the teacher model is a pre-trained model, and the target task is an image recognition task; The sample dataset is input into the teacher model, and the student model is trained by knowledge distillation using the output of each first intermediate subnetwork of the teacher model to obtain the target student model; wherein, the teacher model includes a first input subnetwork, multiple first intermediate subnetworks, and a first output subnetwork; The step of inputting the sample dataset into the teacher model and using the output of each first intermediate subnetwork of the teacher model to perform knowledge distillation training on the student model to obtain the target student model includes: (a) Input the sample data included in the sample dataset into the teacher model and the student model respectively, and obtain the first result output by each first intermediate subnetwork of the teacher model and the second result output by each second intermediate subnetwork of the student model respectively; The student model includes a second input subnetwork, multiple second intermediate subnetworks, and a second output subnetwork, and the first intermediate subnetwork corresponds one-to-one with the second intermediate subnetwork. (b) For each of the first intermediate sub-networks, if the feature value included in the first result is greater than or equal to a preset boundary threshold, the feature value remains unchanged; if the feature value included in the first result is less than the preset boundary threshold, the feature value is modified to the preset boundary threshold to obtain the corrected first result corresponding to the first intermediate sub-network; based on the boundary mean square error between the corrected first result corresponding to the first intermediate sub-network and the second result output by the corresponding second intermediate sub-network, second loss information is obtained; based on the ground truth label of the sample data and the second result output by each of the second intermediate sub-networks, third loss information is obtained; based on the second loss information and the third loss information, first loss information is obtained; the student model is updated based on the first loss information. Repeat steps (a) to (b) until the preset convergence condition is met to obtain the target student model.

2. The method according to claim 1, characterized in that, If the last layer of the intermediate subnetwork is an activation function layer, then the output of the intermediate subnetwork is the output of the penultimate layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

3. The method according to claim 1, characterized in that, If the last layer of the intermediate subnetwork is not an activation function layer, then the output of the intermediate subnetwork is the output of the last layer of the intermediate subnetwork; wherein, the intermediate subnetwork is the first intermediate subnetwork of the teacher model or the second intermediate subnetwork of the student model.

4. The method according to claim 1, characterized in that, The step of obtaining the second loss information based on the boundary mean square error between the corrected first result corresponding to the first intermediate sub-network and the second result output by the corresponding second intermediate sub-network includes: For each of the first intermediate sub-networks, channel mapping is performed on the second result output by the corresponding second intermediate sub-network according to the number of channels of the first intermediate sub-network to obtain the mapped second result; Based on the mean square error between the corrected first result and the mapped second result corresponding to the first intermediate sub-network, the fourth loss information corresponding to the first intermediate sub-network is obtained. The second loss information is obtained by summing or averaging the fourth loss information corresponding to each of the first intermediate sub-networks.

5. A model training device, characterized in that, include: The acquisition module is used to acquire the sample dataset, teacher model, and student model corresponding to the target task; wherein, the teacher model is a model obtained through pre-training, and the target task is an image recognition task; The model training module is used to input the sample dataset into the teacher model and use the output of each first intermediate subnetwork of the teacher model to perform knowledge distillation training on the student model to obtain the target student model; wherein, the teacher model includes a first input subnetwork, multiple first intermediate subnetworks and a first output subnetwork; The model training module is specifically used for: (a) Input the sample data included in the sample dataset into the teacher model and the student model respectively, and obtain the first result output by each first intermediate subnetwork of the teacher model and the second result output by each second intermediate subnetwork of the student model respectively; The student model includes a second input subnetwork, multiple second intermediate subnetworks, and a second output subnetwork, and the first intermediate subnetwork corresponds one-to-one with the second intermediate subnetwork. (b) For each of the first intermediate sub-networks, if the feature value included in the first result is greater than or equal to a preset boundary threshold, the feature value remains unchanged; if the feature value included in the first result is less than the preset boundary threshold, the feature value is modified to the preset boundary threshold to obtain the corrected first result corresponding to the first intermediate sub-network; based on the boundary mean square error between the corrected first result corresponding to the first intermediate sub-network and the second result output by the corresponding second intermediate sub-network, second loss information is obtained; based on the ground truth label of the sample data and the second result output by each of the second intermediate sub-networks, third loss information is obtained; based on the second loss information and the third loss information, first loss information is obtained; the student model is updated based on the first loss information. Repeat steps (a) to (b) until the preset convergence condition is met to obtain the target student model.

6. An electronic device, characterized in that, include: Memory, processor, and computer program instructions; The memory is configured to store the computer program instructions; The processor is configured to execute the computer program instructions to perform the method as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, include: program; When the program is executed by the processor of the electronic device, it performs the method as described in any one of claims 1 to 4.

Images