Model training method and device, and readable storage medium

Abstract

The present disclosure relates to a model training method and device, and a readable storage medium, wherein the method comprises: obtaining a first sample data set, a first teacher model and a first student model corresponding to a target task; wherein the first sample data set comprises at least one first sample data; the first teacher model and the first student model both comprise pre-trained models; performing knowledge distillation according to the first sample data set, the first teacher model and the first student model until a preset convergence condition is met, and obtaining a target student model. Since the first student model is a pre-trained model and has certain performance, the present disclosure can improve the convergence speed of the first student model and improve the knowledge distillation efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer processing, and particularly relates to a model training method and device and readable storage medium. BACKGROUND

[0002] Deep neural networks are increasingly applied in various tasks, and the more complex the task is, the larger the scale of the deep neural network is, and the more computing resources the deep neural network consumes. Therefore, model compression technology is increasingly concerned in actual demand.

[0003] Knowledge distillation is one of the important methods for deep neural network compression. Specifically, a large-scale model is pre-trained as a teacher model, and a small-scale model is selected as a student model, the output of the teacher model is learned by the student model, and a trained student model is obtained. The trained student model is close to the teacher model in performance, but smaller than the teacher model in scale. However, the student model converges slowly and the knowledge distillation efficiency is low by using the above method. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a model training method, device and readable storage medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a model training method, comprising:

[0006] obtaining a first sample data set corresponding to a target task, a first teacher model and a first student model; wherein the first sample data set comprises at least one first sample data; the first teacher model and the first student model each comprise a pre-trained model;

[0007] performing knowledge distillation according to the first sample data set, the first teacher model and the first student model until a preset convergence condition is met, and obtaining a target student model.

[0008] In some possible designs, the method further comprises:

[0009] training an untrained second student model according to a second sample data set corresponding to the target task, and obtaining the pre-trained first student model; wherein the second sample data set comprises at least one second sample data, and each second sample data comprises a corresponding true value label.

[0010] In some possible designs, the method further comprises:

[0011] According to a third sample data set corresponding to the target task, the untrained second teacher model is trained to obtain the pre-trained first teacher model; wherein the third sample data set includes at least one third sample data, and each third sample data includes a corresponding true value label.

[0012] In some possible designs, the knowledge distillation is performed according to the first sample data set, the first teacher model and the first student model until a preset convergence condition is met, and a target student model is obtained, including:

[0013] (A) inputting first sample data in the first sample data set into the first teacher model and the first student model respectively to obtain a first result output by the first teacher model and a second result output by the first student model;

[0014] (B) obtaining first loss information according to the first result, the second result and a true value label of the first sample data;

[0015] (C) updating a weight coefficient of each parameter in the first student model according to the first loss information;

[0016] If the preset convergence condition is met, the target student model is obtained according to the updated first student model; if the preset convergence condition is not met, steps (A) to (C) are executed until the preset convergence condition is met, and the target student model is obtained.

[0017] In some possible designs, the first loss information is obtained according to the first result, the second result and the true value label of the first sample data, including:

[0018] second loss information is obtained according to the first result and the second result;

[0019] third loss information is obtained according to the first result and the true value label of the first sample data;

[0020] the first loss information is obtained according to the second loss information and the third loss information.

[0021] In some possible designs, if the first result includes a result output by an intermediate layer of the first teacher model, and the second result includes a result output by an intermediate layer of the first student model;

[0022] the second loss information is obtained according to the first result and the second result, including:

[0023] obtain the loss information corresponding to the intermediate layer of the first student model according to a mean square error between a result of an intermediate layer output of the first teacher model and a result of an intermediate layer output of the first student model;

[0024] The second loss information includes the loss information corresponding to the output layer of the first student model.

[0025] In some possible designs, the first result includes a result of an output layer output of the first teacher model, and the second result includes a result of an output layer output of the first student model.

[0026] The second loss information includes the loss information corresponding to the output layer of the first student model.

[0027] obtain the loss information corresponding to the output layer of the first student model according to a KL divergence between a result of an output layer output of the first teacher model and a result of an output layer output of the first student model;

[0028] The second loss information includes the loss information corresponding to the output layer of the first student model.

[0029] In some possible designs, the method further includes: performing the target task according to the target student model.

[0030] In a second aspect, an embodiment of the present disclosure provides a model training apparatus, including:

[0031] The obtaining module is configured to obtain a first sample data set corresponding to a target task, a first teacher model, and a first student model; the first sample data set includes at least one first sample data; the first teacher model and the first student model each include a pre-trained model.

[0032] The knowledge distillation module is configured to perform knowledge distillation according to the first sample data set, the first teacher model, and the first student model until a preset convergence condition is met, to obtain a target student model.

[0033] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program.

[0034] The memory is configured to store the computer program.

[0035] The processor is configured to execute the computer program to implement the method in any one of the first aspect.

[0036] In a fourth aspect, the embodiments of the present disclosure provide a readable storage medium, comprising: a program; when the program is executed by a processor of an electronic device, the method of any one of the first aspect is implemented.

[0037] In a fifth aspect, the embodiments of the present disclosure also provide a program product, the program product comprising a computer program stored in a readable storage medium, at least one processor of the model training apparatus can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the model training apparatus to implement the method of any one of the first aspect.

[0038] The embodiments of the present disclosure provide a model training method and device and a readable storage medium, wherein the method comprises: obtaining a first sample data set, a first teacher model and a first student model corresponding to a target task; wherein the first sample data set comprises at least one first sample data; the first teacher model and the first student model both comprise a pre-trained model; performing knowledge distillation according to the first sample data set, the first teacher model and the first student model until a preset convergence condition is met, and obtaining a target student model. Since the pre-trained first student model already has certain performance, the method provided by the present disclosure can improve the convergence speed of the first student model and improve the knowledge distillation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0041] Figure 1 The flowchart of the model training method provided by an embodiment of the present disclosure;

[0042] Figure 2 The flowchart of the model training method provided by another embodiment of the present disclosure;

[0043] Figure 3 The flowchart of the model training method provided by another embodiment of the present disclosure;

[0044] Figure 4 The structural schematic diagram of the model training device provided by an embodiment of the present disclosure;

[0045] Figure 5A structural schematic diagram of an electronic device provided for another embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] In order to enable a more clear understanding of the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other manners different from those described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, and not all the embodiments.

[0048] The execution subject of the model training method provided by the present embodiment can be the model training device provided by the present disclosure, which can be realized by any software and / or hardware. Illustratively, the model training device is an electronic device such as a tablet computer, a mobile phone (such as a folding screen mobile phone, a large-screen mobile phone, etc.), a wearable electronic device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a server, a server cluster, etc. In the following embodiments, the execution subject is taken as an example of the model training device to introduce the model training method provided by the present disclosure in detail.

[0049] Figure 1 The flowchart of the model training method provided by the present embodiment is provided. As shown in the flowchart of the model training method provided by the present embodiment, Figure 1 The method provided by the present embodiment includes:

[0050] S101, obtaining a first sample data set corresponding to a target task, a first teacher model and a first student model; wherein the first sample data set includes at least one first sample data; the first teacher model and the first student model both include a pre-trained model.

[0051] In some cases, the above-mentioned first teacher model can also be referred to as a teacher model, and the above-mentioned first teacher model is a pre-trained teacher model, so it can also be referred to as a pre-trained teacher model, a pre-trained teacher model, etc. Other names; similarly, the above-mentioned first student model is also a pre-trained student model, so it can also be referred to as a pre-trained student model and other names. The present disclosure does not make any limitation on this.

[0052] The first teacher model and the first student model are pre-trained machine learning models for the same target task.

[0053] The first teacher model can include a machine learning model with a large size, a deep level and a large number of parameters, and the first student model can include a machine learning model with a small size, a shallow level and a small number of parameters.

[0054] For example, assuming that the target task is an image recognition task, the first teacher model and the first student model are both image recognition models. Assuming that the first teacher model has 10 intermediate layers and the first student model has 5 intermediate layers.

[0055] In the present scheme, the model training device (i.e., the device for knowledge distillation of the model) stores the architecture of the pre-trained first teacher model and the weight coefficients of the parameters corresponding to the first teacher model. Similarly, the model training device can also store the architecture of the pre-trained first student model and the weight coefficients of the parameters corresponding to the first student model.

[0056] When knowledge distillation of the first teacher model is needed, the model training device loads the architecture of the first teacher model and the weight coefficients of the parameters in the first teacher model, and performs assignment operation to obtain the pre-trained first teacher model. The model training device loads the architecture of the first student model and the weight coefficients of the parameters in the first student model, and performs assignment operation to obtain the pre-trained first student model.

[0057] For example, the process of obtaining the first teacher model and the first student model can be represented by formula (1) and formula (2):

[0058] ST(teacher) = ST(teacher-pretrained); formula (1)

[0059] ST(student) = ST(student-pretrained); formula (2)

[0060] In formula (1), ST(teacher) represents the weight coefficients of the parameters in the first teacher model; ST(teacher-pretrained) represents the weight coefficients of the parameters in the pre-trained first teacher model; "=" represents assignment operation. Formula (1) represents that the weight coefficients of the parameters in the pre-trained first teacher model are assigned to the first teacher model.

[0061] In formula (2), ST (student) represents a weight coefficient of a parameter in the first student model; ST (student-pretrained) represents a weight coefficient of a parameter in the pre-trained first student model; “=” represents an assignment operation. Formula (2) represents that the weight coefficient of the parameter in the pre-trained first student model is assigned to the first student model.

[0062] S102, knowledge distillation is performed according to the first sample data set, the first teacher model and the first student model until a preset convergence condition is met, and a target student model is obtained.

[0063] Specifically, in combination with Figure 1 and Figure 2 It is shown that the knowledge distillation can include the following steps:

[0064] Step a: input each first sample data in the first sample data set to the first teacher model to obtain a first result corresponding to each first sample data output by the first teacher model.

[0065] Step b: input each first sample data in the first sample data set to the first student model, train the first student model, and obtain a second result corresponding to each first sample data output by the first student model.

[0066] Step c: according to the first result corresponding to each first sample data, the second result corresponding to each first sample data and the true value label in each first sample data, the first loss information is calculated.

[0067] Step d: adjust the weight coefficient of the target parameter included in the first student model according to the first loss information.

[0068] Return to execute steps (a) to (d) until the preset convergence condition is met, and a trained target student model is obtained.

[0069] In the present scheme, the first loss information can be obtained according to the second loss information and the third loss information, wherein the first loss information can be recorded as Loss total , the second loss information can be recorded as Loss distill , and the third loss information can be recorded as Loss gt . The second loss information includes a knowledge distillation loss, and the second loss information can be calculated according to the first result and the second result; the third loss information includes the original loss of the first student model, and the third loss information can be obtained according to the first result and the true value label.

[0070] Exemplarily, the first loss information can satisfy formula (3):

[0071] Loss total= λ1*Loss distill + λ2*Loss gt Formula (3)

[0072] wherein λ1 represents a weight coefficient of the second loss information; and λ2 represents a weight coefficient of the third loss information.

[0073] Optionally, λ2 can be set as a constant, for example, constant 1. In the process of knowledge distillation, the proportion of the second loss information and the third loss information can be adjusted by adjusting the value of λ1.

[0074] In actual application, knowledge distillation can be performed on a specific layer (i.e., the distillation layer) in the first teacher model, for example, knowledge distillation can be performed on the input layer, the intermediate layer or the output layer of the first teacher model to migrate the knowledge of the distillation layer of the first teacher model to the first student model; or knowledge distillation can be performed on multiple different layers (i.e., the distillation layers) in the first teacher model, for example, knowledge distillation can be performed on multiple layers among the input layer, the intermediate layer and the output layer of the first teacher model to migrate the knowledge of the distillation layer of the first teacher model to the first student model.

[0075] The above two cases are illustrated by several examples as follows:

[0076] I. Knowledge distillation is performed on the intermediate layer of the first teacher model alone:

[0077] Step s1: input each first sample data into the first teacher model and the first student model to obtain the result of the intermediate layer output of the first teacher model (i.e., the first result) and the result of the intermediate layer output of the first student model (i.e., the second result).

[0078] Step s2: calculate the mean square error of the result of the intermediate layer output of the first teacher model and the result of the intermediate layer output of the first student model to obtain a mean square error value. The mean square error value is the loss information (i.e., the second loss information) corresponding to the intermediate layer of the first student model.

[0079] Step s3: calculate the third loss information according to the true value label in the first sample data and the result of the intermediate layer output of the first student model; and calculate the first loss information by summing or weighted summing the mean square error value and the third loss information according to the weight coefficients corresponding to the mean square error value and the third loss information.

[0080] Step s4: update the weight coefficients of all or part of the parameters of the intermediate layer of the first student model according to the first loss information to obtain the updated first student model.

[0081] If the updated first student model meets the preset convergence condition, a target student model is obtained; if the updated first student model does not meet the preset convergence condition, steps s1 to s4 are returned to be executed until the preset convergence condition is met, and the target student model is obtained.

[0082] II. Knowledge distillation is performed on the output layer of the first teacher model alone:

[0083] Step p1: each first sample data is input into the first teacher model and the first student model respectively, and a result output by an output layer of the first teacher model (i.e., a first result) and a result output by an output layer of the first student model (i.e., a second result) are obtained.

[0084] Step p2: a KL divergence between the result output by the output layer of the first teacher model and the result output by the output layer of the first student model is calculated, and a KL divergence value is obtained. The KL divergence value is loss information (i.e., second loss information) corresponding to the output layer of the first student model.

[0085] Step p3: third loss information is obtained by calculating a true value label in the first sample data and the result output by the output layer of the first student model; the KL divergence value and the third loss information, and the weight coefficients corresponding to the KL divergence value and the third loss information respectively, are summed or weighted summed, and first loss information is obtained.

[0086] The KL (Kullback-Leibler Divergence) divergence can also be referred to as relative entropy, information divergence, or other names.

[0087] Step p4: the weight coefficients of the parameters of the output layer of the first student model are updated according to the first loss information, and an updated first student model is obtained.

[0088] If the updated first student model meets the preset convergence condition, a target student model is obtained; if the updated first student model does not meet the preset convergence condition, steps p1 to p4 are returned to be executed until the preset convergence condition is met, and the target student model is obtained.

[0089] III. Knowledge distillation is performed on the intermediate layer and the output layer of the first teacher model simultaneously:

[0090] Step k1: each first sample data is input into the first teacher model and the first student model respectively, and results output by the intermediate layer and the output layer of the first teacher model and results output by the intermediate layer and the output layer of the first student model are obtained.

[0091] That is, the first result includes: the result of the intermediate layer output of the first teacher model, and the result of the output layer output of the first teacher model; and the second result includes: the result of the intermediate layer output of the first student model, and the result of the output layer output of the first student model.

[0092] Step k2: according to the result of the intermediate layer output of the first teacher model and the result of the intermediate layer output of the first student model, using the first loss function (such as the mean square error function) constructed in advance, the loss information Loss distill(1) corresponding to the intermediate layer of the first student model is obtained; and the result of the output layer output of the first teacher model and the result of the output layer output of the first student model are calculated, using the second loss function (such as the KL divergence function) constructed in advance, to obtain the loss information Loss distill(2) corresponding to the output layer of the first student model; the loss information Loss distill(1) corresponding to the intermediate layer of the first student model and the loss information Loss distill(2) corresponding to the output layer of the first student model are summed or weighted to obtain the final second loss information Loss distill .

[0093] Step k3: according to the result of the intermediate layer output of the first student model and the true value label carried by the first sample data, using the third loss function constructed in advance, the third loss information Loss gt(1) corresponding to the intermediate layer of the first student model is calculated and obtained; and according to the result of the output layer output of the first student model and the true value label carried by the first sample data, the third loss information Loss gt(2) corresponding to the output layer of the first student model is calculated and obtained; the third loss information Loss gt(1) and the third loss information Loss gt(2) are summed or weighted to obtain the final third loss information Loss gt .

[0094] The second loss information Loss distill and the third loss information Loss gt obtained finally are summed or weighted according to the respective weight coefficients to obtain the first loss information; and according to the first loss information, the weight coefficients of the parameters of the intermediate layer and the output layer of the first student model are updated to obtain the updated first student model.

[0095] If the updated first student model satisfies the preset convergence condition, the target student model is obtained; if the updated first student model does not satisfy the preset convergence condition, steps k1 to k3 are returned to be executed until the preset convergence condition is satisfied, and the target student model is obtained.

[0096] The preset convergence condition can include, but is not limited to, a preset evaluation index or a preset number of iterations.

[0097] The method provided in this embodiment comprises the following steps.

[0098] Figure 3 The method provided in this embodiment comprises the following steps. Figure 3 The method provided in this embodiment comprises the following steps.

[0099] S301, training an untrained second student model according to a second sample data set corresponding to the target task, to obtain a pre-trained first student model.

[0100] The weight coefficients of the parameters in the untrained second student model can be determined through random initialization, or can be preset.

[0101] The second sample data set comprises at least one second sample data, and each second sample data comprises a corresponding true value label. It should be noted that the second sample data set and the first sample data set can be the same or different, and the embodiments of the present disclosure do not limit this. In some cases, there can be an intersection between the first sample data included in the first sample data set and the second sample data included in the second sample data set.

[0102] Specifically, the second sample data is input into the untrained second student model to obtain a prediction result output by the second student model; a loss value is obtained according to the prediction result, the true value label in the second sample data, and a loss function of the second student model constructed in advance; and the weight coefficients of the target parameters of the second student model are adjusted according to the loss value. The above steps are repeatedly executed until a convergence condition of the pre-training phase is met, and the pre-trained first student model is obtained.

[0103] For the second student model, the convergence condition of the pre-training phase can be, but is not limited to, a preset evaluation index or a preset number of iterations.

[0104] S302, training an untrained second teacher model according to a third sample data set corresponding to the target task, to obtain a pre-trained first teacher model.

[0105] The weight coefficients of the parameters in the untrained second teacher model can be determined by random initialization or can be preset.

[0106] The third sample data set includes at least one third sample data, and each third sample data includes a corresponding true value label. The third sample data set can be the same as or different from the second sample data set. The more sample data included in the third sample data set, the better the performance of the first teacher model obtained, and therefore, as many sample data as possible should be used to obtain the first teacher model in actual applications.

[0107] Specifically, the third sample data is input into the second teacher model to obtain a prediction result output by the second teacher model; a loss value is obtained according to the prediction result, a true value label of the third sample data, and a loss function of the second teacher model; and the weight coefficients of the target parameters of the second teacher model are adjusted according to the loss value. The above steps are repeatedly executed until a convergence condition of the pre-training phase is met, and the model training device obtains the first teacher model.

[0108] For the second teacher model, the convergence condition of the pre-training phase can be, but is not limited to, a preset evaluation index or a preset number of iterations, and the like.

[0109] It should be noted that the execution order of S301 and S302 is not limited. S301 and S302 can be executed on an integrated device or on a distributed device, and the embodiments of the present disclosure do not limit this.

[0110] For example, when the model training device is a server cluster composed of multiple independent servers, S301 can be executed on one of the servers (assuming server A), and S302 can be executed on another server (assuming server B).

[0111] On this basis, S303 and S304 can be executed on any server in the server cluster, for example, can be executed on server A, can be executed on server B, or can be executed on another server, such as server C.

[0112] S303, obtaining a first sample data set corresponding to the target task, a first teacher model, and a first student model; the first sample data set includes at least one first sample data, and the first teacher model and the first student model each include a pre-trained model.

[0113] S304, performing knowledge distillation according to the first sample data set, the first teacher model, and the first student model until a preset convergence condition is met, to obtain a target student model.

[0114] The steps S303 and S304 of the embodiment are similar to S101 and S102 of the embodiment shown in Figure 1 The detailed description of the embodiment shown in the embodiment S101 and S102 can be referred to Figure 1 The detailed description of the embodiment shown in the embodiment S101 and S102 can be referred to

[0115] In the embodiment, the untrained second student model is trained on the target task to obtain the pre-trained first student model, and the untrained second teacher model is trained on the target task to obtain the pre-trained first teacher model. Then, knowledge distillation is performed according to the first student model and the first teacher model. Since the first student model already has certain performance, the convergence speed of the first student model can be improved, and the knowledge distillation efficiency can be improved.

[0116] Figure 4 The structural schematic diagram of the model training device provided by an embodiment of the present disclosure is shown. Referring to Figure 4 The model training device 400 provided by the embodiment includes:

[0117] The acquisition module 401 is configured to acquire a first sample data set corresponding to a target task, a first teacher model, and a first student model. The first sample data set includes at least one first sample data. The first teacher model and the first student model each include a pre-trained model.

[0118] The knowledge distillation module 402 is configured to perform knowledge distillation according to the first sample data set, the first teacher model, and the first student model until a preset convergence condition is met, and obtain a target student model.

[0119] In some possible designs, the model training device 400 further includes a pre-training module 403.

[0120] The pre-training module 403 is configured to train an untrained second student model according to a second sample data set corresponding to the target task, and obtain a pre-trained first student model. The second sample data set includes at least one second sample data. Each second sample data includes a corresponding true value label.

[0121] In some possible designs, the pre-training module 403 is further configured to train an untrained second teacher model according to a third sample data set corresponding to the target task, and obtain a pre-trained first teacher model. The third sample data set includes at least one third sample data. Each third sample data includes a corresponding true value label.

[0122] In actual applications, the pre-training module 403 can include a first pre-training model and a second pre-training module Figure 4The first pre-training module is configured to train the untrained second student model according to a second sample data set corresponding to the target task, to obtain a pre-trained first student model; and the second pre-training module is configured to train the untrained second teacher model according to a third sample data set corresponding to the target task, to obtain a pre-trained first teacher model.

[0123] In some possible designs, the knowledge distillation module 402 is specifically configured to perform the following steps:

[0124] (A) inputting first sample data in the first sample data set into the first teacher model and the first student model respectively, to obtain a first result output by the first teacher model and a second result output by the first student model; (B) obtaining first loss information according to the first result, the second result, and a true value label in the first sample data; (C) updating weight coefficients of parameters in the first student model according to the first loss information; if a preset convergence condition is met, obtaining the target student model according to the updated first student model; if the preset convergence condition is not met, returning to perform the steps (A) to (C) until the target student model is obtained by meeting the preset convergence condition.

[0125] In some possible designs, the knowledge distillation module 402 is specifically configured to obtain second loss information according to the first result and the second result; obtain third loss information according to the first result and the true value label in the first sample data; and obtain the first loss information according to the second loss information and the third loss information.

[0126] In some possible designs, if the first result includes a result output by an intermediate layer of the first teacher model, and the second result includes a result output by an intermediate layer of the first student model, the knowledge distillation module 402 is specifically configured to obtain loss information corresponding to the intermediate layer of the first student model according to a mean square error between the result output by the intermediate layer of the first teacher model and the result output by the intermediate layer of the first student model; and the second loss information includes the loss information corresponding to the intermediate layer of the first student model.

[0127] In some possible designs, if the first result includes a result output by an output layer of the first teacher model, and the second result includes a result output by an output layer of the first student model, the knowledge distillation module 402 is specifically configured to obtain loss information corresponding to the output layer of the first student model according to a KL divergence between the result output by the output layer of the first teacher model and the result output by the output layer of the first student model; and the second loss information includes the loss information corresponding to the output layer of the first student model.

[0128] In some possible design, the model training apparatus 400 further includes a task execution module 404 (not shown in the figure) configured to execute the target task according to the target student model. Figure 4

[0129] The model training apparatus provided in this embodiment can be used to execute the technical solutions in any of the method embodiments, and has similar implementation principles and technical effects. Refer to the detailed description of the method embodiments, which will not be repeated here.

[0130] Figure 5 A structural schematic diagram of an electronic device is provided for an embodiment of the present disclosure. As shown in the figure, the electronic device 500 provided in this embodiment includes a memory 501 and a processor 502. Figure 5

[0131] The memory 501 can be an independent physical unit, and can be connected to the processor 502 through a bus 503. The memory 501 and the processor 502 can also be integrated together and implemented by hardware.

[0132] The memory 501 is configured to store program instructions, and the processor 502 invokes the program instructions to execute the technical solutions in any of the method embodiments.

[0133] Alternatively, when part or all of the method in the above embodiment is implemented by software, the electronic device 500 can only include the processor 502. The memory 501 for storing programs is located outside the electronic device 500, and the processor 502 is connected to the memory through a circuit / wire, for reading and executing the programs stored in the memory.

[0134] The processor 502 can be a central processing unit (CPU), a network processor (NP), or a combination of the CPU and the NP.

[0135] The processor 502 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. ​​

[0136] The memory 501 can include a volatile memory (Volatile Memory), such as a random-access memory (Random-Access Memory, RAM); the memory can also include a non-volatile memory (Non-volatile Memory), such as a flash memory (Flash Memory), a hard disk (Hard Disk Drive, HDD) or a solid-state disk (Solid-state Drive, SSD); the memory can also include a combination of the above-mentioned kinds of memories.

[0137] The embodiments of the present disclosure further provide a computer readable storage medium, which includes computer program instructions, and the computer program instructions are executed by at least one processor of an electronic device to perform the method of any one of the above embodiments.

[0138] The embodiments of the present disclosure further provide a program product, which includes a computer program stored in a readable storage medium, and at least one processor of the model training apparatus can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the model training apparatus to perform any one of the above method embodiments.

[0139] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element preceded by "comprises... " does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the recited element.

[0140] The above description is merely one specific implementation of the present disclosure, and persons skilled in the art can understand or implement the present disclosure based on the above description. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a first sample data set, a first teacher model and a first student model corresponding to a target task; wherein the first sample data set comprises at least one first sample data; the first teacher model and the first student model each comprise a pre-trained model, and the target task is an image recognition task; inputting each first sample data into the first teacher model and the first student model respectively; Based on the output results of the intermediate layers of the first teacher model and the first student model, the loss information (Loss) corresponding to the intermediate layer of the first student model is obtained using a pre-constructed mean squared error function. distill(1) Calculate the output results of the first teacher model and the first student model, and use the pre-constructed KL divergence function to obtain the loss information corresponding to the output layer of the first student model. distill(2) Based on the loss information corresponding to the intermediate layer of the first student model. distill(1) Loss information corresponding to the output layer of the first student model distill(2) Perform summation or weighted calculation to obtain loss information. distill ; According to the result of the intermediate layer output of the first student model and the true value label carried by the first sample data, the loss information Loss corresponding to the intermediate layer of the first student model is calculated by using a pre-constructed loss function gt(1) ; and according to the result of the output layer output of the first student model and the true value label carried by the first sample data, the loss information Loss corresponding to the output layer of the first student model is calculated gt(2) ; according to the loss information Loss gt(1) and the loss information Loss gt(2) , the final loss information Loss gt is obtained by summation or weighted calculation; to the loss information Loss distill and the loss information Loss gt , according to respective corresponding weight coefficients, summing or weighted summing is performed to obtain first loss information; and according to the first loss information, the weight coefficients of the parameters of the intermediate layer and the output layer of the first student model are updated until a preset convergence condition is met to obtain a target student model.

2. The method of claim 1, wherein, The method further comprises: training an untrained second student model according to a second sample data set corresponding to the target task, to obtain the pre-trained first student model; wherein the second sample data set comprises at least one second sample data, and each second sample data comprises a corresponding true value label.

3. The method of claim 1, wherein, The method further comprises: training an untrained second teacher model according to a third sample data set corresponding to the target task, to obtain the pre-trained first teacher model; wherein the third sample data set comprises at least one third sample data, and each third sample data comprises a corresponding true value label.

4. A model training apparatus characterized by comprising: The method comprises: an obtaining module configured to obtain a first sample data set, a first teacher model and a first student model corresponding to a target task; wherein the first sample data set comprises at least one first sample data; the first teacher model and the first student model each comprise a pre-trained model, and the target task is an image recognition task; a knowledge distillation module configured to input each first sample data into the first teacher model and the first student model respectively. According to the result of the intermediate layer output of the first teacher model and the result of the intermediate layer output of the first student model, the loss information Loss corresponding to the intermediate layer of the first student model is obtained by using a pre-constructed mean square error function distill(1) ; the result of the output layer output of the first teacher model and the result of the output layer output of the first student model are calculated, and the loss information Loss corresponding to the output layer of the first student model is obtained by using a pre-constructed KL divergence function distill(2) ; the loss information Loss corresponding to the intermediate layer of the first student model distill(1) , the loss information Loss corresponding to the output layer of the first student model distill(2) is summed or weighted, and the loss information Loss distill is obtained; According to the result of the intermediate layer output of the first student model and the true value label carried by the first sample data, the loss information Loss corresponding to the intermediate layer of the first student model is calculated by using the pre-constructed loss function gt(1) ; and according to the result of the output layer output of the first student model and the true value label carried by the first sample data, the loss information Loss corresponding to the output layer of the first student model is calculated gt(2) ; according to the loss information Loss gt(1) and the loss information Loss gt(2) , the final loss information Loss gt is obtained by summation or weighted calculation; to the loss information Loss distill and the loss information Loss gt , according to respective corresponding weight coefficients, summing or weighted summing is performed to obtain first loss information; and according to the first loss information, the weight coefficients of the parameters of the intermediate layer and the output layer of the first student model are updated until a preset convergence condition is met to obtain a target student model.

5. An electronic device, comprising: The method comprises: a memory, a processor and computer program instructions; the memory is configured to store the computer program instructions; the processor is configured to execute the computer program instructions to implement the method of any one of claims 1 to 3.

6. A readable storage medium characterized by, The method comprises: a program; when the program is executed by a processor of an electronic device, the method of any one of claims 1 to 3 is implemented.

Images