Learning device and learning method

Abstract

To provide a technique capable of appropriately improving the performance of a student model in the deep layer learning using the knowledge distillation.SOLUTION: Disclosed is an exemplary learning device which trains a student model using a teacher model and includes: an evaluation part for determining the performance difference between the teacher model and the student model; and a change part for changing the teacher model based on the performance difference or changing a weight coefficient at the time of calculating a loss of the student model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a learning device and a learning method. [Background technology]

[0002] Conventionally, a method called knowledge distillation has been known as one of the learning methods of deep learning (see, for example, Patent Document 1). In knowledge distillation, a large-scale teacher model with a large number of parameters is used to train a student model with a small number of parameters. Knowledge distillation can reduce the size of the model while suppressing the deterioration of the model's performance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-71883 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a learning method using knowledge distillation, in the latter half of learning when the performance of the student model approaches that of the teacher model, the performance of the student model may saturate, and the effect of improving performance by knowledge distillation may not be fully obtained. Also, if the performance of the teacher model used is too high, the performance of the student model may not be improved sufficiently.

[0005] In addition, in a learning method using knowledge distillation, there is a concern that, due to the difference in feature space between the teacher model and the student model, for example, bringing the feature map of the student model closer to the feature map of the teacher model may have a negative effect on the learning of the student model in the final stage of learning, which may deteriorate the performance of the student model.

[0006] In view of the above, an object of the present invention is to provide a technology that can appropriately improve the performance of a student model in deep learning using knowledge distillation. [Means for solving the problem]

[0007] An exemplary learning device of the present invention is a learning device that trains a student model using a teacher model, and includes an evaluation unit that determines the performance difference between the teacher model and the student model, and a modification unit that modifies the teacher model or modifies the weighting coefficient when calculating the loss of the student model based on the performance difference.

[0008] An exemplary learning method of the present invention is a learning method in which a teacher model is used to train a student model, and the teacher model is modified based on a performance difference between the teacher model and the student model.

[0009] An exemplary learning method of the present invention is a learning method for training a student model using a teacher model, in which the loss of the student model is calculated using the loss for training data and the loss for the teacher model, and the student model is trained by changing a weighting coefficient that adjusts the balance between the loss for the training data and the loss for the teacher model at a predetermined timing. Effect of the Invention

[0010] According to the exemplary present invention, in deep learning using knowledge distillation, the performance of a student model can be appropriately improved. [Brief description of the drawings]

[0011] [Figure 1] Block diagram showing the hardware configuration of a learning device [Diagram 2] A block diagram showing the functional configuration of a processor included in the learning device. [Diagram 3] FIG. 1 is a schematic diagram for explaining a configuration for changing a teacher model. [Figure 4]FIG. 1 is a schematic diagram showing how the performance of a student model changes when the student model is trained using the learning device of the first embodiment; [Diagram 5] A flowchart showing the flow of training a student model in the learning device according to the first embodiment. [Figure 6] A flowchart showing the flow of training a student model in a learning device according to a second embodiment. [Figure 7] A diagram showing an example in which the weighting factor λ is attenuated every several epochs. [Figure 8] A diagram showing an example of how the weighting factor λ is attenuated every epoch. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <1. First embodiment> FIG. 1 is a block diagram showing a hardware configuration of a learning device 10 according to a first embodiment of the present invention. The learning device 10 is a learning device for machine learning, and uses knowledge distillation as a learning method. Hereinafter, knowledge distillation may be simply referred to as "distillation". The learning device 10 is a learning device that trains a student model using a teacher model. The learning device 10 executes a learning method for training the student model using a teacher model. In detail, the learning device 10 trains the student model using training data and the teacher model. Note that in this embodiment, the learning device 10 trains the student model using multiple teacher models.

[0014] In this embodiment, the training data is training data with a correct answer label. The teacher model is a large-scale deep neural network with a large number of parameters, and is a model that has already been trained. The student model is a small-scale deep neural network with a smaller number of parameters than the teacher model. The student model is a model that will be used for learning (training).

[0015] As shown in FIG. 1, the learning device 10 includes a processor 11 and a storage unit 12. The storage unit 12 stores a teacher model and a student model. The teacher model includes learned parameters. The student model includes parameters that are updated by learning. In this embodiment, there are multiple teacher models, and the multiple teacher models are stored in the storage unit 12.

[0016] The processor 11 performs a function of performing inference on the training data using the teacher model stored in the storage unit 12, and inference on the training data using the student model. The processor 11 performs a function of updating the parameters of the student model so as to reduce an error between the inference result of the teacher model and the training result of the student model. Note that a feature map of the intermediate layer may be used instead of the inference result.

[0017] Fig. 2 is a block diagram showing a functional configuration of the processor 11 included in the learning device 10 according to the first embodiment of the present invention. The functions of the processor 11 are exerted by executing arithmetic processing in accordance with a program stored in the storage unit 12. As shown in Fig. 2, the processor 11 of this embodiment includes, as its functions, an acquisition unit 110, a first inference unit 111, a second inference unit 112, a learning unit 113, an evaluation unit 114, and a change unit 115. In other words, the learning device 10 includes the acquisition unit 110, the first inference unit 111, the second inference unit 112, the learning unit 113, the evaluation unit 114, and the change unit 115. The acquisition unit 110 acquires learning data.

[0018] The first inference unit 111 performs inference on the input learning data using a trained teacher model. The function of the first inference unit 111 is realized by the processor 11 reading the teacher model stored in the storage unit 12. Through the inference, an inference result that is the final result of the inference, a feature map that is an intermediate output, and the like are obtained. In this embodiment, the learning data is image data, and the first inference unit 111 performs object detection on the input image data. The inference result of the first inference unit 111 includes identification of the type and position of the object. The type of object is, for example, a pedestrian, a bicycle, a car, a traffic light, and the like. A bounding box is used to identify the position.

[0019] The second inference unit 112 performs inference on the input learning data using a student model that is the subject of training by the teacher model. The function of the second inference unit 112 is realized by the processor 11 reading the student model stored in the storage unit 12. Through the inference, an inference result that is the final result of the inference, a feature map that is an intermediate output, and the like are obtained. The parameters of the student model are successively updated through learning. In this embodiment, as described above, the learning data is image data, and the second inference unit 112 performs object detection on the input image data. As with the first inference unit 111, the inference result of the second inference unit 112 includes identification of the type and position of the object.

[0020] The learning unit 113 calculates the loss (distillation loss Ldstl) of the student model against the teacher model. The loss (distillation loss Ldstl) against the teacher model is the error of the inference result between the teacher model and the student model, or the error of the feature map of the intermediate layer (intermediate output). This makes it possible to easily calculate the loss function using conventional technology.

[0021] In this embodiment, the distillation loss Ldstl is an error between the feature map of the teacher model and the feature map of the student model. The distillation loss Ldstl may be, for example, the L2 loss represented by the following formula (1). However, the distillation loss Ldstl may be other than the L2 loss, for example, the KL divergence representing the loss between the distribution of the output of the teacher model and the student model.

[0022]

number

[0023] Furthermore, the learning unit 113 calculates a loss for the learning data of the student model. In detail, the learning unit 113 calculates a learning task-specific error (loss) for the correct label of the learning data of the student model. In this embodiment, the learning task is object detection, and the learning task-specific loss Luq includes, for example, a classification loss Lcls and a bounding box regression loss Lbbox. The learning task-specific loss Luq in this embodiment is calculated, for example, by the following formula (2).

[0024] Note that the following formula (2) assumes the use of YOLO, an example of an object detection algorithm. In YOLO, the input image is divided into an S x S grid, and for each grid, the center coordinates (x, y), width and height scales (w, h), and probability (C) of an object existing within B rectangular regions with a certain aspect ratio are predicted. In addition, when an object exists in each rectangular region, the posterior probability p(c) indicating which class the object belongs to is also predicted. In YOLO, the predictions of the rectangular region, the probability of object existence, and the posterior probability of the class are integrated into a single loss function.

[0025]

number

[0026] The learning unit 113 learns the student model so as to minimize the loss L calculated using the loss of the student model with respect to the teacher model (distillation loss Ldstl) and the loss of the student model with respect to the learning data (learning task specific loss Luq). In detail, the learning unit 113 updates the parameters of the student model using the backpropagation method.

[0027] The loss L calculated using the loss for the teacher model and the loss for the training data may be calculated, for example, by the following formula (3). In formula (3), λ is a weighting coefficient. The weighting coefficient λ is a coefficient that adjusts the balance between the loss for the training data and the loss for the teacher model. In this embodiment, the weighting coefficient λ is a constant. L = Luq + λ Ldstl (3)

[0028] The evaluation unit 114 obtains the difference in performance between the teacher model and the student model. As described above, in this embodiment, the performance is the performance of object detection. According to this embodiment, a high-performance trained model capable of performing object detection processing at high speed can be generated by using knowledge distillation.

[0029] In detail, the evaluation unit 114 calculates the performance difference using mAP (mean average precision), which is an evaluation index for the object detection task. That is, the evaluation unit 114 obtains the mAP of the student model and calculates the difference from the mAP of the teacher model. The mAP of the teacher model may be calculated every time, or may be stored in the storage unit 12 in advance. If the learning task is not object detection, a performance evaluation index other than mAP may be used. For example, if the learning task is image segmentation, mIoU (mean intersection over union) may be used as the performance evaluation index.

[0030] The change unit 115 changes the teacher model based on the performance difference between the teacher model and the student model. FIG. 3 is a schematic diagram for explaining a configuration for changing the teacher model. As shown in FIG. 3, in this embodiment, the number of teacher models prepared in advance is not one but multiple. In FIG. 3, N>1. The multiple teacher models have different performances from each other. The teacher model used to train the student model is changed sequentially according to the performance of the teacher model and the student model.

[0031] With this configuration, when the performance of the student model approaches that of the teacher model, the teacher model with higher performance can be changed to train the student model. This improves the performance of the student model compared to training the student model with only one teacher model. In addition, the performance of the teacher model can be gradually changed according to the performance of the student model, so that the student model can be appropriately trained.

[0032] In Fig. 3, the performance of the teacher model increases as the teacher model number increases. That is, in the example shown in Fig. 3, when the performance difference between the teacher model and the student model becomes smaller than a predetermined threshold, the change unit 115 changes the teacher model used for training the student model to a teacher model with higher performance than the teacher model currently being used. The predetermined threshold is an arbitrary value determined, for example, by trial and error.

[0033] Fig. 4 is a schematic diagram showing the change in performance of a student model when training is performed using the learning device 10 according to the first embodiment of the present invention. The horizontal axis of Fig. 4 represents iterations, and more specifically, the number of updates to the parameters of the student model. The vertical axis of Fig. 4 represents performance, for example, the detection rate of a detection target object.

[0034] As shown in Figure 4, when the performance of the student model approaches that of teacher model 1 through training using the initial teacher model, teacher model 1, the teacher model is changed from teacher model 1 to teacher model 2, which has a higher performance than teacher model 1, and training is continued. Then, when the performance of the student model approaches that of teacher model 2, the teacher model is changed from teacher model 2 to teacher model 3, which has a higher performance than teacher model 2, and training is continued. This repetition is performed a number of times equal to the number of teacher models (N) prepared in advance.

[0035] In this manner, in this configuration, training of the student model is started using a teacher model having performance corresponding to the performance of the student model, and the performance of the teacher model used for training is gradually improved as the performance of the student model improves. Therefore, the performance of the student model can be appropriately improved compared to when a high-performance teacher model is used to train the student model from the beginning. In addition, with this configuration, the student model can be trained efficiently.

[0036] Fig. 5 is a flowchart showing a flow of training a student model in the learning device 10 according to the first embodiment of the present invention. Note that, before the start of the flowchart shown in Fig. 5, N (N>1) trained teacher models are created. A known learning method may be used for training the teacher model.

[0037] In step S1, a variable "i" corresponding to the number of the teacher model used for learning is set to zero. The N teacher models are assigned numbers 0, 1, ... N-1 in order from lowest performance to highest performance. When the process of step S1 is completed, the process proceeds to the next step S2.

[0038] In step S2, a variable "epoch" representing the number of epochs is set to zero. The maximum number of epochs is set to M. M epochs is, for example, 100 epochs. When the process of step S2 is completed, the process proceeds to the next step S3.

[0039] In step S3, the student model is trained using the i-th teacher model. The training is training using the above-mentioned knowledge distillation. Here, training for one epoch is performed. In detail, the set of training data is divided into a plurality of subsets according to the batch size, and training using knowledge distillation is performed for each subset (each iteration). For each subset, inference using the i-th teacher model and inference using the student model are performed. Then, for each subset, parameters of the student model are updated using the loss function (see formula (3)) obtained by inference using the i-th teacher model and the student model. When the training of all subsets is completed, the training for one epoch is completed. When the processing of step S3 is completed, the processing proceeds to the next step S4.

[0040] In step S4, the performance of the student model trained in step S3 is evaluated. In this embodiment, mAP, which is an evaluation index for the object detection task, is calculated for the student model trained in step S3. After the mAP is calculated, the process proceeds to the next step S5.

[0041] In step S5, it is determined whether the performance difference between the teacher model and the student model is less than a threshold. Specifically, it is determined whether the difference between the mAP of the teacher model and the mAP of the student model is less than a threshold. The threshold is a fixed value stored in advance in the storage unit 12. If the performance difference is equal to or greater than the threshold (No in step S5), the process proceeds to the next step S6. If the performance difference is less than the threshold (Yes in step S5), the process proceeds to step S8. That is, the processes of steps S6 and S7 are skipped.

[0042] In step S6, 1 is added to the variable "epoch". If the variable "epoch" is zero at this point, the variable "epoch" becomes 1. That is, one epoch of learning (knowledge distillation) using the i-th teacher model is completed. Also, if the variable "epoch" is 50, the variable "epoch" becomes 51. That is, 51 epochs of learning using the i-th teacher model are completed. When the processing of step S6 is completed, the processing proceeds to the next step S7.

[0043] In step S7, it is confirmed whether the variable "epoch" is smaller than the maximum number of epochs M. If the variable "epoch" is smaller than the maximum number of epochs M (Yes in step S7), the process returns to step S3. That is, learning using the i-th teacher model is repeated. If the variable "epoch" has reached the maximum number of epochs M (No in step S7), the process proceeds to the next step S8.

[0044] In step S8, 1 is added to the variable "i". That is, the teacher model used for learning is changed from the i-th teacher model to the i+1-th teacher model. For example, if i=0, then i=1, and the teacher model is changed from 0 to 1. When the process of step S8 is completed, the process proceeds to the next step S9.

[0045] In step S9, it is confirmed whether the variable "i" is smaller than N. That is, it is confirmed whether all the teacher models prepared in advance have been used to train the student model. If the variable "i" is smaller than N (Yes in step S9), all the teacher models have not yet been used to train the student model, and the process returns to step S2. This starts training the student model by knowledge distillation using a new teacher model that has a higher performance than the teacher model used in the previous learning. In addition, the threshold in the process of step S5 may also be changed according to the change in the teacher model. If the variable "i" has reached N (No in step S9), the training (learning) of the student model shown in FIG. 5 is completed, and a new learned (trained) model is completed.

[0046] The trained student model obtained by the process of Fig. 5 can achieve performance close to the highest performance teacher model among the teacher models prepared in advance. In addition, the trained model has a smaller number of parameters than the teacher model, and can be processed quickly.

[0047] <2. Second embodiment> Next, a learning device of the second embodiment will be described. As in the first embodiment, the learning device of the second embodiment also trains a student model by knowledge distillation using a teacher model. The hardware configuration of the learning device of the second embodiment is similar to the hardware configuration of the learning device 10 of the first embodiment shown in FIG. 1. The functional configuration of the processor included in the learning device of the second embodiment is also similar to the functional configuration of the processor 11 included in the learning device 10 of the first embodiment shown in FIG. 2. However, the second embodiment differs from the first embodiment in the details of the functions of the learning unit 113 and the change unit 115 included in the processor. The following description will focus on these different parts. Contents that overlap with the first embodiment will not be described unless there is a particular need to do so.

[0048] The learning unit 113 learns the student model so as to minimize the loss L shown in equation (3), similarly to the first embodiment. However, while the weighting coefficient λ shown in equation (3) is constant in the first embodiment, the weighting coefficient λ is not constant in the second embodiment. This point differs from the first embodiment.

[0049] The change unit 115 changes the weighting coefficient when calculating the loss of the student model based on the performance difference between the teacher model and the student model. With this configuration, for example, the weighting coefficient λ can be changed at a stage where the performance of the student model approaches that of the teacher model, so that the student model can learn in a state where it is less susceptible to the influence of the teacher model.

[0050] In detail, the loss L in the student model is calculated by the formula (3) as described above. That is, in this embodiment, the weighting coefficient is a coefficient λ that adjusts the balance between the loss Luq for the learning data and the loss Ldstl for the teacher model. By reducing the value of λ when the performance of the student model approaches that of the teacher model, it is possible to make the student model less susceptible to the influence of the teacher model while learning. This can reduce the possibility that the teacher model will adversely affect the learning of the student model.

[0051] Fig. 6 is a flowchart showing the flow of training a student model in a learning device according to a second embodiment of the present invention. Note that before the start of the flowchart shown in Fig. 6, a trained teacher model that has been trained using a known method is created. In this embodiment, unlike the first embodiment, the number of teacher models is one.

[0052] In step S11, a variable "epoch" representing the number of epochs is set to zero. The number of epochs to be learned is set to M. The number of epochs M is, for example, 100 epochs. When the process of step S11 is completed, the process proceeds to the next step S12.

[0053] In step S12, the student model is trained using the teacher model. The training is training using the above-mentioned knowledge distillation. Here, training for one epoch is performed. In detail, the set of training data is divided into a plurality of subsets according to the batch size, and training using knowledge distillation is performed for each subset (each iteration). For each subset, inference using the teacher model and inference using the student model are performed. Then, for each subset, parameters of the student model are updated using a loss function (see formula (3)) obtained by inference using the teacher model and the student model. When training of all subsets is completed, training for one epoch is completed. When the processing of step S12 is completed, the processing proceeds to the next step S13.

[0054] In step S13, the performance of the student model trained in step S12 is evaluated. In this embodiment, mAP, which is an evaluation index for the object detection task, is calculated for the student model trained in step S12. After the mAP is calculated, the process proceeds to the next step S14.

[0055] In step S14, it is determined whether the performance difference between the teacher model and the student model is less than a threshold. Specifically, it is determined whether the difference between the mAP of the teacher model and the mAP of the student model is less than a threshold. The threshold is a fixed value stored in advance in the storage unit 12. If the performance difference is less than the threshold (Yes in step S14), the process proceeds to the next step S15. If the performance difference is equal to or greater than the threshold (No in step S14), the process proceeds to step S16. That is, the process of step S15 is skipped.

[0056] In step S15, the weighting coefficient λ included in equation (3) is changed. In detail, the value of the weighting coefficient λ is reduced. For example, the current value of λ is multiplied by a constant smaller than 1 (e.g., 0.5, etc.). That is, in equation (3), the weight of the term expressed as λ×Ldstl is reduced. As a result, the weight of the loss Ldstl for the teacher model in the loss L of the student model is reduced, and the influence of the teacher model can be reduced to allow the student model to learn. When the processing of step S15 is completed, the processing proceeds to the next step S16.

[0057] When the weighting factor λ is changed in step S15, the threshold value in step S14 may be changed accordingly. The threshold value may be fixed even after the weighting factor λ is changed. In this case, after the weighting factor λ is changed for the first time, the weighting factor λ gradually becomes smaller for each epoch. After the weighting factor λ is changed once, the weighting factor λ may not be changed thereafter.

[0058] In step S16, 1 is added to the variable "epoch". If the variable "epoch" is zero at this point, the variable "epoch" becomes 1. That is, one epoch of learning (knowledge distillation) using the teacher model is completed. Also, if the variable "epoch" is 50, the variable "epoch" becomes 51. That is, 51 epochs of learning using the teacher model are completed. When the processing of step S16 is completed, the processing proceeds to the next step S17.

[0059] In step S17, it is confirmed whether the variable "epoch" is smaller than the number of epochs M to be learned. If the variable "epoch" is smaller than the number of epochs M (Yes in step S17), the process returns to step S12, and learning is repeated. If the variable "epoch" has reached the number of epochs M (No in step S17), the training (learning) of the student model shown in FIG. 6 is completed, and a new trained model is completed.

[0060] The trained student model obtained by the process of Figure 6 is less likely to be adversely affected by the teacher model, and its performance can be appropriately improved by using knowledge distillation. In addition, the trained model has a smaller number of parameters than the teacher model, and can be processed quickly.

[0061] As can be seen from the above, in the learning method of the second embodiment, the loss L of the student model is also calculated using the loss Luq for the learning data and the loss Ldstl for the teacher model. In addition, in the learning method of the second embodiment, the weighting coefficient λ that adjusts the balance between the loss Luq for the learning data and the loss Ldstl for the teacher model is changed at a predetermined timing to train the student model. With this configuration, the student model can be trained while adjusting the influence of the teacher model during training of the student model.

[0062] As described above, in this embodiment, the predetermined timing is when the performance difference between the teacher model and the student model becomes smaller than a predetermined threshold value. This allows the weighting coefficient λ to be changed at a stage where the performance of the student model approaches that of the teacher model, so that the student model can be trained in a state where it is less susceptible to the influence of the teacher model.

[0063] However, as another example, the predetermined timing may be every predetermined number of epochs. In this configuration, the weighting coefficient λ can be attenuated every predetermined number of epochs, and the influence of the teacher model can be gradually reduced as the learning using knowledge distillation progresses.

[0064] The predetermined number of epochs may be, for example, every several epochs or every epoch. Fig. 7 is a diagram showing an example in which the weighting coefficient λ is attenuated every several epochs. In Fig. 7, the horizontal axis is the number of epochs, and the vertical axis is the value of λ. In the example shown in Fig. 7, the value of λ with respect to the number of epochs can be expressed by the following formula (4). λ = λ_int*(0.9^(epoch mod 50)) (4) λ_int is the initial value of λ. epoch indicates the current epoch number. In the example shown in equation (4), the weighting coefficient λ is multiplied by 0.9 every 50 epochs. 0.9 is just an example, and may be another number smaller than 1.

[0065] Fig. 8 is a diagram showing an example in which the weighting coefficient λ is attenuated for each epoch. In Fig. 8, the horizontal axis represents the number of epochs, and the vertical axis represents the value of λ. In the example shown in Fig. 8, the value of λ with respect to the number of epochs can be expressed by the following formula (5). λ = λ_int*(cos(π*epoch / M) + 1) / 2 (5) λ_int is the initial value of λ. epoch indicates the current epoch number. M is the number of epochs to be trained. In the example shown in Figure 8, λ is continuously attenuated (every epoch) by so-called cosine annealing.

[0066] <3. Things to keep in mind> In addition to the above embodiment, various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above embodiment should be considered to be illustrative in all respects and not restrictive, and the technical scope of the present invention is indicated by the claims, not the description of the above embodiment, and should be understood to include all modifications that fall within the meaning and scope of the claims. In addition, multiple embodiments and modifications shown in this specification may be implemented in appropriate combinations to the extent possible. [Explanation of symbols]

[0067] 10. Learning device 114...Evaluation section 115...Changes

Claims

1. A learning device for training a student model using a teacher model, the device comprising: a processor; The processor, determining a performance difference between the teacher model and the student model; A learning device that changes the teacher model used to train the student model to a teacher model with higher performance than the teacher model currently being used when the performance difference becomes smaller than a predetermined threshold.

2. A learning device for training a student model using a teacher model, comprising: a processor; The processor, A loss of the student model is calculated using the loss for the training data and the loss for the teacher model; A learning device that trains the student model by changing a weighting coefficient that adjusts the balance between the loss for the learning data and the loss for the teacher model at a predetermined timing.

3. The learning device according to claim 2 , wherein the loss for the teacher model is an error in an inference result between the teacher model and the student model, or an error in a feature map of an intermediate layer.

4. The learning device according to claim 1 , wherein the performance is object detection performance.

5. A learning method for training a student model using a teacher model, comprising: determining a performance difference between the teacher model and the student model; A learning method, comprising: when the performance difference becomes smaller than a predetermined threshold, changing the teacher model used to train the student model to a teacher model that has a higher performance than the teacher model currently being used.

6. A learning method for training a student model using a teacher model, comprising: A loss of the student model is calculated using the loss for the training data and the loss for the teacher model; A learning method in which a weighting coefficient that adjusts the balance between the loss for the learning data and the loss for the teacher model is changed at a predetermined timing to train the student model.

7. The learning method according to claim 6 , wherein the predetermined timing is when a performance difference between the teacher model and the student model becomes smaller than a predetermined threshold value.

8. The learning method according to claim 6 , wherein the predetermined timing is every predetermined number of epochs.

9. A program for causing a computer to execute a learning method for training a student model using a teacher model, comprising: The computer, determining a performance difference between the teacher model and the student model; When the performance difference becomes smaller than a predetermined threshold, changing the teacher model used to train the student model to a teacher model having a higher performance than the teacher model currently being used; A program that serves as a means to execute a program.

10. A program for causing a computer to execute a learning method for training a student model using a teacher model, comprising: The computer, determining a loss for the student model using a loss for the training data and a loss for the teacher model; training the student model by changing a weighting coefficient at a predetermined timing, the weighting coefficient adjusting the balance between the loss for the learning data and the loss for the teacher model; A program that serves as a means to execute a program.

Images