Learning apparatus and learning method

Abstract

To provide a technique for developing distillation learning of a student model more efficiently.SOLUTION: By distillation learning using the output of a first learning model, learning of a second learning model having at least partially the same structure as that of the first learning model is carried out. While the second learning model is learned, one or more of parameters of the first learning model, a structure of the first learning model, parameters of the second learning model, and a structure of the second learning model are dynamically charged.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distillation learning technique for a learning model. [Background technology]

[0002] In recent years, distilled learning has attracted attention in the field of machine learning (Non-Patent Document 1). In distilled learning, the output of a large-scale, high-precision teacher model is generally used as teacher data (soft target), and a lighter student model is trained using the error between the output of the student model and the soft target (soft target error).

[0003] Here, learning refers to, for example, when a hierarchical neural network is used, sequentially and iteratively updating the weight coefficients and other parameters in the neural network by backpropagating the error in the output value obtained as a result of the forward propagation calculation in the neural network.

[0004] Moreover, the training data is a desired output (label value or its distribution) for the input data, and during the above-mentioned learning, learning is performed using learning data consisting of the input data and the training data.

[0005] In addition, the soft target in distillation learning is, for example, the output obtained by using a temperature-added softmax function as the activation function of the output layer. The temperature-added softmax function has the property that as the temperature is increased, the output value of the class corresponding to the correct class becomes smaller, and conversely, the output values ​​of other classes become larger. As a result, the output values ​​(information) of classes other than the correct class contribute more to learning than when normal teacher data (hard targets) are used for learning.

[0006] The soft target error is the error calculated between the soft target and the output of the student model. Generally, cross entropy is used as the error function.

[0007] The teacher model in distillation learning is a model that outputs a soft target when learning a student model in distillation learning, and is generally larger in scale and more accurate than the student model. The student model is generally a lighter model than the teacher model, and is generated by learning using a soft target error in distillation learning.

[0008] Conventionally, in order to obtain a lightweight model, it has been considered to configure an efficient architecture by devising or searching the layer structure and connection state of a neural network (Non-Patent Document 2). In addition, methods such as quantizing the weight coefficients, which are parameters of a neural network, to a small number of bits and pruning to remove layers and connections with low contributions have also been used.

[0009] On the other hand, distilled learning requires a trained model to be used as a teacher model. However, distilled learning has advantages not found in conventional methods for obtaining lightweight models, such as the advantage of obtaining a lightweight model with high accuracy, the advantage of obtaining a model that is less prone to overfitting due to the regularization effect, and the advantage of being able to use unsupervised data for learning.

[0010] The advantages of the regularization effect and the use of unsupervised data are also effective when the network size is not changed. A method called Born Again (Non-Patent Document 3) has been proposed as a method for utilizing the advantages of distillation learning when the network size is not changed. In Born Again, distillation learning is performed using models of the same scale for the teacher model and the student model. At that time, a random value is used as the initial value of the student model. When the distillation learning of the first student model is completed, this time it is used as the teacher model and distillation learning of another student model is performed again. In distillation learning of this other student model, a random value is also used as the initial value. In Born Again, the operation of distillation learning from the random value of this student model and replacement of the teacher model and the student model is repeated multiple times to perform distillation learning of multiple student models. Finally, the ensemble of multiple student models generated is used as the final learning model. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] “Distilling the Knowledge in a Neural Network”, G. Hinton et al.(NIPS2014) [Non-Patent Document 2] “Neural Architecture Search with Reinforcement Learning”, B. Zoph et al.(ICLR2017) [Non-Patent Document 3] “Born-Again Neural Networks”, Tommaso Furlanello et al. (ICML2018) [Non-Patent Document 4] Krizhevsky, A., Sutskever, I., and Hinton, GE “ImageNet classification with deep convolutional neural networks” NIPS, pp. 1106?1114, 2012. [Non-Patent Document 5] K. Simonyan and A. Zisserman. “Very deep convolutional networks for large-scale image recognition“ ICLR, 2015. Summary of the Invention [Problem to be solved by the invention]

[0012] Although Born Again has the advantage that the effect of distillation learning can be obtained even when the network size is not changed, the distillation learning of multiple student models must be performed from random initial values, which increases the cost of learning. In addition, when performing inference using a trained model, it is necessary to perform an ensemble of the outputs of multiple student models, which increases the implementation and computation costs of inference. In this case, inference refers to a process in which target data is input into a trained model, and when classifying, for example, which class the input data corresponds to, the output result of the class classification is obtained. In the present invention, a technology is provided for more efficiently advancing distillation learning of student models. [Means for solving the problem]

[0013] One aspect of the present invention is characterized in that it comprises a learning means for learning a second learning model having at least partially the same configuration as a first learning model by distillation learning using the output of the first learning model, and a control means for dynamically changing one or more of the parameters of the first learning model, the configuration of the first learning model, the parameters of the second learning model, and the configuration of the second learning model during learning of the second learning model. Effect of the Invention

[0014] According to the configuration of the present invention, distillation learning of a student model can be progressed more efficiently. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a block diagram showing an example of the functional configuration of a learning device. [Diagram 2] Flowchart of distillation learning for student model. [Diagram 3] 10 is a flowchart showing details of the process in step S104. [Figure 4] FIG. 3 is a schematic diagram of the process according to the flowchart in FIG. 2. [Diagram 5] FIG. 1 is a diagram for explaining distillation learning, which is a conventional technique. [Figure 6] FIG. 4 is a diagram for explaining the process in step S201. [Figure 7] A schematic diagram showing the temperature fluctuation of the temperature softmax function. [Figure 8] FIG. 13 is a diagram illustrating a process of perturbing the model configurations of a teacher model and a student model. [Figure 9] A schematic diagram showing the process of generating multiple student models through self-distillation learning. [Figure 10] 13A and 13B are diagrams for explaining a case in which self-distillation learning of a student model is performed by imparting fluctuations to an image as learning target data to be input to the student model in learning of the student model. [Figure 11] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device applicable to a learning device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0017] [First embodiment] In this embodiment, a case will be described in which a temperature-added softmax function is used as the activation function of the final output layer of each of the teacher model and the student model in distillation learning, and the temperature of this temperature-added softmax function is fluctuated to perform self-distillation learning of the student model. First, an example of the functional configuration of the learning device according to this embodiment will be described using the block diagram in FIG.

[0018] The storage unit 101 stores learning data used for distillation learning. The learning data includes learning target data and teacher data corresponding to the learning target data. The learning target data may be, for example, still image data, moving image data, or audio data. The teacher data is data for identifying a class in the learning target data.

[0019] The learning unit 102 learns the teacher model using the learning data stored in the storage unit 101. For the teacher model, a convolutional neural network (CNN) having a convolution layer, a pooling layer, and a fully connected layer, which is an example of a hierarchical neural network, is used. In addition, a temperature softmax function is used as the activation function of the final output layer of the teacher model. When the learning of the teacher model is completed, the learning unit 102 stores the learned teacher model in the storage unit 103.

[0020] The learning unit 104 learns the student model by distillation learning using the data to be learned included in the learning data stored in the storage unit 101 and a soft target that is the output of the teacher model stored in the storage unit 103. The student model uses a model having at least a partial configuration that is the same as the teacher model. In other words, the student model may be a model having the same configuration as the teacher model, or a model having a partial configuration that is the same as the teacher model. In any case, the student model also uses a CNN (convolutional neural network) having a convolution layer, a pooling layer, and a full-connection layer, which is an example of a hierarchical neural network. In addition, a temperature-added softmax function is also used as the activation function of the final output layer of the student model. Then, when the learning of the student model is completed, the learning unit 104 stores the learned student model in the storage unit 105.

[0021] The fluctuation adding unit 106 controls the fluctuation to be applied to the entire system. In this embodiment, the fluctuation adding unit 106 dynamically changes the temperatures of the temperature-added softmax function used in the activation function of the final output layer of the teacher model and the temperature-added softmax function used in the activation function of the final output layer of the student model by adding Gaussian fluctuations having a certain mean temperature and standard deviation.

[0022] In other words, every time the number of times the student model has learned increases by LN (LN is an arbitrary natural number, which may vary or may be a fixed value), the fluctuation adding unit 106 sets a random number generated according to a Gaussian distribution having a certain mean temperature and standard deviation to the "temperature (teacher temperature) of the temperature-added softmax function used in the activation function of the final output layer of the teacher model." This makes it possible to dynamically change the teacher temperature in the learning of the student model (during learning).

[0023] Similarly, the fluctuation adding unit 106 sets a random number generated according to a Gaussian distribution having a certain mean temperature and standard deviation to the "temperature (student temperature) of the temperature-added softmax function used in the activation function of the final output layer of the student model" every time the number of times the student model has been learned increases LN times. This makes it possible to dynamically change the student temperature in (during) the learning of the student model.

[0024] Next, distillation learning of a student model by the learning device according to this embodiment will be described with reference to the flowchart in Fig. 2. Also, a schematic diagram of the process according to the flowchart in Fig. 2 is shown in Fig. 4.

[0025] In step S101, the learning unit 102 learns the teacher model using the learning data stored in the storage unit 101. The initial values ​​of the parameters (weighting coefficients, etc.) of the teacher model are not limited to specific values ​​and may be set randomly, or if there is an existing model that matches the teacher model, the parameters may be used as the initial values. As shown in FIG. 4, a teacher model 401 is generated by such a learning process (hard target learning 0) by the learning unit 102. When learning of the teacher model is completed, the learning unit 102 stores the learned teacher model in the storage unit 103.

[0026] In step S102, the learning unit 104 reads out the trained teacher model stored in the storage unit 103, and enables inference using the trained teacher model. In step S103, the learning unit 104 sets initial values ​​for the parameters (weighting coefficients, etc.) of the student model. The parameters of the trained teacher model are set as the initial values ​​of the parameters of the student model. In the example of FIG. 4, the parameters of the teacher model 401 are set as the initial values ​​of the parameters of the student model 402.

[0027] In step S104, the student model is trained by distillation learning using the output of the teacher model. In the example of Fig. 4, self-distillation learning (soft target learning 1) of the student model 402 is performed using the output of the teacher model 401 to generate the student model 403. Before describing the self-distillation learning of the student model according to this embodiment, distillation learning, which is a conventional technique, will be described with reference to Fig. 5.

[0028] Image 501 is data to be learned that is input to teacher model 503, and image 502 is data to be learned that is input to student model 504. Both image 501 and image 502 are images of an animal, a cat. In general, a large-scale model such as Alexnet (Non-Patent Document 4) or VGG (Non-Patent Document 5) is used for the teacher model. On the other hand, a lighter model is generally used for the student model in order to reduce implementation costs and calculation costs during inference.

[0029] A teacher model 503 to which an image 501 is input as data to be learned outputs a distribution (soft target) 505 of output values ​​(likelihood for the class) for each class. p1 indicates the likelihood corresponding to the first class "cat", p2 indicates the likelihood corresponding to the second class "dog", and pi indicates the likelihood corresponding to the i-th class. When a softmax function is used as the activation function, the distribution of output values ​​has a characteristic that the output value of the class corresponding to the correct class (here, the likelihood corresponding to the class "cat") is close to 1, and the output values ​​of other classes are close to 0. Here, if the output value (likelihood) corresponding to the i-th class is softmax_i, the softmax function is a function shown in (Equation 1).

[0030]

number

[0031] Here, u i denotes the input value to the softmax function corresponding to the i-th class, and u jindicates the input value to the softmax function corresponding to the j-th class. The range of the variable j in (Equation 1) is from 1 to the total number of classes.

[0032] However, in distillation learning, a function that makes the distribution of output values ​​smoother, such as a temperature softmax function, is used as the activation function, so output values ​​other than the output value of the class corresponding to the correct answer class (here, the likelihood corresponding to "cat") also have relatively large values. Here, the temperature softmax function is a function shown in (Equation 2) where T is the set temperature and T_softmax_i is the output value (likelihood) corresponding to the i-th class.

[0033]

number

[0034] Here, u i denotes the input value to the temperature softmax function corresponding to the i-th class, and u j indicates the input value to the temperature softmax function corresponding to the jth class. The range of the variable j in (Equation 2) is 1 to the total number of classes. The output value (likelihood) of the teacher model p i is obtained as T_softmax_i using the above (Equation 2).

[0035] Therefore, the output value of the temperature softmax function (here, p i The distribution of the correct class contains not only information about the class that corresponds to the correct class, but also information about other classes, such as their similarity to the correct class, which contributes to learning.

[0036] A distribution 506 of output values ​​(likelihood for the class) for each class is output from a student model 504 to which an image 502 is input as data to be learned. q1 indicates the likelihood corresponding to the first class "cat", q2 indicates the likelihood corresponding to the second class "dog", and qi indicates the likelihood corresponding to the i-th class.

[0037] In general, for example, when a temperature-added softmax function is used for the activation function of the teacher model and the student model, the same temperature as the teacher temperature is also applied to the student temperature. In distillation learning, the output value (likelihood) p i and the output value (likelihood) of the student model, q i From the above, the soft target loss soft_target_loss is calculated using the following (Equation 3).

[0038]

number

[0039] Here, the range of the variable i in (Equation 3) is 1 to the total number of classes. Also, the output value (likelihood) of the student model q i can be calculated according to the following (Equation 4).

[0040]

number

[0041] where v i denotes the input value to the temperature softmax function corresponding to the i-th class in the student model, and v j indicates the input value to the temperature softmax function corresponding to the j-th class in the student model. The range of the variable j in (Equation 4) is 1 to the total number of classes.

[0042] Then, the parameters of the student model 504 are updated based on the soft target loss soft_target_loss calculated according to (Equation 3), thereby learning the student model 504. In other words, the soft target loss soft_target_loss is fed back to the learning of the student model 504.

[0043] In addition, normal teacher data may be further used in training the student model 504. In other words, the student model trained using a soft target may be retrained using the teacher data used in training the teacher model.

[0044] FIG. 5 shows the distribution of training data (hard target) 507. In the distribution 507, the likelihood of only the correct class "cat" (this is the kth class) is 1, and the likelihood of other classes is 0. In this case, the hard target loss hard_target_loss is calculated according to the following (Equation 5).

[0045]

number

[0046] Here, the output value r of the student model 504 i can be calculated according to the following (Equation 6).

[0047]

number

[0048] where v i indicates the input value to the temperature softmax function corresponding to the i-th class in the student model 504, and v j indicates the input value to the temperature softmax function corresponding to the j-th class in the student model 504. The range of the variable j in (Equation 6) is 1 to the total number of classes. Then, this hard target loss hard_target_loss is fed back to the learning of the student model 504.

[0049] In the distilled learning of the conventional technology described above, different models are used for the teacher model and the student model, but in the self-distilled learning according to this embodiment, the same model or at least a partially same model is used for the teacher model and the student model. However, in step S104, the distilled learning is advanced by giving different fluctuations to the system for the teacher model and the student model. The fluctuations given to the system here are, for example, fluctuations in the temperature of the temperature-added softmax function, fluctuations that change a part of the model configuration, or fluctuations applied to the input data to the model. In this embodiment, an example of self-distilled learning using the temperature fluctuations of the temperature-added softmax function will be described using the flowchart of FIG. 3 (a flowchart showing the details of the process in step S104).

[0050] In step S201, the fluctuation adding unit 106 sets fluctuations to be added to the teacher model and the student model respectively. The process in step S201 will be described with reference to FIG.

[0051] FIG. 6(a) is a diagram showing a schematic distribution of teacher temperatures set in the temperature-added softmax function of the teacher model. The horizontal axis represents the teacher temperature, and the vertical axis represents the frequency. Reference numeral 601 indicates the central temperature Tc in the temperature fluctuation, and reference numeral 602 indicates the standard deviation σ of the temperature fluctuation. T This temperature fluctuation is expressed as the mean value of the central temperature Tc and σ T The temperature fluctuation follows a Gaussian distribution with a standard deviation of . If the temperature is T, its probability distribution f T (T) follows the following (Equation 7).

[0052]

number

[0053] However, as shown in FIG. 6(c), each time the number of times the student model has learned increases by LN times, the fluctuation adding unit 106 adds the central temperature Tc as the average value and σ T A probability distribution f that is a Gaussian distribution with standard deviationT The random number (temperature T) generated according to "(T)" is set as the teacher temperature. This enables the fluctuation imparting unit 106 to impart fluctuation to the teacher temperature in the learning of the student model.

[0054] FIG. 6(b) is a diagram showing a typical distribution of student temperatures set in the temperature-added softmax function of the student model. The horizontal axis represents student temperatures, and the vertical axis represents frequency. Reference numeral 603 indicates the central temperature Tc in the temperature fluctuation, and reference numeral 604 indicates the standard deviation σ of the temperature fluctuation. S This temperature fluctuation is expressed as the mean value of the central temperature Tc and σ S It has a Gaussian distribution with standard deviation f S (T) follows the following (Equation 8).

[0055]

number

[0056] However, as shown in FIG. 6(d), each time the number of times the student model has learned increases by LN times, the fluctuation adding unit 106 adds the central temperature Tc as the average value and σ S A probability distribution f that is a Gaussian distribution with standard deviation S The random number (temperature T) generated according to "(T)" is set as the student temperature. This enables the fluctuation imparting unit 106 to impart fluctuation to the student temperature in the learning of the student model.

[0057] Therefore, in step S201, the fluctuation adding unit 106 calculates the probability distribution f T The random number (temperature T) generated according to the probability distribution f S Set the random number (temperature T) generated according to (T) as the student temperature.

[0058] Note that, although an example of imparting fluctuations to both the teacher temperature and the student temperature has been described here, it is not limited to imparting fluctuations to both the teacher temperature and the student temperature, and it is also possible to impart fluctuations to only one of the temperatures.

[0059] In step S202, the learning unit 104 inputs the data to be learned, which is included in the learning data stored in the memory unit 101, to the teacher model 401 read from the memory unit 103 in step S102, and obtains the output value 405 of the teacher model 401 as a soft target.

[0060] In step S203, the learning unit 104 inputs the data to be learned (the same data to be learned as the data to be learned input to the teacher model in step S202) contained in the learning data stored in the memory unit 101 to the student model 402 in which the initial parameter values ​​were set in step S103, and calculates the output value 406 of the student model 402.

[0061] In step S204, the learning unit 104 calculates a soft target loss using the output value 405, which is the soft target calculated in step S202, and the output value 406 calculated in step S203. Then, the learning unit 104 performs learning of the student model (soft target learning 1) by feeding back the calculated soft target loss to the learning of the student model and updating the parameters of the student model.

[0062] If there are no temperature fluctuations, this soft target loss will be 0 and learning will not progress, but by providing temperature fluctuations, the soft target loss will not be 0 and learning will progress. Also, by monitoring the value of the soft target loss, if the value is too small and learning progresses slowly, it is possible to set a larger temperature fluctuation (for example, by making the above-mentioned standard deviation larger).

[0063] In step S205, the learning unit 104 determines whether or not a termination condition for the learning of the student model (self-distillation learning of the student model) is satisfied. The termination condition may be, for example, "the number of times the student model has learned (the number of times steps S201 to S204 have been looped) exceeds a threshold", "the elapsed time since the start of the learning of the student model exceeds a threshold", "the amount of change in the soft target loss is equal to or less than a specified amount", etc.

[0064] If the end condition is satisfied as a result of such a determination, the process proceeds to step S105. On the other hand, if the end condition is not satisfied, the process proceeds to step S201. When the process proceeds to step S105, a student model 403, which is a learning model that has been trained by soft target learning 1, is obtained.

[0065] In step S105, the learning unit 104 inputs the desired learning target data (learning target data to be re-learned) among the learning target data included in the learning data to the student model 403, which is a learning model that has been trained by soft target learning 1, and calculates the output value of the student model 403 according to the above (Equation 6). The learning unit 104 then uses the output value to calculate the hard target loss according to the above (Equation 5), and feeds back the calculated hard target loss to the learning of the student model to update the parameters of the student model, thereby performing learning of the student model (hard target learning 1). Note that hard target learning 1 is not essential and may be omitted as appropriate.

[0066] In step S106, the learning unit 104 determines whether or not a termination condition for the learning of the student model (self-distillation learning of the student model) has been satisfied. Examples of the termination condition include "the number of times the student model has been learned (the number of loops of steps S102 to S105) has exceeded a threshold," "the elapsed time since the start of the learning of the student model has exceeded a threshold," "the amount of change in the hard target loss is equal to or less than a specified amount," and the like. Another termination condition is "when data is obtained indicating that the performance of the student model evaluated based on the output value of the student model to which evaluation data has been input is equal to or higher than a certain level."

[0067] If the result of such a determination is that the termination condition is met, the learning unit 104 stores the student model 404, which is a learned learning model, in the memory unit 105, and the processing according to the flowchart in Fig. 2 ends. On the other hand, if the termination condition is not met, the processing proceeds to step S102. In this way, according to this embodiment, it is possible to more efficiently progress the distillation learning of the student model.

[0068] [Second embodiment] In the following embodiments, the differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed that the second embodiment is the same as the first embodiment. In this embodiment, a case will be described in which a temperature-added softmax function is used for the activation function of the final output layer of the teacher model and the student model, and temperature fluctuations of different properties are given to the teacher model and the student model. This makes it possible to progress self-distillation learning more efficiently. Figures 7(a) to (d) are diagrams that show the temperature fluctuation of the temperature-added softmax function set in this embodiment.

[0069] FIG. 7(a) is a diagram showing a schematic distribution of teacher temperatures set in the temperature-added softmax function of the teacher model. The horizontal axis represents the teacher temperature, and the vertical axis represents the frequency. Reference numeral 701 indicates the central temperature Tc in the temperature fluctuation, and reference numeral 702 indicates the standard deviation σ' of the temperature fluctuation. T This temperature fluctuation is expressed as the mean value of the central temperature Tc and the mean value of σ' T The temperature fluctuation follows a Gaussian distribution with a standard deviation of . If the temperature is T, its probability distribution f T (T) follows the following (Equation 9).

[0070]

number

[0071] However, σ' T fluctuates according to the following (Equation 10).

[0072]

number

[0073] Here, N indicates the current number of learnings, and ω T indicates the frequency of the applied fluctuation. In addition, Nmax indicates the maximum number of times of learning (maximum number of times of learning) that is determined in advance, and σ T is the standard deviation σ' T The maximum value of

[0074] FIG. 7(b) is a diagram showing a typical distribution of student temperatures set in the temperature-added softmax function of the student model. The horizontal axis represents student temperatures, and the vertical axis represents frequency. Reference numeral 703 indicates the central temperature Tc in the temperature fluctuation, and reference numeral 704 indicates the standard deviation σ' of the temperature fluctuation. S This temperature fluctuation is expressed as the mean value of the central temperature Tc and the mean value of σ' S The temperature fluctuation follows a Gaussian distribution with a standard deviation of . If the temperature is T, its probability distribution f S (T) follows the following (Equation 11).

[0075]

number

[0076] However, σ' S fluctuates according to the following (Equation 12):

[0077]

number

[0078] Here, N indicates the current number of learnings, and ω S indicates the frequency of the applied fluctuation. In addition, Nmax indicates the maximum number of times of learning (maximum number of times of learning) that is determined in advance, and σ S is the standard deviation σ' S The maximum value of

[0079] Figure 7(c) shows the range of the time change (change given during the learning process) of the teacher temperature set in the temperature softmax function of the teacher model (i.e., the standard deviation of the temperature fluctuation σ' T ) is a diagram showing a schematic diagram of the temperature fluctuation. The horizontal axis represents the number of learning times, and the vertical axis represents the teacher temperature. Reference numeral 701' represents the central temperature Tc, and reference numeral 702' represents the standard deviation σ' of the temperature fluctuation. T The graph shows the change in the magnitude of

[0080] Figure 7(d) shows the range of time change (change given during the learning process) of the student temperature set in the temperature softmax function of the student model (i.e., the standard deviation of temperature fluctuation σ' S ) is a diagram showing a schematic diagram of the temperature fluctuation. The horizontal axis represents the number of learning sessions, and the vertical axis represents the temperature of the students. Reference numeral 703' represents the central temperature Tc, and reference numeral 704' represents the standard deviation σ' of the temperature fluctuation. S The graph shows the change in the magnitude of

[0081] In this embodiment, the maximum standard deviation σ T and σ S may be set to gradually decrease as the number of learning times increases. In this way, there are various methods for controlling each standard deviation, and the control method is not limited to a specific one. In addition, the parameter to be controlled in the probability distribution is not limited to the standard deviation.

[0082] In this way, in this embodiment, by setting the temperature fluctuations of the teacher model and the student model out of phase, a larger feedback can contribute to learning. This makes it possible to progress self-distillation learning more efficiently.

[0083] [Third embodiment] In this embodiment, a case where fluctuations are applied to the configurations (model configurations) of the teacher model and the student model will be described. As a case where the model configuration is fluctuated, the following description assumes that the full-connect layer of the CNN is dropped out (a number of connections between neurons in the full-connect layer are deleted according to the dropout rate (the value of the weighting coefficient is set to 0)). However, the fluctuations applied to the model configuration are not limited to this. Also, the dropout of the full-connect layer is assumed to be performed randomly on average. This makes it possible to progress self-distillation learning more efficiently.

[0084] Fig. 8 is a diagram showing a schematic diagram of a process of fluctuating the model configurations of the teacher model and the student model. In the following description, a case of self-distillation learning using fluctuations in the model configuration will be described with reference to Fig. 8 and the flowchart in Fig. 3.

[0085] In step S201, the fluctuation adding unit 106 sets fluctuations in each of the teacher model 801 that has performed hard target learning 0 and the student model 802 whose parameters have been initialized. Specifically, a dropout rate is set in each of the teacher model 801 and the student model 802. The fluctuation adding unit 106 may set the same dropout rate for the teacher model 801 and the student model 802, or may set different dropout rates for the teacher model 801 and the student model 802. In addition, the fluctuation adding unit 106 may change the dropout rate of the teacher model 801 and / or the dropout rate of the student model 802 depending on the number of learnings. The dropout rate may be determined in any way, and for example, a random number (a real number in the range of 0 to 1) generated according to the above-mentioned probability distribution may be set as the dropout rate.

[0086] When the dropout rate r1 (where r1 is a real number satisfying 0 < r1 < 1) is set for the teacher model 801, the fluctuation imparting unit 106 sets any connection corresponding to (100×r1)% of the number of connections between neurons in the fully connected layer of the teacher model 801 to 0 (dropout). The fully connected layer 805 is a fully connected layer obtained as a result of performing dropout on the teacher model 801.

[0087] Also, when the dropout rate r2 (where r2 is a real number satisfying 0 < r2 < 1) is set for the student model 802, the fluctuation imparting unit 106 sets any connection corresponding to (100×r2)% of the number of connections between neurons in the fully connected layer of the student model 802 to 0 (dropout). The fully connected layer 806 is a fully connected layer obtained as a result of performing dropout on the student model 802.

[0088] In step S202, the learning unit 104 inputs the data of the learning target included in the learning data stored in the storage unit 101 to the dropout-applied teacher model 801, and obtains the output value of the teacher model 801 as a soft target.

[0089] In step S203, the learning unit 104 inputs the data of the learning target (the same learning target data as the data of the learning target input to the teacher model in step S202) included in the learning data stored in the storage unit 101 to the dropout-applied student model 802, and obtains the output value of the student model 802.

[0090] In step S204, the learning unit 104 obtains a soft target loss using the output value, which is the soft target obtained in step S202, and the output value obtained in step S203. Then, the learning unit 104 feeds back the obtained soft target loss to the learning of the student model, and updates the parameters of the student model, thereby performing the learning of the student model (soft target learning 1).

[0091] Due to some differences in the model configuration between the teacher model and the student model, the soft target loss will not be zero, and this will be fed back into the learning of the student model (the network parameters will be updated).

[0092] In step S205, the learning unit 104 judges whether or not the termination condition of the learning of the student model (self-distillation learning of the student model) is satisfied. If the termination condition is satisfied as a result of such judgment, the process proceeds to step S105. On the other hand, if the termination condition is not satisfied, the process proceeds to step S201. When the process proceeds to step S105, a student model 803, which is a learning model that has been trained by soft target learning 1, is obtained.

[0093] In step S105, the learning unit 104 inputs the desired learning target data (learning target data to be re-learned) among the learning target data included in the learning data to the student model 803, which is a learning model that has been learned by soft target learning 1, and calculates the output value of the student model 803 according to the above (Equation 6). The learning unit 104 then uses the output value to calculate the hard target loss according to the above (Equation 5), and feeds back the calculated hard target loss to the learning of the student model to update the parameters of the student model, thereby performing learning of the student model (hard target learning 1). Note that hard target learning 1 is not essential and may be omitted as appropriate.

[0094] In step S106, the learning unit 104 judges whether or not the termination condition of the learning of the student model (self-distillation learning of the student model) is satisfied. If the termination condition is satisfied as a result of such judgment, the learning unit 104 stores the trained student model 804 in the storage unit 105, and the process according to the flowchart in Fig. 2 is terminated. On the other hand, if the termination condition is not satisfied, the process proceeds to step S102.

[0095] In this way, the difference in the model configuration between the teacher model and the student model creates feedback for learning, which makes it possible to progress self-distillation learning more efficiently.

[0096] [Fourth embodiment] In this embodiment, a case where a plurality of student models are generated by self-distillation learning will be described. This makes it possible to more efficiently learn and generate a plurality of student models.

[0097] Fig. 9 is a diagram showing a schematic diagram of a process for generating multiple student models by self-distillation learning. In the following description, an example of generating multiple student models by self-distillation learning will be described with Fig. 9 in conjunction with the flowchart in Fig. 3. Here, a frame 901 shows a schematic diagram of a process for generating multiple student models by self-distillation learning, and a frame 902 shows a more detailed learning process of a student model 904.

[0098] In step S201, similarly to the first embodiment, the fluctuation adding unit 106 calculates the probability distribution f T (T) is set as the teacher temperature, and the probability distribution f S Set the random number generated according to (T) to the student temperature.

[0099] In step S202, the learning unit 104 inputs the data to be learned, which is included in the learning data stored in the storage unit 101, to the teacher model 903 read from the storage unit 103 in step S102, and obtains the output value of the teacher model 903 as a soft target. Note that the teacher model 903 is a model that has been trained by hard target learning 0.

[0100] In step S203, the learning unit 104 inputs the data to be learned (the same data to be learned as the data to be learned input to the teacher model in step S202) contained in the learning data stored in the memory unit 101 to the student model 906 for which the initial parameter values ​​were set in step S103, and calculates the output value of the student model 906.

[0101] In step S204, the learning unit 104 calculates a soft target loss using the output value, which is the soft target calculated in step S202, and the output value calculated in step S203. Then, the learning unit 104 feeds back the calculated soft target loss to the learning of the student model 906, and updates the parameters of the student model 906 to generate a student model 907 (soft target learning 1).

[0102] In step S205, the learning unit 104 judges whether or not the end condition of the learning of the student model is satisfied. If the end condition is satisfied as a result of such judgment, the process proceeds to step S105. On the other hand, if the end condition is not satisfied, the process proceeds to step S201. At the point when the process proceeds to step S105, a student model 907, which is a learning model trained by soft target learning 1, is obtained.

[0103] In step S105, the learning unit 104 inputs the desired learning target data (learning target data to be re-learned) among the learning target data included in the learning data to the student model 907, which is a learning model that has been learned by soft target learning 1, and calculates the output value of the student model 907 according to the above (Equation 6). The learning unit 104 then uses the output value to calculate the hard target loss according to the above (Equation 5), and feeds back the calculated hard target loss to the learning of the student model to update the parameters of the student model, thereby performing learning of the student model (hard target learning 1). Note that hard target learning 1 is not essential and may be omitted as appropriate.

[0104] In step S106, the learning unit 104 judges whether or not the termination condition of the learning of the student model (self-distillation learning of the student model) is satisfied. If the termination condition is satisfied as a result of such judgment, the learning unit 104 stores the trained student model 904 in the memory unit 105, and the process according to the flowchart in Fig. 2 is terminated. On the other hand, if the termination condition is not satisfied, the process proceeds to step S102.

[0105] As described above, a student model 904 is generated by self-distillation learning 1 using soft target learning 1 and hard target learning 1. Then, for a student model in which the parameters of the student model 904 are set as initial values, soft target learning similar to self-distillation learning 1 and hard target learning similar to hard target learning 1 are performed (this is called self-distillation learning 2) to generate a student model 905. The learning of this student model 905 can progress more efficiently by using the student model 904 as the initial value than learning with a random value as the initial value. When the above self-distillation learning is repeated N times, N student models are generated. When inference is performed, an ensemble of the outputs of the N student models may be performed.

[0106] [Fifth embodiment] In this embodiment, a case will be described in which fluctuations are added to the learning target data input to the student model and the teacher model in the learning of the student model. This makes it possible to progress self-distillation learning more efficiently.

[0107] A case in which self-distillation learning of a student model is performed by fluctuating an image as learning target data to be input to the student model in learning of the student model will be described with reference to FIG.

[0108] Image 1001 is an image of a cat, which is an animal, and is data to be learned and input to teacher model 1103. Image 1002 is an image of a cat, which is an animal, and is data to be learned and input to student model 1104. A fluctuation is added to the pixel values ​​of some or all of the pixels in image 1102. The fluctuation is added by applying a fluctuation to the central pixel value I C is the average value, and σ SI The pixel value is I, and its probability distribution f SI (I) follows the following (Equation 13).

[0109]

number

[0110] Therefore, the fluctuation adding unit 106 adds the "center pixel value I C is the average value, and σ SI A probability distribution f that is a Gaussian distribution with standard deviation SI The random number (pixel value I) generated according to "(I)" is set to the pixel value of the pixel to generate image 1002. This allows the fluctuation adding unit 106 to add fluctuation to the image to be input to the student model in the student model learning. Note that if the image is an RGB image, fluctuation is added to each of the RGB pixel values.

[0111] The teacher model 1003 and the student model 1004 in Fig. 10 are network models having at least a part of the same configuration. Here, VGG is used for both. In the following, we will explain the case where a temperature softmax function is used as the activation function of the final output layer of each of the teacher model and the student model.

[0112] A teacher model 1003 to which an image 1001 is input outputs a distribution (soft target) 1005 of output values ​​(likelihood for that class) for each class. On the other hand, a student model 1004 to which an image 1002 is input outputs a distribution 1006 of output values ​​(likelihood for that class) for each class.

[0113] Then, similarly to the first embodiment, a soft target loss soft_target_loss is calculated from the output value (likelihood) of the teacher model 1003 and the output value (likelihood) of the student model 1004. Then, similarly to the first embodiment, the parameters of the student model 1004 are updated based on the soft target loss soft_target_loss, thereby performing learning (soft target learning) of the student model 1004. Even if the teacher model and student model have the same network configuration and parameters, if the input data are different, the soft target loss will not be 0, and feedback to learning will exist, resulting in progress of learning.

[0114] Then, similarly to the first embodiment, the desired learning target data (learning target data to be re-learned) among the learning target data included in the learning data is input to the student model 1004, which is a learning model that has been trained by soft target learning, and the output value of the student model 1004 is calculated according to the above (Equation 6). The hard target loss is then calculated using the output value according to the above (Equation 5), and the calculated hard target loss is fed back to the learning of the student model to update the parameters of the student model, thereby performing learning of the student model (hard target learning). Note that hard target learning is not essential and may be omitted as appropriate.

[0115] In this way, the difference in the input data between the teacher model and the student model creates feedback for learning, which makes it possible to progress self-distillation learning more efficiently.

[0116] The purpose of the present invention is to provide a method and apparatus that can perform distillation learning and inference at a lower cost by performing self-distillation learning using fluctuations in a model of the same scale, and the method and apparatus can be applied to any method or subject as long as they meet the purpose.

[0117] In this embodiment, fluctuations are imparted to the pixel values ​​of some or all of the pixels in an image 1002 input to a student model 1004, and fluctuations are not imparted to the pixel values ​​of some or all of the pixels in an image 1001 input to a teacher model 1003. However, without being limited to this, fluctuations may be imparted to the pixel values ​​of some or all of the pixels in an image 1001.

[0118] This embodiment can be combined with one or more of the first to fourth embodiments. For example, a fluctuation may be imparted to the temperature (teacher temperature or student temperature) and also to the image input to the teacher model or student model. For example, a fluctuation may be imparted to the configuration of the teacher model or student model and also to the image input to the teacher model or student model.

[0119] [Sixth embodiment] The functional units shown in Fig. 1 may be implemented by hardware, or the functional units other than the storage units 101, 103, and 105 may be implemented by software (computer programs). In the latter case, a computer device capable of executing such a computer program is applicable to the above-mentioned learning device. An example of the hardware configuration of a computer device applicable to the learning device will be described with reference to the block diagram of Fig. 11.

[0120] The CPU 1101 executes various processes using computer programs and data stored in the RAM 1102 and the ROM 1103. As a result, the CPU 1101 controls the operation of the entire computer device, and executes or controls the various processes described above as the processes performed by the learning device.

[0121] The RAM 1102 has an area for storing computer programs and data loaded from the ROM 1103 or the external storage device 1106, and an area for storing computer programs and data received from the outside via the I / F 1107. The RAM 1102 further has a work area used when the CPU 1101 executes various processes. In this way, the RAM 1102 can provide various areas as needed.

[0122] The ROM 1103 stores setting data for the computer device, computer programs and data relating to the startup of the computer device, computer programs and data relating to the basic operation of the computer device, and the like.

[0123] The operation unit 1104 is a user interface such as a keyboard, mouse, touch panel, etc., and the user can operate it to input various instructions and data. For example, the user can operate the operation unit 1104 to input the information described above as known information (threshold value, central temperature, central pixel value, standard deviation, maximum number of learning times, etc.).

[0124] The display unit 1105 has a liquid crystal screen or a touch panel screen, and can display the processing results by the CPU 1101 using images, text, etc. For example, the display unit 1105 can display various information related to learning (soft target loss, hard target loss, threshold, central temperature, central pixel value, standard deviation, maximum number of learning times, etc.) using images, text, etc. This allows the user to see the learning results displayed on the display unit 1105 and input (adjust) parameters such as the threshold, central temperature, central pixel value, standard deviation, and maximum number of learning times by operating the operation unit 1104. The display unit 1105 may be a projection device such as a projector that projects images and text.

[0125] The external storage device 1106 is a large-capacity information storage device such as a hard disk drive device. The external storage device 1106 stores an OS (operating system), computer programs and data for causing the CPU 1101 to execute or control the various processes described above as processes performed by the learning device, and the like. The computer programs and data stored in the external storage device 1106 are loaded into the RAM 1102 as appropriate under the control of the CPU 1101, and become targets for processing by the CPU 1101. Note that the storage units 101, 103, and 105 shown in FIG. 1 can be implemented using memory devices such as the RAM 1102 and the external storage device 1106.

[0126] The I / F 1107 is a communication interface for performing data communication with an external device. The CPU 1101, the RAM 1102, the ROM 1103, the operation unit 1104, the display unit 1105, the external storage device 1106, and the I / F 1107 are all connected to a system bus 1108. Note that the hardware configuration shown in Fig. 11 is merely an example of the hardware configuration of a computer device applicable to the above learning device, and can be modified / altered as appropriate.

[0127] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0128] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0129] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0130] The invention of this specification includes the following learning device, learning method, and computer program.

[0131] (Item 1) A learning means for learning a second learning model having at least a partial same configuration as the first learning model by distillation learning using an output of the first learning model; a control means for dynamically changing one or more of the parameters of the first learning model, the configuration of the first learning model, the parameters of the second learning model, and the configuration of the second learning model during learning of the second learning model; A learning device comprising:

[0132] (Item 2) The learning device described in item 1, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the first learning model, during learning of the second learning model.

[0133] (Item 3) The learning device described in item 2, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the first learning model, in response to temperature fluctuations according to a Gaussian distribution during learning of the second learning model.

[0134] (Item 4) 4. The learning device according to item 3, wherein the control means dynamically changes parameters of the Gaussian distribution depending on the number of times the second learning model has been learned.

[0135] (Item 5) The learning device according to item 1, characterized in that the control means dynamically changes the connections between neurons in a full-connect layer of the first learning model during learning of the second learning model.

[0136] (Item 6) The learning device described in any one of items 1 to 5, characterized in that the control means dynamically changes the pixel values ​​of some or all of the pixels in the image input to the first learning model during learning of the second learning model.

[0137] (Item 7) The learning device described in item 1, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the second learning model, during learning of the second learning model.

[0138] (Item 8) The learning device described in item 7, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the second learning model, during learning of the second learning model, in accordance with temperature fluctuations according to a Gaussian distribution.

[0139] (Item 9) 9. The learning device according to item 8, wherein the control means dynamically changes parameters of the Gaussian distribution depending on the number of times the second learning model has been learned.

[0140] (Item 10) The learning device according to item 1, characterized in that the control means dynamically changes connections between neurons in a full-connect layer of the second learning model during learning of the second learning model.

[0141] (Item 11) The learning device described in any one of items 1 to 10, characterized in that the control means dynamically changes the pixel values ​​of some or all of the pixels in the image input to the second learning model during learning of the second learning model.

[0142] (Item 12) 12. The learning device according to any one of items 1 to 11, wherein the learning means sets parameters of the first learning model to initial values ​​of parameters of the second learning model.

[0143] (Item 13) The learning device described in any one of items 1 to 12, characterized in that the learning means uses teacher data used when learning the first learning model to learn the second learning model that has been learned by the distillation learning.

[0144] (Item 14) The learning device described in any one of items 1 to 13, characterized in that the learning means learns another second learning model in which parameters of a second learning model already learned by the learning means are set, by distillation learning using the output of the first learning model.

[0145] (Item 15) 15. The learning device according to any one of items 1 to 14, wherein the first learning model is a pre-trained model.

[0146] (Item 16) A learning method performed by a learning device, comprising: A learning step in which a learning means of the learning device learns a second learning model having at least a partial same configuration as the first learning model by distillation learning using an output of the first learning model; a control step in which a control means of the learning device dynamically changes one or more of a parameter of the first learning model, a configuration of the first learning model, a parameter of the second learning model, and a configuration of the second learning model during learning of the second learning model; A learning method comprising:

[0147] (Item 17) A computer program for causing a computer to function as each of the means of the learning device described in any one of items 1 to 15.

[0148] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0149] 101: memory unit 102: learning unit 103: memory unit 104: learning unit 105: memory unit 106: fluctuation imparting unit

Claims

1. a learning means for learning a second learning model having at least a partial same configuration as the first learning model by distillation learning using an output of the first learning model; a control means for dynamically changing one or more of a parameter of the first learning model, a configuration of the first learning model, a parameter of the second learning model, and a configuration of the second learning model during learning of the second learning model; A learning device comprising:

2. The learning device according to claim 1 , characterized in that the control means dynamically changes a temperature of a temperature softmax function, which is an activation function of a final output layer of the first learning model, during learning of the second learning model.

3. The learning device according to claim 2, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the first learning model, in response to temperature fluctuations according to a Gaussian distribution during learning of the second learning model.

4. 4. The learning device according to claim 3, wherein the control means dynamically changes parameters of the Gaussian distribution in accordance with the number of times the second learning model has been learned.

5. 2. The learning device according to claim 1, wherein the control means dynamically changes connections between neurons in a fully-connected layer of the first learning model during learning of the second learning model.

6. The learning device according to claim 1 , characterized in that the control means dynamically changes pixel values ​​of some or all of the pixels in an image input to the first learning model during learning of the second learning model.

7. The learning device according to claim 1, characterized in that the control means dynamically changes a temperature of a temperature softmax function, which is an activation function of a final output layer of the second learning model, during learning of the second learning model.

8. The learning device according to claim 7, characterized in that the control means dynamically changes the temperature of a temperature softmax function, which is an activation function of the final output layer of the second learning model, in response to temperature fluctuations according to a Gaussian distribution during learning of the second learning model.

9. 9. The learning device according to claim 8, wherein the control means dynamically changes parameters of the Gaussian distribution in accordance with the number of times the second learning model has been learned.

10. 2. The learning device according to claim 1, wherein the control means dynamically changes connections between neurons in a full-connect layer of the second learning model during learning of the second learning model.

11. The learning device according to claim 1 , characterized in that the control means dynamically changes pixel values ​​of some or all of the pixels in an image input to the second learning model during learning of the second learning model.

12. 2. The learning device according to claim 1, wherein the learning means sets parameters of the first learning model to initial values ​​of parameters of the second learning model.

13. The learning device according to claim 1 , wherein the learning means learns the second learning model that has been learned by the distillation learning using teacher data used in learning the first learning model.

14. The learning device according to claim 1, characterized in that the learning means learns another second learning model in which parameters of a second learning model already learned by the learning means are set by distillation learning using the output of the first learning model.

15. The learning device according to claim 1 , wherein the first learning model is a pre-trained model.

16. A learning method performed by a learning device, comprising: A learning step in which a learning means of the learning device learns a second learning model having at least a partial same configuration as the first learning model by distillation learning using an output of the first learning model; a control step in which a control means of the learning device dynamically changes one or more of a parameter of the first learning model, a configuration of the first learning model, a parameter of the second learning model, and a configuration of the second learning model during learning of the second learning model; A learning method comprising:

17. A computer program for causing a computer to function as each of the means of the learning device according to any one of claims 1 to 15.

Images