Inference device, ensemble model generation device, inference method, ensemble model generation method, and program

Abstract

To provide an inference device, an ensemble model generation device, an inference method, an ensemble model generation method, and a program that generate an ensemble model with improved classification robustness and generalization performance.SOLUTION: An ensemble model generation device 10 includes: a feature extraction unit 12 that has a plurality of feature extractors hwi that extract different features from input data; one identification unit 14 that identifies, on the basis of each of a plurality of feature quantities obtained by the feature extraction unit 12, data and outputs a plurality of identification results; a loss calculation unit 16 that calculates a loss between the plurality of identification results output from the identification unit 14 and a teacher label; and a feature extractor gradient calculation unit 20 that is used to update parameters of a plurality of feature extractors hwi, and calculates the loss obtained by the loss calculation unit 16 and a gradient having a regularization term that promotes diversification of the feature quantities. The ensemble model generation device uses the gradient calculated by the feature extractor gradient calculation unit 20 and updates a parameter of the feature extractor 12.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for generating an ensemble model for performing inference. [Background technology]

[0002] Deep neural networks (DNNs) have demonstrated high performance in various recognition tasks, but they have two issues: (1) low reliability of output probabilities and (2) vulnerability to noise. Low reliability of output probabilities refers to the fact that recognition results are over-confident even when they are incorrect. Vulnerability to noise refers to the fact that performance is significantly degraded by small noise that does not change the meaning of the image.

[0003] These issues can become major problems in real-world applications where recognition results are used for judgment, such as autonomous driving. Model ensembles, which integrate multiple trained DNNs and use them for prediction, are known to be effective ways to mitigate the above problems.

[0004] Although model ensembles have the disadvantage of requiring a larger amount of computation during inference than individual models, in environments with ample computational resources, it is possible to perform inference in approximately the same time as individual models by performing inference for each model in parallel.It is also known that a technique called knowledge distillation can be used to transfer the performance of a trained ensemble model to an individual model, resulting in an individual model with better performance than direct training.

[0005] However, if each model included in an ensemble is exactly the same, integrating them will not improve performance. In a model ensemble, it is important that the predictions of each model in the ensemble are diversified to a certain extent. In DNNs, a method called Deep Ensembles (Non-Patent Document 1) has been proposed as a representative method for ensembling such diverse models, in which the parameters of each model are initialized to different values ​​and trained independently, and this method has become the standard. In general, training a DNN is highly dependent on the initial values. Deep Ensembles utilizes the property that if each model is trained from different initial values, the resulting models will be diverse. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Lakshminarayanan et al. “Simple and scalable predictive uncertainty estimation using deep ensembles.” Advances in neural information processing systems 30 (2017). [Non-patent document 2] Liu, Qiang et al. “Stein variational gradient descent: A general purpose Bayesian inference algorithm.” Advances in neural information processing systems 29 (2016). [Non-patent document 3] Wang, Ziyu et al. "Function Space Particle Optimization for Bayesian Neural Networks." International Conference on Learning Representations (2019). Summary of the Invention [Problem to be solved by the invention]

[0007] The Deep Ensembles described above has demonstrated high performance in terms of both (1) the reliability of output probabilities and (2) robustness against noise, but because it relies solely on differences in initial values ​​for model diversity, it is possible that the number of models required for the desired performance may be large. In other words, because it does not explicitly promote model diversity, it cannot be denied that an ensemble consisting of similar models may be obtained.

[0008] There are also ensemble learning methods that diversify model parameters (Non-Patent Document 2) and ensemble learning methods that diversify model output probabilities (Non-Patent Document 3), but DNNs have a huge number of parameters, and parameter diversity does not necessarily mean model diversity. Also, directly diversifying output probabilities can cause unlearning with real data, so their effectiveness is generally limited. In view of the above background, the present invention proposes a new method for model ensemble. [Means for solving the problem]

[0009] The inference device of the present invention comprises a feature extraction unit having a plurality of feature extractors that extract different features from input data, a classification unit that classifies data based on each of the plurality of feature quantities obtained by the feature extraction unit and outputs a plurality of classification results, and a classification result integration unit that integrates the plurality of classification results obtained by the classification unit.

[0010] With this configuration, it is possible to obtain an appropriate classification result by integrating classification results based on different feature amounts obtained by a plurality of feature extractors.

[0011] In the inference device of the present invention, each feature extractor of the feature extraction unit may be configured to be generated by learning using multiple feature amounts output from each feature extractor when teacher data is input and the loss between the classification result and teacher label output from the classification unit to reduce the loss and diversify the multiple feature amounts.

[0012] In the inference device of the present invention, the feature extraction unit calculates a loss L1(T i , Y) and feature Z i The feature Z is calculated using the following formula, which includes a term L2 that diversifies the feature Z and a regular term L3 that suppresses the divergence of the feature Z. i The gradient of ΔZ i The configuration may be generated by learning using the above.

[0013]

number

[0014] In the inference device of the present invention, the classification unit may be configured to be generated by learning using the classification result output from the classification unit when training data is input and the loss of the training label.

[0015] In the inference device of the present invention, the identification unit may be configured to be generated by learning using a gradient Δθ of a parameter θ of the identification unit calculated by the following equation:

number

[0016] The ensemble model generation device of the present invention includes a feature extraction unit having multiple feature extractors that extract different features from input data; a single classification unit that classifies data based on each of the multiple feature quantities obtained by the feature extraction unit and outputs multiple classification results; a loss calculation unit that calculates a loss between the multiple classification results output from the classification unit and a teacher label; and a feature extractor gradient calculation unit that calculates a gradient used to update parameters of the multiple feature extractors, the feature extractor gradient calculation unit calculating the loss obtained by the loss calculation unit and a gradient having a regularization term that promotes diversification of the feature quantities, and the parameters of the feature extraction unit are updated using the gradient calculated by the feature extractor gradient calculation unit.

[0017] With this configuration, a single common classifier calculates classification results for each of the different feature quantities calculated by the multiple feature extractors, allowing the loss calculator to evaluate the impact of the different feature quantities on the classification results. Therefore, the parameters of the feature extractor can be updated using the loss calculated by the loss calculator and the gradient that promotes diversification of the feature quantities. Diversifying the multiple feature extractors in the feature extractor makes it possible to generate an ensemble model with improved classification robustness and generalization performance.

[0018] In the ensemble model generation device of the present invention, the feature extractor gradient calculation unit calculates a loss L1(T i , Y) and feature Z i The gradient ΔZ used to update the parameters of the plurality of feature extractors is calculated by the following equation, which includes a term L2 that diversifies the feature vectors, and a regular term L3 that suppresses the divergence of the feature vectors. i may be calculated.

number

[0019] The ensemble model generation device of the present invention may include a classifier gradient calculation unit that calculates a gradient used to update parameters of the classifier, based on the loss calculated by the loss calculation unit, and may update the parameters of the classifier using the gradient calculated by the classifier gradient calculation unit.

[0020] In the ensemble model generating device of the present invention, the classifier gradient calculation unit may calculate a gradient Δθ for updating a parameter θ of the classifier by the following equation:

number

[0021] The inference method of the present invention is a method for performing inference using an inference device having a plurality of feature extractors and a common classifier, and includes the steps of inputting input data into a plurality of feature extractors and obtaining a plurality of different feature quantities using the plurality of feature extractors, inputting each of the plurality of feature quantities into a common classifier and obtaining a plurality of classification results corresponding to each of the plurality of feature quantities, and integrating the classification results for each feature quantity obtained by the common classifier.

[0022] The ensemble model generation method of the present invention is a method for determining, by learning, parameters of a plurality of feature extractors and a common classifier in an inference device having the plurality of feature extractors and the classifier, and includes the steps of: inputting input data to the plurality of feature extractors and determining features in each of the plurality of feature extractors; inputting each of the plurality of feature extractors' determined features to the common classifier and determining a plurality of classification results corresponding to each of the plurality of feature quantities; calculating a loss between the plurality of classification results output from the common classifier and a teacher label; and calculating a gradient used to update the parameters of the plurality of feature extractors, the gradient including the loss and a regularization term that promotes diversification of the features; and updating the parameters of the plurality of feature extractors using the gradient.

[0023] The program of the present invention is a program for performing inference on input data, and causes a computer to execute the following steps: inputting the input data to a plurality of feature extractors and obtaining a plurality of different feature quantities using the plurality of feature extractors; inputting each of the plurality of feature quantities to a common classifier and obtaining a plurality of classification results corresponding to each of the plurality of feature quantities; and integrating the classification results for each feature quantity obtained by the common classifier.

[0024] Another aspect of the present invention provides a program for determining, by learning, parameters of a plurality of feature extractors and a common classifier of an inference device having the plurality of feature extractors and the classifiers, the program causing a computer to execute the following steps: inputting input data to the plurality of feature extractors and determining features in each of the plurality of feature extractors; inputting each of the plurality of feature extractors' determined features to the common classifier and determining a plurality of classification results corresponding to each of the plurality of feature quantities; calculating a loss between the plurality of classification results output from the common classifier and a teacher label; and calculating a gradient used to update the parameters of the plurality of feature extractors, the gradient including the loss and a regularization term that promotes diversification of the features; and updating the parameters of the plurality of feature extractors using the gradient. [Effects of the Invention]

[0025] According to the present invention, it is possible to perform inference with improved discrimination robustness and generalization performance. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an ensemble model generating device. [Figure 2] FIG. 10 is a diagram for explaining independent feature extractors. [Figure 3] FIG. 1 is a diagram illustrating a configuration of an inference device. DETAILED DESCRIPTION OF THE INVENTION

[0027] The ensemble model generation device and inference device of this embodiment will be described below with reference to the drawings. Note that the following description merely shows an example of a preferred embodiment and is not intended to limit the invention described in the claims.

[0028] 1 is a diagram showing the configuration of an ensemble model generation device 10 that generates an ensemble model. The ensemble model generation device 10 includes a feature extraction unit 12, a classification unit 14, a loss calculation unit 16, a feature extractor gradient calculation unit 18, and a classifier gradient calculation unit 20. Each component will be described below.

[0029] (Feature extraction section) The feature extraction unit 12 uses N independent feature extractors h wi (i=1, ,N). The feature extractor hw i The specific structure of hw can be a convolutional neural network or a multilayer perceptron. It is also possible to use a network pre-trained on another dataset. i are independent, each feature extractor hw i Other feature extractors hw i feature Z without being affected by i In other words, each feature extractor hw i corresponds to the models to be ensembled, each with a different configuration.

[0030] Figure 2(a) and Figure 2(b) show the independent feature extractor hw iThis figure explains the concept of neural networks. As shown in Figure 2(a), when the layer that receives data from the input layer is divided into two groups and the neurons in each group are not connected, feature extractors A and B are independent. As shown in Figure 2(b), even if the layers appear to be divided into two groups at first glance, if there is a connection between the groups and all the neurons are connected, the whole group forms a single feature extractor C. Figures 2(a) and 2(b) show examples of neural networks with multiple neurons, but the content explained here also applies to DNNs.

[0031] Each feature extractor hw i is the weight parameter w i This weight parameter w i is the target of learning by the ensemble model generation device 10. Each feature extractor h wi takes a batch of training images X as input and computes feature vectors Z i (The following equation) is output.

number

[0032] (Identification section) The classification unit 14 includes a plurality of feature extractors hw i Output Z from i A common classifier c that accepts inputs θ For convenience of explanation, in FIG. 1, each feature extractor hw i The classifier c θ Although multiple discriminators are listed, as can be seen from the fact that the weight parameter θ is the same, these discriminators c θ is a classifier. Classifier c θ Specific structures that can be considered include convolutional neural networks and multilayer perceptrons.

[0033] discriminator c θ has a weight parameter θ. The weight parameter θ is the learning target of the ensemble model generation device 10. θ is the feature extractor hwi Output Z from i (i=1, ,N) as input, and each feature Z i For a vector T with the dimension equal to the number of classes in the problem i Output.

number

[0034] (Loss calculation department) The loss calculation unit 16 calculates the loss θ Output from T i For each of (i=1, ,N), the loss function L1(T i , Y) is calculated. As a loss function, multi-valued cross-entropy loss can be used for classification problems, and squared loss can be used for regression problems.

[0035] (Feature extractor gradient calculation part) The feature extractor gradient calculation unit 18 calculates the gradient of each feature extractor hw i The parameter w i The gradient used to update the loss L1(T i , Y), each feature extractor hw i Feature quantity Z i By adding an L2 term (the following equation) that makes (i=1, ,N) diverse, each feature extractor hw i encourages users to use different features for data classification.

number

[0036] Furthermore, the regularization term L3(Z i ) is also added. i Gradient ΔZ for updating i can be calculated as follows:

number

number

[0037] Here, k is a positive definite kernel, such as a Gaussian kernel, and p is a prior distribution (Gaussian, Cauchy, Laplace, etc.) imposed on the feature. i Using the feature extractor hw i The parameter w i The gradient of Δw i can be calculated as follows:

number

[0038] (Classifier gradient calculation part) The classifier gradient calculation unit 20 calculates the common classifier c θ Specifically, the gradient Δθ used to update the parameter θ of the loss L1(T i , Y) to all feature extractors hw i The gradient Δθ of the average over (i=1, ,N) with respect to θ is calculated using the following equation:

number

[0039] The discriminator gradient calculation unit 20 calculates the feature i As in the case of (1), this gradient Δθ is applied to the optimization method to update the parameters.

[0040] With the above configuration, the loss of training data is reduced, and at the same time, each feature extractor hw constituting the feature extraction unit 12i The parameter w that diversifies i ,θ can be determined. This increase in diversity is possible because each feature extractor hw i The feature Z output from i (i=1, ,N) are used as a common classifier c θ and calculate each feature Z in the same feature space. i This configuration allows for classification based on each feature value Z i Since the positional relationship between the classification boundary and the classification model can be quantified, it is possible to achieve both model diversification and loss reduction. θ If there is no feature extractor hw i The feature value Z output from i Since the feature spaces of each feature Z i It is not possible to compare and evaluate how far away from the decision boundary the

[0041] In the above ensemble learning, N feature extractors hw i The parameters of the feature extractor hw can be trained simultaneously, or the parameters of the feature extractor hw can be trained simultaneously. i The parameters of are fixed and a new feature extractor hw i By adding and training the feature extractor hw i It is also possible to increase the feature extractor hw i This method of learning is useful when the computational resources for learning are limited.

[0042] FIG. 3 is a diagram showing the configuration of an inference device 30 that performs inference using an ensemble model. The inference device 30 includes a feature extraction unit 12, a classification unit 14, and an inference result integration unit 32. The feature extraction unit 12 and the classification unit 14 use the parameters w trained by the ensemble model generation device 10. i ,θ is a neural network model.

[0043] (Feature extraction section) The feature extraction unit 12 extracts the parameters w trained by the ensemble model generation device 10.i N feature extractors hw i The feature extraction unit 12 receives the test image to be inferred as an input, and outputs each feature extractor hw i Feature Z i (i=1, ,N) and calculate the feature value Z i is output to the recognition unit 14.

[0044] (Identification section) The classifier 14 generates a classifier c having a parameter θ trained by the ensemble model generating device 10. θ The classifier c θ is the feature extractor hw i The feature Z output from i (i=1, ,N) and classify it into a vector T i Output (i=1, ,N).

[0045] (Inference result integration part) The inference result integration unit 32 integrates the output T i (i=1, ,N) is integrated and a single classification result P is output. The classification result P is typically considered to be the average after passing it through a softmax function, as shown in the following equation.

number

[0046] Another way to integrate the inference results is to use each feature extractor hw i A possible method is to weight the results based on some degree of reliability and then take the sum.

[0047] The ensemble model generation device 10 and the inference device 30 of this embodiment have been described above, but an example of the hardware of the ensemble model generation device 10 and the inference device 30 is a computer equipped with a CPU, RAM, ROM, a hard disk, a display, a keyboard, a mouse, a communication interface, etc. The ensemble model generation device 10 and the inference device 30 are realized by storing a program having modules that realize each of the above functions in RAM or ROM and executing the program with a CPU. Such programs are also included in the scope of the present invention.

[0048] The scope of the present invention also includes a method for generating an ensemble model using the above-described ensemble model generating device 10 and a method for performing inference using the generated ensemble model with an inference device 30. That is, the method for generating an ensemble model includes: extracting a training image from a plurality of feature extractors hw i and then use multiple feature extractors hw i and a step of obtaining a feature quantity by each of the plurality of feature extractors hw i a step of inputting each of the plurality of feature quantities obtained by the method described above into a common classifier 14 to obtain a plurality of classification results corresponding to each of the plurality of feature quantities; a step of calculating losses between the plurality of classification results output from the common classifier 14 and the teacher labels; and a step of extracting the plurality of feature quantities from the plurality of feature extractors hw i The parameter w i a step of calculating a gradient used to update the feature vector hw, the gradient having a loss and a regularization term that promotes diversification of the feature vector; and a step of using the gradient to update the feature vector hw. i The parameter w i and updating the

[0049] The inference method using the ensemble model uses multiple feature extractors (HW) to extract test images. i and multiple feature extractors hw ia step of inputting each of the plurality of feature quantities to a common classifier 14 to obtain a plurality of classification results corresponding to each of the plurality of feature quantities; and a step of integrating the classification results 32 for each of the feature quantities obtained by the common classifier 14.

[0050] In the ensemble model generating device 10 of this embodiment, the feature extractor gradient calculation unit 20 calculates the gradient of each feature extractor hw i The feature quantity Z obtained by i (i=1, ,N) to diversify, the parameter w i The gradient Δw used to update i and use this gradient to calculate the feature extractor hw i As a result, each feature extractor hw constituting the feature extraction unit 12 learns i The diversity of the feature Z becomes higher. i By ensembling (i=1,...,N), it is possible to perform inference with improved classification robustness and generalization performance.

[0051] This mechanism is easy to understand by considering the example of an image recognition task. In image recognition tasks, there are often multiple features that serve as clues for predicting the label of a given image. For example, to recognize an image of a car as a label, a model can use various parts of the car, such as the tires and mirrors, as clues. In an ensemble, each model does not rely on a single feature, but instead uses different features as clues, which intuitively makes it possible to correctly recognize an image of a car that is partially obscured.

[0052] The inference device 30 of this embodiment includes a feature extractor hw i By diversifying the features output by the and classifying these features with a common classifier, more robust judgments can be made for unknown images, improving generalization performance.

[0053] (Variation) In the above-described embodiment, the label classification problem has been described as an example. However, the ensemble model generation device 10 can also be naturally applied to tasks such as semantic domain segmentation by providing spatial resolution to the output of the classifier (for example, by performing classification on a pixel-by-pixel basis). [Example]

[0054] The following describes the results of an experiment using inference using a model generated by the ensemble model generation device of this embodiment. In the experiment, inference using the model generated by the ensemble model generation device of this embodiment was used as an example, and the gold standard DEEP ENSEMBLES and a single model without ensemble were used as comparative examples. All methods adopted the same learning scheme, except that the inference space and prior parameters determined by standard cross-validation were different.

[0055] ResNet-50 was trained with an ensemble size of 5. Table 1 shows the evaluation results of the example (denoted as "FEATURE-WGD"), DEEP ENSEMBLES, and a single model (denoted as "SINGLE"). [Table 1]

[0056] Regarding the evaluation criteria shown in Table 1, ACCURACY is test accuracy, NLL is negative log-likelihood, BRIER is Brier score (Brier, 1950), and ECE is expected calibration error (Naeini et al., 2015).

[0057] Comparing the Example and the Single Model Example, the accuracy was significantly improved in the Example. Also, comparing the Example and the Comparative Example DEEP ENSEMBLES example, the intra-domain accuracy and calibration score were almost the same, but the Example showed a significant improvement in robustness against damage (i.e., accuracy was +1.5%). [Explanation of symbols]

[0058] 10 Ensemble model generator 12 Feature Extraction Unit 14 Identification unit 16 Loss calculation section 18 Classifier gradient calculation unit 20 Feature extractor gradient calculation unit 30 Reasoning device 32 Results Integration Unit

Claims

1. a feature extraction unit having a plurality of feature extractors for extracting different features from input data; a classification unit that classifies data based on each of the plurality of feature amounts obtained by the feature extraction unit and outputs a plurality of classification results; a classification result integration unit that integrates a plurality of classification results obtained by the classification unit; An inference device comprising:

2. 2. The inference device according to claim 1, wherein each feature extractor of the feature extraction unit is generated by learning using a plurality of feature quantities output from each feature extractor when teacher data is input and a loss between the classification result and teacher label output from the classification unit to reduce the loss and diversify the plurality of feature quantities.

3. The feature extraction unit calculates the loss L between the classification result Ti output from the classification unit and the teacher label Y. 1 (T i , Y) and feature Z i Term L that diversifies 2 and the regular term L that suppresses the divergence of the feature 3 The feature value Z is calculated using the following formula, which includes i The gradient of ΔZ i The inference device according to claim 2, wherein the inference device is generated by learning using the above-mentioned method. [Equation 1]

4. 2. The inference device according to claim 1, wherein the classifier is generated by learning using the loss of the classification result and the teacher label output from the classifier when teacher data is input.

5. The discriminator calculates a gradient Δθ of the parameter θ of the discriminator using the following formula: i 5. The inference device according to claim 4, wherein the inference device is generated by learning using the above. [Equation 2]

6. a feature extraction unit having a plurality of feature extractors for extracting different features from input data; a classification unit that classifies data based on each of the plurality of feature amounts obtained by the feature extraction unit and outputs a plurality of classification results; a loss calculation unit that calculates a loss between a plurality of classification results output from the classification unit and a truth label; a feature extractor gradient calculation unit that calculates a gradient used to update parameters of the plurality of feature extractors, the feature extractor gradient calculation unit calculating a gradient having the loss calculated by the loss calculation unit and a regularization term that promotes diversification of the feature quantities; Equipped with An ensemble model generation device that updates parameters of the feature extraction unit using the gradient calculated by the feature extractor gradient calculation unit.

7. The feature extractor gradient calculation unit calculates the loss L between the classification result Ti output from the classification unit and the teacher label Y. 1 (T i , Y) and feature Z i Term L that diversifies 2 and the regular term L that suppresses the divergence of the feature 3 The gradient ΔZ used to update the parameters of the feature extractors is i The ensemble model generating device according to claim 6, wherein the ensemble model generating device calculates: [Equation 3]

8. a discrimination unit gradient calculation unit that calculates a gradient used to update parameters of the discrimination unit based on the loss calculated by the loss calculation unit; The ensemble model generating device according to claim 6 , wherein the parameters of the classifier are updated using the gradient calculated by the classifier gradient calculation unit.

9. 9. The ensemble model generating device according to claim 8, wherein the classifier gradient calculation unit calculates a gradient Δθ for updating a parameter θ of the classifier using the following equation: [Equation 4]

10. 1. A method for performing inference using an inference device having a plurality of feature extractors and a common classifier, comprising: inputting input data to a plurality of feature extractors and obtaining a plurality of different feature quantities using the plurality of feature extractors; a step of inputting each of a plurality of feature amounts to a common classifier to obtain a plurality of classification results corresponding to each of the plurality of feature amounts; a step of integrating the classification results for each feature amount obtained by the common classification unit; An inference method comprising:

11. 1. A method for determining parameters of a plurality of feature extractors and a common classifier by learning in an inference device having the plurality of feature extractors and the common classifier, comprising: inputting input data to the plurality of feature extractors and obtaining feature quantities in each of the plurality of feature extractors; a step of inputting each of a plurality of feature amounts obtained by the plurality of feature extractors to the common classifier to obtain a plurality of classification results corresponding to each of the plurality of feature amounts; Calculating losses between a plurality of classification results output from the common classifier and a teacher label; Calculating gradients used to update parameters of the plurality of feature extractors, the gradients including the loss and a regularization term that promotes diversification of the feature quantities; updating parameters of the plurality of feature extractors using the gradients; An ensemble model generation method comprising:

12. A program for performing inference on input data, the program comprising: inputting input data to a plurality of feature extractors and obtaining a plurality of different feature quantities using the plurality of feature extractors; a step of inputting each of a plurality of feature amounts to a common classifier to obtain a plurality of classification results corresponding to each of the plurality of feature amounts; a step of integrating the classification results for each feature amount obtained by the common classification unit; A program that executes the following.

13. 1. A program for determining parameters of a plurality of feature extractors and a common classifier by learning in an inference device having the plurality of feature extractors and the common classifier, the program comprising: inputting input data to the plurality of feature extractors and obtaining feature quantities in each of the plurality of feature extractors; a step of inputting each of a plurality of feature amounts obtained by the plurality of feature extractors to the common classifier to obtain a plurality of classification results corresponding to each of the plurality of feature amounts; Calculating losses between a plurality of classification results output from the common classifier and a teacher label; Calculating gradients used to update parameters of the plurality of feature extractors, the gradients including the loss and a regularization term that promotes diversification of the feature quantities; updating parameters of the plurality of feature extractors using the gradients; A program that executes the following.

Images