Learning method and inference method

Abstract

One aspect of the present invention is a learning method including a control step for executing additional pre-training for obtaining, by learning, a target-based pre-trained model used for acquiring a mathematical model for performing inference based on target data, which are data of one or more predetermined modalities. In the additional pre-training, a training subject in which initial values are values in an initial model is trained by using the initial model and extension data, the initial model being a deep learning model pre-trained in advance by using an arbitrary learning method, and the extension data including data that is the same as or different from the target data in addition to the target data. The target-based pre-trained model is a training subject at the timing when a training end condition is satisfied, the training end condition being a predetermined condition relating to the end of the additional pre-training. In the additional pre-training, the training subject is updated so as to reduce at least the difference between features of the target data, inferred by the initial model, and features of the extension data, inferred by the training subject.

Description

Learning and reasoning methods

[0001] The present invention relates to learning and reasoning methods.

[0002] In recent years, there has been active research into technology for obtaining mathematical models for inference through machine learning, as high inference accuracy is expected.In addition, this technology has evolved into a technology where the learning process for obtaining mathematical models through machine learning is divided into two stages: pre-training, where the model is first trained on general-purpose data, and then fine-tuning, where the model is trained on the data of the application, to obtain a mathematical model with improved inference performance.

[0003] Daisuke Niizumi, et al., "Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input," in ICASSP, 2023.Daisuke Niizumi, et al., "Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation," to appear at Interspeech, 2023.Seanie Lee et al. al., “Self-Distillation for Further Pre-training of Transformers,” in ICLR, 2023. Kaiming He, et al., “Masked autoencoders are scalable vision learners,” in CVPR, 2021.

[0004] However, there are cases where the mathematical model obtained by pre-training (hereinafter referred to as "pre-trained model") does not necessarily have good inference accuracy when fine-tuned.

[0005] In view of the above circumstances, an object of the present invention is to provide a pre-training technique for improving the accuracy of estimation of a mathematical model obtained using a pre-trained model.

[0006] One aspect of the present invention is a learning method that includes a control step of performing additional pre-learning to obtain, by learning, a target pre-trained model used to obtain a mathematical model that makes inferences based on target data, which is data of one or more specified modalities.In the additional pre-learning, learning is performed on a learning object whose initial value is the initial model using an initial model that is a deep learning model that has been pre-trained in advance by an arbitrary learning method, and extended data in which data that is the same as or different from the target data is added to the target data.The target pre-trained model is the learning object at the time when a learning termination condition, which is a specified condition for terminating the additional pre-learning, is satisfied.In the additional pre-learning, the learning object is updated so as to at least reduce the difference between the features of the target data inferred by the initial model and the features of the extended data inferred by the learning object.

[0007] One aspect of the present invention is an inference method comprising: a control step of performing additional pre-learning to obtain, by learning, a target pre-trained model used to obtain a mathematical model that makes inferences based on target data, which is data of one or more specified modalities; in the additional pre-learning, learning is performed on a learning object whose initial value is the initial model using an initial model that is a deep learning model that has been pre-trained in advance by an arbitrary learning method, and extended data in which data that is the same as or different from the target data is added to the target data; the target pre-trained model is the learning object at the time when a learning termination condition, which is a specified condition for terminating the additional pre-learning, is satisfied; and in the additional pre-learning, the learning object is updated so as to at least reduce the difference between the features of the target data inferred by the initial model and the features of the extended data inferred by the learning object.

[0008] The present invention makes it possible to improve the accuracy of estimation of a mathematical model obtained using a pre-trained model.

[0009] An explanatory diagram for explaining an inference system of an embodiment. A diagram showing an example of the hardware configuration of a learning device in an embodiment. A diagram showing an example of the flow of processing executed by the learning device in an embodiment. A diagram showing an example of the hardware configuration of an inference device in an embodiment. A diagram showing an example of the flow of processing executed by the inference device in an embodiment.

[0010] (Embodiment) FIG. 1 is an explanatory diagram illustrating an inference system 100 according to an embodiment. The inference system 100 includes a learning apparatus (Learning Apparatus) 1 and an inference apparatus (Inference Apparatus) 2. The inference apparatus 2 performs inference using an inference model, which is a mathematical model obtained by further learning a pre-trained mathematical model (hereinafter referred to as a "target pre-trained model") obtained by the learning apparatus 1. Therefore, the target pre-trained model is a mathematical model used to obtain an inference model, and the target pre-trained model is a pre-trained model for the inference model. Note that the further learning may be, for example, fine tuning or transfer learning.

[0011] An inference model is a mathematical model that performs inference based on target data, which is data of one or more specified modalities. Therefore, when obtaining an inference model using a target pre-trained model, the target pre-trained model is updated based on the results of inputting target data into the target pre-trained model. The trained target pre-trained model is the inference model.

[0012] The target data may be any type of data that can be input to a mathematical model, such as sound data such as speech, acoustics, or environmental sounds, image data, data representing language, data representing graphs, data representing tables, video data, or text data. Therefore, the modality may be, for example, sound data such as speech, acoustics, or environmental sounds, an image, language, a graph, a table, video, or text.

[0013] The learning device 1 includes a control unit 11 including a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, connected via a bus. The control unit 11, for example, executes additional pre-learning. Additional pre-learning is a process for acquiring a target pre-trained model through learning. Hereinafter, the mathematical model to be learned in additional pre-learning is referred to as the learning target model. The learning target model at the time when a learning termination condition, which is a predetermined condition for terminating additional pre-learning, is satisfied is the target pre-trained model.

[0014] The learning termination condition may be any condition related to the termination of additional pre-learning, such as a condition that the number of learning steps reaches a predetermined number. This learning termination condition may be used, for example, when the additional pre-learning is M2D as described in Non-Patent Document 1. The learning termination condition may be, for example, a condition that the change in the output of the model to be trained due to the update is smaller than a predetermined change. The learning termination condition may also be a condition that a user inputs an instruction to terminate the learning device 1 via the interface unit 12 described below.

[0015] In pre-training, the mathematical model at the time when the learning termination condition is met is generally called a pre-trained mathematical model. Therefore, if the term "pre-trained" is used, the model to be trained at the time when the learning termination condition is met is a pre-trained model to be trained.

[0016] In the additional pre-training, training is performed using an initial model, a target dataset, and an extended dataset. The initial model is a deep learning model that has been pre-trained using an arbitrary training method. The deep learning model is a mathematical model whose processing content is updated through deep learning. Therefore, specifically, the deep learning model is a deep neural network. The pre-training to obtain the initial model (i.e., the above training method) may be, for example, Masked Autoencoders (MAE) described in Non-Patent Document 4 or M2D described in Non-Patent Document 1.

[0017] A target dataset is a set of target data. An extended dataset is a set of data (hereinafter referred to as "extended data") in which arbitrary data (hereinafter referred to as "background data"), which may be the same as or different from the target data, is added to the target data. Therefore, if the target data and background data are expressed as tensors belonging to the same additive group, the extended data is the result of adding the tensors of the target data and the tensors of the background data. Therefore, the extended dataset includes extended data in which the background data is different but the target data is the same. Note that the extended data does not need to be created in advance before learning; it may be generated each time by randomly adding the background data and the target data during learning.

[0018] The set of background data may be any data set, such as a so-called large-scale data set, etc. The set of background data may be data used in pre-learning to obtain an initial model, for example.

[0019] In the additional pre-learning, the target model is at least updated so as to reduce the difference between the feature quantities of the target data inferred by a teacher model (described later) and the feature quantities of the augmented data inferred by the target model. In other words, in the additional pre-learning, the target model is trained so as to reduce the distillation loss, which is the loss indicating the difference between the feature quantities of the target data inferred by the teacher model and the feature quantities of the augmented data inferred by the target model.

[0020] The teacher model is a mathematical model used in additional pre-learning and is the initial model itself. Unlike the training model, target data, not augmented data, is input to the teacher model. Unlike the training model, the teacher model is not updated during training of the training model. Therefore, the teacher model is the initial model itself, independent of the time point of training of the training model. Note that since the initial model is a deep learning model, the teacher model has an encoder. Therefore, the teacher model can infer the features of the target data.

[0021] In additional pre-learning, the initial value of the model to be trained is the initial model. The initial value of a mathematical model refers to the initial state of the mathematical model in which the parameter values ​​are updated through training. Therefore, in additional pre-learning, the teacher model remains the initial model, while the model to be trained changes from the initial model.

[0022] Since the initial model is a deep learning model, the training model that uses the initial model as an initial value includes an encoder, which allows the training model to infer the features of the augmented data.

[0023] In addition to the distillation loss, the loss used in the additional pre-training may also include the prediction loss of self-supervised learning (hereinafter referred to as "self-prediction loss"). The self-prediction loss is a loss that does not include the distillation loss, is a loss that is used when only self-supervised learning is performed on the model to be trained, and is a loss that updates the model to be trained so that its value becomes smaller.

[0024] Therefore, for example, when the model to be trained is Masked Modeling Duo (M2D) described in Non-Patent Document 1, the self-prediction loss is the loss L described in Equation (2) of Non-Patent Document 1. For example, when the model to be trained is Masked Modeling Duo for Speech (M2D-S) described in Non-Patent Document 2, the self-prediction loss is the loss L described in Equation (1) of Non-Patent Document 2. ｍ２ｄ may be.

[0025] For example, when the model to be trained is the Masked Autoencoders (MAE) of Non-Patent Document 4, the self-prediction loss may be the loss mean square error (MSE) described in Non-Patent Document 4. For example, when the model to be trained is the Bidirectional Encoder Representations from Transformers (BERT), the self-prediction loss may be the loss due to the prediction errors of the Masked Learning Model (Masked LM) and the Next Sentence Prediction task.

[0026] <Effects of additional pre-training> We will explain the effects of additional pre-training. In the field of machine learning, explanations are often given by replacing the mathematical model with a human. Here, we will use this explanation method to explain the effects of additional pre-training.

[0027] Before explaining, let us explain the meaning of distillation loss. As mentioned above, if we replace the mathematical model with a human, the teacher model can be considered the teacher, and the learning model can be considered the student. If we consider the teacher model as the teacher and the learning model as the student, distillation loss can be said to be a quantity that indicates the degree of agreement between the student's inference results and the answer given by the teacher. This is the meaning of distillation loss.

[0028] As described above, the augmented data used in the additional pre-learning may include data with different background data but the same target data. Considering that the target data is the same, even if the background data is different, the student is updated to infer similar features for the augmented data with different background data but the same target data.

[0029] This is because the features inferred by the teacher for each augmented data set with different background data but the same target data are the same, and the student is updated to reduce the distillation loss. Therefore, through additional pre-training, the student gains the knowledge that even if the input data is different, the encoding results may be the same.

[0030] As a result, students who have learned through additional pre-learning can make inferences based on a broader range of knowledge than a mathematical model updated simply using target data. Therefore, students who have learned through additional pre-learning can make inferences with higher accuracy for a larger number of different input data than a mathematical model updated simply using target data. This improves the accuracy of estimations from the inference model obtained using the target pre-learned model (i.e., the trained learning target model).

[0031] 2 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 having a processor 91 and a memory 92 connected by a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0032] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0033] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 acquires data used for additional pre-learning, such as target data and background data, for example, via the interface unit 12. The control unit 11 generates augmented data based on, for example, the target data and background data.

[0034] The control unit 11, for example, acquires an initial model. The control unit 11 acquires the initial model from a device that transmitted the initial model, for example, via the interface unit 12. Note that the control unit 11 does not necessarily have to acquire the initial model via the interface unit 12. The control unit 11 may execute an initial model acquisition process. The initial model acquisition process is a process of acquiring an initial model by performing predetermined pre-learning on a predetermined deep learning model. The predetermined pre-learning may be any pre-learning that is predetermined.

[0035] The control unit 11 executes, for example, additional pre-learning. The control unit 11 executes, for example, an inference model acquisition process. The inference model acquisition process is a process of obtaining an inference model using a target pre-trained model obtained by additional pre-learning. More specifically, the inference model acquisition process is a process of inputting target data into a target pre-trained model and updating the target pre-trained model based on the results obtained from the target pre-trained model to which the target data has been input, until a predetermined termination condition is met. The target pre-trained model at the time the predetermined termination condition is met is the inference model.

[0036] The control unit 11 acquires, for example, information stored in the storage unit 13. Specifically, the process of acquiring information stored in the storage unit 13 is a read process.

[0037] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data used for additional pre-learning, such as target data and background data. In such a case, the interface unit 12 acquires the data used for additional pre-learning by communicating with the device that transmits the data used for additional pre-learning.

[0038] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the study device 1. In this way, the interface unit 12 accepts input of various information to the study device 1 via the input device, either wired or wireless. Note that the various data acquired by the interface unit 12 described above does not necessarily have to be acquired by the interface unit 12 itself, but may be input to an input device.

[0039] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0040] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 stores information acquired by the interface unit 12, for example.

[0041] 3 is a diagram showing an example of the flow of processing executed by the learning device 1 in this embodiment. For simplicity, in FIG. 3, the example is explained in which the target data and the extended data are pre-stored in the storage unit 13. The control unit 11 acquires an initial model (step S101). Next, the control unit 11 executes additional pre-learning until a learning termination condition is met (step S102). Next, the control unit 11 executes an inference model acquisition process (step S103).

[0042] As described above, the learning device 1 includes a control unit 11 that executes additional pre-learning, which acquires, through training, a target pre-trained model used to acquire a mathematical model that performs inference based on target data, which is data of one or more predetermined modalities. In the additional pre-learning, learning is performed on a learning target whose initial value is the initial model, using an initial model that is a deep learning model that has been pre-trained by an arbitrary learning method and augmented data in which data identical to or different from the target data has been added to the target data. The target pre-trained model is the learning target at the time when a learning termination condition, which is a predetermined condition for terminating the additional pre-learning, is satisfied. In the additional pre-learning, the learning target is updated so as to at least reduce the difference between the feature values ​​of the target data inferred by the initial model and the feature values ​​of the augmented data inferred by the learning target.

[0043] 4 is a diagram showing an example of the hardware configuration of the inference device 2 in an embodiment. The inference device 2 has a control unit 21 including a processor 93 and a memory 94 connected by a bus, and executes a program. By executing the program, the inference device 2 functions as a device including the control unit 21 (inference unit), an interface unit 22, and a memory unit 23.

[0044] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the inference device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0045] The control unit 21 controls the operation of each functional unit included in the inference device 2. The control unit 21 executes, for example, an inference model. The control unit 21 acquires target data via, for example, the interface unit 22.

[0046] The control unit 21 acquires, for example, information stored in the storage unit 23. The process of acquiring information stored in the storage unit 23 is specifically a read process.

[0047] The interface unit 22 includes a communication interface for connecting the inference device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits target data. In such a case, the interface unit 22 acquires the target data by communicating with the device that transmits the target data. The external device is, for example, the learning device 1. In such a case, the interface unit 22 acquires the inference model by communicating with the learning device 1.

[0048] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 22 may be configured as an interface that connects these input devices to the inference device 2. In this way, the interface unit 22 accepts input of various information to the inference device 2 via an input device, either wired or wireless. Note that the various data acquired by the interface unit 22 described above does not necessarily have to be acquired by the interface unit 22 itself, and may be input to an input device.

[0049] The interface unit 22 outputs, for example, various types of information. The interface unit 22 is configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the inference device 2. The display device included in the interface unit 22 outputs, for example, information input to a communication interface or input device included in the interface unit 22.

[0050] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the inference device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 stores information acquired by the interface unit 22, for example.

[0051] 5 is a diagram showing an example of the flow of processing executed by the inference device 2 in this embodiment. For simplicity, FIG. 5 illustrates an example in which the inference device 2 has already acquired an inference model. The control unit 21 acquires target data via the interface unit 22 (step S201). Next, the control unit 21 executes the inference model on the acquired target data (step S202). The processing of step S202 infers a result corresponding to the target data. Thus, in step S202, the control unit 21 performs inference using the inference model.

[0052] <Experimental Results> The following describes the results of an experiment using the learning device 1. In the experiment, a mathematical model that infers a pathology based on respiratory sounds was acquired as an inference model by executing additional pre-learning processing and an inference model. Note that, since respiratory sounds are sounds, the modality in the experiment was sound.

[0053] In the experiments, the ICBHI2017 dataset (hereinafter referred to as "Dataset X") from the International Conference on Biomedical Health Informatics (ICBHI) was used as the target data. Therefore, the amount of sound data available for training was an extremely small number of 539 samples. In the experiments, the self-prediction loss was also used as the loss in the additional pre-training. In the experiments, the training to obtain the initial model was performed using M2D as described in Non-Patent Document 1. Therefore, in the experiments, the self-prediction loss was the loss used in M2D (specifically, the loss L described in Equation (2) in Non-Patent Document 1).

[0054] In the experiment, four types of comparison techniques, namely, a first comparison technique, a second comparison technique, a third comparison technique, and a fourth comparison technique, were used as comparison techniques to compare with the inference model obtained by the learning device 1.

[0055] The first comparison technique is a technique in which a predetermined mathematical model (hereinafter referred to as a "basic experimental model") that has been trained using only a large-scale dataset Q, without using dataset X, is further fine-tuned using dataset X, and the results are used to infer pathology based on respiratory sounds.

[0056] The second comparative technique is a technique in which the results of pre-training a basic experimental model using M2D with dataset X are again performed, and the results of fine-tuning are further performed with dataset X, and the results are used to infer pathology based on respiratory sounds.

[0057] The third comparison technique is a technique in which the results of pre-training a basic experimental model using M2D again using a set of data in which each data item in dataset X is added to each data item in large-scale dataset Q as background data, and then fine-tuning the results using dataset X are used to infer pathology based on respiratory sounds.

[0058] The fourth comparative technique is a technique described in Non-Patent Document 3.

[0059] As a result of the experiment, the average value of the first comparison technique was 60.98. The average value of the second comparison technique was 59.34. The average value of the third comparison technique was 61.38. The average value of the fourth comparison technique was 58.80. In contrast, the average value of the inference model obtained by the learning device 1 was 62.17. Thus, according to the experiment, the inference model obtained by the learning device 1 showed higher inference accuracy than any of the first to fourth comparison techniques.

[0060] The average value represents the average of specificity and sensitivity. Specificity represents the true negative rate. The true negative rate is the proportion of people without the disease who have a negative prediction result. Sensitivity represents the true positive rate. The true positive rate is the proportion of people with the disease who have a positive prediction result. Therefore, the higher the average value, the higher the accuracy of the inference.

[0061] The learning device 1 configured in this manner performs additional pre-learning, which is learning using distillation loss. Therefore, the effects described in "Effects of Additional Pre-learning" are achieved. Therefore, the learning device can improve the accuracy of estimation of the mathematical model (i.e., the inference model) obtained using the target pre-learned model.

[0062] The inference device 2 configured in this manner performs inference using the inference model, thereby improving the accuracy of estimation of the mathematical model (i.e., the inference model) obtained using the target pre-trained model.

[0063] The inference system 100 configured in this manner includes a learning device 1. Therefore, the inference system 100 can improve the accuracy of estimation of a mathematical model (i.e., an inference model) obtained using a target pre-trained model.

[0064] (Variations) Although MAE and M2D have been shown as examples of pre-training up to this point, the pre-training does not necessarily have to be MAE or M2D. For example, if the modality is natural language, BERT (Bidirectional Encoder Representations from Transformers), which is often used in the field of natural language, may be used.

[0065] The inference model acquisition process does not necessarily have to be executed by the control unit 11. The inference model acquisition process may be executed, for example, by another device different from the learning device 1. In such a case, the inference device 2 may acquire the inference model from the other device that executed the inference model acquisition process.

[0066] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0067] The inference device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0068] Note that all or part of the functions of the control unit 11, interface unit 12, and memory unit 13, and the control unit 21, interface unit 22, and memory unit 23 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0069] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0070] REFERENCE SYSTEM 100 1 Learning device 2 Inference device 11 Control unit 12 Interface unit 13 Memory unit 21 Control unit 22 Interface unit 23 Memory unit 91 Processor 92 Memory 93 Processor 94 Memory

Claims

1. A learning method comprising: a control step of executing additional pre-learning to obtain, by learning, a target pre-trained model used to obtain a mathematical model that makes inferences based on target data, which is data of one or more specified modalities; wherein in the additional pre-learning, learning is performed on a learning object whose initial value is the initial model using an initial model that is a deep learning model that has been pre-trained in advance by an arbitrary learning method, and extended data in which data that is identical to or different from the target data is added to the target data; the target pre-trained model is the learning object at the time when a learning termination condition, which is a specified condition for terminating the additional pre-learning, is satisfied; and in the additional pre-learning, the learning object is updated so as to at least reduce the difference between the features of the target data inferred by the initial model and the features of the extended data inferred by the learning object.

2. The learning method according to claim 1, wherein the learning method is Masked Autoencoders or Masked Modeling Duo.

3. The learning method according to claim 1, wherein the modality is sound.

4. An inference method comprising: a control step of executing additional pre-learning to obtain, by learning, a target pre-trained model used to obtain a mathematical model that makes inference based on target data, which is data of one or more specified modalities; in the additional pre-learning, learning is performed on a learning object whose initial value is the initial model using an initial model that is a deep learning model that has been pre-trained in advance by an arbitrary learning method, and extended data in which data that is identical to or different from the target data is added to the target data; the target pre-trained model is the learning object at the time when a learning termination condition, which is a specified condition for terminating the additional pre-learning, is satisfied; and in the additional pre-learning, the learning object is updated so as to at least reduce the difference between the feature amounts of the target data inferred by the initial model and the feature amounts of the extended data inferred by the learning object; and an inference step of performing an inference using a mathematical model obtained by the target pre-trained model obtained by a learning method.

Images