Information processing device, information processing method, and information processing program

Abstract

The present invention provides an information processing device comprising: an augmented data generation unit that refers to training data for training a machine learning model that outputs a prediction result corresponding to input data, and generates augmented data using an augmentation model; a prediction unit that obtains prediction results by respectively inputting the training data and the augmented data into the machine learning model; a machine learning model training unit that refers to the prediction results to train the machine learning model; an indicator acquisition unit that acquires an indicator pertaining to the prediction result corresponding to the augmented data; and an augmentation model training unit that trains the augmentation model on the basis of the indicator. With this arrangement, the present invention can assist with user decision-making.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to techniques for training machine learning models.

[0002] When there is a small amount of training data for training a machine learning model, the generalization error of the machine learning model increases. Therefore, a technique for generating augmented data by transforming training data to artificially enhance the training data is known. Research has also been conducted on techniques for searching for effective transformations for generating augmented data. For example, Patent Literature 1 describes a technique for generating a current data augmentation strategy at each of multiple time steps based on a quality criterion for a data augmentation strategy generated at a previous time step, training a machine learning model using training data using the current data augmentation strategy, and, after the machine learning model has been trained using the current data augmentation strategy, determining a quality criterion for the current data augmentation strategy using the machine learning model. Based on the quality criterion for the determined data augmentation strategy, a final data augmentation strategy is selected, and a final trained machine learning model is generated using the final data augmentation strategy.

[0003] International Publication No. 2019 / 222734

[0004] The related technology described in Patent Literature 1 uses a validation set consisting of one or more pieces of training data that were not used when training a machine learning model to determine quality criteria. However, when the amount of training data is small, it is difficult to secure a sufficient number of both the validation set created by dividing the training data and the remaining training data without bias. Therefore, this related technology has a problem in that when the amount of training data is small, it is difficult to generate augmented data for accurately training a machine learning model.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology for generating augmented data for training a machine learning model with high accuracy even when there is a small amount of training data.

[0006] An information processing device according to an exemplary aspect of the present disclosure includes an extended data generation means for generating extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data, a prediction means for obtaining each prediction result by inputting the training data and the extended data into the machine learning model, a machine learning model training means for training the machine learning model by referring to each prediction result, an index acquisition means for acquiring an index related to the prediction result according to the extended data, and an extended model training means for training the extended model based on the index.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes: an extended data generation process in which at least one processor generates extended data using an extended model by referencing training data for training a machine learning model that outputs prediction results according to input data; a prediction process in which the at least one processor obtains each prediction result by inputting the training data and the extended data into the machine learning model; a machine learning model training process in which the at least one processor trains the machine learning model by referencing each prediction result; an index acquisition process in which the at least one processor acquires an index related to the prediction result according to the extended data; and an extended model training process in which the at least one processor trains the extended model based on the index.

[0008] An information processing program according to an exemplary aspect of the present disclosure is an information processing program that causes a computer to function as an information processing device, and causes the computer to function as: extended data generation means that generates extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data; prediction means that obtains each prediction result by inputting the training data and the extended data into the machine learning model; machine learning model training means that trains the machine learning model by referring to each prediction result; index acquisition means that acquires an index related to the prediction result according to the extended data; and extended model training means that trains the extended model based on the index.

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that augmented data for training a machine learning model with high accuracy can be generated even when there is a small amount of training data.

[0010] Fig. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure; Fig. 2 is a flow diagram showing a flow of an information processing method according to the present disclosure; Fig. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure; Fig. 4 is a flow diagram showing a flow of an information processing method according to the present disclosure; Fig. 5 is a block diagram showing a configuration of a computer that functions as an information processing device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Information Processing Device) The configuration of the information processing device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing device 1. As shown in FIG. 1, the information processing device 1 includes an extended data generation unit 11, a prediction unit 12, a machine learning model training unit 13, an index acquisition unit 14, and an extended model training unit 15. The extended data generation unit 11 generates extended data using the extended model ML11 by referring to training data for training the machine learning model ML12. The machine learning model ML12 is a model that outputs prediction results according to input data. The prediction unit 12 obtains each prediction result by inputting the training data and the extended data to the machine learning model ML12. The machine learning model training unit 13 trains the machine learning model ML12 by referring to each prediction result. The index acquisition unit 14 acquires indexes related to the prediction results according to the extended data. The extended model training unit 15 trains the extended model ML11 based on the indexes. The extended model training unit 15 trains the extended model ML11 based on the above index without referring to the input data for evaluation that was not used in training the machine learning model ML12.

[0014] (Effects of the Information Processing Device) As described above, the information processing device 1 is configured to include the above-described extended data generation unit 11, prediction unit 12, machine learning model training unit 13, index acquisition unit 14, and extended model training unit 15. Therefore, there is no need to use data not used in training to train the extended model ML11. As a result, the information processing device 1 has the effect of being able to generate extended data for training the machine learning model ML12 with high accuracy even when there is little training data.

[0015] (Flow of Information Processing Method) The flow of information processing method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of information processing method S1. As shown in Fig. 2, information processing method S1 includes an extended data generation process S11, a prediction process S12, a machine learning model training process S13, an index acquisition process S14, and an extended model training process S15.

[0016] In the augmented data generation process S11, at least one processor generates augmented data using the augmented model ML11 by referencing training data for training the machine learning model ML12. The machine learning model ML12 is a model that outputs prediction results according to input data. In the prediction process S12, the at least one processor obtains each prediction result by inputting the training data and the augmented data into the machine learning model ML12. In the machine learning model training process S13, the at least one processor trains the machine learning model ML12 by referencing each prediction result. In the index acquisition process S14, the at least one processor acquires an index related to the prediction result according to the augmented data. In the augmented model training process S15, the at least one processor trains the augmented model ML11 based on the index. Note that in the augmented model training process S15, the at least one processor trains the augmented model ML11 based on the index without referencing evaluation input data that was not used in training the machine learning model ML12.

[0017] (Effects of the Information Processing Method) As described above, the extended data generation process S11 is configured to include the above-described extended data generation process S11, prediction process S12, machine learning model training process S13, index acquisition process S14, and extended model training process S15. Therefore, it is not necessary to use data not used in training to train the extended model ML11. As a result, the information processing method S1 has the effect of being able to generate extended data for training the machine learning model ML12 with high accuracy even when there is little training data.

[0018] (Example of Index: First Index) A first index, which is an example of the above-mentioned "index related to the prediction result according to the extended data," will be described. The first index is an index indicating the epistemic uncertainty of the prediction result according to the extended data. In this case, the extended model training unit 15 trains the extended model ML11 so as to increase the first index. The uncertainty of the prediction result is an index indicating the degree to which the prediction result for the data is uncertain. The uncertainty of the prediction result is different from the error rate. While the error rate is an index indicating the proportion of incorrect prediction results among prediction results for multiple data, the uncertainty of the prediction result is an index indicating the degree to which the prediction result for certain data is uncertain.

[0019] For example, suppose the training data is image data of a landscape and is associated with "summer" among the classes representing the four seasons. If the prediction result that the augmented data generated from this training data is "summer" has a high degree of uncertainty, the augmented data may have characteristics of summer image data that were difficult to identify with the previous machine learning model ML12. If the machine learning model ML12 is trained using such augmented data, it is expected to be able to accurately identify a wider variety of summer image data.

[0020] Therefore, by applying the first index and training the augmented model ML11 to increase the uncertainty of the prediction results according to the augmented data, it is possible to generate augmented data with more uncertain prediction results. As a result, it is possible to train the machine learning model ML12 to make more reliable predictions for augmented data that previously produced uncertain predictions, thereby improving the performance of the machine learning model ML12.

[0021] (Example of Index: Second Index) A second index, which is another example of the above-mentioned "index related to the prediction result according to the extended data," will be described. The second index is an index indicating the low likelihood of an incorrect class included in the prediction result according to the extended data. When the second index is used, the training data is associated with a correct class among multiple classes, and the prediction result output from the machine learning model ML12 includes the class likelihood of each of the multiple classes. In this case, the extended model training unit 15 trains the extended model ML11 to increase the second index. Class likelihood is an index indicating the probability that the prediction result is the class. Increasing the index indicating the low likelihood of an incorrect class means lowering the likelihood of an incorrect class. The higher the likelihood of the extended data being a class different from the correct class associated with the original training data, the more likely it is that the data does not retain the meaning of the original training data.

[0022] For example, suppose the training data is image data of a landscape and is associated with "summer" among the classes representing the four seasons. Furthermore, suppose that prediction results based on augmented data generated from this training data show that the resemblance of each of the classes "spring," "autumn," and "winter" is high, in addition to the original correct answer "summer." Such augmented data is likely to not retain the meaning of the original training data (a summer image), for example, because the original training data has been subjected to extreme color transformation, making it difficult to distinguish the seasons. Training the machine learning model ML12 using such augmented data that does not retain the meaning of the original training data does not contribute to improving the performance of the machine learning model ML12. Alternatively, it may even degrade the performance of the machine learning model ML12. Therefore, it is desirable to train the augmented model ML11 so as not to generate augmented data that does not retain the meaning of the original training data (i.e., so that the resemblance of each of the classes "spring," "autumn," and "winter" is low, in addition to the original correct answer "summer").

[0023] Therefore, by applying the second metric and training the extended model ML11 to reduce the likelihood of incorrect classes included in the prediction results based on the extended data, it is possible to generate extended data that more accurately preserves the meaning of the original training data. In other words, the extended data generated by the extended model ML11 is less likely to be difficult to distinguish. As a result, training using such extended data can improve the performance of the machine learning model ML12.

[0024] (Example of index: third index) A third index, which is yet another example of the above-mentioned "index related to the prediction result according to the extended data," will be described. The third index is an index indicating the reliability of the prediction result according to the extended data. In this case, the extended model training unit 15 trains the extended model ML11 so as to increase the third index. The reliability of the prediction result is an index indicating the degree to which the prediction result for the data is determined. The reliability of the prediction result is different from the accuracy rate. While the accuracy rate is an index indicating the proportion of correct prediction results among prediction results for multiple data, the reliability of the prediction result is an index indicating the degree to which the prediction result for certain data is determined.

[0025] For example, suppose the training data is image data of a landscape and is associated with "summer" among the classes representing the four seasons. For augmented data generated from this training data, if the prediction result has low certainty, regardless of whether it is correct ("summer") or incorrect ("spring," "autumn," or "winter"), the augmented data may be outside the domain (landscape images) of the original training data. Training the machine learning model ML12 using such augmented data outside the domain of the original training data does not contribute to improving the performance of the machine learning model ML12. Alternatively, it may even degrade the performance of the machine learning model ML12. Therefore, it is desirable to train the augmented model ML11 so as not to generate augmented data outside the domain of the original training data.

[0026] Therefore, by applying the third metric and training the augmented model ML11 to increase the reliability of prediction results according to the augmented data, it is possible to generate augmented data that preserves the domain of the original training data. As a result, training using such augmented data can improve the performance of the machine learning model ML12.

[0027] (Selection or Combination of Indicators) Note that which of the first, second, and third indices described above to apply may be determined depending on which of the effects obtained by applying each of them is most important. Furthermore, the extended model ML11 may be trained sequentially in any order using the first, second, and third indices. For example, the extended model ML11 may be trained using a certain set of training data to increase the first index, and the trained extended model ML11 may then be further trained using the same set of training data to increase the second index.

[0028] Furthermore, the first index and the third index described above have a trade-off relationship. For example, increasing the first index (uncertainty of the prediction result) decreases the third index (certainty of the prediction result), and vice versa. Therefore, the third index may be maintained at a moderate level by bringing the first index closer to a target value. Alternatively, the first index may be maintained at a moderate level by bringing the third index closer to a target value. In other words, the extended model training unit 15 may train the extended model ML11 so as to bring the above-mentioned indexes closer to their target values.

[0029] This makes it possible to generate augmented data that satisfies the balance between the uncertainty of prediction results and the need to stay within the domain. As a result, by training the machine learning model ML12 using such augmented data, the performance of the machine learning model ML12 can be improved.

[0030] (Example of Use in Combination with Other Indicators) Furthermore, for example, when the second index (low likelihood of a non-correct class) is applied to generate augmented data that maintains the meaning of the original training data, such augmented data may already be data with which the machine learning model ML12 can accurately predict the correct answer. Even if the machine learning model ML12 is trained using such augmented data, the contribution to performance improvement may be small. Furthermore, for example, when the third index (certainty of prediction result) is applied to generate augmented data that does not deviate from the domain of the original training data, such augmented data may already be data with which the machine learning model ML12 can accurately predict the correct answer. Even if the machine learning model ML12 is trained using such augmented data, the contribution to performance improvement may be small.

[0031] Therefore, when applying the second or third index, it may be used in conjunction with another index that indicates the possibility that the machine learning model ML12 is currently unable to accurately predict the correct answer from the augmented data. Examples of other indexes include the fourth index "loss of the machine learning model ML12," the fifth index "strength of the transformation by the augmented model ML11," and the sixth index "difference between the training data and the augmented data." For example, a specific example of the fourth index "loss of the machine learning model ML12" is cross-entropy when the machine learning model ML12 is a classification model. Another specific example of the fourth index is mean squared error when the machine learning model ML12 is a regression model. A specific example of the fifth index "strength of the transformation by the augmented model ML11" is the sum of squares of the decrease from the maximum value that a parameter defining the strength can take. A specific example of the sixth index "difference between the training data and the augmented data" is the distance between the features of the training data and the features of the augmented data in feature space. The extension model training unit 15 may train the extension model ML11 so as to increase the second index or the third index, and to increase at least one of the fourth index, the fifth index, and the sixth index.

[0032] In other words, when applying the second index or the third index, the augmented model training unit 15 may train the augmented model ML11 so as to increase the index and to increase the loss of the machine learning model ML12. Furthermore, when applying the second index or the third index, the augmented model training unit 15 may train the augmented model ML11 so as to increase the index and to increase the strength of the transformation by the augmented model ML11. Furthermore, when applying the second index or the third index, the augmented model training unit 15 may train the augmented model ML11 so as to increase the index and to increase the difference between the training data and the augmented data.

[0033] When the extended model ML11 is trained so as to increase at least one of the fourth, fifth, and sixth indices while increasing the second index, it is possible to generate extended data that is likely to be inaccurate for the machine learning model ML12 in its current state while preserving the meaning of the original training data. As a result, training using such extended data can improve the performance of the machine learning model ML12.

[0034] Furthermore, by training the extended model ML11 so as to increase at least one of the fourth, fifth, and sixth indices while increasing the third index, it is possible to generate extended data that the machine learning model ML12 is unlikely to be able to predict accurately at present while maintaining the domain consistent with the original training data. As a result, training using such extended data can improve the performance of the machine learning model ML12.

[0035] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0036] (Configuration of Information Processing Device) The configuration of the information processing device 1A will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the configuration of the information processing device 1A. As shown in FIG. 3, the information processing device 1A is communicatively connected to a training data storage device 2. Details of the training data storage device 2 will be described later. Also, as shown in FIG. 3, the information processing device 1A includes an extended data generation unit 11, a prediction unit 12, a machine learning model training unit 13, an index acquisition unit 14, and an extended model training unit 15. These functional blocks are configured in substantially the same manner as the functional blocks of the same name provided in the information processing device 1, and are also configured as follows.

[0037] The machine learning model ML12 used by the prediction unit 12 is a model that predicts the class of input data. The machine learning model ML12 is composed of a feature extraction model ML121 and a classification prediction model ML122. The feature extraction model ML121 extracts feature information that indicates the features of the input data. The classification prediction model ML122 predicts the class corresponding to the feature information.

[0038] The machine learning model training unit 13 is configured to train a machine learning model ML12 that is configured from a feature extraction model ML121 and a classification prediction model ML122. The feature extraction model ML121, the classification prediction model ML122, and the machine learning model training unit 13 may be realized by a known technique such as Posterior Networks (see Reference 1).

[0039] [Reference 1] Bertrand Charpentier et. al, "Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts", arXiv:2006.09239v2 [cs.LG] 22 Oct 2020 When using Posterior Networks, the classification prediction model ML122 acquires a probability distribution for each class in the feature space using a method called Normalizing Flow. The classification prediction model ML122 also calculates a posterior probability based on the acquired probability distribution for each class, and outputs a prediction result for the class corresponding to the feature information and an index related to the prediction result. The index acquisition unit 14 acquires the index output from the classification prediction model ML122.

[0040] For example, when using the first index (uncertainty of the prediction result), the uncertainty of the prediction result may be determined by the lowness of the parameter alpha of the corresponding class component in the Dirichlet distribution output by the classification prediction model ML122 and / or the differential entropy of the corresponding class component.

[0041] Furthermore, when the first index (uncertainty of the prediction result) is used, the uncertainty of the prediction result may be determined by the variability of the prediction results output when the same feature information is input multiple times to the classification-prediction model ML122. In this case, the variability may be obtained by sampling the prediction results using a dropout technique that randomly deactivates some of the nodes included in the classification-prediction model ML122.

[0042] Furthermore, when the second index (the lowness of the likelihood of the incorrect class) is applied, the lowness of components other than the corresponding class components of the Dirichlet distribution parameter alpha output by the classification / prediction model ML122 may be used as the lowness of the likelihood of the incorrect class. Alternatively, because the classification / prediction model ML122 itself is still undergoing training, the "lowness of the likelihood of the incorrect class" may be calculated as follows: That is, the "lowness of the likelihood of the incorrect class" may be determined by dividing the lowness of components other than the corresponding class components for the augmented data by the lowness of components other than the corresponding class components for the corresponding training data before conversion, and then normalizing the ratio.

[0043] Furthermore, when the third index (certainty of the prediction result) is used, the relevant class component of the parameter alpha in the Dirichlet distribution output by the classification prediction model ML122 may be applied as the certainty of the prediction result.

[0044] Note that the following description will focus on an example in which the machine learning model ML12 has the above-described configuration, but the machine learning model ML12 is not limited to the above-described configuration and does not necessarily have to be a model that predicts a class. The machine learning model ML12 may be any model that can output, for input data, information indicating a prediction result together with an index related to the prediction result.

[0045] Furthermore, when training the machine learning model ML12 by referring to the prediction results corresponding to the augmented data, the machine learning model training unit 13 may switch the classification prediction model ML122 to another classification prediction model for training. The other classification prediction model may be, for example, a linear classifier, but is not limited to this. In this case, the index acquisition unit 14 may calculate the above-described index by referring to the prediction results output by the other classification prediction model.

[0046] This prevents the class-specific probability distribution in the feature space acquired by the classification-prediction model ML122 based on the training data from being updated by the extended data, thereby preventing the classification-prediction model ML122 from overfitting to the extended data. For example, the machine learning model training unit 13 may switch the classification-prediction model ML122 to another classification-prediction model when training the extended model ML11 by referring to prediction results based on extended data generated when the extended model ML11 has not yet been sufficiently trained. Alternatively, by appropriately switching the classification-prediction model ML122, it is possible to select a structure for the machine learning model ML12 without being limited to this, while still being able to calculate the indices necessary for training the extended model ML11.

[0047] The extended data generator 11 generates extended data from training data and supervised data using an extended model ML11. The extended model ML11 may be a model that, when one or both of the training data and the supervised data are input, outputs parameters to be applied to the training data.

[0048] An example of the extended model ML11 will be described. For example, the extended model ML11 may be a model that generates extended data by performing adaptive transformation with reference to one or both of training data and ground truth data. An example of such an extended model ML11 can be realized using elements of a technology called Teach Augment (see Reference 2).

[0049] [Reference 2] Teppei Suzuki, "TeachAugment: Data Augmentation Optimization Using Teacher Knowledge", arXiv:2202.12513v3 [cs.CV] 28 Mar 2022 When following Teach Augment, the augmented model ML11 is configured by a neural network that includes random noise in its input. The input of the augmented model ML11 may include image data and ground truth data as training data. The output of the augmented model ML11 may include geometric transformation parameters and color transformation parameters to be applied to the image data.

[0050] The extended model ML11 does not have to be a model that performs adaptive transformation as described above. For example, the extended model ML11 may be a model that samples and outputs a parameter corresponding to the transformation strength according to a Gaussian distribution with a finite truncated range. In this case, the extended model ML11 may be trained by adjusting the upper limit of the Gaussian distribution range for each correct answer data or adjusting it commonly. In this case, the extended model training unit 15 generates extended data by transforming the training data using parameters determined using the extended model ML11 regardless of the training data.

[0051] (Information Stored in the Training Data Storage Device) The training data storage device 2 stores multiple training data and correct answer data associated with each training data. The training data is data for training the machine learning model ML12. The training data may be, for example, image data, time-series data, point cloud data, etc., but is not limited to these. The correct answer data indicates the correct class corresponding to the training data among multiple classes.

[0052] (Flow of information processing method S10) The flow of information processing method S10 executed by information processing device 1A will be described with reference to Fig. 4. Fig. 4 is a flow diagram illustrating the flow of information processing method S10. As shown in Fig. 4, information processing method S10 includes steps S101 to S107.

[0053] In step S101, the information processing device 1A acquires, from the training data storage device 2, one piece of training data and the correct answer data associated with the training data.

[0054] Step S102 is an example of an extended data generation process. In step S102, the extended data generation unit 11 references one or both of the training data and the correct answer data and generates extended data using the extended model ML11. For example, when using the extended model ML11 that performs adaptive transformation, the extended data generation unit 11 inputs one or both of the training data and the correct answer data to the extended model ML11 and obtains transformation parameters to be output. Alternatively, when using the extended model ML11 that does not perform adaptive transformation, the extended data generation unit 11 obtains the transformation parameters using the extended model ML11. Furthermore, the extended data generation unit 11 generates extended data by performing a transformation process on the training data using the transformation parameters.

[0055] Step S103 is an example of a prediction process and an index acquisition process. In step S103, the prediction unit 12 uses the machine learning model ML12 to acquire prediction results corresponding to the training data and the augmented data. For example, the prediction unit 12 inputs the training data to a feature extraction model ML121. The feature extraction model ML121 outputs feature information of the training data. The prediction unit 12 also inputs the feature information to a classification prediction model ML122. The classification prediction model ML122 outputs a prediction result of a class corresponding to the training data. Note that the output of the classification prediction model ML122 includes an index related to the prediction result corresponding to the training data, but the index is not used.

[0056] Similarly, the prediction unit 12 inputs the extended data to the feature extraction model ML121. The feature extraction model ML121 outputs feature information of the extended data. The prediction unit 12 also inputs the feature information to the classification prediction model ML122. The classification prediction model ML122 outputs a prediction result of a class according to the extended data. The output of the classification prediction model ML122 also includes an index related to the prediction result according to the extended data. The index acquisition unit 14 acquires the index. For example, the index acquisition unit 14 acquires at least one of a first index, a second index, and a third index related to the prediction result of the extended data from the output of the classification prediction model ML122. The index is used in steps described below.

[0057] Step S104 is an example of a machine learning model training process. In step S104, the machine learning model training unit 13 trains the machine learning model ML12 by referring to the prediction results obtained in step S102. Specifically, for example, the machine learning model training unit 13 trains the feature extraction model ML121 and the classification prediction model ML122 so that the prediction results based on the training data approach the ground truth data associated with the training data. The machine learning model training unit 13 also trains the feature extraction model ML121 and the classification prediction model ML122 so that the prediction results based on the augmented data approach the ground truth data associated with the original training data.

[0058] However, as described above, when training the machine learning model ML12 by referring to the prediction results according to the training data, the machine learning model training unit 13 may switch the classification prediction model ML122 to another classification prediction model. For example, the machine learning model training unit 13 may evaluate the performance of the extended model ML11 at this point, and if the performance does not satisfy a predetermined level, switch the classification prediction model ML122 to another classification prediction model and continue training.

[0059] Step S105 is an example of an extension model training process. In step S105, the extension model training unit 15 trains the extension model ML11 based on the index related to the prediction result corresponding to the extension data obtained in step S103. For example, the extension model training unit 15 may train the extension model ML11 so as to increase any one of the first index, the second index, and the third index. Furthermore, for example, when using the second index or the third index, the extension model training unit 15 may also use the above-mentioned fourth index, fifth index, or sixth index in combination. Furthermore, for example, the extension model training unit 15 may train the extension model ML11 so as to bring the first index or the third index closer to a target value.

[0060] In step S106, the information processing device 1A determines whether to end the training. Examples of the conditions for ending the training include, but are not limited to, reaching a predetermined number of epochs, the parameters of the machine learning model ML12 converging, and the performance of the machine learning model ML12 reaching a predetermined level.

[0061] If it is determined in step S106 that the training should not be terminated, the information processing device 1A repeats the processing from step S101. If it is determined in step S106 that the training should be terminated, the information processing device 1A determines the machine learning model ML12 at this point as the machine learning model ML12 after training. This completes the operation of the information processing method S10.

[0062] [Application Example 1] In each of the exemplary embodiments described above, the information processing device 1A may output some or all of the first index, the second index, and the third index so that the user can recognize them, thereby assisting a user such as a data scientist in making decisions when performing data augmentation (i.e., augmenting training data to generate augmented data).

[0063] [Application Example 2] The exemplary embodiments described above can be applied to the medical and healthcare fields. For example, the exemplary embodiments are useful in fields such as pathological diagnosis or endoscopic image diagnosis. For example, in pathological diagnosis, it is necessary to find abnormal patterns by observing subtle cellular changes and / or tissue characteristics in pathological images. Furthermore, in endoscopic image diagnosis, it is necessary to detect lesions in images captured with an endoscope. Therefore, data augmentation techniques specific to pathological diagnosis or endoscopic image diagnosis include methods such as enlarging specific portions of an image or enhancing the color tone of abnormal areas. However, since it is unclear what data augmentation techniques are effective, the exemplary embodiments described above can be used to train a machine learning model ML12 (hereinafter referred to as image diagnosis AI: artificial intelligence) that performs image diagnosis while optimizing the data augmentation technique.

[0064] Furthermore, training of the augmented model ML11 in such a specific medical scenario needs to be performed to meet medical-specific requirements. For example, image diagnosis used in medical settings is often used to support doctors' decision-making. For example, image diagnosis AI may require fewer false negatives rather than high diagnostic accuracy. In such cases, the present exemplary embodiment can be modified to allow the user to select the criteria for data augmentation.

[0065] Specifically, when preventing missed detection of abnormal patterns in pathological image diagnosis AI, for example, by training the extension model ML11 to increase the second index (low likelihood of an incorrect class), it becomes difficult to generate extended data that predicts that an abnormal pattern is not an abnormal pattern even though it is an abnormal pattern. Also, when preventing missed detection of lesion sites in endoscopic image diagnosis AI, for example, by training the extension model ML11 to increase the second index in the same manner as above, it becomes difficult to generate extended data that predicts that an abnormal pattern is not a lesion even though it is a lesion site.

[0066] In this way, in Application Example 2, the augmented model ML11 can be optimized to meet medical-specific requirements by training the augmented model ML11 to increase an index (e.g., the first index, the second index, or the third index) related to the prediction result according to the augmented data. As a result, it is expected that the accuracy of diagnosis in medical settings such as pathological diagnosis will be improved, and that this will directly contribute to the treatment outcomes of patients.

[0067] [Software Implementation Example] Some or all of the functions of the information processing devices 1 and 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0068] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 5. Figure 5 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0069] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0070] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0071] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0072] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0073] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0074] (Appendix A1) An information processing device comprising: an extended data generation means for generating extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data; a prediction means for obtaining each prediction result by inputting the training data and the extended data into the machine learning model; a machine learning model training means for training the machine learning model by referring to each prediction result; an index acquisition means for acquiring an index related to the prediction result according to the extended data; and an extended model training means for training the extended model based on the index.

[0075] (Supplementary Note A2) The information processing device according to Supplementary Note A1, wherein the index is an index indicating uncertainty of the prediction result according to the extended data, and the extended model training means trains the extended model so as to increase the index.

[0076] (Appendix A3) The information processing device described in Appendix A1, wherein the training data is associated with a correct class among a plurality of classes, the prediction result output from the machine learning model includes the class-likelihood of each of the plurality of classes, the index is an index indicating the low likelihood of the incorrect class included in the prediction result according to the extended data, and the extended model training means trains the extended model to increase the index.

[0077] (Appendix A4) The information processing device according to Appendix A1, wherein the index is an index indicating a degree of certainty of the prediction result according to the extended data, and the extended model training means trains the extended model so as to increase the index.

[0078] (Supplementary Note A5) The information processing device according to Supplementary Note A3 or A4, wherein the extended model training means trains the extended model so as to increase the index and increase a loss of the machine learning model.

[0079] (Supplementary Note A6) The information processing device according to Supplementary Note A3 or A4, wherein the extension model training means trains the extension model so as to increase the index and to increase the strength of the transformation by the extension model.

[0080] (Supplementary Note A7) The information processing device according to Supplementary Note A3 or A4, wherein the extended model training means trains the extended model so as to increase the index and to increase a difference between the training data and the extended data.

[0081] (Supplementary Note A8) The information processing device according to Supplementary Note A2 or A4, wherein the extended model training means trains the extended model so as to bring the index closer to a target value.

[0082] (Appendix A9) The information processing device according to any one of Appendices A1 to A8, wherein the machine learning model is composed of a feature extraction model that extracts feature information indicating features of input data, and a classification prediction model that predicts a class corresponding to the feature information, and the machine learning model training means switches the classification prediction model to another classification prediction model for training when training the machine learning model by referring to the prediction result according to the extended data.

[0083] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0084] (Supplementary Note B1) An information processing method including: an extended data generation process in which at least one processor generates extended data using an extended model by referencing training data for training a machine learning model that outputs prediction results according to input data; a prediction process in which the at least one processor obtains each prediction result by inputting the training data and the extended data into the machine learning model; a machine learning model training process in which the at least one processor trains the machine learning model by referencing each prediction result; an index acquisition process in which the at least one processor acquires an index related to the prediction result according to the extended data; and an extended model training process in which the at least one processor trains the extended model based on the index.

[0085] (Supplementary Note B2) The information processing method according to Supplementary Note B1, wherein the index indicates uncertainty of the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model to increase the index.

[0086] (Appendix B3) The information processing method described in Appendix B1, wherein the training data is associated with a correct class among a plurality of classes, the prediction result output from the machine learning model includes the class-likelihood of each of the plurality of classes, the index is an index indicating the low likelihood of the incorrect class included in the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model to increase the index.

[0087] (Supplementary Note B4) The information processing method according to Supplementary Note B1, wherein the index indicates a degree of certainty of the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model to increase the index.

[0088] (Supplementary Note B5) The information processing method according to Supplementary Note B3 or B4, wherein, in the extension model training process, the at least one processor trains the extension model so as to increase the index and increase a loss of the machine learning model.

[0089] (Supplementary Note B6) The information processing method according to Supplementary Note B3 or B4, wherein in the extension model training process, the at least one processor trains the extension model so as to increase the index and to increase the strength of the transformation by the extension model.

[0090] (Supplementary Note B7) The information processing method according to Supplementary Note B3 or B4, wherein in the extended model training process, the at least one processor trains the extended model so as to increase the index and to increase a difference between the training data and the extended data.

[0091] (Supplementary Note B8) The information processing method according to Supplementary Note B2 or B4, wherein in the extended model training process, the at least one processor trains the extended model so as to bring the index closer to a target value.

[0092] (Appendix B9) The information processing method described in any one of Appendices B1 to B8, wherein the machine learning model is composed of a feature extraction model that extracts feature information that indicates the features of input data, and a classification prediction model that predicts a class corresponding to the feature information, and in the machine learning model training process, the at least one processor switches the classification prediction model to another classification prediction model for training when training the machine learning model by referring to the prediction result according to the extended data.

[0093] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0094] (Appendix C1) An information processing program that causes a computer to function as an information processing device, causing the computer to function as: extended data generation means that generates extended data using an extended model by referencing training data for training a machine learning model that outputs prediction results according to input data; prediction means that obtains each prediction result by inputting the training data and the extended data into the machine learning model, respectively; machine learning model training means that trains the machine learning model by referencing each prediction result; index acquisition means that acquires an index related to the prediction result according to the extended data; and extended model training means that trains the extended model based on the index.

[0095] (Supplementary Note C2) The information processing program according to Supplementary Note C1, wherein the index indicates uncertainty of the prediction result according to the extended data, and the extended model training means trains the extended model so as to increase the index.

[0096] (Appendix C3) An information processing program as described in Appendix C1, wherein the training data is associated with a correct class among a plurality of classes, the prediction result output from the machine learning model includes the class-likelihood of each of the plurality of classes, the index is an index indicating the low likelihood of the incorrect class included in the prediction result according to the extended data, and the extended model training means trains the extended model to increase the index.

[0097] (Supplementary Note C4) The information processing program according to Supplementary Note C1, wherein the index indicates a degree of certainty of the prediction result according to the extended data, and the extended model training means trains the extended model to increase the index.

[0098] (Supplementary Note C5) The information processing program according to Supplementary Note C3 or C4, wherein the extended model training means trains the extended model so as to increase the index and increase a loss of the machine learning model.

[0099] (Supplementary Note C6) The information processing program according to Supplementary Note C3 or C4, wherein the extension model training means trains the extension model so as to increase the index and to increase the strength of the transformation by the extension model.

[0100] (Supplementary Note C7) The information processing program according to Supplementary Note C3 or C4, wherein the extended model training means trains the extended model so as to increase the index and to increase a difference between the training data and the extended data.

[0101] (Supplementary Note C8) The information processing program according to Supplementary Note C2 or C4, wherein the extended model training means trains the extended model so as to bring the index closer to a target value.

[0102] (Appendix C9) The information processing program according to any one of Appendices C1 to C8, wherein the machine learning model is composed of a feature extraction model that extracts feature information indicating features of input data, and a classification prediction model that predicts a class corresponding to the feature information, and the machine learning model training means switches the classification prediction model to another classification prediction model for training when training the machine learning model by referring to the prediction result according to the extended data.

[0103] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0104] (Appendix D1) An information processing device comprising at least one processor, the at least one processor executing: an extended data generation process that generates extended data using an extended model by referencing training data for training a machine learning model that outputs prediction results according to input data; a prediction process that obtains each prediction result by inputting the training data and the extended data into the machine learning model; a machine learning model training process that trains the machine learning model by referencing each prediction result; an index acquisition process that acquires an index related to the prediction result according to the extended data; and an extended model training process that trains the extended model based on the index.

[0105] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0106] (Supplementary Note D2) The information processing device according to Supplementary Note D1, wherein the index is an index indicating uncertainty of the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model so as to increase the index.

[0107] (Appendix D3) The information processing device described in Appendix D1, wherein the training data is associated with a correct class among multiple classes, the prediction result output from the machine learning model includes the class-likelihood of each of the multiple classes, the index is an index indicating the low likelihood of the incorrect class included in the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model to increase the index.

[0108] (Supplementary Note D4) The information processing device according to Supplementary Note D1, wherein the index indicates a degree of certainty of the prediction result according to the extended data, and in the extended model training process, the at least one processor trains the extended model so as to increase the index.

[0109] (Supplementary Note D5) The information processing device according to Supplementary Note D3 or D4, wherein, in the extension model training process, the at least one processor trains the extension model so as to increase the index and increase a loss of the machine learning model.

[0110] (Supplementary Note D6) The information processing device according to Supplementary Note D3 or D4, wherein, in the extension model training process, the at least one processor trains the extension model so as to increase the index and to increase a strength of transformation by the extension model.

[0111] (Supplementary Note D7) The information processing device according to Supplementary Note D3 or D4, wherein in the extended model training process, the at least one processor trains the extended model so as to increase the index and to increase a difference between the training data and the extended data.

[0112] (Supplementary Note D8) The information processing device according to Supplementary Note D2 or D4, wherein, in the extended model training process, the at least one processor trains the extended model so as to bring the index closer to a target value.

[0113] (Appendix D9) The information processing device described in any one of Appendices D1 to D8, wherein the machine learning model is composed of a feature extraction model that extracts feature information indicating features of input data, and a classification prediction model that predicts a class corresponding to the feature information, and in the machine learning model training process, the at least one processor switches the classification prediction model to another classification prediction model for training when training the machine learning model by referring to the prediction result according to the extended data.

[0114] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0115] (Appendix E1) A non-transitory recording medium having recorded thereon an information processing program that causes a computer to function as an information processing device, the information processing program causing the computer to execute: an extended data generation process that generates extended data using an extended model by referencing training data for training a machine learning model that outputs prediction results according to input data; a prediction process that obtains each prediction result by inputting the training data and the extended data into the machine learning model; a machine learning model training process that trains the machine learning model by referencing each prediction result; an index acquisition process that acquires an index related to the prediction result according to the extended data; and an extended model training process that trains the extended model based on the index.

[0116] 1, 1A Information processing device 2 Training data storage device 11 Extended data generation unit 12 Prediction unit 13 Machine learning model training unit 14 Index acquisition unit 15 Extended model training unit ML11 Extended model ML12 Machine learning model ML121 Feature extraction model ML122 Classification prediction model

Claims

1. An information processing device comprising: an extended data generation means for generating extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data; a prediction means for obtaining each prediction result by inputting the training data and the extended data into the machine learning model, respectively; a machine learning model training means for training the machine learning model by referring to each of the prediction results; an index acquisition means for acquiring an index related to the prediction result according to the extended data; and an extended model training means for training the extended model based on the index.

2. The information processing device according to claim 1, wherein the index is an index indicating the uncertainty of the prediction result according to the extended data, and the extended model training means trains the extended model so as to increase the index.

3. The information processing device of claim 1, wherein the training data is associated with a correct class among a plurality of classes, the prediction result output from the machine learning model includes the class-likelihood of each of the plurality of classes, the index is an index indicating the low likelihood of the non-correct class included in the prediction result according to the extended data, and the extended model training means trains the extended model to increase the index.

4. The information processing device according to claim 1, wherein the index indicates the certainty of the prediction result according to the extended data, and the extended model training means trains the extended model to increase the index.

5. The information processing device according to claim 3 or 4, wherein the extended model training means trains the extended model so as to increase the index and to increase a loss of the machine learning model.

6. The information processing device according to claim 3 or 4, wherein the extension model training means trains the extension model so as to increase the index and to increase the strength of transformation by the extension model.

7. The information processing device according to claim 3 or 4, wherein the extended model training means trains the extended model so as to increase the index and to increase a difference between the training data and the extended data.

8. The information processing device according to claim 2 or 4, wherein the extended model training means trains the extended model so as to bring the index closer to a target value.

9. An information processing device as claimed in any one of claims 1 to 8, wherein the machine learning model is composed of a feature extraction model that extracts feature information indicating features of input data, and a classification prediction model that predicts a class corresponding to the feature information, and the machine learning model training means switches the classification prediction model to another classification prediction model for training when training the machine learning model by referring to the prediction result according to the extended data.

10. An information processing method comprising: an extended data generation process in which at least one processor generates extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data; a prediction process in which the at least one processor obtains each prediction result by inputting the training data and the extended data into the machine learning model, respectively; a machine learning model training process in which the at least one processor trains the machine learning model by referring to each prediction result; an index acquisition process in which the at least one processor acquires an index related to the prediction result according to the extended data; and an extended model training process in which the at least one processor trains the extended model based on the index.

11. An information processing program that causes a computer to function as an information processing device, the information processing program causing the computer to function as: extended data generation means that generates extended data using an extended model by referring to training data for training a machine learning model that outputs prediction results according to input data; prediction means that obtains each prediction result by inputting the training data and the extended data into the machine learning model, respectively; machine learning model training means that trains the machine learning model by referring to each prediction result; index acquisition means that acquires an index related to the prediction result according to the extended data; and extended model training means that trains the extended model based on the index.

Images