Machine learning program, machine learning method, and machine learning device

Abstract

To provide a machine learning program, a machine learning method, and a machine learning device which improve prediction accuracy of a model created by machine learning.SOLUTION: A machine learning program causes a computer to execute processing for: acquiring a plurality of first data having a labelled right answer and a plurality of second data different from the plurality of first data; producing a plurality of groups into which any of the plurality of first data are classified by clustering the plurality of first data and the plurality of second data together according to a condition that the plurality of first data and the plurality of second data are not classified into the same group; producing a machine learning model using, as training data, data included in a first group of the plurality of groups; and executing verification of the produced machine learning model using, as verification data, data included in a second group other than the first group of the plurality of groups.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a machine learning program, a machine learning method, and a machine learning device. [Background technology]

[0002] In recent years, the use of machine learning has expanded to various fields. In machine learning, there is supervised learning, which uses training data that is a pair of input and output. In supervised learning, a machine learning model is generated by learning the input-output relationship from multiple training data. In the following, a machine learning model will simply be called a model.

[0003] In supervised learning, it is important to avoid overfitting. Overfitting is a learning result in which the data used for learning can be predicted with good accuracy, but the prediction accuracy decreases for other data. Therefore, in order to avoid overfitting, it is common to perform validation such as hold-out validation and cross validation. Hold-out validation is a validation method in which the teacher data is divided into data for model creation and data for model validation, and if the prediction accuracy of both is equivalent, it is determined that overfitting has not occurred. The teacher data is generally divided randomly. Cross validation is an advanced form of hold-out validation, and is a validation method in which the roles of each teacher data are swapped and hold-out validation is performed multiple times.

[0004] In machine learning methods, parameters called hyperparameters are generally used to adjust the degree of learning, and hyperparameter tuning is often performed to derive optimal hyperparameters through the above-mentioned verification. After hyperparameter tuning, learning is performed using all training data with the optimal hyperparameters to create the final model.

[0005] Here, it is rare that training data can be freely prepared, and it is common to use available data as training data. In this case, the fact that it is "available" already contains a certain bias, which generates data with a high degree of similarity at a probability that is not at the level of chance. This is one of the reasons why training data may contain a large amount of data with a high degree of similarity at a probability that is not at the level of chance.

[0006] When training data that contains a large amount of data with high similarity at a probability other than chance level is randomly divided, the following problems occur. In machine learning, it is desirable to learn global rules that apply to as much data as possible. When there is a large amount of similar data, local rules that apply disproportionately to similar data are likely to be learned. In particular, when the presence of similar data is not at a chance level, there is a high possibility that excessively local rules will be learned, resulting in overfitting. However, when training data is randomly divided, the model validation data also contains a large amount of similar data. Therefore, the prediction accuracy is high in validation using the model validation data. As a result, it is difficult to detect overfitting, making it difficult to create an appropriate model and improving the prediction accuracy of the model.

[0007] One possible solution to this problem is to group data with high similarity and divide the training data by group. This method reduces the possibility that similar data is included in both the model creation data and the model validation data, and if overlearning occurs when training is performed using data with high similarity as training data, the prediction accuracy decreases during validation, making it possible to detect overlearning.

[0008] The following techniques have been proposed as supervised learning techniques. For example, there is a technique that applies a machine learning model constructed using model construction data other than the teacher data to model validation data to perform labeling, and detects mislabeling by comparing the labeling result with the label originally attached to the model validation data. There is also a technique that calculates the distance between the coordinates of expert data whose labeling reliability meets a standard and non-expert data whose labeling reliability is low, and determines the reliability of the non-expert data based on the distance between the coordinates to learn the model. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2018-155522 A [Patent Document 2] JP 2009-282686 A Summary of the Invention [Problem to be solved by the invention]

[0010] However, when highly similar data is grouped and teacher data is divided into groups, it is difficult to set a similarity standard for grouping. For example, there are cases where the similarity standard can be easily set based on knowledge about the data, but there are many cases where this is not the case. If the similarity standard is set too low in grouping, similar data may be included in both the model creation data and the model verification data due to similarity of data belonging to different groups, making it difficult to accurately detect overlearning. As described above, since it is difficult to set the similarity standard appropriately, it is difficult to improve the prediction accuracy of the model with the technology of grouping highly similar data according to knowledge and dividing teacher data into groups.

[0011] The disclosed technology has been made in consideration of the above, and aims to provide a machine learning program, a machine learning method, and a machine learning device that improve the predictive accuracy of a model created by machine learning. [Means for solving the problem]

[0012] In one embodiment of the machine learning program, machine learning method, and machine learning device disclosed in the present application, a computer is caused to execute a process of acquiring a first plurality of data labeled with a correct answer and a second plurality of data different from the first plurality of data, generating a plurality of groups into which any of the first plurality of data is classified by clustering the first plurality of data together according to a condition that the first plurality of data and the second plurality of data are not classified into the same group, generating a machine learning model using data included in a first group among the plurality of groups as training data, and verifying the generated machine learning model using data included in a second group other than the first group among the first plurality of groups as verification data. Effect of the Invention

[0013] In one aspect, the present invention can improve the predictive accuracy of a model created by machine learning. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of a machine learning device according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining hierarchical clustering. [Diagram 3] FIG. 3 is an image diagram of clustering executed by the machine learning device according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining a result of clustering by the clustering unit according to the embodiment. [Diagram 5]FIG. 5 is a flowchart of a process of generating and verifying a model according to the embodiment. [Figure 6] FIG. 6 is a hardware configuration diagram of the machine learning device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, examples of the machine learning program, the machine learning method, and the machine learning device disclosed in the present application will be described in detail with reference to the drawings. Note that the machine learning program, the machine learning method, and the machine learning device disclosed in the present application are not limited to the following examples. EXAMPLES

[0016] 1 is a block diagram of a machine learning device according to an embodiment. As shown in FIG 1, a machine learning device 1 according to the embodiment is connected to a data storage device 2 and a user terminal device 3 via a network.

[0017] The data storage device 2 holds a teacher dataset 21 and a non-teacher dataset 22. The teacher dataset 21 is a data set that collects multiple pieces of teacher data each consisting of an input and an output pair. Teacher data is data for which both an input and an output exist and are available. The teacher dataset 21 may include a group of teacher data that has a high degree of similarity with a probability other than chance level. The teacher data included in this teacher dataset 21 is an example of a "first plurality of data". On the other hand, the non-teacher dataset 22 is a data set that collects multiple pieces of non-teacher data for which an input exists but the output is unknown. The non-teacher data included in this non-teacher data 22 is an example of a "second plurality of data".

[0018] That is, the acquisition conditions of teacher data and non-teacher data are different in terms of whether or not input and output pairs are available. In this way, since the acquisition conditions of teacher data and non-teacher data are different, it is expected that the bias of each data will be different. Therefore, when the teacher data set 21 contains a teacher data group with a high similarity with a probability other than chance level, it is expected that the possibility that data similar to that data group will be included in the non-teacher data set 22 is extremely low.

[0019] A specific example of a teacher dataset 21 including teacher data with a high degree of similarity at a probability other than chance level and a non-teacher dataset 22 corresponding to the teacher dataset 21 will be described. For example, a case where the presence or absence of toxicity is predicted from the chemical structure of a compound in drug discovery can be given as an example. In other words, this is a case where a model is created in which the input is a chemical structure and the output is the presence or absence of toxicity.

[0020] The training data is a pair of information on compounds that have actually been synthesized and whose toxicity has already been measured, and the results of the toxicity measurements. Here, since the synthesis method of compounds (especially compounds that can become medicines) is basically unknown, the synthesis method is derived through research and development. In this case, by basing the synthesis method on the synthesis method of known compounds, it is relatively easy to synthesize compounds that have a similar chemical structure to the desired compound. For these reasons, the training data set 21 in this case will contain a lot of training data representing compounds with similar chemical structures.

[0021] On the other hand, in the non-teacher data, the toxicity is not required, so the compound does not have to be actually synthesized. Therefore, for example, by creating a non-teacher data set 22 including data on a compound to be predicted by the created model as non-teacher data, the bias of the data representing the chemical structure can be made different between the teacher data set 21 and the non-teacher data set 22.

[0022] The user terminal device 3 is a terminal operated by a user who wishes to have the machine learning device 1 predict desired prediction target data. The user uses the user terminal device 3 to input the desired prediction target data to the machine learning device 1. After that, the user uses the user terminal device 3 to obtain the prediction result for the desired prediction target data provided by the machine learning device 1.

[0023] 1, the machine learning device 1 includes a control unit 10. The control unit 10 includes a data acquisition unit 11, a clustering unit 12, a model generation unit 13, a verification unit 14, and a prediction unit 15.

[0024] The data acquiring unit 11 acquires the teacher dataset 21 and the non-teacher dataset 22 held by the data storage device 2 from the data storage device 2. Here, in this embodiment, the teacher dataset 21 and the non-teacher dataset 22 are held by the data storage device 2, which is an external device of the machine learning device 1, but these data may be stored in advance in a storage device (not shown) held by the machine learning device 1.

[0025] The clustering unit 12 performs clustering on each piece of teacher data included in the teacher data set 21 and each piece of non-teacher data included in the non-teacher data set 22 acquired by the data acquisition unit 11, to generate a plurality of clusters.

[0026] Fig. 2 is a diagram for explaining hierarchical clustering. In Fig. 2, the vertical axis represents the distance between clusters, and the horizontal axis represents either teacher data or non-teacher data. Here, the distance between two clusters is the longest distance between data included in one cluster and data included in the other cluster. Also, in Fig. 2, data #0 to #9 exist.

[0027] Fig. 3 is an image diagram of clustering performed by the machine learning device according to the embodiment. The circular dots in Fig. 3 represent teacher data, and the triangular dots represent non-teacher data. Details of the clustering performed by the clustering unit 12 will be described below with reference to Figs. 2 and 3.

[0028] The clustering unit 12 calculates the distance between each piece of teacher data included in the teacher dataset 21 and each piece of non-teacher data included in the non-teacher dataset 22, and sets the shortest distance as a threshold value that is a condition for stopping merging of clusters. That is, the clustering unit 12 stops merging when the distance between two clusters is equal to or greater than the shortest distance between the teacher data and the non-teacher data and is closest.

[0029] For example, the distance L in Fig. 3 is the threshold. Also, the dashed line 106 in Fig. 2 corresponds to the threshold. By setting the shortest distance between teacher data and non-teacher data as the merge stopping condition, the clustering unit 12 can detect a cluster in which there is no cluster in which teacher data and non-teacher data are mixed, when sequentially merging clusters by hierarchical clustering.

[0030] Next, the clustering unit 12 sets each of the teacher data and each of the non-teacher data as an initial cluster for clustering. For example, in the case of Fig. 2, the clustering unit 12 sets each of the data #0 to #9 as an initial cluster. In the case of Fig. 3, the clustering unit 12 sets a cluster including one each of a circle point and a triangle point as an initial cluster.

[0031] Next, the clustering unit 12 calculates the distance between each cluster. Here, the distance between the first cluster and the second cluster is the longest distance between each data included in the first cluster and each data included in the second cluster. Next, the clustering unit 12 judges whether or not the minimum value of the distance between each cluster is less than a threshold value. That is, the clustering unit 12 judges whether or not a pair of clusters whose distance between them is longer than the distance L in FIG. 3 exists. If a pair of clusters whose distance between them is longer than the distance L exists, there is a possibility that a cluster in which teacher data and non-teacher data are mixed has occurred. On the other hand, if the distance between all clusters is shorter than the distance L, there is no cluster in which teacher data and non-teacher data are mixed. Therefore, this judgment allows the clustering unit 12 to perform clustering in a state in which teacher data and non-teacher data are not mixed.

[0032] If the minimum value of the distance between each cluster is less than the threshold, the clustering unit 12 merges the clusters with the minimum distance. Then, the clustering unit 12 calculates the distance between each cluster after the merge, compares the minimum value with the threshold, and repeats the merging until the minimum value exceeds the threshold.

[0033] For example, in FIG. 2, the position where each cluster is connected represents the distance between the clusters. That is, the clustering unit 12 merges the connected clusters one by one while gradually increasing the distance from the position where the distance on the vertical axis of FIG. 2 is 0. For example, the distance between the cluster of data #5 and the cluster of data #7 is the shortest, and the clustering unit 12 first merges the cluster of data #5 with the cluster of data #7. Next, the clustering unit 12 merges the cluster of data #1 with the cluster of data #6. Next, the clustering unit 12 merges the cluster of data #3 with the cluster of data #4, and merges the cluster of data #8 with the cluster including data #5 and #7. Next, the clustering unit 12 merges the cluster of data #2 with the cluster including data #1 and #6.

[0034] On the other hand, when the minimum value of the distance between the clusters is equal to or greater than the threshold, the clustering unit 12 stops merging the clusters. In the case of FIG. 2, the clustering unit 12 gradually increases the distance, and when it determines that the distance between the cluster of data #9 and the cluster including data #1, #2, and #6 exceeds the threshold represented by the dashed line 106, it stops merging the clusters without merging them. In this case, the clustering unit 12 generates clusters 101 to 105. As a result, the clustering unit 12 can generate a cluster including teacher data such as cluster 112 and a cluster including non-teacher data such as cluster 111 as shown in FIG. 3. That is, the clustering unit 12 can generate clusters such that the number of clusters is minimized or as small as possible in a state where there is no cluster including a mixture of teacher data and non-teacher data.

[0035] Fig. 4 is a diagram for explaining the result of clustering by the clustering unit according to the embodiment. Next, the result of clustering by the clustering unit 12 according to the embodiment will be explained with reference to Fig. 4. In Fig. 4, cylinders with no pattern added represent a collection of teacher data, and cylinders with a diagonal line pattern added represent a collection of non-teacher data.

[0036] The clustering unit 12 performs clustering according to the above-mentioned procedure. As a result, the clustering unit 12 confirms that a cluster including both data included in the teacher data 21 and data included in the non-teacher data 22 has been generated for the first time in the result of the N+1th clustering in Fig. 4, and adopts the clustering result by the Nth clustering. As a result, as shown in the division state 203, it is possible to generate clusters in which data included in the teacher data set 21 and data included in the non-teacher data set 22 are not mixed and the number of clusters is as small as possible.

[0037] In contrast, when the first clustering or the second clustering, in which the distance between the clustered data is closer than in the Nth clustering, is performed, the number of clusters is larger than in the division state 203, as shown in the division states 201 and 202. That is, since the number of clusters can still be reduced, the clustering unit 12 does not adopt these clusterings. On the other hand, in the N+1th clustering and subsequent clustering, in which the distance between the clustered data is farther than in the Nth clustering, clusters that are a mixture of data included in the teacher data set 21 and data included in the non-teacher data set 22 are generated. For this reason, the clustering unit 12 does not adopt the N+1th clustering and subsequent clustering.

[0038] Then, the clustering unit 12 extracts clusters including the teacher data from the generated clusters, and outputs information on the extracted clusters to the model generation unit 13.

[0039] The model generation unit 13 receives an input of information on each cluster including the teacher data from the clustering unit 12. Then, the model generation unit 13 divides the acquired clusters into clusters for model generation and clusters for model verification. Then, the model generation unit 13 performs machine learning using the teacher data included in the clusters for model generation as training data to generate a model. Thereafter, the model generation unit 13 outputs information on the generated model and the clusters for model verification to the verification unit 14.

[0040] The verification unit 14 receives an input of the generated model and information on the cluster for model verification from the model generation unit 13. Then, the verification unit 14 applies the teacher data included in the cluster for model verification to the acquired model as verification data, and performs prediction for each of the teacher data included in the cluster for model verification. Next, the verification unit 14 compares the output information included in the teacher data included in the cluster for model verification with the prediction result, and verifies the prediction accuracy of the model.

[0041] 5 is a flowchart of the process of generating and verifying a model according to the embodiment. Next, the flow of the process of generating and verifying a model by the machine learning device 1 according to the embodiment will be described with reference to FIG.

[0042] The data acquiring unit 11 acquires the teacher data set 21 and the non-teacher data set 22 held by the data storage device 2 from the data storage device 2. The clustering unit 12 calculates the distance between each teacher data included in the teacher data set 21 held by the data acquiring unit 11 and each non-teacher data included in the non-teacher data set 22. Then, the clustering unit 12 sets the minimum value of the distances between each teacher data and each non-teacher data as a threshold value that is a condition for stopping merging clusters (step S1).

[0043] Next, the clustering unit 12 sets each of the teacher data and each of the non-teacher data as an initial cluster for clustering (step S2).

[0044] Next, the clustering unit 12 calculates the distance between each cluster (step S3).

[0045] Next, the clustering unit 12 determines whether or not the minimum value of the distance between each cluster is less than a threshold value (step S4).

[0046] If the minimum value of the distance between the clusters is less than the threshold value (step S4: Yes), the clustering unit 12 merges the clusters whose inter-cluster distance is the minimum (step S5). After that, the clustering unit 12 returns to step S3.

[0047] On the other hand, if the minimum distance between the clusters is equal to or greater than the threshold (step S4: No), the clustering unit 12 extracts clusters including the teacher data from the generated clusters (step S6). After that, the clustering unit 12 outputs information about the extracted clusters to the model generation unit 13.

[0048] The model generation unit 13 acquires information on clusters including teacher data from the clustering unit 12. Then, the model generation unit 13 assigns each of the extracted clusters to either model generation or model verification (step S7).

[0049] Next, the model generation unit 13 performs learning using the teacher data included in the cluster assigned for model generation to generate a trained model. The verification unit 14 acquires information on the generated model and the cluster assigned for model verification from the model generation unit 13. Then, the verification unit 14 predicts the model verification data using the acquired model, using the teacher data included in the cluster assigned for verification as model verification data, and performs verification (step S8).

[0050] As described above, the machine learning device according to the present embodiment performs hierarchical clustering using teacher data including a data group with a high similarity at a probability other than chance level and non-teacher data with an unknown output and a different bias from the teacher data. The machine learning device performs clustering by merging clusters with the smallest distance using the shortest distance between the teacher data and the non-teacher data as a threshold. Then, the machine learning device divides the clusters including the teacher data into those for model creation and those for model verification, generates a model by learning using the teacher data belonging to the cluster for model creation, and applies the teacher data belonging to the cluster for model verification to the generated model to verify it.

[0051] By performing clustering by the above-mentioned method, it is possible to reduce the number of clusters in which teacher data and non-teacher data are not mixed. By dividing teacher data together with non-teacher data with different biases, teacher data can be divided into a size that does not mix data with different biases from a highly similar data group that exists at a probability other than the chance level, and appropriate division of teacher data can be realized. Furthermore, by using appropriately divided clusters for model creation and model verification, it is possible to avoid similar data being used for both model creation and model verification, and appropriate verification can be performed with overlearning and the like suppressed. Therefore, the machine learning device according to the present embodiment can improve the prediction accuracy of a model created by machine learning.

[0052] In addition, if the similarity standard is set too high in grouping, the model creation data and the model verification data will diverge significantly. Since machine learning is based on the premise that the divergence between the model creation data and the model verification data is small, it is difficult to improve the prediction accuracy of the model verification data by the model created using the model creation data. As a result, even if machine learning is performed appropriately, there is a risk that overlearning will be determined due to the low prediction accuracy of the model verification data, making it difficult to create an appropriate model. In contrast, the machine learning device according to the present embodiment can prevent the similarity standard from becoming excessive, and can generate model creation data and model verification data that are not diverged, making it possible to perform appropriate verification that suppresses overlearning and the like.

[0053] (Hardware configuration) 6 is a hardware configuration diagram of a machine learning device 1. The machine learning device 1 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94, as shown in FIG.

[0054] The network interface 94 is a communication interface between the machine learning device 1 and an external device. The network interface 94 relays communication between the CPU 91 and the data storage device 2 and user terminal device 3 illustrated in FIG.

[0055] The hard disk 93 is an auxiliary storage device. For example, when the machine learning device 1 has a teacher data set 21 and a non-teacher data set 22 in advance, the data may be held in the hard disk 93. The hard disk 93 also stores various programs including a program for implementing the functions of the control unit 10 including the data acquisition unit 11, the clustering unit 12, the model generation unit 13, the verification unit 14, and the prediction unit 15 illustrated in FIG.

[0056] The memory 92 is a main storage device. For example, a dynamic random access memory (DRAM) can be used as the memory 92.

[0057] The CPU 91 reads out various programs from the hard disk 93, and loads and executes them in the memory 92. In this way, the CPU 91 can realize the functions of the control unit 10 illustrated in FIG. [Explanation of symbols]

[0058] 1 Machine learning device 2 Data storage device 3. User terminal equipment 10 Control section 11 Data Acquisition Section 12 Clustering Department 13 Model Generation Unit 14 Verification Department 15 Prediction Department 21 Training Data Set 22 Unsupervised Dataset

Claims

1. Obtaining a first plurality of data labeled with a correct answer and a second plurality of data different from the first plurality of data; generating a plurality of groups into which any of the first plurality of data is classified by clustering the first plurality of data and the second plurality of data together under a condition that the first plurality of data and the second plurality of data are not classified into the same group; generating a machine learning model using data included in a first group of the plurality of groups as training data; Verification of the generated machine learning model is performed using data included in a second group other than the first group among the plurality of groups as verification data. A machine learning program that causes a computer to execute processing.

2. 2. The machine learning program according to claim 1, wherein the second plurality of data is data that is not labeled with a correct answer, unlike the first plurality of data.

3. 3. The machine learning program according to claim 1, wherein the conditions include a condition that the number of groups generated by the clustering is a minimum.

4. The machine learning program according to any one of claims 1 to 3, characterized in that the clustering process is performed based on a distance between each data included in the first plurality of data and the second plurality of data.

5. 5. The machine learning program according to claim 4, characterized in that the clustering process includes performing clustering such that a distance between two of the groups generated by clustering is set as a maximum distance between the data of one group and the data of the other group, and the shortest distance between the two groups is equal to or greater than a shortest distance between the first plurality of data and the second plurality of data and is closest to the first plurality of data.

6. Obtaining a first plurality of data labeled with a correct answer and a second plurality of data different from the first plurality of data; generating a plurality of groups into which any of the first plurality of data is classified by clustering the first plurality of data and the second plurality of data together under a condition that the first plurality of data and the second plurality of data are not classified into the same group; generating a machine learning model using data included in a first group of the plurality of groups as training data; Verification of the generated machine learning model is performed using data included in a second group other than the first group among the plurality of groups as verification data. A machine learning method characterized in that processing is executed by a computer.

7. Obtaining a first plurality of data labeled with a correct answer and a second plurality of data different from the first plurality of data; generating a plurality of groups into which any of the first plurality of data is classified by clustering the first plurality of data and the second plurality of data together under a condition that the first plurality of data and the second plurality of data are not classified into the same group; generating a machine learning model using data included in a first group of the plurality of groups as training data; Verification of the generated machine learning model is performed using data included in a second group other than the first group among the plurality of groups as verification data. A machine learning device comprising a control unit that executes processing.

Images