Data analysis device

Abstract

The following series of processes are repeated multiple times: the provided labeled teacher data is divided into data for model construction and data for model verification, the model construction data is used to construct a machine learning model, and the model is applied to the model verification data to identify samples (assign labels) (S2~S5). Although the constructed machine learning model changes when the model construction data changes, it should be able to accurately identify with a high probability. Therefore, in the wrong label sample, the original label is inconsistent with the recognition result, resulting in a high possibility of wrong recognition. When the number of wrong recognitions is calculated for each sample to find the error recognition rate, the error recognition rate becomes relatively high in the wrong label sample, so the wrong label sample is determined based on the error recognition rate (S6~S7). In this way, by detecting samples with a high probability of being in the wrong label state contained in the teacher data with high precision, the recognition performance of the machine learning model can be improved.

Description

Technical Field

[0001] The present invention relates to a data analysis device that analyzes data collected by various methods such as data obtained by various analysis devices such as a mass spectrometer, a gas chromatograph (GC), a liquid chromatograph (LC), and a spectrophotometer, and more specifically, to a data analysis device that uses supervised learning as a method of machine learning to identify unlabeled data and assign labels, or predict labels. In addition, in general, multivariate analysis is sometimes not included in the term "machine learning", but in this specification, it is assumed that machine learning includes multivariate analysis. Background Art

[0002] In order to find regularity from a large amount of diverse data and use this regularity to predict or identify data, machine learning is a useful method, and its application field has gradually expanded in recent years. As representative methods of machine learning, support vector machine (SVM=Support Vector Machine), neural network (Neural Network), random forest (Random Forest), adaptive boosting (AdaBoost), deep learning (Deep Learning), etc. are widely known. In addition, as representative methods of multivariate analysis included in broad machine learning, principal component analysis (PCA=Principal Component Analysis), independent component analysis (ICA=Independent Component Analysis), partial least squares (PLS=Partial Least Squares), etc. are widely known (see Patent Document 1, etc.).

[0003] Machine learning is roughly divided into supervised learning and unsupervised learning. For example, in the case of identifying whether a subject has a specific disease based on data collected by an analysis device, as long as a large amount of data can be collected in advance for patients with the disease and normal people who do not have the disease, supervised learning using these data as teacher data can be performed. Recently, in particular, the following attempts have been made in various places: applying supervised learning to mass spectrometry data obtained by mass spectrometry analysis devices to diagnose diseases such as cancer.

[0004] Fig.12 This is an example of a peak matrix that organizes mass spectrometry data on cancer samples and non-cancer samples as teacher data.

[0005] The peak matrix is ​​set as samples in the vertical direction and peak positions (mass-to-charge ratio m / z) in the horizontal direction, and the signal intensity value of each peak is set as the value of the element. Therefore, each element in 1 row of the peak matrix represents the signal intensity value of the peak in each mass-to-charge ratio of a sample, and each element in 1 column represents the signal intensity value of all samples at a certain mass-to-charge ratio. Here, samples 1 to sample n-2 are cancer samples, and each of these samples is assigned a label with a value of "1" indicating that it is cancer. On the other hand, samples n-1 to sample N are non-cancerous samples, and each of these samples is assigned a label with a value of "0" indicating that it is non-cancerous. In this case, the label is a binary label.

[0006] By using such labeled teacher data, it is possible to build a machine learning model that can identify cancer and non-cancer with high accuracy. However, depending on the situation, the teacher data itself is sometimes mislabeled. After all, the determination of cancer and non-cancer (or other diseases and non-disease) is based on the diagnosis of the pathologist. Since it is a human judgment, it is actually impossible to make the error zero. In addition, even if the pathologist's diagnosis is correct, it is also necessary to consider the labeling error caused by the operator's input error when inputting the pathologist's diagnosis as teacher data. Therefore, it is inevitable that a small number of samples with incorrect labels will be mixed in with the large number of samples provided as teacher data.

[0007] As a method to deal with such a situation, the machine learning algorithm is set to obtain high recognition performance even if some samples with incorrect label states are mixed in the teacher data. However, when the tolerance for teacher data with incorrect label states is increased, the reduction in recognition performance is inevitable, and a general machine learning method that allows them to coexist has not yet been realized.

[0008] In addition, other coping methods in the case of samples mixed with wrong label states are: discover and remove samples in wrong label states before building a machine learning model, or replace labels correctly. Although as described in non-patent literature 1, a method for detecting errors in assigned labels by machine learning is proposed, there is no statistical method with high reliability for judging whether samples provided as teacher data are wrong labels in the past. Therefore, the actual situation is that, as to whether wrong labels are included in the data, for example, in medical data, there is only an original method of checking whether the measurement date or the pathologist's diagnosis results are consistent with the labels assigned to the teacher data one by one. Such a method is very labor-intensive and inefficient. In addition, in the case where the pathologist's diagnosis itself is wrong, it is almost impossible to determine whether the sample is really a wrong label by this method.

[0009] Prior art literature

[0010] Patent Literature

[0011] Patent Document 1: Japanese Patent Application Publication No. 2017-32470

[0012] Non-patent literature

[0013] Non-patent document 1: Itabashi et al., "Research on semi-supervised learning by detecting mislabeled data", Proceedings of the National Conference of the Information Processing Society, Vol. 72, No. 2, March 8, 2010, pp. 463-464 Summary of the invention

[0014] Problem that the invention aims to solve

[0015] The present invention is completed to solve the above-mentioned problems, and its purpose is to provide a data analysis device that can build a machine learning model with high recognition performance by accurately determining and removing samples with a high possibility of incorrect label status or replacing labels from a large amount of data provided as teacher data.

[0016] Solutions for solving problems

[0017] The present invention completed in order to solve the above-mentioned problem is a data analysis device, which constructs a machine learning model based on teacher data with labels about multiple samples, uses the machine learning model to identify unknown samples and assign labels, and is characterized in that:

[0018] The method comprises: comprising: a wrong label detection unit for detecting samples of wrong label states in the teacher data; and

[0019] a) a repeated recognition execution unit, which repeats the following series of processes multiple times: constructing a machine learning model using model construction data selected from the teacher data or different from the teacher data as labeled data, applying the constructed machine learning model to model verification data selected from the teacher data to identify samples and assign labels; and

[0020] b) An error label determination unit, which, when a series of processes are repeatedly performed by the repeated recognition execution unit, calculates, for each sample, the number of false recognitions in which the label as the recognition result of the series of processes is inconsistent with the label originally assigned to the data, and determines whether the sample is in an error label state based on the number of false recognitions or the probability of the false recognitions.

[0021] In the data analysis device involved in the present invention, machine learning includes multivariate analysis that performs so-called supervised learning. In addition, in the data analysis device involved in the present invention, the content or type of data used as the analysis object is not limited, but typically, it can be set as analysis data and measurement data collected by various analysis devices. Specifically, it can be set as mass spectrum data obtained by a mass spectrometer, chromatogram data obtained by GC or LC, absorption spectrum data obtained by a spectrophotometer, data obtained by DNA microarray analysis, etc. Of course, data collected by various methods other than these can also be set as objects.

[0022] In the data analysis device involved in the present invention, a machine learning model is constructed based on the provided teacher data with labels for multiple (usually very many) samples, but before that, the wrong label detection unit detects samples in the wrong label state with wrong labels in the provided teacher data. That is, the repeated recognition execution unit selects the model construction data and the model verification data appropriately from the provided teacher data, and uses the former data to construct a temporary machine learning model. Then, by applying the latter data to the temporary machine learning model, the samples selected as the model verification data are respectively identified and labeled. In addition, the model construction data does not need to be the data included in the provided teacher data (that is, the data as the object of determination whether it is in the wrong label state), but can also be completely different labeled data. In addition, the model construction data and the model verification data can also partially overlap or be completely the same. Therefore, all the provided teacher data can also be used as model construction data and model verification data.

[0023] Now, for example, when a sample that is actually cancer but is given a non-cancer label (that is, a sample in a wrong label state) is identified by a certain machine learning model, in most cases, the sample should be identified as cancer. However, since the label attached to the sample is a non-cancer label, it means that the label as the recognition result is inconsistent with the original label, which can be said to be a wrong recognition. On the other hand, when a sample that is given a correct label is identified by the same machine learning model, in most cases, the label as the recognition result is consistent with the original label, and it becomes a correct recognition. In the case of only one machine learning model, even if the label of a certain sample is inconsistent with the label as the recognition result and it is judged to be a wrong recognition, it is essentially impossible to accurately judge whether the original label is correct and the recognition is wrong, or conversely, the recognition itself is correct but the original label is wrong. However, in terms of probability, the possibility of becoming a wrong recognition in the case of a wrong label state is high, so if different multiple machine learning models are used to try to identify the same sample and calculate the number of wrong recognitions, the number of wrong recognitions for the sample in the wrong label state should be increased, and on the other hand, the number of wrong recognitions for the sample with the correct label should be reduced.

[0024] Therefore, the repeated recognition execution unit repeats the above series of processes multiple times for, for example, different model construction data. Even if the machine learning method itself is the same, the machine learning model changes when the model construction data changes, so it becomes a case of using different multiple machine learning models to repeat the recognition. The error label determination unit calculates the number of wrong recognitions when such a series of processes are repeated multiple times for each sample. In other words, the number of wrong recognitions for the same sample is calculated. As described above, for samples in the wrong label state, the number of wrong recognitions becomes relatively large, so the error label determination unit determines whether it is an error label state for each sample based on the calculated number of wrong recognitions or the error recognition rate calculated based on the number of wrong recognitions. It is necessary to determine for each sample whether the number of wrong recognitions is relatively large or relatively small, or whether the error recognition rate is relatively high or relatively low, so of course it is necessary to increase the number of repetitions of the above series of processes to a level sufficient to make the determination.

[0025] As described above, in the data analysis device involved in the present invention, the wrong label detection unit can detect samples with a high probability of label errors in a large amount of teacher data derived from cancer samples. Therefore, by excluding such detected samples from the teacher data to improve the quality of the teacher data, the recognition performance of the machine learning model constructed using the teacher data can be improved. In addition, in the case of binary labels such as cancer and non-cancer, the labels are easy to replace, so it is also possible not to exclude samples with a high probability of being determined to be in an erroneous label state but to replace the labels and keep them as teacher data.

[0026] In the data analysis device involved in the present invention, it is preferably configured as follows: the error label detection unit uses the teacher data after removing the samples determined to be in an error label state by the error label determination unit from the teacher data, and implements the processing performed by the repeated recognition execution unit and the error label determination unit more than once.

[0027] When samples in the wrong label state are removed from the teacher data, the recognition performance of the machine learning model constructed using the removed teacher data is improved. Therefore, according to this structure, even for data that is difficult to determine whether it is in the wrong label state, it can be determined with high reliability, resulting in the improvement of the accuracy of wrong label detection.

[0028] In addition, in the data analysis device involved in the present invention, the data used for model construction does not necessarily need to be the teacher data that is the object of judgment as to whether it is an erroneous label state as described above, but in actual application, it is preferred to select the data used for model construction from the teacher data.

[0029] Therefore, as one aspect of the data analysis device involved in the present invention, it can be set as the following structure:

[0030] The error label detection unit includes a data segmentation unit, which segments the teacher data into data for model construction and data for model verification.

[0031] The duplication recognition execution unit changes the data division performed by the data division unit each time the series of processes are executed.

[0032] In this case, specifically, it is preferable that the data segmentation unit randomly segments the teacher data into the model building data and the model verification data using, for example, a random number table. In addition, in this case, even if the model building data and the model verification data are segmented again, it is possible that each data is identical to the data before the change or to the data that has been identified with a very low probability, but this influence will hardly occur as long as the number of repetitions is large.

[0033] In addition, in the data analysis device involved in the present invention, the repeated identification execution unit can be configured to use only one machine learning method or to use two or more machine learning methods. Of course, when using more than two machine learning methods, the structure of the device (essentially the program for calculation and processing) becomes correspondingly complicated, but the accuracy of error label detection can be improved by appropriately combining different methods. On the other hand, even if there is only one machine learning method, the accuracy of error label detection can be improved by increasing the number of repetitions.

[0034] In addition, in the data analysis device involved in the present invention, the machine learning method used in the repeated recognition execution unit is not particularly limited as long as supervised learning is performed, and it is preferably set to random forest, support vector machine, neural network, linear discriminant method, nonlinear discriminant method, etc. It is preferred that what method to use is appropriately selected according to the type and nature of the data to be analyzed. For example, according to the research of the inventors, it can be confirmed that when identifying whether a subject is cancerous or non-cancerous based on mass spectrometry data obtained by mass spectrometry analysis, the detection accuracy of wrong labels is relatively high when random forest is used.

[0035] In the data analysis device according to the present invention, the error label determination unit may determine the error label state based on various criteria. In one embodiment, the error label determination unit may determine that the sample with the highest error recognition rate is in the error label state.

[0036] In this case, the sample with the highest possibility of being in an erroneous label state is determined to be in an erroneous label state, so it is best to remove multiple samples with a high possibility of being in an erroneous label state by removing the samples determined to be in an erroneous label state one by one as described above, and repeating the processing performed by the repeated recognition execution unit and the erroneous label determination unit.

[0037] In addition, as another aspect, the wrong label determination unit may be configured to determine the number of samples specified by the user as being in the wrong label state in descending order of the wrong recognition rate.

[0038] In this configuration, a plurality of samples with a high probability of being in an erroneous label state can be removed at once, thereby shortening the processing time.

[0039] Furthermore, as yet another aspect, the wrong label determination unit may be configured to determine that a sample with a wrong recognition rate of 100% is in a wrong label state.

[0040] In this structure, a plurality of samples having a high probability of being in an erroneous label state can be removed with high reliability.

[0041] Furthermore, as still another aspect, the error label determination unit may be configured to determine that a sample having an error recognition rate equal to or higher than a threshold value set by a user is in an error label state.

[0042] In addition, in the data analysis device involved in the present invention, when the processing performed by the repeated recognition execution unit and the error label determination unit is repeatedly implemented as described above, the error label detection unit is preferably configured to repeatedly implement the processing performed by the repeated recognition execution unit and the error label determination unit until the error recognition rate becomes below a specified threshold.

[0043] According to this structure, it is possible to more accurately detect samples that may have an erroneous label state. However, depending on the situation, the number of repetitions may become too many, so it is better to set a limit on the number of repetitions or set a limit on the execution time, and end the processing when it conflicts with the limit even if the error recognition rate does not become less than a specified threshold.

[0044] In addition, the data analysis device according to the present invention is preferably configured to further include a result display processing unit that creates a table or a graph based on the recognition result of the wrong label determination unit and displays the table or the graph on a display unit.

[0045] Specifically, for example, by using a curve graph to represent the distribution of the number of misrecognitions or misrecognition rates for each sample in the entire teacher data, the user can easily determine the number of misrecognitions or misrecognition rates that are considered as the criteria for judging samples in the mislabeled state.

[0046] Effects of the Invention

[0047] According to the data analysis device involved in the present invention, it is possible to automatically determine whether the label of the provided teacher data is wrong, and determine the samples with a high probability of wrong label status. Thus, for example, by excluding such samples from the teacher data or replacing the labels, the quality of the teacher data can be improved, and a machine learning model with higher recognition performance than before can be constructed to more accurately identify unknown samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a functional block diagram of a cancer / non-cancer identification device as one embodiment of the data analysis device according to the present invention.

[0049] Figure 2 This is a flowchart of the incorrect label detection process in the cancer / non-cancer identification device of this embodiment.

[0050] Figure 3 This is a flowchart of a modified example of the erroneous label detection process in the cancer / non-cancer identification device of this embodiment.

[0051] Figure 4 FIG. 1 is a schematic diagram of the division process of the teacher data in the cancer / non-cancer classification device of the present embodiment.

[0052] Figure 5 This is an explanatory diagram of data used in a simulation for verifying the mislabel detection capability in the cancer / non-cancer identification device of this embodiment.

[0053] Figure 6 : is a graph showing the relationship between the signal intensity of two marker peaks in the XOR state and the cancer or non-cancer state.

[0054] Figure 7 is a diagram showing the result of false label detection in the case of using linear data as simulation data.

[0055] Figure 8 is a diagram showing the result of false label detection in the case of using linear data as simulation data.

[0056] Fig. 9 is a diagram showing the result of false label detection when nonlinear data is used as simulation data.

[0057] Fig.10 is a diagram showing the result of false label detection when nonlinear data is used as simulation data.

[0058] Fig.11 2 is a diagram showing a display example of an error label detection result.

[0059] Fig.12 This is a diagram showing an example of a peak matrix in which mass spectrum data on cancer specimens and non-cancer specimens are organized as teacher data. DETAILED DESCRIPTION

[0060] Hereinafter, a cancer / non-cancer distinguishing device as one embodiment of the data analyzing device according to the present invention will be described with reference to the drawings.

[0061] Figure 1 1 is a functional block diagram of the cancer / non-cancer identification device of this embodiment.

[0062] The cancer / non-cancer identification device is a device that determines whether mass spectrum data obtained by mass spectrometry analysis of a biological sample from a subject by a mass spectrometry analysis device (not shown) is cancer or non-cancer when unknown sample data is input. The cancer / non-cancer identification device includes a data analysis unit 1 and an operation unit 2 and a display unit 3 as a user interface.

[0063] The data analysis unit 1 includes an error label detection unit 10, an error label sample exclusion unit 17, a machine learning model preparation unit 18, and an unknown data identification unit 19 as functional blocks. In addition, the error label detection unit 10 includes a data segmentation unit 11, a machine learning model construction unit 12, a machine learning model application unit 13, an error recognition count unit 14, an error label sample determination unit 15, and a detection control unit 16 as functional blocks.

[0064] The functional blocks included in the data analysis unit 1 can also be constructed by hardware, but in actual use, it is best to set it to the following structure: a personal computer or a higher-performance workstation is used as a hardware resource, and dedicated software installed on the computer is executed on the computer, thereby concretizing the above-mentioned functional blocks.

[0065] Will come from Fig.12 The mass spectrum data of a large number of samples with cancer or non-cancer labels as shown (data indicating the peak signal intensity for each mass-to-charge ratio of the existing peak) are provided in advance to the data analysis unit 1 as labeled teacher data. The wrong label detection unit 10 detects samples with a high probability of wrong label status in the provided teacher data. The wrong label sample exclusion unit 17 excludes the samples detected by the wrong label detection unit 10 from the teacher data, or replaces the labels assigned to the detected samples. Here, the labels are binary values ​​of cancer: 1, non-cancer: 0, so the replacement of the labels only requires simply changing the values ​​to 1→0, 0→1.

[0066] The machine learning model making unit 18 uses the teacher data after a part of the samples are excluded or the labels are replaced by the wrong label sample exclusion unit 17 to construct a machine learning model. The machine learning method used here can also be the same as the machine learning method used in the wrong label detection unit 10 described later, but it is not necessarily the same. The unknown data identification unit 19 uses the machine learning model constructed by the machine learning model making unit 18 to determine the mass spectrometry data from the unknown sample and assign a label of cancer or non-cancer to the unknown sample. Such a recognition result is output from the display unit 3.

[0067] It is important to reduce as many samples as possible that are mislabeled and may be mixed into the teacher data in order to construct a machine learning model with high recognition performance by the machine learning model creation unit 18. Therefore, in the wrong label detection unit 10 in the cancer / non-cancer identification device of this embodiment, samples with a high possibility of being mislabeled are detected with high accuracy through characteristic processing as described below. Figure 2 is a flowchart of the error label detection process in the cancer / non-cancer identification device of this embodiment, Figure 4 It is a schematic diagram of the segmentation process of labeled teacher data.

[0068] Under the control of the detection control unit 16, the data segmentation unit 11 reads Fig.12 The labeled teacher data as shown in step S1 are the mass spectrometry data of N samples named sample 1, sample 2, ..., sample N-1, sample N, and each sample is given a binary label of cancer: "1" and non-cancer: "0". In general, the larger the number of N, the better, but the required number also varies depending on the nature of the data, so it is desirable to confirm it in advance.

[0069] The data dividing unit 11 divides the teacher data derived from the read-in large amount of samples into model construction data for constructing a machine learning model and model verification data for applying the constructed machine learning model (step S2 ).

[0070] Here, a random number table is used to divide the data obtained from a total of N samples into M data sets, and M-1 of the data sets are used as data for model construction, and the remaining data set is used as data for model verification. In this way, the provided teacher data is divided into data for model construction and data for model verification (refer to Figure 4 ). In addition, M is set to 5 in the simulation verification described later.

[0071] A random number table is used in data segmentation, so the combination of data contained in the data set may be the same when the segmentation is performed again, but the probability is extremely low. In fact, in most cases, the combination of data contained in the data set changes when the segmentation is performed again.

[0072] Next, the machine learning model construction unit 12 uses the model construction data obtained in the above step S2, that is, as teacher data, to construct a machine learning model based on a prescribed method (step S3). The machine learning method used here is not limited as long as it is supervised learning. For example, it can be set to random forest, support vector machine, neural network, linear discriminant method, nonlinear discriminant method, etc.

[0073] The machine learning model application unit 13 applies the model verification data obtained in the above step S2 to the machine learning model constructed in the above step S3 to identify whether each sample is cancerous or non-cancerous and assign a label (step S4). The label assigned to each sample is stored in an internal memory, for example, in correspondence with the sample name. Then, the detection control unit 16 determines whether the series of processes of steps S2 to S4 are repeated a specified number of times P (step S5), and returns to step S2 if the number of repetitions does not reach the specified number of times P.

[0074] When returning to step S2, the data segmentation unit 11 again segments the teacher data derived from a large number of samples into data for model construction and data for model verification. At this time, it is highly likely that the data for model construction and the data for model verification are different combinations from the first time. Even if the machine learning method is set to be the same, when the data for model construction is different, the machine learning model constructed based on the data will of course be different. Therefore, when a machine learning model different from the last time is applied to the model verification data, even if the same samples as the last time are present in the model verification data, the recognition results may be different. In this way, while changing the segmentation of the teacher data, the processing of steps S2 to S5 is repeated a specified number of times P.

[0075] As mentioned above and Figure 4As shown, the combination of samples included in the model verification data usually changes each time the above repetition is performed. However, if P is set to a certain degree of largeness, the same sample will be included in the model verification data multiple times, and each time it is labeled by the processing of step S4. Therefore, after the number of repetitions of the above series of processes reaches the specified number P (step S5: "Yes"), the number of wrong recognition counts 14, for each sample, calculates the number of times the originally assigned label is inconsistent with the label as the recognition result, that is, the number of wrong recognitions (step S6). The number of wrong recognitions is calculated for each sample included in the teacher data read in step S1.

[0076] In the recognition based on the machine learning model, although there is the possibility that it is really cancer but is judged as non-cancer, or vice versa, it is really non-cancer but is judged as cancer, but the probability is low. In other words, when the originally assigned label is inconsistent with the label as the recognition result, that is, in the case of wrong recognition, it can be said that the possibility of the originally assigned label being wrong (wrong label state) is higher than that the recognition itself based on the machine learning model has an error. Of course, it is difficult to judge in this way based on the recognition result of only one time, and it is more appropriate to consider that the originally assigned label is wrong if the number of wrong recognitions increases when the recognition is repeated while changing the machine learning model. Therefore, the wrong label sample determination unit 15 determines the samples with a high probability of wrong label state based on the number of wrong recognitions calculated for each sample (step S7).

[0077] However, the number of times recognition is performed is different for each sample, so it is not necessarily appropriate to compare the number of false recognitions as an absolute value. Therefore, it is best to calculate the false recognition rate based on the number of times recognition is performed and the number of false recognitions for each sample, and determine the sample with a high possibility of a false label state based on the false recognition rate.

[0078] When determining whether or not a state is an erroneous label state based on the false recognition rate, any one of the following determination criteria may be used.

[0079] (1) One sample with the highest misrecognition rate is determined to be in the mislabel state. However, if there are multiple samples with the highest misrecognition rate, all of the multiple samples may be determined to be in the mislabel state.

[0080] (2) The user specifies the number of samples to be determined as being in the wrong label state in advance from the operation unit 2 as a parameter, and the specified number of samples are determined as being in the wrong label state in descending order of the false recognition rate.

[0081] (3) Only samples with a false recognition rate of 100% are judged to be in a false label state. If there are multiple samples with a false recognition rate of 100%, all of the multiple samples may be judged to be in a false label state.

[0082] (4) The user specifies a threshold value of the error recognition rate to be determined as an error label state in advance from the operation unit 2 as a parameter, and samples with an error recognition rate greater than the threshold value are determined to be in an error label state.

[0083] Of course, the above (1) to (4) can be combined appropriately. For example, (1) and (4) can be combined to determine that the sample with the highest error recognition rate above a certain threshold is in the wrong label state. Of course, there may not be any samples in the wrong label state in the provided teacher data. Therefore, basically, it is appropriate that the sample with a low error recognition rate is not in the wrong label state, and conversely, it is appropriate that the sample with an extremely high error recognition rate is in the wrong label state.

[0084] Once the sample of the wrong label state is determined in this way, the wrong label detection result or the wrong recognition detection result can be arranged in a table form or a graph form and displayed on the display unit 3 to be presented to the user (step S8).

[0085] In addition, as described above, the incorrectly labeled sample excluding unit 17 may exclude samples that are likely to be incorrectly labeled as described above from the teacher data or replace the labels to generate teacher data for constructing a machine learning model for actual recognition.

[0086] In addition, generally, in the statistical processing as described above, a method called cross-validation is used to reduce the statistical error. In a strict sense of cross-validation, while changing the data set selected as the model validation data, the following process is performed M times to calculate, for example, the average value of the error recognition rate, wherein the process uses M-1 data sets among the M data sets divided as model building data to construct a machine learning model, and the remaining data set is applied to the machine learning model as model validation data for recognition. In contrast, in the process of the above embodiment, only one process is performed on the data set obtained by dividing in step S2, so it is different from a strict sense of cross-validation. However, by replacing the samples contained in the data set and repeating the process of steps S2 to S5 multiple times, it is possible to obtain substantially the same effect as cross-validation.

[0087] In use Figure 2In the error label detection process described below, after a series of processes of steps S2 to S4 are repeated a predetermined number of times P, samples with a high probability of an error label state are detected at once. However, it is also possible to Figure 3 The flowchart of the error label detection process is modified as shown. Figure 3 The processing of steps S11 to S15 in Figure 2 The processing of steps S1 to S5 in is exactly the same.

[0088] In this example, after step S15 is judged as "yes", one or more samples with the highest error recognition rate for each sample are removed from the teacher data as samples in the wrong label state (step S16). After the quality of the teacher data is improved in this way, return to step S12 and perform the processing of steps S12 to S16 again. In this way, one or more samples with the highest error recognition rate for each sample are again removed from the teacher data as samples in the wrong label state. If the processing of steps S12 to S16 is repeated a specified number of times Q, or the highest error recognition rate is below a specified value, or the change in the error recognition rate converges within a specified range (step S17: "Yes"), the processing is terminated.

[0089] By removing samples with a high probability of being in an incorrectly labeled state in this way in stages, it is possible to avoid mistakenly removing samples that are not incorrectly labeled, and it is possible to more accurately remove only samples that are truly in an incorrectly labeled state, thereby further improving the quality of the teacher data.

[0090] [Evaluation of the Error Label Detection Process by Simulation]

[0091] Next, the results of evaluating whether the samples with the wrong label state are properly detected by the above-mentioned wrong label detection process through simulation are described. In the evaluation through simulation, as described above, the number of divisions M of the data set is set to 5, and the predetermined number of times P is set to 500. In addition, random forest is used as the machine learning method. In addition, as the data (teacher data) used in the evaluation, Figure 5 As shown, both linear data and nonlinear data are used.

[0092] [Method and results of simulation using linear data]

[0093] The linear data mentioned here refers to the data in which the signal intensity difference of all the marker peaks on the mass spectrum is sufficient between cancer and non-cancer. As long as the number of marker peaks is sufficient and the signal intensity difference between the peaks is sufficient, even if the principal component analysis or OPLS-DA (an improved version of PLS-DA (Partial Least Squares Discriminant Analysis) as a kind of discriminant analysis) and other multivariate analysis methods are used, it can be divided into two groups such as cancer and non-cancer. Therefore, here, the data of 10 marker peaks containing basically no signal intensity difference between cancer and non-cancer are used in the simulation. It has been confirmed that even if the principal component analysis is performed on the data, the data cannot be divided into two groups.

[0094] In addition, since the simulated data is known data, the labels are of course 100% correct. Therefore, 10 samples were randomly selected from each of the cancer and non-cancer samples, and the labels of these 20 samples in total were replaced to create artificially mislabeled samples. Then, it was verified whether these 20 samples could be identified as mislabeled samples.

[0095] In a random forest with decision trees as learners, the representative parameter that needs to be adjusted is the number of decision trees. The average correct answer rate in 5-fold cross validation when the number of decision trees is changed was investigated. In the range of 5 to 20 decision trees, the average correct answer rate is 99.6% regardless of the number of decision trees. Therefore, the number of decision trees is set to 10 to try to detect wrong labels.

[0096] exist Figure 7 and Figure 8 The detection results are shown in FIG. Figure 7 This is the result of the incorrect label detection of the sample that was assigned the non-cancer label. Figure 8 This is the result of the wrong label detection of the sample that has been labeled as cancer. Figure 7 and Figure 8 (and the following Fig. 9 and Fig.10 ), the number of times the model verification data is adopted is equivalent to the number of times the recognition processing of step S4 is executed.

[0097] As per Figure 7 and Figure 8As can be seen, the error recognition rate for wrongly labeled samples is 100%, and the error recognition rate for non-wrongly labeled samples is 0%, regardless of cancer or non-cancer. In other words, it can be said that the wrong label detection was completely successful. In addition, in this data, the correct answer rate for cancer / non-cancer judgment in data mixed with wrong labels is 99.6%, but by removing the wrongly labeled samples detected using the above method, the correct answer rate becomes 100%. In other words, it can be confirmed that by removing samples determined to be wrongly labeled samples from the teacher data, it is possible to build a machine learning model with extremely high recognition performance.

[0098] [Simulation method and results using nonlinear data]

[0099] The data collected generally has nonlinear characteristics, and there are fewer completely linear data. Therefore, the ability of the above-mentioned error label detection processing is also evaluated for nonlinear simulation data.

[0100] The nonlinear data mentioned here refers to data that can identify cancer / non-cancer by considering multiple peaks simultaneously, although cancer / non-cancer cannot be identified by a single peak on the mass spectrum. As typical data of such a state, data in which two marker peaks A and B are in an XOR (exclusive OR) state are produced. Figure 6 It is a diagram showing the relationship between the signal intensity of two labeled peaks in an XOR state and the state of cancer or non-cancer. That is, although the two labeled peaks A and B cannot identify cancer / non-cancer separately, if the signal intensity of peaks A and B are both above their respective thresholds Ath and Bth, it is cancer (area [c]), or if the signal intensity of peaks A and B are both less than their respective thresholds Ath and Bth, it is also cancer (area [b]). On the other hand, if the signal intensity of peak B is above threshold Bth and the signal intensity of peak A is less than threshold Ath, it is non-cancer (area [d]), and if the signal intensity of peak A is above threshold Ath and the signal intensity of peak B is less than threshold Bth, it is also non-cancer (area [a]). Therefore, for example, specimen α is cancer.

[0101] Similar to the linear data, 10 samples were artificially mislabeled for cancer and 10 samples were mislabeled for non-cancer (with the same sample numbers). In addition, the marker peaks with the same mass-to-charge ratio as the linear simulation data were selected, but the marker peaks were processed so that 2 of the 10 peaks were in an XOR state.

[0102] For such data, the average correct answer rate in 5-fold cross validation when the number of decision trees was changed was investigated. The average correct answer rate was 99.6% regardless of the number of decision trees in the range of 5 to 20. Therefore, the number of decision trees was set to 10 and an attempt was made to detect incorrect labels.

[0103] exist Fig. 9 and Fig.10 The detection results are shown in FIG. Fig. 9 This is the result of the incorrect label detection of the sample that was assigned the non-cancer label. Fig.10 This is a wrong label detection result for a sample that has been labeled as cancer.

[0104] As per Fig. 9 and Fig.10 As can be seen, the error recognition rate for wrongly labeled samples is 100%, and the error recognition rate for non-wrongly labeled samples is 0%, regardless of cancer or non-cancer. In other words, it can be said that the error label detection is completely successful in this case. In addition, the number of times the model verification data is used for each sample is exactly the same in linear data and nonlinear data, but this is because the random numbers in the random number table used for data segmentation are exactly the same, which will not have any impact on the evaluation results.

[0105] observe Figure 7 to Figure 10 It can be understood that for the wrongly labeled samples, the error recognition rate is 100%, and for the samples assigned the correct label, the error recognition rate is 0%. This is mainly due to the characteristics of the machine learning method (random forest) used in the simulation. In the case where the error recognition rate is so extremely different between the state of the wrong label and the state of the wrong label, it is easy to determine the wrongly labeled sample based on the error recognition rate. On the other hand, when other machine learning methods are used, the error recognition rate is not limited to this.

[0106] Fig.11 This is a diagram showing a schematic relationship between the sorting numbers assigned by sorting the sample numbers in descending order of the false recognition rate and the false recognition rate.

[0107] exist Fig.11 In the figure, the solid line is the error label detection result for the simulation data using the random forest, and the dotted line is an example of the error label detection result for the simulation data using the support vector machine. As shown in the figure, when the support vector machine is used, the error recognition rate sometimes gradually decreases. In addition, sometimes the highest error recognition rate does not reach 100%. Therefore, the user is allowed to specify the threshold value or the value used to determine whether the sample is in the error label state. Figure 3 The method shown to exclude samples with the highest false recognition rate one by one is useful.

[0108] Presenting to the user Fig.11The curve shown in FIG. 1 or the table containing the same information is effective for the user to select the judgment standard for determining whether it is an error label state, or to determine the parameters such as the threshold used for the judgment, or to determine whether the machine learning method used is appropriate. Therefore, in the cancer / non-cancer identification device of the above embodiment, after calculating the error recognition rate of each sample, a curve such as Fig.11 The graph shown or a table equivalent thereto is displayed on the screen of the display unit 3 .

[0109] In the cancer / non-cancer identification device of the above-mentioned embodiment, random forest is used as a machine learning method in the error label detection unit 10, but it is clear that various supervised learning methods that have been exemplified can be used, such as support vector machines, neural networks, linear discriminant methods, nonlinear discriminant methods, etc. It is also possible that which method is appropriate to use varies depending on the nature of the data to be analyzed, so a plurality of machine learning methods are prepared in advance and the user can select arbitrarily.

[0110] Alternatively, you can repeat Figure 2 When processing steps S2 to S5 in Figure 3 In the processing of steps S12 to S15 in , not one machine learning method is used but multiple machine learning methods are used. In addition, when using multiple different types of machine learning methods, of course, even if the model building data is the same, the machine learning models constructed under each machine learning method are different. Therefore, it is also possible that when using multiple different types of machine learning methods, or when machine learning based on other methods is performed after machine learning based on one method is implemented, the re-division of the teacher data is omitted, and the same model building data and model verification data as those previously implemented in the machine learning based on one of the above methods are used to perform machine learning based on the above other methods.

[0111] In addition, in the above-mentioned embodiment, the teacher data derived from the sample is divided into model construction data and model verification data, so the model construction data and the model verification data must be different data, but this is not necessary. For example, it is also possible to arbitrarily (for example, using a random number table) select the model construction data and the model verification data from a large amount of teacher data. Therefore, it is also possible that a part of the model construction data and a part of the model verification data are common. In addition, it is also possible that the model construction data is directly used for the model verification data, that is, the two are exactly the same.

[0112] In addition, the device of the above-mentioned embodiment uses the present invention for analyzing mass spectrum data obtained by a mass spectrometer, but it is clear that the present invention can be applied to all devices that use machine learning to perform certain recognition on various analytical data or measurement data other than this. For example, when it comes to the field of analysis devices similar to mass spectrometers, it is clear that the present invention can be used in devices that analyze chromatographic data obtained by LC devices or GC devices, absorbance spectrum data obtained by spectrophotometric measurement devices, etc. In addition, the present invention can also be used for analyzing data obtained by DNA microarray analysis (data obtained by digitizing images).

[0113] Furthermore, of course, the present invention can also be used in a data analysis device that performs machine learning not only based on data obtained through such machine analysis, but also based on data collected through various other methods to perform identification (labeling).

[0114] That is, the above-described embodiment is merely an example of the present invention, and even if appropriate changes, modifications, additions, etc. are made within the scope of the gist of the present invention in aspects other than those described above, they are naturally included in the claims of the present application.

[0115] Description of Reference Numerals

[0116] 1: Data analysis unit; 10: Error label detection unit; 11: Data segmentation unit; 12: Machine learning model construction unit; 13: Machine learning model application unit; 14: Error recognition count unit; 15: Error label sample determination unit; 16: Detection control unit; 17: Error label sample exclusion unit; 18: Machine learning model production unit; 19: Unknown data identification unit; 2: Operation unit; 3: Display unit.

Claims

1. A data analysis device that uses a machine learning model to identify unknown samples and assign labels, wherein the data analysis device is characterized in that: An error label detection unit is provided, which is configured to perform the following series of processes: Reading a plurality of teacher data to which tags are respectively assigned; A first data set, a second data set and a third data set that are different from each other are obtained from the plurality of teacher data, wherein: The data in the third data set are assigned a first label; Using the first data set to build a first machine learning model; Using the second data set to build a second machine learning model; Assigning a second label to the data in the third data set using the first machine learning model; Assigning a third label to the data in the third data set using the second machine learning model; Obtaining a determination result obtained by determining whether the first label of the data in the third data set is consistent with the second label and the third label; as well as It is determined whether the first label is incorrect based on the determination result.

2. The data analysis device according to claim 1, characterized in that: The wrong label detection unit performs the series of processes performed by the wrong label detection unit one or more times using the teaching data obtained by removing samples determined to be in the wrong label state from the plurality of teaching data.

3. The data analysis device according to claim 1, characterized in that: The wrong label detection unit includes a data segmentation unit, which segments the plurality of teacher data into the first data set, the second data set and the third data set. The error label detection unit changes the data division performed by the data division unit each time the series of processes are executed.

4. The data analysis device according to claim 1, characterized in that: The error label detection unit uses only one machine learning method.

5. The data analysis device according to claim 1, characterized in that: The error label detection unit uses two or more machine learning methods.

6. The data analysis device according to claim 1, characterized in that: The error label detection unit uses random forest as a machine learning method.

7. The data analysis device according to claim 1, characterized in that: The error label detection unit uses a support vector machine as a machine learning method.

8. The data analysis device according to claim 1, characterized in that: The error label detection unit uses a neural network as a machine learning method.

9. The data analysis device according to claim 1, characterized in that: The wrong label detection unit uses a linear discriminant method as a machine learning method.

10. The data analysis device according to claim 1, characterized in that: The error label detection unit uses a nonlinear discrimination method as a machine learning method.

11. The data analysis device according to claim 1, characterized in that: The error label detection unit determines that the sample with the highest error recognition rate is in an error label state.

12. The data analysis device according to claim 1, characterized in that: The error label detection unit determines the number of samples specified by the user as being in an error label state in descending order of error recognition rate.

13. The data analysis device according to claim 1, characterized in that: The error label detection unit determines that a sample with an error recognition rate of 100% is in an error label state.

14. The data analysis device according to claim 1, characterized in that: The error label detection unit determines that a sample whose error recognition rate is greater than or equal to a threshold value set by a user is in an error label state.

15. The data analysis device according to claim 2, characterized in that: The error label detection unit repeatedly performs the process performed by the error label detection unit until the error recognition rate becomes equal to or less than a predetermined threshold value.

16. The data analysis device according to claim 1, characterized in that: The system further includes a result display processing unit that creates a table or a graph based on the recognition result of the wrong label detection unit and displays the table or the graph on a display unit.

Images