Analyzing device, analyzing method and analyzing program

Abstract

To suppress a variability in interpretation between samples.SOLUTION: An analyzing device executes: an obtaining process of obtaining a plurality of sample data sets each including a first sample and a first label corresponding to the first sample; a similarity level calculating process of inputting, to each of a plurality of similarity level evaluation indices for evaluating the similarity level between two pieces of data, two first samples among the plurality of first samples in the sample data sets obtained through the obtaining process, so as to calculate the similarity level between the two first samples for each similarity level evaluation index; and a reliability level calculating process of tallying, for each first sample, the number of other first samples that have the same first label based on the similarity calculated through the similarity level calculating process, and of calculating, for each first sample, a first reliability level representing the lowliness of a variability in interpretation of the first sample based on the tallying result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an analysis device, an analysis method, and an analysis program for analyzing data. [Background technology]

[0002] There is a problem with using artificial intelligence (AI) to reproduce text classified by humans and make automatic judgments. Since human judgments are unstable, consistency cannot be guaranteed. Such fluctuations in judgments are called noisy oracles, and in order to reduce this noisy oracle, which changes depending on a person's physical condition, improvements can be made such as increasing the number of evaluators who make the judgments and reaching a consensus, or increasing the number of samples to be judged to obtain statistical stability, but this requires human and financial costs. Therefore, as a result, it is difficult to perform precise classification based on data that has been judged by humans.

[0003] For this reason, label propagation methods such as those described in the following Patent Documents 1 to 3 have been used in the past. The label propagation method is a method of relabeling labeled samples based on the idea that if samples are close, the labels should also be close. There is also a semi-supervised learning method using the label propagation method, which uses labeled samples to assign pseudo labels to unlabeled samples.

[0004] Patent Document 1 discloses a classification model learning device that builds a classification model with high accuracy even when poor quality teacher data is included. This classification model learning device acquires the coordinates corresponding to expert data whose labeling reliability meets a predetermined standard and non-expert data whose labeling reliability is unknown, calculates the distance from the non-expert data to the expert data, and defines the neighborhood distance by applying a predetermined rule. Next, expert data within the neighborhood distance from the selected non-expert data is searched for, and the same label probability is calculated. The reliability of the non-expert data is determined and added by applying a reliability function based on the probability that the attached label matches the label of expert data within the neighborhood distance. Then, a classification model that labels desired data is learned based on the expert data and the non-expert data to which the reliability is added.

[0005] Patent Document 2 discloses a model creation device that can reduce the risk of overlooking a noteworthy event. This model creation device determines whether a label related to the first data and a label related to the second data are similar based on the degree of similarity between observation information representing a situation in which the first data was observed and observation information representing a situation in which the second data was observed, and when it is determined that they are similar, assigns a label related to the second data as a label related to the first data, and calculates a model that represents the relationship between data information including the first data and the second data, and label information including the assigned label and the label related to the second data.

[0006] Patent Document 3 discloses a processing system that aims to reduce the time required to identify erroneous labels. This processing system includes a first acquisition unit, a second acquisition unit, a third acquisition unit, a discrimination unit, and an extraction unit. The first acquisition unit acquires a plurality of learning data to which labels have been assigned. The second acquisition unit acquires a trained model generated based on the plurality of learning data. The third acquisition unit acquires the discrimination data to which labels have been assigned. The discrimination unit discriminates the discrimination data using the trained model. The extraction unit extracts one or more learning data similar to the discrimination data from the plurality of learning data based on an index relating to the similarity between the discrimination data applied in the trained model and each of the plurality of learning data. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2009-282686 A [Patent Document 2] International Publication No. 2020 / 026395 [Patent Document 3] International Publication No. 2022 / 123905 Summary of the Invention [Problem to be solved by the invention]

[0008] One hypothesis for the cause of fluctuations in human judgment is that the labels fluctuate due to the intelligibility of the input samples. In other words, there is little point in correcting the labels based on the closeness of the samples, and an approach that captures the intelligibility of the samples themselves is desirable, but such an approach is not disclosed in the above Patent Documents 1 to 3.

[0009] The present invention aims to reduce the variation in interpretation between samples. [Means for solving the problem]

[0010] An analysis device according to one aspect of the invention disclosed in the present application is an analysis device having a processor that executes a program and a storage device that stores the program, wherein the processor executes an acquisition process to acquire a plurality of sample data sets having a first sample and a first label that pairs with the first sample; a similarity calculation process to calculate a similarity between the two first samples for each similarity evaluation index by inputting two first samples among the plurality of first samples in the sample data set acquired by the acquisition process into each of a plurality of similarity evaluation indexes that evaluate the similarity between two data; and a reliability calculation process to count, for each of the first samples, the number of other first samples having the same first label based on the similarity calculated by the similarity calculation process, and calculate, for each of the first samples, a first reliability indicating low fluctuation in interpretation of the first sample based on the counting result. Effect of the Invention

[0011] According to the representative embodiment of the present invention, it is possible to improve the reliability of the interpretation of samples with respect to fluctuations. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an analysis device. [Diagram 2] FIG. 2 is an explanatory diagram illustrating an example of the sample DB. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the analysis device. [Figure 4] FIG. 4 is a block diagram showing an example of a detailed functional configuration of the inspection unit. [Diagram 5] FIG. 5 is a diagram illustrating an example of a similarity calculation process using the similarity evaluation index fj according to the first embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a similarity calculation process procedure performed by the similarity calculation unit. [Figure 7] FIG. 7 is an explanatory diagram illustrating an example of the reliability calculation process performed by the reliability calculation unit. [Figure 8] FIG. 8 is a flowchart of an example of a reliability calculation process procedure performed by the reliability calculation unit according to the first embodiment. [Figure 9] FIG. 9 is an explanatory diagram illustrating an example of the learning process performed by the learning unit. [Figure 10] FIG. 10 is a flowchart illustrating an example of a learning process procedure performed by the learning unit. [Figure 11] FIG. 11 is an explanatory diagram of a first similarity calculation process example using a similarity evaluation index according to the second embodiment. [Figure 12] FIG. 12 is an explanatory diagram of a second similarity calculation process example using a similarity evaluation index according to the second embodiment. [Figure 13] FIG. 13 is an explanatory diagram of a similarity calculation process example 3 using a similarity evaluation index according to the second embodiment. [Figure 14] FIG. 14 is a flowchart of an example of a reliability calculation process procedure performed by the reliability calculation unit according to the second embodiment. [Figure 15] FIG. 15 is a diagram illustrating a learning example according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In the embodiment of the present invention, a reliability indicating low variation in interpretation between samples is calculated based on the similarity between samples obtained using various similarity evaluation indices such as a language model, a similarity judgment rule, and a function for calculating a distance measure, and the consistency of the labels paired with the samples, and learning is performed taking into account the variation in interpretation between samples. In other words, if the same label is assigned between close samples due to various interpretations of the samples by using various language models, it can be said that there is no variation in the interpretation between the samples, that is, the samples are likely to be ones for which it is easy to determine that the labels are correct. As a result, the amount of information for each sample increases through the language model, making it possible to obtain a better discrimination model. Hereinafter, the embodiment of the present invention will be specifically described with reference to the attached drawings. EXAMPLES

[0014] <Example of hardware configuration of analytical equipment> FIG. 1 is a block diagram showing an example of a hardware configuration of an analysis device. The analysis device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected by a bus 106. The processor 101 controls the analysis device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transient or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. Examples of the input device 103 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 may be, for example, a display, a printer, or a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0015] <Sample DB (database)> 2 is a diagram illustrating an example of a sample DB 200. The sample DB 200 includes a labeled sample data set group 201 and an unlabeled sample data set group 202.

[0016] The labeled sample dataset group 201 has N labeled sample datasets, which are combinations of labeled samples x1 to xN and labels y1 to yN. N is an integer equal to or greater than 1. i is an integer satisfying 1≦i≦N. When the labeled samples x1 to xN are not distinguished, they are simply referred to as labeled samples xi. When the labels y1 to yN are not distinguished, they are simply referred to as labels yi. The labeled sample dataset is represented as (xi, yi).

[0017] The unlabeled sample dataset group 202 has M sets of unlabeled samples z1 to zN. M is an integer equal to or greater than 1. m is an integer satisfying 1≦m≦M. When the unlabeled samples z1 to zN are not distinguished, they are simply referred to as unlabeled sample zm. A pseudo label w (w1 to wM) is assigned to the unlabeled sample z. The pseudo-labeled sample dataset is represented as (zm, wm). The assignment of the pseudo label w will be described later.

[0018] In addition, when there is no need to distinguish between labeled samples xi and unlabeled samples zm, we simply refer to them as samples x / z. Similarly, when there is no need to distinguish between labels yi and pseudo-labels wm, we simply refer to them as labels y / w.

[0019] <Example of Functional Configuration of Analysis Device 100> 3 is a block diagram showing an example of a functional configuration of analysis device 100. Analysis device 100 includes acquisition unit 301, inspection unit 302, output unit 303, learning unit 304, and prediction unit 305. Specifically, acquisition unit 301, inspection unit 302, output unit 303, learning unit 304, and prediction unit 305 are realized by, for example, causing processor 101 to execute a program stored in storage device 102 shown in FIG.

[0020] The acquiring unit 301 acquires data in the sample DB 200. Specifically, for example, the acquiring unit 301 acquires a labeled sample data set (xi, yi) from the sample DB 200, acquires an unlabeled sample zm, or acquires a pseudo-labeled sample data set (zm, wm).

[0021] The inspection unit 302 calculates the fluctuation of the sample x / z. Specifically, for example, the inspection unit 302 calculates the similarity between the samples x / z using various language models, and calculates the fluctuation of the sample x / z based on the consistency of the labels y / w paired with the sample x / z. In this way, the inspection unit 302 specifies the degree to which the interpretation of the sample x / z fluctuates. Details of the inspection unit 302 will be described later with reference to FIG. 4.

[0022] The output unit 303 outputs the inspection result 330 by the inspection unit 302. Specifically, for example, the output unit 303 displays the inspection result 330 on a display, transmits it to another computer, or prints it. The inspection result 330 is information that specifies the degree to which the interpretation of the sample x / z is unstable. The user can refer to the inspection result 330 to correct the label yi.

[0023] The learning unit 304 learns the discrimination model based on the inspection result 330 by the inspection unit 302. Specifically, for example, the learning unit 304 weights the difference between the discrimination result output from the discrimination model as a result of inputting the sample x / z to the discrimination model and the label y / w by the inspection result 330, and calculates a loss value. Then, the learning unit 304 updates the learning parameters of the discrimination model by backpropagating the error so as to reduce the loss value. Details of the learning unit 304 will be described later with reference to FIG. 9.

[0024] The prediction unit 305 inputs test data 351 to a discriminant model, and outputs a prediction result 352 from the discriminant model. The test data 351 may be sample x / z, or may be the same type of input data as sample x / z.

[0025] Furthermore, the learning unit 304 may use the discrimination result as the prediction result 352 to re-learn the discrimination model.

[0026] [Inspection Department 302] Fig. 4 is a block diagram showing a detailed functional configuration example of the inspection unit 302. In Fig. 4, the inspection unit 302 inspects the fluctuation of interpretation between labeled samples x i. The application of unlabeled samples z m will be described later. The acquisition unit 301 acquires a labeled sample data set (x i, y i ) from the sample DB 200. The inspection unit 302 includes a similarity calculation unit 401 and a reliability calculation unit 402.

[0027] [Similarity calculation unit 401] The similarity calculation unit 401 uses the similarity evaluation indexes f1 to fJ to calculate the similarity between the labeled samples xi for each of the similarity evaluation indexes f1 to fJ. J is an integer equal to or greater than 1. j is an integer that satisfies 1≦j≦J. When the similarity evaluation indexes f1 to fJ are not to be distinguished from one another, they are simply referred to as similarity evaluation indexes fj.

[0028] The similarity evaluation index fj is information for evaluating the similarity between labeled samples xi, and is, for example, a language model or a rule-based table. Specifically, for example, the similarity evaluation index fj has a language model that compares the similarity between samples xi using the average vector of Word2Vec. Since the similarity between samples xi is expressed, for example, by the Euclidean distance, the shorter the Euclidean distance, the more similar the two samples xi are (meaning the higher the similarity).

[0029] The similarity evaluation index fj also includes a language model trained with appropriate entailment or non-entailment judgment data. In this case, the similarity evaluation index fj is composed of a neural network and outputs the score of the activation function (e.g., softmax) of the output layer. The score indicates the entailment probability. If the entailment probability is equal to or greater than a predetermined value, both samples xi are similar (meaning that the similarity is high).

[0030] The similarity evaluation index fj may be a function that calculates the Levenshtein distance, which evaluates the difference in character strings between two samples xi. The shorter the Levenshtein distance, the more similar the two samples xi are (meaning a higher similarity).

[0031] The similarity evaluation index Fj described above is an example, and is not limited to this as long as it is an index capable of calculating the similarity between labeled samples xi.

[0032] The similarity calculation unit 401 outputs the similarity between the labeled samples xi output from each of the similarity evaluation indexes f1 to fJ as a similarity evaluation tensor Ta. The similarity evaluation tensor Ta is an N×N×J tensor including J tensors with N rows and N columns in the similarity evaluation index Fj.

[0033] 5 is an explanatory diagram showing an example of a similarity calculation process using a similarity evaluation index fj according to the first embodiment. A labeled sample x i and a labeled sample x i that is the same as or different from the labeled sample x i are input to the similarity evaluation index fj, and similarities Saj(1,1), ..., Saj(1,N), Saj(2,1), ..., Saj(2,N), ..., Saj(i,1), ..., Saj(i,N), ..., Saj(N,1), ..., Saj(N,N).

[0034] Since the similarity Saj(p,q) (where p and q are sample number i of labeled sample xi) and the similarity Saj(q,p) are the same similarity, the analysis device 100 treats the similarity Saj(q,p) as the similarity Saj(p,q). Since the similarity Saj(i,i) is the similarity between samples xi with the same label, it is the maximum value. When there is no need to distinguish between these, it will simply be written as the similarity Saj.

[0035] FIG. 6 is a flowchart showing an example of a procedure for the similarity calculation process performed by the similarity calculation unit 401.

[0036] (Step S601) The similarity calculation unit 401 judges whether there is a combination of unselected labeled samples xi. If there is no combination of unselected labeled samples xi (step S601: No), the similarity calculation unit 401 ends the similarity calculation process. If there is a combination of unselected labeled samples xi (step S601: Yes), the process proceeds to step S602.

[0037] (Step S602) The similarity calculation unit 401 selects a combination of unselected labeled samples xi, and the process proceeds to step S603.

[0038] (Step S603) The similarity calculation unit 401 calculates the similarity Saj between the samples xi in the combination of the labeled samples xi selected in step S602 for each similarity evaluation index fj, and returns to step S601.

[0039] This ends the loop of steps S601 to S603, and the similarity evaluation tensor Ta is output.

[0040] [Reliability calculation unit 402] Returning to FIG. 4, the reliability calculation unit 402 identifies the consistency of the label yi paired with the labeled sample xi based on the similarity evaluation tensor Ta, and calculates the reliability indicating the fluctuation of the labeled sample xi based on the consistency.

[0041] 7 is an explanatory diagram showing an example of the reliability calculation process by the reliability calculation unit 402. The reliability calculation unit 402 creates ranking tables RT1, ..., RTj, ..., RTJ for each similarity evaluation tensor Ta1, ..., Taj, ..., TaJ. The similarity evaluation tensor Taj is a tensor with N rows and N columns calculated by the similarity evaluation index fj.

[0042] The ranking table RTj is created based on the similarity evaluation tensor Taj. The ranking table RTj is ranked by sorting in descending order of the similarity Saj in the similarity evaluation tensor Taj.

[0043] The reliability calculation unit 402 creates aggregation tables CT1, ..., CTj, ..., CTJ for each ranking table RT1, ..., RTj, ..., RTJ. Specifically, for example, the reliability calculation unit 402 identifies labels yi having the top K similarities Saj in the ranking table RTj. The threshold value K of the rank is an integer equal to or greater than 1 and is set arbitrarily.

[0044] The reliability calculation unit 402 counts the number of same-label samples for the labels yi with the highest similarity Saj in the Kth place. j 1 ,…,k j i ,…,k j N Let the number of identically labeled samples be k. j i is the number of other labeled samples xp, xq in the similarities Sa(i,q), Sa(p,i) with the labeled sample xi when the similarity evaluation index fj is used.

[0045] For example, in the ranking table RTj, the similarity Saj of the labeled sample x1 is two, namely, similarity Saj(1,2) and similarity Saj(1,7). Therefore, the number of identically labeled samples with the label y1 paired with the labeled sample x1 is k j 1 Similarly, the similarity Saj of the labeled sample x2 is Saj(2,5) and Saj(1,2). Therefore, the number of identically labeled samples with the same label y2 that are paired with the labeled sample x2 is k j 2 would result in "2".

[0046] On the other hand, the similarity Saj of the labeled sample x5 is one similarity Saj (2,5). Therefore, the number of identically labeled samples with the same label y5 that are paired with the labeled sample x5 is k j5 The rank of the similarity Saj(5,9) is K+1, so the number of samples with the same label for label y5 is k j 5 is not counted in the.

[0047] In this way, the number of identically labeled samples is k j i is an index value indicating the consistency of a label yi paired with a labeled sample xi when the similarity evaluation index fj is used.

[0048] The reliability calculation unit 402 calculates the number of identically labeled samples k j i By dividing by the rank threshold K, the reliability R j i =k j i Calculate the reliability R j i is an index value that indicates the fluctuation of the interpretation of the labeled sample xi when the similarity evaluation index fj is used. In other words, the reliability R j i The higher the label yi of a labeled sample xi, the more other labeled samples xi have the same label as the label yi, and the less the interpretation fluctuation between the labeled samples xi in the similarity evaluation index fj.

[0049] The reliability calculation unit 402 calculates the reliability R1 of the same label yi from the aggregation table CT1, ..., CTj, ..., CTJ. i ,…,R j i ,…,R J i In other words, the reliability calculation unit 402 obtains the reliability R1 i ,…,R j i ,…,R J iThe average reliability Ri of the label yi is calculated by summing up and dividing by the total number J of similarity evaluation indices fj. The average reliability Ri is an index value indicating the fluctuation in interpretation of the labeled sample xi when the similarity evaluation indices f1 to fJ are used. In other words, the higher the average reliability Ri of a labeled sample xi, the more other labeled samples xi that are assigned the same label as the label yi, indicating that there is less fluctuation in interpretation between the labeled samples xi among the similarity evaluation indices f1 to fJ.

[0050] FIG. 8 is a flowchart of an example of a reliability calculation process procedure performed by the reliability calculation unit 402 according to the first embodiment.

[0051] (Step S801) The reliability calculation unit 402 judges whether the unselected similarity evaluation tensor Taj of N rows and N columns is in the similarity evaluation tensor Ta. If the unselected similarity evaluation tensor Taj of N rows and N columns is not in the similarity evaluation tensor Ta (step S801: No), the process proceeds to step S806. If the unselected similarity evaluation tensor Taj of N rows and N columns is in the similarity evaluation tensor Ta (step S801: Yes), the process proceeds to step S802.

[0052] (Step S802) The reliability calculation unit 402 selects an unselected similarity evaluation tensor Taj of N rows and N columns, and proceeds to step S803.

[0053] (Step S803) The reliability calculation unit 402 creates a ranking table RTj for the selected similarity evaluation tensor Taj, rearranges the similarities Saj in descending order, and proceeds to step S804.

[0054] (Step S804) The reliability calculation unit 402 creates a counting table CTj and counts the number of identical label samples for each of the similarities Saj from 1st to Kth in the ranking table RTj. The reliability calculation unit 402 stores the counted number of identical label samples in the counting table CTj and proceeds to step S805.

[0055] (Step S805) The reliability calculation unit 402 calculates the reliability R for each label y1, ..., yi, ..., yN based on the number of identical label samples. j 1 ,…,R j i ,…,R j N is calculated and stored in the aggregation table CTj, and the process returns to step S801.

[0056] (Step S806) The reliability calculation unit 402 calculates the reliability R1 of the same label yi from the aggregation table CT1, ..., CTj, ..., CTJ. i ,…,R j i ,…,R J i , calculate the average reliability R1, ..., Ri, ..., RN, and end the reliability calculation process.

[0057] The calculated average reliabilities R1, . . . , Ri, .

[0058] [Learning Section 304] Fig. 9 is an explanatory diagram showing an example of the learning process by the learning unit 304. Fig. 10 is a flowchart showing an example of the procedure of the learning process by the learning unit 304.

[0059] (Step S1001) The learning unit 304 acquires the labeled sample data set group 201, and proceeds to step S1002.

[0060] (Step S1002) The learning unit 304 inputs labeled samples x1, ..., xi, ..., xN to the discrimination model 900, obtains discrimination results Y1, ..., Yi, ..., YN from the discrimination model 900, and proceeds to step S1003.

[0061] (Step S1003) The learning unit 304 calculates errors e(x1), ..., e(xi), ..., e(xN) which are the differences between the labels y1, ..., yi, ..., yN and the discrimination results Y1, ..., Yi, ..., YN, and proceeds to step S1004.

[0062] (Step S1004) The learning unit 304 weights the errors e(x1), ..., e(xi), ..., e(xN) by the average reliabilities R1, ..., Ri, ..., RN, and proceeds to step S1005.

[0063] (Step S1005) The learning unit 304 calculates a loss value Lx using a loss function based on the weighted errors R1×e(x1), ..., Ri×e(xi), ..., RN×e(xN), and proceeds to step S1006.

[0064] (Step S1006) The learning unit 304 updates the learning parameters of the discrimination model 900 by backpropagating the error so as to reduce the loss value Lx. This completes the learning process.

[0065] In this way, according to the first embodiment, if the same label yi is always assigned to the labeled sample xi that is closer to each interpretation in various interpretation methods (similarity evaluation indexes f1 to fN) for the labeled sample xi, it is understood that the interpretation between the labeled samples xi is stable and the label yi is likely to be the labeled sample xi that is easy to judge as being correct. Therefore, it is possible to improve the reliability of the interpretation fluctuation between the labeled samples xi.

[0066] In addition, since the average reliability R1 to RN for each labeled sample xi can be obtained through the similarity evaluation indexes f1 to fN, it becomes possible to learn a discrimination model 900 that eliminates uncertainty in the interpretation of the labeled samples xi as described above. EXAMPLES

[0067] A description will be given of Example 2. In Example 1, the interpretation fluctuation between labeled samples x i is calculated as an average reliability Ri. In Example 2, an unlabeled sample z m is further used to calculate the lowness of the interpretation fluctuation between labeled samples x i and unlabeled samples z m as an average reliability Ri.

[0068] In the second embodiment, the differences from the first embodiment will be described, so that the same components as those in the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted.

[0069] [Similarity calculation process using similarity evaluation index fj] Fig. 11 is an explanatory diagram illustrating a similarity calculation process example 1 using the similarity evaluation index fj according to the second embodiment. Fig. 12 is an explanatory diagram illustrating a similarity calculation process example 2 using the similarity evaluation index fj according to the second embodiment. Fig. 13 is an explanatory diagram illustrating a similarity calculation process example 3 using the similarity evaluation index fj according to the second embodiment.

[0070] 11, the similarity calculation unit 401 inputs the labeled sample data set (x1, y1) and each of the unlabeled samples z1, ..., zm, ..., zM to the similarity evaluation indexes f1 to fJ. The similarity calculation unit 401 outputs the similarities Sb1(1,1), ..., Sb1(1,m), ..., Sb1(1,M), ..., Sbj(1,1), ..., Sbj(1,m), ..., Sbj(1,M), ..., SbJ(1,1), ..., SbJ(1,m), ..., SbJ(1,M) from the similarity evaluation indexes f1 to fJ as a similarity evaluation tensor with J rows and M columns.

[0071] 12, the similarity calculation unit 401 inputs the labeled sample data set (xi, yi) and each of the unlabeled samples z1, ..., zm, ..., zM to the similarity evaluation indexes f1 to fJ. The similarity calculation unit 401 outputs the similarities Sb1(i,1), ..., Sb1(i,m), ..., Sb1(i,M), ..., Sbj(i,1), ..., Sbj(i,m), ..., Sbj(i,M), ..., SbJ(i,1), ..., SbJ(i,m), ..., SbJ(i,M) from the similarity evaluation indexes f1 to fJ as a similarity evaluation tensor with J rows and M columns.

[0072] 13, the similarity calculation unit 401 inputs the labeled sample data set (xN, yi) and each of the unlabeled samples z1, ..., zm, ..., zM to the similarity evaluation indexes f1 to fJ. The similarity calculation unit 401 outputs the similarities Sb1(N,1), ..., Sb1(N,m), ..., Sb1(N,M), ..., Sbj(N,1), ..., Sbj(N,m), ..., Sbj(N,M), ..., SbJ(N,1), ..., SbJ(N,m), ..., SbJ(N,M) from the similarity evaluation indexes f1 to fJ as a similarity evaluation tensor with J rows and M columns.

[0073] Since the similarity Sbj(p,q) (where p and q are sample number i of labeled sample xi) and the similarity Sbj(q,p) are the same similarity, the analysis device 100 treats the similarity Sbj(q,p) as the similarity Sbj(p,q). Since the similarity Sbj(i,i) is the similarity between samples xi with the same label, it is the maximum value. When there is no need to distinguish between these, it will simply be written as the similarity Sbj.

[0074] The similarity calculation unit 401 judges whether or not each similarity Sbj is equal to or greater than a predetermined threshold. If the similarity Sbj is equal to or greater than the predetermined threshold, the similarity calculation unit 401 assigns the label yi to the unlabeled sample zm as a pseudo label wm. If the similarity Sbj is not equal to or greater than the predetermined threshold, the similarity calculation unit 401 does not assign the pseudo label wm to the unlabeled sample zm.

[0075] In order to distinguish the unlabeled sample zm to which the pseudo label wm is not assigned, the unlabeled sample zm to which the pseudo label wm is assigned is written as the pseudo labeled sample zm. In addition, when there is no need to distinguish between the similarity Saj and the similarity Sbj, the similarity is written as the similarity Sj.

[0076] In the second embodiment, the similarity calculation unit 401 creates ranking tables RT1, ..., RTj, ..., RTJ, as in the first embodiment. In this case, the samples x / z to be ranked are labeled samples xi and pseudo-labeled samples zm.

[0077] Similarly, the similarity calculation unit 401 creates the aggregation tables CT1, ..., CTj, ..., CTJ in the same manner as in the first embodiment. j i The similarity Sj used to calculate the average reliability Ri is the similarity Sbj, but may include the similarity Saj between labeled samples.

[0078] In this way, according to the second embodiment, by assigning the pseudo label wm, if the same label y / w is always assigned to the sample x / z that is closer to each interpretation in various interpretation methods (similarity evaluation indexes f1 to fN) for the sample x / z, it is understood that the interpretation between the samples x / z is stable and the label y / w is likely to be correct for the sample x / z. Therefore, it is possible to improve the reliability of the interpretation fluctuation between the samples x / z.

[0079] In addition, since the average reliability R1 to RN for each labeled sample x / z can be obtained through the similarity evaluation indexes f1 to fN, it becomes possible to learn a discriminant model 900 by semi-supervised learning that eliminates uncertainty in the interpretation of the above-mentioned samples x / z. EXAMPLES

[0080] Example 3 will be described. Example 3 shows an example of changing the ranking in the ranking table RT1, ..., RTj, ..., RTJ in Example 1 or Example 2. Note that, in Example 3, changes to Example 1 will be described, but in Example 2 as well, only the pseudo-labeled sample zm is included in the target sample x / z.

[0081] In addition, in the third embodiment, the differences from the first and second embodiments will be focused on, so the same components as those in the first and second embodiments will be given the same reference numerals and the description thereof will be omitted.

[0082] 14 is a flowchart illustrating an example of a reliability calculation process procedure performed by the reliability calculation unit 402 according to the second embodiment. Steps S801 to S803, S805, and S806 are the same as those in FIG.

[0083] (Step S1404) The reliability calculation unit 402 creates a counting table CTj and counts the number of identically labeled samples for each similarity Saj equal to or greater than the threshold value α in the ranking table RTj. The threshold value α is a value within the range of the similarity Saj, 0 to 1, and can be set arbitrarily. The reliability calculation unit 402 stores the counted number of identically labeled samples in the counting table CTj and proceeds to step S805.

[0084] In this way, in the first and second embodiments, the labels y / w whose similarities Sj are ranked 1st to Kth are targeted for the number of identical label samples based on relative evaluation of the similarities Sj from 1st to Kth, whereas in the third embodiment, the labels y / w whose similarities Sj are ranked 1st to Kth are targeted for the number of identical label samples based on absolute evaluation of being equal to or greater than the threshold value α.

[0085] In the case of absolute evaluation, labels y / w whose similarity Sj is less than the threshold value α are excluded from the number of identical label samples even if they are within the top K. Therefore, when improving accuracy, it is preferable to select labels y / w that are to be included in the number of identical label samples by such absolute evaluation. On the other hand, in the case of absolute evaluation, since there is a possibility that the number of labels y / w equal to or greater than the threshold value α will be insufficient, it is preferable to select targets for the number of identical label samples by relative evaluation in order to level out the number of labels y / w by securing a certain number of labels y / w.

[0086] Furthermore, the process of step S804 and the process of step S1404 may be selectable by a user operation, thereby improving convenience. EXAMPLES

[0087] Next, a fourth embodiment will be described. In the fourth embodiment, the reliability R j i In this example, instead of calculating the average reliability Ri, a label y / w is output. In the fourth embodiment, the differences from the first to third embodiments will be described, and therefore the same components as those in the first to third embodiments will be denoted by the same reference numerals and description thereof will be omitted.

[0088] For example, when only the labeled sample data set (xi, yi) is applied, the inspection unit 302 calculates the reliability R j i And, the labeled sample data set (xi, yi) is passed to the learning unit 304 without calculating the average reliability Ri.

[0089] Furthermore, when the labeled sample data set (xi, yi) and the unlabeled sample wm are applied, the inspection unit 302 uses the similarity calculation unit 401 to calculate an N×M×J similarity evaluation tensor as shown in FIGS. 11 to 13 to generate a pseudo label zm. Then, the learning unit 304 learns J discriminant models. The learning unit 304 may perform ensemble learning on the J discriminant models.

[0090] 15 is an explanatory diagram showing a learning example according to Example 4. An N×M×J similarity evaluation tensor Tb is composed of J similarity evaluation tensors Tb1, ..., Tbj, ..., TbJ of N rows and M columns.

[0091] The similarity calculation unit 401 generates pseudo labels w1(1,1), ..., w1(1,m), ..., w1(1,M), ..., w1(i,1), ..., w1(i,m), ..., w1(i,M) using the similarity evaluation tensor Tb1.

[0092] The similarity calculation unit 401 generates pseudo labels wj(1,1), ..., wj(1,m), ..., wj(1,M), ..., wj(i,1), ..., wj(i,m), ..., wj(i,M) using the similarity evaluation tensor Tbj.

[0093] The similarity calculation unit 401 generates pseudo labels wJ(1,1), ..., wJ(1,m), ..., wJ(1,M), ..., wJ(i,1), ..., wJ(i,m), ..., wJ(i,M) using the similarity evaluation tensor TbJ.

[0094] The learning unit 304 learns the discrimination model 1500-j for each similarity evaluation index fj. For example, the learning unit 304 inputs pseudo-labeled samples z1, ..., xm, ..., zM to the discrimination model 1500-j and obtains discrimination results Wj1, ..., Wjm, ..., WjM from the discrimination model 1500-j.

[0095] The learning unit 304 calculates errors ej(z1), ..., ej(zm), ..., ej(zM) which are the differences between the pseudo labels wj(1,1), ..., wj(1,m), ..., wj(1,M) and the discrimination results Wj1, ..., Wjm, ..., WjM.

[0096] The learning unit 304 calculates a loss value Ljw using a loss function based on errors ej(z1), ..., ej(zm), ..., ej(zM), and updates the learning parameters of the discriminant model 1500-j by backpropagating the error so that the loss value Ljw becomes smaller. In this way, the discriminant model 1500-j for each similarity evaluation index fj is trained. The discriminant model 1500-j for each similarity evaluation index fj is used in the prediction process of the prediction unit 305.

[0097] Furthermore, the learning unit 304 may perform ensemble learning of the discriminant models 1500-1 to 1500-J to generate a discriminant model that integrates the similarity evaluation indexes f1 to fJ (hereinafter, an integrated discriminant model). The ensemble learning may include the discriminant model 900. The generated integrated discriminant model is used for the prediction process of the prediction unit 305.

[0098] In this way, according to the fourth embodiment, the pseudo label wm is assigned and the discrimination model 1500-j is generated for each similarity evaluation index fj, so that if the pseudo label wm is assigned to a sample zm with a similar label xi with a label yi by various interpretations of the sample zm with a pseudo label, there is no fluctuation in the interpretation of the sample zm with a pseudo label, that is, the sample zm with a pseudo label is likely to be one for which it is easy to determine that the pseudo label wm is correct. Therefore, it is possible to improve the reliability of the fluctuation in the interpretation of the sample zm with a pseudo label.

[0099] Moreover, by performing ensemble learning on the discriminant models 1500-1 to 1500-J, it is possible to further improve the reliability of the fluctuation in the interpretation of the pseudo-labeled sample zm.

[0100] As described above, the above-mentioned analysis device 100 can suppress the variation in interpretation between samples (different labels between different samples) by ensuring the consistency of interpretation between different samples. In addition, the above-mentioned analysis device 100 calculates the reliability indicating the absence of variation in interpretation between different samples based on the extent to which the same labels are shared between different samples. By applying this reliability to learning the discrimination model, the accuracy of the discrimination model can be improved.

[0101] The above-mentioned analysis device 100 can also be configured as follows (1) to (8).

[0102] (1) The analysis device 100 includes a processor 101 that executes a program and a storage device 102 that stores the program. The processor 101 performs an acquisition process (acquisition unit 301) to acquire a plurality of sample data sets (xi, yi) each having a first sample (labeled sample xi) and a first label (yi) that pairs with the first sample, and calculates a similarity Saj(1) between the two first samples for each similarity evaluation index fj by inputting two first samples (xp, xq) among the plurality of first samples in the sample data set (xi, yi) acquired by the acquisition process into each of a plurality of similarity evaluation indexes f1 to fJ that evaluate the similarity between two data. The similarity calculation unit 401 calculates a similarity of the first samples, i, 1), ..., Saj(1,N), ..., Saj(i,1), ..., Saj(i,N), ..., Saj(N,1), ..., Saj(N,N), and a reliability calculation unit 402 calculates, for each first sample, the number of other first samples having the same first label based on the similarity calculated by the similarity calculation process, and calculates, for each first sample, a first reliability Ri indicating the low degree of fluctuation in the interpretation of the first sample based on the aggregation result (aggregation table CTj).

[0103] (2) In the analysis device 100 of (1) above, in the acquisition process, the processor 101 further acquires a plurality of second samples (unlabeled samples zm) that are not labeled, and in the similarity calculation process, the processor inputs combinations of the first samples (labeled samples xi) and the second samples (unlabeled samples zm) to each of the plurality of similarity evaluation indexes f1 to fJ, thereby calculating similarities Sb1(1,1), ..., Sb1(1,M), ..., Sbj between the combinations for each similarity evaluation index fj. (1,1), ..., Sbj(1,M), ..., SbJ(1,1), ..., SbJ(1,M), and based on the similarity, the first label yi in the combination is assigned to the second sample zm as a second label wj(i,m). Based on the similarity, the first reliability is calculated for each of the first samples (labeled samples xi). Based on the similarity, the second reliability indicating low fluctuation in interpretation of the second sample (pseudo-labeled sample zm) is calculated for each of the second samples (pseudo-labeled sample zm).

[0104] (3) In the analysis apparatus 100 of (1) above, in the reliability calculation process, the processor 101 calculates the first reliability for specific first samples whose similarities are up to a predetermined rank K.

[0105] (4) In the analysis apparatus 100 of (1) above, in the reliability calculation process, the processor 101 calculates the first reliability for a specific first sample whose similarity is equal to or greater than a predetermined threshold value α.

[0106] (5) In the analysis device 100 of (1) above, the processor 101 executes an output process (output unit 303) for outputting the first reliability for each of the first samples calculated by the calculation process.

[0107] (6) In the analysis device 100 of (1) above, the processor 101 inputs the multiple first samples (labeled samples x1, ..., xi, ..., xN) into a discrimination model 900 that outputs the results of discriminating input data, and executes a learning process (learning unit 304) to learn the discrimination model 900 based on the errors e(x1), ..., e(xi), ..., e(xN) between the discrimination results Y1, ..., Yi, ..., YN for each of the first samples (labeled samples xi) output from the discrimination model 900 and the first labels y1, ..., yi, ..., yN for each of the first samples (labeled samples xi), and the first reliability R1, ..., Ri, ..., RN for each of the first samples (labeled samples xi).

[0108] (7) In the analysis device 100 of (6) above, the processor 101 executes a prediction process (prediction unit 305) that inputs data to be predicted (test data 351) to the discriminant model 900 and outputs a prediction result 352 obtained from the discriminant model 900.

[0109] (8) The analysis device 100 includes a processor 101 that executes a program and a storage device 102 that stores the program. The processor 101 performs an acquisition process to acquire a plurality of sample data sets (xi, yi) each having a first sample (labeled sample xi) and a first label yi that pairs with the first sample, and a plurality of second samples (unlabeled samples zm) that are not labeled, and inputs a combination (xi, zm) of the first sample (labeled sample xi) and the second sample (unlabeled sample zm) in the sample data set (xi, yi) acquired by the acquisition process to each of a plurality of similarity evaluation indexes f1 to fJ that evaluate the similarity between two data, thereby calculating a similarity Sbj between the combinations (xi, zm) for each of the similarity evaluation indexes fj, and matching the first label yi in the combination (xi, zm) with the second sample (unlabeled sample zm) based on the similarity Sbj. and a learning process is executed to learn each of the discrimination models 1500-1 to 1500-J based on the errors ej(z1), ..., ej(zm), ..., ej(zM) between the discrimination results Wj1, ..., Wjm, ..., WjM for the second samples (pseudo-labeled samples zm) output from each of the discrimination models 1500-1 to 1500-J for the similarity evaluation index fj and the second labels wm for the second samples (pseudo-labeled samples zm).

[0110] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Also, a configuration of another embodiment may be added to a configuration of one embodiment. Also, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0111] Furthermore, each of the aforementioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0112] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

[0113] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0114] 200 Sample DB fj Similarity evaluation index 100 Analyzer 101 Processor 102 Storage Devices 201 labeled sample datasets 202 unlabeled sample datasets 301 Acquisition Department 302 Inspection Department 303 Output section 304 Learning Department 305 Prediction Department 401 Similarity calculation unit 402 Reliability calculation unit 900 Discriminant Model

Claims

1. An analysis apparatus having a processor that executes a program and a storage device that stores the program, The processor, an acquisition process for acquiring a plurality of sample data sets having first samples and first labels paired with the first samples; a similarity calculation process for inputting two first samples among a plurality of first samples in a sample data set acquired by the acquisition process into a plurality of similarity evaluation indexes for evaluating a similarity between two data, thereby calculating a similarity between the two first samples for each of the similarity evaluation indexes; a reliability calculation process for counting, for each first sample, the number of other first samples having the same first label based on the similarity calculated by the similarity calculation process, and calculating, for each first sample, a first reliability indicating low uncertainty in interpretation of the first sample based on a counting result; An analytical device characterized by performing the above.

2. The analysis device according to claim 1 , In the acquisition process, the processor further acquires a plurality of second samples having no labels; In the similarity calculation process, the processor inputs a combination of the first sample and the second sample into each of the multiple similarity evaluation indexes, thereby calculating a similarity between the combinations for each of the similarity evaluation indexes, assigns the first label in the combination to the second sample as a second label based on the similarity, calculates the first reliability for each of the first samples based on the similarity, and calculates a second reliability for each of the second samples based on the similarity, the second reliability indicating low fluctuation in interpretation of the second sample.

1. An analytical device comprising:

3. The analysis device according to claim 1 , In the reliability calculation process, the processor calculates the first reliability for a specific first sample whose similarity is up to a predetermined rank.

1. An analytical device comprising:

4. The analysis device according to claim 1 , In the reliability calculation process, the processor calculates the first reliability for a specific first sample whose similarity is equal to or greater than a predetermined threshold value.

1. An analytical device comprising:

5. The analysis device according to claim 1 , The processor, an output process for outputting the first reliability for each of the first samples calculated by the calculation process;

6. The analysis device according to claim 1 , The processor, An analysis device characterized in that, when the plurality of first samples are input into a discrimination model that outputs the results of discriminating input data, a learning process is executed to learn the discrimination model based on the error between the discrimination results for each of the first samples output from the discrimination model and the first label for each of the first samples, and a first reliability for each of the first samples.

7. The analysis device according to claim 6, The processor, The analysis apparatus executes a prediction process in which prediction target data is input to the discriminant model, and a prediction result obtained from the discriminant model is output.

8. An analysis apparatus having a processor that executes a program and a storage device that stores the program, The processor, an acquisition process for acquiring a plurality of sample data sets having a first sample and a first label paired with the first sample, and a plurality of second samples without a label; a similarity calculation process in which a combination of the first sample and the second sample in the sample data set acquired by the acquisition process is input to each of a plurality of similarity evaluation indices for evaluating a similarity between two pieces of data, thereby calculating a similarity between the combinations for each similarity evaluation index, and assigning the first label in the combination to the second sample as a second label paired with the second sample based on the similarity; An analysis device characterized in that, when the plurality of second samples to which the second labels have been assigned by the similarity calculation process are input into each of the discrimination models for the similarity evaluation index, which output the results of discriminating input data, a learning process is executed to learn each of the discrimination models based on the error between the discrimination results for each of the second samples output from each of the discrimination models and the second labels for each of the second samples.

9. An analysis method performed by an analysis device having a processor that executes a program and a storage device that stores the program, comprising: The processor, an acquisition process for acquiring a plurality of sample data sets having first samples and first labels paired with the first samples; a similarity calculation process for inputting two first samples among a plurality of first samples in a sample data set acquired by the acquisition process into a plurality of similarity evaluation indexes for evaluating a similarity between two data, thereby calculating a similarity between the two first samples for each of the similarity evaluation indexes; a reliability calculation process for counting, for each first sample, the number of other first samples having the same first label based on the similarity calculated by the similarity calculation process, and calculating, for each first sample, a first reliability indicating low uncertainty in interpretation of the first sample based on a counting result; An analysis method comprising the steps of:

10. The processor: an acquisition process for acquiring a plurality of sample data sets having first samples and first labels paired with the first samples; a similarity calculation process for inputting two first samples among a plurality of first samples in a sample data set acquired by the acquisition process into a plurality of similarity evaluation indexes for evaluating a similarity between two data, thereby calculating a similarity between the two first samples for each of the similarity evaluation indexes; a reliability calculation process for counting, for each first sample, the number of other first samples having the same first label based on the similarity calculated by the similarity calculation process, and calculating, for each first sample, a first reliability indicating low uncertainty in interpretation of the first sample based on a counting result; An analysis program characterized by executing the above.

Images