Processing device, processing method, and processing program

Abstract

A processing device 1 is provided with: a labeling unit 51 which labels each input data set using labeling functions with reference to function data 12 including labeling functions that label an input data set or abstain if the input data set cannot be labeled; a discriminator 52 which trains a model, enters each input data set into the trained model, and outputs the discrimination uncertainty of the model, wherein said model learns the label assigned to each input data set, also learns input data 11 with votes that includes the ones of the entered input data sets that have been labeled using a number of labeling functions that is equal to or greater than a threshold value, and outputs, from an input data set, a correct label and the confidence level of each labeling function; and an identification unit 54 which identifies, as presentation data 24, a data set for which the value of an acquisition function considering the discrimination uncertainty 22 is maximized among the data sets in input data 15 with no votes that includes the ones of the entered input data sets that have been labeled using a number of labeling functions that is less than the threshold value. A labeling function newly created for the presentation data 24 is inserted into the function data 12.

Description

[Technical field]

[0001] The present invention relates to a processing device, a processing method, and a processing program. [Background technology]

[0002] Machine learning, especially so-called supervised learning, is widespread in a wide range of fields. In supervised learning, a training dataset is prepared in advance, and a classifier learns based on the training dataset. A training dataset is an input dataset to be analyzed, to which each judgment result such as discrimination, classification, regression, and identity is added. The cost required to create a training dataset with correct answers is a problem in machine learning.

[0003] Active learning and weakly supervised learning are proposed, which computationally augments the training dataset.

[0004] In active learning, an existing training dataset and classifier are used to present a dataset from among the input datasets without correct answers that would improve the performance of the classifier if the correct answer were known. The correct answer is annotated to the presented dataset and added to the training dataset.

[0005] In weakly supervised learning, a function corresponding to the rules and knowledge of the subject who gives the correct answer is implemented in the system, and the system gives the correct answer to the input dataset according to the function. The dataset with the correct answer is added to the training dataset.

[0006] The idea is that by combining active learning and weakly supervised learning, the labeling process can be made more efficient.

[0007] For example, in weakly supervised learning, there is a technique for adding rules in a manner similar to active learning (Non-Patent Document 1). In Non-Patent Document 1, when implementing rules to a group of input datasets, input datasets for which the output majority vote is split or there are no votes are extracted. For an input dataset randomly selected from the extracted input datasets, a rule is added to lead to the correct answer.

[0008] There is also a method of identifying rules that can be expressed as functions based on the form of a function and specific features previously determined by the subject, and asking the subject whether or not to add the rule (Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Benjamin Cohen-Wang and 3 others, "Interactive Programmatic Labeling for Weak Supervision", August 4-8, 2019, Workshop at KDD [Non-Patent Document 2] David Kartchner and 4 others, " ReGAL: Rule-Generative Active Learning for Model-in-the-Loop Weak Supervision ", December 7-12, 2020, Workshop at NeurIPS Summary of the Invention [Problem to be solved by the invention]

[0010] However, the methods described in Non-Patent Document 1 and Non-Patent Document 2 do not take into consideration the expected value of improvement in classifier performance, such as active learning. The methods described in Non-Patent Document 1 and Non-Patent Document 2 may have difficulty in achieving efficient learning, for example, because it takes time to appropriately add rules.

[0011] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technology capable of appropriately presenting an input dataset to which a correct answer should be assigned in weakly supervised learning. [Means for solving the problem]

[0012] A processing device according to one aspect of the present invention includes a labeling unit that refers to function data including a labeling function that labels an input data set or abstains when labeling is not possible, and labels each input data set with the labeling function, a classifier that learns voted input data including the labels assigned to each input data set and input data sets labeled with a threshold number or more of the labeling functions among the input data sets to learn a model that outputs a correct label and a reliability of each labeling function from the input data set, inputs each input data set into the learned model, and outputs an uncertainty of discrimination in the model, and an identification unit that identifies, as presentation data, a data set that maximizes a value of an acquisition function that takes the uncertainty of discrimination into consideration, among each data set of voted input data including input data sets labeled with a threshold number or less of the labeling functions among the input data sets. A labeling function newly created for the presentation data is inserted into the function data.

[0013] A processing method according to one aspect of the present invention includes the steps of: a computer refers to function data including a labeling function that labels an input data set or abstains when labeling is not possible, and labels each input data set with the labeling function; the computer learns voted input data including the labels assigned to each input data set and input data sets that are labeled with a threshold number or more of the labeling functions among the input data sets, learns a model that outputs a correct label and a reliability of each labeling function from the input data set, inputs each input data set into the learned model, and outputs an uncertainty of discrimination in the model; and the computer identifies, as presented data, a data set that has a maximum value of an acquisition function that takes the uncertainty of discrimination into consideration, among each data set of voted input data including input data sets that are labeled with a threshold number or less of the labeling functions among the input data sets. The labeling function newly created for the presented data is inserted into the function data.

[0014] One aspect of the present invention is a processing program that causes a computer to function as the processing device. Effect of the Invention

[0015] According to the present invention, it is possible to provide a technology capable of appropriately presenting an input data set to which a correct answer should be assigned in weakly supervised learning. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating functional blocks of a processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a data structure of input data. [Diagram 3] FIG. 3 is a diagram illustrating an example of the data structure of the function data. [Figure 4] FIG. 4 is a diagram illustrating an example of the data structure of labeled input data. [Diagram 5]FIG. 5 is a flowchart for explaining an outline of the processing of the processing device. [Figure 6] FIG. 6 is a diagram illustrating functional blocks of the classifier. [Figure 7] FIG. 7 is a diagram illustrating an example of a data structure of a classification probability vector. [Figure 8] FIG. 8 is a flowchart illustrating an overview of the processing by the model processing unit in the classifier. [Figure 9] FIG. 9 is a flowchart outlining the process performed by the reliability calculation unit of the model processing unit. [Figure 10] FIG. 10 is a diagram illustrating the hardware configuration of a computer used in the processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0018] A processing device 1 according to an embodiment identifies an input data set to which a new labeling function is to be applied based on an output result of an existing labeling function in weakly supervised learning. The identified input data set is presented to a subject E. The subject E generates a new labeling function for attaching a label to the identified input data set. The newly generated labeling function is added to the existing labeling function. In this way, the processing device 1 determines an input data set for creating a new labeling function, thereby enabling efficient generation of a labeling function.

[0019] 1 includes each of data items including input data 11, function data 12, labeled input data 13, voted input data 14, voteless input data 15, output data 21, classification uncertainty 22, cluster data 23, submitted data 24, and new function data 25, as well as each of functions including a labeling unit 51, a classifier 52, a clustering unit 53, an identifying unit 54, and an updating unit 55. Each of the data items is stored in a memory 902 or a storage 903. Each of the functions is implemented in a CPU 901.

[0020] The input data 11 is data to be labeled by the labeling function. The input data 11 is a collection of multiple input data sets, as shown in Fig. 2. In the embodiment of the present invention, the number of input data sets included in the input data 11 is |D|.

[0021] The function data 12 is data of a labeling function that labels each input data set of the input data 11. The labeling function is a function that labels an input data set or abstains if labeling is not possible. The function data 12 is a collection of multiple labeling functions, as shown in Fig. 3. In the embodiment of the present invention, the number of labeling functions included in the function data 12 is |F|.

[0022] The labeled input data 13 is data in which each input data set of the input data 11 is labeled with a labeling function. As shown in Fig. 4, the labeled input data 13 has values ​​set corresponding to an identifier for identifying the input data set and an identifier for identifying the labeling function. In the example shown in Fig. 4, the first character after the alphabet v is an identifier for identifying the input data set, and the second character is an identifier for the labeling function that processed the input data set.

[0023] If the corresponding labeling function is able to determine the label of the corresponding input data set, the value is set to the identifier of the determined label. On the other hand, if the corresponding labeling function is unable to determine the label of the corresponding input data set, the value is set to a value indicating that the determination was not possible. The value indicating that the determination was not possible is set to a value that is not set in the label identifier, such as 0.

[0024] Each input data set of the input data 11 is classified into either voted input data 14 or no-vote input data 15 depending on whether the processing result of the labeling unit 51 is equal to or greater than a threshold. The threshold is 1 or greater.

[0025] The input data with votes 14 includes input data sets that are labeled with a number of labeling functions equal to or greater than a threshold value among the input data sets of the input data 11. Among the input data, the input data sets that are labeled with labeling functions equal to or greater than a predetermined threshold value are included in the input data with votes 14.

[0026] The no-vote input data 15 includes input data sets labeled with labeling functions of which the number of labeling functions is less than a threshold value among the input data sets. The no-vote input data 15 includes input data sets labeled with labeling functions of which the number of labeling functions is less than a predetermined threshold value among the input data.

[0027] The output data 21 is data resulting from attaching a correct answer label to each data set of the input data 11. The processing device 1 adds new function data 25 to the function data 12, and the classifier 52 repeats the learning process until a predetermined condition is satisfied, and then the classifier 52 outputs the output data 21.

[0028] The classification uncertainty 22 is the classification uncertainty of the learning model in the classifier 52. The classification uncertainty 22 is referred to in order to select a data set to be presented to the subject E from the no-vote input data 15.

[0029] The cluster data 23 indicates the result of dividing each input data set included in the no-vote input data 15 into a plurality of clusters. For example, the cluster data 23 associates a cluster identifier with an identifier of an input data set belonging to that cluster.

[0030] The presented data 24 is data presented to the subject E when the subject E creates a new labeling function. The presented data 24 includes one or more input data sets.

[0031] The new function data 25 is data that specifies a new labeling function generated by the subject E who confirmed the presentation data 24. The new function data 25 includes one or more labeling functions.

[0032] A new labelling function is generated for the input data set contained in the presentation data 24, thereby enabling the performance of the classifier 52 to be improved.

[0033] The labeling unit 51 labels each input data set with a labeling function. The labeling unit 51 stores the labeling result as labeled input data 13. The labeling unit 51 further classifies each input data set of the input data 11 into input data with votes 14 that are labeled with a labeling function equal to or greater than a threshold, and into input data without votes 15 that are labeled with a labeling function less than the threshold (including 0).

[0034] The classifier 52 learns the labels assigned to each input data set of the labeled input data 13 and the voted input data 14, and learns a model that outputs a correct label and the reliability of each labeling function from the input data set. The classifier 52 inputs each input data set of the input data 11 into the trained model, and outputs the classification uncertainty 22 in the model. The classification uncertainty 22 is referenced to identify the presentation data 24. The processing of the classifier 52 will be described in detail later.

[0035] The clustering unit 53 divides each input data set of the no-vote input data 15 into a plurality of clusters. The clustering unit 53 divides each input data set into clusters by referring to the values ​​of real number data indicating the characteristics of each input data set.

[0036] In the embodiment of the present invention, clustering unit 53 uses agglomerative clustering to partition each input data set of no-vote input data 15 into multiple clusters. An example of agglomerative clustering is HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise).

[0037] In agglomerative clustering, each input data set is represented as a leaf on a tree diagram based on the distance between each input data set, and one or more leaves are grouped hierarchically. The parameters indicating how much data grouping is considered may be set in advance or may be automatically adjusted. Agglomerative clustering extracts optimal clusters and outliers that do not belong to any cluster. Agglomerative clustering can extract clusters that ensure semantic grouping without specifying the number of clusters in advance or taking into account uneven distribution of leaves, while adjusting the strength of grouping with parameters.

[0038] The identification unit 54 generates the presentation data 24 to be presented to the subject E. Among the data sets of the input data 15 without votes, the identification unit 54 identifies the data set in which the value of the acquisition function considering the classification uncertainty 22 is maximized as the presentation data 24. The acquisition function is positively correlated with the classification uncertainty 22. After viewing the presentation data 24, the subject E generates a newly created labeling function for the presentation data 24 to generate new function data 25. The identification unit 54 can present the input data set in which the expected value of the improvement in the performance of the classifier 52 is maximized to the subject E. The acquisition function is, for example, batch BALD (J. Kirsch et al., “Batchbald: Efficient and diverse batch acquisition for deep Bayesian active learning,” NueIPS2019). Batch BALD enables the expected value of the improvement in the performance of the classifier 52 expressed by a submodular function to be approximately calculated by a greedy method.

[0039] The identification unit 54 identifies, as the presented data, an input data set belonging to a cluster to which an input data set that has a maximum value of the acquisition function considering the uncertainty of identification 22 belongs. The identification unit 54 calculates the value of the acquisition function considering the uncertainty of identification 22 for each data set of the input data set without votes 15. Here, the values ​​may be calculated for all data sets among the multiple data sets belonging to each cluster, or may be calculated for the number of data sets to be presented to the subject E. The identification unit 54 calculates a representative value for each cluster. The representative value is the maximum value among the input data sets belonging to a certain cluster. The identification unit 54 approximately calculates the value of the acquisition function as the amount of reduction in joint entropy for each cluster, and identifies multiple input data sets of a cluster to which a data set that has a maximum value of the acquisition function belongs, so that it is possible to avoid the selection of only similar data while ensuring a certain degree of closeness between each data set of the input data set without votes 15, and present various data to the subject E. This is expected to improve the efficiency of the labeling function by the subject E.

[0040] The acquisition function batch BALD is a method used when multiple data are presented together in active learning. This takes into account the overlap of multiple data, reducing the possibility that identical or highly similar data are selected.

[0041] The update unit 55 inserts each labeling function included in the new function data 25 into the function data 12. When the function data 12 is updated, the processing device 1 repeats the learning of the classifier 52 by referring to the updated function data 12, and presents the presentation data 24 to the subject E.

[0042] When a new labeling function is added from the presentation data 24, the processing device 1 repeats each process of the labeling unit 51, the discriminator 52, etc., and the discriminator 52 learns the model again. The learning of the model by the discriminator 52 is repeated until a predetermined condition is satisfied. When the predetermined condition is satisfied, the discriminator 52 inputs each input data set to the learned model and outputs the output data 21. The predetermined condition is that the number of input data sets included in the presentation data 24 is 0, the number of added labeling functions, the number of times of learning, the time required for learning, or a combination of these. By generating new function data 25 from the presentation data 24 generated by the processing device 1, it is possible to efficiently increase the function data 12.

[0043] The classifier 52 also optimizes the labels to be assigned to each input data set and the reliability of each labeling function as one model. Compared to optimizing a model for each label to be assigned to each input data set and the reliability of each labeling function, it is possible to improve the performance of the classifier 52 by taking into account the correlation of these data.

[0044] (Processing method) With reference to FIG. 5, a processing method according to an embodiment of the present invention will be described.

[0045] In step S1, the processing device 1 labels each input data set of the input data 11 with each labeling function of the function data 12. The processing device 1 outputs labeled input data 13 in which each input data set is associated with a label assigned by the labeling function.

[0046] In step S2, the processing device 1 divides each input data set of the input data 11 into voted input data 14 and non-voted input data 15. The voted input data 14 includes input data sets that have received votes from labeling functions with a number equal to or greater than the threshold. The non-voted input data 15 includes input data sets that have received votes from labeling functions with a number less than the threshold.

[0047] In step S3, the processing device 1 performs model processing by the classifier 52. In the model processing, a model of the classifier 52 is trained using the labeled input data 13 and the voted input data 14.

[0048] In step S4, it is determined whether it is time to output the learning result. If it is time to output the learning result, in step S10, the output data 21 output by the model processing unit 71 is output. If it is not time to output the learning result, the process proceeds to step S5.

[0049] In step S5, the processing device 1 inputs the input data 11 to the trained model obtained in step S3, and outputs the classification uncertainty 22.

[0050] In step S6, the processing device 1 clusters the non-voting input data 15. In step S7, the processing device 1 identifies a cluster including a data set having the largest value of an acquisition function that takes into account the classification uncertainty 22. The processing device 1 identifies each data set that belongs to the identified cluster as presentation data 24.

[0051] The presentation data 24 obtained in step S7 is presented to the subject E. In step S8, the processing device 1 obtains a new labeling function generated from the presentation data 24. In step S9, the new labeling function obtained in step S8 is added to the function data 12.

[0052] The processing device 1 returns to step S1, refers to each labeling function of the updated function data 12, and performs processing to relabel and update the model.

[0053] (Classifier) The classifier 52 will be described with reference to Fig. 6. The inputs of the classifier 52 are labeled input data 13, voted input data 14, and non-voted input data 15. The outputs of the classifier 52 are output data 21 and classification uncertainty 22.

[0054] The classifier 52 includes a conversion unit 70 and a model processing unit 71 .

[0055] The conversion unit 70 converts each of the data sets of the voted input data 14 and the voted input data 15 into a real number data set. The real number data set is data that expresses the characteristics of the input data set. In the embodiment of the present invention, the model processing unit 71 processes the real number data set. The conversion unit 70 analyzes the meaning of the input data set by embedding using sentence BERT (Bidirectional Encoder Representations from Transformers), for example, to generate a real number data set.

[0056] The conversion unit 70 converts each data set of the voted input data 14 into each real data set to generate voted real data 60. The voted real data 60 is referred to when the model processing unit 71 trains a model, and is also referred to when the trained model outputs the output data 21 and the classification uncertainty 22.

[0057] The conversion unit 70 converts each data set of the no-vote input data 15 into each real data set to generate no-vote real data 61. The no-vote real data 60 is referenced when the model trained by the model processing unit 71 outputs the output data 21 and the classification uncertainty 22.

[0058] The model processing unit 71 includes a reliability calculation unit 72 , a first vector calculation unit 73 , a second vector calculation unit 74 , and a loss function calculation unit 75 .

[0059] The reliability calculation unit 72 calculates the reliability of the labeling function from the relationship between the voted real number dataset indicating the characteristics of the input dataset to be processed of the voted input data 14 and the labels attached to the input dataset to be processed. The reliability of the labeling function is the average of the reliability of the input datasets calculated for the labeling function.

[0060] The reliability calculation unit 72 acquires the labels attached to the input dataset to be processed and the real dataset from the labeled input data 13 and the voted real data 60. The reliability calculation unit 72 calculates the reliability of each labeling function based on the overlaps and inconsistencies that occur between the labeling functions and the real dataset. The reliability calculation unit 72 outputs reliability data 62 by associating the identifier of the labeling function with the reliability of the labeling function.

[0061] The reliability calculation unit 72 includes, for example, a neural network and a processing unit that calculates the reliability of each labeling function from the output of the neural network. The neural network includes a concatenate layer, a linear tanh layer, a cMCdropout (consistent Monte Carlo dropout) layer, and a linear softmax layer. The concatenate layer receives an input data set to be processed of the labeled input data 13 and a real data set corresponding to the input data set to be processed output from a first cMCdropout layer of the second vector calculation unit 74 described later.

[0062] The neural network outputs the reliability of each labeling function for each input data set, regardless of whether or not it has voted. The processing unit refers to the labeled input data 13, extracts the reliability of the input data set for which the labeling function being processed has voted, and outputs the average of the extracted reliability as the reliability of the labeling function being processed.

[0063] The first vector calculation unit 73 weights the labeling function with the reliability to calculate a tentative correct label 63 for the input data set to be processed and a first classification probability vector 64 including the probability that the input data set to be processed corresponds to each label.

[0064] A first classification probability vector 64 is generated for each of the input data sets and contains probabilities corresponding to each label, the sum of the probabilities corresponding to each label being 1, as shown in FIG.

[0065] The first vector calculation unit 73 sets the probability calculated by using a softmax function as the reliability of the labeling function to a value corresponding to the voted labeling function for the input data set to generate a first classification probability vector 64. The first vector calculation unit 73 sets the label assigned by the most reliable labeling function among the voted labeling functions for the input data set to be processed as a provisional correct label 63. The first vector calculation unit 73 calculates the provisional correct label 63 and the first classification probability vector 64 for each input data set of the input data 11.

[0066] For example, when the results of the three labeling functions for a certain input data set are {1,0,0}, the first vector calculation unit 73 outputs the probability of {0.7,0.15,0.15} as the probability of corresponding to each label, taking into account the reliability of each labeling function. When the results of the three labeling functions for another input data set are {0,0,1}, the model processing unit 71 outputs the probability of {0.25,0.25,0.5} as the probability of corresponding to each label. A high probability is set for the results of the labeling function with high reliability, and a low probability is set for the results of the labeling function with low reliability.

[0067] A case will be described in which a contradiction occurs in which the first labeling function discriminates the first label and the third labeling function discriminates the third label for one input data set, such as the discrimination result being {1,0,3}. The first vector calculation unit 73 outputs a probability of {0.55,0.1,0.35}, for example. Even when a contradiction occurs, a high probability is set for the result discriminated by the labeling function with high reliability, and a low probability is set for the result discriminated by the labeling function with low reliability.

[0068] When the result of the discrimination is {0,0,0}, specifically, when it is determined that each labeling function cannot be discriminated, the first vector calculation unit 73 outputs a probability of {0.33,0.33,0.33}, for example, since there is no material for determining the probability corresponding to each label.

[0069] The second vector calculation unit 74 calculates a second classification probability vector 65 from the real number dataset of the input dataset to be processed for the input dataset on which votes were received. During learning, the input dataset to be processed is the input dataset of the voted input data 14. The second vector calculation unit 74 calculates a loss function using the tentative correct answer label 63, and learns a model that calculates the second classification probability vector 65 from the real number dataset of the input dataset to be processed so as to minimize the loss function. The second vector calculation unit 74 inputs the real number dataset of the input dataset to be processed to the learned model, and calculates the second classification probability vector 65 for each of the input datasets.

[0070] A second classification probability vector 65 is generated for each of the input data sets and contains probabilities corresponding to each label, the sum of which equals one, as shown in Figure 7.

[0071] For an input data set to be processed that has a number of votes equal to or greater than a threshold, the second vector calculation unit 74 inputs a real-number data set corresponding to the input data set to a neural network, and outputs a second classification probability vector 65. The neural network includes a cMCdropout layer, a linear ReLU layer, a cMCdropout layer, and a linear softmax layer.

[0072] The loss function calculation unit 75 calculates a function for comparing each of the first classification probability vector 64 and the second classification probability vector 65 with the provisional correct label 63 for the input data set to be processed. This function is positively correlated with the difference between the sum of the probabilities corresponding to each label and the probability of the provisional correct label 63 for the first classification probability vector 64, and is positively correlated with the difference between the sum of the probabilities corresponding to each label and the probability of the provisional correct label for the second classification probability vector 65. In the embodiment of the present invention, since the sum of the probabilities is 1, the function is, for example, a weighted average of the difference between the value of the probability corresponding to the provisional correct label 63 of the first classification probability vector 64 and 1, and the difference between the value of the probability corresponding to the provisional correct label 63 of the second classification probability vector 65 and 1.

[0073] The loss function calculation unit 75 calculates a loss function correlated with each function for each input data set of the voted input data 14. The loss function calculation unit 75 calculates the sum of the functions calculated for the input data set with votes as the loss function.

[0074] The model processing unit 71 learns the model so as to minimize the loss function calculated by the loss function calculation unit 75. The model processing unit 71 learns in accordance with the calculated loss function so that the probability values ​​corresponding to the provisional correct labels 63 of the first classification probability vector 64 and the second classification probability vector 65 for each of the input data sets for which votes have been received approach 1.

[0075] The model processing unit 71 inputs each input data set of the input data 11 to the trained model, and outputs classification uncertainty 22 in the model. The classification uncertainty 22 has a positive correlation with the entropy of each second classification probability vector 65 calculated for each input data set of the input data 11.

[0076] When the model processing unit 71 receives the real number data set converted by the conversion unit 70 for each input data set of the input data 11, the model processing unit 71 calculates a second classification probability vector 65 including probabilities corresponding to each label by the second vector calculation unit 74. The model processing unit 71 outputs, for example, the average of the entropy of the second classification probability vector 65 calculated for each input data set as the classification uncertainty 22.

[0077] After repeating the learning and satisfying a predetermined termination condition, the model processing unit 71 outputs output data 21, which is a classification result, from the second classification probability vector calculated by the trained model. For each input data set, the model processing unit 71 determines the label with the highest probability in the second classification probability vector 65 calculated from the input data set to be processed as the correct label of the input data set to be processed. The model processing unit 71 outputs output data 21 in which the identifier of the input data set to be processed and its correct label are associated with each other.

[0078] The model processing unit 71 adds a Monte Carlo Dropout layer to the input layer of each neural network of the reliability calculation unit 72 and the second vector calculation unit 74, and further adds a Monte Carlo Dropout layer to the positions other than the final layer. In the Monte Carlo Dropout layer, the neural network connecting the nodes is ignored with a certain probability. By providing the Monte Carlo Dropout layer, the model processing unit 71 behaves as if multiple models with different node connections exist not only when learning a model but also when outputting using a learned model, and can estimate the uncertainty of identification. For example, if the output differs each time learning is repeated, it can be determined that the reliability is low, and if the output is the same, it can be determined that the reliability is high.

[0079] (Model processing method) The model processing method by the model processing unit 71 will be described with reference to Figures 9 and 10. Figures 9 and 10 correspond to step S3 in Figure 5.

[0080] In step S101, the model processing unit 71 performs a reliability calculation process.

[0081] The model processing unit 71 performs the process of step S151 for each input data set of the input data 11. In step S151, the model processing unit 71 calculates the reliability of each input data set and each labeling function from the correlation between the labels attached to each input data set of the input data 11 and the voted real number data 60.

[0082] The model processing unit 71 processes steps S152 and S153 for each labeling function. In step S152, the model processing unit 71 identifies an input data set for which the labeling function to be processed has voted. In step S153, the model processing unit 71 calculates the average of the labeling function to be processed in the input data set identified in step S152 as the reliability of the labeling function to be processed.

[0083] When the processes of steps S152 and S153 are completed for each labeling function, the model processing unit 71 performs processes of steps S102 to S105 for each input data set of the voted input data 14.

[0084] In step S102, the model processing unit 71 calculates a first classification probability vector 64 from the reliability of each labeling function calculated in step S101. In step S103, the model processing unit 71 identifies a label corresponding to the highest value among the first classification probability vectors 64 identified in step S102 as a tentative correct label 63. In step S104, the model processing unit 71 calculates a second classification probability vector 65 from the real number data of the input data set to be processed. In step S105, the model processing unit 71 identifies a function of the input data set to be processed from a function indicating the difference between the first classification probability vector 64 and the tentative correct label 63 and a function indicating the difference between the second classification probability vector 65 and the tentative correct label 63.

[0085] When the processes from step S102 to step S105 are completed for each input data set of the voted input data 14, the process proceeds to step S106. In step S106, the model processing unit 71 identifies a loss function in the model from the function of each input data set identified in step S106. In step S107, the model processing unit 71 learns a model that minimizes the loss function identified in step S106.

[0086] (evaluation) Evaluation of the processing device 1 according to the embodiment of the present invention will be described. The conversion unit 70 converts the input data set into real data indicating features using embedding by sentence BERT. The dropout rate of Monte Carlo Dropout in the neural network of the first vector calculation unit 73 and the second vector calculation unit 74 is 0.2 in the input layer and 0.5 in the other layers. The clustering unit 53 performs clustering using HDBSCAN.

[0087] First, an example of the presentation data 24 output by the processing device 1 according to the embodiment of the present invention will be shown.

[0088] [Table 1]

[0089] In the above presented data 24, although the sentence patterns are various, it can be seen that each input data set contains common information such as "earn money" and "make money." Therefore, subject E who observes such presented data 24 can efficiently create a labeling function that can assign labels to the presented data 24.

[0090] According to the processing device 1 according to the embodiment of the present invention, the classifier 52 learns a model so as to minimize one loss function for both the reliability of each labeling function and the labeling of the input data set. Furthermore, the processing device 1 can calculate the uncertainty of the model output using the learned model.

[0091] Moreover, according to the processing device 1, an expected value of the improvement in performance of the classifier 52 expressed by a submodular function is approximated by a greedy method, and an input data set that maximizes the expected value is presented to the subject E. This is expected to improve the efficiency of creating a labeling function by the subject E.

[0092] The processing device 1 of the present embodiment described above is, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. In this computer system, the CPU 901 executes a processing program loaded on the memory 902, thereby realizing each function of the processing device 1.

[0093] The processing device 1 may be implemented in one computer or in multiple computers, and may also be a virtual machine implemented in a computer.

[0094] The processing device 1 program can be stored in a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or can be distributed via a network.

[0095] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0096] 1 Processing equipment 11 Input Data 12 Function Data 13 Labeled input data 14 Input data with votes 15 No Voting Input Data 21 Output Data 22 Uncertainty of Identification 23 Cluster Data 24 Presented Data 25 New Function Data 51 Labeling section 52 Classifier 53 Clustering Department 54 Specific part 55 Update section 60 votes real data 61 Real Data without Votes 62 Reliability Data 63 Tentative correct label 64 First Classification Probability Vector 65 Second Classification Probability Vector 70 Conversion unit 71 Model processing section 72 Reliability calculation unit 73 First Vector Calculation Unit 74 Second Vector Calculation Unit 75 Loss function calculation section 901 CPU 902 Memory 903 Storage 904 Communication equipment 905 Input Device 906 Output Device E subject

Claims

1. Refer to the function data, which contains a labeling function that labels the input dataset or abstains if labeling is not possible. a labeling unit for labeling each input data set with said labeling function; a classifier that learns voted input data including labels assigned to each of the input data sets and input data sets labeled with a threshold or more of the labeling functions from each of the input data sets, learns a model that outputs a correct label and a reliability of each labeling function from the input data set, inputs each of the input data sets to the trained model, and outputs an uncertainty of classification in the model; An identification unit that identifies, as presentation data, a data set in which a value of an acquisition function taking into account the uncertainty of the discrimination is maximized, from among each data set of vote-free input data including an input data set labeled with a labeling function having a number less than a threshold value among the input data sets; The newly created labeling function for the presentation data is inserted into the function data. Processing unit.

2. The classifier is a reliability calculation unit that calculates the reliability of the labeling function based on a relationship between a voted real number dataset indicating characteristics of an input dataset to be processed of the voted input data and a label assigned to the input dataset to be processed; a first vector calculation unit that weights the labeling function by the reliability to calculate a tentative correct label of the input data set to be processed and a first classification probability vector including a probability that the input data set to be processed corresponds to each label; a second vector calculation unit that calculates a second classification probability vector from the voted real number data set; Calculating a function for comparing each of the first classification probability vector and the second classification probability vector with the provisional correct label for the input data set to be processed; a loss function calculation unit that calculates a loss function correlated with each function for each input data set of the voted input data, The model is trained so as to minimize the loss function. The processing device of claim 1 .

3. The function is Regarding the first classification probability vector, there is a positive correlation between the difference between the sum of probabilities corresponding to each label and the probability of the provisional correct label, Regarding the second classification probability vector, there is a positive correlation with the difference between the sum of probabilities corresponding to each label and the probability of the provisional correct label. The processing device according to claim 2 .

4. The confidence of the labeling function is the average of the confidences calculated for the labeling function for each input data set. The processing device according to claim 2 .

5. The classifier calculates, for each of the input data sets, a classification probability vector including probabilities corresponding to each label; The classification uncertainty is positively correlated with the entropy of each classification probability vector. The processing device according to claim 2 .

6. A clustering unit for dividing each input data set of the no-vote input data into a plurality of clusters, The identification unit identifies, as the presentation data, an input data set that belongs to a cluster to which an input data set with the maximum value belongs. The processing device of claim 1 .

7. The computer refers to function data that includes a labeling function that labels the input data set or abstains if it cannot label it, labelling each input data set with said labelling function; The computer learns voted input data including labels assigned to each of the input data sets and input data sets labeled with a threshold or more of the labeling functions from each of the input data sets, learns a model that outputs a correct label and a confidence level of each labeling function from the input data set, inputs each of the input data sets into the trained model, and outputs an uncertainty of classification in the model; The computer includes a step of identifying, as the presented data, a data set that maximizes a value of an acquisition function that takes into account the uncertainty of the discrimination, from among each data set of no-voting input data including an input data set that is labeled with a number of labeling functions less than a threshold value among the input data sets; The newly created labeling function for the presentation data is inserted into the function data. Processing methods.

8. A processing program for causing a computer to function as the processing device according to any one of claims 1 to 6.

Images