A program, apparatus, and method for labeling target data using a classifier.

Abstract

The present invention provides a device, method, and program for assigning labels to target data using a classifier. [Solution] The label assignment device 1 uses a learning model to estimate, for each training data set which serves as the teacher data, one label to be classified from among multiple labels, a characteristic word that serves as the basis for classifying it as that label, and a confidence level. It also includes a classifier group configuration unit that, for each pair of labels and characteristic words, classifies the label according to its confidence level if the training data contains that characteristic word. A classifier input unit inputs the target data to all classifiers. The device executes a program that causes the computer to function as a label assignment unit, which selects one label from among the labels output by each classifier according to the confidence level for each label and assigns the label to the target data. The label estimation unit uses a learning model to estimate the label, characteristic word, and confidence level for each training data set in zero shots. The learning model may also be a large-scale language model.

Description

Technical Field

[0001] The present invention relates to a technique for attaching labels to target data.

Background Art

[0002] A machine learning engine needs to pre-attach labels to a large amount of training data as teacher data. Conventionally, there is a technique for attaching labels to target data using large language models (LLMs) (see, for example, Non-Patent Document 1). According to this technique, accuracy exceeding the manual labeling performance of crowdworkers has been obtained. That is, for the labeling of teacher data, it can be replaced with a large language model without incurring the labor cost of hiring people. Typical large language models include ChatGPT (registered trademark) and Gemini (registered trademark). These are constructed by training a deep neural network having billions of parameters using a large-scale dataset. In particular, Transformer has improved the inference accuracy for tasks for understanding text, images, audio, and videos, and has been utilized in a wide range of applications such as chatbots, search engines, and code generation.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when used zero-shot without fine-tuning a large language model, appropriate labels often cannot be assigned. Suppose the following prompt is sent to a large language model to assign a label to a target text such as an article as the target data. -------------------------------------------------------------------------------- [Prompt]-> Please estimate one label to be classified from the following multiple labels for the target data. Labels: Business / World / Sports / Technology [Target Data]-> Judo played around the world, a martial art rooted in Japan, is celebrated for its focus on technique, balance, and mental discipline. Competitors aim to master throws and holds, showcasing both physical skill and deep respect for tradition. <-[Response 1] Label: World --------------------------------------------------------------------------------

[0005] In this case, the label "World" might be assigned based on the word "world" contained in the target data. However, the article in this target data is actually about Judo, and it would be preferable to assign the label "Sports." This illustrates how being influenced by the impression of a particular word can lead to assigning the wrong label.

[0006] Furthermore, when using a general large-scale language model, labels are assigned by internal parameters, making it unclear what rules govern the labeling process. Therefore, it is crucial to clarify the labeling rules as much as possible. On the other hand, when using a rule-based classifier to assign labels, the configuration of the classifier ultimately has to be done manually.

[0007] Therefore, the present invention aims to provide a program, apparatus, and method for assigning labels to target data using a classifier. [Means for solving the problem]

[0008] According to the present invention, a program that causes a computer to function in order to assign labels to multiple target data, A label estimation method that uses a learning model to estimate, for each training data set which serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it as that label, and the confidence level. A means for constructing a group of classifiers that, for each pair of label and feature word, classifies the label according to its confidence level if the training data contains the feature word, and A classifier input means for inputting the target data into all classifiers, A labeling means that, for the target data, selects one label based on the confidence level of each label output by each classifier and assigns that label. It is characterized by enabling the computer to function in this way.

[0009] According to other embodiments of the program of the present invention, Feature words are obtained by converting features that can be extracted from the training data or target data into words. It is also preferable to make the computer function in this way.

[0010] According to other embodiments of the program of the present invention, The label estimation method uses a learning model to estimate the label, feature word, and confidence level for each training data point in zero shots. It is also preferable to make the computer function in this way.

[0011] According to other embodiments of the program of the present invention, The learning model is a large-scale language model, The label estimation method is, The large-scale language model is instructed to classify each training data point into one of several labels, and is also prompted to estimate the feature words and confidence levels that formed the basis for that classification. Receive labels, feature words, and confidence scores from a large-scale language model. It is also preferable to make the computer function in this way.

[0012] According to other embodiments of the program of the present invention, The learning models are several different types of large-scale language models. The label estimation means sends a prompt to each large-scale language model for each training data set and receives labels, feature words, and confidence scores from each large-scale language model. It is also preferable to make the computer function in this way.

[0013] According to another embodiment of the program of the present invention, It is also preferable that the classifier group configuring means configures the computer to function such that the classifier is configured as a weak monitor.

[0014] According to another embodiment of the program of the present invention, When the classifier group configuring means estimates with the same feature word and the same label but different confidence levels, it calculates the average value or the median value from the different confidence levels, and configures one classifier with the average value or the median value as the confidence level. It is also preferable that the computer functions in this way.

[0015] According to another embodiment of the program of the present invention, It is also preferable that the labeling means adds the confidence levels output from each classifier for each label and selects the label with the largest added confidence level.

[0016] According to another embodiment of the program of the present invention, According to the output from a plurality of classifiers for the training data of the teacher data, for each feature word, calculates the entropy for the confidence level of each label, and for each classifier, outputs the entropy corresponding to the feature word as a reliability coefficient. And further functions as, The labeling means multiplies the confidence level output from the classifier by the reliability coefficient corresponding to the feature word, adds them, and selects the label with the largest added confidence level. It is also preferable that the computer functions in this way.

[0017] According to another embodiment of the program of the present invention, It is also preferable that the classifier group configuring means is configured only with classifiers whose confidence levels are equal to or higher than a predetermined threshold among a plurality of classifiers having the same feature word and different labels.

[0018] ​​​According to the present invention, a device for assigning labels to multiple target data, A label estimation method that uses a learning model to estimate, for each training data set which serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it as that label, and the confidence level. A means for constructing a group of classifiers that, for each pair of label and feature word, classifies the label according to its confidence level if the training data contains the feature word, and A classifier input means for inputting the target data into all classifiers, A labeling means that, for the target data, selects one label based on the confidence level of each label output by each classifier and assigns that label. It is characterized by having the following features.

[0019] According to the present invention, a method for labeling a device that labels multiple target data, The device is The first step involves using a learning model to estimate, for each training data set that serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it to that label, and the confidence level. The second step involves constructing a classifier that, for each pair of label and feature word, classifies the label by its confidence level if the training data contains the feature word. The third step is to input the target data into all classifiers, The fourth step involves selecting one label from the target data based on the confidence level of each label output by each classifier, and assigning that label. It is characterized by having the following features. [Effects of the Invention]

[0020] According to the program, apparatus, and method of the present invention, a classifier can be used to assign labels to target data. [Brief explanation of the drawing]

[0021] [Figure 1]This is a functional configuration diagram of the labeling device according to the present invention. [Figure 2] This is an explanatory diagram of the basic label estimation unit in the present invention. [Figure 3] This is an explanatory diagram of the label estimation unit using a large-scale language model in the present invention. [Figure 4] This is an explanatory diagram showing the output data of the label estimation unit in the present invention. [Figure 5] This is an explanatory diagram showing the multiple classifiers that make up the classifier group component in the present invention. [Figure 6] This is an explanatory diagram of a labeling unit that assigns labels using multiple classifiers according to the present invention. [Figure 7] This is a functional configuration diagram of a labeling device further comprising a confidence coefficient calculation unit. [Figure 8] This is an explanatory diagram of the reliability coefficient calculation unit in the present invention. [Figure 9] This is an explanatory diagram of the labeling unit that assigns labels using the confidence coefficient calculation unit. [Modes for carrying out the invention]

[0022] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0023] Figure 1 is a functional configuration diagram of the labeling device according to the present invention.

[0024] As shown in Figure 1, the labeling device 1 can automatically assign highly accurate labels to target data using a classifier. In other words, the group of classifiers for this purpose can be constructed using a learning model. Of course, the labeling device 1 may be a user-operable terminal or a server connected to a terminal.

[0025] The labeling device 1 includes a label estimation unit 11 that uses a learning model, a classifier group configuration unit 12, for example, three classifiers 121-123, a classifier input unit 13, and a labeling unit 14. These functional components are realized by executing a program that makes the computer installed in the device function. Furthermore, the processing flow of these functional components can also be understood as the labeling method of the device.

[0026] [Label estimation unit 11] The label estimation unit 11 uses a learning model to estimate, for each training data set which serves as the training data, one "label" to be classified from among multiple labels, the "feature word" that served as the basis for classifying it as that label, and the "confidence level" for classifying it as that label. The "learning model" here is one that has been pre-trained by inputting a large amount of training data that serves as the teacher data. Specifically, it is modeled using a neural network and the occurrence probability of "feature words," which are words that represent features. By sending a prompt to the learning model to assign labels to the target data, it is possible to receive a response with the appropriate label. Furthermore, a feature word is a word obtained by converting features that can be extracted from the training data or target data into a word. In other words, according to the present invention, the training data or target data is not limited to text, but may also be multimedia such as numerical sequences, images, videos, and audio, but the focus is on those features that can be converted into words.

[0027] Figure 2 is an explanatory diagram of the basic label estimation unit in the present invention.

[0028] The label estimation unit 11 uses a learning model to estimate the "label" and "feature word and confidence level" for each training data point in zero shots.

[0029] The label estimation unit 11 outputs a response sentence when each training data is input, for example, following the following prompt. Here, the training data is assumed to be a sentence. -------------------------------------------------------------------------------- [Prompt]-> For each sentence, please estimate one label from the following multiple labels, the characteristic words that support that classification, and your confidence level (0-1.0). Labels: Business / World / Sports / Technology -------------------------------------------------------------------------------- [Training Data 1]-> Judo, first introduced to the Olympic program in 1964, continues to showcase its global influence and cultural heritage. The sport's inclusion in the Games offers judoka from diverse nations a prestigious platform to display their skills and embody the principles of respect and discipline inherent in judo. <-[Response 1] Label: Sports Key word: Judo, Confidence level: 0.90 Key word: Olympic, Confidence level: 0.85 -------------------------------------------------------------------------------- [Training Data 2]-> The Olympic Games often drive significant urban development in host cities, such as infrastructure improvements and economic boosts. However, the long-term impact includes challenges like maintaining facilities and managing financial burdens after the event. <-[Response sentence 2] Label:World Key word: Olympic, Confidence level: 0.60 -------------------------------------------------------------------------------- [Training Data 3]-> Judo, a martial art rooted in Japan, is celebrated for its focus on technique, balance, and mental discipline. Competitors aim to master throws and holds, showcasing both physical skill and deep respect for tradition. <-[Response sentence 3] Label: Sports Key word: Judo, Confidence level: 0.70 -------------------------------------------------------------------------------- [Training Data 4]-> ············· <-[Response sentence 4] Label: Business Key word: Olympic, Confidence level: 0.20 --------------------------------------------------------------------------------

[0030] Figure 3 is an explanatory diagram of the label estimation unit using a large-scale language model in the present invention.

[0031] As shown in Figure 3, the label estimation unit 11 utilizes an external large-scale language model 2 connected to the network as its learning model. Here, the large-scale language model 2 is also estimated from unlabeled text in a zero-shot manner without fine-tuning. The large-scale language model 2 can be a general-purpose model such as ChatGPT® or Gemini®, and is not limited to any specific model.

[0032] The label estimation unit 11 sends a prompt to the large-scale language model 2 instructing it to classify each training data, which serves as the training data, into one of several labels. In response, the large-scale language model 2 sends back the label assignment device 1 with the label as a response sentence. Next, the label estimation unit 11 sends a prompt instructing it to estimate the "feature words and confidence levels" that formed the basis for classifying the label. In response, the large-scale language model 2 sends back the "feature words and confidence levels" as a response to the label assignment device 1.

[0033] The label estimation unit 11 outputs a response sentence when it receives a prompt and training data, such as the following. Here, the training data is explained as being in the form of a sentence. -------------------------------------------------------------------------------- [Prompt 11]-> For the text, please determine the single label that best fits the following list of labels. Labels: Business / World / Sports / Technology [Training Data 1]-> Judo, first introduced to the Olympic program in 1964, continues to showcase its global influence and cultural heritage. The sport's inclusion in the Games offers judoka from diverse nations a prestigious platform to display their skills and embody the principles of respect and discipline inherent in judo. <-[Response sentence 11] Label: Sports [Prompt 12]-> This text is labeled "Sports." Estimate the key words and confidence level (0-1.0) that led to this label classification. <-[Response sentence 12] Key word: Judo, Confidence level: 0.90 Key word: Olympic, Confidence level: 0.85 --------------------------------------------------------------------------------

[0034] -------------------------------------------------------------------------------- [Prompt 21]-> For the text, please determine the single label that best fits the following list of labels. Labels: Business / World / Sports / Technology [Training Data 2]-> The Olympic Games often drive significant urban development in host cities, such as infrastructure improvements and economic boosts. However, the long-term impact includes challenges like maintaining facilities and managing financial burdens after the event. <-[Response sentence 21] Label: World [Prompt 22]-> This text has been labeled "World". Please estimate the characteristic words and their confidence level (0-1.0) that led to this label classification. <-[Response sentence 22] Key word: Olympic, Confidence level: 0.60 --------------------------------------------------------------------------------

[0035] -------------------------------------------------------------------------------- [Prompt 31]-> For the text, please determine the single label that best fits the following list of labels. Labels: Business / World / Sports / Technology [Training Data 3]-> Judo, a martial art rooted in Japan, is celebrated for its focus on technique, balance, and mental discipline. Competitors aim to master throws and holds, showcasing both physical skill and deep respect for tradition. <-[Response sentence 31] Label: Sports [Prompt 32]-> This text is labeled "Sports." Estimate the key words and confidence level (0-1.0) that led to this label classification. <-[Response sentence 32] Key word: Judo, Confidence level: 0.70 --------------------------------------------------------------------------------

[0036] In another embodiment, the learning model may be multiple different types of large-scale language models. The label estimation unit 11 sends the aforementioned prompt to each large-scale language model for each training data set and receives "labels" and "feature words and confidence levels" from each large-scale language model. The label estimation unit 11 comprehensively combines this data estimated by each large-scale language model and outputs it to the classifier group configuration unit 12.

[0037] Figure 4 is an explanatory diagram showing the output data of the label estimation unit in the present invention.

[0038] As shown in Figure 4, the label estimation unit 11 outputs the estimated "label" and "feature word and confidence level" for each training data to the classifier group configuration unit 12. Text 1 = Label: Sports Feature word: Judo -> Confidence level: 0.90, Feature word: Olympic -> Confidence level: 0.85 Text 2 = Label: World Key word: Olympic -> Confidence level: 0.60 Text 3 = Label: Sports Key word: Judo -> Confidence level: 0.70 Text 4 = Label Business Key word: Olympic -> Confidence level: 0.20

[0039] [Classifier group configuration unit 12] The classifier group component 12 constructs, for example, four classifiers 121 to 124 using "labels" and "feature words and confidence levels" estimated in zero shots by the learning model from the training data of the teacher data. The classifier group component 12 configures a classifier for each pair of "label" and "feature word" that "classifies the label with its confidence level if the training data contains that feature word." In this case, the classifier group component 12, when estimating different training data with the "same feature word" and "same label" but with different confidence levels, calculates the mean or median from the different confidence levels and constructs a single classifier using that mean or median as the "confidence level".

[0040] [Classifier 121~124] Classifiers 121-124 may also be labeling functions (weakly supervision) that assign labels to text according to rules (see, for example, Non-Patent Document 2). For example, a rule might be that if the target data contains the "feature word", it outputs a default "label" and "confidence level". The classifier is heuristic; if a characteristic word is included in the rule, it outputs a label and confidence level. For example, in the case of spam emails, sentences containing the characteristic words "subscription" and "check it out" will output the default label "spam" with a default confidence level (e.g., 0.8).

[0041] Generally, as the number of labels to be classified increases, it becomes more difficult to set rules. In contrast, according to the present invention, the learning model estimates what kind of classifier group should be constructed based on the training data of the teacher data. Therefore, manual intervention is not required to construct the multiple classifiers for which rules are set.

[0042] Figure 5 is an explanatory diagram showing the multiple classifiers that make up the classifier group component in the present invention.

[0043] <Classifier 121 = Pair of "Label: Sports" and "Feature word: Judo"> Both training data 1 and 3 contain the feature word "Judo" and are therefore classified under the label "Sports," but training data 1 has a confidence level of 0.90, while training data 3 has a confidence level of 0.70. In this case, the mean (or median) is calculated from the different confidence levels, and a single classifier is constructed using this mean (or median) as the confidence level. For "Label: Sports" and "Keyword: Judo," the confidence levels are 0.90 and 0.70. Average confidence score: (0.90 + 0.70) / 2 = 0.80 Classifier 121 outputs "Label: Sports" and "Confidence: 0.80" if the target data includes "Judo" as a rule.

[0044] <Classifier 122 = Pair of "Label: Sports" and "Feature word: Olympic"> Training data 1 contains the feature word "Olympic," so it is classified as "Sports" with a confidence level of 0.85. Classifier 122 outputs "Label: Sports" and "Confidence: 0.85" if the target data includes the Olympics, based on its rules.

[0045] <Classifier 123 = Pair of "Label: World" and "Feature word: Olympic"> Training data 2 contains the feature word "Olympic," so it is classified as "World" with a confidence level of 0.60. Classifier 123 outputs "Label: World" and "Confidence: 0.70" if the target data includes the Olympics, based on its rules.

[0046] <Classifier 124 = Pair of "Label: Business" and "Feature word: Olympic"> Training data 4 contains the feature word "Olympic," so it is classified as "Label: Business" with a "Confidence level: 0.20." Classifier 124 outputs "Label: Business" and "Confidence Score: 0.20" if the target data includes "Olympic" as a rule.

[0047] In another embodiment, the classifier group component 12 may consist only of classifiers whose confidence level is above a predetermined threshold among multiple classifiers for the same feature word and different labels. That is, classifiers with low confidence levels are removed. According to Figure 5, classifier 124 has a confidence score of 0.20. If the predetermined threshold is set to 0.50, the confidence score of classifier 124, which is 0.20, becomes less than or equal to the predetermined threshold of 0.50, and classifier 124 is deleted.

[0048] Figure 6 is an explanatory diagram of the labeling unit in the present invention, which assigns labels using multiple classifiers.

[0049] [Classifier input unit 13] The classifier input unit 13 inputs the target data to all classifiers 121 to 123.

[0050] [Classifier 121~123] Each of the classifiers 121 to 123 outputs a "label" and "confidence level" to the labeling unit 14 when it discovers a characteristic word set for itself in the target data.

[0051] [Labeling unit 14] The labeling unit 14 selects one label from the list of labels output by each classifier 121 to 123 for the target data and assigns that label. Here, the labeling unit 14 adds up the confidence scores output from each classifier 121 to 123 for each label and selects the label with the highest added confidence score.

[0052] According to Figure 6, for the target data, labels and confidence scores are output from classifiers 121 to 123. Classifier 121 -> Label: Sports, Confidence: 0.8 Classifier 122 -> Label: Sports, Confidence: 0.85 Classifier 123 -> Label: World, Confidence: 0.60 Here, we add up the confidence level for each label. Label Sports: Confidence level 0.8 + 0.85 = 1.65 Label World: Confidence level 0.60 Then, the added confidence scores for multiple labels are compared. Label Sports: Confidence level: 1.65 > Label World: Confidence level: 0.60 As a result, the label "Sports," which has the highest added confidence level, is selected and assigned to the target data.

[0053] Furthermore, according to the present invention, as mentioned above, the training data and target data are not limited to text, but may also be numerical sequences, images, videos, audio, etc. For example, it is possible to assign labels for human behavior recognition from a sensor sequence (numerical sequence) detected by one or more sensors. The sensor sequence may be a combination of, for example, a numerical sequence from a living room light fixture, a numerical sequence from a living room motion sensor, and a numerical sequence from a living room television. According to the present invention, multiple classifiers can be automatically constructed using a sensor sequence as training data. These classifiers are rule-based and allow for logical justification of the labels ultimately assigned.

[0054] Figure 7 is a functional configuration diagram of a labeling device that further includes a confidence coefficient calculation unit.

[0055] As shown in Figure 7, during the training phase, a confidence coefficient calculation unit 15 is further provided between each classifier 121-123 and the labeling unit 14. The training data of the teacher data used to construct the group of multiple classifiers is input again from the classifier input unit 13 to each classifier 121-123.

[0056] [Confidence coefficient calculation unit 15] The confidence coefficient calculation unit 15 calculates the entropy for the confidence level of each label for each feature word, based on the output from multiple classifiers to the training data of the training data. Then, for each classifier, it outputs the entropy corresponding to that feature word as a confidence coefficient to the label assignment unit 14.

[0057] A classifier that discovers feature words that have a high confidence level for a particular label is considered to have high confidence in the classified label. On the other hand, a classifier that discovers feature words that have a high confidence level for all labels is considered to have low confidence in the classified label. The confidence level of each classifier is calculated using the entropy of the confidence distribution.

[0058] Specifically, perform the following steps: (S0) The training data of the teacher data is input again to the classifier input unit 13. The classifier input unit 13 inputs the training data to all classifiers 121 to 123. As a result, when multiple classifiers 12 to 123 find a feature word, they output a label and confidence score. For each feature word, the confidence scores output from multiple classifiers 121 to 123 for each label are collected. (S1) First, the confidence distribution X for each label is normalized so that the sum is 1 (normalized distribution normalized(X)). (S2) Next, we calculate the entropy of the normalized distribution X (entropy H(normalized(X))). (S3) Then, the difference between the maximum possible value of entropy and the actual value is taken as the confidence coefficient.

[0059] Figure 8 is an explanatory diagram of a labeling unit that assigns labels using the confidence coefficient calculation unit in the present invention.

[0060] As shown in Figure 8, for each characteristic word, such as Judo, Olympic, and MLB, the confidence level for each label is collected. (About the characteristic word Judo) (S0) Collect the confidence distribution X[0.00, 0.00, 0.80, 0.00] for each label. (S1) Normalization yields the distribution normalized(X)[0, 0, 1, 0]. (S2) The entropy H(normalized(X)) of the distribution is calculated as follows. H(x) = -(0 + 0 + 1 * log2(1) + 0) = 0 (S3) Then, the difference between the maximum possible value of entropy and the actual value is taken as the confidence coefficient. log2(N)-H(x)= 2-0 =2 (confidence coefficient) Similarly, N will be the number of labels (World, Business, Sports, Technology), which is 4. When a feature word like "Judo" does not appear in multiple labels, a high confidence level is output only for the label "Sports." Therefore, the confidence level of the classifier that uses such feature words can be increased.

[0061] (Regarding the characteristic word "Olympic") (S0) Collect the confidence distribution X[0.60, 0.20, 0.85, 0.00] for each label. (S1) Normalization yields the distribution normalized(X) [0.36, 0.12, 0.52, 0.00]. (S2) The entropy H(normalized(X)) of the distribution is calculated as follows. H(x)=-(0.36*log2(0.36)+0.12*log2(0.12)+0.52*log2(0.52)+0)=1.39 (S3) Then, the difference between the maximum possible value of entropy and the actual value is taken as the confidence coefficient. log2(N)-H(x)= 2-1.39 =0.61 (confidence coefficient) When a feature word like "Olympic" appears in a few labels, it doesn't necessarily mean that a high confidence level will be output only for that specific label. Therefore, the confidence level of a classifier that uses such a feature word will be moderate.

[0062] (Regarding the characteristic word MLB) (S0) Collect the confidence distribution X[0.60, 0.90, 0.80, 0.20] for each label. (S1) Normalization yields the distribution normalized(X) [0.44, 0.66, 0.59, 0.15]. (S2) The entropy H(normalized(X)) of the distribution is calculated as follows. H(x)=-(0.44*log2(0.44)+0.66*log2(0.66)+0.59*log2(0.59)+ 0.15 * log2(0.15)) = 1.78 (S3) Then, the difference between the maximum possible value of entropy and the actual value is taken as the confidence coefficient. log2(N)-H(x)= 2-1.78 =0.22 (confidence coefficient) When a feature word like "MLB" appears frequently across multiple labels, it will output a high confidence level regardless of the label (World, Business, Sports, Technology). Therefore, the confidence level of classifiers that use such feature words can be reduced.

[0063] Figure 9 is an explanatory diagram of the labeling unit, which assigns labels using the confidence coefficient calculation unit.

[0064] The target data is input from the classifier input unit 13 to classifiers 121-123. Each of the classifiers 121-123 then outputs a "label" and "confidence level" to the labeling unit 14. Then, the labeling unit 14 multiplies the confidence coefficient for each classifier by the confidence level output from the classifiers 121 to 123 to obtain an updated confidence level.

[0065] According to Figure 9, the labeling unit 14 receives labels and confidence levels from classifiers 121 to 123. Classifier 121 discovers the feature word "Judo" from the target data and outputs the label "Sports" with a confidence level of 0.80. For the feature word "Judo," the aforementioned confidence coefficient of 2 was obtained. Classifier 121 -> Label: Sports, Confidence level 0.80 × Confidence coefficient 2 = Updated confidence level 1.60 Classifier 122 discovers the feature word "Olympic" from the target data and outputs the label "Sports" with a confidence level of 0.85. For the feature word "Olympic," the aforementioned confidence coefficient of 0.61 was obtained. Classifier 122 -> Label: Sports, Confidence level 0.85 × Confidence coefficient 0.61 = Updated confidence level 0.51 Classifier 123 discovers the feature word "Olympic" from the target data and outputs the label "World" with a confidence level of 0.60. For the feature word "Olympic," the aforementioned confidence coefficient of 0.61 was obtained. Classifier 123 -> Label: World, Confidence level 0.60 × Confidence coefficient 0.61 = Updated confidence level 0.37

[0066] Next, the labeling unit 14 adds the updated confidence score for each label. Label Sports: Updated confidence 1.6 + 0.51 = 2.11 Label World: Updated confidence level 0.37 The labeling unit 14 then compares the summed confidence levels for multiple labels. Label Sports: Confidence level 2.11 > Label World: Confidence level 0.37 As a result, the label with the highest added confidence level, "Label:Sports," is selected and assigned to the target data.

[0067] As described in detail above, according to the program, apparatus, and method of the present invention, a classifier can be used to assign labels to target data. According to the present invention, multiple classifiers, each with its own set of rules, can be automatically configured using a learning model. These classifiers can then be used to estimate the labels that should be assigned to the target data. In particular, the learning model can be used in a zero-shot configuration without requiring fine-tuning. Even when a classifier is configured using a zero-shot learning model, the accuracy of label assignment can be improved.

[0068] One application example of the present invention is its use in generating responses in customer service. For example, it can be used to assign appropriate labels to customer questions and business responses. Such label assignment by a classifier is particularly suitable for creating training data for teacher data. Furthermore, by assigning labels using a rule-based classifier, it becomes clear what rules were used to assign those labels. This type of work does not require human intervention such as crowdsourcing, nor does it require fine-tuning of the learning model.

[0069] Furthermore, this will enable, for example, the assignment of rule-based labels to target data with high accuracy, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs): "Build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0070] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The foregoing description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]

[0071] 1. Labeling device 11 Label Estimation Unit 12 Classifier group component 121, 122, 123, 124 classifier 13 Classifier Input Section 14 Labeling Unit 15. Confidence Coefficient Calculation Unit 2. Large-scale language models

Claims

1. A program that causes a computer to function in order to assign labels to multiple target data, A label estimation means that uses a learning model to estimate, for each training data set which serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it as said label, and the confidence level. A classifier group configuration means that constitutes a classifier that, for each pair of label and feature word, classifies the label by the confidence level if the training data contains the feature word, A classifier input means for inputting the target data into all classifiers, A labeling means that selects one label from the target data according to the confidence level of each label output from each classifier and assigns the label. A program characterized by its ability to make a computer function.

2. Feature words are obtained by converting features that can be extracted from the training data or target data into words. The program according to claim 1, characterized in that it causes the computer to function in such a way.

3. The label estimation method uses a learning model to estimate the label, feature word, and confidence level for each training data point in zero shots. The program according to claim 1, characterized in that it causes the computer to function in such a way.

4. The learning model is a large-scale language model, The label estimation method is, The large-scale language model is instructed to classify each training data point into one of several labels, and is also instructed to send prompts to estimate the characteristic words and confidence levels that formed the basis for that classification. Receive labels, feature words, and confidence scores from a large-scale language model. The program according to claim 3, characterized in that it causes the computer to function in such a way.

5. The aforementioned learning model is a set of multiple different types of large-scale language models. The label estimation means sends the prompt to each large-scale language model for each training data set and receives the label, feature word, and confidence score from each large-scale language model. The program according to claim 4, characterized in that it causes the computer to function in such a way.

6. The classifier group configuration means configures the classifier as a weak monitor. The program according to claim 1, characterized in that it causes the computer to function in such a way.

7. The classifier group configuration means, when the same feature word and the same label are estimated with different confidence levels, calculates the mean or median from the different confidence levels and configures a single classifier with the mean or median as its confidence level. The program according to claim 1, characterized in that it causes the computer to function in such a way.

8. The labeling method adds up the confidence scores output from each classifier for each label and selects the label with the highest added confidence score. The program according to claim 1, characterized in that it causes the computer to function in such a way.

9. A confidence coefficient calculation means calculates the entropy of the confidence level of each label for each feature word in accordance with the output from multiple classifiers to the training data of the training data, and outputs the entropy corresponding to the feature word as a confidence coefficient for each classifier. And make it work even better. The labeling means multiplies the confidence level output from the classifier by the confidence coefficient corresponding to the feature word, and selects the label with the highest confidence level obtained by adding these coefficients together. The program according to claim 8, characterized in that it causes the computer to function in such a way.

10. The classifier group configuration means consists only of classifiers whose confidence level is above a predetermined threshold, among multiple classifiers for the same feature word and different labels. The program according to claim 1, characterized in that it causes the computer to function in such a way.

11. A device for assigning labels to multiple target data, A label estimation means that uses a learning model to estimate, for each training data set which serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it as said label, and the confidence level. A classifier group configuration means that constitutes a classifier that, for each pair of label and feature word, classifies the label by the confidence level if the training data contains the feature word, A classifier input means for inputting the target data into all classifiers, A labeling means that selects one label from the target data according to the confidence level of each label output from each classifier and assigns the label. An apparatus characterized by having the following features.

12. A method for labeling a device that labels multiple target data, The aforementioned device is The first step involves using a learning model to estimate, for each training data set which serves as the training data, one label to be classified from among multiple labels, the characteristic word that served as the basis for classifying it to that label, and the confidence level. A second step involves constructing a classifier that, for each pair of label and feature word, classifies the label by the confidence level if the training data contains the feature word. The third step is to input the target data into all classifiers, A fourth step involves selecting one label from the target data according to the confidence level of each label output from each classifier, and assigning that label. A method for labeling an apparatus, characterized by having the following features.

Images