Training data generating program, machine learning program, estimation program, method, and device

Abstract

An information processing device according to the present invention extracts, from combinations of words included in each of a plurality of sentences to which a teacher label is assigned, a combination of words in which an index value indicating a relation between the combination of words and the teacher label satisfies a predetermined condition, determines whether or not the extracted combination of words is included in the same context in each of the plurality of sentences, and generates training data associated with the teacher label by using, as a feature, presence / absence of the combination of words determined to be included in the same context, in each of the plurality of sentences.

Description

Training data generation program, machine learning program, estimation program, method, and device

[0001] The disclosed technology relates to a training data generation program, a training data generation method, a training data generation device, a machine learning program, a machine learning method, a machine learning device, an estimation program, an estimation method, and an estimation device.

[0002] Conventionally, for example, quality assurance personnel would review various numerical data related to a project and documents showing progress, predict or determine whether the project's profitability is declining, and correct the project's progress. There are also systems that use the presence or absence of specific words in project-related documents as features of the documents and predict declining profitability using a machine learning model. In this case, there are also systems that use explainable AI (artificial intelligence) as the machine learning model to analyze the factors behind declining project profitability. The specific words can be specified in advance or automatically acquired.

[0003] As a machine learning technology that uses words contained in a sentence as features, for example, a word semantic relationship extraction device has been proposed that extracts word semantic relationships from text data by performing multi-class supervised learning using an existing thesaurus as a ground truth. This device calculates multiple types of similarity for any pair of words in the text and generates a feature vector with each similarity as an element. The device also assigns a label to each word pair indicating the type of word semantic relationship based on the thesaurus. The device then learns word semantic relationship identification data from the feature vectors and labels as a multi-class classification problem, and uses the word semantic relationship identification data to identify the semantic relationship between two words.

[0004] Furthermore, for example, a word meaning assignment device has been proposed that extracts features for combinations of words contained in an input sentence written in a natural language and determines a higher semantic class that indicates a higher concept of the word based on the features.

[0005] Also proposed is a method for receiving, at a server having one or more processors, labeled training data including multiple groups of words, where each group of words has a predicate, each word having a general word embedding, and extracting, at the server, multiple groups of words within the syntactic context of the predicates. The method also includes concatenating, at the server, the general word embeddings to create a high-dimensional vector space representing the features of each word. The method also includes obtaining, at the server, a model having a learned mapping from the high-dimensional vector space to a low-dimensional vector space and a learned embedding of each possible semantic frame in the low-dimensional vector space. The method then outputs, at the server, the model for storage, where the model is configured to identify a specific semantic frame for the input.

[0006] For example, methods have been proposed for automatic text analysis and ranking of contexts containing words appearing in a corpus, which rank contexts locally for each word by frequency of occurrence, rank contexts globally, and calculate statistics such as log-rank ratio statistics to obtain the similarity of the words in a context, and then order the contexts based on these.

[0007] International Publication No. 2014 / 033799 Japanese Patent Application Publication No. 2009-181408 U.S. Patent Application Publication No. 2016 / 0239739 U.S. Patent Application Publication No. 2005 / 0049867

[0008] However, simply using each word as a feature, such as the presence or absence of a specific word in a sentence, often does not improve the accuracy of estimation by a machine learning model or results in estimation results with low interpretability. This is because, for example, even if "delay" is specified as a specific word indicating a deterioration in profitability, it is difficult to determine whether the "delay" has "occurred" or "resolved."

[0009] Therefore, it is possible to use the feature of whether multiple word combinations appear simultaneously in a sentence. However, word combinations extracted from unrelated sentences do not contribute to improving the estimation accuracy or interpretability of the machine learning model, so it is desirable to exclude them from the features used in the training data. In other words, it is desirable to use word combinations that appear in the same context as features.

[0010] However, it is not easy to determine whether a combination of words is in the same context. For example, if a sentence is divided into sentences using periods and a combination of words appears within a sentence, it may be determined that the combination of words is in the same context. However, this determination method is not sufficient because each word in a combination of words may be used in the same context even if it appears across multiple sentences. Furthermore, determining whether all combinations of words are in the same context is not realistic in terms of processing time.

[0011] In one aspect, the disclosed technology aims to efficiently generate training data that can improve the estimation accuracy and explainability of machine learning models.

[0012] In one aspect, the disclosed technology extracts word combinations from word combinations included in each of a plurality of sentences to which a teacher label has been assigned, the word combinations having an index value indicating a relevance between the word combination and the teacher label that satisfies a predetermined condition. The disclosed technology also determines whether the extracted word combinations are included in the same context in each of the plurality of sentences. The disclosed technology then generates training data in which the presence or absence of the word combinations determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label.

[0013] One aspect is that it has the effect of efficiently generating training data that can improve the estimation accuracy and explainability of a machine learning model.

[0014] 1 is a functional block diagram of an information processing device according to an embodiment of the present invention. FIG. 1 is a schematic diagram showing an example of a set of sentences with teacher labels. FIG. 2 is a diagram showing an example of a vector of word appearance / non-appearance. FIG. 3 is a diagram for explaining index values ​​and priorities of word combinations. FIG. 4 is a diagram showing an example of context determination. FIG. 5 is a diagram showing an example of context determination. FIG. 6 is a diagram showing an example of a vector of word appearance / non-appearance. FIG. 7 is a block diagram showing a schematic configuration of a computer functioning as an information processing device. FIG. 8 is a flowchart showing an example of training data generation processing. FIG. 9 is a flowchart showing an example of machine learning processing. FIG. 10 is a flowchart showing an example of estimation processing. FIG. 11 is a diagram for explaining the difference between the index value of a word combination and the index value of each word.

[0015] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In this embodiment, a case will be described in which a task is executed to determine whether the profitability of a project is deteriorating based on a document related to the project.

[0016] 1 , an information processing device 10 according to this embodiment functionally includes a training data generation unit 12, a machine learning unit 22, and an estimation unit 28. A training data DB (Database) 20 and a machine learning model 26 are stored in a predetermined storage area of ​​the information processing device 10. The training data generation unit 12 is an example of a training data generation device of the present disclosure, the machine learning unit 22 is an example of a machine learning device of the present disclosure, and the estimation unit 28 is an example of an estimation device of the present disclosure.

[0017] First, we will explain the training data generation unit 12. The training data generation unit 12 further includes a first extraction unit 14, a first determination unit 16, and a first generation unit 18.

[0018] The first extraction unit 14 acquires a set of labeled sentences input to the information processing device 10. As shown in FIG. 2, the set of labeled sentences includes a plurality of labeled sentences. Each labeled sentence is text data in which a sentence indicating the progress of each project is written in natural language, and is assigned a labeled sentence indicating whether the project is deteriorating in profitability. In FIG. 2, "Deteriorating profitability: 1" represents a labeled sentence indicating that profitability is deteriorating, and "Deteriorating profitability: 0" represents a labeled sentence indicating that profitability is not deteriorating. Each labeled sentence is also assigned a project number (hereinafter also referred to as "PJ number"), which is identification information for each project.

[0019] The first extraction unit 14 uses a morphological analysis system such as MeCab to perform morphological analysis on each teacher-labeled sentence and extract each word from the teacher-labeled sentence. As shown in FIG. 3 , the first extraction unit 14 generates, for each teacher-labeled sentence, a vector including an element indicating whether the extracted word appears (non-appearance) in the teacher-labeled sentence and the teacher label. In the example of FIG. 3 , each row corresponds to a vector generated from the teacher-labeled sentences of the project indicated by the project number in the “PJ Number” column. Words such as “development,” “delay,” “bug,” “completion,” and “fix” are extracted from the entire set of teacher-labeled sentences. If a word appears in a teacher-labeled sentence, the value of the element for the corresponding word in the vector for that teacher-labeled sentence is set to “1.” If the word does not appear, the value of the element is set to “0.” The “deteriorating profitability” column is a teacher label.

[0020] Based on the extracted words, the first extraction unit 14 comprehensively calculates an index value indicating the association between the word combination and the truth label for word combinations containing one or more words. The index value indicates the significance of the difference in the occurrence rate of the word combination between positive examples and negative examples of the truth label. For example, the index value includes at least one of a chi-square value, normalized mutual information, weight, confidence, and support. The weight indicates the importance of each word combination for classification into positive examples and negative examples, and is a value calculated so that the sum of the weights for all word combinations is 1.

[0021] The upper diagram of Figure 4 shows an example of index values ​​calculated for each word combination. In Figure 4, "weight" is weight, "supp" is support, "conf" is confidence, "chi2" is chi-squared value, and "nmi" is "normalized mutual information." In addition, in the example of the upper diagram of Figure 4, "npos" which is the number of positive examples and "nneg" which is the number of negative examples are also included as index values. In addition, while the example of Figure 4 shows a combination of two words, the word combination may be one word or three or more words.

[0022] The first extraction unit 14 extracts word combinations whose calculated index values ​​satisfy a predetermined condition. For example, the first extraction unit 14 extracts a predetermined number of word combinations in descending order of a specific index value or an integrated index value obtained by integrating multiple index values, or word combinations whose index value or integrated index value is equal to or greater than a predetermined value. The specific index value may be selected as an index value suitable for indicating the significance of the difference in the occurrence rate of word combinations between positive and negative examples of teacher labels. For example, the specific index value may be a chi-square value or normalized mutual information, or may be a weight or reliability. The integrated index value may be the product or weighted sum of each index value, the average of each normalized index value, or the like. The number of word combinations to be extracted may be preset based on the execution time of a determination by the first determination unit 16 (described later) of whether word combinations are included in the same context (hereinafter also referred to as "same context determination").

[0023] 4, the first extraction unit 14 prioritizes the word combinations extracted based on the index values ​​in descending order of index value or integrated index value, and passes them to the first determination unit 16. The priority here refers to the priority when the first determination unit 16 performs context determination on the combinations.

[0024] The first determination unit 16 performs context determination on the word combination passed from the first extraction unit 14. The context determination may be performed based on, for example, the distance between the appearance positions of each word included in the word combination in a sentence, but in this embodiment, the context determination is performed using large language models (LLMs).

[0025] Specifically, the first determination unit 16 selects a word combination in descending order of priority from the word combinations passed from the first extraction unit 14. The first determination unit 16 inputs to the LLM a teacher-labeled sentence containing the selected word combination and a question asking whether the selected word combination is used in the same context within the teacher-labeled sentence. The first determination unit 16 acquires the answer output from the LLM as the determination result of the same-context determination. The first determination unit 16 performs same-context determination on all teacher-labeled sentences containing the selected word combination, and also performs same-context determination on all word combinations passed from the first extraction unit 14.

[0026] 5 to 7 show examples of sentences and questions input to an LLM and responses from the LLM when a Generative Pre-trained Transformer 4 (GPT-4) is used as an example of an LLM. As shown in FIGS. 5 to 7, by using an LLM, it is possible to accurately determine the context of a word combination, regardless of whether the words in the word combination are contained together in a single sentence separated by periods or appear across two or more sentences.

[0027] The first generation unit 18 generates training data in which the presence or absence of word combinations determined to be included in the same context in each of the teacher-labeled sentences is used as a feature quantity and associated with the teacher label.

[0028] For example, as shown in the upper diagram of FIG. 8 , a vector is generated for each teacher-labeled sentence, with elements indicating whether or not a word combination extracted by the first extraction unit 14 appears (non-occurrence) in each teacher-labeled sentence. In the example of the upper diagram of FIG. 8 , each row corresponds to a vector indicating a feature for a project indicated by the project number in the "PJ Number" column. Furthermore, if each word combination appears in the teacher-labeled sentence, the element value is set to "1," and if it does not appear, the element value is set to "0." As shown in the lower diagram of FIG. 8 , the first generation unit 18 changes elements of the vector corresponding to word combinations determined by the first determination unit 16 not to be included in the same context (e.g., elements indicated by bold lines in FIG. 8 ) from 1 to 0. As a result, each vector becomes a feature indicating whether or not each word combination appears in the same context in each teacher-labeled sentence.

[0029] The first generation unit 18 generates training feature data by combining, for example, a vector representing the appearance or non-appearance of a word in a teacher-labeled sentence, as shown in Fig. 3, with a vector representing a combination of words included in the same context, as shown in the lower diagram of Fig. 8. The first generation unit 18 also assigns teacher labels to the training feature data to generate training data. The first generation unit 18 stores the training data generated for each teacher-labeled sentence in the training data DB 20.

[0030] Next, the machine learning unit 22 will be described. The machine learning unit 22 includes a training unit 24.

[0031] The training unit 24 trains the machine learning model 26 using the training data stored in the training data DB 20. When feature data for a sentence to be estimated is input, the machine learning model 26 outputs an estimation result corresponding to the task. The estimation result here is whether or not the profitability of the project corresponding to the sentence to be estimated has deteriorated. Specifically, the training unit 24 updates the parameters of the machine learning model 26 so that the estimation result (label) output when training feature data, which is the portion of the training data excluding the teacher label, matches the teacher label.

[0032] Next, a description will be given of the estimation unit 28. The estimation unit 28 further includes a second extraction unit 30, a second determination unit 32, a second generation unit 34, and an output unit 36.

[0033] The second extraction unit 30 acquires an estimation target sentence input to the information processing device 10. The estimation target sentence is similar to a teacher-labeled sentence, except that no teacher label is assigned to the estimation target sentence. Similar to the first extraction unit 14, the second extraction unit 30 performs morphological analysis of the estimation target sentence using a morphological analysis system to extract each word in the estimation target sentence. The second extraction unit 30 also generates a vector whose elements indicate whether each word included in the training feature data appears in the estimation target sentence (non-appearance). This vector is similar to the vector shown in FIG. 3 , with the teacher label removed.

[0034] The second determination unit 32 determines whether a combination of words included in the training feature data is included in the same context in the sentence to be estimated. The second determination unit 32 may perform the same context determination using LLM, similar to the first determination unit 16.

[0035] Similar to the first generation unit 18, the second generation unit 34 generates estimation feature data by combining vectors regarding the occurrence or non-occurrence of words in the sentence to be estimated with vectors regarding combinations of words included in the same context.

[0036] The output unit 36 ​​inputs the estimation feature data generated by the second generation unit 34 into the machine learning model 26 that has been trained by the training unit 24, obtains the estimation results output from the machine learning model 26, and outputs them.

[0037] The information processing device 10 may be realized by, for example, a computer 40 shown in FIG. 9 . The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 serving as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0038] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 44, which serves as a storage medium, stores a training data generation program 52, a machine learning program 62, and an estimation program 68 for causing the computer 40 to function as the information processing device 10. The training data generation program 52 includes a first extraction process control instruction 54, a first determination process control instruction 56, and a first generation process control instruction 58. The machine learning program 62 includes a training process control instruction 64. The estimation program 68 includes a second extraction process control instruction 70, a second determination process control instruction 72, a second generation process control instruction 74, and an output process control instruction 76. The storage device 44 also includes an information storage area 80 in which information constituting the training data DB 20 and the machine learning model 26 is stored.

[0039] The CPU 41 reads the training data generation program 52 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions included in the training data generation program 52. The CPU 41 operates as the first extraction unit 14 shown in Fig. 1 by executing the first extraction process control instruction 54. The CPU 41 also operates as the first judgment unit 16 shown in Fig. 1 by executing the first determination process control instruction 56. The CPU 41 also operates as the first generation unit 18 shown in Fig. 1 by executing the first generation process control instruction 58.

[0040] The CPU 41 also reads the machine learning program 62 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the machine learning program 62. The CPU 41 operates as the training unit 24 shown in FIG. 1 by executing the training process control instructions 64.

[0041] The CPU 41 also reads out the estimation program 68 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions included in the estimation program 68. The CPU 41 operates as the second extraction unit 30 shown in FIG. 1 by executing a second extraction process control instruction 70. The CPU 41 also operates as the second determination unit 32 shown in FIG. 1 by executing a second determination process control instruction 72. The CPU 41 also operates as the second generation unit 34 shown in FIG. 1 by executing a second generation process control instruction 74. The CPU 41 also operates as the output unit 36 ​​shown in FIG. 1 by executing an output process control instruction 76.

[0042] The CPU 41 also reads information from the information storage area 80 and loads the training data DB 20 and the machine learning model 26 into the memory 43. As a result, the computer 40 that has executed the training data generation program 52, the machine learning program 62, and the estimation program 68 functions as the information processing device 10. The CPU 41 that executes the programs is hardware. Part of the programs may also be executed by the GPU 42.

[0043] The functions realized by each program may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0044] Next, the operation of the information processing device 10 according to this embodiment will be described. When a set of sentences with teacher labels is input to the information processing device 10 and an instruction to generate training data is given, the information processing device 10 executes a training data generation process shown in FIG. 10 . Furthermore, when an instruction to train the machine learning model 26 is given with the generated training data stored in the training data DB 20, the information processing device 10 executes a machine learning process shown in FIG. 11 . When an estimation target sentence is input to the information processing device 10 and an instruction to estimate a deterioration in profitability is given with the trained machine learning model 26 stored in a predetermined storage area of ​​the information processing device 10, the information processing device 10 executes an estimation process shown in FIG. 12 . Note that the training data generation process is an example of a training data generation method of the present disclosure, the machine learning process is an example of a machine learning method of the present disclosure, and the estimation process is an example of an estimation method of the present disclosure.

[0045] First, the training data generation process shown in FIG. 10 will be described.

[0046] In step S10, the first extraction unit 14 acquires a set of teacher-labeled sentences input to the information processing device 10. Next, in step S12, the first extraction unit 14 performs morphological analysis on each teacher-labeled sentence and extracts each word in the teacher-labeled sentence. Next, in step S14, the first extraction unit 14 generates, for each teacher-labeled sentence, a vector including a teacher label and an element indicating whether the extracted word appears (non-appearance) in the teacher-labeled sentence.

[0047] Next, in step S16, the first extraction unit 14 comprehensively calculates, based on the extracted words, index values ​​indicating the relevance between the word combinations and the truth labels for word combinations containing one or more words. Next, in step S18, the first extraction unit 14 extracts a predetermined number of word combinations in descending order of a specific index value or an integrated index value obtained by integrating multiple index values, or word combinations whose index value or integrated index value is equal to or greater than a predetermined value. Then, the first extraction unit 14 prioritizes the extracted word combinations in descending order of index value or integrated index value, and passes them to the first determination unit 16.

[0048] Next, in step S20, the first determination unit 16 performs same-context determination using LLM for the word combinations passed from the first extraction unit 14. Next, in step S22, the first generation unit 18 generates vectors indicating the presence or absence of word combinations determined to be included in the same context in each of the teacher-labeled sentences. The first generation unit 18 also generates training feature data by combining vectors indicating the appearance or non-appearance of words in the teacher-labeled sentences with vectors indicating word combinations included in the same context. The first generation unit 18 then assigns teacher labels to the training feature data to generate training data, stores the training data in the training data DB 20, and the training data generation process ends.

[0049] Next, the machine learning process shown in FIG. 11 will be described.

[0050] In step S30, the training unit 24 retrieves training data from the training data DB 20. Next, in step S32, the training unit 24 uses the retrieved training data to train the machine learning model 26 that outputs an estimation result according to the task when feature data for a sentence to be estimated is input. Next, in step S34, the training unit 24 stores the trained machine learning model 26 in a predetermined storage area, and the machine learning process ends.

[0051] Next, the estimation process shown in Fig. 12 will be described. Note that detailed description of processes similar to the training data generation process will be omitted.

[0052] In step S40, the second extraction unit 30 acquires the sentence to be estimated that has been input to the information processing device 10. Next, in step S42, the second extraction unit 30 performs morphological analysis of the sentence to be estimated using a morphological analysis system to extract each word in the sentence to be estimated. Next, in step S44, the second extraction unit 30 generates a vector whose elements indicate whether each word included in the training feature data appears in the sentence to be estimated (non-appearance).

[0053] Next, in step S46, the second determination unit 32 uses LLM to determine whether word combinations included in the training feature data are included in the same context in the sentence to be estimated. Next, in step S48, the second generation unit 34 generates estimation feature data by combining vectors regarding the appearance or non-appearance of words in the sentence to be estimated and vectors regarding word combinations included in the same context.

[0054] Next, in step S50, the output unit 36 ​​inputs the generated estimation feature data to the machine learning model 26 that has been trained by the training unit 24, and obtains and outputs the estimation result output from the machine learning model 26. Then, the estimation process ends.

[0055] As described above, the information processing device according to this embodiment extracts word combinations whose index values ​​indicating the association between the word combinations and the teacher labels satisfy a predetermined condition from word combinations included in each of a plurality of teacher-labeled sentences. The information processing device also determines whether the extracted word combinations are included in the same context in each of the plurality of teacher-labeled sentences. The information processing device then generates training data in which the presence or absence of word combinations determined to be included in the same context is used as a feature in each of the plurality of teacher-labeled sentences and associated with the teacher labels. This allows the information processing device to determine whether word combinations are included in the same context in order of priority based on their association with the teacher labels and to generate training data to which feature values ​​related to word combinations included in the same context have been added. In other words, training data that can improve the estimation accuracy and explainability of a machine learning model can be efficiently generated.

[0056] Note that the index values ​​used to determine the priority of word combinations for which the same-context determination is performed in the above embodiment are merely examples, and other index values ​​may be used. Furthermore, the greater the difference between the index value for the word combination and the index values ​​for each word included in the word combination, the higher the priority of the same-context determination may be. For example, as shown in FIG. 13 , in the case of the word combination of "development" and "delay," the index value is the difference between the index value for the word combination and the average value of the index values ​​for "development" and "delay." This makes it possible to extract word combinations that are more effective as features when words are combined.

[0057] In the above embodiment, the training data is generated using features that indicate whether a word combination is included in a sentence and are corrected based on the result of the context determination. However, the present invention is not limited to this. The corrected features may be added to the features for the word combination before correction as features that guarantee that the word combination is included in the same context.

[0058] Furthermore, in the above embodiment, the training data generation unit, the machine learning unit, and the estimation unit are configured as one computer, but each unit may be configured as a different computer.

[0059] In the above embodiment, the training data generation program, the machine learning program, and the estimation program are stored (installed) in advance in a storage device, but this is not limiting. The programs according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0060] REFERENCE SIGNS LIST 10 Information processing device 12 Training data generation unit 14 First extraction unit 16 First judgment unit 18 First generation unit 20 Training data DB 22 Machine learning unit 24 Training unit 26 Machine learning model 28 Estimation unit 30 Second extraction unit 32 Second judgment unit 34 Second generation unit 36 ​​Output unit 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 52 Training data generation program 54 First extraction process control command 56 First judgment process control command 58 First generation process control command 62 Machine learning program 64 Training process control command 68 Estimation program 70 Second extraction process control command 72 Second judgment process control command 74 Second generation process control command 76 Output process control command 80 Information storage area

Claims

1. A training data generation program for causing a computer to execute a process including: extracting word combinations from word combinations contained in each of a plurality of sentences to which a teacher label has been assigned, the word combinations having an index value indicating the relevance between the word combination and the teacher label that satisfies a predetermined condition; determining whether the extracted word combinations are contained in the same context in each of the plurality of sentences; and generating training data associated with the teacher label using the presence or absence in each of the plurality of sentences of the word combinations determined to be contained in the same context as a feature.

2. The training data generation program according to claim 1, wherein the index value indicates the significance of the difference in the occurrence rate of the word combination between the positive examples and the negative examples of the teacher label.

3. The training data generation program according to claim 2, wherein the index value includes at least one of a chi-square value, a normalized mutual information, a weight, a confidence, and a support.

4. A training data generation program according to any one of claims 1 to 3, wherein the process of extracting the word combination includes giving a higher priority to extraction of the word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is greater.

5. A training data generation program according to any one of claims 1 to 3, which determines whether the extracted word combinations are included in the same context using a large-scale language model.

6. A machine learning program for causing a computer to execute processing including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation program according to any one of claims 1 to 3.

7. An estimation program for causing a computer to execute a process including: determining whether or not word combinations included as features of the training data are included in the same context in the sentence to be estimated; generating feature data including, as features, the presence or absence in the sentence to be estimated of the word combinations determined to be included in the same context; and outputting the estimation result regarding the task obtained by inputting the feature data into the machine learning model trained by the machine learning program of claim 6.

8. A training data generation method in which a computer executes processes including: extracting word combinations from word combinations included in each of a plurality of sentences to which teacher labels have been assigned, the word combinations having index values ​​that satisfy predetermined conditions and indicate the association between the word combinations and the teacher labels; determining whether the extracted word combinations are included in the same context in each of the plurality of sentences; and generating training data associated with the teacher labels using the presence or absence in each of the plurality of sentences of the word combinations determined to be included in the same context as a feature.

9. The training data generation method according to claim 8, wherein the index value indicates the significance of the difference in the occurrence rate of the word combination between the positive examples and the negative examples of the teacher label.

10. The training data generation method of claim 9, wherein the index value includes at least one of a chi-squared value, a normalized mutual information, a weight, a confidence, and a support.

11. A training data generation method according to any one of claims 8 to 10, wherein the process of extracting the word combination includes giving a higher priority to extraction of the word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is greater.

12. A training data generation method according to any one of claims 8 to 10, wherein a large-scale language model is used to determine whether the extracted word combinations are included in the same context.

13. A machine learning method in which a computer executes processing including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation method according to any one of claims 8 to 10.

14. An estimation method in which a computer performs processes including: determining whether word combinations included as features of the training data are included in the same context in the sentence to be estimated; generating feature data including, as features, the presence or absence in the sentence to be estimated of the word combinations determined to be included in the same context; and outputting the estimation result regarding the task obtained by inputting the feature data into the machine learning model trained by the machine learning method of claim 13.

15. A training data generation device comprising: a first extraction unit that extracts word combinations from word combinations included in each of a plurality of sentences to which a teacher label has been assigned, the word combinations having an index value indicating the association between the word combination and the teacher label that satisfies a predetermined condition; a first judgment unit that judges whether the extracted word combinations are included in the same context in each of the plurality of sentences; and a first generation unit that generates training data associated with the teacher label using the presence or absence in each of the plurality of sentences of the word combinations judged to be included in the same context as a feature.

16. A training data generation device according to claim 15, wherein the index value indicates the significance of the difference in the occurrence rate of the word combination between the positive examples and the negative examples of the teacher label.

17. The training data generating apparatus of claim 16, wherein the index value includes at least one of a chi-squared value, a normalized mutual information, a weight, a confidence, and a support.

18. A training data generation device as described in any one of claims 15 to 17, wherein the first extraction unit gives a higher priority to extraction of a word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is greater.

19. A machine learning device including a training unit that uses training data generated by the training data generation device according to any one of claims 15 to 17 to train a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input.

20. An estimation device comprising: a second determination unit that determines whether word combinations included as features of the training data are included in the same context in the sentence to be estimated; a second generation unit that generates feature data including, as features, the presence or absence in the sentence to be estimated of the word combinations determined to be included in the same context; and an output unit that outputs the estimation result regarding the task obtained by inputting the feature data into the machine learning model trained by the machine learning device of claim 19.

Images