Training data generation program, machine learning program, estimate program, method, and device

Abstract

To efficiently generate training data which can improve estimation accuracy and explanation capability of a machine learning model.SOLUTION: An information processing device: extracts combination of words in which an index value which indicates relevance between the combination of the words and a teacher label satisfies a predetermined condition, from the combinations of the words included in each of a plurality of sentences to which the teacher label is assigned; determines whether or not the extracted combination of the words is included in the same context in each of the plurality of sentences; sets presence / absence in each of the plurality of sentences of the combination of the words determined to be included in the same context as a feature amount; and generates training data associated with the teacher label.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The disclosed technology relates to a training data generation program, a training data generation method, a training data generation device, a machine learning program, a machine learning method, a machine learning device, an estimation program, an estimation method, and an estimation device. [Background technology]

[0002] Conventionally, for example, quality assurance personnel would look at various numerical data related to a project and documents showing progress, predict or determine whether the project's profitability is declining, and correct the project's progress. There are also systems that use the presence or absence of specific words in project-related documents as features of the documents and predict declining profitability using a machine learning model. In this case, there are also systems that use explainable AI (artificial intelligence) as the machine learning model to analyze the factors behind declining project profitability. Specific words can be specified in advance or acquired automatically.

[0003] As a machine learning technology that uses words contained in a sentence as features, for example, a word semantic relationship extraction device has been proposed that extracts word semantic relationships from text data by performing multi-class supervised learning using an existing thesaurus as the correct answer. This device calculates multiple types of similarity for any pair of words in the text and generates a feature vector with each similarity as an element. The device also assigns a label to each word pair indicating the type of word semantic relationship based on the thesaurus. The device then learns word semantic relationship identification data from the feature vectors and labels as a multi-class classification problem, and uses the word semantic relationship identification data to identify the semantic relationship between two words.

[0004] Furthermore, for example, a word meaning assignment device has been proposed that extracts features for combinations of words contained in an input sentence written in a natural language and determines a higher semantic class that indicates a higher concept of the word based on the features.

[0005] Also proposed is a method for receiving labeled training data including multiple groups of words at a server having one or more processors. The method further includes extracting multiple groups of words within the syntactic context of the predicates, each group of words having a predicate and each word having a general word embedding. The method further includes creating a high-dimensional vector space representing the features of each word by concatenating the general word embeddings at the server. The method further includes obtaining a model at the server that includes a learned mapping from the high-dimensional vector space to a low-dimensional vector space and learned embeddings for each possible semantic frame in the low-dimensional vector space. The method then outputs the model for storage by the server, where the model is configured to identify a specific semantic frame for the input.

[0006] For example, methods have been proposed for automatic text analysis and ranking of contexts containing words that appear in a corpus, which rank contexts locally for each word by frequency of occurrence, globally rank contexts, and calculate statistics such as log-rank ratio statistics to obtain the similarity of the words in a context, and then order the contexts based on these. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2014 / 033799 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-181408 [Patent Document 3] US Patent Application Publication No. 2016 / 0239739 [Patent Document 4] US Patent Application Publication No. 2005 / 0049867 Summary of the Invention [Problem to be solved by the invention]

[0008] However, simply treating each word as a feature, such as when the presence or absence of a specific word in a sentence, as a feature of that sentence, often does not improve the accuracy of estimation by a machine learning model or results in estimation results with low interpretability. This is because, for example, even if "delay" is specified as a specific word indicating a deterioration in profitability, it is difficult to distinguish whether the "delay" has "occurred" or has "been resolved."

[0009] Therefore, it is possible to use the feature of whether multiple word combinations appear simultaneously in a sentence. However, word combinations extracted from unrelated sentences do not contribute to improving the estimation accuracy or interpretability of the machine learning model, so it is desirable to exclude them from the features used in the training data. In other words, it is desirable to use word combinations that appear in the same context as features.

[0010] However, it is not easy to determine whether a combination of words is in the same context. For example, if a sentence is divided into sentences using periods and a combination of words appears within a sentence, it may be determined that the combination of words is in the same context. However, this determination method is not sufficient because each word in a combination of words may be used in the same context even if it appears across multiple sentences. Furthermore, determining whether all combinations of words are in the same context is not realistic in terms of processing time.

[0011] In one aspect, the disclosed technology aims to efficiently generate training data that can improve the estimation accuracy and explainability of machine learning models. [Means for solving the problem]

[0012] In one aspect, the disclosed technology extracts word combinations from word combinations included in each of a plurality of sentences to which a teacher label has been assigned, the word combinations having an index value indicating a relevance between the word combination and the teacher label that satisfies a predetermined condition. The disclosed technology also determines whether the extracted word combinations are included in the same context in each of the plurality of sentences. The disclosed technology then generates training data in which the presence or absence of the word combinations determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label. [Effects of the Invention]

[0013] One aspect is that it has the effect of efficiently generating training data that can improve the estimation accuracy and explainability of a machine learning model. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a functional block diagram of an information processing device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of a set of sentences with teacher labels. [Figure 3] FIG. 10 is a diagram showing an example of a vector of word occurrence / non-occurrence. [Figure 4] FIG. 10 is a diagram illustrating index values ​​and priorities of word combinations. [Figure 5] FIG. 10 is a diagram illustrating an example of context determination. [Figure 6] FIG. 10 is a diagram illustrating an example of the context determination. [Figure 7] FIG. 10 is a diagram illustrating an example of the context determination. [Figure 8] FIG. 10 is a diagram illustrating an example of an occurrence / non-occurrence vector of a word combination. [Figure 9] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as an information processing device. [Figure 10] 10 is a flowchart illustrating an example of a training data generation process. [Figure 11]10 is a flowchart illustrating an example of machine learning processing. [Figure 12] 10 is a flowchart illustrating an example of an estimation process. [Figure 13] FIG. 10 is a diagram illustrating the difference between the index value of a word combination and the index value of each word. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In this embodiment, a case will be described in which a task is executed to determine whether the profitability of a project is deteriorating based on a document related to the project.

[0016] 1, an information processing device 10 according to this embodiment functionally includes a training data generation unit 12, a machine learning unit 22, and an estimation unit 28. A training data DB (Database) 20 and a machine learning model 26 are stored in a predetermined storage area of ​​the information processing device 10. The training data generation unit 12 is an example of a training data generation device of the present disclosure, the machine learning unit 22 is an example of a machine learning device of the present disclosure, and the estimation unit 28 is an example of an estimation device of the present disclosure.

[0017] First, we will explain the training data generation unit 12. The training data generation unit 12 further includes a first extraction unit 14, a first determination unit 16, and a first generation unit 18.

[0018] The first extraction unit 14 acquires a set of labeled sentences input to the information processing device 10. As shown in FIG. 2, the set of labeled sentences includes a plurality of labeled sentences. Each labeled sentence is text data in which a sentence indicating the progress of each project is written in natural language, and is assigned a labeled sentence indicating whether the project is deteriorating in profitability. In FIG. 2, "Deteriorating profitability: 1" represents a labeled sentence indicating that profitability has deteriorated, and "Deteriorating profitability: 0" represents a labeled sentence indicating that profitability has not deteriorated. Each labeled sentence is also assigned a project number (hereinafter also referred to as "PJ number"), which is identification information for each project.

[0019] The first extraction unit 14 uses a morphological analysis system such as MeCab to perform morphological analysis on each teacher-labeled sentence and extract each word from the teacher-labeled sentence. As shown in FIG. 3, the first extraction unit 14 generates a vector for each teacher-labeled sentence, including an element indicating whether the extracted word appears (non-appearance) in the teacher-labeled sentence and the teacher label. In the example of FIG. 3, each row corresponds to a vector generated from the teacher-labeled sentence of the project indicated by the project number in the "PJ Number" column. Words such as "development," "delay," "bug," "completion," and "fix" are extracted from the entire set of teacher-labeled sentences. If a word appears in a teacher-labeled sentence, the value of the element for the corresponding word in the vector for that teacher-labeled sentence is set to "1." If the word does not appear, the value of the element is set to "0." The "profitability deterioration" column is a teacher label.

[0020] Based on the extracted words, the first extraction unit 14 comprehensively calculates an index value indicating the association between the word combination and the teacher label for word combinations containing one or more words. The index value indicates the significance of the difference in the occurrence rate of the word combination between positive examples and negative examples of the teacher label. For example, the index value includes at least one of a chi-square value, normalized mutual information, weight, confidence, and support. The weight indicates the importance of each word combination for classification into positive examples and negative examples, and is a value calculated so that the sum of the weights for all word combinations is 1.

[0021] The upper diagram of Figure 4 shows an example of index values ​​calculated for each word combination. In Figure 4, "weight" is weight, "supp" is support, "conf" is confidence, "chi2" is chi-squared value, and "nmi" is "normalized mutual information." In the example in the upper diagram of Figure 4, "npos," which is the number of positive examples, and "nneg," which is the number of negative examples, are also included as index values. In addition, while the example in Figure 4 shows a combination of two words, the word combination may be one word or three or more words.

[0022] The first extraction unit 14 extracts word combinations whose calculated index values ​​satisfy a predetermined condition. For example, the first extraction unit 14 extracts a predetermined number of word combinations in descending order of a specific index value or an integrated index value obtained by integrating multiple index values, or word combinations whose index value or integrated index value is equal to or greater than a predetermined value. The specific index value may be selected as an index value suitable for indicating the significance of the difference in the occurrence rate of word combinations between positive and negative examples of teacher labels. For example, the specific index value may be a chi-square value or normalized mutual information, or may be a weight or reliability. The integrated index value may be the product or weighted sum of each index value, the average of normalized index values, or the like. The number of word combinations to be extracted may be preset based on the execution time of a determination by the first determination unit 16 (described later) of whether word combinations are included in the same context (hereinafter also referred to as "same context determination").

[0023] As shown in the lower diagram of Fig. 4, the first extraction unit 14 prioritizes the word combinations extracted based on the index values ​​in descending order of index value or integrated index value, and passes them to the first determination unit 16. The priority here refers to the priority when the first determination unit 16 performs context determination on the combinations.

[0024] The first determination unit 16 performs context determination on the word combination passed from the first extraction unit 14. The context determination may be performed based on, for example, the distance between the appearance positions of each word included in the word combination in a sentence, but in this embodiment, it is performed using large language models (LLM).

[0025] Specifically, the first determination unit 16 selects a word combination in descending order of priority from the word combinations passed from the first extraction unit 14. The first determination unit 16 inputs to the LLM a teacher-labeled sentence containing the selected word combination and a question asking whether the selected word combination is used in the same context within the teacher-labeled sentence. The first determination unit 16 acquires the answer output from the LLM as the determination result of the context determination. The first determination unit 16 performs context determination on all teacher-labeled sentences containing the selected word combination, and also performs context determination on all word combinations passed from the first extraction unit 14.

[0026] 5 to 7 show examples of sentences and questions input to an LLM and answers from the LLM when using GPT-4 (Generative Pre-trained Transformer 4) as an example of an LLM. As shown in FIGS. 5 to 7, by using an LLM, it is possible to accurately determine the context of each word in a word combination, regardless of whether the words are contained together in a single sentence separated by periods or appear across two or more sentences.

[0027] The first generation unit 18 generates training data in which the presence or absence of word combinations determined to be included in the same context in each of the teacher-labeled sentences is used as a feature quantity and associated with the teacher label.

[0028] For example, as shown in the upper diagram of FIG. 8, a vector whose elements indicate whether or not a word combination extracted by the first extraction unit 14 appears (non-appearance) in each teacher-labeled sentence is generated for each teacher-labeled sentence. In the example of the upper diagram of FIG. 8, each row corresponds to a vector indicating a feature for a project indicated by the project number in the "PJ Number" column. If each word combination appears in the teacher-labeled sentence, the element value is set to "1," and if it does not appear, the element value is set to "0." As shown in the lower diagram of FIG. 8, the first generation unit 18 changes elements of the vector corresponding to word combinations determined by the first determination unit 16 not to be included in the same context (e.g., elements indicated by bold lines in FIG. 8) from 1 to 0. As a result, each vector becomes a feature indicating whether or not each word combination appears in the same context in each teacher-labeled sentence.

[0029] The first generation unit 18 generates training feature data by combining, for example, a vector regarding the appearance or non-appearance of a word in a teacher-labeled sentence as shown in FIG. 3 with a vector regarding a combination of words included in the same context as shown in the lower diagram of FIG. 8. The first generation unit 18 also assigns teacher labels to the training feature data to generate training data. The first generation unit 18 stores the training data generated for each teacher-labeled sentence in the training data DB 20.

[0030] Next, we will explain the machine learning unit 22. The machine learning unit 22 includes a training unit 24.

[0031] The training unit 24 trains the machine learning model 26 using training data stored in the training data DB 20. When feature data for a sentence to be estimated is input, the machine learning model 26 outputs an estimation result corresponding to the task. The estimation result here is whether or not the profitability of the project corresponding to the sentence to be estimated has deteriorated. Specifically, the training unit 24 updates the parameters of the machine learning model 26 so that the estimation result (label) output when training feature data, which is the portion of the training data excluding the teacher label, matches the teacher label.

[0032] Next, a description will be given of the estimation unit 28. The estimation unit 28 further includes a second extraction unit 30, a second determination unit 32, a second generation unit 34, and an output unit 36.

[0033] The second extraction unit 30 acquires the sentence to be estimated that is input to the information processing device 10. The sentence to be estimated is similar to the sentence with a teacher label, except that the sentence to be estimated is not assigned a teacher label. Similar to the first extraction unit 14, the second extraction unit 30 performs morphological analysis of the sentence to be estimated using a morphological analysis system to extract each word in the sentence to be estimated. The second extraction unit 30 also generates a vector whose elements indicate whether each word included in the training feature data appears in the sentence to be estimated (non-appearance). This vector is similar to the vector shown in FIG. 3 with the teacher label removed.

[0034] The second determination unit 32 determines whether or not a combination of words included in the training feature data is included in the same context in the sentence to be estimated. The second determination unit 32 may perform the same context determination using LLM, similar to the first determination unit 16.

[0035] Similar to the first generation unit 18, the second generation unit 34 generates estimation feature data by combining vectors regarding the occurrence or non-occurrence of words in the sentence to be estimated with vectors regarding combinations of words included in the same context.

[0036] The output unit 36 ​​inputs the estimation feature data generated by the second generation unit 34 into the machine learning model 26 that has been trained by the training unit 24, obtains the estimation results output from the machine learning model 26, and outputs them.

[0037] The information processing device 10 may be realized by, for example, a computer 40 shown in Fig. 9. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0038] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 44, which serves as a storage medium, stores a training data generation program 52, a machine learning program 62, and an estimation program 68 for causing the computer 40 to function as the information processing device 10. The training data generation program 52 includes a first extraction process control instruction 54, a first determination process control instruction 56, and a first generation process control instruction 58. The machine learning program 62 includes a training process control instruction 64. The estimation program 68 includes a second extraction process control instruction 70, a second determination process control instruction 72, a second generation process control instruction 74, and an output process control instruction 76. The storage device 44 also includes an information storage area 80 in which information constituting the training data DB 20 and the machine learning model 26 is stored.

[0039] The CPU 41 reads the training data generation program 52 from the storage device 44, expands it in the memory 43, and sequentially executes the control instructions of the training data generation program 52. The CPU 41 operates as the first extraction unit 14 shown in FIG. 1 by executing the first extraction process control instruction 54. The CPU 41 also operates as the first judgment unit 16 shown in FIG. 1 by executing the first judgment process control instruction 56. The CPU 41 also operates as the first generation unit 18 shown in FIG. 1 by executing the first generation process control instruction 58.

[0040] Furthermore, the CPU 41 reads the machine learning program 62 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the machine learning program 62. The CPU 41 executes the training process control instructions 64 to function as the training unit 24 shown in FIG.

[0041] The CPU 41 also reads out the estimation program 68 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions included in the estimation program 68. The CPU 41 operates as the second extraction unit 30 shown in FIG. 1 by executing a second extraction process control instruction 70. The CPU 41 also operates as the second judgment unit 32 shown in FIG. 1 by executing a second determination process control instruction 72. The CPU 41 also operates as the second generation unit 34 shown in FIG. 1 by executing a second generation process control instruction 74. The CPU 41 also operates as the output unit 36 ​​shown in FIG. 1 by executing an output process control instruction 76.

[0042] Furthermore, the CPU 41 reads information from the information storage area 80 and loads the training data DB 20 and the machine learning model 26 into the memory 43. As a result, the computer 40 that has executed the training data generation program 52, the machine learning program 62, and the estimation program 68 functions as the information processing device 10. The CPU 41 that executes the programs is hardware. Part of the programs may also be executed by the GPU 42.

[0043] The functions realized by each program may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0044] Next, the operation of the information processing device 10 according to this embodiment will be described. When a set of sentences with teacher labels is input to the information processing device 10 and an instruction to generate training data is given, the information processing device 10 executes a training data generation process shown in FIG. 10. Furthermore, when an instruction to train the machine learning model 26 is given with the generated training data stored in the training data DB 20, the information processing device 10 executes a machine learning process shown in FIG. 11. When an estimation target sentence is input to the information processing device 10 and an instruction to estimate a deterioration in profitability is given with the trained machine learning model 26 stored in a predetermined storage area of ​​the information processing device 10, the information processing device 10 executes an estimation process shown in FIG. 12. Note that the training data generation process is an example of a training data generation method of the present disclosure, the machine learning process is an example of a machine learning method of the present disclosure, and the estimation process is an example of an estimation method of the present disclosure.

[0045] First, the training data generation process shown in FIG. 10 will be described.

[0046] In step S10, the first extraction unit 14 acquires a set of teacher-labeled sentences input to the information processing device 10. Next, in step S12, the first extraction unit 14 performs morphological analysis on each teacher-labeled sentence and extracts each word in the teacher-labeled sentence. Next, in step S14, the first extraction unit 14 generates, for each teacher-labeled sentence, a vector including an element indicating whether the extracted word appears (non-appearance) in the teacher-labeled sentence and the teacher label.

[0047] Next, in step S16, the first extraction unit 14 comprehensively calculates index values ​​indicating the association between word combinations and truth labels for word combinations containing one or more words based on the extracted words. Next, in step S18, the first extraction unit 14 extracts a predetermined number of word combinations in descending order of a specific index value or an integrated index value obtained by integrating multiple index values, or word combinations whose index value or integrated index value is equal to or greater than a predetermined value. Then, the first extraction unit 14 prioritizes the extracted word combinations in descending order of index value or integrated index value, and passes them to the first determination unit 16.

[0048] Next, in step S20, the first determination unit 16 performs same-context determination using LLM for the word combinations passed from the first extraction unit 14. Next, in step S22, the first generation unit 18 generates a vector indicating the presence or absence in each of the teacher-labeled sentences of word combinations determined to be included in the same context. The first generation unit 18 also generates training feature data by combining the vector indicating the appearance or non-appearance of words in the teacher-labeled sentences with the vector indicating the word combination included in the same context. The first generation unit 18 then assigns teacher labels to the training feature data to generate training data, stores the training data in the training data DB 20, and the training data generation process ends.

[0049] Next, the machine learning process shown in FIG. 11 will be described.

[0050] In step S30, the training unit 24 retrieves training data from the training data DB 20. Next, in step S32, the training unit 24 uses the retrieved training data to train the machine learning model 26 that outputs an estimation result according to the task when feature data for a sentence to be estimated is input. Next, in step S34, the training unit 24 stores the trained machine learning model 26 in a predetermined storage area, and the machine learning process ends.

[0051] Next, the estimation process shown in Fig. 12 will be described. Note that detailed description of processes similar to the training data generation process will be omitted.

[0052] In step S40, the second extraction unit 30 acquires the sentence to be estimated that has been input to the information processing device 10. Next, in step S42, the second extraction unit 30 performs morphological analysis of the sentence to be estimated using a morphological analysis system to extract each word in the sentence to be estimated. Next, in step S44, the second extraction unit 30 generates a vector whose elements indicate whether each word included in the training feature data appears in the sentence to be estimated (non-appearance).

[0053] Next, in step S46, the second determination unit 32 uses LLM to determine whether word combinations included in the training feature data are included in the same context in the sentence to be estimated. Next, in step S48, the second generation unit 34 generates estimation feature data by combining vectors regarding the appearance or non-appearance of words in the sentence to be estimated and vectors regarding word combinations included in the same context.

[0054] Next, in step S50, the output unit 36 ​​inputs the generated estimation feature data to the machine learning model 26 that has been trained by the training unit 24, and obtains and outputs the estimation result output from the machine learning model 26. Then, the estimation process ends.

[0055] As described above, the information processing device according to this embodiment extracts word combinations whose index values ​​indicating the association between the word combinations and the teacher labels satisfy a predetermined condition from word combinations included in each of a plurality of teacher-labeled sentences. The information processing device also determines whether the extracted word combinations are included in the same context in each of the plurality of teacher-labeled sentences. The information processing device then generates training data in which the presence or absence of word combinations determined to be included in the same context is used as a feature in each of the plurality of teacher-labeled sentences and associated with the teacher labels. This allows the information processing device to determine whether word combinations are included in the same context in order of priority based on their association with the teacher labels and to generate training data to which feature values ​​related to word combinations included in the same context have been added. In other words, training data that can improve the estimation accuracy and explainability of a machine learning model can be efficiently generated.

[0056] Note that the index values ​​used to determine the priority of word combinations for which the same-context determination is performed in the above embodiment are merely examples, and other index values ​​may be used. Furthermore, the greater the difference between the index value for the word combination and the index values ​​for each word included in the word combination, the higher the priority of the same-context determination may be. For example, as shown in FIG. 13 , in the case of the word combination of "development" and "delay," the index value is the difference between the index value for the word combination and the average value of the index values ​​for "development" and "delay." This makes it possible to extract word combinations that are more effective as features when words are combined.

[0057] In the above embodiment, the training data is generated using features that indicate whether a word combination is included in a sentence and are corrected based on the result of the context determination. However, the present invention is not limited to this. The corrected features may be added to the features for the word combination before correction as features that guarantee that the word combination is included in the same context.

[0058] Furthermore, in the above embodiment, the training data generation unit, the machine learning unit, and the estimation unit are configured as one computer, but each unit may be configured as a different computer.

[0059] In the above embodiment, the training data generation program, the machine learning program, and the estimation program are stored (installed) in advance in a storage device, but this is not limiting. The programs according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0060] The following additional notes are provided regarding the above-described embodiments.

[0061] (Appendix 1) extracting word combinations, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, for which an index value indicating a relevance between the word combination and the truth label satisfies a predetermined condition; determining whether the extracted word combinations are included in the same context in each of the plurality of sentences; generating training data in which the presence or absence of the combination of words determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label; A training data generation program for causing a computer to execute a process including the above.

[0062] (Appendix 2) The training data generation program according to claim 1, wherein the index value indicates the significance of the difference in the occurrence rate of the word combination between positive examples and negative examples of the teacher label.

[0063] (Appendix 3) 3. The training data generation program according to claim 2, wherein the index value includes at least one of a chi-square value, a normalized mutual information, a weight, a confidence, and a support.

[0064] (Appendix 4) The training data generation program according to any one of Supplementary Note 1 to Supplementary Note 3, wherein the process of extracting the word combination includes giving a higher priority to extraction of the word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is larger.

[0065] (Appendix 5) 5. The training data generation program according to claim 1, wherein a large-scale language model is used to determine whether the extracted word combinations are included in the same context.

[0066] (Appendix 6) A machine learning program for causing a computer to execute a process including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation program according to any one of Supplementary Note 1 to Supplementary Note 5.

[0067] (Appendix 7) determining whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; generating feature data including, as a feature, the presence or absence of the combination of words determined to be included in the same context in the sentence to be estimated; The feature data is input to the machine learning model trained by the machine learning program of Appendix 6, and the estimation result regarding the task obtained is output. An estimation program for causing a computer to execute a process including the above.

[0068] (Appendix 8) extracting word combinations, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, for which an index value indicating a relevance between the word combination and the truth label satisfies a predetermined condition; determining whether the extracted word combinations are included in the same context in each of the plurality of sentences; generating training data in which the presence or absence of the combination of words determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label; A training data generation method in which a computer executes a process including:

[0069] (Appendix 9) 9. The training data generation method according to claim 8, wherein the index value indicates the significance of the difference in the occurrence rate of the word combination between positive examples and negative examples of the teacher label.

[0070] (Appendix 10) 10. The training data generation method of claim 9, wherein the index value includes at least one of a chi-squared value, a normalized mutual information, a weight, a confidence, and a support.

[0071] (Appendix 11) The training data generation method according to any one of Supplementary Note 8 to Supplementary Note 10, wherein the process of extracting the word combination includes giving a higher priority to extraction of the word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is larger.

[0072] (Appendix 12) The training data generation method according to any one of Supplementary Note 8 to Supplementary Note 11, wherein it is determined whether or not the extracted word combinations are included in the same context using a large-scale language model.

[0073] (Appendix 13) A machine learning method in which a computer executes a process including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation method according to any one of Supplementary Note 8 to Supplementary Note 12.

[0074] (Appendix 14) determining whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; generating feature data including, as a feature, the presence or absence of the combination of words determined to be included in the same context in the sentence to be estimated; The feature data is input to the machine learning model trained by the machine learning method of Appendix 13, and the estimation result regarding the task obtained is output. The estimation method includes a process performed by a computer.

[0075] (Appendix 15) a first extraction unit that extracts, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, word combinations whose index values ​​indicating the association between the word combinations and the truth labels satisfy a predetermined condition; a first determination unit that determines whether the extracted word combinations are included in the same context in each of the plurality of sentences; a first generation unit that generates training data in which the presence or absence of the word combinations determined to be included in the same context in each of the plurality of sentences is used as a feature and is associated with the teacher label; A training data generation device comprising:

[0076] (Appendix 16) 16. The training data generation device according to claim 15, wherein the index value indicates the significance of the difference in appearance rate of the word combination between positive examples and negative examples of the teacher label.

[0077] (Appendix 17) 17. The training data generation device of claim 16, wherein the index value includes at least one of a chi-squared value, a normalized mutual information, a weight, a confidence, and a support.

[0078] (Appendix 18) The training data generation device according to any one of Supplementary Note 15 to Supplementary Note 17, wherein the first extraction unit assigns a higher extraction priority to a word combination for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is greater.

[0079] (Appendix 19) a training unit that uses training data generated by the training data generation device according to any one of Supplementary Note 15 to Supplementary Note 18 to train a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input. Machine learning devices including:

[0080] (Appendix 20) a second determination unit that determines whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; a second generation unit that generates feature data including, as a feature, whether or not the combination of words determined to be included in the same context is present in the sentence to be estimated; an output unit that outputs the estimation result regarding the task obtained by inputting the feature data into the machine learning model trained by the machine learning device of Supplementary Note 19; An estimation device comprising: [Explanation of symbols]

[0081] 10. Information processing equipment 12 Training data generation unit 14 1st extraction part 16 1st Judgment Section 18 1st generation part 20 Training Data DB 22 Machine Learning Department 24 Training Department 26 Machine Learning Models 28 Estimation part 30 Second extraction part 32 Second Judgment Section 34 Second generation part 36 Output section 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / Output Devices 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 52 Training data generation program 54 First Extraction Process Control Instruction 56 First judgment process control instruction 58 First Generation Process Control Instruction 62 Machine Learning Programs 64 Training Process Control Instructions 68 Estimation Program 70 Second Extraction Process Control Instruction 72 Second decision process control instruction 74 Second generation process control instruction 76 Output Process Control Instructions 80 Information storage area

Claims

1. extracting word combinations, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, for which an index value indicating a relevance between the word combination and the truth label satisfies a predetermined condition; determining whether the extracted word combinations are included in the same context in each of the plurality of sentences; generating training data in which the presence or absence of the combination of words determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label; A training data generation program for causing a computer to execute a process including the above.

2. 2. The training data generation program according to claim 1, wherein the index value indicates the significance of a difference in the occurrence rate of the word combination between positive examples and negative examples of the teacher label.

3. 3. The training data generation program according to claim 2, wherein the index value includes at least one of a chi-square value, a normalized mutual information, a weight, a confidence, and a support.

4. The training data generation program according to any one of claims 1 to 3, wherein the process of extracting the word combinations includes giving a higher priority to extraction of the word combinations for which the difference between the index value for the word combination and the index values ​​for each word included in the word combination is greater.

5. 4. The training data generation program according to claim 1, wherein a large-scale language model is used to determine whether the extracted word combinations are included in the same context.

6. A machine learning program for causing a computer to execute a process including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation program according to any one of claims 1 to 3.

7. determining whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; generating feature data including, as a feature, the presence or absence of the combination of words determined to be included in the same context in the sentence to be estimated; The feature data is input to the machine learning model trained by the machine learning program of claim 6, and the estimation result regarding the task obtained by inputting the feature data is output. An estimation program for causing a computer to execute a process including the above.

8. extracting word combinations, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, for which an index value indicating a relevance between the word combination and the truth label satisfies a predetermined condition; determining whether the extracted word combinations are included in the same context in each of the plurality of sentences; generating training data in which the presence or absence of the combination of words determined to be included in the same context in each of the plurality of sentences is used as a feature and associated with the teacher label; A training data generation method in which a computer executes a process including:

9. 9. A machine learning method in which a computer executes a process including training a machine learning model that outputs an estimation result according to a task when feature data about a sentence to be estimated is input, using training data generated by the training data generation method according to claim 8.

10. determining whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; generating feature data including, as a feature, the presence or absence of the combination of words determined to be included in the same context in the sentence to be estimated; The machine learning model trained by the machine learning method of claim 9 is input with the feature data, and the estimation result regarding the task obtained is output. The estimation method includes a process performed by a computer.

11. a first extraction unit that extracts, from word combinations included in each of a plurality of sentences to which truth labels have been assigned, word combinations whose index values ​​indicating the association between the word combinations and the truth labels satisfy a predetermined condition; a first determination unit that determines whether the extracted word combinations are included in the same context in each of the plurality of sentences; a first generation unit that generates training data in which the presence or absence of the word combinations determined to be included in the same context in each of the plurality of sentences is used as a feature and is associated with the teacher label; A training data generation device comprising:

12. A training unit that uses the training data generated by the training data generation device according to claim 11 to train a machine learning model that outputs an estimation result according to a task when feature data on a sentence to be estimated is input. Machine learning devices including:

13. a second determination unit that determines whether a combination of words included as a feature of the training data is included in the same context in the sentence to be estimated; a second generation unit that generates feature data including, as a feature, whether or not the combination of words determined to be included in the same context is present in the sentence to be estimated; an output unit that outputs the estimation result regarding the task obtained by inputting the feature data into the machine learning model trained by the machine learning device of claim 12; An estimation device comprising:

Images