Data generation apparatus and method

Abstract

To make it possible to improve quality of a data set.SOLUTION: A data generation apparatus according to the present embodiment includes a selection unit and a determination unit. The selection unit selects an event group in which at least a part of a plurality of second event ranges overlap, the second event ranges being ranges of character sequences estimated by a plurality of different methods with respect to a document of teaching data and being different from first event ranges being ranges of character sequences defined with respect to the document. The determination unit determines an additional event which is the second event range to be added to the teaching data from the event group.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments of the present invention relate to a data generating device, a method, and a learning device. [Background technology]

[0002] One of the most popular tasks in natural language processing is text range extraction, such as named entity recognition using sequence labeling. For machine learning applications of sequence labeling, datasets are prepared in which documents are pre-labeled with labels specifying text ranges, but mislabeling may occur. For such datasets, a method exists to estimate sentences that may contain mislabeling and reduce the weight of the estimated mislabeled sentences, thereby reducing the impact of mislabeling and improving the precision of inference for trained models trained using the dataset.

[0003] However, when performing tasks such as causal relationship extraction, which preprocess text ranges extracted by sequence labeling, it is important to extract all text ranges that are likely to have causal relationships. In other words, recall, which indicates whether strings that should be labeled have been correctly labeled, is more important than precision, which indicates the percentage of correct labels assigned. Therefore, the above-mentioned method only reduces the weight of mislabeled sentences and does not improve recall. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zihan Wang et al., “CrossWeigh: Training Named Entity Tagger from Imperfect Annotations”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Pages 5154-5163, Hong Kong, China, November 3-7, 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to improve the quality of datasets. [Means for solving the problem]

[0006] The data generating device according to this embodiment includes a selection unit and a determination unit. An event group is selected that is different from a first event range, which is a range of character strings defined for a document of training data, and that overlaps at least a portion of multiple second event ranges, which are ranges of character strings estimated for the document using multiple different models. The determination unit determines additional events, which are the second event ranges, to be added to the training data from the event group. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing a data generating device according to an embodiment of the present invention. [Figure 2] FIG. 4 is a diagram showing an example of teaching data stored in a teaching data storage unit. [Figure 3] 10 is a flowchart illustrating an example of an event generation process of the data generating device. [Figure 4]A diagram showing an example of using partial data from the first round of k-fold cross-validation. [Figure 5] A diagram showing an example of using the second partial data of k-fold cross validation. [Figure 6] FIG. 10 is a diagram showing an example of a method for generating an event group. [Figure 7] FIG. 10 is a diagram showing an example of selecting a candidate group from an event group. [Figure 8] FIG. 10 is a diagram showing an example of determining an additional event. [Figure 9] FIG. 10 is a diagram showing an example of using an event range. [Figure 10] FIG. 10 is a diagram showing an example in which an additional event is added by the data generating device. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of a data generating device. DETAILED DESCRIPTION OF THE INVENTION

[0008] The data generation device, method, and learning device according to the present embodiment will be described in detail below with reference to the drawings. Note that in the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

[0009] The data generating device according to this embodiment will be described with reference to the block diagram of FIG. The data generating device 10 according to this embodiment includes a teaching data storage unit 101, a dividing unit 102, a learning unit 103, an estimation unit 104, an estimation result storage unit 105, a selecting unit 106, a determining unit 107, and an adding unit 108. The teaching data storage unit 101 and the learning unit 103 are collectively referred to as a learning device.

[0010] The training data storage unit 101 stores training data. The training data is a dataset in which a document containing multiple sentences is associated with a text range (hereinafter referred to as an event range) arbitrarily specified for a character string included in the document. An event according to this embodiment refers to an occurrence indicated in the document. An event range is assumed to be, for example, a range of character strings indicating the cause or result of a problem, but is not limited to an event, and may be any text range specified for another purpose, such as a named entity. The event range of the dataset may be assigned manually, for example.

[0011] The dividing unit 102 receives the teaching data stored in the teaching data storage unit 101 and divides the teaching data into a plurality of partial data. In this embodiment, it is assumed that k-fold cross validation (k is a positive number equal to or greater than 2) is performed, for example, and the dividing unit 102 divides the teaching data into k pieces of partial data. Furthermore, the dividing unit 102 generates a plurality of sets of partial data by varying the division positions in the teaching data.

[0012] The learning unit 103 uses the training data to train a model and generate a trained model. For example, the learning unit 103 uses one of the k pieces of partial data as inference data and the remaining k-1 pieces of partial data as training data to train a model and generate k trained models. Furthermore, the k trained models are considered as one set, and a trained model is generated for each set of k pieces of partial data. Note that the k trained models generated corresponding to one set of multiple partial data are also referred to as one trained model set.

[0013] The estimation unit 104 estimates an event range in a document of the training data for each of a plurality of different trained model sets trained using the training data.

[0014] The estimation result storage unit 105 stores the event range estimated by the estimation unit 104 in association with the document, for example, as a label indicating the range of the corresponding character string in the document.

[0015] The selection unit 106 selects an event group that differs from the event range already defined in the training data and in which multiple event ranges estimated using multiple different methods for the documents of the training data at least partially overlap. The multiple event ranges estimated using multiple different methods refer to, for example, multiple event ranges estimated by the estimation unit 104 for each trained model set. Note that multiple different methods may refer to multiple event ranges estimated for the training data from different perspectives. In other words, the positions of sentences in the documents of the training data may be swapped, the network structure of the model may be changed, the hyperparameters of the model may be changed, or the event ranges may be estimated manually.

[0016] The determination unit 107 determines additional events, which are a range of events to be added to the teaching data, from the event group. The adding unit 108 adds the additional event to the teaching data and registers it in the teaching data storage unit 101 . The teaching data storage unit 101 and the estimation result storage unit 105 may be located outside the data generating device 10, such as on an external server, as long as the data generating device 10 can access them as needed.

[0017] Next, an example of the teaching data stored in the teaching data storage unit 101 will be described with reference to FIG. The training data shown in FIG. 2 is an example in which labels 22 are assigned to character strings in a document 21. Specifically, labels 22 are assigned to constituent units (also called tokens), such as characters or morphemes, that make up the document 21, to specify an event range 23. For example, assuming that the document 21 contains events such as "crack in the pipe" and "water leaked," labels 22 of "B-Event," "I-Event," and "O" are assigned to the morphemes that make up the document 21, "As a result, we discovered that the crack in the pipe caused water to leak," to specify the event range 23. More specifically, "B-Event / I-Event / I-Event" are assigned to the morpheme "crack in the pipe," respectively, to define the event range 23 "crack in the pipe." Similarly, the event range 23 "water leaked" is defined.

[0018] "B-Event" indicates the start position of an event in document 21. "I-Event" indicates that it is an element that constitutes an event, following the structural unit to which "B-Event" is attached. "O" indicates that it is not an element that constitutes an event, that is, it is outside the event range.

[0019] Next, an example of an additional event generation process of the data generating device 10 according to this embodiment will be described with reference to the flowchart of FIG.

[0020] In step S301, the dividing unit 102 divides the training data into multiple partial data. The division method of the training data may be, for example, to divide the training data evenly into k partial data in order to perform k-fold cross validation. Note that the method is not limited to k-fold cross validation, and any method may be used as long as it generates appropriate partial data that can generate multiple trained model sets. In step S302, the learning unit 103 learns a model using multiple partial data sets and generates a trained model set consisting of multiple trained models. The learning process in the learning unit 103 will be described later with reference to FIGS. 4 and 5.

[0021] In step S303, the estimation unit 104 estimates the event range included in the document of the training data using the trained model set. The estimated event range is stored in the estimation result storage unit 105. In step S304, the estimation unit 104 determines whether the event range estimation process using the trained model set in step S303 has been performed a predetermined number of times. Specifically, for example, a counter is set, and the counter value is incremented by one each time the event range estimation process in step S303 is performed, and it is determined whether the counter value matches the predetermined number of iterations. If the event range estimation process has been performed the predetermined number of iterations, the process proceeds to step S306. If the event range estimation process has not been performed the predetermined number of iterations, the process proceeds to step S305.

[0022] In step S305, the dividing unit 102 divides the instruction data again into a plurality of partial data at a division position different from the previous division position for the instruction data, and then proceeds to step S302 to repeat the same processing. In step S306, the selection unit 106 compares the event ranges estimated for each trained model set between the trained model sets. As a result of the comparison, the selection unit 106 selects an event range that is not included in the training data.

[0023] In step S307, the selection unit 106 generates one or more event groups by grouping the multiple event ranges selected in step S306. For example, multiple event ranges whose overlapping degree is equal to or greater than a threshold are grouped into an event group. Details of the event group generation process in steps S306 and S307 will be described later with reference to FIG. 6. In step S308, the selection unit 106 selects one or more candidate groups that are more likely to be teaching omissions rather than estimation errors from among the one or more event groups.

[0024] In step S309, the determination unit 107 determines an additional event to be added to the teaching data from one or more candidate groups selected in step S308. In step S310, the adding unit 108 adds the determined additional event to the teaching data and registers it in the teaching data storage unit 101. That is, the adding unit 108 updates the teaching data stored in the teaching data storage unit 101. The updated teaching data is also called updated teaching data.

[0025] Next, model learning using a plurality of partial data and event estimation using the learned model in steps S301 to S303 will be described with reference to FIGS. 4 and 5. FIG. The upper diagram of FIG. 4 shows a conceptual diagram of partial data relative to teaching data, and the lower diagram of FIG. 4 is a table showing the allocation of partial data used for learning and estimation.

[0026] In this embodiment, it is assumed that five-fold cross-validation is performed. That is, in the upper diagram of FIG. 4, the training data is divided into five partial data 401, labeled "A" through "E." Here, of the five partial data 401, four partial data 401 are used as training data, and the remaining partial data 401 is used as estimation data. For example, if the training data is a document consisting of 10,000 sentences, it is sufficient to divide it into five partial data of 2,000 sentences each, with 8,000 sentences used as training data and the remaining 2,000 sentences used as estimation data.

[0027] Specifically, as shown in the lower diagram of FIG. 4, when partial data "B, C, D, E" are used as training data, a model is trained using the four partial data "B, C, D, E" of the training data, and the remaining partial data A is used as estimation data "A." An existing method may be used to train the model. For example, only documents from the training data "B, C, D, E" are used as input data, and the model is trained using the documents from the training data "B, C, D, E" and the sets of labels assigned to the documents as correct answer data. The difference between the output data from the model for the input data and the correct answer data is evaluated using an error function, and backpropagation processing is performed to minimize the error function, thereby generating a trained model. For ease of explanation, the trained model that estimates estimation data A is referred to as "trained model A." The estimation unit 104 uses trained model A to estimate the event range included in estimation data A.

[0028] Next, when the training data is replaced and "A, C, D, E" is used as the training data, the model is trained using the four partial data of the training data "A, C, D, E", and trained model B is generated in the same manner as trained model A. The estimation unit 104 uses trained model B to estimate the event range included in the estimation data "B".

[0029] In this way, the training data and the estimation data are swapped in order so that all partial data are assigned as estimation data, and the event range estimation process is performed using the trained model. As a result, the event range estimation process can be performed once for the entire document of the training data by the event range estimation process from trained model A to trained model E.

[0030] Here, the five trained models A to E shown in Figure 4 are collectively referred to as trained model set 1. In the example of Figure 4, the first event range estimation process is performed using trained model set 1.

[0031] Next, FIG. 5 shows a case where the dividing unit 102 divides the teaching data at a position different from the dividing position of the teaching data in the upper diagram of FIG. The upper diagram of Figure 5 is a conceptual diagram of partial data similar to the upper diagram of Figure 4, but the instruction data is divided at a different position than the upper diagram of Figure 4. The dashed lines indicate the division positions shown in the upper diagram of Figure 4, and the solid lines indicate the new division positions. For example, a portion of the beginning of the instruction data is part of partial data "E'". In this way, multiple new partial data "A', B', C', D', E'" are generated.

[0032] The lower diagram of Figure 5, like the lower diagram of Figure 4, is a table showing the allocation of partial data used for learning and estimation. The learning unit 103 and the estimation unit 104 process model learning and event range estimation using the trained model in the same way as in Figure 4. As a result, a second event range estimation process is performed on all documents in the training data using trained model set 2 "A', B', C', D', E'".

[0033] Because the document is divided at different positions, the sets of sentences (character strings) included in the partial data are different between the cases of Figure 4 and Figure 5. Therefore, the trained model, which is the learning result using the partial data, is also different between the cases of Figure 4 and Figure 5. In this way, the division unit 102 generates multiple sets of multiple partial data with different division positions, making it possible to perform k-fold cross-validation multiple times and smooth out fluctuations in the estimation results for each trained model.

[0034] 4 and 5, it is assumed that the content of the partial data is changed for each round of the event range estimation process by changing the division position in the instruction data, but this is not limited to this. For example, the division position for the instruction data may not be changed, and the sentences in the instruction data may be randomly rearranged and then partial data may be generated for each round of the event range estimation process. In other words, any method for generating partial data may be used as long as the sentences included in the partial data are set to be different for each round of the event range estimation process.

[0035] Furthermore, as long as the event range estimation process is performed multiple times for the entire document of the training data, the process is not limited to performing k-fold cross-validation using partial data multiple times. For example, models with multiple different network structures may be trained in advance using other training data, and the event range estimation process may be performed for the entire document of the training data using the trained models with different network structures. For example, by preparing multiple models with different network structures, such as a recursive neural network (RNN) model, a long short-term memory (LSTM) model, a transformer model, or a BERT model, and performing the event range estimation process, different event range estimation results can be obtained.

[0036] Furthermore, multiple different trained models may be generated by training a certain model with different hyperparameters, such as the number of layers of the neural network, the number of units, the activation function, the dropout rate, etc. Since different hyperparameters are expected to result in different output results for the trained model to some extent, it is possible to obtain estimation results for multiple different event ranges.

[0037] Furthermore, the results of manually setting event ranges for documents of teaching data by multiple users may be used. Since it is conceivable that the ranges recognized as event ranges differ depending on the user, different event range estimation results can be obtained.

[0038] Next, a method for generating an event group will be described with reference to FIG. Figure 6 shows the event ranges obtained by multiple event range estimation processes using a trained model set (simply referred to as a model set in Figure 6). The horizontal direction in Figure 6 indicates the direction in which sentences progress in the document of the training data. The vertical direction in Figure 6 indicates the type of trained model set.

[0039] For ease of explanation, character strings are illustrated with dashed lines, and the event ranges in the training data and the event ranges 601 estimated for each model set are shown. Here, an example is described in which multiple partial data are generated four times at different division positions for the training data, and the event range estimation process is performed four times using different trained model sets 1 to 4. Because the trained model sets 1 to 4 are each different, the estimated event ranges 601 are different even for the same document.

[0040] The selection unit 106 selects an estimated event range that does not appear in the teaching data from the event ranges 601 estimated for each model set. As a method for determining whether an event range does not appear in the teaching data, for example, if a range of character strings estimated as an event range by a trained model set overlaps even partially with a character string in the event range in the teaching data, the selection unit 106 determines that the estimated event range appears in the teaching data. On the other hand, if the estimated range of character strings does not overlap with the event range of the teaching data, the selection unit 106 may determine that the estimated event range does not appear in the teaching data.

[0041] Furthermore, the selection unit 106 may determine that the estimated event range does not appear in the teaching data when the degree of overlap between the estimated event range and the event range of the teaching data is less than a threshold value. Furthermore, the selection unit 106 may determine that the estimated event range does not appear in the teaching data when the last n morphemes (n is a positive number equal to or greater than 1) in the estimated event range do not overlap with the teaching data.

[0042] Next, the selection unit 106 generates an event group 610 by grouping together events with similar event ranges 601 from among the event ranges 601 that do not appear in the training data. A method for determining whether the event ranges 601 are similar may involve cross-sectionally comparing the event ranges of each trained model set, and determining that the event ranges are similar if the character strings in the event ranges overlap by one or more characters. Note that the event ranges 601 may be determined to be similar if the degree of overlap of the character strings in the event ranges 601 is equal to or greater than a threshold, for example, if the overlap is equal to or greater than n percent. Alternatively, the event ranges 601 may be determined to be similar if any of the last n morphemes in the event ranges 601 overlap. Furthermore, a combination of these determination methods or another determination method may be used. Note that the selection unit 106 does not generate an event group for the third event range 601 of each model set along the sentence progression direction, since this event range overlaps with the training data.

[0043] In the example of FIG. 6, three event groups 610, 611, and 612 are generated, which are groups in which the event ranges estimated by each trained model set overlap, using a determination method that generates an event group when "character strings in the event range overlap by one or more characters." For example, in event group 610, the event ranges estimated by each trained model set are not the same character string, but include estimated variations. To specifically explain event group 610, consider a case in which each trained model set estimates an event range for the sentence "There were no welding defects in the pipes." In model set 1, "poor welding in the pipes" is estimated as event range 601, and in model set 3, "the defect is" is estimated as event range 601.

[0044] Next, an example of selecting a candidate group including an event range to be added from an event group will be described with reference to FIG. The selection unit 106 selects an event group containing a number of events equal to or greater than the threshold as the candidate group 701. In the example of FIG. 7 , for example, if the threshold is set to “3,” the number of events included in the event group 610 is “4,” the number of events included in the event group 611 is “4,” and the number of events included in the event group 612 is “2,” so the selection unit 106 selects the event group 610 and the event group 611 as the candidate group 701. Note that the selection unit 106 may select an event group in which the number of event ranges 601 included in the event group is equal to or greater than a predetermined ratio relative to the number of times the event range estimation process is performed as the candidate group 701. Specifically, for example, if the predetermined ratio is 70 percent and the event range estimation process is performed 10 times, the selection unit 106 selects an event group containing seven or more event ranges as the candidate group 701. This allows for identifying event ranges that do not exist in the training data by majority vote while taking into account estimated fluctuations, thereby improving the likelihood of adding only training omissions rather than estimation errors to the trained model.

[0045] Next, an example of determining an additional event will be described with reference to FIG. FIG. 8 shows the candidate group 701 shown in FIG. 7. The determination unit 107 determines an additional event from the event ranges included in the candidate group 701. As a method for determining an additional event, for example, the event range 601 having the largest number of identical character strings selected as an event range among the event ranges belonging to the candidate group 701 is determined as the additional event 801. For example, in the example of FIG. 8, in the first candidate group 701 (event group 610) in the direction of the sentence progression, the event ranges 601 estimated by model set 3 and model set 4 are identical character string ranges, so the number of identical event ranges selected is "2." The event ranges estimated by other model sets 1 and 2 are not identical to the other event ranges in the first candidate group, so the number of identical event ranges selected is "1." Therefore, the determination unit 107 determines the event ranges estimated by model set 3 and model set 4 in the first candidate group 701 as the additional event 801. Similarly, in the second candidate group 701 (event group 611), the event ranges estimated by model set 2 and model set 4 are the same character string range, and the selected number is "2." Also, since the selected number for the other event ranges of model set 1 and model set 3 is "1," the event ranges 601 estimated by model set 2 and model set 4 are determined as additional events 801.

[0046] Note that even if an event range satisfies the conditions of the above-described method for determining an additional event, if the event range ends with an unnatural part of speech such as a particle, a colon, parentheses, or other specific symbols, the determination unit 107 may not determine the event range as an additional event 801. Furthermore, if there are multiple non-overlapping event ranges ranked high in the candidate group in terms of the number of overlapping event ranges, the determination unit 107 may determine the multiple non-overlapping event ranges as additional events 801, or may use a combination of at least one of these determination methods.

[0047] Furthermore, when registering an additional event in the training data, the adding unit 108 may also register a weight for each sentence constituting a document, the weight for the sentence including the event range from which the event group was generated. For example, when an event group is generated, a sentence including an event range belonging to the event group is a part that was not previously labeled in the training data, and may have low reliability as training data. Therefore, the adding unit 108 may weight the sentence including the event range from which the event group was generated so that it is weighted lower than the sentence including the event range previously assigned to the training data. Furthermore, the adding unit 108 may weight the token label so that the weight of only the range of the additional event is lower, rather than the weight of the entire sentence. Furthermore, the adding unit 108 may weight the labels of all tokens constituting a sentence so that the weights are lower.

[0048] Next, an example of using the event range generated by the data generating device 10 according to this embodiment will be described with reference to FIGS. The left image in Figure 9 shows the document to be processed, and assumes that the event range has already been extracted, as in the case of instruction data. The extracted event range is displayed in a box. In this way, the event range is extracted from the target document, a process known as sequence labeling. The right image in Figure 9 is a graph showing the causal relationships between events. Causal relationships between events can be inferred and the relationships can be displayed.

[0049] FIG. 10 shows a case where an additional event is added by the data generating device 10 to the target document shown on the left side of FIG. Assume that the data generating device 10 according to this embodiment executes an event range estimation process on a target document, and adds an event range of "models with flood prevention measures" as an added event 1001. In this way, if the target document is training data, even if the training data has omitted setting an event range, the event range that should have been labeled can be added as an added event 1001. In addition, the event range estimation results and additional events may be used not only for estimating causal relationships but also as target data for keyword searches, and can be applied to any application where there is an advantage in extracting the event range completely.

[0050] The learning unit 103 may train a model using updated teaching data that has been updated by adding an additional event to existing teaching data, thereby generating a trained model. By learning using the updated teaching data, a trained model with a high recall rate can be generated, and an appropriate event range can be extracted.

[0051] Next, an example of the hardware configuration of the data generating device according to the above embodiment is shown in FIG. The data analysis device includes a CPU (Central Processing Unit) 31, a RAM (Random Access Memory) 32, a ROM (Read Only Memory) 33, a storage 34, a display device 35, an input device 36, and a communication device 37, all of which are connected via a bus. Note that the display device 35 does not necessarily have to be included as a hardware configuration of the data generation device 10.

[0052] The CPU 31 is a processor that executes arithmetic processing, control processing, etc. in accordance with a program. The CPU 31 uses a predetermined area of ​​the RAM 32 as a working area and executes various processes in cooperation with programs stored in the ROM 33, storage 34, etc. The RAM 32 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory), and functions as a work area for the CPU 31. The ROM 33 is a memory that stores programs and various types of information in a non-rewritable manner.

[0053] The storage 34 is a device that writes and reads data to a magnetic recording medium such as an HDD, a semiconductor storage medium such as a flash memory, a magnetically recordable storage medium such as an HDD (Hard Disc Drive), an optically recordable storage medium, etc. The storage 34 writes and reads data to the storage medium in accordance with control from the CPU 31. The display device 35 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 35 displays various information based on a display signal from the CPU 31.

[0054] The input device 36 is an input device such as a mouse, a keyboard, etc. The input device 36 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 31. The communication device 37 communicates with external devices via a network under the control of the CPU 31 .

[0055] According to the present embodiment described above, multiple event range estimation processes are performed on the document of the training data using multiple different methods, and an event group is generated based on the overlap of the event ranges obtained in each estimation process. From the event group, additional events, which are event ranges to be added to the training data, are determined and registered in the training data. This allows data that should originally be labeled as an event range to be added to the training data, even though it has not been labeled as an event range. Furthermore, for example, if all event ranges estimated by a trained model that are not in the training data are simply added as positive examples, the recall rate will increase, but there is also the possibility that they are simply estimation errors, and they will be registered as noise data, which may reduce the precision rate. However, according to this embodiment, for example, by using k-fold cross-validation, the event range estimation process is performed multiple times for the training data documents using different trained model sets, and by taking into account the overlap of the event ranges obtained with each trained model set, it is possible to increase the probability of determining a more likely event range as an additional event, rather than an estimation error. As a result, the quality of the dataset can be improved.

[0056] The instructions shown in the processing procedures described in the above-described embodiments can be executed based on a software program. A general-purpose computer system can store this program in advance and, by loading this program, achieve effects similar to those achieved by the control operations of the data generation device and learning device described above. The instructions described in the above-described embodiments can be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, or Blu-ray Disc), a semiconductor memory, or a similar recording medium. The recording medium may take any storage format as long as it is readable by a computer or embedded system. A computer can load the program from the recording medium and execute the instructions described in the program on a CPU based on the program, thereby achieving operations similar to those of the data generation device and learning device described in the above-described embodiments. Of course, the computer may acquire or load the program via a network. In addition, an OS (operating system), database management software, network middleware, etc. running on a computer may execute some of the processes required to realize this embodiment based on instructions from a program installed on the computer or embedded system from a recording medium. Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like. Furthermore, the number of recording media is not limited to one, and cases where the processing in this embodiment is executed from multiple media are also included in the recording media in this embodiment, and the media may have any configuration.

[0057] The computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored on a recording medium, and may be configured as either a device consisting of a single device such as a personal computer or a microcomputer, or a system in which multiple devices are connected to a network. Furthermore, the computer in this embodiment is not limited to a personal computer, but also includes an arithmetic processing unit, a microcomputer, etc. included in information processing equipment, and is a general term for equipment or devices that can realize the functions in this embodiment by a program.

[0058] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0059] 10...data generation device, 21...document, 22...label, 23,601...event range, 31...CPU, 32...RAM, 33...ROM, 34...storage, 35...display device, 36...input device, 37...communication device, 101...teaching data storage unit, 102...division unit, 103...learning unit, 104...estimation unit, 105...estimation result storage unit, 106...selection unit, 107...determination unit, 108...addition unit, 401...partial data, 610,611,612...event group, 701...candidate group, 801,1001...addition event

Claims

1. a selection unit that selects an event group that is different from a first event range that is a range of character strings specified for a predetermined purpose and that is defined for a document of the teaching data and that overlaps at least a portion of a plurality of second event ranges that are ranges of character strings related to the predetermined purpose and that are estimated for the document using a plurality of different methods; a determination unit that determines a second event range having a large number of event ranges with the same character string among the plurality of second event ranges in the event group as an additional event that is a second event range to be added to the teaching data; A data generating device comprising:

2. 2. The data generating device according to claim 1, wherein the selection unit performs a cross-sectional comparison between the plurality of second event ranges, and selects the plurality of second event ranges as the event group when a degree of overlap of character strings is equal to or greater than a threshold.

3. The data generating device according to claim 1 or claim 2, further comprising an estimation unit that estimates the second event range for the document for each of a plurality of different trained models trained using the teaching data.

4. a division unit that divides the teaching data into a plurality of partial data; a learning unit that learns a model using some of the plurality of partial data and generates a trained model; an estimation unit that estimates the second event range for a sentence corresponding to remaining partial data among the plurality of partial data by using the trained model, 3. The data generating device according to claim 1, wherein the generation of the trained model and the estimation of the second event range are repeated so that the second event range is estimated for each of the plurality of partial data.

5. the dividing unit generates a plurality of sets of the plurality of partial data by dividing the teaching data at different positions; The learning unit generates a trained model set including a plurality of trained models for each of the plurality of partial data sets, The data generating device according to claim 4 , wherein the estimation unit estimates the second event ranges for each of the partial data sets using the trained model set.

6. 3. The data generating device according to claim 1, wherein each of the plurality of second event ranges estimated by the different methods is an event range set by a plurality of users for the document.

7. The teaching data is assigned a weight for each sentence or token that constitutes the document, The smaller the weight for the additional event, the greater the proportion of the influence of the additional event in training the model using training data including the additional event.

7. The data generation device according to claim 1, wherein a weight for the additional event is set in advance to be lower than a weight for a character string included in the teaching data in order to reduce an influence when learning a model using teaching data including the additional event.

8. A selection unit selects an event group that is different from a first event range, which is a range of character strings specified for a predetermined purpose, defined for a document of teaching data, and that overlaps at least a portion of a plurality of second event ranges, which are ranges of character strings related to the predetermined purpose estimated for the document by a plurality of different methods; a determination unit determining, among the plurality of second event ranges in the event group, a second event range having a large number of event ranges with the same character string as an additional event, which is a second event range to be added to the teaching data.

Images