Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device capable of generating new data on the basis of existing data and performing data augmentation while ensuring diversity of the data.SOLUTION: An information processing device comprises: a summary sentence generation unit to which a first text dataset, which is the original sentences assigned with a first annotation label, is input, the first annotation label being used as training data of a text classification model, and which generates the summary sentences of the original sentences; a summary sentence labeling unit which assigns a second annotation label to the summary sentence; and an augmented-dataset determination unit which determines whether to use a second text dataset, which is the summary sentences to which the second annotation labels are assigned as an augmented dataset for the training data of the text classification model on the basis of the first annotation label and the second annotation label.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program that expand learning data. [Background technology]

[0002] Data augmentation is a machine learning technique used to reduce overfitting when training machine learning models by training the model on several slightly modified copies of existing data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-185551 [Patent Document 2] Japanese Patent Application Publication No. 2020-126360 [Patent Document 3] Japanese Patent Application Publication No. 2019-220144 [Non-patent literature]

[0004] [Non-Patent Document 1] Shorten, C., & Khoshgoftaar, TM (2019). A survey on image data augmentation for deep learning. Journal of big data, 6(1), 1-48. [Non-patent document 2] Wei, J., & Zou, K. (2019). Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196. Summary of the Invention [Problem to be solved by the invention]

[0005] The method used in Patent Documents 1 and 2 is a thesaurus method using a thesaurus, which is a data expansion method mainly used in natural language processing where data expansion is difficult.

[0006] There are two main reasons why data augmentation in natural language processing is difficult. First, because natural language is a grammatical entity, changing the order of words changes the meaning of an existing sentence. Second, because natural language is a sentence in which all words are organically linked to create meaning, adding or removing words arbitrarily can cause the original meaning of the sentence to be lost.

[0007] To solve this problem, natural language data expansion is typically performed under strict rules. The thesaurus method described above is the most representative method. However, the thesaurus method has the following three problems. First, it is difficult to ensure data diversity by simply replacing arbitrary words with synonyms. Second, according to Non-Patent Document 2, thesaurus methods perform worse than methods that arbitrarily delete or rearrange words in a sentence. Third, because they focus on ensuring data quantity rather than data quality, they are not suitable for training currently complex deep learning models. In other words, the thesaurus method cannot create a robust model due to the above issues, resulting in little improvement in the performance of classification models that use current pre-trained language models.

[0008] In view of the above circumstances, an object of the present disclosure is to create new data based on existing data and to perform data expansion while ensuring data diversity. [Means for solving the problem]

[0009] An information processing device according to an embodiment of the present disclosure includes: a summary generator that receives a first text dataset, which is an original text to which a first annotation label is added and is used as training data for a text classification model, and generates a summary of the original text; an abstract labeling unit that assigns a second annotation label to the abstract; an extended dataset determination unit that determines, based on the first annotation label and the second annotation label, whether to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extended dataset of the training data of the text classification model; It is equipped with:

[0010] An information processing method according to an embodiment of the present disclosure includes: The computer of the information processing device A first text dataset is input, the first text dataset being an original text to which a first annotation label is added, and the original text is used as training data for a text classification model. A summary of the original text is generated. assigning a second annotation label to the summary sentence; Based on the first annotation label and the second annotation label, it is determined whether to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extended dataset of the training data of the text classification model.

[0011] An information processing program according to an embodiment of the present disclosure includes: The computer of the information processing device, a summary generator that receives a first text dataset, which is an original text to which a first annotation label is added and is used as training data for a text classification model, and generates a summary of the original text; an abstract labeling unit that assigns a second annotation label to the abstract; an extended dataset determination unit that determines, based on the first annotation label and the second annotation label, whether or not to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extended dataset of the training data of the text classification model. Operate as. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to create new data based on existing data and perform data expansion while ensuring data diversity.

[0013] The effects described here are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates a functional configuration of an information processing device according to an embodiment of the present disclosure. [Figure 2] 1 shows an operation flow of the information processing device. [Figure 3] An example of a summary is shown below. [Figure 4] An example of verification of data augmentation will be described. [Figure 5] Examples of parameters are shown below. [Figure 6] The results of verifying data augmentation are shown below. [Figure 7] The results of verifying data augmentation are shown below. [Figure 8] 1 shows the difference between the results of data augmentation presented in this embodiment and data augmentation using existing thesaurus methods. DETAILED DESCRIPTION OF THE INVENTION

[0015] 1. Overview of this embodiment

[0016] Text classification is one of the most representative tasks in natural language processing, and has been developed using most natural language processing models, from machine learning to deep learning. Research into text classification is still ongoing to improve its accuracy.

[0017] To train a text classification model for natural language processing, supervised learning is performed in which annotation-labeled text data, in which annotation labels are attached to text, is used as training data, and the work of classifying text is carried out using the model obtained as a result of the learning.

[0018] In machine learning methods, achieving high classification accuracy requires the use of large amounts of annotated text data as training data. However, the process of assigning appropriate labels to text and generating training data is typically done manually and is costly. While it is possible to obtain text data via the Internet and generate training data, this can sometimes lead to legal issues due to issues such as personal information and copyright of the text data.

[0019] As a method for securing training data for machine learning, a method of expanding existing training data for machine learning has been proposed. This method, called data augmentation, is mainly used in the field of image processing. For example, as shown in Non-Patent Document 1, the accuracy of machine learning is improved by using an augmented data set in which the data is expanded by applying simple processing such as RGB conversion, rotation, and zoom to images that serve as training data.

[0020] Although data augmentation methods have been proposed for natural language processing, the simple data augmentation methods used in image processing cannot be applied due to the characteristics of natural language, which forms a single meaning through the organic connection of words different from images. Therefore, as in Non-Patent Document 2, data augmentation of language data is limited to limited methods such as thesaurus methods that replace any word in a sentence with a synonym, methods that delete any word, and methods that change the order of words.

[0021] The method described in Non-Patent Document 2 effectively improves performance in general statistically based machine learning and recurrent neural networks. However, this method has limitations in that it is difficult to ensure data diversity. This limitation means that there is no significant performance improvement in pre-trained language models, which have recently become mainstream in deep learning.

[0022] This embodiment aims to solve the above problems and ensure data diversity, and provides a data expansion method that can generate text with a grammatical structure different from that of existing documents while preserving the meaning of the existing documents, thereby ensuring the quality of the expanded data set, and a text classification program that uses the expanded data set.

[0023] This embodiment aims to solve the above problems by using a text generation AI that uses deep learning and performing data augmentation using generative summarization.

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0025] 2. Information processing equipment

[0026] FIG. 1 shows a functional configuration of an information processing device according to an embodiment of the present disclosure.

[0027] The information processing device 100 is used as an annotated text data expansion device for expanding annotation-labeled text data. The information processing device 100 includes an input unit 102, a calculation unit 103, and a storage unit 104. The input unit 102 includes a text data input unit 105 and a learning text data input unit 106. The calculation unit 103 includes a text data separation unit 107, a summary sentence generation unit 108, a summary sentence label assignment unit 109, and an expanded data set determination unit 110. The storage unit 104 includes a separated text data storage unit 111 and an expanded data set storage unit 112.

[0028] FIG. 2 shows the operation flow of the information processing device.

[0029] The text data input unit 105 inputs a first text dataset 210, which is original text 212 to which a first annotation label 211 has been assigned, to the text data separation unit 107. The first text dataset 210 is used as training data for the text classification model 114. The type of the first text dataset 210 is not particularly limited as long as it is text data to which a first annotation label 211 has been assigned to text, which is original text 212. A typical example of the first text dataset 210, which is original text 212 to which a first annotation label 211 has been assigned, is the IMDB dataset to which positive and negative labels have been assigned to movie reviews.

[0030] The text classification model 114 is constructed using a first text dataset 210, and when an arbitrary original sentence 212 is input, it assigns an annotation label 211 to the original sentence. The text classification model 114 is a text classifier that is trained by separating all existing data (i.e., the first text dataset 210, which is the original sentence 212 to which the first annotation label 211 has been assigned) into training data 201, test data 202, and validation data 203 in a ratio of 8:1:1.

[0031] The text data separation unit 107 separates a first text data set 210, which is an original text 212 to which a first annotation label 211 has been assigned, into training data 201, test data 202, and verification data 203 at a predetermined ratio (e.g., 8:1:1) (step S11). The first text data set 210 separated at this ratio is used as training data 201, which is 80% of the original data, test data 202, which is 10%, and verification data 203, which is 10%. The first text data set 210 separated into training data 201, test data 202, and verification data 203 is stored in a separated text data storage unit 111.

[0032] The training text data input unit 106 inputs the training data 201 (original sentences 212 to which first annotation labels 211 have been added) stored in the separated text data storage unit 111 to the summary generation unit 108 (step S12). The summary generation unit 108 generates a summary sentence 222 of the original sentences 212 of the training data 201 using the summary generation model 113 (step S13).

[0033] The summary generation model 113 is a deep learning model that is fine-tuned using a pre-trained language model, such as BART-Large, T5-Base, or T5-Large.

[0034] The summary sentence labeling unit 109 inputs the summary sentence 222 into the text classification model 114 and assigns the second annotation label 221 to the summary sentence 222. This generates a second text dataset 220, which is the summary sentence 222 assigned the second annotation label 221 (step S14).

[0035] The extended data set judgment unit 110 judges whether the first annotation label 211 assigned to the original text 212 is the same as the second annotation label 221 assigned to the summary text 222 of this original text 212 (step S15).

[0036] If the first annotation label 211 and the second annotation label 221 are the same (step S15, YES), the extended dataset determination unit 110 determines to use the second text dataset 220, which is the summary sentence 222 to which the second annotation label 221 is assigned, as the extended dataset 230. The extended dataset determination unit 110 stores the second text dataset 220, which is the summary sentence 222 to which the second annotation label 221 is assigned, in the extended dataset storage unit 112 (step S16). The extended dataset 230 is a text dataset that extends the first text dataset 210 (i.e., is added to the first text dataset 210).

[0037] On the other hand, if the first annotation label 211 and the second annotation label 221 are different (step S15, NO), the extended dataset determination unit 110 deletes the second text dataset 220, which is the summary sentence 222 to which the second annotation label 221 is assigned. Then, the information processing device 100 executes the processes from step S13 onwards on the next training data 201 (the original sentence 212 to which the first annotation label 211 is assigned).

[0038] As described above, the extended dataset determination unit 110 determines, based on the first annotation label 211 and the second annotation label 221, whether or not to use the second text dataset 220, which is the summary sentence 222 to which the second annotation label 221 is assigned, as the extended dataset 230 of the training data 201 of the text classification model 114.

[0039] The reason for checking whether the first annotation label 211 and the second annotation label 221 match for the newly generated summary sentence 222 is to ensure the quality of the extended dataset 230. The summary sentence 222 is text data that is shorter than the existing original sentence 212 and is generated by grasping the core meaning of the original sentence 212, which is source language data. If a portion that maintains the meaning of the existing original sentence 212 is deleted in the process of generating the summary sentence 222, the first annotation label 211 and the second annotation label 221 will not match, resulting in a decrease in the quality of the overall training data. In other words, to prevent the extended dataset 230 from disrupting the overall training data, summary sentences 222 that have a mismatch between the first annotation label 211 and the second annotation label 221 are deleted.

[0040] When classification is completed for all of the training data 201 (original sentences 212 to which first annotation labels 211 are assigned), the process of generating the extended data set 230 for the training data 201 ends (step S1).

[0041] The training text data input unit 106 inputs the second text data set 220 that the extended data set determination unit 110 has determined to be used as the extended data set 230 to the summary sentence generation unit 108. The summary sentence generation unit 108 generates a summary sentence of the original text (summary sentence 222) of the extended data set 230. Based on the training data and a predetermined ratio, a text classification process is performed using the annotation-labeled text data.

[0042] When the final training dataset is used for training the text categorization model 114, the text data separation unit 107 separates the first text dataset 210 into training data 201, test data 202, and verification data 203 at a predetermined ratio, but does not separate the test data 202 and verification data 203 from the extended dataset 230. In other words, even after the extended dataset 230 is created, only the first text dataset 210 is used for the test data 202 and verification data 203.

[0043] The final text classification model 114 is trained using the extended dataset 230 in the extended dataset storage unit 112 and the first text dataset 210 in the separated text data storage unit 111 (step S2). The training data 201, which accounts for 80% of the first text dataset 210 stored in the separated text data storage unit 111, and the extended dataset 230 in the extended dataset storage unit 112 are used to train the text classification model 114.

[0044] The text classification model 114 is tested and validated (step S3) using 10% of the test data 202 and 10% of the validation data 203 of the first text dataset 210. "Testing" refers to evaluating the final text classification model 114. "Validation" refers to fine-tuning the parameters of the text classification model 114.

[0045] The machine learning model used to train the text classification model 114 may be a classification model fine-tuned using a pre-trained language model. For example, three models may be used to validate data augmentation methods: a Convolutional Neural Network (CNN), a Bidirectional Gated Recurrent Unit (GRU) with Bahdanau Attention, and a Bidirectional Encoder Representations from Transformers (BERT-Base).

[0046] 3. Working Example

[0047] Figure 3 shows examples of summary data obtained using three methods (T5-Base (Example 1), T5-Large (Example 2), and BART-Large (Example 3)) for arbitrary text data from the IMDB dataset, which is movie review data.

[0048] Example 1 is an example of a summary created using T5-Base, which is a fine-tuned summary model using a pre-trained language model T5-Base for the original text. Compared to the existing original text, it can be seen that there are no overlapping sentences and that the new sentence was created using only the core of the sentence.

[0049] Example 2 is an example of a summary sentence using T5-Large, which is a fine-tuned pre-trained language model using the same original text as in Example 1, with the T5-Large as the summary model. The difference between T5-Base and T5-Large is the number of parameters used to train the language model; all other aspects are the same. As in Example 1, Example 2 was also compared with the original text, and no overlapping sentences were found.

[0050] Example 3 shows an example of a summary sentence generated using BART-Large, a pre-trained language model fine-tuned as a summary model for the original text. In Example 3, it can be seen that overlaps occur in all sentences compared with the original text. The relevant sentences are shown as 301, 302, and 303 in Figure 3.

[0051] The datasets used for validation were the IMDB dataset and the Amazon® Review dataset. The IMDB dataset contains 50,000 pieces of text data with annotation labels. The Amazon® Review dataset contains 1.6 million pieces of text data with annotation labels. Because the Amazon® Review dataset is too large for this validation, we randomly selected 50,000 pieces of data, the same size as the IMDB dataset.

[0052] A dataset with 50,000 annotation labels was divided into proportions of 1%, 2%, 4%, 10%, 20%, 25%, 50%, and 100%, and data expansion according to the size of the dataset was verified. Accuracy was used as the metric for verification, and since the proportions of positive and negative labels were similar as shown in Figure 4, no other measures were taken to balance the data. Tests for all proportions were performed independently, with the hyperparameters set as shown in Figure 5, and the final accuracy obtained for each training run was averaged.

[0053] Figures 6 and 7 show the results of verifying data augmentation. The number of data refers to the total amount of data before data augmentation. For example, if the number of data is 500, 400 pieces of annotation-labeled data, or 80% of 500, are used for training, and test data 202 and validation data 203, each accounting for 10%, or 50 pieces of data each, are used for testing and validation. Data augmentation is then performed only on the 400 pieces of data separated for training. In addition, considering the possibility of differences in accuracy due to differences in data quality between data intervals, a classification model was trained and verified for all data intervals.

[0054] The numbers at the top of Figures 6 and 7 are the accuracy results after data augmentation, and the numbers at the bottom are the difference from the accuracy without data augmentation. Upward and downward triangles are used to visualize the difference. An upward triangle 305 means that the model's performance improved after data augmentation. A downward triangle means that performance decreased after data augmentation. For example, the number 66.92 (upward triangle 6.45) 304 in Figure 6 means that data augmentation improved performance by 6.45% compared to the conventional model.

[0055] Figure 8 shows a table showing the differences between the results of data augmentation presented in this embodiment and data augmentation using existing thesaurus methods. For thesaurus data augmentation, we used the EDA method described in Non-Patent Document 2. We performed all four methods described in that paper, calculated the averages, and compared the results. The metrics are the same as those in Figure 6. Based on the same amount of data augmented, the above figures represent the accuracy of the data augmentation presented in this embodiment, and the following figures represent the differences from data augmentation using existing thesaurus methods.

[0056] CNN, a relatively simple deep learning model, showed significant performance improvements in many comparisons, while the complex models RNN and BERT showed significant performance improvements when there was little data, but when there was sufficient data, there was no improvement and in fact performance often decreased. However, when there was little data, performance definitely improved, and there was a clear difference in performance compared to Figure 8, which used the conventional thesaurus method. There was a significant performance gap between the simple model CNN and the complex model BERT, and there were cases where performance was significantly higher even when there was a sufficient amount of data.

[0057] This embodiment can be applied to the creation of training data for machine learning of text classification models. In this embodiment, only generative summarization models fine-tuned using BART-Large, T5-Base, and T5-Large were used for data augmentation for verification. However, as new generative summarization models are released over time, the scope of data augmentation is not limited to the examples shown here.

[0058] 4. Conclusion

[0059] The goal of data augmentation is to create new data based on existing data while ensuring data diversity. However, existing natural language data augmentation methods use simple methods such as replacing specific words with words with similar meanings or changing the order of words, making it impossible to ensure diversity. Data augmentation methods using generative AI such as GANs (generative adversarial networks) are also being researched, but they do not offer significant differentiation in that they simply generate data similar to existing data. I propose data augmentation using generative summarization to solve existing problems. Generative summarization is a method of summarizing an original document by generating new sentences that reflect the core context, even if they were not present in the original document. Natural language data augmentation using generative summarization has the following three advantages over existing text data augmentation methods using thesaurus.

[0060] First, it ensures data diversity by generating new sentences that did not exist in the original document. Second, it can remove relatively unnecessary parts from the document and use only the core parts as data. Third, since deep learning generative summarization models are diverse, data can be secured for each model.

[0061] The generation of new sentences that differ from existing documents and their use as data has been explained using the example in Figure 3. In many cases, new sentences were generated, ensuring data diversity.

[0062] By deleting relatively unnecessary parts from the document and retaining only the core content that is useful for text classification, we can also confirm the performance improvement due to data expansion, as shown in Figures 6, 7, and 8.

[0063] As performance improvements were observed in all generative summarization models, the scope of this data extension can be expanded as other generative summarization models are devised in the future.

[0064] According to this embodiment, a generative summarization method for natural language processing mechanically expands a small amount of text data with annotation labels, matches the labels of existing data, prevents performance degradation due to data expansion, and generates text data that is grammatically correct for natural language. Going beyond existing text data expansion methods that rely on thesaurus methods, the present invention provides a method for expanding annotated text data, a program for expanding annotated text data, an apparatus for expanding annotated text data, and a method for training a text classification model, which ensure data diversity by creating new sentences and improve the quality of the text data expansion.

[0065] Although the embodiments and modified examples of the present technology have been described above, the present technology is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present technology. [Explanation of symbols]

[0066] 100 Information processing device 102 Input section 103 Arithmetic section 104 Storage section 105 Text data input section 106 Learning text data input section 107 Text data separation unit 108 Summary sentence generator 109 Summary Labeling Unit 110 Extended Data Set Judgment Unit 111 Separated text data storage unit 112 Extended Data Set Storage 113 Summary Generation Model 114 Text Classification Model 201 training data 202 Test Data 203 Verification Data 210 First Text Dataset 211 First Annotation Label 212 Original text 220 Second Text Dataset 221 Secondary Annotation Label 222 Summary 230 Extended Dataset

Claims

1. a summary generator that receives a first text dataset, which is an original text to which a first annotation label is added and is used as training data for a text classification model, and generates a summary of the original text; an abstract labeling unit that assigns a second annotation label to the abstract; an extended dataset determination unit that determines, based on the first annotation label and the second annotation label, whether to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extended dataset of the training data of the text classification model; An information processing device comprising:

2. 2. The information processing device according to claim 1, The summary generation unit generates the summary using a summary generation model, which is a deep learning model fine-tuned using a pre-trained language model. Information processing device.

3. 2. The information processing device according to claim 1, the text classification model is constructed using the first text dataset, and when an arbitrary original sentence is input, assigns an annotation label to the original sentence; the summary label assignment unit assigns the second annotation label by inputting the summary to the text classification model; The extended data set determination unit determines, when the first annotation label and the second annotation label are identical, to use a second text data set, which is the summary sentence to which the second annotation label is assigned, as the extended data set, thereby extending the first text data set. Information processing device.

4. 4. The information processing device according to claim 3, further comprising a text data separation unit that separates the first text data set into the training data, test data, and verification data at a predetermined ratio; At least a portion of the first text data set separated as the training data is input to the summary generator and used for training to fine-tune the text classification model. Information processing device.

5. 5. The information processing device according to claim 4, The extended data set is input to the summary generation unit, The summary generator generates a summary of the original text of the extended dataset, and the summary is used for training to fine-tune the text classification model. Information processing device.

6. 6. The information processing device according to claim 5, the first text data set is separated into the training data, the test data, and the validation data in predetermined proportions and used to train the text classification model; The augmented data set is not separated into test data and validation data and is used to train the text classification model. Information processing device.

7. The computer of the information processing device A first text dataset is input, the first text dataset being an original text to which a first annotation label is added, and the original text is used as training data for a text classification model. A summary of the original text is generated. assigning a second annotation label to the summary sentence; Based on the first annotation label and the second annotation label, it is determined whether to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extension dataset of the training data of the text classification model. Information processing methods.

8. The computer of the information processing device, a summary generator that receives a first text dataset, which is an original text to which a first annotation label is added and is used as training data for a text classification model, and generates a summary of the original text; an abstract labeling unit that assigns a second annotation label to the abstract; an extended dataset determination unit that determines, based on the first annotation label and the second annotation label, whether or not to use a second text dataset, which is the summary sentence to which the second annotation label is assigned, as an extended dataset of the training data of the text classification model. An information processing program that operates as a

Images