Document classification learning device, document classification device and program

Abstract

To improve accuracy in document classification by training a language model so that teacher data of low-frequency labels is not deficient and an influence of restrictions on the number of tokens of input data is not received.SOLUTION: A division unit 10 of a learning device for document classification 1 inputs the whole of a text and a label corresponding to it and divides the text into sentence units in accordance with a preliminarily set rule to generate a plurality of divided texts consisting of a prescribed number of sentences and gives the same label as the inputted label to generate labeled teacher data of the divided texts. A learning unit 11 uses the labeled teacher data of the divided texts to perform fine tuning of a pre-trained language model stored in a storage unit 12.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to document classification technology for natural language processing, and in particular to a document classification learning device that fine-tunes a pre-trained natural language processing model (hereinafter referred to as a "language model"), a document classification device that classifies documents using the language model learned by the document classification learning device, and a program. [Background technology]

[0002] A conventional technique for solving classification problems in a specific field is known, which is to develop BERT (Bidirectional Encoder Representations from Transformers). This technique pre-trains a language model using general-purpose data, such as Wikipedia (registered trademark), which is composed of vocabulary that is not biased toward a specific field, and fine-tunes the pre-trained language model using data from the specific field when implementing the classification problem.

[0003] BERT is a general-purpose language model that incorporates an architecture called a Transformer.

[0004] BERT is pre-trained by using general-purpose data, such as Wikipedia (registered trademark), which is composed of vocabulary that is not biased towards a specific field, as training data. BERT is fine-tuned by using labeled training data from a specific field in which a classification problem is to be performed on a pre-trained general-purpose language model.

[0005] For example, BERT BASE The model consists of 12 layers, a 768-dimensional hidden layer, and 12 attention layers.

[0006] In recent years, pre-trained general-purpose language models such as BERT (see, for example, Non-Patent Document 2) and RoBERTa (see, for example, Non-Patent Document 3), which are based on Transformers (see, for example, Non-Patent Document 1), have been applied to machine translation, automatic summarization, etc.

[0007] Research is also underway into language models specialized for specific domains or languages, such as PubMedBERT (see, for example, Non-Patent Document 4), which is specialized for the medical field, and the Japanese version of BERT (see, for example, Non-Patent Document 5).

[0008] FIG. 13 is a block diagram showing an example of the configuration of a conventional document classification learning device and document classification device, and FIG. 14 is a flowchart showing an example of the processing thereof.

[0009] This document classification learning device 101 is a device that learns a language model for categorizing documents, and includes a learning unit 110 and a storage unit 111 that stores pre-trained language models.

[0010] The learning unit 110 receives labeled training data, which is the entire sentence that is the gist of a document and its label (classification data that indicates the category of the document), as training data (step S1401).The learning unit 110 then uses the labeled training data to fine-tune the language model stored in the storage unit 111 (step S1402).As a result, a language model fine-tuned for document classification is generated.

[0011] The document classification device 102 is a device that classifies documents into categories, and includes an estimation unit 112 and a storage unit 113 that stores a language model fine-tuned by the document classification learning device 101 .

[0012] The estimation unit 112 inputs a sentence that is the summary of the document to be classified (step S1403), estimates a label for the document using a language model stored in the memory unit 113 (step S1404), and outputs a label that is the classification result of the document (step S1405).

[0013] However, when fine-tuning a pre-trained language model, the distribution of labels in the labeled training data, which is training data, may become biased. It has been reported that when a language model is fine-tuned with a small amount of training data for some labels, classification accuracy decreases when documents are classified using that language model (see, for example, Non-Patent Document 6).

[0014] In addition, the number of tokens that can be input during fine tuning (the smallest unit when inputting to a language model (almost the same as the number of words)) depends on the size of the input data, and in BERT, a language model based on a transformer, there is a limit of 512 tokens. [Prior art documents] [Non-patent literature]

[0015] [Non-Patent Document 1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, ANGomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need”, NIPS 2017, pp. 5998-6008 (2017) [Non-patent document 2] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of NAACL-HLT 2019, pp. 4171-4186 (2019) [Non-patent document 3] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa:A Robustly Optimized BERT Pretraining Approach”, arXiv preprint arXiv:1907.11692(2019) [Non-patent document 4] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Domain-specific language model pretraining for biomedical natural language processing”, arXiv:2007.15779(2020) [Non-Patent Document 5] "NICT BERT Japanese Pre-trained Model", [online], National Institute of Information and Communications Technology, Data-Driven Intelligent Systems Research Center, [Retrieved January 26, 2022], Internet<URL:https: / / alaginrc.nict.go.jp / nict-bert / index.html> [Non-patent document 6] J. Phang, T. Fevry, and SRBowman, “Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks”, arXiv:1811.01088(2018) Summary of the Invention [Problem to be solved by the invention]

[0016] Fig. 15 is a diagram illustrating that, in the prior art, a part of the gist (sentence) of a document, which is input data, is truncated. As shown in Fig. 15, a language model such as BERT is fine-tuned during learning using the entire sentence that is the gist of the document and its label as learning data. Furthermore, when estimating labels for classifying documents, the language model is used to estimate labels for the sentence that is the gist of the document to be classified.

[0017] As mentioned above, when fine-tuning a language model, the maximum number of data tokens that can be input is 512. Therefore, if the sentence that is the gist of a document exceeds 512 tokens, the gist of the document will be divided into a part that can be input to the language model and a part that cannot be input to the language model, as shown in Fig. 15.

[0018] For this reason, in the conventional document classification learning device 101 shown in Figure 13, when fine-tuning a language model, any sentence exceeding 512 tokens is truncated, making it impossible to construct a language model that corresponds to the entire sentence, which is the gist of the document. Using such a language model to estimate a document label results in a problem of reduced classification accuracy.

[0019] 13, as mentioned above, the distribution of labels in the labeled training data used to fine-tune a language model can be biased. If the language model is fine-tuned with the biased label distribution and document labels are estimated using this language model, classification accuracy will decrease.

[0020] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a learning device for document classification, a document classification device, and a program that can improve the accuracy of document classification by learning a language model without a shortage of training data for low-frequency labels and without being affected by the limit on the number of tokens in the input data. [Means for solving the problem]

[0021] In order to solve the above problem, the document classification learning device of claim 1 is a document classification learning device that fine-tunes a pre-trained language model for classifying documents based on sentences contained in the documents and labels assigned to the documents, and is characterized by comprising: a first segmentation unit that inputs the sentences and the labels, segments the sentences into sentence units, and generates a plurality of segmented sentences consisting of a predetermined number of sentences; and a learning unit that fine-tunes the language model using labeled training data consisting of the segmented sentences generated by the first segmentation unit and the labels input by the first segmentation unit.

[0022] Furthermore, the document classification learning device of claim 2 is characterized in that, in the document classification learning device of claim 1, the text is assumed to be composed of n sentences (n is an integer greater than or equal to 2), and the first division unit generates n divided sentences, each of which consists of a predetermined number of sentences between 1 and (n-1).

[0023] Furthermore, the document classification device of claim 3 is a document classification device that estimates a label of a document to be classified using a language model fine-tuned by the document classification learning device of claim 1 or 2, and is characterized by comprising: a second segmentation unit that inputs a sentence contained in the document, segments the sentence into sentence units, and generates a plurality of segmented sentences each consisting of a predetermined number of sentences; an estimation unit that uses the language model to estimate a probability value for each of the labels of the segmented sentence for each of the plurality of segmented sentences generated by the second segmentation unit; and a label integrating unit that determines the label of the document by integrating the labels using the probability value for each of the labels for each of the plurality of segmented sentences estimated by the estimation unit.

[0024] Furthermore, the document classification device of claim 4 is characterized in that, in the document classification device of claim 3, the text is assumed to be composed of n (n is an integer greater than or equal to 2) sentences, and the second division unit generates n divided texts, each of which consists of a predetermined number of sentences between 1 and (n-1).

[0025] Furthermore, the program of claim 5 is characterized in that the program causes a computer constituting a learning device for document classification that fine-tunes a pre-trained language model for classifying a document based on sentences contained in the document and labels assigned to the document to function as a first segmentation unit that inputs the sentences and the labels, divides the sentences into sentence units, and generates a plurality of divided sentences consisting of a predetermined number of sentences, and a learning unit that fine-tunes the language model using labeled training data consisting of the divided sentences generated by the first segmentation unit and the labels input by the first segmentation unit.

[0026] The program of claim 6 is characterized in that it causes a computer constituting a document classification device that estimates labels of documents to be classified using a language model fine-tuned by the document classification learning device of claim 1 or 2 to function as: a second segmentation unit that inputs a sentence included in the document, divides the sentence into sentence units, and generates a plurality of divided sentences consisting of a predetermined number of sentences; an estimation unit that uses the language model to estimate a probability value for each of the labels of the divided sentences generated by the second segmentation unit; and a label integrating unit that determines the label of the document by integrating the labels using the probability value for each of the labels of the plurality of divided sentences estimated by the estimation unit. [Effects of the Invention]

[0027] As described above, according to the present invention, it is possible to train a language model without a shortage of training data for low-frequency labels and without being affected by the limit on the number of tokens in the input data. Furthermore, by using such a language model, it is possible to improve the accuracy of document classification. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating an overview of the present invention. [Figure 2] 1 is a block diagram showing an example of the configuration of a document classification learning device according to an embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an example of processing performed by a document classification learning device according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating the gist of a document (a sentence consisting of five sentences) and the sentences after division. [Figure 5] FIG. 10 is a diagram showing a specific example of a document summary (a sentence consisting of six sentences). [Figure 6] FIG. 10 is a diagram showing a specific example of a sentence after division into one sentence. [Figure 7] FIG. 10 is a diagram showing a specific example of a sentence after division into three sentences. [Figure 8] 1 is a block diagram illustrating an example of the configuration of a document classification device according to an embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating an example of processing performed by the document classification device according to the embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of the number of documents (number of papers) for each label. [Figure 11] 1A and 1B are diagrams showing experimental results of the prior art and an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating a specific example of an experimental result. [Figure 13] FIG. 1 is a block diagram showing an example of the configuration of a conventional document classification learning device and document classification device. [Figure 14] 1 is a flowchart showing an example of processing by a conventional document classification learning device and document classification device. [Figure 15]FIG. 1 is a diagram illustrating that, in the prior art, a part of the gist (sentence) of a document, which is input data, is truncated. DETAILED DESCRIPTION OF THE INVENTION

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. [Summary of the Invention] First, an overview of the present invention will be described. Fig. 1 is a diagram illustrating the overview of the present invention. The present invention is characterized in that, during learning, sentences included in a document to be learned are divided into sentence units, a plurality of sentences (post-division sentences) each consisting of a predetermined number of sentences are generated, and a pre-trained language model is fine-tuned using labeled training data of the plurality of post-division sentences.

[0030] Furthermore, the present invention is characterized in that, during estimation, the text contained in the document to be classified is divided into sentence units, multiple divided sentences consisting of a predetermined number of sentences are generated, a fine-tuned language model is used to estimate a probability value for each label for each of the multiple divided sentences, and the probability values ​​are used to integrate the multiple labels, thereby determining the label of the document to be classified.

[0031] This artificially increases the amount of training data for low-frequency labels during training, thereby resolving the shortage of training data for low-frequency labels. Also, since segmented sentences that are shorter than conventional sentences can be input to the language model, there is no need to censor sentences before inputting them to the language model.

[0032] Therefore, the language model can be trained without a shortage of training data for low-frequency labels and without being affected by the limit on the number of tokens in a sentence input to the language model.Using such a language model can improve the accuracy of document classification.

[0033] [Learning device for document classification] Next, a description will be given of a document classification learning device according to an embodiment of the present invention. Fig. 2 is a block diagram showing an example of the configuration of a document classification learning device according to an embodiment of the present invention, and Fig. 3 is a flowchart showing an example of the processing performed by the device.

[0034] This document classification learning device 1 is a device that fine-tunes a pre-trained language model for document classification based on sentences contained in the documents and labels assigned to the documents. Document classification learning device 1 includes a segmentation unit 10, a learning unit 11 that performs fine-tuning, and a memory unit 12 that stores the pre-trained language model.

[0035] The division unit 10 receives as input an entire sentence, which is the gist of a document used as learning data, and a corresponding label (a label assigned to the entire sentence) (step S301). Then, the division unit 10 divides the input sentence into sentence units according to a preset rule, and generates a plurality of divided sentences, each consisting of a predetermined number of sentences (step S302).

[0036] As a result, a plurality of divided sentences are generated from the entire sentence in units of a predetermined number of sentences according to a preset rule, and a plurality of new divided sentences are generated that are shorter in size than the input sentence (fewer characters than the input sentence).

[0037] The division unit 10 generates labeled training data for the segmented sentences (the segmented sentences and their corresponding labels) by assigning the same label as the input label to each of the generated segmented sentences (step S303).The division unit 10 then outputs the labeled training data for the segmented sentences to the learning unit 11 for each of the segmented sentences generated from the entire input sentence.

[0038] This allows multiple new labeled training data to be generated in accordance with pre-set rules, corresponding to one set of input sentence and label data (one labeled training data in conventional technology), thereby virtually increasing the number of labeled training data.

[0039] Generally, a sentence expresses a coherent emotion, thought, topic, etc., and may consist of one sentence or multiple sentences. The same is true for the segmented sentences in the embodiment of the present invention.

[0040] Figure 4 is a diagram explaining the gist of a document (a sentence consisting of five sentences) and the sentences after division. Figure 4(1) shows the gist of a document (a sentence consisting of five sentences). This sentence is assumed to be composed of "Sentence 1, Sentence 2, Sentence 3, Sentence 4, Sentence 5". Sentence 1 is the first sentence in the sentence consisting of five sentences, sentence 2 is the second sentence, sentence 3 is the third sentence, sentence 4 is the fourth sentence, and sentence 5 is the fifth sentence.

[0041] Figure 4(2) shows a post-division sentence consisting of one sentence obtained by dividing the sentence in Figure 4(1), i.e., a post-division sentence divided into one sentence. This post-division sentence consisting of one sentence consists of five sentences: "Sentence 1," "Sentence 2," "Sentence 3," "Sentence 4," and "Sentence 5." The preset rule in this case indicates that by dividing the sentence consisting of five sentences, five post-division sentences consisting of one sentence will be generated.

[0042] Figure 4(3) shows the divided sentence consisting of two sentences obtained by dividing the sentence in Figure 4(1), that is, the divided sentence consisting of two sentences. <bos> <sep>"Article 1", "Article 1" <sep>"Article 2", "Article 2" <sep>"Article 3", "Article 3" <sep>Sentence 4" and "Sentence 4 <sep>The predefined rule in this case indicates that a sentence consisting of five sentences should be divided into five divided sentences each consisting of two sentences.

[0043] still, <bos>is a special token that indicates the beginning of a sentence (sentence after division), <sep>is a special token that indicates the boundary of a sentence. <bos> <sep>The divided sentence of "Sentence 1" contains only one sentence, <bos>By treating the special tag as one sentence, the sentence after division is considered to consist of two sentences.

[0044] Figure 4(4) shows the divided sentence consisting of three sentences obtained by dividing the sentence in Figure 4(1), that is, the divided sentence consisting of three sentences. <bos> <sep>Article 1 <sep>"Text 2", "Text 1" <sep>Text 2 <sep>"Text 3", "Text 2" <sep>Text 3 <sep>"Text 4", "Text 3" <sep>Text 4 <sep>Sentence 5" and "Sentence 4 <sep>Text 5 <sep> <eos>", and the number is 5. In this case, the preset rule indicates that a sentence consisting of five sentences is to be divided into five divided sentences each consisting of three sentences.

[0045] still, <eos>is a special token that indicates the end of a sentence (sentence after division). <bos> <sep>Article 1 <sep>The divided sentences of "Sentence 2" and "Sentence 4" <sep>Text 5 <sep> <eos>" each contains only two sentences, <bos>and <eos>By considering each of the special tags as a sentence, these divided sentences are made up of three sentences.

[0046] 4 shows a case where a sentence consisting of five sentences is divided to generate a divided sentence consisting of one sentence, a divided sentence consisting of two sentences, and a divided sentence consisting of three sentences, but a divided sentence consisting of four sentences may also be generated. In this case, the number of divided sentences consisting of four sentences is also five.

[0047] 4, the number of generated segmented sentences consisting of one sentence, the number of generated segmented sentences consisting of two sentences, and the number of generated segmented sentences consisting of three sentences are set to 5, but they may be 4 or less. Also, different numbers of segmented sentences may be generated depending on the number of sentences contained in the generated segmented sentences. For example, the number of segmented sentences consisting of one sentence may be set to 5, the number of segmented sentences consisting of two sentences may be set to 4, and the number of segmented sentences consisting of three sentences may be set to 2.

[0048] Also, while FIG. 4 shows an example in which the gist of a document is a sentence consisting of five sentences, generalizing this, if the gist of a document is a sentence consisting of n sentences, post-segmentation sentences are generated from the sentence consisting of n sentences by dividing it into 1 sentence, 2 sentences, ..., and (n-1) sentences. n is an integer greater than or equal to 2. In this case, the maximum number of post-segmentation sentences for each of the 1-sentence division, 2-sentence division, ..., and (n-1) sentence division is n. Note that it is also possible to generate post-segmentation sentences with a number less than n for each of the 1-sentence division, 2-sentence division, ..., and (n-1) sentence division.

[0049] Figure 5 shows a specific example of a document abstract (a six-sentence text). As shown in Figure 5, this text is "In the last decade ··· research communities. In response to the COVID-19 ··· in a sub-optimal manner. In this article, ··· research output. We provide ··· the scientific process. We call for ··· even more rapidly. We provide ··· https: / / osf.io / renxy / ." and consists of six sentences.

[0050] FIG. 6 is a diagram showing a specific example of a divided sentence obtained by dividing the sentence shown in FIG. 5 into one sentence. This is a specific example of a divided sentence consisting of one sentence obtained by dividing the sentence shown in FIG.

[0051] As shown in Figure 6, the sentences after splitting into one sentence are "In the last decade ··· research communities.", "In response to the COVID-19 ··· in a sub-optimal manner.", "In this article, ··· research output.", "We provide ··· the scientific process.", "We call for ··· even more rapidly.", and "We provide ··· https: / / osf.io / renxy / .", which is six in total.

[0052] Figure 7 shows a specific example of a divided sentence that has been divided into three sentences. This is a specific example of a divided sentence consisting of three consecutive sentences that is obtained by dividing the sentence shown in Figure 5.

[0053] As shown in Figure 7, the sentence after division into three sentences is <bos> <sep>In the last decade ··· research communities. <sep>In response to the COVID-19 ··· in a sub-optimal manner.」、「In the last decade ··· research communities. <sep>In response to the COVID-19 ··· in a sub-optimal manner. <sep>In this article, ··· research output.」、···、「We provide ··· the scientific process. <sep>We call for ··· even more rapidly. <sep>We provide ··· https: / / osf.io / renxy / . and "We call for ··· even more rapidly. <sep>We provide ··· https: / / osf.io / renxy / . <sep> <eos>" and the number is 6.

[0054] Note that Fig. 7 shows a specific example of a segmented sentence consisting of three consecutive sentences, but the three sentences do not necessarily have to be consecutive, and a segmented sentence consisting of three sentences may be generated by any combination of three sentences in the original sentence consisting of six sentences shown in Fig. 5. This is also true for a segmented sentence consisting of two sentences, a segmented sentence consisting of four sentences, and a segmented sentence consisting of five sentences, and a segmented sentence may be generated by combining sentences in any position.

[0055] 2 and 3, the learning unit 11 receives labeled training data of the segmented sentences (the segmented sentences and their corresponding labels) for each of the multiple segmented sentences from the segmentation unit 10. Then, the learning unit 11 uses the labeled training data of the segmented sentences to train the language model stored in the storage unit 12 by fine tuning (step S304).

[0056] This allows the language model to be trained using divided sentences that are shorter in size than the input sentence.

[0057] Furthermore, for example, as shown in Figure 4(2), if the division unit 10 generates only multiple divided sentences consisting of one sentence, the learning unit 11 fine-tunes the language model using each of the labeled training data of the multiple divided sentences consisting of one sentence.

[0058] Furthermore, for example, as shown in Fig. 4(3), when the segmentation unit 10 generates only a plurality of segmented sentences each consisting of two sentences, the learning unit 11 fine-tunes the language model using each of the labeled training data of the plurality of segmented sentences each consisting of two sentences. The same applies to the segmented sentence consisting of three sentences shown in Fig. 4(4), and also to the segmented sentence consisting of four sentences, which is not shown in Fig. 4.

[0059] Furthermore, for example, as shown in Figures 4(2) and (3), when the division unit 10 generates multiple divided sentences consisting of one sentence and multiple divided sentences consisting of two sentences, the learning unit 11 fine-tunes the language model using labeled training data for the multiple divided sentences consisting of one sentence and labeled training data for the multiple divided sentences consisting of two sentences.

[0060] As described above, in the document classification learning device 1 according to an embodiment of the present invention, the segmentation unit 10 inputs an entire sentence and its corresponding labels, segments the sentence into sentences according to preset rules, and generates segmented sentences consisting of a predetermined number of sentences. The segmentation unit 10 then generates labeled training data for the segmented sentences by assigning the same labels as the input labels.

[0061] The learning unit 11 fine-tunes the pre-trained language model stored in the storage unit 12 using labeled training data of the segmented sentences.

[0062] This allows us to artificially increase the labeled training data using a set of input sentences and labels, and can resolve the shortage of training data for low-frequency labels when the number of labels for sentences is uneven.

[0063] Furthermore, because the language model can be trained using labeled training data of segmented sentences that are shorter than the input sentence, it is possible to avoid truncating the segmented sentences to be input to the language model. In other words, even if the entire input sentence exceeds 512 tokens and cannot be input to the language model, by segmenting it into segmented sentences of less than 512 tokens, all of the multiple segmented sentences corresponding to the entire input sentence can be input to the language model.

[0064] Therefore, the language model can be trained without a shortage of training data for low-frequency labels and without being affected by the limit on the number of tokens in a sentence input to the language model.The document classification device 2 shown in Figure 8, which will be described later, can improve the accuracy of document classification by using the language model trained by the document classification learning device 1.

[0065] [Document Classification Device] Next, a document classification device according to an embodiment of the present invention will be described. Fig. 8 is a block diagram showing an example of the configuration of a document classification device according to an embodiment of the present invention, and Fig. 9 is a flowchart showing an example of the processing performed by the device.

[0066] This document classification device 2 is a device that estimates labels of documents to be classified using a language model fine-tuned by the document classification learning device 1 shown in Figure 2. Document classification device 2 includes a segmentation unit 13, an estimation unit 14, a label integration unit 15, and a storage unit 16 in which the language model fine-tuned by the document classification learning device 1 is stored.

[0067] The division unit 13 receives an input of an entire sentence that is the gist of a document to be classified (step S901). Then, similar to the division unit 10 shown in Fig. 1, the division unit 13 divides the input sentence into sentence units according to a predetermined rule, and generates a plurality of divided sentences each consisting of a predetermined number of sentences (step S902). The division unit 13 outputs the plurality of divided sentences generated from the gist of the document to be classified to the estimation unit 14.

[0068] As a result, a plurality of divided sentences are generated from the entire sentence in units of a predetermined number of sentences according to a preset rule, and a plurality of new divided sentences are generated that are shorter in size than the input sentence (fewer characters than the input sentence).

[0069] The division unit 13 may divide the input text into a predetermined number of sentence units according to the same preset rules as those used by the division unit 10 of the document classification learning device 1 shown in Figure 2. The division unit 13 may also divide the input text into a predetermined number of sentence units according to rules different from the preset rules used by the division unit 10.

[0070] The estimation unit 14 receives a plurality of segmented sentences corresponding to the gist of the document to be classified from the segmentation unit 13. Then, for each of the plurality of segmented sentences, the estimation unit 14 estimates a probability value for each label corresponding to the segmented sentence using a language model stored in the storage unit 16 (step S903). Then, the estimation unit 14 outputs the probability value for each label for each of the plurality of segmented sentences to the label integration unit 15.

[0071] The label integrating unit 15 receives the probability value for each label for each of the plurality of segmented sentences from the dividing unit 13, and integrates the labels of the plurality of segmented sentences using these probability values ​​(step S904).Then, the label integrating unit 15 outputs the integrated label as the classification result of the document, that is, as the label of the document (step S905).

[0072] For example, the label integration unit 15 integrates labels by an average value integration method using the probability values ​​for each label for multiple segmented sentences. That is, the label integration unit 15 calculates the average value of the probability values ​​for each label from the probability values ​​for each label for multiple segmented sentences, identifies the label corresponding to the largest average value of the probability values ​​for each label, and integrates the identified label as the label of the document.

[0073] Specifically, for the first segmented sentence, the probability value of label A is P1A, the probability value of label B is P1B, and the probability value of label C is P1C, for the second segmented sentence, the probability value of label A is P2A, the probability value of label B is P2B, and the probability value of label C is P2C, and for the third segmented sentence, the probability value of label A is P3A, the probability value of label B is P3B, and the probability value of label C is P3C.

[0074] Then, the label integration unit 15 calculates the average values PA, PB, and PC of the probability values of labels A, B, and C from the probability values P1A, ···, P3C of labels A, B, and C for the first, second, and third post - split sentences by the following formula. [Equation 1] PA=(P1A+P2A+P3A) / 3 PB=(P1B+P2B+P3B) / 3 ···(1) PC=(P1C+P2C+P3C) / 3

[0075] Then, the label integration unit 15 identifies the label corresponding to the maximum average value among the average values PA, PB, and PC of the probability values of labels A, B, and C, and integrates the identified label as the label of the document. For example, when the average value PC of the probability value of label C is the maximum (when PA, PB < PC), label C is integrated as the label of the document.

[0076] In addition, the label integration unit 15 may integrate the labels by a majority - decision integration method using the probability values for each label of a plurality of post - split sentences. That is, for each of the plurality of post - split sentences, the label integration unit 15 identifies the label corresponding to the maximum probability value among the probability values for each label of the post - split sentence, and obtains the label of the document by a majority decision of the identified labels.

[0077] Specifically, let the probability value of label A for the first post - split sentence be P1A, the probability value of label B be P1B, and the probability value of label C be P1C. Let the probability value of label A for the second post - split sentence be P2A, the probability value of label B be P2B, and the probability value of label C be P2C. Also, let the probability value of label A for the third post - split sentence be P3A, the probability value of label B be P3B, and the probability value of label C be P3C. In this case, assume P1A, P1B < P1C, P2A > P2B, P2C, and P3A, P3B < P3C.

[0078] Then, for the first segmented sentence, the label integrating unit 15 identifies a label C corresponding to the largest probability value P1C among the probability values ​​P1A, P1B, and P1C of the labels A, B, and C. Furthermore, for the second segmented sentence, the label integrating unit 15 identifies a label A corresponding to the largest probability value P2A among the probability values ​​P2A, P2B, and P2C of the labels A, B, and C. Furthermore, for the third segmented sentence, the label integrating unit 15 identifies a label C corresponding to the largest probability value P3C among the probability values ​​P3A, P3B, and P3C of the labels A, B, and C.

[0079] Then, the label integration unit 15 integrates the label C as the label of the document by majority vote of the labels C, A, and C identified for the first, second, and third divided sentences.

[0080] Here, there may be cases where the label integrating unit 15 is unable to integrate into one label even by this majority vote integration method. Therefore, when multiple labels are integrated by the majority vote integration method, the label integrating unit 15 integrates into one label by the above-mentioned average value integration method.

[0081] However, there may be cases where the label integration unit 15 is unable to integrate the labels into one label even by the above-described average value integration method. Therefore, the label integration unit 15 identifies the label corresponding to the maximum (highest) probability value among the multiple labels obtained by the average value integration method, and integrates the identified label as the label of the document.

[0082] In this case, the label integrating unit 15 may not be able to integrate the labels into one label using the integration method with the highest value. Therefore, the label integrating unit 15 uses a random number or the like to identify one label from among the multiple labels obtained using the integration method with the highest value, and integrates the identified label as the label of the document.

[0083] As a result, labels are estimated for the multiple segmented sentences generated from the entire sentence, and the label of the document is integrated from the labels of the multiple segmented sentences.

[0084] As described above, according to the document classification device 2 of the embodiment of the present invention, the division unit 13 inputs the entire sentence that is the gist of the document to be classified, divides the sentence into sentence units according to preset rules, and generates multiple divided sentences.

[0085] The estimation unit 14 uses the language model stored in the storage unit 16 to estimate a probability value for each label corresponding to each of the plurality of segmented sentences corresponding to the gist of the document to be classified.

[0086] The label integration unit 15 integrates the labels of the plurality of divided sentences into a document label using the probability value for each label for each of the plurality of divided sentences, and outputs the document label.

[0087] The language model stored in the storage unit 16 is a model that has been trained so that there is no shortage of training data for low-frequency labels and that it is not affected by the limit on the number of tokens in a sentence that can be input to the language model. Therefore, by using such a language model, it is possible to improve the accuracy of document classification.

[0088] [Experimental results] Next, we will explain the experimental results. The documents used were from the CORD-19 (The Covid-19 Open Research Dataset) database, a resource of scientific papers related to the novel coronavirus disease (COVID-19), and the pre-trained language models used were BERT and PubMedBERT.

[0089] Figure 10 shows an example of the number of documents (number of papers) per label in CORD-19 papers, for a total of 4,483 papers. The horizontal axis shows the types of labels (25 types) that are paper categories, and the vertical axis shows the number of papers corresponding to each label. Figure 10 shows that there is a bias in the distribution of the number of papers for each label.

[0090] Of the 4,483 papers with the distribution shown in Figure 10, 80% were used as training data, 6% as validation data, and 14% as test data. Note that the abstracts of each paper are generally 512 tokens or less.

[0091] Figure 11 shows experimental results for the conventional technology and an embodiment of the present invention. Figure 11(1) shows experimental results when a BERT language model is used in the conventional document classification learning device 101 and document classification device 102 shown in Figure 13. Figure 11(2) shows experimental results when a PubMedBERT language model is used in the conventional document classification learning device 101 and document classification device 102 shown in Figure 13.

[0092] Figure 11(3) shows experimental results when the PubMedBERT language model is used and the one-sentence segmentation method and the average value integration method are used in the document classification learning device 1 according to the embodiment of the present invention shown in Figure 2 and the document classification device 2 according to the embodiment of the present invention shown in Figure 8. Figure 11(4) shows experimental results when the three-sentence segmentation method is used instead of the one-sentence segmentation method in Figure 11(3).

[0093] Figure 11(5) shows the experimental results when the PubMedBERT language model is used in the document classification learning device 1 according to the embodiment of the present invention shown in Figure 2 and the document classification device 2 according to the embodiment of the present invention shown in Figure 8, and when a one-sentence division method and a majority voting integration method are used. Figure 11(6) shows the experimental results when a three-sentence division method is used instead of the one-sentence division method in Figure 11(5).

[0094] 11(1) to 11(6) show that the embodiment of the present invention outperforms the conventional technique in terms of Macro-f1. In other words, the embodiment of the present invention can improve the document classification accuracy compared to the conventional technique.

[0095] 11(3) to 11(6) show that in the embodiment of the present invention, the segmentation method of dividing a sentence into three sentences is superior to the segmentation method of dividing a sentence into one sentence in terms of Macro-f1. In other words, in the embodiment of the present invention, the segmentation method of dividing a sentence into three sentences can improve the accuracy of document classification more than the segmentation method of dividing a sentence into one sentence.

[0096] In this way, it is desirable that the segmentation unit 10 of the document classification learning device 1 and the segmentation unit 13 of the document classification device 2 segment the original sentence to generate segmented sentences so that the size of the segmented sentences does not exceed 512 tokens. In particular, it is even more desirable that the segmentation units 10 and 13 segment the original sentence to generate segmented sentences so that the size of the segmented sentences is close to 512 tokens.

[0097] Figure 12 is a diagram explaining a specific example of the experimental results, showing the experimental results when one paper with the document abstract (a six-sentence text) shown in Figure 5 was used out of 4,483 papers with the distribution shown in Figure 10. The true label in this case is label id=15 (scientific communication and education). In other words, the document abstract shown in Figure 5 should be classified as label id=15.

[0098] In the case of Figure 11(2), that is, when the PubMedBERT language model is used in the conventional document classification learning device 101 and document classification device 102 shown in Figure 13, the label into which the document shown in Figure 5 was classified by the document classification device 102 was label id=3 (genomics), and an incorrect result was obtained.

[0099] In contrast, in the case of Figure 11(5), that is, in the document classification learning device 1 according to the embodiment of the present invention shown in Figure 2 and the document classification device 2 according to the embodiment of the present invention shown in Figure 8, when the PubMedBERT language model is used and the one-sentence division method and the majority voting integration method are used, the label with which the document shown in Figure 5 was classified by the document classification device 2 is label id=15, and the correct result is obtained.

[0100] More specifically, the document classification device 2 estimated label id=15 for the segmented sentence in Figure 12(1), estimated label id=15 for each of the segmented sentences in Figures 12(2) to (5), and estimated label id=2 for the segmented sentence in Figure 12(6).These labels were then integrated using a majority vote integration method, and the label classified by the document classification device 2 was label id=15, which was a correct result.

[0101] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0102] A normal computer can be used as the hardware configuration of document classification learning device 1 and document classification device 2 according to the embodiments of the present invention. Each of document classification learning device 1 and document classification device 2 is configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0103] The functions of the segmentation unit 10, learning unit 11, and memory unit 12 provided in the document classification learning device 1 are each realized by having a CPU execute a program describing these functions. Similarly, the functions of the segmentation unit 13, estimation unit 14, label integration unit 15, and memory unit 16 provided in the document classification device 2 are each realized by having a CPU execute a program describing these functions.

[0104] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk), an optical disk (e.g., a CD-ROM, a DVD), or a semiconductor memory and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0105] 1,101 Learning devices for document classification 2,102 Document Classification Device 10,13 Split part 11,110 Learning Department 12,16,111,113 Storage section 14,112 Estimation part 15 Label Integration Unit< / eos> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / bos> < / eos> < / bos> < / eos> < / sep> < / sep> < / sep> < / sep> < / bos> < / eos> < / eos> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / bos> < / bos> < / sep> < / bos> < / sep> < / bos> < / sep> < / sep> < / sep> < / sep> < / sep> < / bos>

Claims

1. 1. A document classification learning device that fine-tunes a pre-trained language model for classifying documents based on sentences contained in the documents and labels assigned to the documents, comprising: a first division unit that receives the sentence and the label, divides the sentence into sentence units, and generates a plurality of divided sentences each consisting of a predetermined number of sentences; a learning unit that fine-tunes the language model using labeled training data that includes the segmented sentences generated by the first segmentation unit and the labels input by the first segmentation unit; A learning device for document classification comprising:

2. 2. The document classification learning device according to claim 1, Assuming that the text is composed of n sentences (n is an integer of 2 or more), The first dividing portion is The document classification learning device generates n divided sentences, the divided sentences consisting of a predetermined number of sentences between 1 and (n-1).

3. 3. A document classification device that estimates labels of documents to be classified using a language model fine-tuned by the document classification learning device according to claim 1, a second division unit that inputs a sentence included in the document, divides the sentence into sentence units, and generates a plurality of divided sentences each consisting of a predetermined number of sentences; an estimation unit that estimates a probability value for each of the labels of the segmented sentences generated by the second segmentation unit using the language model; a label integration unit that obtains the label of the document by integrating the labels using the probability value for each of the labels for each of the plurality of segmented sentences estimated by the estimation unit; A document classification device comprising:

4. 4. The document classification device according to claim 3, Assuming that the text is composed of n sentences (n is an integer of 2 or more), The second division portion is The document classification learning device generates n divided sentences, the divided sentences consisting of a predetermined number of sentences between 1 and (n-1).

5. a computer constituting a learning device for document classification that fine-tunes a pre-trained language model for classifying documents based on sentences contained in the documents and labels assigned to the documents; a first division unit that receives the sentence and the label, divides the sentence into sentence units, and generates a plurality of divided sentences each consisting of a predetermined number of sentences; and A program for functioning as a learning unit that fine-tunes the language model using labeled training data consisting of the segmented sentences generated by the first segmentation unit and the labels input by the first segmentation unit.

6. A computer constituting a document classification device that estimates labels of documents to be classified using a language model fine-tuned by the document classification learning device according to claim 1 or 2, a second division unit that inputs a sentence included in the document, divides the sentence into sentence units, and generates a plurality of divided sentences each consisting of a predetermined number of sentences; an estimation unit that estimates a probability value for each of the labels of the segmented sentences generated by the second segmentation unit using the language model; and A program for functioning as a label integration unit that determines the label of the document by integrating the labels using the probability value for each label for each of the multiple divided sentences estimated by the estimation unit.

Images